Using txtfetch as a LangChain and LlamaIndex document loader

LangChain and LlamaIndex both ship a long list of built-in document loaders, covering PDFs, S3 buckets, Notion, and a dozen other sources. Each one wraps some underlying parsing library. txtfetch doesn’t have an official loader in either ecosystem yet. What it does have is a plain HTTP endpoint. That’s enough to write a thin custom loader yourself in about fifteen lines, without waiting on us to ship one.

Why write your own instead of waiting

A custom loader that calls /v1/extract gives you the same one-endpoint handling as the rest of txtfetch’s guides: PDFs, Office docs, scanned images, and HTML. That covers multi-column PDFs, DOCX/PPTX/XLSX, and OCR fallback. You don’t pin a parsing library in your requirements.txt for each format. The loader itself is a thin adapter. It calls the endpoint, reads extracted_text out of the JSON response, and hands it to the framework’s document object. There’s no hidden complexity to wait for an official package to solve.

A LangChain loader

LangChain’s BaseLoader interface just needs a load() method that returns a list of Document objects:

import requests
from langchain_core.document_loaders import BaseLoader
from langchain_core.documents import Document

class TxtfetchLoader(BaseLoader):
    def __init__(self, file_path: str, api_key: str):
        self.file_path = file_path
        self.api_key = api_key

    def load(self) -> list[Document]:
        with open(self.file_path, "rb") as f:
            resp = requests.post(
                "https://api.txtfetch.com/v1/extract",
                headers={"Authorization": f"Bearer {self.api_key}"},
                files={"file": f},
            )
        resp.raise_for_status()
        body = resp.json()
        if body["status"] != "success":
            raise ValueError(body.get("error", "extraction failed"))
        return [Document(
            page_content=body["extracted_text"],
            metadata={"source": self.file_path},
        )]

Use it exactly like any built-in loader:

loader = TxtfetchLoader("quarterly-report.pdf", api_key=TXTFETCH_KEY)
docs = loader.load()
# feed docs into your text splitter, then your vector store, as usual

A LlamaIndex reader

LlamaIndex’s equivalent is a BaseReader with a load_data() method:

import requests
from llama_index.core.readers.base import BaseReader
from llama_index.core.schema import Document

class TxtfetchReader(BaseReader):
    def __init__(self, api_key: str):
        self.api_key = api_key

    def load_data(self, file_path: str) -> list[Document]:
        with open(file_path, "rb") as f:
            resp = requests.post(
                "https://api.txtfetch.com/v1/extract",
                headers={"Authorization": f"Bearer {self.api_key}"},
                files={"file": f},
            )
        resp.raise_for_status()
        body = resp.json()
        if body["status"] != "success":
            raise ValueError(body.get("error", "extraction failed"))
        return [Document(text=body["extracted_text"], metadata={"source": file_path})]

Both loaders are deliberately minimal: a single request, a status check, and a document object. There’s no batching, retry, or concurrency built in. For ingesting more than a handful of files at once, see the batch ingestion guide. It covers the concurrency and retry pattern to wrap around this same call. Full endpoint details (auth, request shape, response shape) are in the docs and the PDF extraction reference.

Feeding the URL variant instead of local files

If your documents already live somewhere reachable over HTTP (an S3 presigned URL, a CMS export), skip the local file and pass ?url=:

resp = requests.post(
    "https://api.txtfetch.com/v1/extract",
    headers={"Authorization": f"Bearer {api_key}"},
    params={"url": document_url},
)

Same response shape, one fewer download step in your loader.

What this is and isn’t

This is a self-written adapter around a plain HTTP API. It’s not an official txtfetch package on PyPI or npm. It’s also not a LangChain- or LlamaIndex-maintained integration. If you’d rather not maintain even fifteen lines of loader code, an official SDK is on our roadmap. Today, the pattern above is the fastest path to plugging txtfetch into either framework. The loader itself doesn’t change no matter what document type it’s parsing. Only the file you point it at does. See the PDF extraction guide for the reasoning behind /v1/extract’s single-endpoint design. Once you have documents loaded, chunking strategy is the next decision to make before embedding.

Building this into a production pipeline and want a hand? Get in touch and we’ll set you up with an API key.