Using txtfetch as a LangChain and LlamaIndex document loader

LangChain and LlamaIndex both ship a long list of built-in document loaders — for PDFs, for S3 buckets, for Notion, for a dozen other sources — each wrapping some underlying parsing library. txtfetch doesn’t have an official loader in either ecosystem yet. What it does have is a plain HTTP endpoint, which is enough to write a thin custom loader yourself in about fifteen lines, without waiting on us to ship one.

Why write your own instead of waiting

A custom loader that calls /v1/extract gives you the same one-endpoint handling of PDFs, Office docs, scanned images, and HTML that the rest of txtfetch’s guides cover — multi-column PDFs, DOCX/PPTX/XLSX, OCR fallback — without a parsing library pinned in your requirements.txt for each format. The loader itself is a thin adapter: it calls the endpoint, reads extracted_text out of the JSON response, and hands it to the framework’s document object. There’s no hidden complexity to wait for an official package to solve.

A LangChain loader

LangChain’s BaseLoader interface just needs a load() method that returns a list of Document objects:

import requests
from langchain_core.document_loaders import BaseLoader
from langchain_core.documents import Document

class TxtfetchLoader(BaseLoader):
    def __init__(self, file_path: str, api_key: str):
        self.file_path = file_path
        self.api_key = api_key

    def load(self) -> list[Document]:
        with open(self.file_path, "rb") as f:
            resp = requests.post(
                "https://api.txtfetch.com/v1/extract",
                headers={"Authorization": f"Bearer {self.api_key}"},
                files={"file": f},
            )
        resp.raise_for_status()
        body = resp.json()
        if body["status"] != "success":
            raise ValueError(body.get("error", "extraction failed"))
        return [Document(
            page_content=body["extracted_text"],
            metadata={"source": self.file_path},
        )]

Use it exactly like any built-in loader:

loader = TxtfetchLoader("quarterly-report.pdf", api_key=TXTFETCH_KEY)
docs = loader.load()
# feed docs into your text splitter, then your vector store, as usual

A LlamaIndex reader

LlamaIndex’s equivalent is a BaseReader with a load_data() method:

import requests
from llama_index.core.readers.base import BaseReader
from llama_index.core.schema import Document

class TxtfetchReader(BaseReader):
    def __init__(self, api_key: str):
        self.api_key = api_key

    def load_data(self, file_path: str) -> list[Document]:
        with open(file_path, "rb") as f:
            resp = requests.post(
                "https://api.txtfetch.com/v1/extract",
                headers={"Authorization": f"Bearer {self.api_key}"},
                files={"file": f},
            )
        resp.raise_for_status()
        body = resp.json()
        if body["status"] != "success":
            raise ValueError(body.get("error", "extraction failed"))
        return [Document(text=body["extracted_text"], metadata={"source": file_path})]

Both loaders are deliberately minimal — a single request, a status check, and a document object. There’s no batching, retry, or concurrency built in; for ingesting more than a handful of files at once, see the batch ingestion guide for the concurrency and retry pattern to wrap around this same call. Full endpoint details — auth, request shape, response shape — are in the docs and the PDF extraction reference.

Feeding the URL variant instead of local files

If your documents live somewhere already reachable over HTTP — an S3 presigned URL, a CMS export — skip the local file entirely and pass ?url=:

resp = requests.post(
    "https://api.txtfetch.com/v1/extract",
    headers={"Authorization": f"Bearer {api_key}"},
    params={"url": document_url},
)

Same response shape, one fewer download step in your loader.

What this is and isn’t

This is a self-written adapter around a plain HTTP API — not an official txtfetch package on PyPI or npm, and not a LangChain- or LlamaIndex- maintained integration. If you’d rather not maintain even fifteen lines of loader code, an official SDK is on our roadmap; today, the pattern above is the fastest path to plugging txtfetch into either framework. Whatever document type it’s parsing — see the PDF extraction guide for the reasoning behind /v1/extract’s single-endpoint design — the loader itself doesn’t change; only the file you point it at does. Once you have documents loaded, chunking strategy is the next decision to make before embedding.

Building this into a production pipeline and want a hand? Get in touch and we’ll set you up with an API key.