Chunk → embed → index, end to end.

txtfetch handles extraction; the loader hands the rest of the pipeline documents your framework already knows how to chunk, embed, and index.

langchain-python

Load with TxtfetchLoader, split with a LangChain text splitter, embed and index with any LangChain-supported embedding model and vector store, then query it. This extends the tested examples/build_pipeline.py from the LangChain Python quickstart — only the embed/index/query steps are new.

LangChain (Python)

from langchain_core.vectorstores import InMemoryVectorStore
from langchain_openai import OpenAIEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter

from langchain_txtfetch import TxtfetchLoader

# 1. Load — one Document per file/URL, extracted via txtfetch
loader = TxtfetchLoader(
    files=["whitepaper.pdf"],
    urls=["https://example.com/spec.docx"],
)
documents = loader.load()

# 2. Chunk
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)
chunks = splitter.split_documents(documents)

# 3. Embed + index
vector_store = InMemoryVectorStore(OpenAIEmbeddings())
vector_store.add_documents(chunks)

# 4. Query
results = vector_store.similarity_search("What does the whitepaper conclude?", k=4)
for doc in results:
    print(doc.metadata["source"], "->", doc.page_content[:80])

llamaindex-python

LlamaIndex folds chunking, embedding, and indexing into a single call — VectorStoreIndex.from_documents(...) — once TxtfetchReader has loaded the source documents.

LlamaIndex (Python)

from llama_index.core import VectorStoreIndex
from llama_index.readers.txtfetch import TxtfetchReader

reader = TxtfetchReader()
documents = reader.load_data(files=["whitepaper.pdf"], urls=["https://example.com/spec.docx"])

# Chunking, embedding, and indexing all happen inside from_documents(...)
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
print(query_engine.query("What does the whitepaper conclude?"))

notes

Both paths accept files and URLs in the same loader call — mix a local upload with a hosted spec sheet in one pipeline. Large sources are handled the same way the API handles them directly: the loader's underlying SDK call transparently waits out the async job path, so load()/load_data() still return finished Documents either way.

Stop parsing. Start shipping.

Create an account and get an API key in minutes — the free Hobby plan needs no card.

Get started →