https://txtfetch.com/docs/recipe/
Chunk → embed → index, end to end.
txtfetch handles extraction; the loader hands the rest of the pipeline documents your framework already knows how to chunk, embed, and index.
langchain-python
Load with TxtfetchLoader, split with a LangChain text splitter, embed and index with any LangChain-supported embedding model and vector store, then query it. See extract text into a vector store for the same pipeline in plain Python, plus the constraints of five named stores. This extends the tested examples/build_pipeline.py from the LangChain Python quickstart. Only the embed/index/query steps are new.
from langchain_core.vectorstores import InMemoryVectorStore
from langchain_openai import OpenAIEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_txtfetch import TxtfetchLoader
# 1. Load — one Document per file/URL, extracted via txtfetch
loader = TxtfetchLoader(
files=["whitepaper.pdf"],
urls=["https://example.com/spec.docx"],
)
documents = loader.load()
# 2. Chunk
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)
chunks = splitter.split_documents(documents)
# 3. Embed + index
vector_store = InMemoryVectorStore(OpenAIEmbeddings())
vector_store.add_documents(chunks)
# 4. Query
results = vector_store.similarity_search("What does the whitepaper conclude?", k=4)
for doc in results:
print(doc.metadata["source"], "->", doc.page_content[:80])llamaindex-python
LlamaIndex folds chunking, embedding, and indexing into a single call: VectorStoreIndex.from_documents(...). This runs once TxtfetchReader has loaded the source documents.
from llama_index.core import VectorStoreIndex
from llama_index.readers.txtfetch import TxtfetchReader
reader = TxtfetchReader()
documents = reader.load_data(files=["whitepaper.pdf"], urls=["https://example.com/spec.docx"])
# Chunking, embedding, and indexing all happen inside from_documents(...)
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
print(query_engine.query("What does the whitepaper conclude?"))notes
Both paths accept files and URLs in the same loader call. Mix a local upload with a hosted spec sheet in one pipeline. Large sources are handled the same way the API handles them directly. The loader's underlying SDK call transparently waits out the async job path. So load()/load_data() still return finished Documents either way.
Before picking a chunk size and overlap for the splitter step above, try the chunk previewer. Paste extracted text into it and see the resulting chunk boundaries and token estimates. It also flags extraction-damage signals that work against your chunks. Everything runs in your browser, before the text ever reaches your embedding model.
Watch it run, then take a key.
The playground replays a real recorded response for every sample document.
Open the playground →