search⌘KGet started

Extract text into Typesense

An open-source search engine that keeps every indexed field in RAM and publishes no per-document size cap. Here's what changes between the text txtfetch returns and the first row you write to Typesense.

The problem

Typesense holds every indexed field in memory. It publishes no size limit for one document, so a large document is not rejected. It costs RAM instead. Every extra character stays in memory for as long as the collection exists.

what typesense needs

Constraints that shape splitting and the write step, cited and dated
Where long text stops countingNo per-document cap is documented. The documented limit is 10 GB per bulk import request. Typesense rejects a larger request. It does not truncate it.
Max request size10 GB per bulk import request (JSONL over HTTP).
Built-in document extractionNone. Typesense indexes the JSON documents you send. It does not extract text from files.
Keeps the full textYes. Typesense stores every field. Set index: false on a field to keep it on disk and out of memory.
Document ID formatA string. Typesense makes one for you if a document has no id field.
Bulk write APIdocuments.import_(...). It takes JSONL and reports success or an error for each line.

No accuracy, speed, or recall figure is measured against Typesense on this page. See /benchmarks for txtfetch's own measured extraction numbers.

Splitting for Typesense

Index one document per page or section, each with a parent_id field that points back to the source file. Keep sections small. Every indexed field stays in RAM, so smaller sections cost less memory. To find real section breaks, use the page numbers in format=json output. Or test your text in the chunk previewer at /tools/chunk-preview.

the pipeline

Extract, split, and bulk-write. No framework, just the txtfetch Python SDK and Typesense's own client.

Python
import os

import typesense
from txtfetch import Txtfetch

txtfetch = Txtfetch(api_key=os.environ["TXTFETCH_KEY"])
client = typesense.Client(
    {
        "nodes": [{"host": os.environ["TYPESENSE_HOST"], "port": "443", "protocol": "https"}],
        "api_key": os.environ["TYPESENSE_API_KEY"],
        "connection_timeout_seconds": 5,
    }
)

# collections.create raises if the collection is already there, so a second
# run of this script needs the guard.
try:
    client.collections.create(
        {
            "name": "docs",
            "fields": [
                {"name": "parent_id", "type": "string"},
                {"name": "section", "type": "int32"},
                {"name": "text", "type": "string"},
            ],
        }
    )
except typesense.exceptions.ObjectAlreadyExists:
    pass


def split_text(text: str, size: int = 2000, overlap: int = 200) -> list[str]:
    sections = []
    start = 0
    while start < len(text):
        end = start + size
        sections.append(text[start:end])
        start = end - overlap
    return sections


# 1. Extract
result = txtfetch.extract(file="whitepaper.pdf")

# 2. Split
sections = split_text(result.extracted_text)

# 3. Bulk write — import_() takes JSONL, one document per line.
documents = [
    {"id": f"whitepaper-{i}", "parent_id": "whitepaper.pdf", "section": i, "text": section}
    for i, section in enumerate(sections)
]
client.collections["docs"].documents.import_(documents)

Every call above was resolved against typesense 2.0.0, in September 2026. Client APIs move, so check the vendor's own docs for the version you pin.

large batches

A large file, or a slow OCR pass, routes to an async job automatically. That returns a 202 plus a job_id to poll, and ?async=true forces that path for any request. Pass webhook_url instead, and txtfetch calls you back when the text is ready, rather than you polling for it. See async jobs & webhooks for the full lifecycle.

Submit (async)
curl -X POST "https://api.txtfetch.com/v1/extract?url=https://example.com/whitepaper.pdf&async=true" \
  -H "Authorization: Bearer $TXTFETCH_KEY"

# {"status": "processing", "job_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6"}
Poll
curl "https://api.txtfetch.com/v1/extract/3fa85f64-5717-4562-b3fc-2c963f66afa6" \
  -H "Authorization: Bearer $TXTFETCH_KEY"

# {"status": "processing", "job_id": "..."} while running, then the same
# {"status": "success", "extracted_text": "...", "metadata": {...}} shape
# POST /v1/extract would have returned synchronously.
curl
curl -X POST "https://api.txtfetch.com/v1/extract?url=https://example.com/whitepaper.pdf" \
  -H "Authorization: Bearer $TXTFETCH_KEY" \
  --data-urlencode "webhook_url=https://example.com/webhooks/txtfetch"

A retried request, from a client timeout or a dropped response, shouldn't extract and write the same document twice. Send the same Idempotency-Key on a retry and txtfetch returns the first result instead of running the job again. See idempotency for the full guarantee.

curl
curl -X POST https://api.txtfetch.com/v1/extract \
  -H "Authorization: Bearer $TXTFETCH_KEY" \
  -H "Idempotency-Key: 3f29b6e4-9c1a-4b8e-9c2a-1e6f0a2d5b3c" \
  -F file=@contract.pdf
JavaScript
// The SDK auto-generates and reuses an Idempotency-Key across its own
// retry chain. Pass your own to control it explicitly:
await txtfetch.extract({ file: "./contract.pdf", idempotencyKey: "3f29b6e4-..." });
Python
# The SDK auto-generates and reuses an Idempotency-Key across its own
# retry chain. Pass your own to control it explicitly:
client.extract(file="contract.pdf", idempotency_key="3f29b6e4-...")

txtfetch ships no connector, plugin, or client for Typesense. The pipeline above is the whole integration. Extract with txtfetch. Then write the result with Typesense's own client.

frequently asked questions

Does txtfetch write directly to Typesense?
No. txtfetch returns the extracted text over one HTTP call. You define the collection, split the text, and run the import with Typesense's own client.
Is there really no size limit on one Typesense document?
None is published. The documented limit is 10 GB per bulk import request. Every indexed field lives in RAM, so a large document costs memory instead of failing. Your limit is the RAM your cluster has.
Does txtfetch benchmark accuracy or speed against Typesense?
No. Typesense is a search engine, not a text-extraction service. There is no accuracy or speed comparison to make. See /benchmarks for txtfetch's own extraction numbers.

sources

Related

other engines

Wire the extraction step in.

The RAG recipe shows the whole path, from a file to the Typesense upsert.

Read the RAG recipe →

Get an API key →