search⌘KGet started

Extract text into Elasticsearch

A search engine with a built-in Tika step that stops reading at 100,000 characters by default. Here's what changes between the text txtfetch returns and the first row you write to Elasticsearch.

The problem

Elasticsearch can extract text itself. Its attachment processor decodes a base64 file and runs Apache Tika inside your cluster. txtfetch runs the same library, outside it. But the processor stops at 100,000 characters by default, and its docs name no OCR step. Send txtfetch's text instead, and you don't need the processor at all.

what elasticsearch needs

Constraints that shape splitting and the write step, cited and dated
Where long text stops counting100,000 characters by default, set by the attachment processor's indexed_chars option. The processor drops the rest of the text. Set indexed_chars to -1 to remove the cap. Or skip the processor and index txtfetch's text as a plain field.
Max request size100 MB per HTTP request by default (http.max_content_length).
Built-in document extractionYes. The attachment processor runs Apache Tika inside the cluster. Its docs name no OCR step.
Keeps the full textYes, in _source by default. If you use the attachment processor, set remove_binary: true. Otherwise the base64 file stays in the index next to its text.
Document ID formatAny UTF-8 string up to 512 bytes. Elasticsearch generates one automatically if you omit it.
Bulk write APIThe Bulk API (_bulk), one JSON action line per document.

No accuracy, speed, or recall figure is measured against Elasticsearch on this page. See /benchmarks for txtfetch's own measured extraction numbers.

The extraction step it already has

Elasticsearch already runs Tika. That is a fair reason to skip txtfetch, so here is what it costs. The attachment processor decodes a base64 file, passes it to Tika, and writes the text to a field. That work runs on your ingest nodes and takes CPU from indexing. The encoded file also rides inside the index request. The processor stops at 100,000 characters by default. Its docs name no OCR step for scanned pages. txtfetch runs Tika and Tesseract outside your cluster, behind one HTTP call.

Splitting for Elasticsearch

Index one document per page or section. Give each one a parent_id field that points back to the source file. Keep each text field under 1,000,000 characters too. Past that point highlighting fails, because index.highlight.max_analyzed_offset stops there by default. To find real section breaks, use the page numbers in format=json output. Or test your text in the chunk previewer at /tools/chunk-preview.

the pipeline

Extract, split, and bulk-write. No framework, just the txtfetch Python SDK and Elasticsearch's own client.

Python
import os

from elasticsearch import Elasticsearch, helpers
from txtfetch import Txtfetch

txtfetch = Txtfetch(api_key=os.environ["TXTFETCH_KEY"])
es = Elasticsearch(os.environ["ELASTICSEARCH_URL"], api_key=os.environ["ELASTICSEARCH_API_KEY"])


def split_text(text: str, size: int = 2000, overlap: int = 200) -> list[str]:
    sections = []
    start = 0
    while start < len(text):
        end = start + size
        sections.append(text[start:end])
        start = end - overlap
    return sections


# 1. Extract
result = txtfetch.extract(file="whitepaper.pdf")

# 2. Split
sections = split_text(result.extracted_text)

# 3. Bulk write — indexed_chars only limits Elasticsearch's own attachment
# processor. Since txtfetch already extracted the text, index it as a plain
# field and skip that processor entirely.
actions = [
    {
        "_index": "docs",
        "_id": f"whitepaper-{i}",
        "_source": {"parent_id": "whitepaper.pdf", "section": i, "text": section},
    }
    for i, section in enumerate(sections)
]
helpers.bulk(es, actions)

Every call above was resolved against elasticsearch 9.5.1, in September 2026. Client APIs move, so check the vendor's own docs for the version you pin.

large batches

A large file, or a slow OCR pass, routes to an async job automatically. That returns a 202 plus a job_id to poll, and ?async=true forces that path for any request. Pass webhook_url instead, and txtfetch calls you back when the text is ready, rather than you polling for it. See async jobs & webhooks for the full lifecycle.

Submit (async)
curl -X POST "https://api.txtfetch.com/v1/extract?url=https://example.com/whitepaper.pdf&async=true" \
  -H "Authorization: Bearer $TXTFETCH_KEY"

# {"status": "processing", "job_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6"}
Poll
curl "https://api.txtfetch.com/v1/extract/3fa85f64-5717-4562-b3fc-2c963f66afa6" \
  -H "Authorization: Bearer $TXTFETCH_KEY"

# {"status": "processing", "job_id": "..."} while running, then the same
# {"status": "success", "extracted_text": "...", "metadata": {...}} shape
# POST /v1/extract would have returned synchronously.
curl
curl -X POST "https://api.txtfetch.com/v1/extract?url=https://example.com/whitepaper.pdf" \
  -H "Authorization: Bearer $TXTFETCH_KEY" \
  --data-urlencode "webhook_url=https://example.com/webhooks/txtfetch"

A retried request, from a client timeout or a dropped response, shouldn't extract and write the same document twice. Send the same Idempotency-Key on a retry and txtfetch returns the first result instead of running the job again. See idempotency for the full guarantee.

curl
curl -X POST https://api.txtfetch.com/v1/extract \
  -H "Authorization: Bearer $TXTFETCH_KEY" \
  -H "Idempotency-Key: 3f29b6e4-9c1a-4b8e-9c2a-1e6f0a2d5b3c" \
  -F file=@contract.pdf
JavaScript
// The SDK auto-generates and reuses an Idempotency-Key across its own
// retry chain. Pass your own to control it explicitly:
await txtfetch.extract({ file: "./contract.pdf", idempotencyKey: "3f29b6e4-..." });
Python
# The SDK auto-generates and reuses an Idempotency-Key across its own
# retry chain. Pass your own to control it explicitly:
client.extract(file="contract.pdf", idempotency_key="3f29b6e4-...")

txtfetch ships no connector, plugin, or client for Elasticsearch. The pipeline above is the whole integration. Extract with txtfetch. Then write the result with Elasticsearch's own client.

frequently asked questions

Does txtfetch write directly to Elasticsearch?
No. txtfetch returns the extracted text over one HTTP call. You split the text and send the bulk request with Elasticsearch's own client.
Why not send the file to Elasticsearch's attachment processor?
You can. It runs Tika on your own ingest nodes and stops at 100,000 characters by default. Its docs name no OCR step. txtfetch runs Tika and Tesseract outside the cluster and returns the full text.
Does txtfetch benchmark accuracy or speed against Elasticsearch?
No. Elasticsearch is a search engine, not a text-extraction service. There is no accuracy or speed comparison to make. See /benchmarks for txtfetch's own extraction numbers.

sources

Related

other engines

Wire the extraction step in.

The RAG recipe shows the whole path, from a file to the Elasticsearch upsert.

Read the RAG recipe →

Get an API key →