search⌘KGet started

Extract text into OpenSearch

A fork of Elasticsearch whose Tika plugin installs separately and stops at 100,000 characters by default. Here's what changes between the text txtfetch returns and the first row you write to OpenSearch.

The problem

OpenSearch keeps the attachment idea as a separate plugin, ingest-attachment. That is one more install to run and patch. The plugin runs Tika and stops at 100,000 characters by default. Its docs name no OCR step. txtfetch needs no plugin at all.

what opensearch needs

Constraints that shape splitting and the write step, cited and dated
Where long text stops counting100,000 characters by default, set by the plugin's indexed_chars option. The plugin drops the rest of the text. Set indexed_chars to -1 to remove the cap. Or skip the plugin and index txtfetch's text as a plain field.
Max request size100 MB per HTTP request by default (http.max_content_length). OpenSearch kept the setting name from its fork.
Built-in document extractionYes, through the ingest-attachment plugin, a separate install that runs Apache Tika. Its docs name no OCR step.
Keeps the full textYes, in _source by default, the same as any other field.
Document ID formatAny UTF-8 string up to 512 bytes. OpenSearch generates one automatically if you omit it.
Bulk write APIThe Bulk API (_bulk), one JSON action line per document.

No accuracy, speed, or recall figure is measured against OpenSearch on this page. See /benchmarks for txtfetch's own measured extraction numbers.

The extraction step it already has

OpenSearch can run Tika too, through the ingest-attachment plugin. You install it first with ./bin/opensearch-plugin install ingest-attachment. It decodes a base64 file, passes it to Tika, and writes the text to a field. It stops at 100,000 characters by default. Its docs name no OCR step for scanned pages. txtfetch runs Tika and Tesseract outside your cluster, with no plugin to install.

Splitting for OpenSearch

Index one document per page or section. Give each one a parent_id field that points back to the source file. To find real section breaks, use the page numbers in format=json output. Or test your text in the chunk previewer at /tools/chunk-preview.

the pipeline

Extract, split, and bulk-write. No framework, just the txtfetch Python SDK and OpenSearch's own client.

Python
import os

from opensearchpy import OpenSearch, helpers
from txtfetch import Txtfetch

txtfetch = Txtfetch(api_key=os.environ["TXTFETCH_KEY"])
client = OpenSearch(
    hosts=[os.environ["OPENSEARCH_URL"]],
    http_auth=(os.environ["OPENSEARCH_USER"], os.environ["OPENSEARCH_PASSWORD"]),
    use_ssl=True,
)


def split_text(text: str, size: int = 2000, overlap: int = 200) -> list[str]:
    sections = []
    start = 0
    while start < len(text):
        end = start + size
        sections.append(text[start:end])
        start = end - overlap
    return sections


# 1. Extract
result = txtfetch.extract(file="whitepaper.pdf")

# 2. Split
sections = split_text(result.extracted_text)

# 3. Bulk write — the ingest-attachment plugin's own indexed_chars cap only
# applies if you route the raw file through it. Index the already-extracted
# text as a plain field instead.
actions = [
    {
        "_index": "docs",
        "_id": f"whitepaper-{i}",
        "_source": {"parent_id": "whitepaper.pdf", "section": i, "text": section},
    }
    for i, section in enumerate(sections)
]
helpers.bulk(client, actions)

Every call above was resolved against opensearch-py 3.2.0, in September 2026. Client APIs move, so check the vendor's own docs for the version you pin.

large batches

A large file, or a slow OCR pass, routes to an async job automatically. That returns a 202 plus a job_id to poll, and ?async=true forces that path for any request. Pass webhook_url instead, and txtfetch calls you back when the text is ready, rather than you polling for it. See async jobs & webhooks for the full lifecycle.

Submit (async)
curl -X POST "https://api.txtfetch.com/v1/extract?url=https://example.com/whitepaper.pdf&async=true" \
  -H "Authorization: Bearer $TXTFETCH_KEY"

# {"status": "processing", "job_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6"}
Poll
curl "https://api.txtfetch.com/v1/extract/3fa85f64-5717-4562-b3fc-2c963f66afa6" \
  -H "Authorization: Bearer $TXTFETCH_KEY"

# {"status": "processing", "job_id": "..."} while running, then the same
# {"status": "success", "extracted_text": "...", "metadata": {...}} shape
# POST /v1/extract would have returned synchronously.
curl
curl -X POST "https://api.txtfetch.com/v1/extract?url=https://example.com/whitepaper.pdf" \
  -H "Authorization: Bearer $TXTFETCH_KEY" \
  --data-urlencode "webhook_url=https://example.com/webhooks/txtfetch"

A retried request, from a client timeout or a dropped response, shouldn't extract and write the same document twice. Send the same Idempotency-Key on a retry and txtfetch returns the first result instead of running the job again. See idempotency for the full guarantee.

curl
curl -X POST https://api.txtfetch.com/v1/extract \
  -H "Authorization: Bearer $TXTFETCH_KEY" \
  -H "Idempotency-Key: 3f29b6e4-9c1a-4b8e-9c2a-1e6f0a2d5b3c" \
  -F file=@contract.pdf
JavaScript
// The SDK auto-generates and reuses an Idempotency-Key across its own
// retry chain. Pass your own to control it explicitly:
await txtfetch.extract({ file: "./contract.pdf", idempotencyKey: "3f29b6e4-..." });
Python
# The SDK auto-generates and reuses an Idempotency-Key across its own
# retry chain. Pass your own to control it explicitly:
client.extract(file="contract.pdf", idempotency_key="3f29b6e4-...")

txtfetch ships no connector, plugin, or client for OpenSearch. The pipeline above is the whole integration. Extract with txtfetch. Then write the result with OpenSearch's own client.

frequently asked questions

Does txtfetch write directly to OpenSearch?
No. txtfetch returns the extracted text over one HTTP call. You split the text and send the bulk request with OpenSearch's own client.
Why not install the ingest-attachment plugin?
You can. It is one more plugin to install and patch. It stops at 100,000 characters by default, and its docs name no OCR step. With txtfetch, you extract over HTTP and bulk-write the result.
Does txtfetch benchmark accuracy or speed against OpenSearch?
No. OpenSearch is a search engine, not a text-extraction service. There is no accuracy or speed comparison to make. See /benchmarks for txtfetch's own extraction numbers.

sources

Related

other engines

Wire the extraction step in.

The RAG recipe shows the whole path, from a file to the OpenSearch upsert.

Read the RAG recipe →

Get an API key →