> Source: https://txtfetch.com/blog/batch-and-large-document-ingestion
> Plain-text twin — every page on txtfetch.com has one. https://txtfetch.com/text

---

# Ingesting large documents and big batches without falling over

Jun 23, 2026 · 3 min read · rag, batch, llm-ingestion, api

Two kinds of scale break ingestion pipelines that work fine in a demo. One is a single very large document. The other is a very large number of ordinary ones. They look different. But they stress the same two things: how long a single request takes, and how many you can run at once.

## Large single documents

[`/v1/extract`](https://txtfetch.com/extract/pdf) is a synchronous endpoint. There’s no async job or webhook mode. You send a request and get `extracted_text` back on the same connection. For a 500-page report or a multi-hundred-slide deck, the request itself takes longer. Your client needs a longer timeout too. Don’t use the short timeout you’d set for a two-page memo:

```bash
curl -X POST https://api.txtfetch.com/v1/extract \
  -H "Authorization: Bearer $TXTFETCH_KEY" \
  -F file=@annual-report-500pg.pdf \
  --max-time 300
```

Many HTTP client libraries default to a short timeout, sometimes 10 or 30 seconds. Raise it explicitly for extraction calls. Otherwise you’ll discover the hard way that your largest documents are the ones that fail silently.

## Large batches

A backfill means ingesting an existing document archive, not a steady trickle of new uploads. There the constraint shifts. It’s no longer “how long does one request take” but “how many can I run at once without overwhelming either side.” A plain `for` loop making requests one at a time works. But it wastes most of your batch window waiting on network I/O. Some client-side concurrency, capped at a sane number, gets through a large batch faster than a sequential wait:

```python
import asyncio
import httpx

SEM = asyncio.Semaphore(8)  # cap concurrent in-flight requests

async def extract(client, path, api_key):
    async with SEM:
        with open(path, "rb") as f:
            resp = await client.post(
                "https://api.txtfetch.com/v1/extract",
                headers={"Authorization": f"Bearer {api_key}"},
                files={"file": f},
                timeout=120,
            )
        body = resp.json()
        if body["status"] != "success":
            raise ValueError(f"{path}: {body.get('error')}")
        return path, body["extracted_text"]

async def run_batch(paths, api_key):
    async with httpx.AsyncClient() as client:
        results = await asyncio.gather(
            *(extract(client, p, api_key) for p in paths),
            return_exceptions=True,
        )
    return results
```

`return_exceptions=True` matters here: one bad file in a ten-thousand-file backfill shouldn’t take down the other 9,999. Collect the failures separately and retry or route them to a review queue instead of letting one exception cancel the whole batch.

## Retry and backoff

Treat a failed request as retryable unless the response tells you otherwise. An explicit `{"status": "error", "error": "..."}` for a genuinely unreadable file (corrupt, unsupported, empty) isn’t worth retrying. Retrying the same broken input just wastes another request. A transport-level failure (timeout, connection reset) is usually worth a retry. Add backoff so a batch of failures doesn’t turn into a retry storm:

```python
async def extract_with_retry(client, path, api_key, attempts=3):
    for attempt in range(attempts):
        try:
            return await extract(client, path, api_key)
        except (httpx.TimeoutException, httpx.TransportError):
            if attempt == attempts - 1:
                raise
            await asyncio.sleep(2 ** attempt)
```

There’s no idempotency key or job ID here. Each request is independent. A retry just means “send the same file again,” not resuming a partially completed job. That’s the tradeoff of a synchronous, no-job-state API. It’s simpler to reason about. But the client owns concurrency and retry logic instead of polling a job status endpoint.

## Where this fits into cost

Once ingestion completes reliably at whatever batch size you’re running, the next question is usually cost. What does it cost to run this repeatedly? See the [per-document vs per-page pricing guide](https://txtfetch.com/blog/per-document-vs-per-page-pricing) for how document count and page count factor in.

If scanned documents are part of the batch, read the [OCR guide](https://txtfetch.com/blog/ocr-scanned-documents-api) too. OCR requests can take longer than native-text extraction. Account for that in your timeout and concurrency tuning.

Planning a backfill and want to talk through concurrency limits? [Get in touch](https://txtfetch.com/contact) and we’ll set you up with an API key.

keep reading

-   **[Using txtfetch as a LangChain and LlamaIndex document loader](https://txtfetch.com/blog/langchain-llamaindex-document-loader)**

    There's no official txtfetch SDK yet. Wiring the endpoint into a LangChain or LlamaIndex loader takes about fifteen lines.

    May 5, 2026 · 3 min read

-   **[Chunking strategies for RAG: from clean text to good retrieval](https://txtfetch.com/blog/chunking-strategies-for-rag)**

    Fixed-size, recursive, and structure-aware chunking all assume clean extracted text. Extraction quality bounds chunk quality, whichever one you pick.

    Apr 7, 2026 · 3 min read

-   **[Parsing DOCX, PPTX, and XLSX into clean text for LLM ingestion](https://txtfetch.com/blog/parse-office-docs-docx-pptx-xlsx-for-llms)**

    python-docx, python-pptx, and openpyxl each have their own API and their own way of failing on real files. One endpoint replaces all three.

    Mar 10, 2026 · 3 min read

See also: [RAG & LLM ingestion →](https://txtfetch.com/solutions/rag-ingestion)

## Try it on your own file.

The free reader runs in your browser. Nothing gets uploaded.

[Open the file reader →](https://txtfetch.com/tools/file-to-text)

[Get an API key →](https://app.txtfetch.com/signup)