search⌘KGet started

Extract text into Postgres full-text search

Built into Postgres, and it silently clamps every word position past 16,383. Here's what changes between the text txtfetch returns and the first row you write to Postgres full-text search.

The problem

A tsvector stores each lexeme with the positions where it appears. Postgres caps positions at 16,383. It silently sets every later position to 16,383, with no error. So in a long document stored as one row, the tail of the text collapses onto one position. Plain word matches still work. Phrase and proximity searches on that tail do not.

what postgres full-text search needs

Constraints that shape splitting and the write step, cited and dated
Where long text stops countingWord positions stop at 16,383. Postgres silently sets every later position to 16,383. It also keeps at most 256 positions per lexeme and drops the rest. Only a tsvector over 1 MB raises an error.
Max request sizeNone from full-text search. Normal Postgres statement and row limits apply. Use COPY or executemany for a bulk load.
Built-in document extractionNone. to_tsvector() parses text you already have. It does not extract text from files.
Keeps the full textYes. The source text is a normal column in the same row as its generated tsvector.
Document ID formatNone from full-text search. Use the primary key type your table declares, such as bigserial, uuid, or text.
Bulk write APINo bulk endpoint. Use executemany(...) for a moderate batch, or COPY for a large one.

No accuracy, speed, or recall figure is measured against Postgres full-text search on this page. See /benchmarks for txtfetch's own measured extraction numbers.

Splitting for Postgres full-text search

Index one row per page or section, with a parent_id column that points back to the source file. The limits apply per row. Keep each row well under 16,383 words, so no position gets clamped. To find real section breaks, use the page numbers in format=json output. Or test your text in the chunk previewer at /tools/chunk-preview.

the pipeline

Extract, split, and bulk-write. No framework, just the txtfetch Python SDK and Postgres full-text search's own client.

Python
import os

import psycopg
from txtfetch import Txtfetch

txtfetch = Txtfetch(api_key=os.environ["TXTFETCH_KEY"])

conn = psycopg.connect(os.environ["DATABASE_URL"], autocommit=True)
conn.execute(
    """
    CREATE TABLE IF NOT EXISTS docs (
        id bigserial PRIMARY KEY,
        parent_id text,
        section int,
        content text,
        search tsvector GENERATED ALWAYS AS (to_tsvector('english', content)) STORED
    )
    """
)
conn.execute("CREATE INDEX IF NOT EXISTS docs_search_idx ON docs USING gin (search)")


def split_text(text: str, size: int = 2000, overlap: int = 200) -> list[str]:
    sections = []
    start = 0
    while start < len(text):
        end = start + size
        sections.append(text[start:end])
        start = end - overlap
    return sections


# 1. Extract
result = txtfetch.extract(file="whitepaper.pdf")

# 2. Split — Postgres silently clamps every word position past 16,383, so
# one long row loses phrase and proximity search on its tail.
sections = split_text(result.extracted_text)

# 3. Bulk write — the generated column builds the tsvector automatically.
rows = [("whitepaper.pdf", i, section) for i, section in enumerate(sections)]
with conn.cursor() as cur:
    cur.executemany(
        "INSERT INTO docs (parent_id, section, content) VALUES (%s, %s, %s)",
        rows,
    )

Every call above was resolved against psycopg 3.3.6, in September 2026. Client APIs move, so check the vendor's own docs for the version you pin.

large batches

A large file, or a slow OCR pass, routes to an async job automatically. That returns a 202 plus a job_id to poll, and ?async=true forces that path for any request. Pass webhook_url instead, and txtfetch calls you back when the text is ready, rather than you polling for it. See async jobs & webhooks for the full lifecycle.

Submit (async)
curl -X POST "https://api.txtfetch.com/v1/extract?url=https://example.com/whitepaper.pdf&async=true" \
  -H "Authorization: Bearer $TXTFETCH_KEY"

# {"status": "processing", "job_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6"}
Poll
curl "https://api.txtfetch.com/v1/extract/3fa85f64-5717-4562-b3fc-2c963f66afa6" \
  -H "Authorization: Bearer $TXTFETCH_KEY"

# {"status": "processing", "job_id": "..."} while running, then the same
# {"status": "success", "extracted_text": "...", "metadata": {...}} shape
# POST /v1/extract would have returned synchronously.
curl
curl -X POST "https://api.txtfetch.com/v1/extract?url=https://example.com/whitepaper.pdf" \
  -H "Authorization: Bearer $TXTFETCH_KEY" \
  --data-urlencode "webhook_url=https://example.com/webhooks/txtfetch"

A retried request, from a client timeout or a dropped response, shouldn't extract and write the same document twice. Send the same Idempotency-Key on a retry and txtfetch returns the first result instead of running the job again. See idempotency for the full guarantee.

curl
curl -X POST https://api.txtfetch.com/v1/extract \
  -H "Authorization: Bearer $TXTFETCH_KEY" \
  -H "Idempotency-Key: 3f29b6e4-9c1a-4b8e-9c2a-1e6f0a2d5b3c" \
  -F file=@contract.pdf
JavaScript
// The SDK auto-generates and reuses an Idempotency-Key across its own
// retry chain. Pass your own to control it explicitly:
await txtfetch.extract({ file: "./contract.pdf", idempotencyKey: "3f29b6e4-..." });
Python
# The SDK auto-generates and reuses an Idempotency-Key across its own
# retry chain. Pass your own to control it explicitly:
client.extract(file="contract.pdf", idempotency_key="3f29b6e4-...")

txtfetch ships no connector, plugin, or client for Postgres full-text search. The pipeline above is the whole integration. Extract with txtfetch. Then write the result with Postgres full-text search's own client.

frequently asked questions

Does txtfetch write directly to Postgres?
No. txtfetch returns the extracted text over one HTTP call. You split the text and run the INSERT yourself, with psycopg or any other Postgres client.
Does Postgres warn me when a row goes over a limit?
Only for the 1 MB tsvector cap, which raises an error. Positions past 16,383 are silently set to 16,383. Positions past 256 for one lexeme are silently dropped. A word over 2 KB gets a notice and stays out of the index. So split the document before you insert it.
Does txtfetch benchmark accuracy or speed against Postgres full-text search?
No. Postgres full-text search is a database feature, not a text-extraction service. There is no accuracy or speed comparison to make. See /benchmarks for txtfetch's own extraction numbers.

sources

Related

other engines

Wire the extraction step in.

The RAG recipe shows the whole path, from a file to the Postgres full-text search upsert.

Read the RAG recipe →

Get an API key →