Re-index only what changed.

What txtfetch does not do

txtfetch does not remember which files it saw before. So it cannot tell which of your files changed since last night. A full nightly run pays for every document, changed or not. Change detection is your pipeline's job.

There is no content cache

Every successful extraction counts against your quota. That is true whether the bytes changed since your last call or not.

An Idempotency-Key is not a cache

It keys on the header value plus your API key, not on document content. A different file under the same key is rejected, not deduplicated. The record expires after 1 day.

A ?url= call always re-fetches

fetch-url.js sends no If-None-Match or If-Modified-Since header. txtfetch never asks the origin whether the document changed.

The response carries no content hash and no engine version

A success body has a fixed field set, and none of them is a hash. To detect a change in the output, compute a hash yourself.

txtfetch keeps nothing past the 1-day expiry, so it has no store to compare your next upload against. Where your file goes lists every place your bytes rest, and for how long. Idempotency explains what the Idempotency-Key does protect: a retried request.

Four steps to re-index only what changed

Detect a change before you call the API, not after. Each step below builds on the last.

1. Hash the source bytes before you call the API. Take a SHA-256 hash of each document's raw bytes. Keep it in a manifest, keyed by document id. Before your next run, hash the bytes again and skip the call when the hash matches. For a URL source, do your own conditional GET first. Send an If-None-Match or If-Modified-Since header against the origin. Only POST the bytes to txtfetch when the origin reports a change.
2. Hash the output text too. Keep a second hash, of the output text, next to the source hash. Hash extracted_text, or markdown, or the elements array, whichever format you request. Re-embed only when that second hash changes. This step catches two cases the source hash misses. An engine upgrade can change the extracted text for a file whose bytes never changed. A premium-tier call can also return different text on a re-run.
3. Derive stable chunk ids. Build each chunk's id from the document id plus its chunk index. Do not use a random or auto-incrementing value. Re-extract the same document twice, and it must produce the same chunk ids both times. That lets a re-index find and replace the old rows instead of appending duplicates. Check the id-format rule for your own store before you pick a scheme.
4. Delete before you upsert. When a document changes or is removed, delete its old chunk ids first. Then upsert the new ones. A changed document can produce fewer chunks than it did before. Skip the delete step, and the extra old chunks stay in the index. They keep surfacing stale text to your retriever.

Step 3 must follow your store's own id rule. Weaviate is a strict example. Its row in the ingestion guide gives this id format:

A UUID. Weaviate can generate one deterministically from your own fields with generate_uuid5(...) if you don't supply one.

See every Weaviate constraint. Each store in the ingestion guide lists its id rule the same way.

What it costs, worked out

A full nightly run calls the API once per document. Hashing first calls it once per changed document. Take a corpus of 10,000 documents. Use 2% daily churn as the example rate, and a 30-day month.

The full run needs 300,000 calls a month. That is 30× the Developer quota of 10,000. It is also more than the Scale quota of 100,000. The incremental run needs 6,000 calls a month. That fits inside the Developer quota.

Check the quota table on /pricing against your own churn rate.

A manifest, in plain Python

The snippet uses the standard library only. hashlib makes both hashes, and json stores the manifest. Replace the file loop with your own document list. Add your delete and upsert calls where the comment marks them.

import hashlib
import json
import os

from txtfetch import Txtfetch

MANIFEST_PATH = "manifest.json"
client = Txtfetch(api_key=os.environ["TXTFETCH_KEY"])


def sha256_bytes(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()


def load_manifest() -> dict:
    if not os.path.exists(MANIFEST_PATH):
        return {}
    with open(MANIFEST_PATH) as f:
        return json.load(f)


def save_manifest(manifest: dict) -> None:
    with open(MANIFEST_PATH, "w") as f:
        json.dump(manifest, f, indent=2)


def reindex(paths: list[str]) -> None:
    manifest = load_manifest()

    for path in paths:
        doc_id = os.path.basename(path)
        with open(path, "rb") as f:
            source_bytes = f.read()
        source_hash = sha256_bytes(source_bytes)

        record = manifest.get(doc_id)
        if record and record["source_hash"] == source_hash:
            continue  # unchanged bytes: skip the call entirely

        result = client.extract(file=source_bytes)
        output_hash = sha256_bytes(result.extracted_text.encode("utf-8"))

        if record and record["output_hash"] == output_hash:
            manifest[doc_id] = {"source_hash": source_hash, "output_hash": output_hash}
            continue  # same source, same text: nothing to re-embed

        # source or output changed: delete the doc's old chunk ids first,
        # then upsert new chunks built from result.extracted_text
        # (each chunk id derived from doc_id + its chunk index).

        manifest[doc_id] = {"source_hash": source_hash, "output_hash": output_hash}

    save_manifest(manifest)

It uses the same from txtfetch import Txtfetch client as the ingestion pipelines. A response carries status, extracted_text, metadata.content_type, metadata.bytes, metadata.chars, and metadata.ocr. Hash extracted_text only. The metadata can stay the same while the text changes.

Frequently asked questions

Does the Idempotency-Key skip unchanged files?

No. An Idempotency-Key is keyed on the header value plus your API key, not on document content. Its record expires after one day. It replays a retried request. It does not detect an unchanged file across separate calls.

Does a ?url= call use ETag or Last-Modified?

No. txtfetch's server-side fetch sends no conditional-request headers, so it always re-fetches and re-extracts. To skip an unchanged URL, do your own conditional GET against the origin first. Only call txtfetch when the origin says the document changed.

Is the output the same every run?

On the standard tier, yes. The same bytes produce the same text every run. An engine upgrade can still change that output over time, and a response carries no engine version to tell you it happened. The premium tier calls a vision-language model and sets no temperature. Two runs of the same file can differ. Premium is off by default.

Do deleted files leave chunks behind?

Only if your pipeline doesn't delete them. txtfetch keeps no record of your index, so it can't tell you a source file is gone. Delete a removed document's chunk ids from your manifest and your index in the same run you drop it from the source.