Does txtfetch write directly to Pinecone?

No. txtfetch returns extracted text over one HTTP call. Chunking, embedding, and the upsert itself stay a caller-side step, exactly like the pipeline above shows.

Can I store the full chunk text in Pinecone?

Only up to the 40 KB per-vector metadata ceiling, and metadata only holds strings, numbers, booleans, and lists of strings. A longer chunk, or a chunk record with nested fields, needs a separate store keyed by the same vector id.

Does txtfetch benchmark accuracy or speed against Pinecone?

No. Pinecone is a vector database, not a text-extraction service, so there's no accuracy or speed comparison to make. See /benchmarks for txtfetch's own measured extraction numbers.

Extract text into Pinecone

A managed, serverless vector database with a hard ceiling on per-vector metadata. Here's what changes between the text txtfetch returns and the first row you write to Pinecone.

The problem

Pinecone is fully managed, so there's no schema to design. But its metadata is capped at 40 KB per vector, and only holds strings, numbers, booleans, and lists of strings. A chunk record that carries its own JSON blob, the way you might store it elsewhere, doesn't fit as-is.

what pinecone needs

Constraints that shape chunking and the write step, cited and dated
Per-vector metadata limit	40 KB per vector, filterable fields only. Values must be a string, number, boolean, or list of strings. No nested objects.
Stores the chunk text itself	Not as a dedicated field. Put the chunk text in metadata, inside that 40 KB, or keep it in your own store, keyed by the vector id.
ID format	An ASCII string, 1 to 512 characters.
Max vector dimensions	20,000 dimensions per vector.
Batch upsert size	2 MB per upsert request. Pinecone's own guidance keeps a single call near 1,000 vectors, fewer at higher dimensions.
Index type	Serverless indexes only today. Metric is cosine, dotproduct, or euclidean, set once at index creation.

No accuracy, speed, or recall figure is measured against Pinecone on this page. See /benchmarks for txtfetch's own measured extraction numbers.

the pipeline

Extract, chunk, embed, and upsert. No framework, just the txtfetch Python SDK and Pinecone's own client.

Python

import os

from openai import OpenAI
from pinecone import Pinecone
from txtfetch import Txtfetch

txtfetch = Txtfetch(api_key=os.environ["TXTFETCH_KEY"])
oai = OpenAI()
pc = Pinecone(api_key=os.environ["PINECONE_API_KEY"])
index = pc.Index("docs")


def chunk_text(text: str, size: int = 1000, overlap: int = 100) -> list[str]:
    chunks = []
    start = 0
    while start < len(text):
        end = start + size
        chunks.append(text[start:end])
        start = end - overlap
    return chunks


# 1. Extract
result = txtfetch.extract(file="whitepaper.pdf")

# 2. Chunk
chunks = chunk_text(result.extracted_text)

# 3. Embed
response = oai.embeddings.create(model="text-embedding-3-small", input=chunks)

# 4. Upsert — metadata is capped at 40 KB per vector, so only an excerpt of
# the chunk goes here. Keep the full text in your own store, keyed by this
# same id, if you need it back verbatim.
vectors = [
    {
        "id": f"whitepaper-{i}",
        "values": item.embedding,
        "metadata": {"source": "whitepaper.pdf", "text": chunks[i][:2000]},
    }
    for i, item in enumerate(response.data)
]

index.upsert(vectors=vectors, namespace="docs")

Every call above was resolved against pinecone 7.3.0, in September 2026. Client APIs move, so check the vendor's own docs for the version you pin.

Chunking for Pinecone

Pinecone's 40 KB metadata ceiling is the real constraint on chunk size, not any limit on the vector itself. Store a short excerpt in metadata, not the full chunk, once your chunks run long. Check the actual token counts and boundaries your chunker produces in the chunk previewer before you commit to a size.

large batches

A large file, or a slow OCR pass, routes to an async job automatically. That returns a 202 plus a job_id to poll, and ?async=true forces that path for any request. Pass webhook_url instead, and txtfetch calls you back when the text is ready, rather than you polling for it. See async jobs & webhooks for the full lifecycle.

Submit (async)

curl -X POST "https://api.txtfetch.com/v1/extract?url=https://example.com/whitepaper.pdf&async=true" \
  -H "Authorization: Bearer $TXTFETCH_KEY"

# {"status": "processing", "job_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6"}

Poll

curl "https://api.txtfetch.com/v1/extract/3fa85f64-5717-4562-b3fc-2c963f66afa6" \
  -H "Authorization: Bearer $TXTFETCH_KEY"

# {"status": "processing", "job_id": "..."} while running, then the same
# {"status": "success", "extracted_text": "...", "metadata": {...}} shape
# POST /v1/extract would have returned synchronously.

curl

curl -X POST "https://api.txtfetch.com/v1/extract?url=https://example.com/whitepaper.pdf" \
  -H "Authorization: Bearer $TXTFETCH_KEY" \
  --data-urlencode "webhook_url=https://example.com/webhooks/txtfetch"

A retried request, from a client timeout or a dropped response, shouldn't extract and upsert the same document twice. Send the same Idempotency-Key on a retry and txtfetch returns the first result instead of running the job again. See idempotency for the full guarantee.

curl

curl -X POST https://api.txtfetch.com/v1/extract \
  -H "Authorization: Bearer $TXTFETCH_KEY" \
  -H "Idempotency-Key: 3f29b6e4-9c1a-4b8e-9c2a-1e6f0a2d5b3c" \
  -F file=@contract.pdf

JavaScript

// The SDK auto-generates and reuses an Idempotency-Key across its own
// retry chain. Pass your own to control it explicitly:
await txtfetch.extract({ file: "./contract.pdf", idempotencyKey: "3f29b6e4-..." });

Python

# The SDK auto-generates and reuses an Idempotency-Key across its own
# retry chain. Pass your own to control it explicitly:
client.extract(file="contract.pdf", idempotency_key="3f29b6e4-...")

txtfetch ships no connector, plugin, or client for Pinecone. The pipeline above is the whole integration. Extract with txtfetch, then write to Pinecone with Pinecone's own client, the same as you would for any other source of text.

frequently asked questions

Does txtfetch write directly to Pinecone?: No. txtfetch returns extracted text over one HTTP call. Chunking, embedding, and the upsert itself stay a caller-side step, exactly like the pipeline above shows.
Can I store the full chunk text in Pinecone?: Only up to the 40 KB per-vector metadata ceiling, and metadata only holds strings, numbers, booleans, and lists of strings. A longer chunk, or a chunk record with nested fields, needs a separate store keyed by the same vector id.
Does txtfetch benchmark accuracy or speed against Pinecone?: No. Pinecone is a vector database, not a text-extraction service, so there's no accuracy or speed comparison to make. See /benchmarks for txtfetch's own measured extraction numbers.

sources

Pinecone: Python SDK overviewAccessed 2026-09
Pinecone: Upsert referenceAccessed 2026-09
Pinecone: Indexing overview (metadata limits and types)Accessed 2026-09
Pinecone: Upsert data guide (batch and request-size limits)Accessed 2026-09

RAG & LLM ingestion, the wider use case this pipeline feeds.
RAG recipe: chunk, embed, index, the same pipeline through LangChain or LlamaIndex instead.
Chunk previewer, to test your own extracted text before you write the first row.
Async jobs & webhooks, the full lifecycle for a large batch.

other stores

Wire the extraction step in.

The RAG recipe shows the whole path, from a file to the Pinecone upsert.

Read the RAG recipe →

Get an API key →

Extract text into Pinecone

The problem

Chunking for Pinecone

Related

Wire the extraction step in.