Is a token the same as a word?

No. A tokenizer often splits a single word into multiple sub-word tokens. Common words may be a single token, while rarer or compound words split into several. Token counts and word counts diverge, sometimes substantially.

Can I get an exact token count without a real tokenizer?

Not precisely. A character-based estimate, roughly four characters per token, is a useful sizing guide. Actual counts are model-specific, though, and can differ meaningfully on non-English text or heavy punctuation.

Token

The unit an LLM actually counts and charges for, usually a word piece rather than a whole word. A character-based estimate of it is a heuristic, not the real count.

definition

A token is the unit of text an LLM's tokenizer produces by splitting a string into sub-word pieces. It is what the model's context limit and usage-based pricing are actually measured in, rather than characters or words.

Also called: LLM token, token count

in-plain-terms

A tokenizer, typically a byte-pair-encoding scheme, breaks text into pieces that are often smaller than a whole word. Common words may be one token, while rarer or compound words split into several. This means a rough "characters divided by four" estimate is a useful sizing heuristic, not a real token count. Actual counts are model-specific, and the same text can tokenize to meaningfully different totals across different embedding or chat models.

Token counts matter twice in a RAG pipeline. Once for chunk sizing, staying under an embedding model's input limit. And once for cost, since most LLM APIs charge per token processed and generated.

why-it-matters

Sizing chunks off a character-count heuristic instead of a real tokenizer can under- or over-shoot an embedding model's actual input limit. This is especially true on non-English text or heavy punctuation, where the rule of thumb is least accurate.
Token counts, not word or character counts, are what usage-based LLM pricing is metered against. A rough estimate is fine for chunk sizing but isn't a substitute for a real tokenizer when cost accuracy matters.

how-to-check

See both an estimated token count and the exact character count for pasted text, side by side. This lets you tell the estimate from the one number that isn't an estimate. Estimate tokens for your text

related-terms

Chunking →

faq

Is a token the same as a word?: No. A tokenizer often splits a single word into multiple sub-word tokens. Common words may be a single token, while rarer or compound words split into several. Token counts and word counts diverge, sometimes substantially.
Can I get an exact token count without a real tokenizer?: Not precisely. A character-based estimate, roughly four characters per token, is a useful sizing guide. Actual counts are model-specific, though, and can differ meaningfully on non-English text or heavy punctuation.

related-reading

Into an LLM pipeline

See the term in real output.

Drop a file into the free reader and watch it happen.

Open the file reader →

Get an API key →