A is a piece of text a language model processes. Before the model reads your , the text is split into tokens. A token may be a whole word, part of a word, a single character, or punctuation.
Why the count is hard to predict
The way text is split varies from one model to another, so the same sentence can contain different numbers of tokens in different models. Long words often split into several tokens. You therefore cannot work out the number of tokens directly from the number of words.
Where you encounter tokens
- Length limits: the context window is measured in tokens and includes both the prompt and the response.
- Prices: services used through an API often charge by the number of input and output tokens.
- Usage limits: a service may limit how many tokens can be used in a given amount of time.
When the count matters
If a length limit or price matters, do not estimate tokens by converting them to words in your head. Use the service’s documentation or a token counter.