OpenAI’s large language models (sometimes referred to as GPT’s) process text using tokens, which are common sequences of characters found in a set of text. The models learn to understand the statistical relationships between these tokens, and excel at producing the next token in a sequence of tokens.
You can think of tokens as the “letters” that make up the “words” and “sentences” that AI systems use to communicate.
A helpful rule of thumb is that one token generally corresponds to ~4 characters of text for common English text. This translates to roughly ¾ of a word (so 100 tokens ~= 75 words).
The process of breaking text down into tokens is called tokenization. This allows the AI to analyze and “digest” human language into a form it can understand. Tokens become the data used to train, improve, and run the AI systems.
Why Do Tokens Matter? There are two main reasons tokens are important to understand:
Strategies for Managing Tokens
Because tokens are central to how LLMs work, it’s important to learn strategies to make the most of them:
