Token
Also: Tokens · Tokenizer · Tokenisierung
The smallest unit of text a language model works with – often a word or part of a word.
Before a model processes text, a tokenizer splits it into tokens. Common words are a single token; rare or long words are split into several parts. In English, a token roughly corresponds to three quarters of a word; languages with long compound words, such as German, usually need more tokens.
Tokens are the currency of language models: context windows, speed and API prices are measured in tokens – separately for input and output tokens.
General information, not legal or professional advice.