Next-token prediction

A language model does not produce text in one go but token by token. At each step it computes a probability distribution over all possible next tokens, picks one and appends it – then the process starts again.

This simple principle explains both the strengths (fluent, fitting language) and the weaknesses (plausible-sounding errors) of language models. Parameters such as temperature control how random the choice is.