Loss function

The loss function produces a number for every prediction: the smaller, the better. The goal of training is to minimise this value across all training data.

The choice of loss function determines what the model considers “important”. For language models, for example, it measures how well the actual next token was predicted.