Benchmark

Benchmarks consist of fixed task collections – such as knowledge questions, maths, programming or reasoning. The results help measure progress and position models.

But they have limits: models may be optimised for well-known tests, and good scores do not guarantee good performance in your own use case. For businesses, their own tests with real tasks (“evals”) are therefore important.