Pre-training
Also: Pretraining
The first and most expensive training phase, in which a model learns from huge amounts of general data.
In pre-training, a language model learns to continue text from trillions of tokens – such as web pages, books and code. This builds broad basic knowledge about language and the world.
A purely pre-trained “base model” only continues text and follows instructions poorly. Only post-training with instruction tuning and RLHF turns it into a helpful assistant.
General information, not legal or professional advice.