Transformer

The transformer was introduced in the research paper “Attention Is All You Need”. Its core is the attention mechanism, which lets the model look at all parts of an input at once and weigh which ones matter for understanding.

Because transformers can be trained in parallel on GPUs very efficiently, models could for the first time be scaled to enormous amounts of data. The architecture is now also used for images, audio, video and proteins.