Attention
Also: Self-Attention · Aufmerksamkeitsmechanismus
A mechanism that lets a model determine, for each word, which other words in the context are relevant to it.
In the sentence “The bank was by the river, so I sat on it”, attention helps the model recognise that “it” refers to “bank” – and that a riverbank or bench is meant, not a financial institution.
Attention compares each token with all others. The computing effort therefore grows sharply with text length – one reason why very large context windows are expensive.
General information, not legal or professional advice.