Each token can gather information from other allowed token positions according to learned similarity scores. A mask may restrict which positions are visible, such as preventing a language model from attending to future tokens during generation.
Self-attention is attention in which queries, keys, and values are derived from the same sequence.
Each token can gather information from other allowed token positions according to learned similarity scores. A mask may restrict which positions are visible, such as preventing a language model from attending to future tokens during generation.