An attention sink is a token — often one of the first few in a sequence — that soft-max attention consistently assigns unusually high weight to, regardless of its actual semantic relevance, because softmax requires attention scores to sum to 1 and the model needs somewhere to route 'unneeded' attention mass. StreamingLLM (Xiao et al., 2023) showed that dropping early tokens from a fixed-size sliding-window cache causes attention to collapse, but keeping just a handful of sink tokens permanently in the window restores stability, enabling long streaming generation on a fixed memory budget. A 2026 paper from Microsoft researcher Alexia Jolicoeur-Martineau found that sliding-window attention with sinks, applied at inference with no retraining, outperforms models retrofitted to linear attention during post-training.