It computes compatibility scores between a query and a set of keys, normalizes those scores into weights, and combines the corresponding values. The result is a context-dependent summary instead of a fixed compression of all inputs.
An attention mechanism lets a model assign different importance to available pieces of information when producing a representation or output.
It computes compatibility scores between a query and a set of keys, normalizes those scores into weights, and combines the corresponding values. The result is a context-dependent summary instead of a fixed compression of all inputs.