Each head projects queries, keys, and values into its own subspace and produces a weighted combination. The head outputs are concatenated and projected again, allowing the layer to combine multiple patterns of interaction.
Multi-head attention runs several attention operations in parallel so a model can learn different relationships within the same input.
Each head projects queries, keys, and values into its own subspace and produces a weighted combination. The head outputs are concatenated and projected again, allowing the layer to combine multiple patterns of interaction.