The query-side representation selects relevant information from the source-side representation through attention weights. It is commonly used to connect encoder and decoder states or condition generation on text, images, audio, or retrieved context.
Cross-attention lets queries from one sequence attend to keys and values from another source.
The query-side representation selects relevant information from the source-side representation through attention weights. It is commonly used to connect encoder and decoder states or condition generation on text, images, audio, or retrieved context.