GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

Introduced Grouped-Query Attention (GQA), interpolating between Multi-Head Attention (MHA) and Multi-Query Attention (MQA) with intermediate key-value groups, preserving model quality while maintaining MQA inference speed.

Directly guides architectural decisions, alignment strategy, and serving infrastructure for task-text-generation.

Assumptions

• Empirical distribution regularity holds and target domain adheres to pretraining linguistic/visual support

Limitations

• Resource scaling, inference memory requirements, and alignment robustness vary with model size and hardware topology