Fused Attention

What is Fused Attention?

Fused attention, or fused head attention, is the merging of the weights in the attention heads. This is a type of parameter sharing, and there are variants including MHA, Medusa attention, and other more obscure methods.

Fused Attention: Book Excerpts and Blog Articles

Free online book excerpts with full text chapters online and free PDF downloads, and the Aussie AI blog, including related articles:

• Width pruning is a type of structured pruning that reduces the "width" of the internal layers of a neural network. In early models, the "width" was the number of internal neurons in the hidden layers, although in newer Transformers, it often refers to the number of attention heads. Like all types of pruning, the goal is model compression and faster inference by avoiding computation associated with the pruned parts. ... more about Width pruning »
• Parameter sharing, also called "weight sharing", is the use of the same parameters by different structures of the Transformer. Parameter sharing and pruning are similar techniques, both being forms of model compression, but they are not the same. Pruning avoids doing some computations, whereas parameter sharing still does all the computations, but with shared parameters (reducing the total number of stored weights).... more about Parameter sharing »

Research on Fused Attention

Research papers on fused attention heads:

• Jielin Jiang, Hongxiang Xu, Xiaolong Xu, Yan Cui, Jintao Wu. Transformer-Based Fused Attention Combined with CNNs for Image Classification. Neural Processing Letters (2023). https://doi.org/10.1007/s11063-023-11402-1 https://link.springer.com/article/10.1007/s11063-023-11402-1
• Xiang Zhang and Lijun Yin, Mar 2022, Multi-Modal Learning for AU Detection Based on Multi-Head Fused Transformers, https://arxiv.org/pdf/2203.11441.pdf
• NVIDIA, 2023, fused_attn.h: Enums and functions for fused attention, https://nvidia.github.io/TransformerEngine/api/c/fused_attn.html
• Chen, Yilong ; Zhang, Linhao ; Shang, Junyuan ; Zhang, Zhenyu ; Liu, Tingwen ; Wang, Shuohuan ; Sun, Yu, June 2024, DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusion, https://arxiv.org/abs/2406.06567 https://ui.adsabs.harvard.edu/abs/2024arXiv240606567C/abstract
• Hui Wu, Yi Gan, Feng Yuan, Jing Ma, Wei Zhu, Yutao Xu, Hong Zhu, Yuhua Zhu, Xiaoli Liu, Jinghui Gu, Peng Zhao, 23 Jun 2024 (v2), Efficient LLM inference solution on Intel GPU, https://arxiv.org/abs/2401.05391 (Disaggregated the KV cache between prefill and decoding tokens, since theh KV cache size is known for prefill, thereby reducing memory fragmentation, and also applying kernel fusion to several modules include the scaled dot product attention.)
• Agarwal, Saurabh, Aug 2024, Minimizing Data Movement in Machine Learning Systems, Ph.D. Thesis, Computer Sciences, University of Wisconsin--Madison, https://digital.library.wisc.edu/1711.dl/MKLIYRPB24A5R9D https://search.library.wisc.edu/digital/AMKLIYRPB24A5R9D PDF: https://asset.library.wisc.edu/1711.dl/QXSTVAIXECHQA8L/R/file-62b54.pdf?dl https://www.proquest.com/openview/c1ae2a92106d7ec681a7296cd163e0c1/1 (Dataflow optimization in training and also "clustered head attention" for memory-efficient inference, an extension of multi-head attention similar to layer-wise head fusion/pruning.)
• Janek Haberer, Ali Hojjat, Olaf Landsiedel, 26 Sep 2024, HydraViT: Stacking Heads for a Scalable ViT, https://arxiv.org/abs/2409.17978 https://github.com/ds-kiel/HydraViT
• Anonymous authors, Oct 2024, Forget the Data and Fine-Tuning! Just Fold the Network to Compress, https://openreview.net/pdf?id=W2Wkp9MQsF
• Aaron Klein, Jacek Golebiowski, Xingchen Ma, Valerio Perrone, Cedric Archambeau, 3 May 2024, Structural Pruning of Pre-trained Language Models via Neural Architecture Search, https://arxiv.org/abs/2405.02267 (Post-training structured pruning of sub-networks based on NAS, also with weight sharing and several different focus areas of pruning including attention heads, FFNs, and layers.)
• Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, Zijia Chen, Ameya Sunil Mahabaleshwarkar, Shih-Yang Liu, Matthijs Van Keirsbilck, Min-Hung Chen, Yoshi Suhara, Yingyan Lin, Jan Kautz, Pavlo Molchanov, 20 Nov 2024, Hymba: A Hybrid-head Architecture for Small Language Models, https://arxiv.org/abs/2411.13676
• Saurabh Saini, Kapil Ahuja, and Akshat S. Chauhan, 20 Sep 2025, Block-Fused Attention-Driven Adaptively-Pooled ResNet Model for Improved Cervical Cancer Classification, https://arxiv.org/abs/2405.01600
• Fatih Ilhan, Gaowen Liu, Ramana Rao Kompella, Selim Furkan Tekin, Tiansheng Huang, Zachary Yahn, Yichang Xu, Ling Liu, 25 Mar 2026, Attention-aware Inference Optimizations for Large Vision-Language Models with Memory-efficient Decoding, https://arxiv.org/abs/2603.23914

More Attention Research Topics

Related LLM research areas for long context optimization of the attention methods include:

• Attention optimization (main page)
• Local attention
• Linear attention
• Sparse attention
• Multi-Head Attention (MHA)
• Muti-Query Attention (MQA)
• Group-Query Attention (GQA)
• Flash attention
• Paged attention

Other topics in attention research:

• Low-rank matrix attention
• Medusa attention
• Block attention
• Cross attention
• Fused head attention
• Hybrid local-global attention
• FFT attention
• QKV computation optimizations
• Additive attention
• Multiplicative attention
• Graph attention
• Chunked attention
• Attention sink
• Attention steering
• Bilinear attention
• Attention-free methods
• Mixture-of-Heads (MOH) Attention (MoE+MHA)
• Star attention
• Ring attention

More AI Research

Read more about:

• Inference Optimizations
• Loop Optimizations
• Code Optimizations
• « Research Home