Aussie AI

Addition Optimization

Last Updated 27 August, 2025

by David Spuler, Ph.D.

Addition is not the main bottleneck when compared to multiplication, but there are various ways to improve addition, or to use addition in optimization of neural networks.

Addition has a role in optimization techniques such as:

Adder networks (and other types of multiplication-free networks)
Add-as-integer approximate multiplication
Logarithmic models (because logarithms convert multiplications to additions)
Binary quantization or ternary quantization (only requires additions and/or subtractions, or neither if bitwise operators used)
Approximate addition algorithms
Max-Plus networks (using addition and maximum operations)
Log-sum-exp (LSE) networks

Addition operations are a secondary bottleneck in a vanilla Transformer architecture, but they can be reduced for extra speedup. Some ways to only remove the addition operations from Transformers, without changing any multiplications, include:

Bias vector pruning (i.e., don't add any bias vectors in FFNs)
Residual connection pruning (i.e., remove skip connections)

Addition Optimization

Research papers on optimizing the arithmetic addition operation:

O. Yildiz. 2017. Training methodology for a multiplication free implementable operator based neural networks. Master’s thesis, Middle East Technical University. URL https://hdl.handle.net/11511/26664. >>>>>>> .r1599
David Spuler, March 2024, Chapter 53. Arithmetic Optimization Research, Generative AI in C++: Coding Transformers and LLMs, https://www.amazon.com/dp/B0CXJKCWX9
Yunsheng Li, Yinpeng Chen, Xiyang Dai, Dongdong Chen, Mengchen Liu, Lu Yuan, Zicheng Liu, Lei Zhang, Nuno Vasconcelos, MicroNet: Improving Image Recognition with Extremely Low FLOPs, 2021, https://ieeexplore.ieee.org/abstract/document/9857393 PDF: https://openaccess.thecvf.com/content/ICCV2021/papers/Li_MicroNet_Improving_Image_Recognition_With_Extremely_Low_FLOPs_ICCV_2021_paper.pdf
Sue Ann Campbell, Delay independent stability for additive neural networks, January 2001, Differential Equations and Dynamical Systems 9(3), https://www.researchgate.net/publication/268246122_Delay_independent_stability_for_additive_neural_networks PDF: https://www.math.uwaterloo.ca/~sacampbe/preprints/deds01.pdf
Xu Y, Xu C, Chen X, Zhang W, Xu C, Wang Y. Kernel based progressive distillation for adder neural networks. 2020 arXiv preprint arXiv:2009.13044 http://arxiv.org/abs/2009.13044
Vincenzo Liguori, 9 Jun 2024, Procrastination Is All You Need: Exponent Indexed Accumulators for Floating Point, Posits and Logarithmic Numbers, https://arxiv.org/abs/2406.05866
Xi Zhang, Huihui Liu, Junrui Xi, Menglu Chen, Tao Zhu, 30 Mar 2025, High-Dimensional Evolutionary Algorithm Based Design of Semi-Adder, https://arxiv.org/abs/2503.23322
Zhenhua Xu, Qichen Liu, Zhebo Wang, Wenpeng Xing, Dezhang Kong, Mohan Li, Meng Han, 25 Aug 2025, Fingerprint Vector: Enabling Scalable and Efficient Model Fingerprint Transfer via Vector Addition, https://arxiv.org/abs/2409.08846

Approximate Addition Algorithms

Early papers examined the use of approximate addition. This is probably only of interest to hardware designers now.

V. Gupta, D. Mohapatra, S.P. Park, A. Raghunathan, “IMPACT: IMPrecise adders for low-power approximate computing”, International Symposium on Low Power Electronics and Design (ISLPED), pp. 409–414, 2011, https://dl.acm.org/doi/10.5555/2016802.2016898
V. Gupta, D. Mohapatra, A. Raghunathan, K. Roy, “Low-Power Digital Signal Processing Using Approximate Adders”, IEEE Transaction on CAD of Integrated Circuits and Systems 32(1): 124-137, 2013, https://dl.acm.org/doi/10.1109/TCAD.2012.2217962
M. Shafique, W. Ahmad, R. Hafiz, J. Henkel, “A Low Latency Generic Accuracy Configurable Adder”, IEEE/ACM Design Automation Conference (DAC), 2015, https://ieeexplore.ieee.org/abstract/document/7167270
R. Ye, T. Wang, F. Yuan, R. Kumar, Q. Xu, “On reconfiguration-oriented approximate adder design and its application”, International Conference on Computer-Aided Design (ICCAD), pp.48-54, 2013, PDF: https://www.cse.cuhk.edu.hk/~qxu/ye-iccad13.pdf
J. Miao, K. He, A. Gerstlauer, M. Orshansky, “Modeling and synthesis of quality-energy optimal approximate adders”, International Conference on Computer Aided Design (ICCAD), pp. 728-735, 2012, https://ieeexplore.ieee.org/document/6386754
A. B. Kahng, S. Kang, “Accuracy-configurable adder for approximate arithmetic designs”, IEEE/ACM Design Automation Conference (DAC), pp.820-825, 2012, https://ieeexplore.ieee.org/document/6241600
S. Mazahir, O. Hasan, R. Hafiz, M. Shafique, J. Henkel, “An Area-Efficient Consolidated Configurable Error Correction for Approximate Hardware Accelerators”, ACM/EDAC/IEEE 53rd Design Automation Conference (DAC), 2016, https://ieeexplore.ieee.org/document/7544339
N. Zhu, W.-L. Goh, K.-S. Yeo, “An enhanced low-power high-speed Adder for Error-Tolerant application”, 12th International Symposium on Integrated Circuits (ISIC), 2009, https://ieeexplore.ieee.org/document/5403865
Ryu, H. Kim, W. Yi and J.-J. Kim, "BitBlade: Area and energy-efficient precision-scalable neural network accelerator with bitwise summation", Proc. 56th Annu. Design Autom. Conf., pp. 1-6, Jun. 2019. https://ieeexplore.ieee.org/document/8807054
Ao Ren, Ji Li, Zhe Li, Caiwen Ding, Xuehai Qian, Qinru Qiu, Bo Yuan, Yanzhi Wang, 2017, "SC-DCNN: Highly-scalable deep convolutional neural network using stochastic computing", ACM SIGPLAN Notices, vol. 52, no. 4, pp. 405-418, 2017. https://arxiv.org/abs/1611.05939 (Stochastic method with multiplication and addition approximations via AND gates and multiplexers.)
Salar Shakibhamedan, Amin Aminifar, Nima TaheriNejad, Axel Jantsch, 2024, EASE: Energy Optimization through Adaptation — A Review of Runtime Energy-Aware Approximate Deep Learning Algorithms, https://eclectx.org/Publications/2024_M13.pdf (Survey paper on techniques for adaptive inference with a focus on approximations of inference, including loop performance, stochastic algorithms, approximate arithmetic, quantization, pruning and low-rank.)