Input Compression

Last Updated 11 August, 2026

by David Spuler, Ph.D.

What is Input Compression?

Input compression is an LLM inference optimization that reduces an input text sequence to a shorter sequence of text or tokens for the LLM to process faster. LLM inference efficiency is improved if there are fewer tokens for the LLM to process. For details on the various specific sub-techniques of compressing input text or tokens, see also:

Token reduction
RAG optimizations (e.g., retriever returns fewer chunks, or use smaller chunks)
Multi-step reasoning optimizations
Token pruning (input token pruning)
Dynamic token pruning
Prompt compression
Context compression
Token merging
Token skipping
Token dropping
Zero padding removal
Length pruning

Input Compression: Book Excerpts and Blog Articles

Free online book excerpts with full text chapters online and free PDF downloads, and the Aussie AI blog, including related articles:

Token compression is an LLM inference optimization that reduces an input text sequence to a shorter sequence of text or tokens. There are various methods that can be used before sending the tokens to the LLM, or during the LLM inference processes. Efficiency of inference is improved when there are fewer tokens for the LLM to process... more about Token Compression »
Token folding is an LLM inference optimization that "folds" or merges two or more tokens together. This is a type of "token reduction" that results in fewer overall tokens for the LLM to process. Folding of tokens can occur in prompt preprocessing or dynamically during inference computations ... more about Token folding »
Token dropping is an LLM optimization that reduces token processing by "dropping" some of them. It is similar to "token pruning", but often refers to dropping tokens during the training phase, whereas token pruning is mostly an inference optimization ... more about Token dropping »
Token fusion is an LLM inference optimization that fuses two or more input tokens together. This results in fewer tokens for the LLM to process. Static token fusion occurs prior to LLM processing (during prompt preprocessing), whereas dynamic token fusion occurs during inference (when important tokens can be identified) ... more about Token fusion »
Tokenization: The tokenizer does not receive as much attention in the research literature as other parts of large language models. This is probably because the tokenization phase itself is not a bottleneck in either inference or training, when compared to the many layers of multiplication operations on weights. However, the choice of the tokenizer algorithm, and the resulting size of the vocabulary, has a direct impact on the speed (latency) of model inference ... more about Tokenization »
Token merging is an LLM inference optimization that merges two or more adjacent tokens into a single token. This optimization reduces the number of tokens for a GPU to process, thereby directly improving speed, and reducing compute. Token merging is a type of token reduction, and is closely related to token pruning, which is simply removing tokens from processing. Merging of multiple tokens together can be performed before sending the prompt to the LLM, or dynamically during inference based on analysis of which tokens are important ... more about Token merging »
Token pruning is a type of model "length pruning" that aims to address the cost of processing input sequences and the related embeddings. It is closely related to "embeddings pruning", but is orthogonal to depth pruning (e.g. layer pruning) or width pruning (e.g. attention head pruning) ... more about Token pruning »
Token reduction is an LLM inference optimization method that aims to speed up LLMs by reducing the number of tokens processed. This is relevant to any inference process, but is particularly applicable to optimizing multi-step reasoning algorithms, such as Chain-of-Thought, because their interim reasoning steps generate long sequences of tokens. ... more about Token reduction »
Token selection is the final phase of LLM processing, also called the "decoding algorithm", whereby the output token is chosen. There are numerous variants including greedy decoding, top-k decoding, top-p decoding, and many more. Most models work by outputting one token at a time, but newer models use "multi-token prediction" (MTP) or "multi-token decoding" to output two or more tokens in parallel... more about Token selection »
Token skipping is an LLM inference optimization methods that "skips" the processing of some input tokens. Fewer total tokens to process means less GPU compute cost and faster inference. Variants of this technique include token pruning, token dropping, and token merging ... more about Token skipping »
Vision attention optimization is a speed optimization for the inference in Vision Transformers (ViTs). This uses techniques similar to image attention optimization, such as pruning or merging patches or sections of the images ... more about Vision attention optimization »
Vision token merging is the combining of two or more areas of an image for faster AI processing. Vision models tokenize the images into segments, which must each be processed by the LLM. Merging them together allows the model to process only a single area, with reduced accuracy ... more about Vision token merging »
Vision token pruning is the removal and avoidance of processing portions of an image or video stream. The idea generalizes token pruning in text models, which remove words, to avoid processing some segments of visual images. This is an effective speedup for image processing or machine vision because many of the areas of a large image are not important, and their ... more about Vision token pruning »
Vision token reduction is an LLM inference optimization technique for vision models that involves pruning or merging tokens. Tokenization of vision data is based on image tokenization, and can be optimized in various ways. Vision token pruning is avoiding the processing of redundant or unimportant segments of images, such as recurring blank background regions. ... more about Vision token reduction »
Length pruning is weight pruning on one of the three axes of pruning. The other two axes are width pruning (e.g. attention head pruning) and depth pruning (e.g. layer pruning and early exit). All three types of pruning are mostly orthogonal to each other and can be combined into triple pruning. The main types of length pruning along the "lengthwise" dimension of the inputs are... more about Length pruning »
Prompt compression is the shortening of LLM prompts automatically so that they contain fewer tokens and can be processed more efficiently. This isn't used on short user queries, but there are many other parts of a prompt that are longer, such as the entire conversational history, retrieved documents, or other types of "context" for the prompt. For this reason, this technique is often called "context compression" or "token reduction".... more about Prompt compression »

Research on Input Compression

Some of the general papers on token compression strategies:

Wangbo Zhao, Jiasheng Tang, Yizeng Han, Yibing Song, Kai Wang, Gao Huang, Fan Wang, Yang You, 18 Mar 2024, Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation, https://arxiv.org/abs/2403.11808 (PEFT and adaptive inference and token pruning in Vision Transformers.)
Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, Yan Yan, 25 Mar 2024, LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models, https://arxiv.org/abs/2403.15388 Code: https://llava-prumerge.github.io/ (Compresses input images based on redundant sections.)
Maxim Bonnaerens, Nov 2023, Resource-Efficient Deep Learning for Computer Vision, Ph.D. thesis, Ghent University, https://biblio.ugent.be/publication/01HEMGWENRT8C255N2RD9KAEJC/file/01HEMGZ9JYP8NXPSQJZM14ACT9 (Examines various vision Transformer optimizations including a NAS approached based on building blocks and also combined token pruning/merging for input compression.)
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, Maosong Sun, 3 Aug 2024, MiniCPM-V: A GPT-4V Level MLLM on Your Phone, https://arxiv.org/abs/2408.01800 Code: https://github.com/OpenBMB/MiniCPM-V
Wei Chen, Zhiyuan Li, Shuo Xin, Yihao Wang, 28 Aug 2024, Dolphin: Long Context as a New Modality for Energy-Efficient On-Device Language Models, https://arxiv.org/abs/2408.15518 https://huggingface.co/NexaAIDev/Dolphin (Using vision transformer architecture to process longer text.)
Bosheng Qin, Juncheng Li, Siliang Tang, Yueting Zhuang, 24 Nov 2022, DBA: Efficient Transformer with Dynamic Bilinear Low-Rank Attention, https://arxiv.org/abs/2211.16368
Keda Tao, Can Qin, Haoxuan You, Yang Sui, Huan Wang, 22 Nov 2024, DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models, https://arxiv.org/abs/2411.15024
M Xu, D Cai, W Yin, S Wang, X Jin, X Liu - ACM Computing Surveys, 2024, Resource-efficient Algorithms and Systems of Foundation Models: A Survey, https://dl.acm.org/doi/pdf/10.1145/3706418
Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, Xiuyu Li, Yunhao Fang, Yukang Chen, Cheng-Yu Hsieh, De-An Huang, An-Chieh Cheng, Vishwesh Nath, Jinyi Hu, Sifei Liu, Ranjay Krishna, Daguang Xu, Xiaolong Wang, Pavlo Molchanov, Jan Kautz, Hongxu Yin, Song Han, Yao Lu, 5 Dec 2024, NVILA: Efficient Frontier Visual Language Models, https://arxiv.org/abs/2412.04468
Hao Li, Changyao Tian, Jie Shao, Xizhou Zhu, Zhaokai Wang, Jinguo Zhu, Wenhan Dou, Xiaogang Wang, Hongsheng Li, Lewei Lu, Jifeng Dai, 12 Dec 2024, SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Folding, https://arxiv.org/abs/2412.09604 (Reducing the precision of input images via token folding, with a special decoder at the end to ensure output is high precision.)
OpenAI, Dec 2024, OpenAI o1 and new tools for developers, https://openai.com/index/o1-and-new-tools-for-developers/ ("Lower latency: o1 uses on average 60% fewer reasoning tokens than o1-preview for a given request.")
Guoxuan Chen, Han Shi, Jiawei Li, Yihang Gao, Xiaozhe Ren, Yimeng Chen, Xin Jiang, Zhenguo Li, Weiyang Liu, Chao Huang, 17 Dec 2024 (v2), SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separator, https://arxiv.org/abs/2412.12094 http://sepllm.github.io/
Tianqiao Liu, Zui Chen, Zitao Liu, Mi Tian, Weiqi Luo, 13 Sep 2024, Expediting and Elevating Large Language Model Reasoning via Hidden Chain-of-Thought Decoding, https://arxiv.org/abs/2409.08561 (Compressing the interim token sequences in Chain-of-Thought.)
Yu Kang, Xianghui Sun, Liangyu Chen, Wei Zou, 16 Dec 2024, C3oT: Generating Shorter Chain-of-Thought without Compromising Effectiveness, https://arxiv.org/abs/2412.11664 (Token pruning and prompt compression for Chain-of-Thought.)
Haoran You, Connelly Barnes, Yuqian Zhou, Yan Kang, Zhenbang Du, Wei Zhou, Lingzhi Zhang, Yotam Nitzan, Xiaoyang Liu, Zhe Lin, Eli Shechtman, Sohrab Amirghodsi, Yingyan Celine Lin, 22 Dec 2024, Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers, https://arxiv.org/abs/2412.16822
Fabio Montello, Ronja Güldenring, Simone Scardapane, Lazaros Nalpantidis, 13 Jan 2025, A Survey on Dynamic Neural Networks: from Computer Vision to Multi-modal Sensor Fusion, https://arxiv.org/abs/2501.07451 (Survey of adaptive inference optimizations: early exit, dynamic routing, token skimming.)
J Köpke, A Safan, 2024, Efficient llm-based conversational process modeling, Business Process Management Workshops, https://isys.uni-klu.ac.at/PDF/BPM_2024_paper_1442.pdf (Examines and improves the token costs of prompt strategies in conversational sessions.)
Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, Wangding Zeng, 16 Feb 2025, Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention, https://arxiv.org/abs/2502.11089
Xiaoran Liu, Ruixiao Li, Mianqiu Huang, Zhigeng Liu, Yuerong Song, Qipeng Guo, Siyang He, Qiqi Wang, Linlin Li, Qun Liu, Yaqian Zhou, Xuanjing Huang, Xipeng Qiu, 24 Feb 2025, Thus Spake Long-Context Large Language Model, https://arxiv.org/abs/2502.17129 (Impressive survey of many techniques to improve efficiency and accuracy of long context processing in both inference and training, covering text, video and multimodal models.)
Kele Shao, Keda Tao, Kejia Zhang, Sicheng Feng, Mu Cai, Yuzhang Shang, Haoxuan You, Can Qin, Yang Sui, Huan Wang, 30 Jul 2025 (v3), When Tokens Talk Too Much: A Survey of Multimodal Long-Context Token Compression across Images, Videos, and Audios, https://arxiv.org/abs/2507.20198 https://github.com/cokeshao/Awesome-Multimodal-Token-Compression
Naderdel Piero, Zacharias Cromwell, Nathaniel Wainwright, Matthias Nethercott, 8 Aug 2025, Contextual Reinforcement in Multimodal Token Compression for Large Language Models, https://arxiv.org/abs/2501.16658
John Harvill, Ziwei Fan, Hao Wang, Luke Huan, Anoop Deoras, Yizhou Sun, Hao Ding, 20 Aug 2025, Lossless Token Sequence Compression via Meta-Tokens, https://arxiv.org/abs/2506.00307
Jingyu Xiao and Zhongyi Zhang and Yuxuan Wan and Yintong Huo and Yang Liu and Michael R.Lyu, 15 Sep 2025, EfficientUICoder: Efficient MLLM-based UI Code Generation via Input and Output Token Compression, https://arxiv.org/abs/2509.12159
Dongwon Jo, Jiwon Song, Yulhwa Kim, Jae-Joon Kim, 28 Oct 2025, FastKV: KV Cache Compression for Fast Long-Context Processing with Token-Selective Propagation, https://arxiv.org/abs/2502.01068
Xueyi Chen, Keda Tao, Kele Shao, Huan Wang, 21 Oct 2025, StreamingTOM: Streaming Token Compression for Efficient Video Understanding, https://arxiv.org/abs/2510.18269
Mandar Karhade, MD. PhD., Feb 2026, The End of Token Inflation with DeepSeek OCR-2: How “Context Optical Compression” Re-Engineers Document Processing from First Principles, https://ithinkbot.com/the-end-of-token-inflation-with-deepseek-ocr-2-8acdd653c2cf
Yi Wang, Haofei Zhang, Qihan Huang, Anda Cao, Gongfan Fang, Wei Wang, Xuan Jin, Jie Song, Mingli Song, Xinchao Wang, 23 Mar 2026, Rethinking Token Reduction for Large Vision-Language Models, https://arxiv.org/abs/2603.21701 https://github.com/MArSha1147/MetaCompress
Bowen Peng, Subho Ghosh, Jeffrey Quesnelle, 7 May 2026, Long Context Pre-Training with Lighthouse Attention, https://arxiv.org/abs/2605.06554