Prefix Tuning

Last Updated 10 August, 2026

by David Spuler, Ph.D.

What is Prefix Tuning?

Prefix tuning is an LLM fine-tuning optimization that uses a vector of prefix tokens with values that are defined during fine-tuning. Hence, prefix tuning is similar to prompt tuning and other types of Parameter-Efficient Fine-Tuning (PEFT).

Prefix Tuning: Book Excerpts and Blog Articles

Free online book excerpts with full text chapters online and free PDF downloads, and the Aussie AI blog, including related articles:

Soft prompts are numeric vectors that represent prompting directions, rather than simple English text prompts. They are used as an alternative to fine-tuning by directly modifying the numbers in dynamic activation computations. Soft prompts are related to techniques such as prompt tuning, prefix tuning, attentiong steering, and activation patching. ... more about Soft prompts »

Research on Prefix Tuning

Research papers on prefix tuning include:

Mengwei Xu, Wangsong Yin, Dongqi Cai, Rongjie Yi, Daliang Xu, Qipeng Wang, Bingyang Wu, Yihao Zhao, Chen Yang, Shihe Wang, Qiyang Zhang, Zhenyan Lu, Li Zhang, Shangguang Wang, Yuanchun Li, Yunxin Liu, Xin Jin, Xuanzhe Liu, 16 Jan 2024, A Survey of Resource-efficient LLM and Multimodal Foundation Models, https://arxiv.org/abs/2401.08092 Project: https://github.com/UbiquitousLearning/Efficient_Foundation_Model_Survey (Broad survey with many optimizations including this topic.)
M Xu, D Cai, W Yin, S Wang, X Jin, X Liu - ACM Computing Surveys, 2024, Resource-efficient Algorithms and Systems of Foundation Models: A Survey, https://dl.acm.org/doi/pdf/10.1145/3706418
Xiang Lisa Li, Percy Liang, 1 Jan 2021, Prefix-Tuning: Optimizing Continuous Prompts for Generation, https://arxiv.org/abs/2101.00190 (Precursor to prompt tuning.)
Andrea Matarazzo, Riccardo Torlone, 3 Jan 2025, A Survey on Large Language Models with some Insights on their Capabilities and Limitations, https://arxiv.org/abs/2501.04040 (Broad survey with many LLM topics covered from history to architectures to optimizations.)
Tong Xiao, Jingbo Zhu, 16 Jan 2025, Foundations of Large Language Models, https://arxiv.org/abs/2501.09223 (Huge 230 page paper on many topics such as training, prompting, alignment, and long context.)
Liu Yang, Ziqian Lin, Kangwook Lee, Dimitris Papailiopoulos, Robert Nowak, 16 Jan 2025, Task Vectors in In-Context Learning: Emergence, Formation, and Benefit, https://arxiv.org/abs/2501.09240
Dan Zhang, Tao Feng, Lilong Xue, Yuandong Wang, Yuxiao Dong, Jie Tang, 23 Jan 2025, Parameter-Efficient Fine-Tuning for Foundation Models, https://arxiv.org/abs/2501.13787
Runjin Chen, Andy Arditi, Henry Sleight, Owain Evans, Jack Lindsey, 29 Jul 2025, Persona Vectors: Monitoring and Controlling Character Traits in Language Models, https://arxiv.org/abs/2507.21509
Zeyu Huang, Tianhao Cheng, Zihan Qiu, Zili Wang, Yinghui Xu, Edoardo M. Ponti, Ivan Titov, 24 Sep 2025, Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling, https://arxiv.org/abs/2507.01679