Aussie AI

Soft Prompts

Last Updated 21 August, 2025

by David Spuler, Ph.D.

What are Soft Prompts?

Soft prompts are numeric vectors that represent prompting directions, rather than simple English text prompts. They are used as an alternative to fine-tuning by directly modifying the numbers in dynamic activation computations. Soft prompts are related to techniques such as prompt tuning, prefix tuning, attentiong steering, and activation patching. The general classes of algorithms that work directly on model numbers in "latent space" (embeddings) include mechanistic interpretability and representation engineering.

Research on Soft Prompts

Research papers include:

Y Cong, 2024, Research for Enhancing Processing and Computational Efficiency in LLM, 2024 2nd International Conference on Image, https://www.atlantis-press.com/article/126004157.pdf
Zongqian Li, Yinhong Liu, Yixuan Su, Nigel Collier, 17 Oct 2024 (v2), Prompt Compression for Large Language Models: A Survey, https://arxiv.org/abs/2410.12388
Tong Xiao, Jingbo Zhu, 16 Jan 2025, Foundations of Large Language Models, https://arxiv.org/abs/2501.09223 (Huge 230 page paper on many topics such as training, prompting, alignment, and long context.)
Huanxuan Liao, Shizhu He, Yupu Hao, Xiang Li, Yuanzhe Zhang, Jun Zhao, Kang Liu, Jan 2025, SKIntern: Internalizing Symbolic Knowledge for Distilling Better CoT Capabilities into Small Language Models, Proceedings of the 31st International Conference on Computational Linguistics, pages 3203–3221 January 19–24, 2025, https://aclanthology.org/2025.coling-main.215.pdf
Ruijun Feng, Hammond Pearce, Pietro Liguori, Yulei Sui, 8 Jan 2025, CGP-Tuning: Structure-Aware Soft Prompt Tuning for Code Vulnerability Detection, https://arxiv.org/abs/2501.04510
Lingzhi Yuan, Xinfeng Li, Chejian Xu, Guanhong Tao, Xiaojun Jia, Yihao Huang, Wei Dong, Yang Liu, XiaoFeng Wang, Bo Li, 7 Jan 2025, PromptGuard: Soft Prompt-Guided Unsafe Content Moderation for Text-to-Image Models, https://arxiv.org/abs/2501.03544
Yingyi Ma, Zhe Liu, Ozlem Kalinli, 9 Dec 2024, Effective Text Adaptation for LLM-based ASR through Soft Prompt Fine-Tuning, https://arxiv.org/abs/2412.06967
Zhepeng Wang, Runxue Bao, Yawen Wu, Jackson Taylor, Cao Xiao, Feng Zheng, Weiwen Jiang, Shangqian Gao, Yanfu Zhang, 20 Sep 2024, Unlocking Memorization in Large Language Models with Dynamic Soft Prompting, https://arxiv.org/abs/2409.13853
Shuai Fu, Xiequn Wang, Qiushi Huang, Yu Zhang, 26 Aug 2024, Nemesis: Normalizing the Soft-prompt Vectors of Vision-Language Models, https://arxiv.org/abs/2408.13979
Robert Belanec, Simon Ostermann, Ivan Srba, Maria Bielikova, 23 Oct 2024 (v2), Task Prompt Vectors: Effective Initialization through Multi-Task Soft-Prompt Transfer, https://arxiv.org/abs/2408.01119
Thilak Shekhar Shriyan, Janavi Srinivasan, Suhail Ahmed, Richa Sharma, Arti Arya, March 2025, SwarmPrompt: Swarm Intelligence-Driven Prompt Optimization Using Large Language Models, Proceedings of the 17th International Conference on Agents and Artificial Intelligence (ICAART 2025), Volume 3, pages 86-93, https://www.scitepress.org/Papers/2025/130903/130903.pdf
Hui Xiang, Jinqiao Shi, Ting Zhang, Xiaojie Zhao, Yong Liu, Yong Ma, 22 Jul 2025, PromptAL: Sample-Aware Dynamic Soft Prompts for Few-Shot Active Learning, https://arxiv.org/abs/2507.16424