Aussie AI

In-Context Learning (ICL)

Last Updated 29 August, 2025

by David Spuler, Ph.D.

What is In-Context Learning (ICL)?

In-Context Learning (ICL) is the general idea of using knowledge from the LLM's input prompt in answering a question. This doesn't sound very revolutionary these days, since we're all familiar with RAG architectures, but there was a time when it was a novel concept. When researchers put all their energy into pre-training the parametric knowledge of a model, it wasn't immediately obvious that it could be "augmented" with extra facts, just by putting them into the input string.

After all, the RAG technique itself was once an unproven research paper. The authors of the first RAG paper have gone on the record saying that, if they'd known how popular it would become, they would have chosen a better name!

Augmentation of knowledge via extra context tokens in the middle of the prompt is no longer new bananas. ICL is the underpinning idea behind various LLM prompt augmentation methods:

Research on ICL

Research papers on ICL include:

João Monteiro, Étienne Marcotte, Pierre-André Noël, Valentina Zantedeschi, David Vázquez, Nicolas Chapados, Christopher Pal, Perouz Taslakian, 23 Apr 2024, XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference, https://arxiv.org/abs/2404.15420
Andrea Matarazzo, Riccardo Torlone, 3 Jan 2025, A Survey on Large Language Models with some Insights on their Capabilities and Limitations, https://arxiv.org/abs/2501.04040 (Broad survey with many LLM topics covered from history to architectures to optimizations.)
Tong Xiao, Jingbo Zhu, 16 Jan 2025, Foundations of Large Language Models, https://arxiv.org/abs/2501.09223 (Huge 230 page paper on many topics such as training, prompting, alignment, and long context.)
Son, M., Won, Y.-J., & Lee, S. (2025). Optimizing Large Language Models: A Deep Dive into Effective Prompt Engineering Techniques. Applied Sciences, 15(3), 1430. https://doi.org/10.3390/app15031430 https://www.mdpi.com/2076-3417/15/3/1430
Fabio Matricardi, Jan 18, 2025, How a Small Language Model Can Achieve 100% Accuracy: In Context Learning is Underrated — ICL is the secret key to reach performance boosting — teach to an AI how to say “I don’t know” — part 2, https://generativeai.pub/how-a-small-language-model-can-achieve-100-accuracy-323a789ffa83
Xiaoran Liu, Ruixiao Li, Mianqiu Huang, Zhigeng Liu, Yuerong Song, Qipeng Guo, Siyang He, Qiqi Wang, Linlin Li, Qun Liu, Yaqian Zhou, Xuanjing Huang, Xipeng Qiu, 24 Feb 2025, Thus Spake Long-Context Large Language Model, https://arxiv.org/abs/2502.17129 (Impressive survey of many techniques to improve efficiency and accuracy of long context processing in both inference and training, covering text, video and multimodal models.)
Benoit Dherin, Michael Munn, Hanna Mazzawi, Michael Wunder, Javier Gonzalvo, 21 Jul 2025, Learning without training: The implicit dynamics of in-context learning, https://arxiv.org/abs/2507.16003
Jathin Korrapati, Patrick Mendoza, Aditya Tomar, Abein Abraham, 13 Aug 2025, Can Transformers Break Encryption Schemes via In-Context Learning?, https://arxiv.org/abs/2508.10235
Shugang Hao, Hongbo Li and Lingjie Duan, 14 Aug 2025, To Theoretically Understand Transformer-Based In-Context Learning for Optimizing CSMA, https://arxiv.org/abs/2508.09146
Shahriar Golchin, Yanfei Chen, Rujun Han, Manan Gandhi, Tianli Yu, Swaroop Mishra, Mihai Surdeanu, Rishabh Agarwal, Chen-Yu Lee, Tomas Pfister, 22 Jul 2025, Towards Compute-Optimal Many-Shot In-Context Learning, https://arxiv.org/abs/2507.16217
Jihyung Lee, Jin-Seop Lee, Jaehoon Lee, YunSeok Choi, Jee-Hyong Lee, 22 Jul 2025, DCG-SQL: Enhancing In-Context Learning for Text-to-SQL with Deep Contextual Schema Link Graph, https://arxiv.org/abs/2505.19956
Yongyi Yang, Hidenori Tanaka, Wei Hu, 17 Jul 2025, Provable Low-Frequency Bias of In-Context Learning of Representations, https://arxiv.org/abs/2507.13540
Erfan Pirmorad, 20 Jul 2025, Exploring the In-Context Learning Capabilities of LLMs for Money Laundering Detection in Financial Graphs, https://arxiv.org/abs/2507.14785
Xing Shen, Justin Szeto, Mingyang Li, Hengguan Huang, Tal Arbel, 29 Jun 2025, Exposing and Mitigating Calibration Biases and Demographic Unfairness in MLLM Few-Shot In-Context Learning for Medical Image Classification, https://arxiv.org/abs/2506.23298
Shuo Chen, Jianzhe Liu, Zhen Han, Yan Xia, Daniel Cremers, Philip Torr, Volker Tresp, Jindong Gu, 21 Jul 2025, True Multimodal In-Context Learning Needs Attention to the Visual Context, https://arxiv.org/abs/2507.15807
Yijing Lin, Mengqi Huang, Shuhan Zhuang, Zhendong Mao, 20 Jul 2025, RealGeneral: Unifying Visual Generation via Temporal In-Context Learning with Video Models, https://arxiv.org/abs/2503.10406
Hongbo Li, Lingjie Duan and Yingbin Liang, 28 Jul 2025, Provable In-Context Learning of Nonlinear Regression with Transformers, https://arxiv.org/abs/2507.20443
Kacper Kadziolka and Saber Salehkaleybar, 31 Jul 2025, Causal Reasoning in Pieces: Modular In-Context Learning for Causal Discovery, https://arxiv.org/abs/2507.23488
Kwesi Cobbina and Tianyi Zhou, 30 Jul 2025, Where to show Demos in Your Prompt: A Positional Bias of In-Context Learning, https://arxiv.org/abs/2507.22887
Huiyi Chen, Jiawei Peng, Kaihua Tang, Xin Geng, Xu Yang, 30 Jul 2025, Enhancing Multimodal In-Context Learning for Image Classification through Coreset Optimization, https://arxiv.org/abs/2504.14200
Patrik Kenfack, Samira Ebrahimi Kahou, Ulrich A\"ivodji, 1 Aug 2025, Towards Fair In-Context Learning with Tabular Foundation Models, https://arxiv.org/abs/2505.09503
Thomas F Burns, Tomoki Fukai, Christopher J Earls, 4 Aug 2025, Associative memory inspires improvements for in-context learning using a novel attention residual stream architecture, https://arxiv.org/abs/2412.15113
Ruixing Zhang, Bo Wang, Tongyu Zhu, Leilei Sun, Weifeng Lv, 5 Aug 2025, Urban In-Context Learning: Bridging Pretraining and Inference through Masked Diffusion for Urban Profiling, https://arxiv.org/abs/2508.03042
Simon Lepage, Jeremie Mary and David Picard, 5 Aug 2025, Markov Chain Estimation with In-Context Learning, https://arxiv.org/abs/2508.03934
Usman Anwar, Johannes Von Oswald, Louis Kirsch, David Krueger, Spencer Frei, 5 Aug 2025, Understanding In-Context Learning of Linear Models in Transformers Through an Adversarial Lens, https://arxiv.org/abs/2411.05189
Yanshu Li, Yi Cao, Hongyang He, Qisen Cheng, Xiang Fu, Xi Xiao, Tianyang Wang, Ruixiang Tang, 8 Aug 2025, M$^2$IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineering, https://arxiv.org/abs/2504.04633
Hengzhe Zhang, Qi Chen, Bing Xue, Wolfgang Banzhaf, Mengjie Zhang, 8 Aug 2025, LLM-Meta-SR: In-Context Learning for Evolving Selection Operators in Symbolic Regression, https://arxiv.org/abs/2505.18602
Chenrui Liu, Falong Tan, Chuanlong Xie, Yicheng Zeng and Lixing Zhu, 12 Aug 2025, In-Context Learning as Nonparametric Conditional Probability Estimation: Risk Bounds and Optimality, https://arxiv.org/abs/2508.08673
Jaeyeon Kim, Sehyun Kwon, Joo Young Choi, Jongho Park, Jaewoong Cho, Jason D. Lee, Ernest K. Ryu, 12 Aug 2025, Task Diversity Shortens the ICL Plateau, https://arxiv.org/abs/2410.05448
Trevine Oorloff, Vishwanath Sindagi, Wele Gedara Chaminda Bandara, Ali Shafahi, Amin Ghiasi, Charan Prakash, Reza Ardekani, 13 Aug 2025, Stable Diffusion Models are Secretly Good at Visual In-Context Learning, https://arxiv.org/abs/2508.09949
Dake Bu, Wei Huang, Andi Han, Atsushi Nitanda, Taiji Suzuki, Qingfu Zhang, Hau-San Wong, 13 Aug 2025, Provably Transformers Harness Multi-Concept Word Semantics for Efficient In-Context Learning, https://arxiv.org/abs/2411.02199
Chuanliu Fan, Zicheng Ma, Jun Gao, Nan Yu, Jun Zhang, Ziqiang Cao, Yi Qin Gao, Guohong Fu, 17 Aug 2025, ProtTeX-CC: Activating In-Context Learning in Protein LLM via Two-Stage Instruction Compression, https://arxiv.org/abs/2508.12212
Chase Goddard, Lindsay M. Smith, Vudtiwat Ngampruetikorn, David J. Schwab, 18 Aug 2025, When can in-context learning generalize out of task distribution?, https://arxiv.org/abs/2506.05574
Aleksandra Bakalova, Yana Veitsman, Xinting Huang, Michael Hahn, 22 Aug 2025, Contextualize-then-Aggregate: Circuits for In-Context Learning in Gemma-2 2B, https://arxiv.org/abs/2504.00132
Fernando Martinez-Lopez, Tao Li, Yingdong Lu, Juntao Chen, 8 Aug 2025, In-Context Reinforcement Learning via Communicative World Models, https://arxiv.org/abs/2508.06659
Aditya Varre, Gizem Y\"uce, Nicolas Flammarion, 18 Aug 2025, Learning In-context $\pmb{n}$-grams with Transformers: Sub-$\pmb{n}$-grams Are Near-stationary Points, https://arxiv.org/abs/2508.12837
Quan Nguyen and Thanh Nguyen-Tang, 20 Aug 2025, One-Layer Transformers are Provably Optimal for In-context Reasoning and Distributional Association Learning in Next-Token Prediction Tasks, https://arxiv.org/abs/2505.15009