Prefix Sharing

  • Last Updated 20 May, 2026
  • by David Spuler, Ph.D.

What is Prefix Sharing?

Prefix sharing is an LLM inference optimization that shares a common prefix across multiple queries. This is a generalization of KV prefix caching to the case where concurrent queries in a batch API have the same prefix. A common prefix can arise in a multi-query batch in common ways:

  • Common document used with multiple queries
  • Global instructions prepended to every query

There are two main ways to implement prefix sharing:

  • Batched prefix sharing — detecting a common prefix in a group of batch queries for reuse of the prefix computations.
  • Prefix caching — storing the computed KV data in a "prefix cache" datastore, which is used later by any number of queries, whether batched or interactive.

Batched prefix sharing is where a group of LLM queries submitted in a batch have the same prefix. For example, they could all be queries about a particular large document. The common prefix can be used to optimize this group of inference queries. This method has the advantage of not needing a datastore for KV cache data, but has a scope limited to a single multi-query batch job.

The optimization of prefix sharing is similar to KV caching, but it does not need storage for the long-term. Multiple queries with a common prefix can share the prefill computations for that prefix, in particular the KV values computed for that prefix of tokens.

Prefix Sharing: Book Excerpts and Blog Articles

Free online book excerpts with full text chapters online and free PDF downloads, and the Aussie AI blog, including related articles:

Research on Prefix Sharing

Research papers on prefix sharing in a multi-query batch inference job:

  • Zhen Zheng, Xin Ji, Taosong Fang, Fanghao Zhou, Chuanjie Liu, Gang Peng, 29 Nov 2024, BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batching, https://arxiv.org/abs/2412.03594
  • Vikranth Srivatsa∗, Zijian He∗, Reyna Abhyankar, Dongming Li, Yiying Zhang, 2024, Preble: Efficient Distributed Prompt Scheduling for LLM Serving, University of California, San Diego, https://escholarship.org/content/qt1bm0k1w0/qt1bm0k1w0.pdf (Evalulates prompt sharing including full inference cache or a partial prefix-based computation of a global KV cache for the prefill phase. Also schedules GPUs based on prefill versus decoding phase requirements.)
  • Bingli Liao, Danilo Vasconcellos Vargas, 13 Jul 2024, Beyond KV Caching: Shared Attention for Efficient LLMs, https://arxiv.org/abs/2407.12866 (Layerwise weight sharing in attention.)

What is Prefix Caching?

Prefix caching is the LLM optimization of storing the KV data values from a prefix token sequence for use in a later query. The storage of a prefix KV cache to optimize LLM inference is a special case of prefix sharing. The computed KV data from prefill of a common prefix can be stored on disk or in memory to be later used by LLM inference queries with a common prefix.

Implementing a prefix cache of KV data is more complicated than a simple batched query prefix sharing method. Extra infrastructure is needed:

  • Datastore of KV data mapped to token prefixes.
  • Network transmission of KV data between multiple GPUs if there is a cache hit.
  • Each query needs to lookup its token sequence in the index of prefix token sequences.

Nevertheless, prefix caching is a powerful optimization being used in several industry inference platforms. When they're talking about "cached tokens" then it's almost certainly doing prefix caching behind the scenes. Prefixes arise naturally in several AI use cases:

  • Conversational history in every chatbot session is a prefix from a prior query.
  • Prepended global instructions or "hidden prompts" of any LLM engine.
  • Same document used repeatedly across multiple queries (e.g., same RAG chunk or internet document search).

The main limitation of a prefix cache is in RAG architectures. Because a retriever may return multiple RAG chunks, in any order, there is not necessarily a common prefix for each RAG chunk in a query. There are various ways to address this, but prefix caching is less effective with RAG.

Research on Prefix Caching

Research papers on KV prefix caching:

KV Caching Optimizations

The main purpose of the prompt processing or prefill phase of inference is to generate the KV cache for use in the decoding phase. Hence, if you can pre-compute and cache the KV cache data, then the need for the prefill phase disappears!

Various types of caching in Transformers include:

More AI Research

Read more about: