New OpenAI Inference Optimization is Precalculation

  • June 30th, 2026
  • by David Spuler, Ph.D.

What is the New Technique from OpenAI?

There's been a lot of articles recently reporting an OpenAI advance in inference optimization that claims to reduce GPU compute costs by 50% or more. The facts seem to be:

  • "New technique" that has been discovered.
  • Inference optimization (not training).
  • Text-based queries (not images or video).
  • Halves compute costs (50% or more).
  • General query answering (public ChatGPT users).
  • Limited to "guest" users (who aren't logged in or paid users of ChatGPT).

What's the new technique?

Candidate Recent Techniques and Breakthroughs

To be clear, I have no inside information about the technique, and am just looking at the available information and combining some logic with my expertise. I've been researching this area for years, and have compiled a list of over 700 LLM inference optimization techniques and recently published a book on LLM inference optimization research breakthroughs.

Let's look at some possibilities (including some suggested in various articles):

Shorter system prompt. If OpenAI could cut half the tokens out of its system prompt, then that surely cuts a lot of short-query costs. However, this is well-known and OpenAI would certainly have been tweaking the wording of its system prompt, being well aware of the cost of every extra token in the system prompt.

System prompt KV caching. On the other hand, the cost of each token in the system prompt is partially resolved by precalculating the "KV cache" of the system prompt. If you never change the system prompt (except rarely), then the KV cache data never changes, and you use that for every user. But this is not a new breakthrough.

KV pinning. One way to further improve system prompt KV caching is to store it on every server in memory. This avoids the memory cost of loading the cache and/or the network cost of sending the same cache data again and again. However, this technique is known, and also isn't a compute optimization as such (load cost and network traffic).

Conversation History KV pinning. Another speedup is to store the KV cache for not only the system prompt, but for the multi-turn conversation with a user. However, storing the conversation on a server is a well-known technique, both in multi-user modes (e.g., using "sticky sessions" to route users to the same server every time), and in single-user modes for on-device inference. Furthermore, it mainly reduces the network cost of sending KV caches around, rather than compute.

Flash attention-style kernel advancements. The latest research on Flash Attention v4 achieved about 71% of theoretical GPU utilization. Hence, there is room for improvement, and a new attention technique is possible, but I don't see a 50% gain here.

Sparse attention improvements. There's been some massive gains in compute cost for "sparse attention" and especially "token-selection sparse attention" such as DeepSeek Sparse Attention (DSA) and more recent advances in this area. The DSA approach is still quadratic in the cost of its indexer, and there have been some subsequent advances in reducing this indexation cost. Let's leave a new variant here on the list of possibilities.

Distillation. This was suggested in some articles about the OpenAI breakthrough, but it's hardly a new technique. Using model compression combined with routing simple queries to smaller models on some queries is also well-known.

Speculative decoding. This is a long-standing parallelization technique, and there are some recent breakthroughs such as FR-Spec (with Eagle), DFlash (diffusion-based) and DSpark (from DeepSeek). There's also some possibility of doing a large improvement with "guided decoding" but that's specific to programming use cases. However, I don't see that improving the acceptance rates would reduce overall compute by 50%, because overall acceptance rates aren't that low to begin with that they'd double.

Vocabulary trimming ("shortlisting"). The idea here is that models have vocabularies of 50,000-150,000 distinct tokens (e.g., words) in the models' "vocabulary", but there's only about 5,000 common words used in English. Hence, if you just do the compute for those 5,000 words (on a "shortlist") rather than 150,000 computations, then you've reduced compute by 30 times. Unfortunately, models don't actually work this way, and most of the compute is done on an internal "hidden dimension" of weight channels, which is usually 4,096 or 8,192. Reducing the tokens doesn't change that dimension and doesn't reduce most compute. The only places where vocab trimming cuts compute is: (a) at the very start, (b) at the very end ("decoding" or "LM Head"), and (c) a few obscure places including FR-Spec in Eagle speculative decoding, and the "LM Head" of early exiting layers. Hence, vocabulary trimming is a useful optimization technique but only in combination with something else, such as precalculation, which I explore below.

Early exiting of layers. If you could skip half of the model layers, then that would be a 50% reduction in compute. Early exiting has been well-known for years, but suffers from accuracy problems. A new technique would have to resolve that problem without increasing cost. A related technique is "self-speculative decoding" that makes use of internal model layers as part of speculative decoding's verification mode. You can certainly reduce compute with the early exiting technique, whether in speculative decoding or separately. For example, you could use vocabulary trimming in the LM Head for confidence calculations, and thereby reduce the extra cost of early exiting to very low, but the accuracy concerns remain.

Overall, none of these seem likely. This main reason that none of these techniques are the one is simple: they're not new techniques.

New Technique Idea: System Prompt Compute Precalculation

Precalculation of system prompt attention compute is a candidate. Let's take the precalculation of the KV cache for system prompt one step further. This KV cache data is easy to precompute and then store for "KV pinning." But the attention calculation is still using each token's KV cache in the compute.

Can we precompute that?

Let's make some assumptions:

  • 500 word system prompt
  • 50 word user query

Each token has to "pay attention" to each other (prior) token, which means GPU matrix multiplications. The system prompt tokens avoid paying attention to each other via KV cache precomputation. Hence, the total cost is: 

     50*500 computations

It's actually about half that, because tokens use "causal masking" and don't peek ahead. But let's stick with this over-estimate for now.

Some obvious ways to speed it up don't work. The compute here depends on:

  • Position of each of the 50 tokens (each word could appear at word 1 or word 50 or anywhere in-between).
  • Tokens at each position (each of the 50,000 to 150,000 words have distinct embedding vectors).

In order to precompute all of that, we have: 

   50*50,000 = 2.5 million output vectors

Note that we have not precomputed all compute, even if we achieve this precomputation. What we have precomputed is that attention of the query tokens to the system prompt tokens, and the intra-system prompt attention is also handled via system prompt KV caching and pinning. However, each of the 50 user query tokens also has to "attend" to the other query tokens, but this is much smaller: 

    50 * 50 tokens = 2,500 computations

The output vector is an "activation vector" of dynamic results (now precalcuated and stored). Each precalculated vector has the size of the hidden dimension (e.g., 4,096/8,192 or 4k/8k) of BF16 numbers (16-bit floating-point numbers of size 2 bytes each). Hence, storing this all as a cache: 

    50*50,000*4,096*2 = 20,480,000,000 bytes ~ 20,000 Megabytes ~ 20 Gigabytes

That's quite reasonable! It might even fit into GPU memory: Blackwell has 96 GB of VRAM, and even a desktop PC RTX 4090 has 24GB of GPU RAM. But we need to store model weights and activation outputs in the GPU RAM as well. Even so, we could store a lot of this data "near" the GPU in other fast HBM RAM.

But here's where we can greatly reduce it: token shortlisting. If we only precompute for user queries that use the 5,000 most common tokens (and dynamically compute any other tokens), then that's a ten-fold reduction in precomputed memory we need down to 2GB for a 50-token precompute of 10% shortlisted tokens. Hence, there's room for a longer query precompute range or a higher percentage of shortlisted tokens.

Limitations and Assumptions. That all sounds great, but it's assuming a lot:

  • Storing only one final "hidden state" or "activations" vector.
  • No longer user queries (e.g., editing use cases).
  • No "augmentation" (e.g., plugins or internet searches)
  • No tool usage
  • No conversational history

My calculations are based on only storing a "hidden state" result vector for 5,000 tokens in 50 distinct positions. However, the intra-query attention will need KV caches so as to finalize the compute for the 50*50 computations. Hence, the precalculation idea needs to save the KV cache data. This is also available during the precalculation, but it blows up my memory size estimates.

The other problem is the 50 token estimate for user queries (excluding the system prompt). A significant portion of user questions are not short, but editing or review type questions are around 10% of queries to ChatGPT (see OpenAI's own research by Chatterji et al., Sep 2025). The longer queries are often "review my document" or "debug my code" type queries. That's a lot more than 50 tokens, although there's still the other 90% of queries that may be shorter.

Furthermore, even if the user's prompt is short, answering the myriad of varying queries to ChatGPT requires information that goes beyond the model parameters. These are bigger chunks of text that are added to the prompt, and therefore are token positions requiring compute. It's more than 50 tokens.

One possible solution for these extra tokens is again precalculation (or caching). There's a lot of research in "non-prefix KV caching" and "position-independent caching" that involves stitching together precomputed KV caches from unrelated blocks of text. Any internet searches or tool outputs could theoretically be precomputed (for common searches) or cached after the first time. A lot of this research involves precomputing the KV cache data for "chunks" in RAG architectures, but that's a lot more static and less variable than internet searches.

Nevertheless, maybe a significant percentage of internet searches or tool output requests are overlapping and could be cached or precomputed. Note that these caches could pay attention to the system prompt after caching or precomputation. To keep the precomputation of the 50 query tokens alive, the augmented data could be appended rather than prepended, or could be always put in a known position.

But that's assuming that token positions are absolute. Recently, there's been research on "KV shifting" or "KV correction" where a KV cache can be moved to a different token position, which could be relevant here. For example, techniques are where you can store KV cache data without RoPE positional encoding, or alternatively, you can do "RoPE reversal" to take that out later. Hence, precomputing for 50 user query tokens could still be used in combination with precomputed bigger chunks from plugins or tools and then KV shifted.

Finally, another problem with my 50 token estimate is that the second and later questions in a session have a conversational history. This includes the user's prior query, any tokens from the plugin or tool data sources, and ChatGPT's prior answer. All of those will blow up my estimate of 50 tokens.

So, it's possible that precomputation of the attention compute could work for 50 tokens (or slightly longer) for a subset of queries that only require parametric knowledge (and only on the first query). Separating these parameter-only queries is not actually that difficult (it's already routing to plugins, search, or tools), but raises the concerns: (a) what percentage of queries is that, and (b) does that give us a massive 50% speedup overall?

Anyway, that's my theory about the OpenAI breakthrough: precalculation.

References

  1. Stephanie Palazzolo, June 2026, OpenAI Discovers New Way to Cut Inference Costs in Half, The Information, https://www.theinformation.com/newsletters/ai-agenda/openai-discovers-new-way-cut-inference-costs-half
  2. Sara Quinn, June 30, 2026, OpenAI just slashed guest ChatGPT running costs by over fifty percent, https://www.singularitymoments.com/content/openai-just-slashed-guest-chatgpt-running-costs-by-over-fifty-percent/
  3. David Spuler, May 30, 2026, LLM Inference Optimization: State-of-the-Art Research, table of contents, https://www.amazon.com/dp/B0H3FKR39T.
  4. Ted Zadouri, Markus Hoehnerbach, Jay Shah, Timmy Liu, Vijay Thakkar, Tri Dao, 5 Mar 2026, FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling, https://arxiv.org/abs/2603.05451
  5. David Spuler, June 16, 2026 (updated), 500+ LLM Inference Optimization Techniques, https://www.aussieai.com/blog/llm-inference-optimization
  6. Aaron Chatterji, Tom Cunningham, Christopher Ong, Carl Shan, David Deming, Zoe Hitzig, Kevin Wadman, Sep 15, 2025, How People Use ChatGPT, https://openai.com/index/how-people-are-using-chatgpt/, https://cdn.openai.com/pdf/a253471f-8260-40c6-a2cc-aa93fe9f142e/economic-research-chatgpt-usage-paper.pdf
  7. David Spuler, June 25, 2026, C++ Caching & Precomputation: Data Structure and Algorithm Optimizations, https://www.amazon.com/dp/B0H6M2W9J7