Aussie AI Blog

What's New in Speculative Decoding?

  • March 3rd, 2025
  • by David Spuler, Ph.D.

What's New in Speculative Decoding?

Speculative decoding is one of the earliest LLM efficiency improvements that parallelized a lot of decoding steps. And yet, there seems to be a never-ending supply of research papers on the topic of speculative decoding.

So, what's new? Here are some of the more recent research areas:

  • Draft model accuracy — more papers on this, as always; forgive me if I yawn!
  • Multiple parallel draft models — ongoing improvements to this idea.
  • Multi-query prompt lookup decoding — this generalizes prompt lookup decoding to scour not only the current prompt context, but also any previous queries in the history.
  • Distributed speculative decoding — optimal use of speculative decoding when inference processing is distributed over multiple GPUs or multiple servers.
  • Long context speculative decoding — examination of particular optimizations when applying speculative decoding to long context or ultralong contexts.
  • Vision and multimodal speculative decoding — visual tokenization is very different.

Read more about types of speculative decoding.

Free AI and C++ Books

Generative AI programming books:

  1. The Sweetest Lesson: Your Brain Versus AI, November 2025: full text online, free PDF available
  2. RAG Optimization: Accurate and Efficient LLM Applications, June 2025: full text online, free PDF available
  3. Generative AI Applications: Planning, Design and Implementation, November 2024: full text online, free PDF available
  4. Generative AI in C++ (Spuler, March 2024): full text online, free PDF available, table of contents, bonus materials, reference lists, source code

CUDA C++ GPU Programming Books:

  1. CUDA C++ Optimization: Coding Faster GPU Kernels, July 2024: full text online, bonus materials, free PDF available
  2. CUDA C++ Debugging: Safer GPU Kernel Programming, July 2024: full text online, free PDF available

Modern C++ Programming Books

  1. C++ AVX Optimization: CPU SIMD Vectorization, 2025: full text online, free PDF available
  2. C++ Ultra-Low Latency: Multithreading and Low-Level Optimizations, 2025: full text online, free PDF available
  3. Advanced C++ Memory Techniques: Efficiency and Safety, 2025: full text online, free PDF available
  4. Efficient C++ Multithreading: Modern Concurrency Optimization, 2025: free PDF available
  5. Efficient Modern C++ Data Structures: Container and Algorithm Optimizations, 2025: free PDF available
  6. C++ Low Latency: Multithreading and Hotpath Optimizations, 2025: free PDF available
  7. Safe C++: Fixing Memory Safety Issues, Oct 2024: full text online, free PDF available

More AI Research Topics

Read more about: