Aussie AI Blog
500+ LLM Inference Optimization Techniques
-
Updated: May 12, 2026
-
by David Spuler, Ph.D.
LLM Inference Optimization
We do a lot of research on inference optimization techniques, so here's a very long list of all the techniques about which we have research papers. There's more than 500 (600+ now!), but see the blog post links below if you only want to know about the latest LLM inference techniques.
Update in May, 2026: Some of the newer research areas added to the list:
- Infinite context
- Context extension
- Shallow prefill
- Algebraic integer number system
- Kernel synthesis
- Infill/FIM optimizations
- Infill suffix KV correction
- KV pinning (e.g., for system prompt)
- Whole layer fused kernels
- KV shifting
- KV reversal
- KV correction
- KV layer propagation
- RoPE efficiency optimizations
Update in April, 2026: Some of the newer techniques added to this list:
- Gated attention
- Per-Layer Embeddings (PLE)
- Bulging attention (per-layer weight increases)
- Bulging FFNs (per-layer FFN weight increases)
- Partial RoPE (p-RoPE)
- RoPE rescaling
- KV sharding
- Hybrid MoE (dense FFN)
- Layerwise Pipelined Prefill-Decoding
- KIVI attention
- SnapKV
- K=V (KV sharing)
- Prefill first-layer precomputation
- Prefill last-layer FFN skipping
- Prefill first-token optimizations
- Tool integration optimizations
- Reasoning caching
Update in Feb 2026: As we head into 2026, some of the more recent areas of attention include:
- Native FP4/FP8 in Blackwell/Rubin GPUs
- Fused and shared epilogues/prologues (type of kernel fusion)
- Thread block clusters (Blackwell/Rubin)
Research areas that remain as hot as always include:
- KV cache compression (eviction, KV token pruning, etc.)
- Scheduling (e.g. PD disaggegation)
- Speculative decoding
- Reasoning token reduction (CoT optimization, path pruning, etc.),
- Low-bit quantization kernels
- Outlier handling to improve quantization accuracy,
- Block-scaled quantization vs block-floating point numeric representations
- BF16x9 emulation (optimization of FP32 computations)
- FP64 compute emulation algorithms
And no doubt much more to come in 2026!
Update in March 2025: well, now we're into 2025 and this list has outgrown its title. There are over 600 items on the list below, all of which are related to LLM efficiency. The main change in 2025 is that the recent releases of "reasoning models" has spawned a new area of research in optimizing the efficiency of LLM reasoning algorithms such as Chain-of-Thought.
Free AI C++ books: for more about LLM optimization, read books online or download a PDF:
- Generative AI in C++, David Spuler, March 2024, full text online, free PDF, bonus materials, source code
- CUDA C++ Optimization, David Spuler, June 2024, full text online, bonus materials, free PDF
- C++ Ultra Low Latency, David Spuler, July 2025, full text online, free PDF
Popular articles: additional research articles on faster LLM inference:
- Promising LLM Inference Optimization Techniques (Sep 2025)
- Hot LLM inference optimization research (August 2024)
- LLM reasoning model efficiency
More lists: lots of general efficiency optimization information:
- 100 CUDA Optimization Techniques
- List of 100+ AI Smartness Techniques
- 600+ low latency C++ techniques
LLM Inference Optimizations List
Here's the list! It's over 600 and growing!
-
Reasoning Efficiency Optimization (REO): it's the latest hot research area in 2025!
- Reasoning inference optimization (RIO) (blog article)
- Chain-of-Thought (CoT) optimization
- — CoT token reduction
- — CoT step skipping
- — CoT path reduction
- — CoT early stopping
- — CoT reasoning decoding
- — Constrained CoT
- — Coconut
- — Concise CoT
- — Hidden CoT (interim steps in latent space)
- — CoT prompt sequence optimizations
- — CoT sparsity
- — CoT distillation
- — Long context CoT
- — Small Reasoning Model (SRM)
- — Reasoning tokens
- — Adaptive inference time compute
- — One-step reasoning models (e.g. DeepSeek R1's long answers)
- — Augmented scaffold + Small Reasoning Model
- — Reasoning caching
Inference Modes and Token API optimizations: - "Fast mode" inference (e.g. from OpenAI or Anthropic)
- Cached tokens
- Batched tokens
- Low batch size inference
- Priority batching
- API model routing features
Model compression main subtypes: - Model compression (overview)
- — Pruning (overview)
- — Quantization (overview)
- — Knowledge Distillation (KD)
- — Parameter sharing (weight sharing)
- — Low-rank matrices
- — Small Language Models (SLMs)
- — Data compression algorithms
Pruning main types: - Dynamic pruning
- Hybrid pruning
- Unstructured pruning
- Semi-Structured Pruning
- Structured pruning
Layerwise structured pruning subtypes (depth dimension): - Depthwise structural pruning (overview)
- — Static layer pruning
- — Layer pruning
- — Dynamic layer pruning
- — Layer skipping
- — Layer approximation
- — Shallow decoder architecture
- — Layer reordering
- — Layer Importance
Early exiting (dynamic layerwise pruning): Pruning all the layers from the point of exit: - Early exit (overview)
- — Confidence-based exit policy
- — Patience-based exit policy
- — Entropy-based exit policy
- — Learned exit points or exit policies
- — Early exit KV cache fixes
- — Early exit knowledge distillation
- — Early exit speculative decoding
- — Early exit in training
- — Layer freezing
Width-wise structured pruning subtypes: - Widthwise structural pruning (overview)
- — Attention head pruning
- — Slimmable networks (width pruning)
- — FFN pruning
- — Channel pruning
- — Filter pruning
Length-wise structured pruning subtypes: - Lengthwise structural pruning (longitudinal/input/end-to-end):
- — Token pruning (input pruning)
- — Dynamic token pruning
- — Prompt compression
- — Context compression
- — Token merging
- — Token skipping
- — Token dropping
- — Zero padding removal
- — Token reduction
- — Token compression
- — Input text compression
Model dimension embedding pruning subtypes: - Embedding-dimension pruning
- — Embedding pruning
- — Embedding matrix compression (embedding pruning)
- — Embedding low-rank matrix factorization
- — Unembedding matrix (output embeddings)
Hybrid multi-dimensional pruning: - Multi-dimensional pruning
- — Dual pruning
- — Triple pruning
- — Quadruple pruning
- — 3D CNN model pruning
- — Pyramid inference
Transformer component pruning: - Normalization pruning
- Positional embeddings pruning
- Softmax pruning
- Skip connection pruning (residual connection removal)
Unstructured pruning subtypes: - Unstructured pruning (overview)
- — Magnitude pruning
- — Movement pruning
- — Gradual pruning
Quantization theory and major subtypes: - Post-Training Quantization (PTQ)
- Quantization-Aware Training (QAT)
- Activation Quantization
- Outlier-aware quantization (outlier management)
- Dequantization
Quantization overall algorithms: - Uniform quantization
- Non-Uniform quantization
- Symmetric quantization
- Asymmetric quantization
- GPTQ: Gradient PTQ
- AQLM: Activation-Quantization Low-Bit Method
- SpQR: Sparse Quantized Representations
Integer quantization subtypes: - Integer quantization (overview)
- — Integer-only arithmetic quantization
- — Fixed-point quantization (integer)
- — Low-bit integer quantization (overview)
- — Binary quantization
- — Ternary quantization
- — 2-bit quantization (INT2)
- — 3-bit quantization (INT3)
- — 4-bit quantization (INT4)
- — 5-bit quantization (INT5)
- — 6-bit quantization (INT6)
- — 7-bit quantization (INT7)
- — 8-bit quantization (INT8)
- — 9-bit quantization (INT9)
- — 10-bit quantization (INT10)
- — 11-bit quantization (INT11)
- — 12-bit quantization (INT12)
- — 16-bit INT16 quantization
- — 32-bit INT32 quantization
- — W4A4 quantization
- — W4A4KV4 quantization
Floating-point quantization subtypes: - Floating-point quantization
- — FP4 quantization
- — FP6 quantization
- — FP8 quantization
- — FP16 quantization
- — FP32 quantization
Quantization error mitigation and metrics: - Quantization errors
- Outlier mitigation methods
- — Mean-Squared Error (MSE)
- — SNR degradation
Outlier mitigation in quantization: - AWQ: Activation‑Aware Weight Quantization
Other uncommon quantization subtypes: - Logarithmic power-of-two quantization (bitshift quantization)
- Double bitshift power-of-two quantization
- Division quantization
- Cluster-based quantization (Weight clustering)
- Hashing-based weight clustering
- Dyadic quantization
- Fake quantization
- Simulated quantization
- Stochastic quantization (probabilistic)
Mixed-precision quantization subtypes: - Mixed-precision quantization
Granularity-level quantization subtypes: - Granular quantization (overview)
- — Layerwise Quantization
- — Blockwise Quantization
- — K-quantization
- — Vector quantization
Knowledge distillation subtypes: - Knowledge Distillation (overview)
- — Ensemble Distillation
- — Unnatural instructions (data sets)
- — Dataset Distillation
- — Black Box Distillation
- — White Box Distillation
Parameter/weight sharing subtypes: - Parameter/Weight sharing (overview)
- — Activation sharing
- — Layer fusion
- — Clustering (Weights)
- — Attention head fusion
- — FFN fusion (sharing parameters)
- — KV cache layer fusion (depthwise)
- — KV cache head fusion (widthwise)
Activation function optimizations: - Activation function optimizations (overview)
- — Activation function approximation
- — Integer-only activation functions
- — Fused activation functions (kernel fusion)
- — Fused RELU
- — Fused GELU
- — Fused SwiGLU
- — Activation alternatives/replacements
- — Activation function pruning/removal (bilinear layers)
- — Activation function reordering
Normalization optimization types: - Normalization algorithm optimizations (overview)
- — Approximate normalization
- — Norm reordering (pre-norm/post-norm)
- — Integer-only normalization
- — Normalization alternatives/replacements
- — Fused normalization (e.g. "fused LayerNorm" in kernel fusion)
Softmax optimization types: - Softmax optimizations (overview)
- — Softmax pruning
- — Approximate Softmax
- — Softmax alternatives/replacements
- — Integer-only Softmax
- — Fused Softmax
Feed-Forward Network (FFN) optimization types: - FFN optimizations (overview)
- — FFN pruning
- — FFN approximation
- — Fused add-bias
- — Bias vector pruning
- — FFN sparsity
- — FFN alternatives/replacements
- — Integer-only FFN
- — FFN fusion (shared parameters)
- — Inter-FFN fusion (merging two FFNs)
- — Intra-FFN fusion (with piecewise linear approximations) (merging two linear projections in one FFN)
- — Bias vector addition optimizations
- — Bias vector pruning (no bias!)
- — FFN matrix merging (similar to "intra-FFN fusion")
- — Bulging FFN (per-layer FFN size increases)
MatMul/GEMM optimization types: - MatMul/GEMM kernel optimizations (overview)
- — Faster matrix multiplication (e.g. Winograd, Strassen)
- — Approximate matrix multiplication
- — Transpose cache
- — Fused multiply-add (FMA)
- — Fused transpose
- — Vector dot product optimization
- — Sparse MatMul/GEMM
- — Tiled MatMul
- — Triangular MatMul optimizations (causal masking in attention)
- — Tiled skipping
Positional Encoding optimizations: - Positional encoding optimization (overview)
- — RoPE (Rotary Positional Encoding)
- — Pruning positional encoding (removal/NoPE)
- — Positional encoding approximation
- — Integer-only positional encoding
- — Partial RoPE (p-RoPE)
- — RoPE rescaling
- — Attention with Linear Biases (ALiBi)
- — Relative Attention Biases (RAB)
NAS subtypes: - Neural Architecture Search (NAS)
- — Dynamic NAS
- — Embedding Size Optimization (embeddings NAS)
Platform-specific optimization subtypes: - On-device inference (native phone and PC AI)
- AI Phones
- AI PCs (desktops/laptops)
- Edge device inference (IoT/mobile/PC)
- Hybrid cloud-on-device inference
Decoding algorithm subtypes: - Decoding algorithms (overview)
- — Non-autoregressive decoding
- — Greedy decoding
- — Top-k decoding
- — Top-p decoding
- — Min-P Sampling
- — Flash decoding
- — Beam search decoding
- — Edit decoding
- — Contrastive decoding
- — Approximate top-k algorithms
- — Bidirectional decoding
- — Constrained decoding
Parallel Decoding algorithms: - Parallel decoding
- — Blockwise parallel decoding
- — n-gram parallel decoding
- — Lookahead decoding
- — Medusa decoding
- — Consensus decoding
- — Mutually-guided decoding
- — Multi-token generation
- — Eagle decoding
Speculative decoding subtypes: - Speculative decoding (overview)
- — Generalized speculative decoding
- — Aggressive decoding
- — Lookup decoding
- — Retrieval lookup decoding
- — Prompt lookup decoding
- — Multi-query prompt lookup decoding (across entire LLM history)
- — Self speculative decoding
- — Tree speculative decoding
- — Superposed decoding
- — Hierarchical speculative decoding
- — Heuristic speculative decoding
- — Multi-token speculative decoding
- — Sequential speculative decoding
- — Eagle speculative decoding
- — Redrafting
Parameter Efficient Fine-Tuning (PEFT) subtypes: - PEFT (overview)
- — LoRA
- — Multi-LoRA inference
- — QLoRa (Quantized Low-Rank Adapters)
- — LoRA inference optimizations (load/unload)
- — Prompt Tuning (Extended Vocabulary PEFT)
- — Prefix Tuning
Mixture-of-Experts (MoE): - — Mixture of Experts (MoE)
- — MoE-specific compute optimizations
- — Hybrid MoE (dense FFN)
- — Shared experts
- — MoE routing optimizations
- — MoE gating optimizations
Tool Integration Optimizations: LLMs using tools has gone mainstream, and there is also newer research on speeding it up: - Tool optimizations
- — Tool execution pipelining (overlap with prefill or decode)
- — Speculative tool execution
- — Tool token reduction
- — Concise tool output
- — Disaggregated tool execution
- — Multi-tool parallel execution
Ensemble multi-LLM subtypes: - Ensemble inference (overview of multi-model AI engines)
- — Model selection algorithms
- — Big-little architectures
- — Cascades
- — Collaborative inference
- — Consensus decoding
- — Swarm ensemble architectures
- — Committee ensemble architectures
- — Ensemble averaging
- — Easy-hard queries
- — Submodels (Many-Models-in-One)
- — Distributed Inference
Orchestration, Deployment and Serving: - Cloud inference servers
- Orchestration frameworks
- Scheduling optimizations
- Serving
- Load balancing
- Batching
- Static Batching
- Dynamic Batching
- Continuous batching
- Deployment
- Serverless
- Networking optimizations
- In-flight batching
Attention optimization subtypes: - Attention optimizations (overview)
- — Multi-Head Attention (MHA)
- — Group Query Attention (GQA)
- — Multi-Query Attention (MQA)
- — Sparse attention
- — Local attention
- — Memory-efficient attention algorithms
- — Flash Attention
- — Paged Attention
- — Linear attention
- — Cross attention
- — Tree attention
- — Sliding window attention
- — Approximate attention heads
- — Attention alternatives/replacements
- — Fused MHA
- — Low-rank matrix attention
- — Medusa attention
- — Block attention
- — Cross attention
- — Fused head attention
- — Hybrid local-global attention
- — FFT attention
- — Additive attention
- — Multiplicative attention
- — Graph attention
- — Attention sink
- — Attention steering
- — Bilinear attention
- — Attention-free methods
- — Star attention
- — Ring attention
- — Flex attention
- — Razor attention
- — Contiguous QKV tensor
- — Relative Attention Bias (RAB)
- — Lightning attention
- — Multihead Latent Attention (MLA (DeepSeek)
- — FFT attention
- — Round attention
- Delta attention
- Gated attention
- KIVI attention
- K=V (KV compute sharing)
- Bulging attention (per-layer attention module size increases)
Attention compute optimizations: - — Chunked attention
- — QKV computation optimizations
- — Mixture-of-Heads (MOH) Attention (MoE+MHA)
- Mixture-of-Attention (MoA) (MoE attention)
Long context optimizations (attention): - — Long context models
- — Length generalization
- — Quadratic attention complexity
- — Long RAG
- — Infinite context models
- — Context extension (e.g., extending RoPE/YaRN)
Caching optimizations: - Caching (overview)
- — Inference Cache (text-to-text)
- — Inference cache (global KV caching)
- — Prompt caching
- — Input Similarity-Based Caching (frame skipping in video)
- — Semantic caching (text-to-text)
- — Semantic KV caching
- — Vector database caching
- — Chatbot caching
- — Vector Caching (Vector hashing)
- — Caching vector dot products
- — Caching general theory
KV cache optimizations: - KV Caching (overview)
- — KV cache global (multi-query KV caching)
- — KV cache reuse
- — Global semantic KV caching (difficult!)
- — Context cache (global KV caching)
- — Prefix KV Caching
- — KV cache recomputation with early exit
- — Session KV cache (multi-turn KV caching)
- — Substring/fused/concatenated KV cache (Lengthwise-fused KV caching)
- — Paged KV caching (related to paged attention)
- — KV cache offloading (to CPU)
- — KV sharding
KV cache memory size reduction: - KV cache compression
- — KV cache quantization
- — KV cache sparsity
- — KV cache token pruning
- — Salient token-based KV cache token pruning
- — KV cache eviction policies
- — KV cache layer fusion
- — KV cache layer pruning
- — KV Cache low-rank matrix factorization
- — Cyclic KV cache (Rolling buffer KV cache or circular KV cache)
- — KV cache token merging
- — KV head fusion
- — KV head pruning
- — KV mixed-precision quantization
- — KV context compression
- — KV block pruning
- — SnapKV
Non-Multiplication AI Models: - Zero-Multiplication Models (overview)
- — Binary quantization
- — Ternary quantization
- — 2-bit quantization (INT2)
- — Adder networks
- — Bitshift-add networks
- — Bitshift power-of-2 quantization (logarithmic quantization)
- — Double bitshift quantization
- — Add-as-integer networks
- — Logarithmic Models
- — Bitwise neural networks
- — Diff-squared networks
- — Log-sum-exp (LSE) networks
- — Max-Plus networks
- — Min-Max-Plus networks
- — Morphological networks
- — Trigonometric approximate inference
- — Weightless Neural Networks (WNNs)
- — XNOR networks
- — Hadamard elementwise matrix multiplication models
- — Other addition-related zero-multiplication networks
- — Table lookups replace multiplication
- — Other multiplication-free neural networks
Advanced Number System optimizations: - Advanced Number Systems (overview)
- — Posit number system (PNS)
- — Residue number system (RNS)
- — Dyadic numbers
- — Double-base number system (DBNS)
- — Dynamic number systems
- — Hybrid number systems
- — Tropical algebra (max-plus)
- — MiniMax algebra
- — Multi-dimensional logarithmic number system (MDLNS)
- — Multiple-Base Number System (MBNS)
- — Semi-Logarithmic Number System (SLNS)
- — Lattice algebra
- — Algebraic integer number system
Logarithmic Number System optimizations: - Logarithmic number system (LNS) (overview)
- — End-to-end LNS logarithmic model
- — LNS addition and subtraction
- — LNS in AI models
- — LNS Hardware Acceleration
- — LNS mathematical and algorithmic theory
- — LNS algebra
- — LNS extensions
Prefill phase optimizations: - Prefill optimizations (overview)
- — Chunked prefill
- — Disaggregated prefill scheduling (Phase splitting)
- — Deep prefill, shallow decoder architecture
- — Mini-prefill recomputation
- — Prefill first-layer precomputation
- — Prefill last-layer FFN skipping
- — Prefill first-token optimizations
- — Layerwise Pipelined Prefill-Decoding
- — Shallow prefill
Parallel Programming Optimization Techniques: - Parallelization techniques (overview)
- — Hardware acceleration
- — Hardware-software co-design
- — Vectorization
- — Pipelining (pipeline parallelism)
- — Overlapping (new)
- — Overlapping communications and computation (new)
- — Overlapping rematerialization (new)
- — Overlapping memory access & computation (new)
- — Offloading
- — Partitioning
- — Dataflow optimizations
- — Sharding
- — Overlapping
- — Data parallelism
- — Query parallelism
- — Tensor parallelism
- — Model parallelism
- — Prefetching
- — Speculative execution
- — Sequence Parallelism
- — Skeleton-of-Thought (Query Parallelism)
Hardware Optimizations: - Hardware Acceleration (overview)
- — Software accelerations
- — Hardware-software co-design
- — GPU
- — GPU software platforms
- — Multi-GPU
- — CPU Execution
- — Single Instruction Multiple Data (SIMD)
- — AVX (AVX/AVX-2/AVX-512)
- — ARM NEON
- — Neural Processing Unit (NPU)
- — Overclocking CPU
- — Overclocking GPU
- — Assembly language
RAG Architecture Optimizations: - RAG architectures (overview)
- — RAG cache
- — RAG optimizations
- — RAG retriever datastore indexing
- — Advanced RAG
- — Speculative RAG
- — Reranker in RAG
- — Chunk-specific global KV caching
- — Chunk-specific prefix KV caching
- — RAG Knowledge Graph
- — RAG Ontologies/Taxonomies
- — RAG fusion
- — Mini-RAG (single-document RAG)
Sparsity Optimizations: - Sparsification techniques (overview)
- — Activation Sparsity
- — Dynamic Sparsity
- — Block sparsity
- — Vector sparsity
- — Tensor sparsity
- — Sparse matrix kernels
- — Outlier-aware sparsification
- — N:M sparsity
- — 2:4 sparsity (supported in NVIDIA "Sparse Tensor Cores")
Memory Utilization Optimizations: - Memory optimization techniques (overview)
- — Parameter sharing
- — Model compression
- — Low-bit integer quantization
- — Binary quantization
- — Ternary quantization
- — Layer fusion
- — Recomputation: trading time for space
- — Memory-bound versus CPU-bound
- — Data locality optimization
- — Compute-in-Memory (CIM) architectures (also called PIM)
- — Memory cache management algorithms
- — Kernel operator fusion
- — Flash Inference (FlashInfer)
- — Checkpointing
- — Offloading
- — SSD storage
Numerical representation subtypes: - Floating-point representations (overview)
- — Floating Point Bit Tricks
- — Block floating-point arithmetic
- — Fixed point number system (FXP) optimizations
- — Floating point number system (FLP) optimizations
- — Foating point bitwise arithmetic
- — FTZ/DAZ floating point CPU settings
Kernel optimizations: - Kernel optimizations (overview)
- — Kernel operator fusion (merging, aka "kernel fusion" or "fusion")
- — Fused epilogues (post-MatMul fusion: fused MatMul then activation/normalization)
- — Fused prologues (pre-MatMul fusion: fused activation/normalization then MatMul)
- — Kernel fission (splitting one kernel apart)
- — Kernel tiling
- — Operator reordering
- — Graph operator fusion (Deep learning compilers)
- — Kernel synthesis
- — Whole-layer fused kernels (fused attention with FFN/GLU)
Infill / Fill-in-Middle (FIM) optimizations: - Infill harness optimizations
- Infill prefix reordering
- Infill suffix KV correction
Computation optimizations: - Advanced AI Mathematics
- Approximate activation functions
- Caching / memoization
- Computation reuse
- Precomputation
- Source code precomputation
- Conditional computation
- Approximations
- Integer-only arithmetic quantization
- Weight precomputations
- Zero-skipping
- — Low-Level Zero Skipping
- — High-Level Zero Skipping
- Negative skipping
- Approximate caching
- End-to-End integer inference
- Padding usage
- Incremental inference (new)
- BF16x9 emulation of FP32 computations (on Blackwell GPU)
- FP64 arithmetic emulation using 8-bit/16-bit/32-bit computations
- Thread block clusters (Blackwell/Rubin)
Arithmetic optimizations: - Integer operations
- Addition optimizations
- Bitwise operation tricks
- Approximate addition
- Multiplication algorithms
- Approximate division
- Approximate multiplication
- Bitwise operator inference
- Bitserial operations
- Division optimizations
- Logarithmic approximate multiplication
- Integer Dot Product
- Vector dot product optimization
Advanced matrix algebra optimizations: - Matrix Algebra (overview)
- — Approximate matrix multiplication
- — Butterfly matrices
- — Monarch matrices
- — Sparse matrices (sparsification)
Low-rank matrix optimizations: - Low-rank matrix factorization (overview)
- — Tensor decomposition
- — Tucker decomposition
- — Embedding low-rank matrix factorization
- — KV Cache low-rank matrix factorization
Transformer architectural optimizations: - Transformer architectures (overview)
- — Transformer low-level optimizations (overview)
- — Adaptive Inference (dynamic inference)
- — Integer-only Transformers
- — Approximate Transformers
- — Decoder-Only Architectures
- — Encoder-Only Architectures
- — Encoder-Decoder Architectures
Transformers and LLMs: - Open source models
- Inference frameworks
- Open source frameworks
Next-Generation Transformer architectures: - Next-generation architectures (overview)
- — Hybrid Transformer architectures
- — Newer Transformer architectures
- — BERT (encoder)
- — State Space Models (SSMs)
- — Mamba
- — RWKV
- — Knowledge graph AI architectures
- — Compound AI architectures
- — Large Concept Model (LCM)
General Classes of Optimization Techniques: - Dynamic inference (adaptive inference)
- Skipping
- Heuristics
- Probabilistic optimizations
- Approximate computing
- Code optimizations
- Deep learning compilers
- Incremental algorithms
- Fuzzy logic
- Inference budget (with adaptive inference)
Loop Optimizations: - Loop optimizations (overview)
- — Inference loop optimizations
- — Loop fusion (merging loops)
- — Loop unrolling
- — Loop perforation
- — Loop reordering
- — Loop tiling
- — Loop reversal
- — Loop fission (splitting a loop)
- — Loop interleave
- — Loop interchange
- — Loop coalescing
- — Loop-invariant code motion ("hoisting")
- — Loop distribution
- — Pointer arithmetic
- — Loop peeling (unrolling first iterations)
- — Loop splitting— Loop sentinel
- — Loop collapsing
- — Loop normalization
- — Loop strip mining (Loop sectioning)
- — Loop skewing
- — Loop spreading
Low-Level Coding Efficiency: - Code optimizations (overview)
- — Constant folding
- — Common subexpression elimination
- — Algebraic identities
- — Strength reduction
- — Type consistency
- — Reciprocal multiplication
- — References vs pointers
- — Compile-time optimizations
- — Pointer arithmetic
- — Algorithm-level optimizations
- — Lazy evaluation
- — Memory reduction heuristics
Data Structures for AI optimization: - — Hashing
- — Perfect hashing
- — Look-up tables (LUTs)
- — Bloom filters
- — Trees
- — Tries
- — Bloom filters
- — Bitserial operations
- — Permutation arrays
- — Radix trees (compressed tries) used in Radix Attention for prefix KV caches.
Vector Data Structures: - — Parallel data structures
- — Bit vectors
- — Vector hashing
- — Locality-Sensitive Hashing (LSH)
- — Vector dot product caching
- — Bit signatures (vector algorithm)
- — K-means clustering (vector algorithm)
- — Hyper-Cube (vector algorithm)
Convolution Optimizations in CNNs: - Convolution optimizations (overview)
- — Grouped convolutions
- — Depth-wise separable convolutions
Tokenization and Vocabulary Optimizations: - Tokenization (overview)
- — Tokenizer and model inference latency
- — Semantic tokenization
- — Tokenization for Machine Vision
- — Tokenization of non-English languages
- Vocabulary optimizations:
- — Vocabulary size
- — Lexical shortlisting
- — Vocabulary trimming
- — Vocabulary expansion
- — Dynamic vocabulary pruning
Overall summaries of AI optimizations: - — Deslugging AI engines
- — Accuracy-degrading optimizations
- — Accuracy-retaining optimizations
- — Uncommon inference optimizations
Not Enough?
More inference optimization resources:
- What's Hot in Inference Optimization?
- Inference optimization research overview
- Research Blog
- Patents in inference optimization
AI Books from Aussie AI
|
The Sweetest Lesson: Your Brain Versus AI: new book on AI intelligence theory:
Get your copy from Amazon: The Sweetest Lesson |
|
RAG Optimization: Accurate and Efficient LLM Applications:
new book on RAG architectures:
Get your copy from Amazon: RAG Optimization |
|
Generative AI Applications book:
Get your copy from Amazon: Generative AI Applications |
|
Generative AI programming book:
Get your copy from Amazon: Generative AI in C++ |
|
CUDA C++ Optimization book:
Get your copy from Amazon: CUDA C++ Optimization |
|
CUDA C++ Debugging book:
Get your copy from Amazon: CUDA C++ Debugging |
Free AI and C++ Books
Generative AI programming books:
- The Sweetest Lesson: Your Brain Versus AI, November 2025: full text online, free PDF available
- RAG Optimization: Accurate and Efficient LLM Applications, June 2025: full text online, free PDF available
- Generative AI Applications: Planning, Design and Implementation, November 2024: full text online, free PDF available
- Generative AI in C++ (Spuler, March 2024): full text online, free PDF available, table of contents, bonus materials, reference lists, source code
CUDA C++ GPU Programming Books:
- CUDA C++ Optimization: Coding Faster GPU Kernels, July 2024: full text online, bonus materials, free PDF available
- CUDA C++ Debugging: Safer GPU Kernel Programming, July 2024: full text online, free PDF available
Modern C++ Programming Books
- C++ AVX Optimization: CPU SIMD Vectorization, 2025: full text online, free PDF available
- C++ Ultra-Low Latency: Multithreading and Low-Level Optimizations, 2025: full text online, free PDF available
- Advanced C++ Memory Techniques: Efficiency and Safety, 2025: full text online, free PDF available
- Efficient C++ Multithreading: Modern Concurrency Optimization, 2025: free PDF available
- Efficient Modern C++ Data Structures: Container and Algorithm Optimizations, 2025: free PDF available
- C++ Low Latency: Multithreading and Hotpath Optimizations, 2025: free PDF available
- Safe C++: Fixing Memory Safety Issues, Oct 2024: full text online, free PDF available
More AI Research Topics
Read more about: