Elements of Memory Optimization
-
Book Excerpt from "Generative AI in C++" (full text online)
-
by David Spuler, Ph.D.
Elements of Memory Optimization
Before we delve into complex AI memory management optimizations like model compression or kernel tiling, let's look at the fundamental building blocks. How do we represent data in memory inside an AI engine to promote memory efficiency?
Some of the lower-level types of memory optimizations include:
- Contiguous memory blocks
- Linearizing multidimensional matrices/tensors
- C++ memory management optimizations
For the last category, there are also a variety of basic C++ programming techniques to minimize CPU memory usage overall. See chapter 14 for information on C++ memory reduction.