Majid Hadian | Principal Engineer
Google

Majid Hadian is Principal Engineer with Google DeepMind. He joined Google in 2016 after his Ph.D. in Mathematics from the Max Planck Institute for Mathematics and a stint as CalTech faculty.

Appearances:

Future of Memory and Storage - Day 3 @ 15:25

Closing Keynote - AI Memory Compression and Quantization

AI workloads, especially KV cache for Transformer based model serving and embedding vectors and their indices for context engineering, are placing growing pressure on memory resources such as DRAM and HBM for holding KV cache data and NAND flash SSDs for holding embeddings and HNSW indices in vector databases. Memory capacity needed to look up embeddings inline with Generative AI conversations can exceed 100 terabytes of DRAM and NAND Flash, and active KV cache can exceed the parameter memory of a 3B Llama model at a 128K token context window. As we move to extract more embeddings to hold business and end user context in our vector databases, and as our multimodal models switch to thousands of dimensions and deal with more than a million tokens for coding tasks, the pressure on memory is driving up costs of AI infrastructure and ultimately subscription charges for AI-enabled services. In this session, we look at two techniques being used to address this issue: an architectural technique from ZeroPoint Technologies that can inline and transparently compress memory data by 2x even at cacheline grain using hardware engines and an algorithmic technique from Google DeepMind team that exploits properties of AI embedding vector data type to compress memory capacity by more than 4x.

Per Stenström, Chief Scientist Officer & Co-Founder, ZeroPoint Technologies