Sandeep Kaipu | Engineering Manager
Broadcom

Future of Memory and Storage - Day 3 @ 13:05

Distributed AI Memory Architectures: Scaling LLM Inference with GPUDirect RDMA and InfiniBand

Modern large language models (LLMs) such as DeepSeek-R1, Llama-3.1-405B, and emerging trillion-parameter architectures are fundamentally constrained by memory capacity, memory bandwidth, and data movement latency. Serving these models increasingly requires distributed inference architectures spanning multiple GPU nodes, where memory hierarchy and interconnect performance become the dominant system bottlenecks.

This session presents a deep technical exploration of how distributed GPU memory systems and high-speed interconnect technologies enable large-scale AI inference. Drawing from a production reference architecture deployed on VMware Private AI infrastructure, the talk examines how GPUDirect RDMA over InfiniBand enables direct GPU-to-GPU memory transfers across nodes while bypassing CPU memory copies and minimizing PCIe overhead.

The session analyzes the architectural building blocks required for scalable AI memory systems, including NVIDIA HGX platforms, NVLink/NVSwitch intra-node GPU fabrics, RDMA-enabled inter-node networking, and distributed GPU memory orchestration across Kubernetes clusters.

Sandeep Kaipu, Engineering Manager, Broadcom

×

Sandeep Kaipu / Distributed AI Memory Architectures: Scaling LLM Inference with GPUDirect RDMA and InfiniBand

Sandeep Kaipu, Engineering Manager,
Broadcom

Sandeep Kaipu is an Engineering Manager at Broadcom. With over 20 years of experience in enterprise AI infrastructure and engineering leadership, he has driven large-scale platform innovations, holds multiple patents, and authored the book “AI Engineering Leadership”. He advises academic...

CloseMore