San Chang | Director
Wolley

Future of Memory and Storage - Day 3 @ 08:35

Making Persistent Context Practical on NVMe-oC: Adaptive KV Tiering Across GPU, DRAM, CXL, and SSD

Long-context LLM serving is a persistent-context problem: KV cache grows with context length and concurrency, so state spills beyond GPU memory into a many-tier hierarchy across host DRAM, CXL memory, and SSD. Under load, eviction and refill form a bidirectional KV stream. Keeping GPUs saturated needs overlapping compute and transfer, which depends on pinned DMA staging. That pinned staging budget is fixed by platform constraints, so KV bursts can saturate it first, causing thrash and making many-tier persistence infeasible.

We present a tiering runtime on NVMe-oC hardware that combines CXL memory and SSD, built by extending LMCache and SPDK. We enable persistent-context by adaptively splitting a pinned DMA budget into two non-competing regions: a store buffer for GPU evictions and a prefetch buffer for GPU refills. Hot and cold tagging per KV offload unit guides placement, and very cold blocks bypass staging via direct GPU-to-SSD DMA through our SPDK path. On a 2-GPU system serving 40 to 70B models at high concurrency, we observe 30 to 50% higher throughput on the same hardware.

San Chang, Director, Wolley

×

San Chang / Making Persistent Context Practical on NVMe-oC: Adaptive KV Tiering Across GPU, DRAM, CXL, and SSD

San Chang, Director,
Wolley

Dr. Yu-Ming Chang has served as Director at Wolley since 2016, leading technology and product development in storage-class memory (SCM) controllers and NVMe-over-CXL solutions. Before joining Wolley, he was a manager at the Emerging System Lab of Macronix Taiwan and a technical consultant for the...

CloseMore