Oleg Kragel | Senior Technologist
Sandisk

Future of Memory and Storage - Day 2 @ 16:05

When VRAM Isn’t Enough: Storage-Centric Context Management for LLMs

Large language model inference is increasingly constrained by per-session KV cache growth in multi-turn and long-context workloads. Many KV-offload approaches treat SSD as a generic spillover tier and optimize for averages, leading to unpredictable tail latency, rehydration read amplification from fragmentation, SSD-unaware packing that turns continuation into many small reads, and read/write interference that destabilizes QoS.We present a storage-centric study of KV-cache persistence and rehydration using a behavioral simulator that models batched inference pipeline, captures host-staging backpressure, and makes layout/indexing first-class in read planning. We compare request-end flushing vs token-streaming persistence and alternative packing/placement policies that shape the SSD I/O stream. We translate I/O and latency results into SSD requirements (bandwidth, mixed-workload QoS, endurance) and deliver quantified rehydration costs plus actionable KV layout guidelines to reduce fragmentation and stabilize tail latency. Attendees will leave with an actionable framework for reasoning about KV offload tradeoffs and storage design priorities for LLM inference.

Oleg Kragel, Senior Technologist, Sandisk

×

Oleg Kragel / When VRAM Isn’t Enough: Storage-Centric Context Management for LLMs

Oleg Kragel, Senior Technologist,
Sandisk

Oleg is a systems architect at Sandisk with 14 years spanning SanDisk’s acquisition by Western Digital (2016) and Sandisk’s return as an independent company (2025), focused on NAND flash management algorithms and performance analysis. He holds B.S. and M.S. degrees in Computer Science from...

CloseMore