Yu-Ming Huang | Project department manager
Macronix
Yu-Ming Huang is a project department manager at Macronix with 13 years of experience, focusing on error handling for non-volatile memory and the ongoing development of AI technologies. He received the B.S. degree in Communication Engineering from National Central University, Taiwan, in 2011, and the M.S. degree in Communication Engineering and Ph.D. degrees in Electronics Engineering from National Chiao Tung University, Taiwan, in 2013 and 2019, respectively. He also served as a Technical Program Committee member of Design Automation Conference in 2020 and 2021.
Appearances:
Future of Memory and Storage - Day 1 @ 09:20
Dual-tier SSD Case Study: KV Cache Offloading
KV cache offload to storage has emerged as a key mechanism to accelerate and reduce the cost of LLM inference. By strategically treating high-speed storage as a vast repository of pre-computed model state, inference platforms can instantly recall information for recurring prompts or multi-turn dialogues, drastically reducing the time-to-first-token and cost per token. However, due to the significant KV cache sizes, the massive data volumes generated by long-context models can quickly saturate even the fastest storage during high-concurrency workloads.In this talk, we discuss the storage challenges in involved in maintaining predictable KV cache access time at scale. We also show how a SOTA open-source inference platform can, without any changes, leverage a novel distributed tiered storage system that seamlessly combines a fast, low-latency, and high-endurance storage pool for hot KV cache data with a capacity-optimized pool for cost-effective storage of massive KV cache datasets. The storage system combines an optimized software-defined stack with heterogeneous SSDs and showcases the advantages of a flexible, DT-SSD architecture that exposes its internal data tiering.
Created by
Terrapinn is proud to be a member of isla.
Working together to build sustainable events