FLINT: Efficiently Leveraging High Bandwidth Flash for Capacity-Scalable LLM Inference Acceleration

By Geraldo F. Oliveira ^†, Vamanan Arulchelvan ^†, Jia Xinglei ^‡, Arash Tavakkol ^†, Lukas Cavigelli ^†, Onur Mutlu ^§, Xiangyu Zhu ^†, Renzo Andri ^†, Zhou Ke ^∇, Ahmet Caner Yüzügüler ^†, Mohammad Sadrosadati ^†, Shai Bergman ^†, Ji Zhang ^‡
† Huawei Technologies Switzerland AG
^‡ Huawei Technologies Co., Ltd. 
^§ ETH Zürich
^∇ HUST

Abstract

LLM inference is increasingly constrained by accelerator memory capacity rather than compute throughput. This constraint is especially acute in single-accelerator and small-node inference systems, where limited on-package memory capacity restricts the size of deployable models. HBF is an emerging 3D-stacked NAND flash technology that provides multi-terabyte near-accelerator capacity, making it a promising capacity tier for storing LLM weights. However, existing HBF-based proposals face three adoption challenges: they (1) rely on coarse-grained static prefetching for LLM weights aiming to hide the microsecond-level read latency of the NAND flash device while maximizing HBF's read throughput, (2) expose NAND flash management tasks (e.g., refresh operations) to the accelerator-visible critical inference path, and (3) miss optimization opportunities to specialize and optimize the flash-management mechanisms to the workload behavior.

Our goal is to design an efficient HBF substrate that integrates HBF as a memory-capacity tier alongside HBM while addressing these three challenges. To this end, we propose FLINT, a workload-driven HBF substrate for capacity-scalable LLM inference. FLINT introduces three mechanisms: (1) a hardware burst-buffer controller that dynamically coalesces and pipelines HBF reads aiming to utilize existing NAND flash buffers while sustaining high HBF bandwidth, (2) a phantom-plane refresh mechanism, which removes refresh from the critical inference path by moving refresh-related NAND flash operations outside the read foreground back via low-cost resource duplication, and (3) a read-only FTL, which replaces SSD-class support for arbitrary writes with a compact table that translates logical weight bursts to physical HBF locations.

We evaluate FLINT across dense and mixture-of-experts (MoE) LLM inference models. Our evaluation shows that FLINT improves decode throughput by 1,205×, 2.2×, and 6.2×, and reduces energy consumption by 408×, 1.1×, and 6.8× that of an SSD-equipped GPU system, an HBM-only GPU system, and a prior hybrid HBM+HBF GPU system, respectively. FLINT meets a 50ms time per output token (TPOT) service-level objective (SLO) with 3.1× fewer GPU packages than the HBM-only GPU system. FLINT adds small area cost to an HBF die (3.1%) and HBF base die (3.9 mm2 at 7nm).

To read the full article, click here