Custom HBM as a System: Co-designing the Base Die, Die to Die Power, and Package Thermals for AI Accelerators

Hsin-wei Tseng, Director, Product Management, Marvell

HBM is advancing faster than any memory in the data center — HBM3, 3E, 4, 4E, 5 in just a few years — each generation pushing bandwidth, capacity, and power higher. Yet the JEDEC HBM5 standard is still maturing, opening a gap between what AI accelerators need and what the standard delivers. This session presents a custom HBM (cHBM) architecture that closes that gap the HBM memory controller and PHY move off the compute die onto a custom logic base die, with the host connected by a die-to-die (D2D) interface replacing the standard HBM PHY. Customizing the base die frees compute-die area, adds capacity, and lowers I-O power — and lets teams ship ahead of the standards cadence. We then tackle the two hard problems it creates D2D power (a bidirectional link with adaptive power management) and package thermals (co-designing TIM, lid, and integrated cooling). We close on the open, multi-vendor die-to-die interfaces the OCP community needs to make custom memory real.