One Abstraction, Many Transports: Push-Mode Communication from Chiplet to Rack

Sai Rahul Chalamalasetti, Sr. Principal AI Systems Architect, d-Matrix

SRAM-based accelerators are turning low-latency inference into a practical reality, and 3D- stacked DRAM extends that model to far larger memory capacity and bandwidth. As models outgrow any single accelerator, the system-level question of how to partition and connect them now dominates inference performance. Interfaces and protocols built for yesterday's workloads don't scale to modern inference; wrapping a highly capable accelerator in legacy interconnect quickly becomes the bottleneck.

At d-Matrix, we've built a communication model that stays consistent to software even as the underlying transport changes, whether a message crosses a chiplet, a package, a node, or a rack, and even as the accelerator changes underneath it. This talk shows how push-mode streaming communication over Ethernet, between homogeneous or heterogeneous nodes, underpins the disaggregation techniques shaping modern inference, including heterogeneous AFD with GPUs. We cover how JetStream, our network interface for low-latency inference clusters built around Corsair, enables this today, and how the same model carries forward to Raptor, our 3D-DRAM- based accelerator, through NVL-based scale-up and RDMA scale-out.