Stacking memory on top of a processor sounds like a packaging detail. It turns out to be a maintenance policy. Nvidia and the rest of the chip makers now bond high-bandwidth memory to the GPU through an interposer layer, using a process called Chip-on-Wafer-on-Substrate, and Data Center Knowledge reports that the card which comes out the other end cannot be serviced in the field.
HBM, said letter by letter as aitch bee em, is the apartment-building answer to a commuting problem. Rather than spreading memory across a suburb of sockets, the stack goes straight up next to the processor so the data has a shorter trip. Bandwidth improves. Serviceability does not.
Manoj Sukumaran, principal analyst for data center compute and networking at Omdia, splits the memory wall into two problems that usually get discussed as one. Bandwidth is how fast data gets from memory to the processing unit. Capacity is how much of a model can sit close enough to matter, and he flags that as a particular concern in the decode phase of inference, in model storage, and in the large reasoning tasks that increasingly define what AI is asked to do.
Each step down the hierarchy costs speed. Going from on-GPU SRAM out to HBM, then to DRAM, then out to NAND flash, bandwidth commonly drops by something approaching a factor of ten at every hop. When a model outgrows the fast tiers, the most expensive silicon in the building sits idle waiting for its groceries to arrive, which is not a sentence any finance department enjoys hearing.
Then there is the failure arithmetic, and this is where the packaging choice comes back around. Sukumaran puts GPU mean time between failure at around 20,000 hours, a number that sounds comfortable until it is multiplied out. A 20,000-GPU cluster loses roughly one card an hour. Because the memory is now bonded to the processor rather than seated in a slot, a bad stack is no longer a swap. It is a return.
Compute Express Link, or CXL, pronounced as its letters, goes after the capacity side instead. It lets a rack treat memory as a shared pool rather than a private allocation stranded in each server, and the 3.0 revision is intended to extend that pooling across multiple hosts through multi-level switching. Penguin Solutions used GTC 2026 to announce a CXL-based MemoryAI KV cache server aimed at getting data to GPU clusters faster.
Inference itself splits into two phases that stress memory in opposite directions. Pre-fill, where the prompt is read in parallel and key-value pairs are written into a cache, leans on compute. Decode, where the model emits one token at a time out of that cache, leans on bandwidth and eats capacity, which is why software has joined the fight. Google's TurboQuant compresses those cached values to as little as 3.5 bits each.
Reference designs have multiplied to cope. AMD, Dell Technologies, HPE, Intel and Lenovo have all published AI design frameworks, colocation providers including Equinix and Digital Realty sell turnkey infrastructure pods, and neocloud operators such as CoreWeave and Lambda publish memory layouts of their own. None of that makes the wall go away. It decides, in advance, which side of the wall the budget lands on.
Leave a Reply