MinIO has launched MemKV, a context store that turns petabytes of NVMe flash into a shared memory tier a GPU cluster can reach over RDMA at microsecond latency.
The problem it addresses is eviction. When a KV cache overflows GPU memory, or a session lands on a different node, the context is discarded and the next request rebuilds it with a full prefill recomputation. MinIO says that in a typical production deployment this consumes more than half of all GPU cycles, and that the industry calls the waste the recompute tax.
Benchmarks came from the vendor's own lab, on Llama 3.1 70B at production concurrency. Time to first token at 64K context was 53 seconds without MemKV and 703 milliseconds with it. At 128K, the company says the system without MemKV did not slow down but stopped, out of memory.
Read what that comparison actually is. Both numbers come from the vendor, and the baseline is the same cluster with MemKV removed rather than a competitor's product installed. Fifty-three seconds is what a GPU does when there is nowhere to put evicted context, which is precisely the condition the product exists to eliminate. It measures the size of the problem, not a lead over anyone selling a fix for it.
The money claim is built the same way. MinIO's worked example takes 128 GPUs, contexts of 64K to 128K tokens, cache miss rates above 50 percent and cloud GPU rates of $2 to $4 an hour, and arrives at a recompute tax of up to $2 million a year. The company presents that as arithmetic rather than a projection, which is fair as far as it goes. It is arithmetic on four inputs MinIO selected, and "up to" is carrying weight. The hourly rate MinIO quotes spans a factor of two on its own, so running the same calculation from the bottom of that range, with a miss rate at the floor of what the post calls routine, yields a materially smaller answer than the headline figure.
On rivals, the post is blunt: "Every storage vendor in the G3.5 ecosystem is claiming context memory support."
G3.5 is the tier MinIO is naming, sitting between local NVMe offload that strands context on one server and general shared storage that has the capacity but adds milliseconds. The accusation is that competitors are extending platforms built for durable data, with erasure coding and lifecycle management attached, into a path carrying context that lives for hours. MemKV, by contrast, is described as an ARM64 native single binary embedded in the storage tier, moving 2 to 16 MB blocks straight from NVMe to GPU memory.
One number deserves more scrutiny than it will get: a petabyte of context memory on standard NVMe drives, MinIO says, costs under $80,000, less than a single GPU. That sets media cost against accelerator cost, and drives are the cheapest part of a tier that also needs fabric, enclosures and software.
Blocks & Files reports that MemKV is built for Nvidia's STX architecture and runs within it, with the caching software hosted on BlueField-4 DPUs, and that MinIO claims GPU utilization rising from 50 percent to over 90 percent in the 128-GPU example.
Bootnote: the G-numbers are tiering shorthand for where inference data sits. G3 is local NVMe offload, fast but stranded on the server holding it. G4 is shared external storage, which scales and has a namespace but answers in milliseconds. G3.5 is the gap MinIO says it fills, and naming a tier one is first to ship into is an old and effective move.
Leave a Reply