AI inference could deliver 3.7 times more output per GPU, according to WEKA, based on a benchmark using the Nebius AI cloud. The explanation for this claimed gain lies in the changing nature of AI workloads. How does this gain come about, and why does the current setup leave so much untapped?
GPUs are increasingly used for daily AI operations, or inferencing, but the surrounding infrastructure is still designed for training models. The factor of 3.7, measured in output tokens per GPU-hour, stems from the 169,000 tokens measured on an Nvidia HGX B300 node with eight GPUs running agentic tasks for 8 hours. The node ran DeepSeek-V4-Pro and utilized 512 active sessions, with 32 requests simultaneously.
With WEKA’s storage layer, the node managed to handle 2.4 times as many requests as a node using only GPU memory. In addition, the response time was consistently faster: the time-to-first-token was within 5 seconds 97 percent of the time, compared to 44 percent when using only HBM.
Interestingly, a widely used metric, GPU utilization, would not reflect any of this. In both cases, the GPUs may or may not be working 100% of the time, but that says very little. Behind that percentage lies either a lot of extra work that keeps the GPU busy but does not make it productive.
Slower is faster
HBM, the memory placed directly next to the GPU on the same package, is considered one of the most expensive components of AI infrastructure. It’s incredibly fast memory with enormous bandwidth, but with limited capacity. If your GPU has 270 GB of HBM but your agents need 271 GB, the system must discard context from memory, which it will then have to recalculate later. Falling back to “regular” system memory (DRAM) is an option, but that’s limited to a single server.
Typically, AI workloads run on a cluster of servers that operate based on availability, which means a session might end up on a server that doesn’t have that context in the next step. For latency-sensitive tasks, such as agentic sessions where the end user is waiting for the output of an AI tool, this is a scenario you want to avoid.
One solution to that capacity problem is somewhat paradoxical. WEKA uses the Augmented Memory Grid (AMG) for this purpose, a layer built on top of its NeuralMesh storage platform. In the test, NeuralMesh pooled the local NVMe drives of eight B300 nodes, six per node, into a single shared layer for AI context. This refers to context that an AI model has already processed and needs again to generate output, known as the KV cache. An AI model processes a prompt during the prefill phase. In the HBM configuration used in the Nebius test, 60 percent of that prefill work consisted of recalculating previously processed context, essentially duplicating effort. With NVMe storage, which is orders of magnitude slower than HBM but offers much greater capacity, this figure was only 6.8 percent.
Incidentally, the DRAM route was not included in the test. According to WEKA and Nebius, there was no DRAM offload implementation for DeepSeek-V4 in July, when the test was conducted. They acknowledge that there are alternative setups that do make effective use of DRAM. Compared to an approach that makes heavy use of system memory, the improvement would likely be less significant.
How it works
That’s not to say the architecture isn’t sophisticated. NeuralMesh is a distributed shared-nothing architecture, in which each node independently handles data, metadata, and I/O. The location of data is determined only at runtime. The AMG’s speed lies primarily in the connectivity layer, taking advantage of technologies such as RDMA and NVIDIA’s GPUDirect Storage to pull the KV cache directly from NVMe into GPU memory, without the CPU, and thus DRAM, intervening. WEKA states that context is also preserved across sessions and node failures. This is because data in DRAM and HBM is lost when the power goes out.
WEKA has some competitors that differ in their implementation but share the same goals regarding faster, scalable AI inference. Nvidia’s Inference Context Memory Storage platform (ICMS, also known as CMX), for example, runs on a full Nvidia stack in terms of DPUs, networking, and, of course, GPUs. VAST Data tested CoreWeave KV-cache offload via Nvidia Dynamo late last year, and DDN is doing something similar with Infinia at the Dutch company Nebul. Accelerating AI inference is thus being addressed by multiple vendors at the architectural level.
Unlike Nvidia’s platform, AMG is not tied to the Rubin generation: according to WEKA, it is already running on H100 systems at OCI and B300 systems at Nebius, and has been commercially available since November 2025.
Cached cash
Those who find the infrastructure details irrelevant will likely be interested in the costs. Pricing for AI workloads is typically entirely token-based, with “cached tokens” offering a cost advantage. This refers to data that the model has already “seen” and is therefore stored in a location that can be accessed more quickly. This is likely where the majority of the global DRAM supply has gone for players like OpenAI.
The provider is able to deliver this data to the GPU much faster than new data, resulting in a lower token price. Those running long agentic sessions rely on more and more cached input as time and context progress. A layer like AMG makes reusing this data even more attractive without having to recalculate everything from scratch.
It will take some time before we know how AI can best be utilized to maximize returns on the customer side. On the supplier side, the key is to achieve returns by identifying efficiencies. The low-hanging fruit has already been picked. Now it’s a matter of reinventing the entire architecture to optimize data processing. WEKA is far from the only player in this space, but it does have an advantage with its persistent memory. Above all, it’s a matter of pinpointing the problem: every AI provider must achieve the same efficiency gains and now has yet another indication that their GPU utilization can be significantly improved. The end of GPU utilization