Almost no one measures their own AI consumption. The vast majority of organizations have taken the technology on the road, but only a tenth actively consider the energy costs associated with AI.
Research commissioned by SambaNova suggests that organizations are unprepared for the impending explosion in energy consumption caused by Rodrigo Liang, CEO of SambaNova Systems, highlights the issues which this will cause. “Without a proactive approach to more efficient AI hardware and energy consumption, particularly in the face of increasing demand from AI workflows, we risk undermining the very progress AI promises to deliver. By 2027, my expectation is that more than 90% of leaders will be concerned about the power demands of AI and will monitor consumption as a KPI that corporate boards will track closely.”
Energy costs not felt
Corporate leaders overwhelmingly (70 percent) do realize that training AI models takes an enormous amount of energy. Fewer respondents (60 percent) know that the same (though to a lesser extent) is true of AI inferencing. Only 13 percent of organizations that have started working with AI measure how much energy is consumed.
These figures are unsurprising. Many organizations rely on an API pricing model to access LLMs from OpenAI, Anthropic or others. As a result, the main concern is one focused on cost. Said costs are currently still heavily subsidized; AI companies typically run large losses as they continue to spend on R&D and offer their products at competitive prices. Even the immensely popular OpenAI needs billions in investment to stay in business. But for organizations, this pricing model is a layer of abstraction too far. They do not look at exact energy costs, because they can only make estimates of them. This is a consequence of the way GenAI is currently consumed.
However, it is evident organizations do care about the energy costs involved. Especially as agentic AI is on the rise, 56 percent of respondents acknowledge that energy efficiency will play a crucial role in future strategic planning. On top of that, 37 percent expect increasing stakeholder pressure to improve AI energy efficiency – another 42 percent are anticipating such pressure to emerge soon.
Where the SambaNova research hits a sore spot is that the current paradigm will not hold for long. GPU-driven GenAI workloads have captured the imagination of executives worldwide since ChatGPT. Because of the gargantuan scale that LLMs require, only a few industry players have the capital and infrastructure to host AI training of the biggest models. Smaller models (SLMs), domain-specific purposes and local inferencing are steps in the right direction to reduce overall energy consumption, but not all will be able to run on GPUs.
Looking for the alternative
SambaNova talks about a lack of “infrastructure preparedness” for AI adoption. The proactive approach mentioned by SambaNova’s Liang focuses on an alternative to GPU acceleration. GenAI running can also be done on other accelerators that deal more efficiently with the billions of parameters and training data. For this purpose, SambaNova has created the RDU (Reconfigurable Dataflow Unit). Instead of holding tens of gigabytes of on-chip memory, this AI chip possesses terabytes to store multiple LLMs simultaneously.
SambaNova is not the only one eager to knock Nvidia’s GPUs off its throne. Cerebras, maker of gigantic wafer-scale AI chips, and Groq are also looking for a working alternative. The question is whether the aforementioned hyperscalers seek such a switch (or, in Google’s case, continue to rely on its own TPUs) before AI’s energy costs are already beyond the pale.
Also read: Executives have sky-high expectations of AI