OpenAI is dissatisfied with the speed of Nvidia’s AI chips for inference tasks and has been looking for alternatives since last year. The focus is on chips with more built-in memory for faster processing, especially for software development.
This is according to Reuters, based on sources. OpenAI is said to have spoken with chip startups Cerebras and Groq about faster inference solutions. Inference is the process by which an AI model such as ChatGPT responds to user queries. However, Nvidia signed a $20 billion licensing deal with Groq, which halted talks between OpenAI and Groq.
The ChatGPT maker is looking for hardware that will ultimately provide about 10 percent of its future inference computing power. Sources say OpenAI is dissatisfied with the speed at which Nvidia’s hardware generates answers to specific problems, including software development and AI-to-AI communication.
SRAM chips for faster processing
OpenAI’s search for alternatives focuses on chips with large amounts of SRAM memory on the same chip. This architecture offers speed advantages for chatbots and other AI systems that process millions of user requests. Inference requires more memory than training because the chip spends relatively more time retrieving data from memory than performing mathematical operations.
Within OpenAI, the problem was particularly evident with Codex, the code generation solution. The company is aggressively marketing this tool. OpenAI employees reportedly attribute part of Codex’s poor performance to Nvidia’s GPU-based hardware. CEO Sam Altman said on January 30 that customers using OpenAI’s programming models “place a lot of value on speed for coding tasks.”
Nvidia defends its position
NVIDIA CEO Jensen Huang called reports of tensions with OpenAI “nonsense” and said NVIDIA is planning a huge investment in OpenAI. “Customers continue to choose NVIDIA for inference because we deliver the best performance and total cost of ownership at scale,” according to an Nvidia statement. An OpenAI spokesperson said the company relies on Nvidia for the vast majority of its inference capacity and that Nvidia delivers the best price-performance ratio.
After the news broke, OpenAI CEO Sam Altman wrote on X that Nvidia makes “the best AI chips in the world” and that OpenAI hopes to “gigantic customer for a very long time.”