Elon Musk has announced that his AI development company xAI plans to launch a “Gigafactory of compute” supercomputer in the fall of 2025. The supercomputer will be developed in collaboration with Oracle.
According to the eccentric tech entrepreneur, the supercomputer is needed to develop the next version of xAI’s Grok-LLM. Elon Musk has calculated that training and running this new LLM will require as many as 100,000 GPUs, writes The Information. xAI’s Grok 2 LLM will be trained on 20,000 GPUs.
Therefore, in Elon Musk’s logic, instead of purchasing them separately, it is convenient to combine them into an all-encompassing supercomputer or a Gigafactory of compute. He himself will personally lead the construction process, which should be completed by the fall of 2025.
Nvidia H100 GPUs
The GPUs to be used within the xAI supercomputer are Nvidia H100 GPUs. Combined, they would be as much as four times larger than the current largest GPU clusters, such as the GPU clusters Meta uses to develop its Llama LLMs.
The new supercomputer is being developed in collaboration with Oracle. Exactly what this collaboration consists of is not known. Oracle and xAI have not yet responded to the rumours about the new supercomputer.
$6 billion in Series B round
xAI is also gaining traction as an AI developer and pushing the pressure on other major developers, such as OpenAI and Google. Recently, the developer managed to raise $6 billion in a Series B investment round.
Investors in this round included Valor Equity Partners, Vy Capital, Andreessen Horowitz, Sequoia Capital and Fidelity Management & Research Company. In addition, striking Prince Alwaleed Bin Talal and Kingdom Holding from Saudi Arabia also participated in this investment round.
According to xAI, the capital raised will be used to bring the first AI models to market, build an advanced infrastructure and accelerate research into future technologies. Whether by advanced infrastructure xAI refers to the supercomputer to be developed is unknown.
Also read: Chatbot Grok-1.5 by Elon Musk available next week