Genesis can reduce training times from decades to hours using 3D worlds generated from text.
On Thursday, according to Ars Technica, a large group of researchers from universities and the private sector presented Genesis. This new open-source computer simulation system allows robots to practice tasks in a simulated reality. And this is 430,000 times faster than in the “real” world. Researchers can also use an AI agent to generate 3D physics simulations from text inputs.
Thanks to accelerated simulation, a neural network controlling robots can spend the virtual equivalent of decades learning to pick up objects, walk or handle tools in just a few hours of computer time.
Huge acceleration
“One hour of compute time gives a robot 10 years of training experience. That’s how Neo was able to learn martial arts in a blink of an eye in the Matrix Dojo,” wrote Genesis paper co-author Jim Fan on X. Fan, who says he played a “small role” in the research, has previously worked on several robot simulation projects for Nvidia.
One launches Genesis as robotics researchers look for better tools to test and train robots in virtual environments before they are deployed in the real world. Fast, accurate simulations help robots learn complex tasks faster and reduce the need for expensive physical testing.
The Genesis platform, developed by a group led by Zhou Xian of Carnegie Mellon University, processes physics calculations up to 80 times faster than existing robot simulators. Consider Nvidia’s Isaac Gym. It uses graphics cards similar to video games to simultaneously run up to 100,000 simulations. This is crucial for training the neural networks that will control future robots in the real world.
“If an AI can control 1,000 robots to perform 1 million skills in 1 billion different simulations, then it may ‘just work’ in our real world, which is simply another point in the vast space of possible realities”, Fan wrote in his post on X.
Generating dynamic worlds
The team also announced that it is possible to generate so-called “4D dynamic worlds”—probably called “4D” because they simulate a 3D world in motion over time. The system uses vision-language models (VLMs) to generate complete virtual environments based on text descriptions, similar to “prompts” in other AI models. In doing so, it uses Genesis’ own simulation APIs to create the worlds.
The AI-generated worlds reportedly include realistic physics, camera movements and object behaviour based on text commands. The system then produces physically accurate ray-traced videos and data that robots can use for training.
Faster creation of robot tests
This text-based system allows researchers to create complex testing environments for robots by typing natural language commands instead of programming them manually. “Traditionally, simulators require a huge amount of manual effort from artists: 3D assets, textures, scene layouts, etc. But every component in the workflow can be automated,” Fan wrote.
Genesis’ engine can also generate character animation, interactive 3D scenes, facial animation and more. This opens possibilities for artistic projects but can also lead to more realistic AI-generated games and videos, building a simulated world in data rather than just the statistical representation of pixels, as with video diffusion models.
Genesis is still in active development on GitHub, where the team accepts community contributions.
The power of Python
The platform differs from other 3D robot training simulators using Python for both the user interface and the core physics engine. Other engines use C++ or CUDA for their underlying computations while providing Python APIs. Genesis, however, takes a Python-first approach.
Importantly, the Genesis platform’s nonproprietary nature makes super-fast robot training simulations available for free to any researcher via simple Python commands on standard computers with ordinary hardware.
Fan said in his announcement of Genesis that running robot simulations previously required complex programming and specialized hardware. “Robotics should be a lunar mission initiative owned by all humanity,” he wrote.