Researchers at Nvidia have developed a deep learning engine capable of creating 3D objects from a single image. The tool can be used by architects or game developers, for example, to add objects and mock-ups with ease.
With GANverse3D, as Nvidia calls the tool, objects in a two-dimensional image are blown up into a 3D model. In a blog post about the tool, Nvidia focuses mainly on scanning cars, although it also shows examples with animals. The neural network only needs to be fed one picture of a car to create a complete model. As an example, Nvidia shows how a photo of Knight Rider car KITT can be transformed into a 3D object.
Easily add objects to virtual worlds
Nvidia thinks that GANverse3D will be particularly useful for architects, designers and game developers. They can use the tool to easily add new objects to their environments without having to have 3D modelling expertise or spend a lot of money on existing renders.
“We turned a GAN model into a very efficient data generator so we can create 3D objects from any 2D image on the web,” said Wenzheng Chen, research scientist at NVIDIA and lead author on the project.
“Because we trained on real images instead of the typical pipeline, which relies on synthetic data, the AI model generalizes better to real-world applications,” said NVIDIA researcher Jun Gao, an author on the project.
Usage of a GAN
As there are few datasets that record the same object from different angles, 3D datasets such as ShapeNet are usually used to create an AI that can turn 2D images into 3D models. However, such systems do not handle real-world data, such as images on the internet, as well.
Therefore, Nvidia used a generative adversial network (GAN) model to use the images on the internet as data generators. The final model was trained on 55,000 images of cars and achieved better results than a model based on the Pascal3D dataset.
The whole thing is an extension to the Nvidia Omniverse platform and can run on Nvidia RTX GPUs. More information about the project can be found in the paper written about the project by the researchers of the Nvidia AI Research Lab in Toronto. Whether and when end-users will be able to play with the technology themselves is not known.
Tip: Nvidia announces new workstation GPUs and conversational AI
41 minutes ago
