Nvidia unveils Cosmos-Transfer1: a new AI model for simulation-based robotics training

Nvidia has introduced Cosmos-Transfer1, a new artificial intelligence (AI) model designed to enhance simulation-based training for robots.

This diffusion-based conditional world model enables highly customisable, multimodal world generation, allowing AI-powered robotic systems to train in diverse virtual environments.

The Santa Clara-based tech giant has made Cosmos-Transfer1 available as open source under a permissive license, making it accessible to researchers and developers through popular online repositories such as GitHub and Hugging Face. The primary advantage of the model lies in its granular control over generated simulations, paving the way for more realistic and adaptable robotic training scenarios.

Simulation-based training has become an essential aspect of AI-powered robotics, also known as physical AI. Unlike traditional factory robots programmed for specific, repetitive tasks, AI-driven robotics require exposure to a variety of real-world scenarios to perform more dynamic functions.

Nvidia’s Cosmos-Transfer1 is part of the Cosmos Transfer World Foundation Models (WFMs), designed to process structured video inputs such as segmentation maps, depth maps, and LiDAR scans. These inputs are then transformed into photorealistic video outputs, which serve as realistic environments for training AI-powered robots.

A recent paper published in arXiv highlights that Cosmos-Transfer1 offers greater customization compared to previous models. The AI enables developers to adjust the weight of conditional inputs based on spatial location, resulting in more precise control over world generation. Additionally, the model supports real-time simulation, accelerating the training process and enhancing the adaptability of robotics applications.

Cosmos-Transfer1 is a diffusion-based AI model with seven billion parameters, built specifically for video denoising in latent space. It features a control branch that modulates inputs and supports a combination of text and video data to generate photorealistic output videos.

The model accommodates four types of control input videos - canny edge detection, blurred RGB images, segmentation masks, and depth maps.

These advanced capabilities make the AI model particularly useful for simulation environments, robotics training, and AI research.

Nvidia has tested Cosmos-Transfer1 on Blackwell and Hopper series chipsets, running inference on the Linux operating system. The model is released under the Nvidia Open Model License Agreement, allowing both academic and commercial applications.

For developers and researchers eager to explore its capabilities, Cosmos-Transfer1 can be downloaded from Nvidia’s GitHub and Hugging Face repositories. Additionally, the company has announced plans to release a more powerful AI model with 14 billion parameters in the near future, further expanding its capabilities in AI-driven robotics training.