Nvidia releases DreamDojo, a robot ‘world model’ trained on 44,000 hours of human video

Nvidia, in collaboration with several universities, unveiled DreamDojo, a system designed to train robots using vast amounts of human video. DreamDojo leverages a massive dataset, DreamDojo-HV, comprising 44,000 hours of diverse human actions. This dataset is significantly larger than previous datasets used for robot world model training. The system employs a two-phase training process: learning general physics from human videos and then fine-tuning for specific robot hardware. This approach significantly reduces the need for expensive, robot-specific data collection, a major bottleneck in robotics. DreamDojo achieves real-time interaction speeds, enabling applications like live teleoperation and on-the-fly planning. The system has demonstrated success across multiple humanoid robot platforms, showcasing its versatility in various environments. Nvidia views robotics as a major growth area, fueled by increased AI infrastructure spending and a "once-in-a-generation" opportunity. DreamDojo's simulation capabilities allow for extensive testing before physical deployment, enhancing robot adaptability. The research signals Nvidia's shift towards robotics, viewing the future of computing as physical rather than solely digital. DreamDojo embodies the concept that robots can learn by observing human behavior in the real world.