ACE Robotics, in collaboration with S-Lab from Nanyang Technological University, and with participation from Beijing Jiaotong University and the University of Michigan, has made Puffin-World publicly available. This unified multimodal world model considers physics, geometry, and appearance as integral three-dimensional states of the world.
The released package includes the code, models, and dataset Puffin-16M. The goal of this development is to enable perception and simulation ready for robots, rather than just predicting the next frame based on RGB images.
Most generative world models focus on predicting future pixels. However, Puffin-World integrates modeling of a gravity-aware physical state, which includes latitude indicators, depth-recovered geometry state, and an appearance state represented in RGB format. All these elements are linked through an Omni-Camera representation that combines absolute and relative camera frames.
Absolute landmarks anchor observations relative to real gravity and orientation, while relative ray geometry supports continuous viewpoint changes and composite motions. The model can estimate absolute roll, pitch, and vertical field of view from a single image, and also synthesize trajectories from arbitrary viewpoints, reconstruct dense geometry, and perform closed-loop exploration by correcting drift using gravity and position during camera movement in the generated scene, utilizing textual or visual data.
The training scale is provided by the Puffin-16M dataset, which contains approximately 15 million 'vision-language-camera' triplets in Puffin-Cam-15M and 1 million complex trajectories in Puffin-Traj-1M. These trajectories cover pitch, yaw, roll, and composite motions across various environments: indoor, outdoor, synthetic, and road scenes.
The team also used Puffin-World to add absolute camera pose labels to 28 public datasets covering about 44.5 million images, available for further research. This data contribution aims to fill the gap in absolute pose present in camera corpora based only on relative data, as they cannot train gravity alignment.
On four public camera-to-world benchmarks—Stanford2D3D, MegaDepth, TartanAir, and LaMAR—Puffin-World demonstrates state-of-the-art median errors. On Stanford2D3D, median errors for roll, pitch, and vertical field of view decrease to 0.29°, 0.53°, and 1.62°, respectively. Furthermore, the model shows the least roll within one degree on MegaDepth (0.28°), TartanAir (0.31°), and LaMAR (0.26°).
Design materials also indicate leading PSNR and LPIPS scores in multi-view rendering for RealEstate10K and low angular errors on test benches for arbitrary viewpoint camera control. The architecture combines a vision encoder aligned with geometry, a language model, a diffusion generator, and a lightweight connector. This allows understanding, generation, and reconstruction to utilize a unified structure without task-specific external geometry modules.
For embodied intelligence systems (embodied stacks), the open release provides a single control point capable of perceiving absolute camera physics, simulating controlled viewpoints, and recreating coherent three-dimensional worlds. This represents a shift from simple frame prediction to modeling how a physical scene exists and how it can be explored.
