SenseTime has made its embodied AI evaluation toolkit, SenseNova-RoboRSI, and a 27-billion parameter model publicly available. This release follows research where SenseTime tested the large GPT-6-Astra model across twelve simulated benchmarks and eight real robots. All presented results are based on SenseTime's own tests.
SenseTime introduced SenseNova-RoboRSI as a framework for evaluating embodied AI systems, along with an embodied AI model named SenseNova-RoboRSI-27B-Preview. The study involved testing the large GPT-6-Astra model on 12 simulated robotics benchmarks and 8 physical robots.
GPT-6-Astra Testing Results
According to SenseTime data, when GPT-6-Astra gained access to training demonstrations and step limits were increased tenfold from the official value, the model reached saturation on the LIBERO and SIMPLE benchmarks and achieved 81.9% success on RoboDojo. In one instance, nine RoboDojo tasks that previously failed zero times out of five attempts were successfully completed in all five attempts after the model was allowed to continue interacting.
However, this came at a high cost. In extended tests with a larger budget, stacking bowls took about 131 minutes and was estimated to cost around $19 in inference, while placing a handle in a holder took 263 minutes and cost approximately $55. SenseTime also found that GPT-6-Astra demonstrates the most difficulty with long-horizon tasks, struggles with skill transfer between tasks, and has issues with deformable objects and fine motor control on real robots.
Subsequently, the company tested a combination of a slow reasoning model with a fast action model. The best such combination achieved 54% success in 42 closed-loop RoboDojo tasks, which verified every result, although SenseTime notes that the cost issue remains relevant.
The open-source toolkit packages each benchmark and its dependencies into a separate Docker image and provides a unified HTTP API for interacting with the environment and native evaluation. SenseTime asserts that researchers can conduct end-to-end evaluation using a configuration file without needing to manually configure every simulator.
Characteristics of the 27B Model
The 27-billion parameter model is designed to embed proven experience directly into the model weights, reducing the need for trial-and-error actions. SenseTime reports that this model scored 26.13 on RoboDojo with a task success rate of 20.83%, surpassed the 90 score on LIBERO-Pro, and increased its success rate in RoboCasa365 from 26.0% to 46.5%, while reducing token usage by 73%. The company claims it performs embodied AI tasks at one-fiftieth the cost compared to GPT-6-Astra.
Furthermore, SenseTime insists that future embodied AI benchmarks should be based on real human needs, require robots to learn through interaction, and account for thinking time as real-world time, since the kettle continues to heat up while the model considers a solution. None of the presented results have been independently verified.
