Alibaba's Qwen released Qwen-UI-Agent on August 20. This model is a Graphical User Interface (GUI) based agent that covers operation on phones, desktop computers, web, and deep search. The core idea of the model is to understand the user interface, simulate human actions such as clicks, data entry, and swipes, in order to perform tasks.
Since most software does not provide an API, the graphical interface is considered the most universal input. Qwen-UI-Agent bets that the screen is the largest interface, acting as a universal executor that uses one model for four different environments.
A key challenge for this model is working with real devices, as many GUI agents are trained in simulators and show failures on actual hardware. Alibaba developed a mobile environment including over 100 real phones and more than 150 applications for training and evaluation on physical devices, and also created a benchmark of 400 tasks called MobileWorld-Real.
On mobile devices, MobileWorld achieved a result of 82.1 percent, surpassing models such as GPT-5.6 Sol, Claude Opus 4.8, and Seed 2.1 Pro. MobileWorld-Real demonstrated the best results among flagship devices, including Gemini 3.1 Pro, while AndroidDaily reached 97.5 percent. On desktop computers, OSWorld-Verified scored 79.5 percent, and in the browser, WebArena took first place with a score of 73.6 percent; using OSWorld-v2 reduced the number of execution steps by 58 percent. Demonstrations showed the ability to search via Amap, summarize from Xiaohongshu, and book in DingTalk across five or six applications.
The screen-interacting model faces more serious security issues than performance issues. Qwen-UI-Agent integrates security assessment directly into the task execution process: upon receiving illegal or high-risk requests, the agent refuses the operation and terminates. When encountering sensitive scenarios, such as sending red envelopes, deleting files, or requesting privacy authorization, it stops and awaits confirmation. For example, when instructed to 'send 500 yuan to mom via Alipay,' the agent fills in the amount and note, then pauses at the payment stage to receive confirmation. If details like flight dates or cabin class are missing, it first requests this information.
The model's action space is mixed: besides GUI, it can execute Command Line Interface (CLI) commands, with CLI actions accounting for almost half of all desktop actions. In a 170-step task, the agent verified financial data, ran analytical scripts, created reports in Excel, PPT, and Word formats, and adjusted layouts using visual verification. Meanwhile, the GUI was responsible for gathering information from the web, and the CLI for analysis. Training is aggressive: online reinforcement learning supports trajectories longer than 100 steps using about 10,000 parallel environments. The agent also offers proactive services: for instance, if a flight cancellation notification is received, it suggests a rebooking plan, and tasks can be transferred between phone and desktop, such as retrieving a receipt from the photo library and subsequently creating a summary table in Excel on the desktop. Thus, the agent moves from waiting for instructions to caring for the user.
The world outside of APIs is significantly larger than the world inside them. Qwen-UI-Agent chose a direct approach: not to try to change the world, but to learn how to use it as it exists, while the technical report, project page, and code have been published. For ordinary users, the relevant question remains when these capabilities will be integrated into everyday phone assistants to solve trivial and complex tasks.