Alibaba Tongyi Labs Open-Sources OpenGCUA: Full-Stack Generalist Computer-Use Agent Cutting OSWorld Steps by 53%

Background and the Problem

Computer-use agents powered by multimodal foundation models have promised end-to-end automation of desktop operating systems. However, most prevailing implementations remain tethered to naive visual-only execution loops: the agent captures a full-screen screenshot, predicts coordinates, issues an isolated click or keystroke, and waits for another screenshot. This paradigm suffers from acute engineering bottlenecks: excessive token burn, multi-second round-trip latencies, and total catastrophic failure upon unexpected popups or minor interface latency. Routine batch workflows—such as transferring fifty rows of tabular data into a web CRM—require hundreds of fragile, serialized steps. To fundamentally reconstruct this interaction paradigm, Alibaba Tongyi Labs (Tongyi-MAI) introduced and open-sourced OpenGCUA in technical report arXiv:2610.22200.

Architecture and How It Works

1. Code-Execution-as-Action Paradigm:

• Replaces isolated GUI action atoms with expressive, unified programmatic execution blocks;
• Synthesizes and executes native Python and Bash scripts combining pyautogui, Playwright, and OS system calls to process loops, condition handling, and file parsing locally within a single turn;

1. Hybrid Perception with Autonomous Observation Routing:

• Seamlessly reconciles pixel-level visual understanding with operating system accessibility trees (AX APIs) and application interfaces;
• Delegates observation strategy to the model itself, allowing it to bypass pixels in favor of deterministic accessibility nodes or terminal pipes whenever available;

1. Open Production Infrastructure:

• Features a fully operational macOS real-device harness operating from the macOS menu-bar for unconstrained physical machine control;
• Includes an extensible Cross-Device SDK standardizing programmatic agent control across desktop OS, web browsers, and mobile environments.

Benchmarks and Measured Results

Evaluated rigorously against the realistic, high-complexity OSWorld-v2 benchmark across eight frontier models (Table 2, Figure 2):

• Consistent +19.6 Points Partial Score Lift:
• Across eight leading commercial and open models (including GPT-6 Astra, Claude Opus 5, and Gemini 3.8 Flash), integrating OpenGCUA increased mean partial success scores by +19.6 points;
• 53.0% Reduction in Task Execution Steps:
• Slashes average interaction steps per task by 53.0% compared to traditional batched GUI step execution, turning sluggish multi-step UI crawls into instantaneous script executions;
• Closes the performance deficit between lightweight open-weight models and proprietary flagship systems by offloading mechanical execution onto deterministic script sandboxes.

Getting Started for Developers

• Paper & Source Repository: Available via arXiv:2610.22200 and GitHub under the Tongyi-MAI organization (Apache 2.0);
• Implementation Strategy: Engineering teams developing RPA solutions, desktop copilots, or cross-application QA bots should abandon brittle coordinate-only clickers in favor of OpenGCUA's hybrid code-action harness to attain exponential gains in task speed, reliability, and cost-efficiency.