Alibaba's Tongyi Lab has officially unveiled MAI-UI, a pioneering family of foundation Graphical User Interface (GUI) agents. This new system offers native integration of multi-platform (MCP) tool utilization, dynamic user interaction capabilities, intelligent device-cloud collaboration, and advanced online reinforcement learning (RL). These features collectively establish leading performance in generalized GUI grounding and mobile GUI navigation, notably outperforming existing solutions such as Gemini 2.5 Pro, Seed1.8, and UI-Tars-2 within the Android environment.
MAI-UI was developed to address critical deficiencies in early GUI agents, specifically focusing on seamless agent-user engagement, robust MCP tool integration, and a flexible device-cloud architecture that prioritizes user privacy by processing sensitive data locally while leveraging powerful cloud models when necessary.
Understanding MAI-UI: A Multimodal Approach
The MAI-UI family comprises multimodal GUI agents built upon the Qwen3 VL architecture, available in various model sizes including 2B, 8B, 32B, and a 235B A22B variant. These models are engineered to interpret natural language directives and visual UI screenshots, subsequently generating structured actions executable in a live Android environment.
The agent's action repertoire extends beyond standard operations like clicking, swiping, text entry, and system button presses. MAI-UI innovatively includes explicit actions for providing direct answers to user queries, requesting clarification for ambiguous goals, and invoking external tools via MCP calls. This allows the agent to execute complex tasks by blending GUI interactions, direct linguistic responses, and API-level operations within a single workflow.
Core Innovations Driving MAI-UI's Performance
From a technical standpoint, MAI-UI integrates three crucial components: a self-evolving navigation data pipeline that incorporates both user interactions and MCP scenarios; an online RL framework capable of scaling across hundreds of parallel Android instances and managing extensive contexts; and a native device-cloud collaborative system that intelligently routes task execution based on its current state and privacy considerations.
Advanced GUI Grounding and Instruction Reasoning
A fundamental requirement for any GUI agent is effective grounding—the ability to map free-form language (e.g., 'open monthly billing settings') to the correct on-screen control. MAI-UI employs a unique UI grounding strategy inspired by prior work on multi-perspective instruction descriptions. Instead of relying on a single caption per UI element, the training pipeline generates diverse views, such as appearance, function, spatial location, and user intent. These multiple instructions serve as reasoning evidence, guiding the model to pinpoint the correct bounding box. This approach significantly mitigates the impact of imprecise or incomplete instructions, a known issue in existing datasets.
MAI-UI models demonstrate impressive accuracy on public GUI grounding benchmarks, achieving 73.5% on ScreenSpot Pro with adaptive zoom, 91.3% on MMBench GUI L2, 70.9% on OSWorld G, and 49.2% on UI Vision. These figures surpass Gemini 3 Pro and Seed1.8 on ScreenSpot Pro and markedly outperform earlier open models on UI Vision.
Self-Evolving Navigation Data and the MobileWorld Benchmark
Navigating dynamic mobile applications is more complex than grounding, requiring the agent to maintain context across multiple steps, potentially across different applications, while interacting with users and tools. To cultivate robust navigation behaviors, Tongyi Lab utilizes a self-evolving data pipeline. Initial tasks are derived from app manuals, engineered scenarios, and filtered public data. Parameters are varied to broaden coverage, and object-level substitutions are applied while maintaining task relevance. Multiple agents, alongside human annotators, execute these tasks in Android environments, generating extensive trajectories. A judge model then evaluates these trajectories, preserving high-quality segments for subsequent supervised training rounds. This adaptive approach ensures the data distribution continuously evolves with the current policy.
MAI-UI's navigation capabilities are evaluated on MobileWorld, a proprietary benchmark featuring 201 tasks across 20 applications. MobileWorld encompasses pure GUI tasks, agent-user interaction tasks requiring natural language dialogue, and MCP-augmented tasks demanding tool calls. On this rigorous benchmark, MAI-UI achieves an overall success rate of 41.7%, representing a substantial 20.8-point improvement over the strongest end-to-end GUI baselines.
Scalable Online Reinforcement Learning
To ensure resilience in dynamic mobile application environments, MAI-UI integrates an online RL framework where the agent directly interacts with containerized Android Virtual Devices. This environment leverages rooted AVD images and backend services within Docker containers, providing standardized operations over a service layer. It supports over 35 self-hosted applications spanning e-commerce, social media, productivity, and enterprise categories.
The RL setup employs an asynchronous on-policy method, GRPO, built upon verl. It combines tensor, pipeline, and context parallelism, mirroring Megatron-style training, enabling the model to learn from trajectories up to 50 steps long and managing extensive token sequences. Rewards are assigned by rule-based verifiers or model judges that detect task completion, with penalties for repetitive or unproductive behaviors. Only recent, successful trajectories are retained in task-specific buffers to promote stable learning.
Scaling this RL environment has practical implications. Researchers demonstrated that increasing the number of parallel GUI environments from 32 to 512 resulted in approximately a 5.2 percentage point improvement in navigation success. Furthermore, extending the allowed environment steps from 15 to 50 added around 4.3 points to performance. On the AndroidWorld benchmark, which assesses online navigation in a standard Android app suite, the largest MAI-UI variant achieved a 76.7% success rate, outperforming UI-Tars-2, Gemini 2.5 Pro, and Seed1.8.
Key Highlights of MAI-UI
- Unified GUI Agent Family: MAI-UI represents a Qwen3 VL-based family of GUI agents, ranging from 2B to 235B A22B, specifically engineered for real-world mobile deployment with built-in agent-user interaction, MCP tool calls, and sophisticated device-cloud routing.
- State-of-the-Art Grounding and Navigation: The models deliver impressive accuracy across benchmarks: 73.5% on ScreenSpot Pro, 91.3% on MMBench GUI L2, 70.9% on OSWorld G, and 49.2% on UI Vision. They also establish a new benchmark of 76.7% on AndroidWorld mobile navigation, surpassing UI-Tars-2, Gemini 2.5 Pro, and Seed1.8.
- Realistic MobileWorld Performance: On the MobileWorld benchmark, MAI-UI 235B A22B achieves 41.7% overall success, including 39.7% on pure GUI tasks, 51.1% on agent-user interaction tasks, and 37.5% on MCP-augmented tasks, significantly outperforming competitors like Doubao 1.5 UI TARS (20.9%).
- Scalable Online RL: MAI-UI leverages an online GRPO-based RL framework over containerized Android environments, demonstrating performance gains from increased parallel environments and extended step budgets.
This article is a rewritten summary based on publicly available reporting. For the original story, visit the source.
Source: MarkTechPost
