Alibaba's Qwen team has introduced QwenGyre, an innovative elastic reinforcement learning (RL) framework designed to train AI agents for extremely long-horizon tasks. This new system promises to significantly enhance the development of sophisticated agents, particularly those involved in complex coding endeavors.
QwenGyre operates as an online RL system, enabling the training of coding agents within unmodified, 'black-box' environments. Its primary focus is on agent rollouts that can extend for hours and process close to one million tokens, a scope referred to as 'XLong-horizon' tasks. The framework was successfully implemented and tested on Qwen's flagship Qwen 3.8 2.4T model, as well as the Qwen 3.6 122B model.
Addressing XLong-Horizon Challenges
The research team defines XLong-horizon tasks as agent operations that span hours and involve numerous model and tool interactions, such as generating entire code repositories from specifications. At this scale, two primary challenges emerge:
- Inefficient GPU Utilization: Traditional setups often suffer from idle GPUs. In a colocated environment, training may stall, awaiting the slowest execution. Asynchronous systems, with their fixed GPU pools, cannot dynamically share spare capacity, leading to underutilization.
- Complex Trajectory Management: Agent interactions, particularly in coding environments like Claude Code, can involve branching execution paths, sub-agents, and retries. Treating each path as a unique trajectory dramatically inflates training costs due to duplicated shared prefixes.
The Elastic Scheduler: Dynamic Resource Allocation
QwenGyre tackles GPU inefficiency through an intelligent elastic scheduler. This system partitions GPUs into 'cells,' each capable of either serving agent rollouts or hosting a full training replica. A central core cell maintains the authoritative model weights, while satellite cells can dynamically join the training process.
- The scheduler monitors the volume of queued and active agent executions.
- As executions conclude, cells can transition to training mode, provided sufficient capacity remains for ongoing work.
- Model calls are seamlessly rerouted via a proxy, allowing the agent harness to continue uninterrupted.
- Key-Value (KV) cache blocks are migrated using RDMA during rerouting, minimizing the need for prefix recomputation.
- Satellite cells can pull core weights and integrate into ongoing training batches through streaming.
- Average role switching times are remarkably fast: 8.52 seconds from rollout to training, and 3.46 seconds for the reverse.
Smart Trajectory Processing for Efficient Learning
To address the challenge of complex, branching agent histories, QwenGyre incorporates a sophisticated trajectory processor. It records precise token IDs and behavior log-probabilities, storing interactions as prefix-shared trajectory trees. This innovative approach offers several benefits:
- Partial Scoring: Even timed-out executions contribute to the score based on their preserved workspace, ensuring that incomplete work is not discarded.
- Bounded Sampling: A maximum of five trajectories are retained per execution, ranked by importance (e.g., main agent > sub-agent), significantly reducing data bloat.
- Deduplication: Shared targets are counted only once, and training focuses exclusively on policy-generated tokens.
- Fair Weighting: Losses are averaged within each execution, preventing an overemphasis on executions with numerous branches.
This bounded sampling strategy was found to match uncapped training scores while reducing forward-backward pass time by 25.2%.
Performance Benchmarks
QwenGyre underwent rigorous benchmarking using Claude Code 2.1.220 as the agent harness, with consistent GPU budgets and staleness levels. The results demonstrate significant speed improvements:
- NL2RepoBench (Qwen 3.6 122B): 1.42x to 1.53x faster than Async, 1.36x to 1.47x faster than Colocate.
- DeepSWE (Qwen 3.6 122B): 1.569x to 1.572x faster than Async, 1.610x to 1.824x faster than Colocate.
- TerminalBench (Qwen 3.6 122B): 1.383x to 1.430x faster than Async, 1.809x to 1.849x faster than Colocate.
On the larger Qwen 3.8 2.4T model, QwenGyre completed 48 steps of the NL2RepoBench in 75.42 hours, a substantial improvement over Async (134.55 hours) and Colocate (91.47 hours). This also led to an increase in the evaluation pass rate from 52.48% to 58.54%.
Conclusion
QwenGyre represents a notable advancement in scalable reinforcement learning, particularly for agents tackling complex, long-duration tasks. By intelligently managing GPU resources and efficiently processing intricate agent behaviors, the framework significantly reduces training times and improves model performance. While its technical capabilities are impressive, public code release for QwenGyre has not yet been announced.
This article is a rewritten summary based on publicly available reporting. For the original story, visit the source.
Source: MarkTechPost
