TerminalHorizon Released: Automated Data Synthesis Engine for Long-Horizon Terminal Agents Powers 27B Model to 55.8% on Terminal-Bench 2.0

Background and the Problem

Autonomous software engineering agents and terminal assistants operating in Linux environments are advancing from one-line bash utilities toward complex, multi-hour engineering tasks. However, training foundation models for terminal autonomy faces fundamental data bottlenecks:

1. Short-Horizon Trap: Existing datasets offer trivial 3-to-5 turn tool invocation snippets, failing to teach models how to explore unfamiliar filesystems, adjust policies upon execution failures, and synthesize final deliverables over prolonged trajectories.
2. Synthetic Environment Pre-solving: Automated benchmark generators frequently oversimplify configurations or pre-install dependencies, eliminating the realistic debugging challenges engineers face.
3. Prohibitive Annotation Overhead & Hallucinations: Manual curation of 50+ turn workflows is unsustainable, while naive synthetic generation suffers catastrophic drift and logical contradictions.

Architecture and How It Works

Researchers from USTC, Alibaba Group (AMAP), and SUSTech introduce TerminalHorizon (arXiv:2610.13166, project: zs1314.github.io/TerminalHorizon):

1. Real-World Workflow Reverse-Engineering: Seeds task creation from authentic production codebases, deploying independent execution sandboxes to identify implicit environmental prerequisites and missing configurations.
2. Evidence-Guided Dynamic Reconstruction: Reconstructs environments around uncovered real-world complexities, requiring autonomous agents to actively probe, diagnose, and repair system states.
3. Unified Multi-Stage Expansion: Organically scales tasks along software development life-cycles under a coherent objective, naturally driving trajectories to 50+ interaction turns.
4. Scaled High-Quality Data Pipeline: Synthesizes 1.5K verified task environments and 2,852 rigorous long-horizon trajectories with an average length of 52.8 interaction turns.

Benchmarks and Measured Results

Supervised fine-tuning of Qwen3.5-27B yields TerminalHorizon-27B, demonstrating remarkable performance across 12 rigorous evaluation suites:

1. 55.8% on Terminal-Bench 2.0: TerminalHorizon-27B reaches 55.8% task accuracy, delivering a massive 14.2-point improvement over the base Qwen3.5-27B model (41.6%).
2. Data Efficiency Superiority: Achieves SOTA terminal performance using under 3,000 highly curated long-horizon trajectories, outperforming prior models trained on orders of magnitude larger datasets like Terminal-Universe-27B.
3. Consistent Gains Across 12 Benchmarks: Exhibits strong transferability across SWE-bench software repair, shell scripting, and multi-step reasoning benchmarks without degradation.

Getting Started for Developers

The TerminalHorizon datasets (shunzou05/TerminalHorizon-3K) and methodology offer an engineering roadmap for agent developers: shift training data synthesis toward multi-stage evidence gathering, leverage independent sandbox execution loops for realistic error feedback, and fine-tune open-weight backbones for reliable command-line autonomy.