SWR: Self-Supervised Scaling of Scientific Terminal Environments for Coding Agents Surges Terminal-Bench 2 to 53.56%

核心背景与行业痛点

Terminal agents (such as SWE-bench and Terminal-Bench systems) possess remarkable autonomy across Linux bash shells, DevOps scripting, and repository management. However, deploying agents into sophisticated scientific workflows—such as structural biology, computational fluid dynamics, and materials science—encounters a chronic bottleneck: high-fidelity sandbox environments with deterministic verifiers are severely lacking. Manually authoring specialized tasks, reference outputs, and semantic rubrics requires prohibitive engineering overhead, limiting scientific post-training data diversity.

架构亮点与底层机制

Researchers from Tsinghua University and Tencent present Software-in-the-Loop Reconstruction (SWR, arXiv:2610.02710), a self-supervised environment synthesis paradigm:

1. Bootstrapping From Existing Executable Workflows: Bypasses manual task creation by tapping into existing mature software workflows across production science domains. By running multiple input configurations, SWR partitions output instances into public demonstration samples and private hidden evaluation sets.
2. Black-Box Program Reconstruction: Given natural language goals, input schemas, and public observations, an agent must construct an editable, standalone program inside a clean terminal without seeing the underlying proprietary pipeline source code.
3. Hierarchical Anti-Shortcut Verifier: Executes the synthesized script across hidden configurations. The verifier validates floating-point semantic accuracy, structural schema compliance, and inspects code execution to eliminate hardcoded lookup shortcuts.

权威 Benchmark 与实测跑分对比

Instantiated across 500 realistic scientific workflows and 46 distinct software families spanning six major STEM disciplines:

1. 1,422 Verified Scientific Trajectories: Within 3 attempts per task, Qwen3.8-Max solves 838 arduous computational tasks, generating 1,422 rigorously verified execution traces (expanded to 3,000 reconstruction examples).
2. Terminal-Bench 2 Surges to 53.56%: Fine-tuning Qwen3.8-27B exclusively on SWR data elevates average accuracy on Terminal-Bench 2 from 47.94% to 53.56% across three seeds (+5.62 percentage points).
3. Dominates Token-Matched Baseline Controls: Outperforms all four token-matched synthetic corpus control baselines across every single reported evaluation dimension, verifying the cross-domain transferability of software-in-the-loop training.

开发者实战落地与开箱指南

The SWR synthesis framework, verifier rules, and software harnesses are documented on arXiv (arXiv:2610.02710). AI for Science organizations and enterprise DevOps engineering teams can plug existing internal CLI tools and computational workflows into SWR to generate vast pools of high-fidelity terminal training environments without manual annotation, fueling self-evolving post-training for scientific agents.