ProgramGym Open-Sourced: Scaling 1,648 Verifiable Environments for Long-Horizon Program Reproduction Agents
Current software engineering benchmarks and training pipelines predominantly evaluate coding agents on localized bug fixing within existing repositories (such as SWE-bench) or isolated code snippet generation. They fail to teach agents how to explore unfamiliar reference systems, formulate architectural blueprints, and reproduce complete, multi-file software projects from scratch under long-horizon constraints. Researchers introduce ProgramGym (arXiv:2610.13395, repo: github.com/Acm1011/ProgramGym, dataset: ProgramGym/programgym-1648), an automated framework scaling verifiable training environments for program reproduction agents. ProgramGym consists of Program-Env, a feedback-driven engine that converts open-source repositories into verified runtime sandboxes paired with black-box test suites across C, C++, Go, and Rust (generating ProgramGym-1648), alongside Program-Agent, an iterative workflow enforcing role isolation across Analysis, Implementation, and Review stages with revision management. Fine-tuning Qwen3.8-27B on ProgramGym trajectories raises ProgramBench test pass rates from 34.87% to 42.84% (+7.97 points), outperforming GPT-5.4 under mini-SWE-agent (37.70%). Coupling SFT with the Program-Agent scaffold boosts performance to 56.25%, eclipsing Claude Opus 4.7 (50.90%) and Gemini 3.5 Flash (53.60%), with datasets and codebase open-sourced.
Key Takeaways
- ProgramGym open-sources 1,648 verifiable multi-language environments for whole-program reproduction coding agents
- Qwen3.8-27B-SFT achieves 42.84% on ProgramBench (+7.97 points), surpassing GPT-5.4 under mini-SWE-agent
- Coupled with the Program-Agent scaffold, test pass rates reach 56.25%, outperforming Claude Opus 4.7 and Gemini 3.5 Flash