JetBrains Unveils SWE-TaskFlow: Aligning Coding Agent Benchmarks with Real Developer IDE Workflows
Mainstream coding agent benchmarks like SWE-bench frame evaluation around issue-derived tasks: feeding an agent a bug description and expecting an end-to-end patch. But does this reflect genuine developer practices? JetBrains Research analyzes 4,782 production sessions of professional software engineers inside JetBrains IDEs (arXiv:2610.09633). For long interactive sessions (>=3 turns, comprising 33% of production traffic), user requests span a diverse mix of intent types—project codebase Q&A, strategic planning, code review, refactoring, and test execution—with users frequently transitioning between types mid-session. To align benchmarks with real developer task flows, JetBrains introduces SWE-TaskFlow, an approach that transforms issue-based benchmarks via prompt-splitting and verifiable repository QA into calibrated multi-stage workflows guided by a TaskFlow Alignment Score (TFAS). A pilot study across 700 SWE-Bench Pro tasks demonstrates that sequential multi-step interaction protocols nearly double agent inference expenditure without stable improvements in resolve rates, proving that the interaction protocol itself is a critical, previously unmeasured dimension of agent evaluation.
Key Takeaways
- JetBrains analyzes 4,782 production IDE sessions, proving real software engineers navigate dynamic multi-intent task flows
- Introduces SWE-TaskFlow and TFAS to transform issue benchmarks into realistic multi-turn developer interaction protocols
- Pilot across 700 SWE-Bench Pro tasks reveals multi-turn protocols double token costs without stable success gains