⚡

Developer 3s Key Decision Metrics

TL;DR Verdict
🏆 Benchmark Rank72s 必看
Lifts replay verifier pass rates fro
Full Leaderboard→
💰 Cost & Plans比价
API & Subscription
Compare Plans↗
🛠️ Ecosystem & IDERules
Agent Error Dataset · AET Pipeline · WebShop
Setup IDE Rules↗

An unsuccessful agent rollout contains rich diagnostic signals beyond sparse negative rewards. UIUC researchers introduced the Agent Error Dataset (AED), the largest scale failure dataset comprising 50,228 error-diagnosis pairs across 9,961 source tasks, 33 environments, 19 harness families, and 23 policy models. Powered by a 5-stage Agentic Error-to-Training (AET) pipeline, first-proposal repairs lift verifier pass rates from 18.4% to 51.1% (+32.7 percentage points). Furthermore, post-training Qwen3-8B on AED boosts exact-step failure diagnosis agreement from 47.2% to 63.6%.

Key Takeaways

  • ✓Features 50,228 curated error-diagnosis pairs across 33 environments, 19 harnesses, and 23 policy models
  • ✓Lifts replay verifier pass rates from 18.4% to 51.1% (+32.7 percentage points) with first-proposal corrections
  • ✓Boosts Qwen3-8B exact-step failure diagnosis agreement from 47.2% to 63.6%, demonstrating superiority over success-only training
🧭

Finished reading? Explore benchmark rankings & pricing

Real-world SWE-bench scores & $20/mo vs API cost break-even calculator

🏆 Leaderboard→💰 Compare Plans↗
🔗

Project Links & Resources

Direct Access
Direct access to official project resources and documentation
🌐
arXiv:2609.40111 - Agent Error Dataset: Scaling 50,000 Error-Diagnosis Pairs for Failure Analysis and Error-Aware Post-Training
arxiv.org/abs/2609.40111
↗
📖
Hugging Face Daily Papers - Agent Error Dataset
huggingface.co/papers/2609.40111
↗
🔬

In-Depth Technical Analysis

核心背景与行业痛点 Traditional LLM agent training predominantly relies on sparse binary rewards, discarding unsuccessful rollouts that contain rich diagnostic information regarding state transitions, faulty decisions, and stack traces. Lacking structured failure data, current agents struggle with localized self-correction and recurrent pitfalls across complex multi-turn trajectories. ### 架构亮点与底层机制 UIUC researchers introduced the Agent Error Dataset (AED), the largest-scale failure analysis dataset comprising 50,228 error-diagnosis pairs across 9,961 source tasks, 33 environments, 19 harness families, and 23 policy models. AET (Agentic Error-to-Training) powers the data collection across 5 stages: natural failure harvesting, causal diagnosis and repair generation, recorded evidence validation, controlled checkpoint replay, and multi-view training derivation (diagnosis vs. recovery). ### 权威 Benchmark 与实测跑分对比 Empirical validation across controlled replay and post-training demonstrates massive improvements: 1. Pass Rate Spikes by 32.7%: Across 3,062 matched replay pairs, first-proposal corrections increase verifier pass rates from 18.4% to 51.1% (+32.7 percentage points). 2. Exact-Step Failure Diagnosis at 63.6%: Fine-tuning Qwen3-8B on AED's full-diagnosis slice elevates exact-step diagnosis agreement with teacher labels from 47.2% to 63.6% on a 943-case holdout, outperforming prompted frontier baselines (54.7%). 3. Repair Training Outperforms Success-Only: In WebShop-lite policy training, action-level repair training achieves 6.67 percentage points higher score than conventional success-only training. ### 开发者实战落地与开箱指南 AED provides ready-to-train JSONL datasets formatted for SFT and DPO post-training. Practitioners training autonomous coding agents can leverage AED to train dedicated runtime diagnostic critics or fine-tune core policies with innate error recovery behaviors, markedly improving agent resilience in production environments.