Research21d ago
Can Agents Predict Their Own Failure? Outcome-Grounded Confidence Calibration in LLM Agents Unlocks Reliable Execution
Controlling acting LLM agents requires step-level confidence signals to decide whether to execute an action, abstain, or hand off control to humans. Research from NYU introduces outcome-grounded confidence calibration for multi-step agents. Rather than calibrating confidence against immediate local actions, the framework aligns per-step verbalized confidence with the ultimate multi-step episode outcome using GRPO rollouts and proper scoring rules. Evaluated across ALFWorld, TextWorld, and WebShop on Qwen2.5-7B, confidence AUROC jumps to 0.68–0.85, steps within the top-10% confidence bracket succeed 4.3x more frequently, and expected calibration error plummets from 0.42 to below 0.12.
Tooling6d ago
Factory launches Droid 1.0: programmatic tool calling by default, hundreds of tool calls per model request, leads native harnesses on its own Terminal-Bench 4 runs
AI coding agent company Factory announced Droid 1.0 on X and its blog at 22:31 UTC on Oct 29, calling it the largest update yet to its agent harness. The key change makes programmatic tool calling (PTC) the default: the model writes a JavaScript program that runs up to hundreds of tool calls in a sandbox in one model request. In Factorys own Terminal-Bench 4 runs, Droid scored 57.6% with GPT-6.1 Sol ($0.79 per passed task) and 65.2% with Claude Opus 5.5 ($4.73), ahead of Codex, Claude Code, Cursor, OpenCode and Devin on the same models.
Agents28d ago
DAEDALUS: Bootstrapping Reusable Agent Memory via Self-Generated Tasks and Autonomous Dual-Agent Practice Without Human Oracles
LLM agents operating in novel API or terminal environments frequently repeat identical errors due to an absence of procedural memory. Existing memory bootstrapping approaches rely heavily on curated prompt playbooks or oracle verifiers that require extensive domain foresight. Illuin Technology releases DAEDALUS (arXiv:2610.08048), an autonomous dual-agent framework bootstrapping reusable memory entirely from self-generated practice without human oracles. DAEDALUS pairs an Explorer agent—which probes the execution environment to formulate curriculum-style tasks—with a Solver agent. When the solver stumbles, it extracts candidate heuristics that are committed into a persistent memory bank only after proving their efficacy across repeated verification trials. Across AppWorld, tau^2-bench, and AutomationBench, DAEDALUS boosts average success rates by up to 15.9 percentage points and raises Pass^5 rates by 2.2x over memory-less baselines, with generated procedural memories transferring seamlessly across heterogeneous foundation model families.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.