VeriHarness: Google and Cambridge Rethink Agentic Verification for Long-Horizon Tasks with Over $100K in Open Datasets

核心背景与行业痛点

As coding and workflow automation agents execute intricate long-horizon tasks across file systems and cloud sandboxes, test-time output verification without ground-truth answers or formal grading rubrics has emerged as the defining performance hurdle. The standard industry mitigation—sampling multiple rollouts followed by majority voting or consensus aggregation—exhibits systemic vulnerabilities: across long reasoning horizons, candidate divergence often exposes correct edge-case handling, while uniform consensus frequently conceals shared, systematic model hallucinations.

架构亮点与底层机制

Google Cloud AI Research and the University of Cambridge formulate VeriHarness (arXiv:2610.00972) to transform standard generation engines into agentic verifiers:

1. Workspace & Tool Scaffolding: Equips the baseline model with an isolated execution sandbox, environmental introspection tools, and reusable programmatic verification skills.
2. Disagreement Resolver: Actively detects divergent assertions across alternative rollouts and compiles targeted verification scripts to adjudicate conflicting claims against live execution feedback.
3. Consensus Challenger: Stress-tests uncontentious, shared assertions across rollouts, proactively probing for omitted requirements and deceptive false positives.
4. Evidence-Backed Artifact Revision: Synthesizes execution insights to patch candidate artifacts through surgical feedback iterations, while enabling verification skills to self-improve over time.

权威 Benchmark 与实测跑分对比

Evaluated across five rigorous long-horizon workspace benchmarks utilizing premier commercial frontier models:

1. Top Selection Accuracy Across All Benchmarks: VeriHarness achieves the highest candidate selection accuracy across all five evaluated benchmarks compared to existing self-consistency and critic baselines.
2. +6.2 to +6.4 Point Performance Gains: Elevates single-rollout execution scores by +6.2 points on Gemini 3.5 Flash and +6.4 points on Claude Opus 4.8.
3. $100K+ Open-Source Benchmark Dataset: Releases approximately 26,000 verified agent rollouts across all five workspace suites generated at an evaluation cost exceeding $100,000 to foster open-source verification research.

开发者实战落地与开箱指南

VeriHarness is available on the Google Research GitHub repository (google-research/veriharness). Software teams deploying production-grade coding assistants and long-horizon autonomous agents can deploy VeriHarness as an execution-based verification gateway. By pitting alternative execution traces against live environmental evidence rather than blind heuristic voting, systems can dramatically curb silent failures in complex software modifications.