When software engineers interact with LLM coding agents, natural language requirements are frequently ambiguous, underspecified, or missing critical boundary rules. Existing coding agents face a difficult trade-off: silently hallucinating assumptions to produce syntactically valid but functionally flawed code, or incessantly asking trivial clarifying questions that disrupt developer focus and cause cognitive fatigue. Researchers from Peking University and Wuhan University introduce CONTRA (arXiv:2610.01769, repo: github.com/fangz-cs/Contra), a training-free framework for selective clarification in LLM code generation. CONTRA employs a three-tier pipeline: broad question discovery, semantic- and execution-based question qualification, and history-aware adaptive stopping. By executing candidate code variants in isolated environments conditioned on alternative answers, it verifies whether a question triggers stable, behavior-changing differences before prompting the user. On ClarifyCodeBench, CONTRA achieves macro-average question F1 of 41.20% and TKQR of 38.32%, outperforming leading baselines by 13.88 and 12.33 percentage points respectively. On GPT-5.5, it lifts recall from 15.12% to 36.72% while maintaining high precision. CONTRA is released as an open-source extension for Claude Code.
Key Takeaways
- ✓CONTRA framework published, leveraging execution-based behavior differential testing to stop agent hallucinations and fatigue
- ✓Achieves 41.20% macro-average F1 on ClarifyCodeBench, outperforming leading clarification baselines by 13.88 points
- ✓More than doubles key question recall on GPT-5.5 from 15.12% to 36.72%, with open-source plugin released for Claude Code
Key Decision Metrics at a Glance
Heavy Claude Code use: compare subscription limits and API bills
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Background and the Problem
When developers supply natural language specifications to coding agents, prompts are inherently plagued by ambiguities, unspecified constraints, or boundary edge cases. Current coding agents suffer from a fundamental dilemma: either silently making arbitrary assumptions that produce insidious logic bugs, or generating floods of pedantic clarification questions that derail developer focus.
Architecture and How It Works
Researchers from Peking University and Wuhan University propose CONTRA (arXiv:2610.01769, repo: github.com/fangz-cs/Contra), a training-free framework for selective clarification in LLM code generation:
- Broad Question Discovery: Systematically surfaces candidate questions exposing ambiguous program requirements while filtering out questions already resolved by the prompt.
- Execution-Based Question Qualification: The core breakthrough of CONTRA. Generates candidate code implementations conditioned on alternate plausible user answers and evaluates them within execution sandboxes on identical test inputs. Questions that do not produce stable, behavior-changing output variations are discarded as redundant.
- History-Aware Adaptive Stopping: Tracks dynamic dialogue state across rounds, computing marginal information gain to selectively trigger questions or finalize execution without user interruption.
- Production Claude Code Plugin: Open-sources a native plugin architecture compatible with Claude Code and OpenHands harnesses.
Benchmarks and Measured Results
Evaluated on ClarifyCodeBench across GPT-5.5, GPT-5-mini, Grok 4.5, and Gemini 3.1 Pro:
- SOTA F1 and TKQR Gains: CONTRA achieves macro-average F1 of 41.20% and TKQR of 38.32%, outperforming existing clarification baselines by 13.88 and 12.33 percentage points respectively.
- Doubling Recall Without Sacrificing Precision: On GPT-5.5, lifts key question recall from 15.12% to 36.72% while maintaining 50.15% precision, putting an end to silent requirement hallucinations.
- Suppressing Spurious Inquiries: On GPT-5-mini, boosts precision from 13.58% to 36.20%, preventing question fatigue in developer workflows.
Getting Started for Developers
CONTRA is open-sourced on GitHub (fangz-cs/Contra). Software engineering teams can install the Claude Code plugin to automatically eliminate requirements ambiguity, integrate shadow sandbox execution loops to verify whether ambiguities alter code behavior before prompting users, and enforce question turn budgets to ensure seamless developer productivity.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.