AWS open-sourced awslabs/Agent-EvalKit: run evals inside Claude Code, Kiro CLI, or Kilo Code via slash commands like /evalkit.plan and /evalkit.report. Six phases cover plan, data, OpenTelemetry traces, run, score, and code-linked fixes; a travel-research demo scored Response Quality 83.9%, Tool Parameter Accuracy 64.5%, Faithfulness 32.3%—pretty replies still invented temps and FX rates over empty tool results. An AWS ML blog documents the systematic eval approach.

Key Takeaways

  • ✓Open source: awslabs/Agent-EvalKit with slash commands in Claude Code / Kiro / Kilo.
  • ✓Pipeline: plan → data → OTel traces → run → score → code-linked fixes.
  • ✓Insight: demo Faithfulness only 32.3%; empty tool results still hallucinate facts.
ADSponsored