KaliBench Sets Deterministic Benchmark for Cybersecurity CLI Tool Use: 8B Model Rivals 685B MoE via Verifiable Rewards
Large language models are rapidly deployed across cybersecurity workflows to translate analysts' intent into command-line interface (CLI) executions. However, existing benchmarks emphasize either high-level knowledge QA or unconstrained agent rollouts, failing to measure precise parameter binding across real-world security tooling where minor syntax glitches invalidate execution. Researchers from MBZUAI introduce KaliBench, a fine-grained natural-language-to-CLI benchmark on Kali Linux spanning 8,504 query-command pairs across 1,642 tools, 23 capability dimensions, and 5 security phases. Auditing reveals no open-weight model surpasses 42% exact accuracy without tool hints. Furthermore, reinforcement learning with KaliBench's runtime-free verifiable rewards elevates an 8B model to rival a 685B MoE frontier model.
Key Takeaways
- First fine-grained cybersecurity CLI benchmark spanning 1,642 Kali Linux tools and 8,504 verified query-command pairs
- Reveals open-weight models fail to exceed 42% exact CLI accuracy in unconstrained real-world settings
- Introduces runtime-free verifiable rewards, empowering an 8B model to match a 685B parameter MoE architecture