KaliBench releases its data, scoring, Docker evaluation and training recipes.
KaliBench tests whether language models can compose security commands
2026-10-02Frontier watchTRIAL
KaliBench introduces 8,504 verified query-command pairs spanning 1,642 Kali Linux subtools and 23 capability dimensions. It separates 3,504 training examples from 5,000 held-out tests and releases scoring, Docker evaluation, data and RedSage-K training recipes. In the authors' evaluation, the best unrestricted open-weight model remained below 42% accuracy.
What this means for you
For teams building security agents, KaliBench turns command construction into a reproducible capability test with public artifacts. A one-day baseline can reveal whether a model understands exact flags and composition before anyone grants it a live shell.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Dataset | 8,504 verified query-command pairs; 3,504 train and 5,000 held-out |
|---|---|
| Coverage | 1,642 Kali Linux subtools across 23 dimensions |
| Artifacts | data, exact scoring, Docker evaluation and SFT/GRPO recipes |
| Author Result | best unrestricted open-weight model below 42% accuracy |
| Safety | evaluation can score command structure without granting a model a live target |
Technical impact 3
Engineering value 4
Adoption signal 2
Maturity 3
Verification cost 3
Risk 3