The paper presents a verifier for AI-generated GPU kernels built on twelve adversarial property-based gates, most of them tolerance-free; it agrees with reference correctness code 98.5% of the time. The authors used it to audit 2,638 publicly accepted machine-generated kernels: 39.5% were broken beyond any tolerance argument, and 62.1% had at least one violation. Field-standard tests accepted 1,487 kernels that the verifier rejected.
Verifier finds ~40% of accepted LLM-generated GPU kernels are broken
2026-08-13Frontier watchTRIAL
The paper presents a verifier for AI-generated GPU kernels built on twelve adversarial property-based gates, most of them tolerance-free; it agrees with reference correctness code 98.5% of the time. The authors used it to audit 2,638 publicly accepted machine-generated kernels: 39.5% were broken beyond any tolerance argument, and 62.1% had at least one violation. Field-standard tests accepted 1,487 kernels that the verifier rejected.
Imported from the existing research archive. Migration preserves the recorded judgment and is not a fresh verification.
What this means for you
For any team shipping LLM-generated code validated only by loose tests, this is a directly actionable warning: property-based adversarial gates belong in kernel and code CI.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Kernels Audited | 2638 |
|---|---|
| Broken Beyond Tolerance | 39.5% |
| At Least One Violation | 62.1% |
| Agreement With Reference | 98.5% |
| Code | Zenodo artifact, no GitHub repo |
Technical impact 3
Engineering value 4
Adoption signal 2
Maturity 2
Verification cost 3
Risk 2