PTXBench is a benchmark for evaluating and adapting LLMs on GPU kernel optimization with architecture-specific PTX, NVIDIA's low-level IR. On GEMM and attention workloads across H100 and B200, it measures three things: functional correctness, whether the selected target instructions actually execute at runtime, and speedup over frontier libraries such as cuBLAS and cuDNN. The headline finding is negative: no evaluated model consistently matches the tuned libraries, success rates fall sharply on complex attention-backward workloads, and executing the target instructions does not translate into competitive performance. The authors also adapt Qwen3.6-27B with repair-conditioned SFT, which improves several tasks but generalizes unevenly; data coverage, balance, and reasoning-teacher quality matter beyond dataset size. PTXBench is positioned as an auditable testbed for measuring and improving how well LLMs exploit evolving GPU architectures.
PTXBench: LLMs emit exotic PTX instructions but can't beat tuned libraries
PTXBench is a benchmark for evaluating and adapting LLMs on GPU kernel optimization with architecture-specific PTX, NVIDIA's low-level IR. On GEMM and attention workloads across H100 and B200, it measures three things: functional correctness, whether the selected target instructions actually execute at runtime, and speedup over frontier libraries such as cuBLAS and cuDNN. The headline finding is negative: no evaluated model consistently matches the tuned libraries, success rates fall sharply on complex attention-backward workloads, and executing the target instructions does not translate into competitive performance. The authors also adapt Qwen3.6-27B with repair-conditioned SFT, which improves several tasks but generalizes unevenly; data coverage, balance, and reasoning-teacher quality matter beyond dataset size. PTXBench is positioned as an auditable testbed for measuring and improving how well LLMs exploit evolving GPU architectures.
Imported from the existing research archive. Migration preserves the recorded judgment and is not a fresh verification.
What this means for you
Kernel generation is where LLMs could compound hardest: hardware refreshes every year and kernel engineers are scarce. For teams building kernel-agent pipelines, the actionable part is the negative result — treat speedup, not instruction selection, as the success signal, because today's models can emit exotic instructions without beating tuned libraries. This is one of the first benchmarks to go below CUDA to architecture-specific PTX on both H100 and B200, with runtime verification that claimed instructions actually execute; it extends the KB's GPU-kernel evidence line (ev-20260813-06) from verification toward capability measurement.
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Scope | GEMM + attention workloads; H100 + B200; architecture-specific PTX |
|---|---|
| Metrics | functional correctness · target-instruction runtime execution · speedup vs frontier libraries |
| Findings | success rates drop sharply on attention-backward · instruction execution ≠ competitive performance · no model consistently matches frontier libraries |
| Adaptation | SFT on Qwen3.6-27B; repair-conditioned training helps per-task; generalization uneven; data coverage/balance/teacher quality matter beyond size |
| Code | no repo link on abstract page; benchmark claimed as auditable testbed |
| Relation To Kb | extends GPU-kernel × LLM evidence line started by arXiv 2608.12700 contract-grade verifier (ev-20260813-06) |