PTXBench: LLMs emit exotic PTX instructions but can't beat tuned libraries

PTXBench is a benchmark for evaluating and adapting LLMs on GPU kernel optimization with architecture-specific PTX, NVIDIA's low-level IR. On GEMM and attention workloads across H100 and B200, it measures three things: functional correctness, whether the selected target instructions actually execute at runtime, and speedup over frontier libraries such as cuBLAS and cuDNN. The headline finding is negative: no evaluated model consistently matches the tuned libraries, success rates fall sharply on complex attention-backward workloads, and executing the target instructions does not translate into competitive performance. The authors also adapt Qwen3.6-27B with repair-conditioned SFT, which improves several tasks but generalizes unevenly; data coverage, balance, and reasoning-teacher quality matter beyond dataset size. PTXBench is positioned as an auditable testbed for measuring and improving how well LLMs exploit evolving GPU architectures.

Kernel generation is where LLMs could compound hardest: hardware refreshes every year and kernel engineers are scarce. For teams building kernel-agent pipelines, the actionable part is the negative result — treat speedup, not instruction selection, as the success signal, because today's models can emit exotic instructions without beating tuned libraries. This is one of the first benchmarks to go below CUDA to architecture-specific PTX on both H100 and B200, with runtime verification that claimed instructions actually execute; it extends the KB's GPU-kernel evidence line (ev-20260813-06) from verification toward capability measurement.