Kernel-engineering evaluation filters false speedups and weak correctness tests

METR’s February 2025 kernel-engineering study adapts KernelBench and adds tasks from newer ML workloads. It removes cases with nearly constant outputs or insufficient input variation, checks generated solutions and rejects timing artifacts that create misleading speedups. Performance is measured against reference implementations using the best valid result from repeated attempts. The study is limited to single-GPU inference, fixed shapes and approximate output matching; it lacks a matched human baseline and does not establish training stability or multidevice performance. Its main evaluation lesson is that correctness and timing methodology must be audited together.