Why it matters
OpenAI and Paradigm’s February 2026 EVMbench release evaluates three distinct security capabilities using 117 historical vulnerabilities from 40 audits. Detection is scored against known findings; patching must remove exploitability while retaining functionality; exploitation is evaluated by replaying transactions in isolated local blockchain environments. The setup includes custom graders and checks against grader abuse. The benchmark does not capture all real-world contract security: detection cannot reliably adjudicate novel findings, and exploit grading omits timing-dependent behavior, mainnet state and multichain interactions. Strong exploit scores therefore do not establish equally strong auditing or repair.
My takeaway: Evaluate discovery, remediation and exploit validation separately. Run historical cases in isolated environments, retain functionality tests for patches, and report coverage gaps instead of treating one aggregate score as security assurance.