Why it matters
METR’s March 2024 example protocol develops an agent on one task set, then measures performance on held-out tasks with repeated runs and uncertainty estimates. It discusses aggregation across task difficulty, confidence intervals and the effects of sequential stopping. The proposed risk thresholds are illustrative and depend on assumptions about future elicitation and the threat being assessed. The protocol is a historical beta methodology for general autonomy; it does not turn a passing aggregate score into evidence that every deployment is safe.
My takeaway: Specify task splits, run budgets, weights and the stopping rule before comparing systems. Log interventions and failed runs, report uncertainty, and investigate setup errors without silently turning held-out solutions into development guidance.