Why it matters
METR’s March 2024 study compares model versions and agent scaffolds on 195 tasks across twelve families. Later post-training versions of the same base models improved measured autonomy, while the additional scaffold gains studied were small and not statistically significant. The task selection, limited enhancement methods and lack of model-weight access constrain generalization. The study demonstrates why an evaluation should specify the exact model version, tools, prompts and selection procedure; it does not establish a ceiling on gains from future elicitation.
My takeaway: Compare one model or scaffold change at a time on a fixed held-out task set. Count retry and judge-model compute in the budget, retain uncertainty estimates, and avoid attributing a gain to prompting when the underlying model version also changed.