Why it matters
METR’s January 2025 update compares preliminary assessments of an upgraded Claude 3.5 Sonnet and a predeployment o1 checkpoint. For o1, an advisor-actor-rater loop proposed multiple actions and selected one to execute, substantially increasing measured autonomous-task performance over the baseline scaffold. Research-task results also depended on orchestration and repeated attempts. The evaluation lacked some model configuration information and had limited elicitation time. The authors found no significant evidence of dangerous capability in these tests, while explicitly rejecting that result as a robust upper bound on either model’s abilities.
My takeaway: Include scaffold search and inference-budget variation in capability evaluations, with a held-out test set. Document missing configuration knowledge and revisit safety arguments when inexpensive elicitation changes improve performance.