Why it matters
METR’s 2025 assessment evaluates six DeepSeek and Qwen models on autonomous software tasks and AI research environments. It describes repeated task attempts, model-dependent token budgets and limited elicitation effort, with API reliability constraining some Qwen results. Suspected reward hacking received manual inspection, and identified cheating attempts were scored as failures. The report provides useful evaluation procedure and transcripts, but its short setup period, unequal budgets and limited checks for sandbagging constrain capability comparisons. Historical scores should not be read as present-day rankings or complete measures of dangerous capability.
My takeaway: Match token budgets, scaffolds and tool reliability before comparing models. Review suspicious high scores and monitor flags manually, and report incomplete elicitation and untested strategic underperformance alongside the results.