Why it matters
METR’s July 2025 analysis asks whether task-completion horizons transfer beyond its original software evaluations. It reanalyzes nine existing benchmarks, fitting success against human completion time where suitable measurements are available. The relationship is useful in several settings but differs across domains and weakens when proxies such as video length or task fees stand in for human effort. Some fits require constrained assumptions, and agents, datasets and scaffolds are not matched across benchmarks. The study supports checking the metric’s validity locally rather than treating a single horizon as a universal measure of autonomy.
My takeaway: Before reporting a time horizon, test whether measured human effort predicts success in your task distribution. Publish fit assumptions, uncertainty and domain-specific results; do not substitute unrelated duration or price proxies without validation.