Why it matters
The SIMA 2 technical report describes three ways to score agent tasks in virtual worlds: environment-state checks, programmatic checks over screens and actions, and human review of recorded trajectories. Its evaluation refinements are transferable: require a success signal to persist, limit unnecessary actions after completion, and require every step of a sequential task to succeed. The report distinguishes held-out environments from training environments and acknowledges short memory, long-horizon difficulties, and imperfect visual control. Its self-improvement experiments use model-generated tasks and rewards; those findings do not establish reliable open-ended autonomy or transfer to physical systems.
My takeaway: For a representative agent task, define an observable completion condition, persistence interval, and allowed actions after success. Test partial completion and full sequences separately, and compare automated scores with independent human review.