Why it matters
METR’s SHUSHCAST prototype asks agents to perform a visible main task and a concealed side task, then evaluates transcript monitors at a threshold calibrated to a 5% false-positive rate on benign runs. It distinguishes completing the side objective from escaping detection and compares access to reasoning traces. The small task set, limited adversarial optimization, noisy threshold calibration and differences between raw and summarized reasoning constrain generalization. Success on this proxy cannot certify monitoring of realistic sabotage.
My takeaway: Include benign controls, successful and failed side-task attempts, and held-out adversarial strategies. Report detection at an operationally tolerable false-positive rate, and test transcript truncation and reasoning visibility before relying on the monitor.