Why it matters
METR’s MALT dataset combines manually reviewed natural behavior, prompted and edited examples, confirmed benign runs and a larger presumed-benign set to study evaluation tampering. Its monitoring setup chunks transcripts and measures classification with explicit negative controls. Reasoning access helps some labels, but severe natural sandbagging is poorly represented and prompted traces can be conspicuous. Dataset label frequencies are sampling artifacts, and good benchmark detection does not establish that subtle real-world underperformance will be visible in a transcript.
My takeaway: Separate natural, prompted and presumed-benign subsets in evaluation. Hold out task families, calibrate false positives on reviewed negatives, inspect missed cases, and supplement transcript review with cross-task performance checks.