Jason Ma’s publisher notes describe using a video-based progress model to find robot failures, then collecting human demonstrations of recovery and fine-tuning the policy. Napkin folding exposes both bad grasps and quality failures that a nominal task-completion measure can miss. The reported 24-hour success rate belongs to a particular folding evaluation, with no sample size supplied in the notes. Transfer to a new site and recovery from unfamiliar mistakes require separate evidence.
Robot reliability: evaluate recovery and new-site transfer separately
Related research
More curated notes connected through Model Evaluation and AI Engineering.
Pacing model development in an era of cyber-critical capabilities
OpenAI says preliminary evidence that Astra may meet its Critical cybersecurity threshold led it to pause frontier reinforcement-learning work for two weeks and keep its largest planned run on hold. New safeguards include stronger workload and network isolation, continuous boundary testing, token-level monitoring that escalates suspicious tool activity, and broader alignment checks for deception, reward hacking, and unauthorized access.
A blueprint for democratic governance of frontier AI
OpenAI proposes a three-part U.S. frontier-AI governance model: harmonize emerging state safety laws into a federal baseline, strengthen CAISI as an evaluation and standards institution, and coordinate a broader resilience program. Proposed controls include severe-risk evaluations, transparency reports, independent audits, safety-incident reporting, model-weight security, whistleblower protection, and periodic technical assessments.
OpenAI’s Frontier Governance Framework
OpenAI's 22-page Frontier Governance Framework maps its frontier-model processes to California's Transparency in Frontier AI Act and the EU AI Act's general-purpose AI code. It documents lifecycle risk assessment, cyber-offense and other risk tiers, mitigation and residual-risk decisions, critical-incident handling, security risk management, model reporting, external review, responsibility allocation, and change control.