METR’s early-2025 experiment randomized AI access across 246 real issues completed by 16 experienced open-source developers in repositories they knew well. With the tools available then, mainly Cursor with Claude 3.5 and 3.7 Sonnet, AI access increased completion time by 19%, despite participants believing it had accelerated them. The result concerns this sample, workflow and historical toolset; it does not measure novice programmers, unfamiliar projects or current models.
METR’s March 2025 research measures how agent success varies with the time human experts need to complete software and reasoning tasks. A fitted success curve yields a horizon at a chosen reliability level, while hierarchical resampling estimates uncertainty across task families, tasks and attempts. The original paper also extrapolated historical growth, but that forecast depends on the trend continuing and on transfer beyond the measured tasks.
METR’s early-2025 DeepSeek-R1 assessment uses autonomous software tasks and research-engineering environments through a third-party model provider. Transcript inspection found hallucinated tool results, malformed calls and difficulty recovering from mistakes, showing how interface behavior can limit measured performance. The report also acknowledges modest elicitation, incomplete checks for strategic underperformance and potentially biased exclusion of server-error runs.
METR’s February 2025 kernel-engineering study adapts KernelBench and adds tasks from newer ML workloads. It removes cases with nearly constant outputs or insufficient input variation, checks generated solutions and rejects timing artifacts that create misleading speedups. Performance is measured against reference implementations using the best valid result from repeated attempts.
METR’s early-2025 DeepSeek-V3 report combines autonomous-task testing, research-engineering tasks and a held-out knowledge check. The authors observed repeated actions, hallucinated tool outputs and misread numerical feedback, while scaffold changes substantially improved performance. They explicitly caution that elicitation was limited, curated transcripts were not representative and excluded server-error runs could inflate scores. The study did not robustly test deliberate underperformance.
Play video
This vLLM walkthrough explains the tradeoff behind draft-model speculative decoding: a smaller model proposes tokens and the target model verifies them, adding draft computation and memory in exchange for possible faster generation. Acceptance rate, prompt length, output length, sampling settings, and concurrency all affect the result.
Play video
Doug Guthrie’s observability workshop follows a support agent from nested model and tool spans to failed-order-lookup analysis. A custom support-workflow classification reveals failures that a generic issue grouping missed. Selected traces become evaluation cases, with deterministic checks for objective requirements and model judges calibrated against human review. The workflow connects a proposed code change to new failure cases and the existing regression suite.
Play video
Browserbase’s verifier design generates a task rubric and retrieves relevant screenshots for each criterion alongside action history and the final answer. It separates evidence of effort from evidence of completion: a sensible attempt to buy an unavailable product must not become a successful purchase. It also aims to avoid repeatedly penalizing downstream consequences of one earlier error.
Play video
Crusoe describes a Slurm-on-Kubernetes recovery path that marks a failed GPU node down, signals and requeues its jobs, replaces unhealthy capacity, and restarts training from an application checkpoint. The demonstration injects an XID 79 failure signal; it does not physically break a GPU. Recovery depends on replacement policy, spare nodes and checkpoint save/load code.
Play video
Bogdan Gaza describes operational lessons from DatologyAI’s large synthetic-data pipeline: batch object-store metadata listing, size partitions to bound lost work, checkpoint partial outputs, and coordinate Ray CPU heads with GPU workers. Curated source documents seed rephrasing, while a shared workflow connects curation, generation, training and evaluation.
Play video
Jakub Hojsan uses an API migration to show why a plausible diff can be misread when a coding agent relies on stale model knowledge. A version-change rule triggers retrieval of upstream documentation or changelogs, while query-specific excerpts and retrieval traces make the evidence inspectable without loading entire pages. The method separates tool availability from actually invoking verification.
Play video
Jan Čurn demonstrates mcpc, a CLI client that exposes MCP sessions through searchable tool descriptions, structured JSON output and background tasks. Progressive discovery reduces the initial catalog in model context; shell composition lets intermediate results flow between operations without being restated by the model. Authentication and persistent sessions are client responsibilities.
Play video
Suraj Gupta’s publisher notes distinguish agents doing recurring work from agents proposing improvements to that work. Warp’s triage example turns human feedback into a skill-change pull request, while persistent memory retains investigation findings with editing and provenance controls. A separate evaluation loop compares models on recurring task classes.
Play video
Roland Gavrilescu’s publisher notes propose preserving an agent’s prompts, skills, evaluations, tools and environment choices as a versioned configuration. Failures become tests, repeated procedures become skills, and human feedback defines what counts as useful work. Candidate changes should then face production experiments before promotion.
Play video
Aditya Gautam’s publisher notes describe separating video perception, retrieval and review so short content changes remain tied to timestamps and source clips. Production traces help distinguish model errors from tool or retrieval failures before retraining. Small specialized models, caching and frame compression reduce work, but each shortcut can hide relevant content.
ThreatDown’s analysis of an exposed container registry describes a botnet that compromises unauthenticated Docker daemons, establishes host persistence, and installs an unchanged Hermes Agent framework with malicious persona instructions. Operators send post-compromise tasks through Telegram, with AI API keys named as priority targets. The surrounding scripts handle infection, persistence and propagation; the evidence does not make those stages autonomous model decisions.
With the introduction of models that require data sharing with third-party providers—such as Claude Fable 5—organizations need a way to centrally enforce data retention policies. Amazon Bedrock gives you control over whether your prompts and model outputs are retained after an inference request completes.
Play video
AI Engineer session on AgentCraft: Putting the Orc in Orchestration, presented by Ido Salomon. It adds practical context for how teams are building and operating AI systems in production.
Play video
AI Engineer session on Agents need more than a chat - Jacob Lauritzen, CTO Legora. It adds practical context for how teams are building and operating AI systems in production.
Play video
AI Engineer session on Full Workshop: Build Your Own Deep Research Agents - Louis-François Bouchard, Paul Iusztin, Samridhi. It adds practical context for how teams are building and operating AI systems in production.
Play video
AI Engineer session on The Future of MCP, presented by David Soria Parra, Anthropic. It adds practical context for how teams are building and operating AI systems in production.
Play video
AI Engineer session on From Chaos to Choreography: Multi-Agent Orchestration Patterns That Actually Work, presented by Sandipan Bhaumik. It adds practical context for how teams are building and operating AI systems in production.
Play video
AI Engineer session on Agentic Engineering: Working With AI, Not Just Using It, presented by Brendan O'Leary. It adds practical context for how teams are building and operating AI systems in production.
Play video
AI Engineer session on Platforms for Humans and Machines: Engineering for the Age of Agents, presented by Juan Herreros Elorza. It adds practical context for how teams are building and operating AI systems in production.