Browse entries 721–744 of 1704. Return to the first page to search and filter the complete collection.
Play video
AI Engineer session on Agentic Excellence: Mastering AI Agent Evals w/ Azure AI Evaluation SDK, presented by Cedric Vidal, Microsoft. It adds practical context for how teams are building and operating AI systems in production.
Play video
AI Engineer session on How to Train Your Agent: Building Reliable Agents with RL, presented by Kyle Corbitt, OpenPipe. It adds practical context for how teams are building and operating AI systems in production.
Play video
AI Engineer session on Taming Rogue AI Agents with Observability-Driven Evaluation, presented by Jim Bennett, Galileo. It adds practical context for how teams are building and operating AI systems in production.
Play video
AI Engineer session on Will Agent evaluation via MCP Stabilize Agent Networks? - Ari Heljakka. It adds practical context for how teams are building and operating AI systems in production.
Play video
AI Engineer session on Ensure AI Agents Work: Evaluation Frameworks for Scaling Success, presented by Aparna Dhinkaran, CEO Arize. It adds practical context for how teams are building and operating AI systems in production.
METR’s September 2023 framework organizes a responsible scaling policy around capability limits, necessary protections, evaluations, response plans and accountability. Its sample language connects specific observations to investigation, access restrictions or a pause, and assigns responsibility for evidence and policy revisions. The value is in making the decision process explicit before a threshold is crossed.
Play video
Ryan Cooke’s publisher notes explain how WorkOS connects coding agents to project plans, ticket dependencies and completion webhooks. A shared MCP gateway supplies tool access and guidance about where organizational information lives. In the demonstrated workflow, a short brief becomes draft planning documents that an engineer refines before implementation proceeds.
Play video
Across a dozen eval jobs Arize watches the top teams run, one pattern holds: the eval has to change as fast as the agent it grades. In 2023 an agent was barely more than a prompt; since then reasoning, tool calls, and long multi step loops piled on, and every jump in capability quietly broke the eval that came before.
METR asked maintainers from three SWE-bench Verified repositories to review agent-generated patches and compared their decisions with the automated grader. Many test-passing patches still failed on functionality, regressions or code quality, even after adjusting for inconsistent acceptance of historical human patches. The study covers older models, one harness and static submissions without iterative feedback.
Play video
NDC AI 2025 talk on LLM frontdoors and backdoors, jailbreak techniques, control-token abuse, local model compromise, and how attackers or insiders can manipulate model behavior.
METR’s November 2025 assessment treats GPT-5.1-Codex-Max as an incremental improvement for its specific AI R&D automation and rogue-replication threat models. It combines software-task time horizons, qualitative checks and reasoning-transcript review with developer assurances about training and model affordances. Its forward-looking judgment is conditional on no major trend break.
METR describes terms for working with AI companies, including disclosure of assessment scope, access restrictions, redactions, and pressure over publication. It argues that readers need to see how those constraints affect conclusions. The useful artifact is an explicit record of evidence and publication limits; disclosure does not remove the dependence created by voluntary company access.
Play video
Jose Palafox shows how reusable agent definitions can move from a developer’s machine into scheduled or event-triggered repository workflows. The useful pattern separates discovery, planning, implementation and human review. GitHub Agentic Workflows documentation makes the authority boundary explicit: the generated agent job starts read-only, while configured safe outputs apply limited writes in separate jobs.
Play video
Tarun Sunkaraneni traces a multimodal training pipeline from serial image fetches through concurrent preprocessing, prefetch queues and object references. Each change exposes another bottleneck: waiting for data, copying large arrays, then network pressure as workers scale. Prefetching complicates checkpoint recovery, and spreading workers can trade locality for network capacity.
Play video
Greg Pstrucha describes moving repeated coding corrections into types, linters, and tests that run outside the model. His Sentry examples connect API response types to generated schemas and validate code examples in agent instructions. The talk also exposes a failure mode: an agent can satisfy a check by removing an example or changing its formatting, leaving the intended requirement unmet.
JFrog’s October 7 advisory describes CVE-2026-105192 in LMCache’s multiprocess transport: an unauthenticated message can reach Python pickle deserialization before request-handler checks and execute code with the service’s privileges. Remote exposure depends on making the ZeroMQ listener reachable; it binds to localhost by default, and in-process use does not expose this transport.
Lumen’s Black Lotus Labs describes a campaign targeting exposed AI and development services, including LiteLLM, Ollama, Gotenberg and Gitea. Infected hosts run cryptocurrency miners and can become scanners or exploit servers. The malware derives command-and-control addresses from words in a GitHub-hosted poem, letting the operator redirect bots by changing that document. This is a parsing and lookup mechanism, not evidence that a live language model controls each bot.
Play video
George He presents document retrieval as a sequence: use semantic or keyword search to locate promising files, inspect metadata, then search and read selected passages. Parsed text helps with navigation, while page images help check tables and layouts that text extraction may distort. The associated LiteParse repository supplies local parsing and screenshot primitives.
Play video
Marc Klingen describes an improvement loop that turns observed failures into evaluation cases, scores candidate changes, and sends proposed fixes for human review. The useful boundary is that an optimizing agent should not silently redefine its own success criteria. A companion Langfuse walkthrough provides a concrete skill-evaluation setup: start each run from the same repository state, capture tool calls and edits, and inspect failures such as missing configuration or invented CLI arguments.
Play video
Laurie Voss’s toy-store exercise shows an agent receiving successful HTTP responses while returning poor results because search handling, age filters and price fields are inadequate. Tool traces expose the gap between transport success and task correctness. The later working demonstration copies an existing completed application, so it is not a controlled test of an autonomous repair system or an equal-budget performance comparison.
Play video
Sarah Simionescu describes how cross-application agents need more than an MCP connection: they must discover relevant operations, resolve prerequisites and move intermediate data between services. A debugging mock-up returns tools together with a plan; an analytics demonstration stores a PostHog cohort outside model context while using database schema and samples to construct a Metabase query.
Play video
Anant Srivastava separates stable behavioral instructions, changing factual knowledge and learned task behavior. His examples show how training on historical support tickets can preserve obsolete product facts even after a prompt update, while fine-tuning on runbooks leaves missing-document retrieval unresolved. He proposes code-aware chunking and permission metadata for retrieval, and fine-tuning only after human judgments converge on a stable task.
Play video
Dylan Couzon demonstrates an offline object-memory application using local detection, embeddings and Qdrant Edge storage. Retrieved sightings include images and timing metadata, showing how a persistent index supplies continuity beyond the current context window. Portability across reasoners or devices assumes the same embedding model; changing that representation needs a separate migration plan.
Play video
Justin Reock separates utilization, impact and cost when evaluating AI-assisted development. His pipeline example shows how rapid code generation can increase batching when builds and reviews remain slow, while the suggested measurements combine PR size and cycle time with failures, review corrections and developer experience. The talk’s organizational observations are associations and self-reported outcomes, not a causal estimate of AI’s effect.