Topic

AI Red Teaming

Methods, case studies, and tooling for red teaming AI systems end to end.

ai red teamingllm red teamingjailbreakadversarial testingpyritpromptfoo
Evergreen Overview

AI red teaming is the practice of testing AI-enabled systems the way an adversary, abusive user, or curious operator would interact with them in production. The real work usually sits in the surrounding application context rather than in isolated model prompts.

What AI red teaming includes
  • Prompt abuse, indirect injection, and trust-boundary failures
  • Tool misuse, privilege expansion, and unsafe action chains
  • System-level evaluation of how the model, workflow, and controls behave together
What teams usually need to answer
  • What an attacker can influence, read, or trigger through the model
  • Where approvals, isolation, monitoring, or policy controls are missing
  • Which failures are model problems versus product and architecture problems
Who this page is for
  • People studying AI evaluation and red-team programs
  • Product and platform teams launching copilots or agents
  • Leaders who need concrete examples of AI risk in operational systems
References

Current notes, events, and source material

These items are included because they add useful evidence, framing, implementation detail, or upcoming context for teams working in this area.

The Hacker News AI Security July 13, 2026 news

New MemGhost Attack Plants Persistent False Memories in AI Agents Through One Email

Give an AI assistant a memory and access to your inbox, and you hand an attacker a way to rewrite what it thinks it knows about you. A single email can trick that agent into saving a false "fact" about the user, hide the change, and quietly steer its answers in later sessions. When it works, the person reads an ordinar

The Hacker News AI Security July 10, 2026 news

Researcher Details WhatsApp-to-Host Attack Chain Using Three OpenClaw Flaws

Details have emerged about three now-patched security flaws in the OpenClaw personal artificial intelligence (AI) assistant that, if successfully exploited, could enable credential theft, privilege escalation, and arbitrary code execution on the host. A brief description of the high-severity vulnerabilities is as follo

The Hacker News AI Security July 9, 2026 news

Top AI Agents Built to Catch Malicious Code Can Be Tricked Into Running It

Ask an AI coding agent to scan open-source code for security holes, and it might run the attacker's code on your own machine instead. That is the finding in a proof-of-concept published Wednesday by the AI Now Institute, an attack it calls "Friendly Fire." It works against Anthropic's Claude Code and OpenAI's Codex whe

The Hacker News AI Security July 9, 2026 news

GhostApproval Symlink Flaws Could Let Malicious Repos Run Code in AI Coding Agents

Researchers at Wiz found that a flaw in six popular AI coding assistants lets a booby-trapped code project quietly take control of a developer's computer. The assistant asks permission to edit one harmless-looking file, but the write lands on a sensitive one instead. The affected tools are Amazon Q Developer, Anthropic

The Hacker News AI Security July 8, 2026 news

AI Coding Agents Found Triggering Endpoint Security Rules Built to Catch Attackers

Sophos looked at a week of its own endpoint data and found that AI coding agents such as Claude Code, Cursor, and OpenAI Codex are setting off detection rules written to catch human intruders. The agents are not malicious. They just do a lot of things that, to a behavioral engine, look exactly like an attack. Decryptin

The Hacker News AI Security July 8, 2026 news

CISA Adds 4 Actively Exploited Adobe, Joomla, and Langflow Flaws to KEV

The U.S. Cybersecurity and Infrastructure Security Agency (CISA) on Tuesday added four security flaws to its Known Exploited Vulnerabilities (KEV) catalog, citing evidence of active exploitation. The vulnerabilities are listed below - CVE-2026-48282 (CVSS score: 10.0) - A path traversal vulnerability in Adobe ColdFusio

The Hacker News AI Security July 8, 2026 news

GitHub Copilot Refuses Harmful Requests in Chat, Then Writes Them in Code

An AI coding assistant that refuses to answer a dangerous request in its chat box can answer it anyway if the same request is broken into small, ordinary-looking steps inside a code editor. That is the finding of a new study of GitHub Copilot by researchers Abhishek Kumar and Carsten Maple. The models they tested throu

The Hacker News AI Security July 7, 2026 news

Public GitHub Issue Could Trick GitHub Agentic Workflows Into Leaking Private Repo Data

A public issue can trick GitHub Agentic Workflows into leaking the contents of an organization's private repositories, researchers at Noma Security have shown. The attacker needs only to open a normal-looking issue on a public repository, with no stolen credentials and no access to the organization. If that organizatio

The Hacker News AI Security July 7, 2026 news

Writer AI Flaw Could Let Agent Previews Leak Session Tokens Across Tenants

Cybersecurity researchers have disclosed details of a now-patched critical session isolation vulnerability in Writer, an enterprise generative artificial intelligence (AI) platform, that could result in cross-tenant compromise. The one-click vulnerability has been codenamed WriteOut by the Sand Security Research team.

POC Prison: Why agentic systems never escape the lab and how to fix that in 90 days - Luise Freese video thumbnail Play video
NDC Conferences YouTube July 7, 2026 video

POC Prison: Why agentic systems never escape the lab and how to fix that in 90 days - Luise Freese

This talk was recorded at NDC Copenhagen in Copenhagen, Denmark. #ndccopenhagen #ndcconferences #developer #softwaredeveloper Attend the next NDC conference near you: https://ndcconferences.com https://ndccopenhagen.com/ Subscribe to our YouTube channel and learn every day: / @NDC Follow our Social Media! https://www.f

Put an Agent Inside Your App in 10 Minutes or Less with the GitHub Copilot SDK - Daniel Ward video thumbnail Play video
NDC Conferences YouTube July 7, 2026 video

Put an Agent Inside Your App in 10 Minutes or Less with the GitHub Copilot SDK - Daniel Ward

This talk was recorded at NDC Copenhagen in Copenhagen, Denmark. #ndccopenhagen #ndcconferences #developer #softwaredeveloper Attend the next NDC conference near you: https://ndcconferences.com https://ndccopenhagen.com/ Subscribe to our YouTube channel and learn every day: / @NDC Follow our Social Media! https://www.f

Building an AI-Powered Personal Companion Mobile App - Gerald Versluis - NDC Copenhagen 2026 video thumbnail Play video
NDC Conferences YouTube July 7, 2026 video

Building an AI-Powered Personal Companion Mobile App - Gerald Versluis - NDC Copenhagen 2026

This talk was recorded at NDC Copenhagen in Copenhagen, Denmark. #ndccopenhagen #ndcconferences #developer #softwaredeveloper Attend the next NDC conference near you: https://ndcconferences.com https://ndccopenhagen.com/ Subscribe to our YouTube channel and learn every day: / @NDC Follow our Social Media! https://www.f

Microsoft Security Blog July 6, 2026 news

5 insights from Frost & Sullivan’s 2025 Frost Radar™ for Cloud Security Posture Management

Read five key learnings from the Frost & Sullivan 2025 Frost Radar™ for CSPM to learn how CSPM is evolving from point-in-time compliance to continuous risk management. The post 5 insights from Frost & Sullivan’s 2025 Frost Radar™ for Cloud Security Posture Management appeared first on Microsoft Security Blog .

The Hacker News AI Security June 22, 2026 news

Stop Your Legacy Infrastructure from Hijacking Your AI Agents

Earlier this month, I spoke at the Gartner Security & Risk Management Summit about a blind spot most security programs are still not accounting for - how attackers are circumventing AI security programs by using legacy infrastructure to hijack AI agents. AI adoption is moving faster than security programs can account f

The Hacker News AI Security June 11, 2026 news

New Attacks Trick OpenClaw AI Agent Into Running Code and Leaking Secrets

Two security teams have shown, in separate research published this week, that OpenClaw, the popular self-hosted AI agent, can be driven to run attacker-controlled code or hand over sensitive data through ordinary-looking inputs. Imperva buried instructions inside shared contacts, vCards, and location pins that the agen

Krebs on Security May 12, 2026 news

Patch Tuesday, May 2026 Edition

Artificial intelligence platforms may be just as susceptible to social engineering as human beings, but they are proving remarkably good at finding security vulnerabilities in human-made computer code. That reality is on full display this month with some of the more widely-used software makers -- including Apple, Googl

Anthropic Frontier Red Team April 7, 2026 news

Assessing Claude Mythos Preview’s cybersecurity capabilities

Claude Mythos Preview is a new general-purpose language model that is strikingly capable at computer security tasks. This post provides technical details for researchers and practitioners who want to understand exactly how we have been testing this model, and what we have found over the past month. We hope this will sh

Krebs on Security March 8, 2026 news

How AI Assistants are Moving the Security Goalposts

AI-based assistants or "agents" -- autonomous programs that have access to the user's computer, files, online services and can automate virtually any task -- are growing in popularity with developers and IT workers. But as so many eyebrow-raising headlines over the past few weeks have shown, these powerful and assertiv

Anthropic Frontier Red Team December 18, 2025 news

Project Vend: Phase Two

In June, we revealed that we'd set up a small shop in our San Francisco office run by an AI shopkeeper. It did not do particularly well. We made some adjustments for phase two of Project Vend. The idea of an AI running a business doesn't seem as far-fetched as it once did. But the gap between 'capable' and 'completely