Adversarial machine learning attacks, taxonomies, and mitigations across the ML lifecycle.
Adversarial ML gives useful language for thinking about evasion, poisoning, privacy attacks, and model misuse. It becomes especially valuable when teams need to distinguish classic model risk from the broader application-layer risks found in modern LLM systems.
These items are included because they add useful evidence, framing, implementation detail, or upcoming context for teams working in this area.
NIST finalizes AI 100-2e2025, providing a terminology and taxonomy for adversarial machine learning across predictive and generative AI systems.
PMLR Volume 299 collects fourteen peer-reviewed CAMLIS papers spanning typographic prompt injection, system-level AI red teaming, white-box LLM backdoors, scam agents, LLM attack defenses, poisoned-model restoration, security knowledge graphs, cloud identity analysis, and production cyber-defense agents. Individual entries provide stable abstracts, citations, and open PDFs, with code or supplemental material where available.
Play video
Edward Raff and collaborators introduce Maximum Violated Multi-Objective attacks for manipulating financial statements while simultaneously reducing model-generated fraud scores. Their evaluation finds roughly 20 times more successful dual-objective attacks than standard methods; in about half of tested cases, earnings could be inflated 100–200% while fraud scores fell 15%.
Unit 42 presents a two-forward-pass method for identifying feed-forward neurons causally tied to a target behavior. In Qwen3-4B, disabling 50 of 350,208 neurons changed the refusal format on 80% of 520 harmful prompts; across 13 tested models, an FFN/Skip ratio explained 81% of measured vulnerability to small targeted changes.
Play video
Sanket Badhe presents ScamAgent, an autonomous multi-turn framework that combines planning, conversational memory, deceptive framing, and text-to-speech to produce realistic scam calls. Evaluation against current model safeguards shows that distributing malicious intent across apparently benign turns can bypass prompt-level refusal and content filtering.
Play video
The AgentFuzz researchers present directed greybox fuzzing for finding paths from attacker-controlled natural-language input to security-sensitive agent operations. Their evaluation across 20 open-source agents combines generated seed prompts, semantic and distance feedback, and argument-aware mutation, reporting 34 high-risk zero-days and 23 assigned CVEs.
NVIDIA argues that most proposed model CVEs actually describe vulnerable serving applications, unsafe serialization and supply-chain formats, access-control failures, or statistical behaviors shared by a model class. A narrow exception may exist for deliberately poisoned training that creates a reproducible backdoor in a specific weight artifact.
NVIDIA shows how a deliberately placed Pickle-backed model can beacon through a DNS canary when loaded, turning unauthorized use of a model artifact into a detection signal. The technique complements access controls and safer formats such as safetensors; it does not make untrusted Pickle files safe to load.
garak v0.16.0 begins context-aware scanning with technique and intent annotations and IntentProbe, adds native Anthropic and adaptive-attack plugins, and introduces a unified selection grammar plus revised report output.
Play video
Ashley Song and collaborators evaluate test-time compute strategies on two operational cybersecurity agents: a container vulnerability analysis workflow and a server-alert triage system. The study examines whether allocating more inference-time reasoning can improve both answer accuracy and consistency across repeated runs.
Play video
BlackIce packages fourteen open-source responsible-AI, LLM-security, and adversarial-ML tools into a reproducible, version-pinned container with a unified command-line interface. The CAMLIS presentation explains tool selection, coverage, dependency isolation, image architecture, and a working assessment demonstration rather than presenting the bundle as a substitute for test design.
garak 0.14.0 adds JSON configuration, redesigned HTML reports, detector calibration and benchmarking guidance, new generator integrations, and broader report records. It also changes the eval and digest JSONL formats and aggregate scoring, making the release operationally important for teams that parse reports or compare historical baselines.
TrustFall shows how project-defined MCP configuration can turn a generic “trust this folder” decision into unsandboxed command execution in several coding agents, with zero-click variants in unattended CI. The vendor-authored research traces the issue to conflating permission to read or edit a workspace with permission to start repository-supplied executables.
NVIDIA analyzed nearly 140 GB of Meta's Kaggle for Code corpus and found more than 140 active plaintext third-party credentials, widespread pickle deserialization, common import typos, and no imports of several adversarial-testing libraries. The study cautions that isolated competition notebooks still shape code and habits that migrate into production.
Trail of Bits bypassed multiple agent-skill scanners with compiled Python hidden beside benign source and with prompt-like prose that persuaded an LLM classifier to accept a malicious configuration. The experiments show recurring blind spots around unreferenced files, binaries, assets, and ambiguous installer behavior, and also explain why legitimate skills can contain patterns that look malicious.
Play video
Text2VLM is a reproducible pipeline that extracts harmful concepts from text-only safety datasets and renders them as typographic images for multimodal evaluation. Human validation supports the transformation pipeline, and tests of open-source visual language models find greater prompt-injection susceptibility when the same concepts arrive through images instead of plain text.
GuardFall tests 11 open-source coding and computer-use agents against shell-command transformations that evade string and regex deny lists, including quote removal, $IFS expansion, command substitution, and encoded payloads. The study finds configuration- and model-dependent failures and shows that local or auto-approve modes can turn untrusted repository content into host command execution; it is vendor-authored research, not an independent benchmark.
Google introduced Gemini Omni Flash, a multimodal model that combines text, image, audio, and video references to generate and iteratively edit video through natural-language conversation. Generated videos include a SynthID watermark.
Play video
The NDSS-backed research identifies six inference-time cache attacks across vLLM, SGLang, GPTCache, and related stacks. Weak prefix and image cache keys plus semantic near-match errors can make distinct inputs share cached state, enabling poisoned responses, information leakage, and moderation bypass; the authors provide experimental artifacts and vendor disclosures.
NVIDIA demonstrates a model supply-chain attack in which a privileged adversary edits a tokenizer JSON file so visible words map to different token IDs. The change can make the model interpret "deny" as "allow" or corrupt decoded output while leaving the model weights untouched.
Play video
Fuzzinglabs researchers explain how threat modeling, file-format fuzzing, and plugin analysis exposed an authentication bypass and memory-corruption issues in Ollama plus command injection in NVIDIA Triton Inference Server's model-configuration pipeline. The Pwn2Own case study also examines RedisAI, ChromaDB, and container-runtime attack surfaces.
Adversa's IICL study evaluates a few-shot jailbreak that interleaves benign and harmful demonstrations and uses short output-field labels to shift model behavior. Across more than 3,500 probes, ten models, and seven ablations, results vary materially with example order and field names; the work is vendor-authored and its model-specific attack rates should be independently reproduced.
SymJack demonstrates that a user-approved, apparently harmless copy command can write through a repository-controlled symlink into executable agent configuration, producing code execution when the tool restarts. The vendor-authored study reports variants across six coding agents and highlights a gap between approval text, shell semantics, and the resolved filesystem target.
Release notes for garak, an LLM vulnerability scanning and evaluation toolkit. Relevant to tracking new probes, detectors, and repeatable red-team workflows.