Why it matters
Microsoft AI Red Team engineer Nina Chikanov shows how PyRIT supported a ten-day multimodal Sora assessment and a GPT-5 operation spanning roughly one million conversations and eighteen harm areas. The workflow combines labeled datasets, custom targets, prompt transformations, single- and multi-turn attacks, scorers, retries, rate limits, and a shared evidence store while documenting important automation gaps.
My takeaway: Build repeatable operations from versioned datasets, target adapters, transformations, scoring criteria, and retained conversations. Calibrate automated scorers against human review, label results by harm area, budget time for custom targets, and keep manual paths for video, UI, agentic, application-security, and other scenarios that PyRIT cannot evaluate reliably.