CAMLIS · November 8, 2026

CTI-REALM: evaluate detection agents on query results and investigation steps

CTI-REALM: evaluate detection agents on query results and investigation steps video thumbnail
Why it matters

CTI-REALM tests whether an agent can turn threat intelligence into executable detections. The research behind this CAMLIS presentation uses telemetry from attacks emulated on isolated Linux, Azure Kubernetes Service and Azure environments. Agents work in a separate containerized evaluation environment, inspect schemas, query logs and submit Sigma rules with KQL detection queries. The 25-task set is a subset of the 50-task set. Scoring combines intermediate investigation checkpoints with detection quality: query results receive deterministic checks, while an LLM judge assesses parts of the report and rule quality. The overall reward is therefore not a standalone detection-accuracy percentage. Cloud scenarios were harder, but they also contain more complex tasks, so the category comparison mixes platform and difficulty. The benchmark is Azure-centric and does not test production-scale query performance or long-baseline anomaly detection. It provides a reproducible workflow evaluation, not certification that generated rules can be deployed without review.

My takeaway: Use development tasks to tune tools and reserve held-out tasks for comparison. Inspect checkpoint scores and actual matched events, then test proposed rules against representative benign traffic and production-scale telemetry. Validate Sigma and KQL outputs separately before promoting a detection.
Keep exploring

More curated notes connected through Model Evaluation and AI Red Teaming.

OpenAI News · framework

Pacing model development in an era of cyber-critical capabilities

OpenAI says preliminary evidence that Astra may meet its Critical cybersecurity threshold led it to pause frontier reinforcement-learning work for two weeks and keep its largest planned run on hold. New safeguards include stronger workload and network isolation, continuous boundary testing, token-level monitoring that escalates suspicious tool activity, and broader alignment checks for deception, reward hacking, and unauthorized access.

OWASP GenAI Security Project · guide

OWASP Top 10 for Agentic Applications for 2026

OWASP's community guide organizes agentic-system risk into ten categories, including goal hijacking, tool misuse, identity and privilege abuse, memory poisoning, insecure inter-agent communication, cascading failures, and rogue-agent behavior. It provides a shared taxonomy and mitigation starting point rather than a certification checklist or evidence that a deployed system is secure.

OpenAI News · framework

A blueprint for democratic governance of frontier AI

OpenAI proposes a three-part U.S. frontier-AI governance model: harmonize emerging state safety laws into a federal baseline, strengthen CAISI as an evaluation and standards institution, and coordinate a broader resilience program. Proposed controls include severe-risk evaluations, transparency reports, independent audits, safety-incident reporting, model-weight security, whistleblower protection, and periodic technical assessments.