Microsoft's new VS Code skill cuts AI agent data leaks from 30% to 5.9%
Microsoft has released a new open-source VS Code skill called `run-assert-eval` that automates the full security lifecycle for AI agents — from identifying vulnerabilities to verifying that fixes actually hold. Released on September 24, 2026 under the MIT license, the tool is a direct response to a growing enterprise problem: written AI safety policies rarely enforce themselves, and standard code review misses agent-specific failures like cross-account data leakage. The result is available now on GitHub with no regional restrictions.
The four-stage pipeline
The tool chains four steps into one repeatable workflow. First, a component called Clarity maps out potential failure scenarios. Next, ASSERT converts those risks into measurable eval tests. The third stage — Agent Control Specification (ACS) — generates a runtime security policy that blocks dangerous actions as they happen. Finally, the agent reruns the same frozen test set to prove the policy works without breaking anything else.
ASSERT and ACS had already shipped as separate tools in June 2026, per Microsoft Command Line. `run-assert-eval` is what ties them together into a single VS Code command. Critically, the test set and judge remain identical across baseline and governed runs — only the policy changes, so you can see exactly what the fix did.
The billing agent test
Microsoft's demo is concrete and easy to follow. A billing support agent was supposed to work strictly within a single customer account. Without any governance applied, it exposed other customers' data in 30% of conversations — 12 out of 40 sessions. After `run-assert-eval` generated and applied a policy blocking calls with mismatched account identifiers, that rate dropped to 5.9%. Crucially, the agent did not start over-refusing legitimate requests. As AlphaSignal notes, an agent that blocks everything has zero vulnerabilities and zero utility — the tool is designed to avoid that trade-off.
This maps directly onto OWASP LLM05:2025 (Improper Output Handling), one of the most commonly cited failure modes in production AI deployments.
Context and competition
`run-assert-eval` ships with 7 agent domain examples and 14 risk suites grounded in AILuminate, NIST AI RMF, and OWASP LLM Top 10. It builds on PyRIT (Python Risk Identification Tool), which Microsoft integrated into Azure in spring 2025 for model-level red teaming. The main open-source alternative remains NVIDIA's Garak framework, which offers 120+ vulnerability probes but focuses on model evaluation rather than the full agent lifecycle. `run-assert-eval` targets the gap between threat modeling and runtime enforcement — territory Garak doesn't cover.
For enterprise teams now navigating AI governance requirements, the tool turns compliance from a policy document into a repeatable, measurable test run. To use it, you'll need an updated VS Code with agent session support in Dev Containers for isolated environments.
