Background
The U.S. Cybersecurity and Infrastructure Security Agency (CISA) — the federal body designated as national coordinator for critical infrastructure security and resilience — has released a series of increasingly prescriptive policy documents requiring operators of essential services to embed adversarial security testing into the lifecycle of any artificial intelligence (AI) system they deploy.
<cite index="3-1">As the National Coordinator for critical infrastructure security and resilience, CISA is responsible for facilitating a Secure by Design approach to AI-based software across the digital ecosystem and helping protect critical infrastructure from malicious uses of AI.</cite>
Key Guidance Documents
<cite index="6-1">On December 3, 2025, CISA, together with cybersecurity agencies from Australia, Canada, Germany, the Netherlands, New Zealand, the United Kingdom, and the U.S. National Security Agency and FBI, released a landmark document: *Principles for the Secure Integration of Artificial Intelligence in Operational Technology*.</cite> <cite index="13-5">It focuses on machine learning, Large Language Model (LLM)-based AI, and AI agents because of the complex security considerations and challenges they pose.</cite>
<cite index="11-8,11-9">The guidance document describes four key principles for integrating AI into operational technology, detailing the issues that infrastructure operators should consider as they adopt AI. The advice covers general risk awareness, need and risk assessment, AI model governance, and operational fail-safes.</cite>
In May 2026, CISA extended its reach further. <cite index="4-3">CISA, alongside the Australian Cyber Security Centre and other international partners, published new guidance on the secure adoption of agentic AI, outlining cybersecurity risks tied to deploying these systems.</cite> <cite index="4-6">While the benefits are clear, the agencies warn that these systems introduce new risks, including expanded attack surfaces, privilege escalation, behavioral misalignment, and limited auditability.</cite>
Red Teaming and Adversarial Testing Requirements
<cite index="3-2,3-3">To effectively mitigate against critical failures, physical attacks, and cyberattacks, AI software developers must prioritize conducting rigorous safety and security testing to understand how an AI system can fail or be exploited. AI red teaming is a foundational component of the safety and security evaluations process.</cite>
<cite index="17-1">At the development stage, the agencies require comprehensive threat modeling before integration — specifically, adversarial testing, red-teaming against prompt injection scenarios, and hardening of agent behavior before any production deployment.</cite> <cite index="17-3">Agents must be configured to fail-safe by default, escalating to human reviewers when encountering uncertainty rather than proceeding on a best-effort basis.</cite>
<cite index="3-9">CISA has already begun to receive requests from partners to conduct penetration and penetration testing on Large Language Models (LLMs) and expects demand for these services to grow as partners increasingly adopt AI tools.</cite>
<cite index="6-3">The guidance's emphasis on AI Software Bills of Materials (AI-SBOMs), vendor transparency, human-in-the-loop oversight, continuous testing, and AI red-teaming indicates that regulators may soon expect organizations to implement comprehensive governance and risk-management structures.</cite>
Broader Regulatory Context
<cite index="8-6,8-7">CISA updated a list of goals that it hopes utilities, water treatment facilities, hospitals, and other critical infrastructure operators will use to protect their systems from hackers. Version 2.0 of CISA's Cross-Sector Cybersecurity Performance Goals (CPGs), released in December 2025, "incorporates three years of operational insights, and addresses emerging threats through data-driven, actionable guidance."</cite>
<cite index="18-1,18-2">The White House Executive Order on AI (June 2, 2026) explicitly tasked CISA with issuing binding operational directives to defend civilian federal systems; CISA Binding Operational Directive (BOD) 26-04 followed eight days later, compressing the most critical vulnerability remediation window to three calendar days.</cite>
<cite index="14-7,14-8">AI has the potential to introduce improvements and rapidly change many areas. However, deploying AI may make critical infrastructure systems that support the nation's essential functions, such as supplying water, generating electricity, and producing food, more vulnerable.</cite>
Outlook
<cite index="6-4">Supply chain accountability, explicit data usage policies, and documentation of model dependencies are likely to become regulatory requirements, ensuring both operators and vendors are responsible for AI security and functional safety.</cite> <cite index="12-12">However, a lack of cybersecurity know-how within the operational technology (OT) sector could derail these efforts.</cite> Compliance timelines and enforcement mechanisms for non-federal operators remain under discussion, with CISA indicating further rulemaking is anticipated as the broader AI governance landscape consolidates.