AI Red Teaming, Defined
AI red teaming is the practice of deliberately attacking your own AI system to find how it fails before an adversary does. Testers use adversarial prompts and inputs to make a model leak data, break its rules or produce harmful output, then record each weakness so the team can fix it.
The idea comes from security where a red team plays the attacker so a defending team can find its blind spots. NIST describes red teaming as a group authorised to emulate an adversary against an organisation’s security posture. Applied to AI, the target is behaviour, not the servers around it.
That focus is the whole point. A generative model is probabilistic so it can be steered into responses it was never meant to give. Red teaming measures the distance between what you intend an AI system to do and what a determined user can make it do.
This has moved from a lab curiosity to a board concern fast. Organisations now put AI in front of customers and staff through chatbots, copilots and agents and each one inherits the model’s failure modes. Regulators and courts have started to treat those failures as the deploying company’s responsibility.
How AI Red Teaming Works
A red team engagement follows a loop rather than a single test. The detail varies by system but the shape stays the same.
- Set objectives: Decide what a failure means for this system from leaking personal data to giving unsafe instructions or making an unauthorised transaction.
- Map the attack surface: Identify every way input reaches the model including the chat box, uploaded files, connected data sources and tool calls.
- Run the attacks: Probe the model with adversarial prompts and inputs, chaining techniques the way a real attacker would.
- Report and retest: Document each weakness with a reproduction, hand it to the owners to fix, then confirm the fix holds.
Across those stages, a handful of weakness classes come up again and again. The OWASP Top 10 for LLM Applications updated for 2025 is the reference list and prompt injection has held the top spot for two editions.
- Prompt injection: Crafted input that overrides the system’s instructions. Direct injection comes from the user and indirect injection hides in content the model reads such as a web page or document.
- Jailbreaks: Prompts that talk the model past its safety rules into banned or harmful output.
- Data leakage: Coaxing out personal data, secrets or another user’s information the model should never reveal.
- Data and model poisoning: Corrupting training or reference data so the model behaves the way an attacker wants.
- Excessive agency: In systems that can act, tricking the model into unsafe tool use such as sending an email or changing a record.
None of this shows up in a normal application scan. A classic penetration test checks the code and infrastructure around the model. Red teaming goes after the model’s judgement itself which is where these systems actually break.
Manual, Automated and Continuous Red Teaming
Red teaming is delivered in a few different ways and the terms below describe how tests are run rather than what they look for. Most mature programmes mix them.
Manual red teaming puts human experts in front of the system for a fixed engagement. People are still the best source of novel, context-specific attacks and they reason about your particular business logic. The trade-off is scope. A manual test is deep but it is a snapshot of one moment and it cannot cover every prompt.
Automated red teaming uses tooling to generate and fire large volumes of adversarial prompts from known jailbreaks to fuzzed variations. It runs far more tests than a human could and repeats them on demand. Automation buys breadth and speed, though it is weaker at the creative, situation-specific attacks a person would spot.
Continuous red teaming runs testing on an ongoing cadence instead of once a year. AI systems change constantly through model updates, new prompts and fine-tuning and fresh attack techniques appear every week. A system that passed last quarter can fail today so the testing is wired into the release pipeline and repeated on every change.
Continuous automated red teaming combines the last two. Automated adversarial tests run continuously or on a trigger such as a new model version, so regressions are caught as they appear rather than months later. It turns testing from an annual event into a standing property of the system.
You do not have to choose one. Automated and continuous testing give constant coverage and catch regressions and human red teamers give depth on the attacks that matter most. Programmes that hold up over time use both and treat the automated layer as the floor rather than the ceiling.
The Business Impact of Untested AI
The core reason to test is that AI systems fail in ways standard checks miss. Stanford’s 2026 AI Index found that when leading models were tested against jailbreak attempts with adversarial prompts, safety performance dropped across every model tested. Passing normal use is not the same as passing adversarial use.
The volume of failures is rising with adoption. The AI Incident Database recorded 362 documented AI incidents in 2025, up from 233 in 2024, according to the same index. As more of the business runs on AI each untested weakness has more chances to cause harm.
Regulators have attached real money to this. Under the EU AI Act, breaching provider or deployer obligations can draw fines up to 15 million euros or 3% of worldwide annual turnover and prohibited practices reach 35 million euros or 7%. Adversarial testing is now written into the law for the largest models.
The liability is not limited to fines. Courts have held companies responsible for what their AI tells customers so a chatbot’s mistake becomes the company’s mistake. A single confident but wrong answer can turn into a refund, a lawsuit or a headline. The cases below show how that plays out.
Real-World Cases
Three public examples show both sides of AI red teaming, the value of doing it and the cost of skipping it.
The DEF CON Generative Red Team Challenge
In August 2023 the AI Village at DEF CON ran the largest public generative-AI red teaming event to date. Over two and a half days in Las Vegas, 2,244 people probed models from developers including Anthropic, Google, Meta and OpenAI.
Participants exchanged more than 164,000 messages across 21 challenges, hunting for bias, harmful output and security flaws. The exercise was supported by the White House science office and aligned with the NIST AI Risk Management Framework.
The lesson is straightforward. Opening a system to many skilled testers at once surfaces failures no internal team would find alone and it does so before those failures reach real users.
Air Canada’s Chatbot Invented a Policy
In November 2022 a grieving customer asked Air Canada’s support chatbot about bereavement fares. The bot told him he could claim the discount after flying. That policy did not exist.
When the airline refused the refund, the customer took it to the British Columbia Civil Resolution Tribunal. In February 2024 the tribunal held Air Canada liable for the chatbot’s misinformation and rejected its argument that the bot was a separate entity responsible for itself.
The award was small, about 800 Canadian dollars in total, roughly 650 of it in damages but the precedent was not. Pre-deployment testing that checked the chatbot’s answers against the airline’s real policies would have caught the invented rule before a customer relied on it.
DPD’s Chatbot Went Rogue
In January 2024 a frustrated customer of the parcel firm DPD prompted its support chatbot to drop its rules. The bot swore at him, then wrote a poem calling DPD the worst delivery firm in the world.
The screenshots went viral within hours and DPD disabled the AI part of the chatbot the same day. DPD said an error following a system update was to blame and it disabled the AI element the same day.
This is prompt injection in its plainest form. Red teaming for injection and output filtering, run before the update shipped, would have flagged that the bot could be talked into anything.
AI Red Teaming and Compliance
Testing AI is shifting from good practice to legal duty. Four regimes matter most for organisations operating in Sweden and the EU.
The EU AI Act (Regulation (EU) 2024/1689) is the clearest driver. Article 55 requires providers of the most capable general-purpose AI models to conduct and document adversarial testing to find and reduce systemic risks. Article 15 requires high-risk AI systems to be accurate, robust and secure against adversarial inputs and data poisoning.
Most organisations are not building frontier models but many deploy AI in ways that touch these duties and the obligations for the largest models have applied since August 2025. The direction is set and adversarial testing is the expected way to meet the robustness bar.
The NIST AI Risk Management Framework and its Generative AI Profile are voluntary and US-based but they have become a common baseline. Both name AI red teaming as a core part of pre-deployment testing which is why buyers and auditors increasingly ask for it.
In Sweden, Cybersäkerhetslagen and NIS2 raise the bar for essential and important entities. Cybersäkerhetslagen (SFS 2025:1506) took effect on 15 January 2026 and requires continuous monitoring, incident handling and testing the effectiveness of security measures under Article 21 with the board accountable under Article 20. An AI system that is important to your operations falls inside that duty and incidents are reported to MCF (formerly MSB).
Financial entities carry a further layer. DORA requires ICT risk management and threat-led testing which extends to the AI systems those firms increasingly rely on. Where an AI service is critical, its resilience is in scope.
The through-line across all four is the same. If an AI system matters to your business or your customers you are expected to show that you have tested it against attack before trusting it in production.
How to Build an AI Red Teaming Programme
You do not need a frontier-lab budget to start. A workable programme rests on three foundations and a handful of concrete moves.
- People: A mix of security testers who think like attackers and staff who understand your product and its data.
- Process: A defined scope, a fixed cadence and a clear owner for every finding so issues get fixed rather than filed.
- Technology: Tooling that automates the repetitive attacks and plugs into your release pipeline for continuous coverage.
Then put it into practice with a few concrete steps.
- Inventory every AI system you run and rank them by the harm a failure would cause.
- Define what a serious failure looks like for each one before you test.
- Re-run the tests on every model or prompt change, including minor updates.
- Test the whole path into the model including files, connected data and any tools it can call.
- Give each finding a severity, an owner and a deadline, then retest to confirm the fix.
- Keep the evidence because the EU AI Act and NIS2 both expect you to show your working.
A perfect model does not exist. A defensible one is a system whose weaknesses you find and close faster than an attacker can reach them, rechecked every time it changes.
