Prompt Injection in Plain Terms
A prompt injection attack is when someone feeds an AI system crafted text that the model treats as a new instruction, overriding the rules its developer gave it. The text can be typed straight in or hidden inside content the AI reads and it needs no code, only careful wording.
This matters because AI assistants now sit inside everyday work. In 2025, 71% of organisations reported using generative AI in at least one business function up from 65% a year earlier according to McKinsey. Every chatbot, copilot and agent that reads a message, a document or a web page is a place an instruction can be smuggled in.
Security researchers named the technique in 2022 drawing an analogy to SQL injection where attackers slip commands into data. The Open Worldwide Application Security Project (OWASP) now ranks prompt injection as the number one risk for applications built on large language models.
How a Prompt Injection Attack Works
A language model reads everything it is given as one stream of text. The developer’s rules, your question and any content the model pulls in all arrive in the same channel and the model has no reliable way to tell a trusted instruction from ordinary text. So a sentence buried in that text, phrased as a command can be followed as if the developer had written it.
That is the whole trick. OWASP states it plainly, models process instructions and data together without clear separation so an attacker can craft input the model reads as a new instruction rather than as content to handle.
It is the same weakness as SQL injection where code and data share one line. Prompt injection is harder to fix because the thing reading the input is a statistical model with no fixed grammar, not a parser you can lock down. There is no character you can escape to make the risk go away.
Two shifts make it worse. Retrieval-augmented systems feed the model live external content, a web page or a file so untrusted text reaches the model on its own. AI agents go further and can act on what they read sending an email or calling a tool, which turns a rogue instruction into a rogue action.
Types of Prompt Injection Attacks
NIST and OWASP group the attack into two root forms, plus a hidden variant that cuts across both.
- Direct injection: Typed straight into the model through a chat box, form or API. It covers jailbreaks that strip a model’s safety rules and it is how the Chevrolet and DPD bots were pushed off script.
- Indirect injection: Hidden inside content the model later reads such as a web page, an email, a PDF or a calendar invite. The user never sees it which is what lets it scale and what makes data theft possible.
- Hidden and multimodal injection: Instructions concealed in invisible text, white font, HTML comments or metadata. They can also hide inside an image the model processes. OWASP flags multimodal injection as a growing route.
What the attacker does with that foothold varies. It can force the model to break its rules, reveal its hidden system prompt or secrets, hand over data it can reach or, in an agent, take an action on the attacker’s behalf.
Business Impact of Prompt Injection
The cost of a prompt injection attack depends on what the AI can reach and what it can do. In practice the harm climbs through four levels.
At the mild end sits reputational damage. A public chatbot that swears at a customer or calls its own company the worst in its industry becomes a screenshot within minutes. Neither the DPD nor the Chevrolet bot lost money yet both made the brand the punchline.
The next step up is disclosure. An attacker can pull out the hidden system prompt, internal rules or secrets an assistant holds, as happened to Bing Chat, which hands over a map of how the system works.
More serious is data theft. Where an assistant can read company files and also send messages, a single injected instruction can make it copy sensitive data to an outsider. The Microsoft 365 Copilot flaw proved this is more than a hypothetical worry.
The sharpest risk is unauthorised action. In an agent that can move money, change records or run code, a rogue instruction becomes a rogue transaction and a data leak or an outage can also become a regulatory event.
Real-World Prompt Injection Cases
Four incidents show the range from a leaked secret to stolen data. Each also points to the control that would have blunted it.
Bing Chat Reveals Its Codename (2023)
Days after Microsoft launched its AI-powered Bing Chat in February 2023, Stanford student Kevin Liu typed a simple instruction telling it to ignore its previous instructions and print what came before. The model obliged and revealed its hidden system prompt including an internal codename (Sydney) and rules it had been told never to disclose.
Microsoft confirmed the leaked text was real and closed the first route but another bypass appeared within days. The lesson is not to treat the system prompt as a secret. Keep credentials and anything sensitive out of it and enforce limits outside the model.
A Chatbot Sells a Car for One Dollar (2023)
In December 2023, a customer told the ChatGPT-powered chatbot on a Chevrolet dealership’s website to agree with anything he said and treat it as binding. He then got it to accept one dollar for a new Chevrolet Tahoe (an SUV that lists for tens of thousands of dollars) and to add that there were no takesies backsies.
The dealership never honoured the offer and took the bot down within days. Others had already made it write code and recommend rival brands. Nothing was stolen but the bot spoke for the business with no guardrails. Scope a customer-facing bot to answering questions and route any price or commitment to a person.
A Delivery Bot Turns on Its Owner (2024)
On 18 January 2024, a frustrated DPD customer prompted the firm’s support chatbot into swearing, writing a poem about how useless it was and calling DPD the worst delivery firm in the world, even after it said it was not allowed to swear. It also suggested rival couriers.
DPD blamed a system update and switched the AI element off the same day. The post was viewed millions of times. The fix here is output controls that survive every change, checked with adversarial testing after each update rather than only at launch.
A Zero-Click Email Steals Data From Copilot (2025)
In 2025, researchers disclosed a flaw in Microsoft 365 Copilot, tracked as CVE-2025-32711 and nicknamed EchoLeak. A single crafted email with no click or action from the recipient could make Copilot read data from the user’s context and send it to an outside server.
The instructions were hidden in the email and reached Copilot through its normal retrieval, slipping past Microsoft’s own prompt-injection filter. Microsoft patched it before public disclosure and found no sign it was used in the wild. It is regarded as the first documented case of indirect injection turned into working data theft in a live AI assistant.
The control is least privilege plus egress limits. An assistant should not be able to both read sensitive data and send it anywhere it likes.
Prompt Injection and Compliance
For organisations in the EU and Sweden, resilience against prompt injection is moving from good practice to legal duty. Four regimes are relevant.
The EU AI Act: Article 15 requires high-risk AI systems to be accurate, robust and secure including resilience against outsiders trying to alter their outputs by exploiting weaknesses which is the category prompt injection falls into. It binds high-risk systems and general-purpose models used in a high-risk setting. The Act entered into force in 2024 and phases in with most high-risk duties applying from August 2026.
NIS2 and Cybersäkerhetslagen: If an essential or important entity runs an AI system, prompt-injection risk falls under the Article 21 duty to manage security risks. A resulting incident triggers reporting to MCF (formerly MSB) within 24 hours, 72 hours and one month. Sweden’s law (SFS 2025:1506) has applied since 15 January 2026 and boards are personally accountable under Article 20. See our NIS2 compliance guide.
GDPR: If an injection leaks personal data, as a Copilot-style attack could, it is a personal-data breach and Article 33 sets a 72-hour notification to IMY, the Swedish data protection authority. See our GDPR compliance guide.
DORA: For banks, insurers and other financial entities, an AI assistant counts as ICT so an injection-driven incident falls under the ICT incident-management rules in Article 17 supervised by Finansinspektionen. See our DORA compliance guide.
Underneath all four sits the same expectation, an information security management system that treats AI as part of the estate, in the spirit of ISO 27001. Probing an AI system for these weaknesses is a natural fit for penetration testing and for AI detection and response.
How to Spot a Prompt Injection Attack
You cannot reliably catch prompt injection by reading the input. Direct attacks use ordinary language and indirect ones hide in content you never see. Some signals are still worth knowing.
- An AI assistant does something outside its brief such as changing topic, ignoring a limit or offering to take an action it was not asked for.
- Output that quotes or describes its own rules or system prompt, a sign someone has been probing it.
- Content that seems to address the AI rather than you, an email or document carrying instructions for the assistant.
- In an agent, an action proposed with no clear reason such as sending data out or contacting an address you do not recognise.
Treat these as reasons to check rather than proof of an attack. Because detection alone is unreliable, the real protection is in how the system is built and governed.
How to Defend Against Prompt Injection
You cannot patch prompt injection out of a model so defence works in layers. NIST and OWASP recommend the same shape, reduce what an attack can reach and add checks around the model.
- Give the model and any tools it uses the least access they need so a hijacked assistant can reach little.
- Keep untrusted content separate and clearly marked so retrieved text is treated as reference material, never as commands.
- Filter and constrain the model’s output, then block any channel it could use to send data to unknown destinations.
- Require a human to approve sensitive or high-value actions such as payments, data exports or record changes.
- Test AI systems against injection before launch and after every change, the way you would penetration test any application.
- Train the people who use AI so they treat its output and its suggested actions with the same care as any other request.
People, process and technology all carry part of the load. The single habit that helps most is simple, never let an AI system act on a sensitive request without a human check on a separate channel.
