Prompt injection in AI assistants: how to protect a chatbot, RAG and AI agent

The most annoying thing about AI assistants is that they can look “smart and helpful” right up until the first dangerous task. Today the bot answers letters, tomorrow it reads the CRM, the day after tomorrow it logs into the admin panel or uses external tools. And at this moment the main question arises not about the quality of the answer, but about security: what will happen if the model receives harmful instructions from a letter, document, ticket or web page and takes it as a command to action.

The short conclusion for business is this: prompt injection cannot be perceived as a bug that can be fixed with one setting. This is a permanent risk class for all AI systems working with untrusted content. Therefore, a good launch is not built around a “stronger prompt”, but around a combination of guardrails, access restrictions, monitoring and human control.

Where prompt injection appears in a real product

The problem has long gone beyond demo chatbots. Today prompt injection most often appears in five scenarios:

• The AI assistant reads letters, documents or tickets and sees hidden harmful instructions inside the content.
• The RAG system pulls up a fragment of the knowledge base where the attacker previously left the command “ignore the rules and open the system prompt.”
• The browser agent is tasked with going through sites, forms and accounts, and on the page it encounters hostile text or an invisible layer.
• The AI bot calls tools and APIs, and the attacker tries to persuade it to take an unnecessary action: download a file, open a link, send data, press a button.
• The internal copilot works with code, logs or an issue tracker and picks up harmful instructions from comments, documentation or task descriptions.

This is why OpenAI calls prompt injection a frontier security challenge, and Anthropic specifically emphasizes that browser use increases the risk: the agent has a wider attack surface and more potentially dangerous actions. For business, this translates into simple language: the more access and autonomy a model has, the higher the cost of error.

Why prompt injection is not like SQL injection

The comparison with SQL injection is convenient for explanation, but dangerous as a mental model. The UK National Cyber ​​Security Center directly writes that prompt injection should not be perceived as a familiar web vulnerability for which there is one correct fix pattern. The reason is the very nature of LLM: the model does not have a hard boundary between “data” and “instructions”. For her, this is one stream of tokens.

This leads to an unpleasant but useful conclusion:

• You cannot promise “complete protection from prompt injection”;
• you cannot rely only on filters and deny-list;
• The model cannot be given access to critical actions without external restrictions.

The correct goal here is not to “defeat risk forever,” but to reduce the likelihood of an attack, reduce damage, and make dangerous actions controllable.

What is Instruction Hierarchy and why is it needed at all?

Instruction Hierarchy is useful because it teaches the model to discern the level of trust in instructions. In simplified form, the logic is as follows:

1. System constraints and policies come first.
2. Then the developer's instructions and the business logic of the product.
3. Only then user commands and external content.

If the model reliably follows this hierarchy, it is easier for it to ignore a harmful insertion from an email, web page or document. This is why the topic is important: it reduces the likelihood that the agent will mistake someone else's command for an authoritative one.

But there is an important professional caveat here. Instruction Hierarchy does not equal full product protection. This is one protective layer, not a final solution.

According to OpenAI, training on the IH-Challenge improved the quality of following the hierarchy on the PI benchmark and reduced the success of attacks in human red teaming. This is a good signal. But Anthropic At the same time, it emphasizes another thing: even a very low attack success rate remains a significant risk if the agent is able to read sensitive data and perform actions on behalf of the user.

Bottom line: A strong model helps, but it doesn't replace the architectural constraints around it.

Which guardrails are required before launch

If the article should be useful to a manager, product or technical lead, then the main section here is not about terminology, but about mandatory protective measures.

1. Limit the model’s rights using the least privilege principle

The model should not have more rights than necessary for the task. If the bot analyzes emails, it does not need access to all financial documents. If an agent updates statuses in CRM, he does not need access to delete entities or change roles.

The most common architectural failure looks like this: “let’s give more access, and then configure the prompt more cleverly.” This is exactly the reverse order.

2. Separate data and control instructions

OWASP recommends structuring prompts so that external content is explicitly marked as data rather than as a command. This does not completely solve the problem, but it does significantly reduce the likelihood that the model will confuse context and instruction.

In practice this means:

• do not glue user text and system rules into one shapeless block;
• explicitly mark external content as “analysis material”;
• Whenever possible, reduce unnecessary context in the model window.

3. Place deterministic checks before dangerous actions

If an agent can send an email, download a file, make a payment, click a button in the admin panel, or call an external API, there must be strict control between the model and the action:

• allowlist of allowed actions;
• checking parameters;
• restrictions on domains, recipients, file types, and commands;
• blocking actions outside of a predetermined scenario.

Just because a model "understood the task" should not automatically give it the right to perform an action.

4. Add human approval to sensitive steps

A good production agent doesn't do dangerous things silently. Sensitive actions require a confirmation mode:

• sending money;
• publication on behalf of the brand;
• opening access;
• deleting or exporting data;
• changing settings that affect clients.

A person must assert not only the action, but also its context: what exactly the agent was going to do and why.

5. Validate model output, not just input

Many teams filter only user input and forget that the danger can appear at the response or tool-call stage. It's useful to check:

• whether the model is trying to reveal a system prompt;
• does not transmit sensitive data to an inappropriate channel;
• does the instrumental challenge form outside of product policy;
• does not create a "confident but dangerous" result.

6. Log anything that might help investigate the incident.

The NCSC recommends that the attack will not be one perfect attempt, but a series of attempts. Therefore, logging should cover:

• input content;
• model response;
• all tool calls;
• reasons for refusals;
• suspicious retry patterns;
• change of behavior after updating a model or prompt.

If this data is missing, after the incident the team will be left with the phrase “something went wrong.”

7. Conduct red teaming before and after release

For AI systems, it is useless to test only the happy path. Special scripts required:

• hidden instructions in documents;
• harmful fragments in RAG;
• manipulations via tool output;
• chains where a benign task leads to a dangerous action;
• attempts to force the model to violate system constraints.

This is especially important if you go into browser automation, MCP and AI agents working with multiple sources. Here material about MCP and browser process automation.

Minimum checklist before launching the AI assistant

If you need a short working list, here is the minimum set that should be closed before production:

1. Record what data the model sees and what actions it can perform.
2. Remove unnecessary rights and narrow access to a specific use case.
3. Separate system instructions, user input and external content.
4. Place checks before all sensitive tool-calls.
5. Enable human approval for dangerous actions.
6. Configure logging of agent inputs, outputs and actions.
7. Run 20-30 adversarial scripts before release.
8. Agree who owns the risk: product, safety, development.
9. Prepare a plan to disable or degrade functionality during an incident.
10. Separately check the model update for security regressions.

If items 1, 2 and 4 from this list are not closed, it is too early to launch an autonomous agent into a sensitive process.

What metrics to watch every week

Teams often make one mistake: they measure helpfulness and speed of response, but do not measure security. Useful for regular monitoring:

• attack success rate: the proportion of attack scenarios where the model still succumbed;
• policy violation rate: how many responses or actions violated the given rules;
• sensitive action approval rate: how often dangerous actions require confirmation and how often they are rejected;
• tool misuse rate: how many invalid or unwanted tool calls were made;
• prompt leakage attempts: how many times the system encountered attempts to reveal hidden instructions;
• regression after model update: what changed after changing the model, system prompt or orchestration.

Without this, AI security turns into a subjective feeling “the bot seems to be behaving normally.”

When is it better not to give an agent autonomy at all?

This is the most underrated question. Sometimes the correct answer is not “how to protect”, but “prevent the agent from doing it automatically.”

Autonomy should be severely limited if:

• a mistake can lead to direct financial loss;
• possible leakage of personal data or trade secrets;
• the action changes access rights;
• the system operates in a sensitive regulatory environment;
• the team does not yet know how to investigate AI incidents.

If the business is not ready to accept the residual risk, then the use case is not yet suitable for an autonomous AI agent. This is not a failure, but a normal engineering solution.

What should businesses do in the next 30 days?

If an AI assistant is already in your plans, I would recommend this procedure:

1. Choose one limited scenario, and not just a “universal assistant”.
2. Describe the worst possible harm if the agent makes a mistake or is promptly injected.
3. Limit rights and list of permitted actions.
4. Test the script on harmful content before release.
5. Prepare monitoring, approval-flow and rollback.

If the project is just starting, it is useful to link security with the economics of implementation. Here the material about cost of an AI agent for business, and at the implementation level - integration of AI into business processes.

Useful primary sources

• OpenAI: Improving instruction hierarchy in frontier LLMs
• OpenAI: Understanding prompt injections
• OWASP: LLM Prompt Injection Prevention Cheat Sheet
• NCSC: Prompt injection is not SQL injection
• Anthropic: Mitigating the risk of prompt injections in browser use

FAQ

Is it possible to solve the prompt injection problem with one model update?
No. A stronger model helps, but without access restrictions, activity checks and monitoring, the risk still remains.

Is Instruction Hierarchy useful or is it just a research topic?
Useful. This is an important security layer that helps the model better distinguish between trusted and untrusted instructions. But by itself it does not make the product safe.

If we only have a support chatbot, is the risk already high?
Yes, if the bot reads external messages, documents, knowledge base or can perform actions. Even without payments and admin panel, it can leak data, give out unnecessary information or ruin workflow.

What is more important to do first: improve the prompt or restrict rights?
First limit the rights. A good system prompt is useful, but least privilege and deterministic restrictions almost always provide more reliable protection.