Adding an AI assistant, copilot or agent introduces trust boundaries that need explicit testing. Prompt injection is one important risk: untrusted content can influence model behaviour and, when permissions allow it, connected actions.
What it is
A model may follow instructions embedded in content that should be treated only as data. A support ticket, summarised web page or retrieved document can carry such instructions. Impact depends on the surrounding application, available data and tool permissions.
Why traditional testing misses it
Conventional application testing remains necessary, but a SQL injection check does not establish that an AI workflow respects its trust boundaries. Test what happens when hostile content reaches retrieval, output consumers or connected tools. System-prompt disclosure is not automatically equivalent to private-data compromise.
What to do about it
- Treat all model input as untrusted, including content the model retrieves itself.
- Constrain model-connected tools, actions and data with least privilege and application-enforced authorisation.
- Separate and identify untrusted data where possible, without relying on separation as a complete defence.
- Test authorised abuse scenarios informed by OWASP GenAI guidance and the application's actual permissions.
Treat the model's output as untrusted. Validate tool parameters and permissions outside the model, require approval for high-risk actions, and record residual limitations. See the OWASP prompt-injection guidance for the baseline.