Input Validation · Content & Policy Screening
Score prompts for harmful content
Verify that every prompt is scored by a content classifier for violence, self-harm, hate, and sexual content against configurable thresholds. Prompts that exceed those thresholds are rejected or sanitized before reaching the model context.
OWASP AISVS v1.01, requirement 2.2.1
In plain English
Score every prompt for harmful content, covering violence, self-harm, hate and sexual content, and stop prompts that exceed your thresholds before the model sees them. Decide the thresholds for your audience. A children's tutoring app and a trust-and-safety tool need very different settings.
What good looks like
- A content classifier runs on every prompt, on every turn.
- Each category has a threshold that was chosen deliberately and written down.
- Prompts over a threshold are rejected or cleaned before they reach the model.
- Blocked prompts get a clear, neutral response and are logged.
How teams usually do it
- The moderation service from your model provider or cloud, for example the OpenAI Moderation API, Azure AI Content Safety or Amazon Bedrock Guardrails content filters.
- An open moderation model in your own stack, for example Llama Guard or ShieldGemma.
- For self-harm, routing to a supportive response with signposting to help. A flat refusal is the wrong answer there.
Suggested evidence
- Classifier configuration and thresholds, with the reasoning behind them.
- Test results using sample prompts in each category.
- Block and false-positive rates from production.
Common mistakes
- Leaving thresholds at defaults that do not suit your audience.
- Screening the first message of a conversation and skipping later turns.
- Failing open. If the moderation call times out and the prompt goes through anyway, an attacker only has to slow the classifier down.
Not sure which requirements apply to your system? Take the assessment, or browse every guide in this chapter.
Requirement text from the OWASP AI Security Verification Standard, licensed under CC BY-SA 4.0. This guidance is shared under the same licence.