Implementation guide
OWASP AISVS v1.01
Adversarial Robustness · Model Alignment, Safety, and Robustness Testing and Training
Track harmful-content rate for regressions
Verify that an automated evaluator measures harmful-content rate and flags regressions beyond a defined threshold.
OWASP AISVS v1.01, requirement 11.1.5
In plain English
Measure the rate of harmful output automatically, on a fixed test set, and flag any version where the rate rises past a set amount. A number tracked over time catches the gradual slide that pass-or-fail tests miss. This is a Level 3 control.
What good looks like
- An automated evaluator scores the harmful-content rate on a fixed, versioned test set.
- A regression threshold is defined, such as a rise of more than a set number of percentage points.
- A regression blocks release or triggers a review.
- The evaluator is itself checked against human judgement from time to time.
How teams usually do it
- A classifier-based or model-graded evaluator in the release pipeline, for example Llama Guard or a rubric-driven judge model.
- Enough test prompts and repeated runs for differences to be statistically meaningful.
- Rates charted per version and per category.
Suggested evidence
- Rates per version.
- The threshold and the record of a flagged regression.
- Results of the evaluator's own accuracy check.
Common mistakes
- Measuring the rate and never defining what counts as a regression.
- A test set so small that a single response moves the rate.
- Changing the test set and comparing with old numbers.
Not sure which requirements apply to your system? Take the assessment, or browse every guide in this chapter.
Requirement text from the OWASP AI Security Verification Standard, licensed under CC BY-SA 4.0. This guidance is shared under the same licence.