Implementation guide
OWASP AISVS v1.01
Training Data Integrity & Traceability · Training Data Quality and Security Assurance
Check security-relevant models for bias
Verify that models used in security-relevant decisions are evaluated for bias patterns.
OWASP AISVS v1.01, requirement 1.3.3
In plain English
If a model helps make security decisions, such as flagging fraud, blocking content or scoring risk, test whether it performs worse for some groups or kinds of input. A weak spot is unfair to the people affected and gives an attacker a place to hide.
What good looks like
- Security-relevant models are evaluated separately on each relevant group or input category, including language, region and device type where they apply.
- False-positive and false-negative rates are compared across those slices.
- Large gaps are investigated, then fixed or accepted in writing with a reason.
- The evaluation is repeated when the model or its training data changes.
How teams usually do it
- Slice-based evaluation: run the test set through the model and report the error rates per slice.
- Open-source fairness toolkits such as Fairlearn or Aequitas to compute the standard metrics.
- Thresholds for an acceptable gap agreed in advance with the people who act on the model's decisions.
Suggested evidence
- Evaluation reports broken down by group or category.
- Records of how the largest gaps were handled.
- Evidence the evaluation was rerun after the last model change.
Common mistakes
- Reporting one overall accuracy figure that hides poor performance on a subgroup.
- Testing only the slices that are easy to measure.
- Testing once at launch and never again.
Not sure which requirements apply to your system? Take the assessment, or browse every guide in this chapter.
Requirement text from the OWASP AI Security Verification Standard, licensed under CC BY-SA 4.0. This guidance is shared under the same licence.