Adversarial Robustness · Model Runtime Anomaly Detection
Protect the safety feedback loop
Verify that the safety violation feedback pipeline includes poisoning detection and human review gates to prevent adversarial manipulation of the improvement mechanism.
OWASP AISVS v1.01, requirement 11.4.3
In plain English
Many teams improve their models from reports of safety problems: user flags, moderator decisions, red-team findings. That pipeline is a way into your training data. Check feedback for poisoning and have a person approve it before it changes the model. This is a Level 3 control.
What good looks like
- Feedback data is checked for poisoning patterns before use.
- A person reviews and approves feedback before it enters training or changes a filter.
- The source of each piece of feedback is recorded.
- No single source or small group can dominate an update.
How teams usually do it
- The poisoning checks from 1.3.1 applied to feedback data, plus a review queue.
- Caps on each account's contribution, and reputation weighting for sources.
- Detection of coordinated reporting, such as many new accounts flagging the same content.
Suggested evidence
- Pipeline configuration.
- Review records.
- Contribution limits and an example of rejected feedback.
Common mistakes
- Training directly on thumbs-up and thumbs-down data.
- Auto-updating a blocklist or classifier from user reports, which lets attackers get legitimate content blocked or harmful content waved through.
Not sure which requirements apply to your system? Take the assessment, or browse every guide in this chapter.
Requirement text from the OWASP AI Security Verification Standard, licensed under CC BY-SA 4.0. This guidance is shared under the same licence.