Training Data Integrity & Traceability · Training Data Quality and Security Assurance
Detect poisoned training data
Verify that training and fine-tuning pipelines implement poisoning detection techniques to identify potential data poisoning or unintentional corruption in training data.
OWASP AISVS v1.01, requirement 1.3.1
In plain English
Check training and fine-tuning data for records planted to change how the model behaves, and for accidental corruption. Poisoning needs surprisingly few records, so data from users, the web or third parties should be checked before the model sees it.
What good looks like
- Every training and fine-tuning run includes automated checks on data that is new since the last run.
- Outliers, near-duplicates, sudden shifts in label balance and repeated unusual phrases are flagged.
- Flagged records are held back until a person has reviewed them.
- A failed check stops the run.
How teams usually do it
- Schema and distribution checks between data versions, for example with Great Expectations or TensorFlow Data Validation.
- Label-error detection (for example cleanlab) to find records whose labels disagree with what similar records say.
- Research-grade detectors such as activation clustering or spectral signatures, both available in the Adversarial Robustness Toolbox, where the risk justifies the effort.
Suggested evidence
- Pipeline configuration showing the checks and the thresholds that block a run.
- Reports from recent runs: what was flagged, who looked at it and what was decided.
Common mistakes
- Checking the original dataset and ignoring the feedback and fine-tuning data added later, which is the easiest place for an outsider to inject records.
- Checks that only log warnings.
- Feeding user thumbs-up and thumbs-down data straight into training with no review.
Not sure which requirements apply to your system? Take the assessment, or browse every guide in this chapter.
Requirement text from the OWASP AI Security Verification Standard, licensed under CC BY-SA 4.0. This guidance is shared under the same licence.