Implementation guide
OWASP AISVS v1.01
Adversarial Robustness · Membership-Inference and Model-Inversion Mitigation
Train on sensitive data with differential privacy
Verify that training on sensitive datasets employs differentially-private optimization.
OWASP AISVS v1.01, requirement 11.2.4
In plain English
When you train on sensitive data, use differentially private training. It limits, with a mathematical guarantee, how much any one record can influence the model, so the model cannot be made to reveal an individual's data.
What good looks like
- Training on sensitive datasets uses a differentially private optimiser.
- The privacy budget (epsilon) is chosen deliberately, with a written justification, and recorded.
- The cost in accuracy is measured and accepted.
- Budget spent across repeated training runs and tuning is accounted for.
How teams usually do it
- DP-SGD through a maintained library: Opacus for PyTorch, TensorFlow Privacy, or JAX Privacy.
- Pre-training on public data and applying differential privacy to the fine-tuning on sensitive data, which keeps more accuracy.
- Reporting epsilon and delta in the model's documentation.
Suggested evidence
- Training configuration showing the optimiser, clipping norm and noise level.
- The privacy budget and its justification.
- Accuracy with and without differential privacy.
Common mistakes
- A privacy budget so loose it gives little real protection.
- Ignoring the budget spent on hyperparameter tuning runs.
- Treating differential privacy as a replacement for removing data you never needed (see 1.1.1).
Not sure which requirements apply to your system? Take the assessment, or browse every guide in this chapter.
Requirement text from the OWASP AI Security Verification Standard, licensed under CC BY-SA 4.0. This guidance is shared under the same licence.