Implementation guide
OWASP AISVS v1.01
Adversarial Robustness · Model Alignment, Safety, and Robustness Testing and Training
Run an alignment test suite on every update
Verify that a version-controlled alignment test suite is run on every model update or release.
OWASP AISVS v1.01, requirement 11.1.2
In plain English
Keep a test suite for safe behaviour in version control, and run it on every model update or release. Model behaviour shifts with each new version, fine-tune and prompt change. Without a repeatable suite you find out from your users.
What good looks like
- The suite is in version control and changes to it are reviewed.
- It runs automatically on every model, prompt or configuration change.
- Results are stored per version and compared with the previous one.
- A failed run blocks the release, or needs a named person to accept the risk.
How teams usually do it
- An evaluation job in the release pipeline, using a framework such as Inspect, promptfoo or garak.
- A new test added each time an incident or red-team finding exposes a gap.
- Each test run several times, since model output varies from run to run.
Suggested evidence
- The suite's repository history.
- Results for recent releases.
- A release that was blocked or delayed by the suite.
Common mistakes
- A test suite kept in someone's notebook.
- Running the suite and releasing whatever the result.
- A suite that never changes, which the model eventually passes without being any safer.
Not sure which requirements apply to your system? Take the assessment, or browse every guide in this chapter.
Requirement text from the OWASP AI Security Verification Standard, licensed under CC BY-SA 4.0. This guidance is shared under the same licence.