AI-Assisted Secure Coding (Appendix C) · AI Tool Qualification & Threat Modeling
Attack-test tools before adopting them
Verify that each tool goes through adversarial robustness testing before onboarding. The testing is repeated after any material change to the model or to the system prompts. Coverage includes automated prompt-injection probes, jailbreak suites, and indirect-injection corpora delivered through realistic PR and issue surfaces.
OWASP AISVS v1.01, requirement AC.2.3
In plain English
Attack-test each tool before you adopt it, and again after any material change to its model or system prompts. Deliver the attacks the way a real attacker would, through pull requests, issues and comments. Typing them into the chat box alone tests the wrong path.
What good looks like
- Adversarial testing is completed before onboarding.
- Coverage includes automated prompt-injection probes, jailbreak suites and indirect-injection samples delivered through realistic PR and issue content.
- Testing is repeated after a material model or prompt change.
- Results are recorded per tool and version, and failures are fixed or accepted in writing.
How teams usually do it
- A test repository holding poisoned PRs, issues, comments, commit messages and files, which you point each new tool at.
- Open-source probing tools such as garak, PyRIT or promptfoo for the automated suites.
- Canary secrets in the test environment to show whether an injection could exfiltrate anything.
Suggested evidence
- Test results per tool and version.
- The test corpus.
- The record of a re-test after a model change.
Common mistakes
- Testing only with direct prompts typed by a tester.
- Testing once at adoption while the vendor updates the model every few weeks.
- Testing in a sandbox that has none of the permissions the tool will hold in production.
Not sure which requirements apply to your system? Take the assessment, or browse every guide in this chapter.
Requirement text from the OWASP AI Security Verification Standard, licensed under CC BY-SA 4.0. This guidance is shared under the same licence.