You can hand a repository to a capable AI coding model and ask it to find security problems, and for part of the job that works. This note is about the part where it doesn't, and about where we are the commodity.
We are not claiming our AI is better, because it isn't. The figures below come from our own scans of Vybscan, including one that goes against us.
Where does the model win?
On a small repository, reviewing your own code, a model with repo access is competitive with us and sometimes better. It reads intent, follows logic that no rule expresses, and explains itself in plain language.
If that is all you need, use the model.
Four things the model cannot do
1. Dependency risk is a data problem
A design partner checked every dependency finding in one report against their own lockfiles: 163 of 163 true positives, zero fabricated CVEs, correct KEV enrichment, conservative severity. Asked from source code alone, a model invents advisory numbers. Whether crypto-js 3.3.0 is affected by a given CVE is not in the code.
2. Scale outruns the context window
One customer monorepo scans 2,197 source files; another's scan pool holds 5,063. No context window fits that. We cap semantic review at 300 files, pick them by risk ranking, and list the choice on the report. Pasting a repository into a chat also gets a subset, chosen less carefully, and you are never told which files were skipped.
3. Findings have to persist
Two scans four hours apart differed by 13 findings with no relevant file touched. Generation is non-deterministic, even with stable signatures, sticky findings and revalidation in place. A one-shot review has no memory of what you dismissed, no diff against last week, no audit trail, and no way to tell fixed from forgotten.
4. It has to run when nobody asks
The model reviews code when someone remembers to prompt it. Vybscan runs on every pull request and on a schedule, for years, at a known per-scan cost, and blocks a merge when it should.
The evidence against us
We built an agentic verifier: a model with tools to read and search the whole repository, asked to decide whether each finding was real. We scored it against 86 human-adjudicated findings on a corpus that is 94% false positives.
It never deleted a true positive and fabricated almost nothing, so the safety machinery held. Its judgement still lost to a constant answer.
We rejected it and recorded the result in the codebase so nobody rebuilds it in three months.
It is the second measurement pointing the same way: the model is not our moat, and the semantic layer is the most replaceable part of what we sell.
What are you actually buying?
Use the model on its own
- The repository is small enough to read in one sitting
- You only care about your own code, not your dependencies
- A point-in-time answer is enough
- A person is available to ask every time it matters
You need the tool
- Your risk lives in dependencies, so the answer is data you don't have
- The codebase is bigger than a context window, so selection must be principled and disclosed
- Findings must persist: dismissed stays dismissed, fixed is provably fixed
- It must run on every change without anyone asking, and block a merge
The model is the commodity, and it improves every year without us. What we sell is the advisory data, the triage state, and unattended operation over years.
On today's evidence we do not reason better than a frontier model, and we are not asking you to pay as if we did.
If you want to see how the tool side of that trade-off compares across vendors, we have compared Vybscan with eleven SCA and SAST tools. For an example of the dependency data the model does not have, see what slopsquatting is and how to protect against it.
Frequently asked questions
Can I just use an AI coding model to find security bugs in my code?
For part of the job, yes. On a small repository, reviewing your own code, a model with repository access is competitive with a dedicated scanner and sometimes better: it reads intent, follows logic that no rule expresses, and explains itself in plain language. If that is all you need, use the model.
Why can't an AI model check my dependencies for known vulnerabilities?
Because dependency risk is a data problem, not a reasoning problem. Whether a given package version is affected by a given CVE is not in the source code. It lives in advisory databases such as OSV, CISA KEV, EPSS and CVSS. Asked from source code alone, a model invents advisory numbers. A design partner checked 163 dependency findings from one Vybscan report against their own lockfiles and found 163 true positives and zero fabricated CVEs.
How consistent are AI security findings between runs?
Not very. Two Vybscan scans four hours apart differed by 13 findings with no relevant file touched, because model generation is non-deterministic. That drift happened even with stable signatures, sticky findings and revalidation in place. A one-shot chat review has none of that machinery, so there is no memory of what you dismissed, no diff against last week, and no way to tell a fixed issue from a forgotten one.
What did Dheemai's agentic verifier experiment show?
We built an agentic verifier, a model with tools to read and search a whole repository, and asked it to decide whether each finding was real. Scored against 86 human-adjudicated findings on a corpus that is 94% false positives, it got 27 right, 26 wrong, and declined to decide on 33. Answering "not real" every time would have scored 94%; the verifier scored 31% and skipped 38% of the work. We rejected it and recorded the result.
When do I need a security scanner instead of a model?
When any of these is true: your risk lives in dependencies, so the answer is data you do not have; the codebase is bigger than a context window, so file selection must be principled and disclosed; findings must persist, so dismissed stays dismissed and fixed is provably fixed; or it must run on every change without anyone asking, and block a merge. If none of those apply, use the model on its own.
All figures measured on Vybscan production and staging scans, September 2026. Dependency verification performed independently by the customer against their own lockfiles and OSV. Verifier scored offline against a commit-pinned, human-adjudicated corpus; re-runnable for $1–2.
See what Vybscan flags on your repos
SCA, secrets, and SAST on every GitHub pull request, with a built-in dashboard.
Try Vybscan →