Why not just ask a coding agent to review your repo?

← Back to Blog

On five codebases with known security bugs, Vybscan found almost four times as many as a Claude Code security agent, for half the AI spend. It also keeps track of them from one scan to the next.

Teams often ask us why they should pay for a scanner when Claude Code can already read their repository. We tested it. We ran a security-review agent and Vybscan on the same five codebases, scored both against the same answer keys, and checked every extra finding by hand. We also worked out what each one costs over a month.

What we compared

We used five codebases with 188 known security defects between them.

• The codebases were OWASP juice-shop, cal.diy (an open-source scheduling app), dub (a link shortener), gogs (a Git server written in Go) and kanboard (a PHP project tool). Each has an answer key, built from published advisories and their fixes or from a full manual review.
• For the agent we used ponytail-sec, an open-source security plugin for Claude Code, on Claude Opus 4.8. We ran it three times per codebase, because it gives a different answer on each run.
• Vybscan ran a standard full scan, the same one a customer gets.
• A finding counted when it matched a defect in the key. We checked every other finding by hand, for both tools.

Agent numbers are the average of its three runs. Our earlier post, ponytail-sec vs Vybscan, has the full detail.

Results

Vybscan found 149 of the 188 known defects. The agent found 40 per run and spent twice as much.

All four rows are measured. The agent’s findings were more often correct, but there were far fewer of them. Its three runs also disagreed with each other. Together they found 55 defects, so no single run gave the whole picture.

The gap was widest on large codebases with harder bugs. On gogs the agent found about 2 of 45 published advisories per run. Vybscan found 28.

Vybscan found more known defects on every codebase

Share of each codebase’s answer-key defects found. Latest Vybscan full scans (October–November 2026) and three ponytail-sec runs per codebase (4 November 2026).

Cost over a month

The agent costs the same every time it runs. Vybscan’s cost falls after the first scan.

The agent in our test read only part of each codebase, in 20 to 35 steps over 4 to 12 minutes. One that reads every source file would cost more. At Claude Opus 5.5 list prices, a full read of all five codebases comes to about $105, and at least $47. We estimated that from the size of each codebase; we did not run it.

For one cal.diy-sized codebase (about 4,800 source files) with a weekly full review and 40 pull requests, a month looks like this:

Vybscan stays cheap because it concentrates on what changed. A pull-request scan costs cents, and a weekly re-scan spends little on code it has already reviewed.

We also asked whether Vybscan would find more if it read more. Of the 31 defects it missed, 28 were in code it had already reviewed. Reading more would raise the cost several times over and add almost nothing.

Keeping track between scans

A security review turns into a list your team works through over months. That list has to stay stable.

An agent starts from scratch on every run. Its three runs on the same code gave three different lists. A finding you fixed or rejected can return next week in new words, and one you never got to can drop off without notice.

Vybscan keeps its state between scans.

• Rule-based checks give the same result on the same code.
• An AI finding stays on the list until the code changes or a later review confirms it is gone.
• A finding you mark as a false positive stays dismissed on later scans, with a record of who dismissed it, when and why.
• Every scan is recorded, so you can see when each issue was found, fixed or introduced.
• When the code changes and the issue is no longer there, the finding is closed as fixed.

Dependencies: what an agent cannot check

Most of the code in an application comes from open-source packages, and an agent cannot tell you which of them are vulnerable.

• An agent knows only the advisories that existed when its model was trained. A vulnerability published last week is invisible to it. Vybscan checks every package version against current advisory data.
• In our 16 agent runs, it did not name a single package with a known vulnerability. None of its 75 notes on dependencies cited a CVE or an advisory.
• A lockfile can list thousands of direct and indirect packages. Vybscan reads them all, across npm, PyPI, Go, Maven and Gradle, Composer, NuGet, Cargo, RubyGems and Conda.
• New advisories keep appearing for code that has not changed. Vybscan re-checks dependencies every day and flags a new issue without a commit or a rerun.
• Vybscan also checks whether your code actually imports the vulnerable package, and it flags typosquats and known malicious packages.

What else comes with it

An agent reviews code when you ask it to. Vybscan also covers the other parts of a security review, and it runs on its own.

An agent also needs a machine with your code checked out, shell access, and a model account that someone pays for. Vybscan connects to GitHub or GitLab and sees only the repositories you choose.

Where a coding agent is the better tool

An agent is the better choice for some jobs.

• Investigating one specific question, such as whether a member of one team can read another team’s bookings. An agent can follow that through the code one file at a time.
• Precision. 77% of the agent’s findings were real, compared with 58.6% for Vybscan. If you only have time to read five findings, the agent’s five are more likely to be right.
• Writing fixes. Vybscan can export its findings for a coding agent to work through, so you can use Vybscan to find and track issues and an agent to fix them.

The agent also found bugs that Vybscan did not catch, including a stored XSS and an SSRF in kanboard. We have added checks for both, and they ship in the next Vybscan release.

Limits of this test

• Five codebases in TypeScript, Go and PHP is a small sample.
• We ran one open-source agent plugin with its default settings. Your own prompts, or pointing an agent at specific files, would give different numbers.
• The cost of an agent that reads every file is an estimate. We did not measure its recall or precision.
• Agent costs use Anthropic list prices on 5 November 2026. Vybscan costs are what our scans spent.
• Both tools found real bugs that the answer keys missed. We checked those by hand and counted them as real for both.

Summary

In our test Vybscan found almost four times as many known defects, spent a few dollars of AI per codebase per month, and kept a record of what the team decided. An agent has to be started, and paid for in full, every time.

The two work well together. Vybscan checks every change, and your agent fixes what it finds.