Ten security scanners on RealVuln, scored fairly: Vybscan at 1/8 the cost (plus SCA)

Every security scanner says it finds real bugs. Few say how often it is wrong. We wanted numbers for Vybscan that anyone can check. So we tested its code scanning (SAST) on four sets of apps. Two are real open-source projects. Two come from RealVuln, an open benchmark that also publishes results for nine other scanners.

This post shares what we found. It covers the good numbers and the weak ones.

Precision and recall, in plain words

Two numbers tell you how good a scanner is.

• Recall is how many of the real bugs it finds. Low recall means missed bugs.
• Precision is how many of its findings are real. Low precision means false alarms. Developers stop trusting a tool that cries wolf.

The two pull against each other. A scanner can flag everything and catch every bug, but then most of its findings are noise. The goal is high numbers on both.

What we tested

Vybscan ran with its production settings. That means Semgrep rules plus an AI review on Claude Haiku 4.5, capped at 300 files per app.

What is RealVuln?

RealVuln is an open benchmark for security scanners. Kolega.ai built it because older benchmarks fall short. Some use tiny made-up test files that a scanner can be tuned to pass. Others are vendor studies nobody else can repeat. You can see its live results on the RealVuln dashboard.

• What is in it. Version 3.0.0 has 140 apps in Python and TypeScript/JavaScript. All of them are vulnerable on purpose.
• What it labels. Each bug gets a file, a line and a CWE weakness type. On the TypeScript and JavaScript apps, nothing is marked safe.
• How it scores. A finding counts if the file matches, the CWE is accepted, and the line is within 10 of the label. Everything else is a false positive.
• What it publishes. The apps, the labels, the scoring code, and every scanner's raw findings and cost. That openness is what made this post possible.

One note for fairness. Kolega.ai also makes one of the ten scanners on the list, Kolega DevSec Max. We show its results exactly as published.

Results on real code

These scores use the list we ask users to fix. That is the main findings table on the Vybscan page. They come from the latest scans, on 24 September 2026. Answer keys grow as we find new real bugs, so earlier posts, such as why recall comes first, show slightly different counts.

The cal.com key is the hardest test of precision. It marks 2,901 places in the code as safe. Vybscan left 2,860 of them alone. That is a specificity of 98.6%. Still, 58% precision on a large real app means we have work to do.

Why benchmark scores undercount real bugs

RealVuln's official scores look low for every scanner. The best precision on its list is about 55%. We looked into why.

RealVuln lists only some of the bugs in each app. It does not claim to list them all. So any real bug missing from its key counts as a false positive.

We read every Vybscan finding that RealVuln rejected. We checked each one against the code. When we were unsure, we called it a false positive. The result:

• On the community apps, 176 rejected findings were real bugs. That is 43% more than the 413 bugs RealVuln labels.
• On the SaaS apps, 456 rejected findings were real bugs.

Many of the SaaS bugs are easy to explain. AI agents wrote those apps. The benchmark planted known bugs and labelled them. But the agents also wrote bugs by accident, and nobody labelled those.

Some examples of real bugs with no label: a sign-up form that lets anyone choose the admin role, a password reset link handed back to whoever asked for it, and a login throttle that trusts a header any client can set.

All ten scanners, scored the same way

We then corrected the scores for all ten scanners with the same list of verified bugs. If any scanner reported one of those bugs, it got credit for precision. Recall still uses only RealVuln's own labels.

This view has one bias, and we want to be clear about it. Our list of verified bugs came from Vybscan's findings. A real bug that only another scanner found is not on it. So the correction helps us a little more than it helps others. The official numbers sit next to the corrected ones, so you can compare both.

Community apps (21 apps, 413 labels)

SaaS apps (20 apps, 687 labels)

Sorted by corrected precision. Cost is each scanner's own recorded spend for the whole set. *GLM-5.3 has no published result for one app in each set. On the community apps, Vybscan read only JavaScript and TypeScript files but is scored over all 413 labels. Over JavaScript and TypeScript labels alone, its community recall is 61.3%.

Where Vybscan stands

• Precision is competitive. Corrected, Vybscan is 3rd of 10 on the community apps and 4th on the SaaS apps.
• Recall is where we invest most. Kolega DevSec Max, GPT Daybreak Blue and GPT-5.6 Sol find 20 to 40 points more of RealVuln's labelled bugs. On real code our recall is 85% to 92%. We explain why recall comes first for us in this post.
• Cost is our strength. The two leaders that publish a cost spent about eight times what we spent on the community apps and seven times on the SaaS apps.

For a team that scans every pull request, cost matters. A scan that costs cents can run on every change. A scan that costs dollars per app tends to run once a week, or not at all.

Why a general LLM is not a security program

A benchmark scores one scan of one snapshot. Real security needs scans on every change, memory of what your team dismissed, context about what actually runs, and a record an auditor can check. We cover this in depth in Why not just use the model?

This benchmark measures only part of Vybscan

RealVuln tests code scanning only. The other nine entries are code reviewers, and nothing else. Vybscan does more in the same scan, for the same cost:

• Dependency scanning (SCA). Known vulnerabilities from OSV and GitHub advisories, for npm, PyPI, Go, Maven, Gradle, Composer and NuGet.
• Malicious packages. Checks against the OpenSSF malicious packages feed, plus typosquat and dependency-confusion detection.
• Reachability. It checks whether your code actually uses the vulnerable part of a package, so you fix what matters first.
• Licences. It flags packages whose licence clashes with your policy.
• Secrets and infrastructure. Hardcoded keys, plus risky Terraform, Dockerfile, Compose and GitHub Actions settings.

How accurate is the dependency scanning? We keep answer keys for three public codebases, built from OSV for the exact package versions each one uses. Scored on 26 September 2026:

*Juice Shop commits no lockfile, so package versions are resolved fresh at each scan. Its key is older than 25 newer vulnerable versions, so extra findings cannot be scored fairly against it.

All of it runs on every pull request, on GitHub and GitLab, with one dashboard. A code-only agent run, however cheap, covers none of this. Read more in our SCA and SAST comparison.

How to read these numbers

• One run per app. The AI review is not fully repeatable. A rerun can move scores by a few points.
• We built two of the answer keys. That is a possible bias. The cal.com key is closed and strict. The Juice Shop key follows OWASP's own challenge list.
• We triaged the disputed findings ourselves. Nobody outside Dheemai has reviewed that work yet. Our triage file lists every finding with its reason. We will share it on request.

We would welcome one change to RealVuln. Marking safe code, as our cal.com key does, would let it measure precision properly. That would help every scanner and every buyer.

Frequently asked questions

What is the RealVuln benchmark?

An open benchmark for security scanners from Kolega.ai. It has 140 deliberately vulnerable apps. It labels each bug with a file, a line and a CWE, and it publishes every scanner's raw findings.

How accurate is Vybscan?

It finds 91.7% of known bugs in a cal.com fork and 85.0% in OWASP Juice Shop. On RealVuln, corrected for missing labels, its precision is 56.5% and 60.1% on the two sets.

Why is benchmark precision so low for every scanner?

Benchmarks label only some bugs. A real bug with no label counts as a false positive. We found 176 such bugs on the community apps and 456 on the SaaS apps.

Does Vybscan scan dependencies too?

Yes. Besides code, it scans dependencies for known vulnerabilities, malicious packages, typosquats and licence issues. On three public codebases it found 147 of 147 known vulnerable packages. It also finds hardcoded secrets and risky infrastructure settings.

What does Vybscan cost to run?

$6.51 for 21 community apps and $17.73 for 20 SaaS apps. The recall leaders that publish a cost spent seven to eight times more.

Want the full detail? Read how Vybscan compares on dependency and code scanning in our SCA and SAST comparison.

A note on these numbers: Vybscan changes every week. We add rules, tune the AI review and fix what the answer keys show us. We expect these scores to keep going up, and we will publish new results as they do.

Sources: RealVuln benchmark on GitHub. RealVuln dashboard. Why Kolega built RealVuln. OWASP Juice Shop. MITRE CWE.