ponytail-sec vs Vybscan: one finds less, one finds more

← Back to Blog

Vybscan found 3.7 times as many known security bugs as ponytail-sec on five open-source codebases, at half the cost per run. ponytail-sec was more precise on three of the five. It also found three real bugs our rules missed, and we have since written rules for all three.

ponytail-sec is an open-source plugin for Claude Code, MIT-licensed, by Andy Pitcher. It is a set of prompts, not a scanning engine. You run /ponytail-sec-audit inside Claude Code, and the model on your account reads the repository. Its stated goal is “the smallest change that ruins an attacker’s day”: a few high-leverage fixes, not a long list.

Vybscan is a code scanner. It runs 411 Semgrep rules, a secrets pass, dependency scanning and an AI review on every repository. The two tools answer different questions. We still wanted to know, with numbers, what each one finds.

How we tested

We used five codebases that each have an answer key: a list of known bugs, every one located to a file and line.

• Same code. Both tools scanned the same commit of each repository.
• ponytail-sec, three runs. An AI audit gives a different answer each time, so we ran it three times per repository and report the average. It ran headless (claude -p) on Claude Opus, the Claude Code default, with version 0.2.6 of the plugin.
• Vybscan, one full scan. Each repository’s latest full scan, on 2 November 2026. Dub’s is from 8 October, the last scan at the commit its key describes.
• Same grading. Both tools were graded by the same code, with the same line matching and the same false-positive lists.
• Both triaged. Any finding the key did not cover, from either tool, we read against the source and marked real, false or unconfirmed. Unconfirmed findings are left out of precision.
• What counts. ponytail-sec files each finding as a vulnerability or as hardening. We counted its vulnerabilities, and treated hardening like Vybscan’s own hardening drawer: shown, but not scored.
• Cost is the measured model spend for one run. It leaves out compute. Vybscan’s cost also covers its dependency and secrets scans.

Results

Across all five codebases, Vybscan found 149 of 188 known bugs (79.3%), and ponytail-sec found 40 (21.3%). On average one ponytail-sec run cost $0.57 per known bug found, and one Vybscan scan cost $0.08.

• Recall is the share of known bugs a tool reported where the reader counts them. For ponytail-sec it is the average of three runs.
• Precision is the share of reported findings that are real. Vybscan reports far more findings: 318 counted findings across the five codebases, against 72 for ponytail-sec (three-run average).
• Cost is Vybscan’s recorded model spend for that scan, and ponytail-sec’s average for one run. Dub’s Vybscan cost is the average of its two most recent full scans, because the 8 October scan’s cost record is gone.

What ponytail-sec does well

ponytail-sec says little, and most of it is right. On Gogs, six of its seven counted findings were real. On Cal.diy, five of seven. A developer reading its report wastes little time.

It also found three real bugs that Vybscan’s rules did not:

We wrote a rule for each of the three, as general patterns rather than for one file. Each rule finds its bug, stays quiet on Kanboard’s fixed release, and adds no new findings on Laravel, DVWA, Cal.diy, Novu, Juice Shop or the 140 apps of the RealVuln benchmark. The rules ship in the next Vybscan release. That is the most useful outcome of this comparison for us.

Its reports are also good to read. Each finding names the smallest fix and how likely that fix is to break something. Its hardening advice for Kubernetes and Docker is concrete.

Where it falls short

ponytail-sec misses most known bugs. Its best result was Juice Shop, at 43.3% recall. On the four real-world codebases it found between 3.7% and 8.3% of known bugs per run.

• Runs disagree. Each run picked different bugs. On Cal.diy one run found one known bug and another found two. Across all three runs together, it found four of the 36.
• Training apps read as fine. In its default mode it treats planted flaws as part of the app’s design. On DVWA it reported none of the 57 known bugs, calling them “intentional training content”. For Juice Shop we told it to treat the app as production.
• Some claims ignore the framework. Five of its Juice Shop findings said users could act on other users’ data by changing a user ID. Juice Shop’s server replaces that ID with the logged-in user’s before the handler runs, so none of the five holds.
• It costs more to run. One run on Opus cost between $2.57 and $6.53 on average per codebase. One Dub run cost $11.71.
• No vulnerable-package check. Its dependency pass asks whether each package is needed and maintained. In 16 runs it named no package with a known vulnerability: none of its 75 dependency notes cites a CVE or an advisory.

Our own numbers, honestly

On Cal.diy, Vybscan’s recall today is 80.6%, lower than the 91.7% we published on 26 September. Its precision is 42.9%, against 58% then. Same code, same answer key. Here is why.

• Our review step rejected two real bugs. Since 14 October, the AI review that checks each finding has marked two known bugs as “likely not a bug”. One is a webhook URL check that does not resolve DNS, so it can be bypassed by DNS rebinding. The review read the check and trusted it. Review verdicts are kept for unchanged code, so the mistake repeats on every scan.
• One bug moved to the wrong section. A shell script that pipes a download straight into a shell is filed under “Dev / Build” as an operator script. It still shows, but outside the count.
• AI findings vary between scans. Four to six of Cal.diy’s 36 bugs are found by the AI pass alone, and they come and go. Across five scans at the same commit since 14 October, recall ranged from 80.6% to 88.9%.

The first two are ours to fix. The third is the price of AI review, and why our rules matter: a rule finds the same bug every time.

Limits of this comparison

• Keys built partly from our scans. The Juice Shop, Cal.diy and Dub keys were written in part by reading Vybscan’s findings. Our findings were therefore labelled before this test, and ponytail-sec’s were not. We triaged every unlabelled finding of both tools by hand to even that out.
• One reviewer. That triage was done once, by one person, with no second reader.
• Rules written from the same advisories. Many Vybscan rules for Gogs and Kanboard were written after studying these very advisories. That favours Vybscan on those two.
• Different models. ponytail-sec ran on Claude Opus. Vybscan’s AI pass runs on Claude Haiku 4.5, with a separate review model. A different model would give ponytail-sec different results.
• Different goals. ponytail-sec is built to name a few high-leverage fixes, not to list every bug. Recall is not what it optimises for.
• Small samples. Three runs per codebase, five codebases. Dub has only four known bugs, so one bug moves its recall by 25 points.

Which to use

Use ponytail-sec as a second reader. Run it on a change you are about to ship, or on a repository you know well, and read its few findings closely. They are usually right, and some will be things a scanner misses.

Use a scanner to know what is in your code. Most known bugs in these five codebases were invisible to an AI audit on any single run. A scanner checks every file on every commit, the same way each time, and tells you which dependencies carry known vulnerabilities.

The two work well together. ponytail-sec found three bugs we had missed. Each is now a rule that checks every repository we scan, every time.

Sources: ponytail-sec on GitHub (version 0.2.6, MIT licence) · Ten security scanners on RealVuln, scored fairly (26 September 2026) · Kanboard 1.2.54.