The face-off: AI versus rule-based scanners
In the quiet, meticulous world of software security, two kinds of detectors keep watch over our code: the veteran static analyzers that run on rules, and the newer, flexible minds built from large language models. The former move like seasoned searchlights, scanning for known patterns of danger—SQL injections, hardcoded secrets, outdated libraries—triggering alarms only when they match something already written in their rulebooks. The latter, AI-driven systems, read code more like humans do: by understanding context, tracing data flows across files, and guessing where trouble might be hiding even if it hasn’t appeared in a prewritten pattern. It’s a clash of philosophies: precision and predictability on one side, broad reasoning and context on the other.
This study, conducted by Damian Gnieciak and Tomasz Szandala at the Wroclaw University of Science and Technology in Poland, puts six approaches head-to-head to answer a deceptively simple question: which detects real vulnerabilities better, and at what cost? The lineup is six distinct tools: three industry-standard static analyzers—SonarQube, CodeQL, and Snyk Code—and three large language models hosted on GitHub Models—GPT-4.1, Mistral Large, and DeepSeek V3. The aim isn’t to crown a single winner, but to illuminate the trade-offs each approach makes when faced with real-world code cribs containing 63 known vulnerabilities across common categories like SQL injection, hardcoded secrets, and deprecated dependencies.
The opening takeaway is provocative: the language-based scanners often outperform their static cousins on a broad measure of effectiveness. They lean into what the authors call better recall—the ability to find more of the actual problems—even when the patterns aren’t neatly codified in a rule. But that strength comes with a cost: more false alarms and, crucially for developers, a struggle to pinpoint the exact line and column of the vulnerability. The result is not a simple win for AI or tradition, but a drama of context and discipline playing out in real-time in CI pipelines and code reviews. The study’s authors argue for collaboration, not conquest: a hybrid pipeline that uses AI-driven triage to surface likely risks, followed by deterministic scanners to verify and localize issues with high assurance. The punchline is less “which tool is best?” than “how can we design a workflow that leverages the strengths of both?”
How they measured real risk in ten real projects
To test these detectors in a credible, reproducible way, the researchers built a disciplined, transparent benchmark. They gathered ten real-world C# projects, each peppered with a curated mix of vulnerability types, totaling 63 known weaknesses. That’s not a toy dataset: it mirrors the messy reality developers face, where bugs aren’t always obvious and not every suspicious signal sticks to a tidy rule. Each vulnerability has a ground truth label, which lets the team compute classic metrics: precision, recall, and the F1 score, a harmonic mean that balances the two. They also tracked practical factors that matter in the wild—how long each analysis takes, and how much developer effort is required to vet the findings.
An especially important aspect of the study is how results are presented and interpreted. The researchers used the Static Analysis Results Interchange Format (SARIF) to standardize outputs from the rule-based tools, an important step for apples-to-apples comparisons. When the large language models produced results, the team treated localization a bit differently due to how tokenization in transformers can blur exact line numbers. This nuance matters: even when a model surfaces the right file and a plausible vulnerability, its line-and-column precision might lag behind traditional scanners. That gap underlines a practical truth about AI in code: it often understands the scene, but it can stumble on the exact coordinates.
The ProjectAnalyzer—an open-source tool built for the study—helps operationalize the comparison. It collects the code files, composes a single aggregated prompt representing the entire project, and then queries each model via GitHub Models. The responses are parsed and translated into SARIF-like reports for uniform evaluation. The authors don’t just publish numbers; they provide a blueprint for how to run a similar, fair comparison in a real engineering team’s environment. That kind of openness matters because it turns a laboratory result into a practical, repeatable workflow that practitioners can adopt and stress-test.
The numbers tell a nuanced story about memory and context
If you’re scanning for vulnerabilities, you want a detector that doesn’t miss them, while not drowning you in false alarms. On this measure, the language models have the edge. Across the ten projects, GPT-4.1, Mistral Large, and DeepSeek V3 achieved average F1 scores around three-quarters to almost eight-tenths, markedly higher than the static tools. The best-performing language model, GPT-4.1, posted an average F1 near 0.80, with Mistral Large close behind at ~0.75 and DeepSeek V3 around 0.75 as well. By contrast, the trusty trio of SonarQube, CodeQL, and Snyk Code hovered in the 0.26 to 0.55 range, with Snyk Code doing the best among them, but still well below the AI group on average.
Where does the advantage come from? The study highlights a core strength of LLMs: their ability to reason across file boundaries and follow data flows through modules that a traditional scanner might treat as separate islands. In practice, that means an LLM can infer how a tainted input could propagate through layers and reach a vulnerable sink, even if the code path isn’t a textbook pattern. It’s a different kind of intelligence—forging a narrative of how data moves through a program rather than ticking a checklist of suspicious signatures. The payoff is especially clear for complex vulnerabilities that rely on broader program context rather than a single-line signature.
But the benefits aren’t free. The same broad reasoning that boosts recall also leads to more false positives, and in some cases, a flood of alerts that require developers to sift signal from noise. The study notes that SnykCode, a hybrid approach combining ML with traditional analysis, often sits in a middle ground: better than some pure static tools on the F1 metric, yet not immune to the same challenges the AI-heavy methods face. And even when the models detect a problem, their localization can be imperfect. Tokenization quirks in transformers—how the model slices text into tokens—sometimes misplace the reported region of the bug, complicating quick triage in tight CI windows.
What this means for building safer software
The practical upshot is not a single, white-knuckle triumph of AI over old-school tooling. It’s a pragmatic, human-centered prescription: use AI-based code analysis as a broad, context-aware triage tool early in development, then lean on deterministic, rule-based scanners to confirm and precisely localize vulnerabilities for critical audits. The authors explicitly advocate a hybrid pipeline: let language models skim the forest for likely trouble, then deploy established scanners to map out the exact trees and branches that need attention. In other words, AI can widen the net, but humans—and strict, rule-based tooling—still must close the net with precision where it counts.
That hybrid approach mirrors how teams already blend different skills and tools in software engineering. It also acknowledges real-world constraints: speed, cost, and the pain of false positives. Some tools, like CodeQL, trade immediacy for depth, showing longer runtimes that are acceptable in CI pipelines when the goal is thorough coverage. AI models, by contrast, can process larger contexts quickly but may require careful prompting and post-processing to keep false alarms manageable. The study’s cautionary notes about data privacy and potential hallucinations—where models invent deprecated-library warnings or obsolete versions—are more than academic quibbles. They shape how we deploy these systems in practice, especially in regulated industries or where code bases touch sensitive data.
A shared, open doorway for the field
A notable achievement of this work is not just the empirical findings but the infrastructure it builds to advance the field. By releasing the benchmark and a JSON-based result harness, the authors invite other researchers and practitioners to reproduce, challenge, and extend the comparison as threat vectors evolve. The project’s GitHub-hosted code, including a practical toolchain for running the analysis and generating reports, lowers the barrier to experimentation. In an industry that too often relies on anecdotes and warmed-up marketing pitches, this kind of openness helps move the conversation from “does it work on-paper?” to “how does it work inside our daily workflows?”
The study also anchors a conversation about who should lead in code analysis. The authors emphasize the value of combining human engineering judgment with evolving AI capabilities, rather than ceding control to one kind of detector. That stance resonates beyond vulnerability detection: it mirrors broader AI-adoption wisdom in tech industries, where augmenting expert work with capable tools tends to deliver the best balance of speed, reliability, and accountability.
What to watch for as the field moves forward
If you’re building or auditing software, the implications are practical and a little humbling. Don’t expect a single tool to solve every problem. Instead, design your pipeline to exploit complementary strengths: start with AI-powered triage to surface a broad map of risk, then apply deterministic scanners to confirm findings and pin down locations with surgical precision. The best deployments will likely blend both worlds, with human reviewers stepping in to adjudicate ambiguous signals and to assess risk in the context of your product and organization.
There are also important caveats to heed. The authors document how some models can hallucinate, flagging stale or irrelevant dependencies, or misreport the version of a library. They note that language models cannot reliably deliver exact line-and-column localization in the SARIF sense, a limitation you must accommodate when integrating with tooling that requires precise triage and reproducible audits. Data privacy remains a live concern: some models collect inputs, chat histories, or metadata that could expose sensitive information unless care is taken with policy and prompts. These realities aren’t roadblocks so much as design constraints that shape how you choose vendors, configure prompts, and build governance around AI-assisted code analysis.
From Poland to the coding world: a scholarly invitation
The study closes with a clear invitation: the field should embrace open benchmarks, transparent evaluation, and deliberate hybrid workflows that reflect how software teams actually work. The authors—affiliates of the Wroclaw University of Science and Technology—present a case study in how rigorous, context-rich testing can illuminate a path forward for tooling that used to feel like a choice between two extremes. If there’s a larger takeaway, it’s this: the future of secure software will be built not by one tool, but by a chorus of approaches that play to their strengths, with humans guiding the tempo and integrity of the performance.
Lead researchers Damian Gnieciak and Tomasz Szandala from the Wroclaw University of Science and Technology led the study, and their open benchmark invites the global community to join the conversation. The work also underscores the value of practical research—tools, datasets, and code are released so engineers can run their own tests, learn, and iterate. In a world where software increasingly sits at the center of every business, the message is hopeful: we can make security smarter without sacrificing trust, speed, or accountability. The fusion of human judgment with AI-powered analysis isn’t just possible; it’s already beginning to redefine how we defend the code that underpins our digital lives.
