N-Day-Bench – Can LLMs find real vulnerabilities in real codebases? (opens in new tab)

(ndaybench.winfunc.com)

91 pointsmufeedvh26d ago29 comments

N-Day-Bench tests whether frontier LLMs can find known security vulnerabilities in real repository code. Each month it pulls fresh cases from GitHub security advisories, checks out the repo at the last commit before the patch, and gives models a sandboxed bash shell to explore the codebase.

Static vulnerability discovery benchmarks become outdated quickly. Cases leak into training data, and scores start measuring memorization. The monthly refresh keeps the test set ahead of contamination — or at least makes the contamination window honest.

Each case runs three agents: a Curator reads the advisory and builds an answer key, a Finder (the model under test) gets 24 shell steps to explore the code and write a structured report, and a Judge scores the blinded submission. The Finder never sees the patch. It starts from sink hints and must trace the bug through actual code.

Only repos with 10k+ stars qualify. A diversity pass prevents any single repo from dominating the set. Ambiguous advisories (merge commits, multi-repo references, unresolvable refs) are dropped.

Currently evaluating GPT-5.4, Claude Opus 4.6, Gemini 3.1 Pro, GLM-5.1, and Kimi K2.5. All traces are public.

Methodology: https://ndaybench.winfunc.com/methodology

Live Leaderboard: https://ndaybench.winfunc.com/leaderboard

Live Traces: https://ndaybench.winfunc.com/traces

29 comments

sigmoid1025d ago

Interesting, but there is something really off here. Probably caused by a harness bug, but it heavily screws output and I wouldn't trust anything about this leaderboard right now. Consider this case:

https://ndaybench.winfunc.com/cases/case_874d1b0586784db38b9...

GPT 5.4 allegedly failed, but if you look at the trace, you'll see that it simply couldn't find the file specified in the input prompt. It gave up after 9 steps of searching and was then judged as "missed."

Claude Opus 4.6 somehow passed with grade "excellent", but if you look at its trace, it never managed to find the file either. It just ran out of tool calls after the allowed 24 steps. But instead of admitting defeat, it hallucinated a vulnerability report (probably from similar code or vulnerabilities in its training corpus), which was somehow judged to be correct.

So if you want this to be remotely useful for comparing models, the judging model definitely needs to look at every step of finding the bug, not just the final model output summary.

Aurornis25d ago

Good find. This appears to be another vibe coded vanity project where the output was never checked.

All of the online spaces where LLMs are discussed are having a problem with the volume of poorly vibecoded submissions like this. Historically I’ve really enjoyed Show HN type submissions but this year most of the small projects that get shared here and on other social medias turn out to be a waste of my time due to all of the vibecoding and how frequently the projects don’t do what they say they do when you look into the details.

sacrelege26d ago

Thanks for putting N-Day-Bench together - really interesting benchmark design and results.

I'd love to see how the model we serve, Qwen3.5 122B A10B, stacks up against the rest on this benchmark. AI Router Switzerland (aiRouter.ch) can sponsor free API access for about a month if that helps for adding it to the evaluation set.

ra26d ago

Nice. I've been thinking of doing something similar in our local jurisdiction (Australia).

Are you able to share (or point me toward) any high-level details: (key hardware, hosting stack, high-level economics, key challenges)?

I'd love to offer to buy you a coffee but I won't be in Switzerland any time soon.

sacrelege25d ago

Ah thanks, I love coffee

At a high level, it's a mix of our own GPU capacity plus the ability to burst into external nodes when things get busy. Right now we're running a bunch of RTX PRO 6000s, which basically forces you into workstation/server boards since you need full x16 PCIe 5.0 lanes per card.

We operate a small private datacenter, which gives us some flexibility in how we deploy and scale hardware. On the software side, we're currently LiteLLM as a load balancer in front of the inference servers, though I'm in the process of replacing that with a custom rust based implementation.

We've only been online since the beginning of this month, so I can't really say much about the economics yet, but we've had some really nice feedback from early customers so far. :)

Tepix26d ago

Interesting. How fast is your service? Do you guarantee a certain number of tokens/s?

sacrelege25d ago

We typically observe throughput of around 100–110 toks/s, and for larger context sizes this ranges between 90–100 toks/s.

While we don't guarantee a fixed toks/s rate, we scale by provisioning external GPU nodes during peak demand. These nodes run our own dockerized environment over a secure tunnel.

Our goal is to ensure a consistent baseline performance of at least 60–80 toks/s, even under high load.

Tepix25d ago

Sounds good. I saw that you use the FP8 version of the model. Do you also quantize the KV cache?

1 more reply

mbbutler26d ago

It would be helpful to add in some cases that do not contain any vulnerabilities to assess false-positive rate as well.

mufeedvhOP26d ago

This is a good idea.

Will incorporate false-positive rates into the rubric from the next run onwards.

At winfunc, we spent a lot of research time taming these models to eradicate false-positive rates (it's high!) so this does feel important enough to be documented. Thanks!

cortesoft26d ago

Any code that is certain that it doesn't have any vulnerabilities is going to be pretty trivial to verify.

Cynddl26d ago

> Each case runs three agents: a Curator reads the advisory and builds an answer key, a Finder (the model under test) gets 24 shell steps to explore the code and write a structured report, and a Judge scores the blinded submission. The Finder never sees the patch. It starts from sink hints and must trace the bug through actual code.

Curator, answer key, Finder, shell steps, structured report, sink hints… I understand nothing. Did you use an LLM to generate this HN submission?

It looks like a standard LLM-as-a-judge approach. Do you manually validate or verify some of the results? Done poorly, the results can be very noisy and meaningless.

yorwba26d ago

Yeah, the LLM judge is a bit too gullible. GLM 5.1 here https://ndaybench.winfunc.com/traces/trace_585887808ff443cca... claims that onnx/checker.cc doesn't reject hardlinks, even though it does (and the model output even quotes the lines that perform the check). The actual patch https://github.com/onnx/onnx/commit/4755f8053928dce18a61db8f... instead adds using std::filesystem::weakly_canonical to catch path traversal through symlinks. It also adds a Python function that does the same (?) checks when saving files. Honestly, even that patch seems LLM-generated to me, the way it duplicates code in a bunch of places instead of channeling all file accesses through a single hardened function.

Anyway, GLM 5.1 gets a score of 93 for its incorrect report.

rohansood1526d ago

I worked in AppSec in the past, made sense to me. Maybe you aren't the target audience?

You don't really need manual verification for these, the CVEs (vulnerabilities) are public and can be programmatically validated.

muldvarp26d ago

Manual verification that the "judge" judges correctly.

Also, how exactly do you programmatically validate CVEs?

rohansood1525d ago

Most open-source CVEs will have a patch linked in their disclosure. You can get vulnerable code via the git diff, then just verify if it is part of the LLM's finding.

johnfn26d ago

Is this really that hard to parse?

Curator and Finder are the names of the agents. "answer key" - haven't you ever taken a test in high school? It's an explanation of the answer. "shell steps" I presume means it gets to run 24 commands on the shell. "structured report" - do I really need to explain to you what a report is? "sink hints" - I admit I didn't know this one, but a bit of searching indicates that it's a hint at where the vulnerability lies.

peyton26d ago

> Did you use an LLM to generate this HN submission?

Must have.

> The Finder will never see the patch.

I wasn’t worried that this eval would show the answer to the model before evaluating it. Seems requirements leaked into this post.

croemer25d ago

Heavily vibe coded, the judge can even change the weights and that's presented as a feature ("conscious tradeoff"), see methodology section 7:

> The rubric is fixed across all cases. Five dimensions, weighted: target alignment (30%), source-to-sink reasoning (30%), impact and exploitability (20%), evidence quality (10%), and overclaim control (10%).

> There's no server-side arithmetic that recomputes the overall score from dimension scores and weights. The Judge LLM produces the entire score object in one pass. This is a conscious trade-off: it avoids the brittleness of post-hoc formula application at the cost of giving the Judge more interpretive latitude than a mechanical scorer would have.

How on earth is a post-hoc formula application "brittle"? Classic LLM giving bogus reasons instead of the real ones (laziness).

StrauXX26d ago

Do you plan on adding more models in the future? I would love to see how other OSS modles like Gemma, GPT-OSS and Qwen fare.

zurfer25d ago

Really cool. One thing wonder: Are they allowed to search the internet? If so, how do you filter out results after the vuln got published?

croemer25d ago

Traces being public is nice, but shouldn't the whole harness be open source? Otherwise, it's hard to trust.

RALaBarge25d ago

I can say without a shadow of a doubt: yes.

Rohinator26d ago

Very curious how Claude Mythos will perform here

spicyusername26d ago

I'd love to see some of the open source models in there

PeterStuer26d ago

You mean like Kimi-K2.5 or GLM 5.1?

spicyusername25d ago

Qwen, Gemma, Nemotron, etc

withinboredom26d ago

I didn’t read tfa, but can we also have it be able to distinguish when a vulnerability doesn’t apply? As an open source contributor, people open nonsensical security issues all the time. It’s getting annoying.

j / k navigate · click thread line to collapse

29 comments

sigmoid1025d ago

Interesting, but there is something really off here. Probably caused by a harness bug, but it heavily screws output and I wouldn't trust anything about this leaderboard right now. Consider this case:

https://ndaybench.winfunc.com/cases/case_874d1b0586784db38b9...

So if you want this to be remotely useful for comparing models, the judging model definitely needs to look at every step of finding the bug, not just the final model output summary.

Aurornis25d ago

Good find. This appears to be another vibe coded vanity project where the output was never checked.

sacrelege26d ago

Thanks for putting N-Day-Bench together - really interesting benchmark design and results.

ra26d ago

Nice. I've been thinking of doing something similar in our local jurisdiction (Australia).

Are you able to share (or point me toward) any high-level details: (key hardware, hosting stack, high-level economics, key challenges)?

I'd love to offer to buy you a coffee but I won't be in Switzerland any time soon.

sacrelege25d ago

Ah thanks, I love coffee

We've only been online since the beginning of this month, so I can't really say much about the economics yet, but we've had some really nice feedback from early customers so far. :)

Tepix26d ago

Interesting. How fast is your service? Do you guarantee a certain number of tokens/s?

sacrelege25d ago

We typically observe throughput of around 100–110 toks/s, and for larger context sizes this ranges between 90–100 toks/s.

While we don't guarantee a fixed toks/s rate, we scale by provisioning external GPU nodes during peak demand. These nodes run our own dockerized environment over a secure tunnel.

Our goal is to ensure a consistent baseline performance of at least 60–80 toks/s, even under high load.

Tepix25d ago

Sounds good. I saw that you use the FP8 version of the model. Do you also quantize the KV cache?

1 more reply

mbbutler26d ago

It would be helpful to add in some cases that do not contain any vulnerabilities to assess false-positive rate as well.

mufeedvhOP26d ago

This is a good idea.

Will incorporate false-positive rates into the rubric from the next run onwards.

At winfunc, we spent a lot of research time taming these models to eradicate false-positive rates (it's high!) so this does feel important enough to be documented. Thanks!

cortesoft26d ago

Any code that is certain that it doesn't have any vulnerabilities is going to be pretty trivial to verify.

Cynddl26d ago

Curator, answer key, Finder, shell steps, structured report, sink hints… I understand nothing. Did you use an LLM to generate this HN submission?

It looks like a standard LLM-as-a-judge approach. Do you manually validate or verify some of the results? Done poorly, the results can be very noisy and meaningless.

yorwba26d ago

Anyway, GLM 5.1 gets a score of 93 for its incorrect report.

rohansood1526d ago

I worked in AppSec in the past, made sense to me. Maybe you aren't the target audience?

You don't really need manual verification for these, the CVEs (vulnerabilities) are public and can be programmatically validated.

muldvarp26d ago

Manual verification that the "judge" judges correctly.

Also, how exactly do you programmatically validate CVEs?

rohansood1525d ago

Most open-source CVEs will have a patch linked in their disclosure. You can get vulnerable code via the git diff, then just verify if it is part of the LLM's finding.

johnfn26d ago

Is this really that hard to parse?

peyton26d ago

> Did you use an LLM to generate this HN submission?

Must have.

> The Finder will never see the patch.

I wasn’t worried that this eval would show the answer to the model before evaluating it. Seems requirements leaked into this post.

croemer25d ago

Heavily vibe coded, the judge can even change the weights and that's presented as a feature ("conscious tradeoff"), see methodology section 7:

How on earth is a post-hoc formula application "brittle"? Classic LLM giving bogus reasons instead of the real ones (laziness).

StrauXX26d ago

Do you plan on adding more models in the future? I would love to see how other OSS modles like Gemma, GPT-OSS and Qwen fare.

zurfer25d ago

Really cool. One thing wonder: Are they allowed to search the internet? If so, how do you filter out results after the vuln got published?

croemer25d ago

Traces being public is nice, but shouldn't the whole harness be open source? Otherwise, it's hard to trust.

RALaBarge25d ago

I can say without a shadow of a doubt: yes.

Rohinator26d ago

Very curious how Claude Mythos will perform here

spicyusername26d ago

I'd love to see some of the open source models in there

PeterStuer26d ago

You mean like Kimi-K2.5 or GLM 5.1?

spicyusername25d ago

Qwen, Gemma, Nemotron, etc

withinboredom26d ago

j / k navigate · click thread line to collapse