How I used o3 to find a remote 0-day vulnerability in the Linux kernel (ksmbd) (opens in new tab)

Love the rigor of running an experiment many times to see how often it got the desired outcome. Most people would just fire off a prompt and be disappointed if the model didn’t find the bug!

Seems a clever technique for anything that needs strong defense against hallucination. Kind of an “average across runs”. Manually auditing results isn’t very scalable (cf. the author says they missed that the LLM caught the second half of the bug in some runs but they missed that detail). In future an LLM could do that bit too so the technique becomes scalable. One can imagine being given a meta-report of what’s in all the reports produced by the runs.

We're approaching convincing David Gerard from last week

> I'll be convinced when LLMs start making valuable pull requests, non-obvious corner cases or non-trivial bugs in mature FOSS projects

https://pivot-to-ai.com/2025/05/13/if-ai-is-so-good-at-codin...