The problem
How it works
He first tested o3 through the API on a use-after-free he had already found by hand (CVE-2025-37778), feeding about 3.3k lines of session-setup code with the llm and files-to-prompt command-line tools and repeating each experiment 100 times. He then gave it about 12k lines covering all command handlers, with no agent framework or tool use.
Results and limitations
He reports o3 found the benchmark bug in 8 of 100 runs (Claude Sonnet 3.7 managed 3), and on the larger code set it flagged a new use-after-free in the session logoff handler, now CVE-2025-37899; it also showed his earlier fix was incomplete.
Before you try it
Very noisy: he estimates about one real finding per 50 false ones on the large run, and the known bug was found in only 1 of 100 runs there. It depends on an expert checking every report, used a single benchmark bug, and 100 large runs cost about $116.
Tools and repositories
No ranked repository is involved. This use case still stands on its own source.
- Tools
- OpenAI o3 · llm CLI · files-to-prompt
- Prerequisites
- Requires security expertise to triage reports and API budget for repeated runs.
Try it yourself: Review a code region for one bug class, repeatedly
Fill the fields to adapt the prompt. Empty fields stay as visible [placeholders].
You are assisting a security review of [Component under review] in [Source path], which I am authorized to audit. Focus only on [Bug class]. I will give you [Code provided] plus notes on how data is shared between connections and threads. For each candidate issue, give the file and function, the exact sequence of events that triggers it, why existing locks or reference counts do not prevent it, and your confidence. Say plainly when you find nothing. Do not write exploit code, contact maintainers or run anything against live systems; I will verify each report by hand and handle disclosure myself.Expect many false positives, run several passes, and report only issues a human has confirmed, through responsible disclosure.
Sources
- How I used o3 to find CVE-2025-37899, a remote zeroday vulnerability in the Linux kernel's SMB implementation (opens in a new tab)
First-person account · sean.heelan.io · Published 22 May 2025 · Source checked 9 Nov 2026, 22:15 UTC
What this source supports
- o3 via API with no agent framework
- Found the benchmark bug in 8 of 100 runs; Sonnet 3.7 in 3
- Found new use-after-free CVE-2025-37899 in the logoff handler
- Roughly 1:50 signal-to-noise on the large run; about $116 for 100 runs