Every few weeks a vendor announces that AI coding agents can now “write secure code” or “auto-remediate vulnerabilities.” I’ve stopped taking these claims at face value, because most of them are measured against synthetic benchmarks the model has effectively memorized. Innodata just released something different: a benchmark built from thousands of real, historically exploited CVEs, replayed as live attacks against sealed copies of the vulnerable software, across Python, TypeScript, JavaScript, Rust, C, and Go.

The methodology is the interesting part. Innodata’s security engineers took each real-world flaw, rebuilt it by hand inside an isolated, offline environment, and re-ran the original attack. An AI agent’s patch is only scored as successful if the attack no longer works and the software still functions correctly. That’s a materially harder bar than “the linter is happy” or “the unit tests pass” — most vulnerabilities don’t get caught by either of those.

The number that matters

When shown exactly where a vulnerability was located, the best open-weight models patched at most 23% of flaws. Unguided — no hint about where the bug is, which is the realistic scenario for a security agent scanning your codebase — that number was lower before fine-tuning: 18.4%. After one round of fine-tuning on a slice of the suite’s data, unguided patch success rose to 41.2%.

Sit with that for a second. Even with fine-tuning specifically for this task, on a benchmark the model was trained toward, unguided real-world vulnerability patching tops out around 41%. That’s the ceiling for a model tuned on the exact distribution of the test. Your production codebase, running frameworks and patterns the training data didn’t cover, will land somewhere below that.

Why this matters for how you run code review

I’ve watched teams start treating AI-generated security fixes the same way they treat AI-generated feature code — skim it, run CI, merge if green. That’s a mistake specifically for security patches, for a reason this benchmark makes concrete: passing tests is not the same as closing the vulnerability. A patch can make the reported symptom go away (crash fixed, error suppressed) while leaving the actual exploitable path open, or introduce a narrower but still-exploitable variant of the same flaw. Innodata’s scoring method — replay the actual attack against the actual patch — is the only rigorous way to know the difference, and it’s not something your CI pipeline does by default.

Concretely, here’s what I’d change in a team’s workflow based on this data:

1. Route AI-generated security patches through a different review lane than feature PRs. Not a rubber stamp with an extra checkbox — an actual second reviewer with security context, or a SAST/DAST re-scan specifically targeting the CVE class the patch claims to fix.

2. Don’t trust “the vulnerability is fixed” from the same agent that wrote the fix. Self-assessment is the weakest signal here. If your agent says “patched,” verify with an independent scanner or exploit-replay step — the same principle Innodata used to build the benchmark, just at your scale.

# Minimal verification pattern for an AI-generated security patch
git diff main --stat  # scope check: does this touch more than it should?
semgrep --config=p/security-audit ./src  # independent static pass
# if you have a PoC or CVE reproduction, replay it against the patched build
./repro-exploit.sh --target ./build/patched

3. Track patch success rate by CVE class internally. If your agent handles SQL injection patches reliably but struggles with race-condition-class bugs (a common gap — race conditions require reasoning about execution order, not just pattern matching against known-bad code shapes), that’s exactly the kind of signal that should route certain bug classes to human-only review.

4. Treat “41.2% after fine-tuning” as a ceiling, not a floor, when a vendor pitches you an AI security tool. Ask what benchmark they’re quoting and whether it resembles Innodata’s replay-based methodology or a softer static-analysis pass rate. The gap between those two numbers is usually the gap between marketing and reality.

The honest takeaway

AI coding agents are a genuinely useful first pass at security remediation — triaging, suggesting a direction, drafting a patch attempt — and a genuinely risky final authority. The data here isn’t an argument against using them; it’s an argument against removing the human checkpoint that verifies the fix actually closes the hole. 41% unguided success, on a benchmark tuned in the model’s favor, is a good reason to keep that checkpoint in place for a while yet.