1Password's AI patching benchmark is misleading

1Password’s FLAWED report, published on August 6, 2026, gives defenders a misleading picture of AI patching. Its headline says models produced clean fixes only 26% of the time. That figure includes experiments that deliberately instructed agents to apply the wrong fix, along with experiments in which agents could not compile or test their patches. The report risks making defenders less effective by discouraging them from using technology that could help them fix more vulnerabilities. Teams that take its headline at face value may leave repairable vulnerabilities unaddressed. We want our work to help defenders fix more vulnerabilities. This post shares real-world data on human and agent patch quality from our consulting projects and Patch the Planet. We’re also releasing two agent skills: post-patch-validation to help agents test fixes, and review-walkthrough to help engineers review them. How the experiment produces a misleading headline Our review of 1Password’s code and data found four choices that make its 26% clean-fix rate a misleading guide to ordinary patching work.1 The sample was selected for difficult fixes. The authors chose six vulnerabilities because their fixes were complex. Clean-fix rates ranged from 3% to 60% across those bugs, so the average depends heavily on which vulnerabilities made the list.2 Two prompts tell agents to apply the wrong fix. Those prompts account for 22% of the data. Combining them with ordinary repair attempts makes the reported rate depend partly on how often the researchers chose to give agents bad advice. More than a third of the trials prohibit testing. One evaluation mode prevents agents from building or running code and accounts for 36% of the data. The headline combines those trials with experiments in which agents could test their patches and act on the results. The models ran at different reasoning settings. GPT-5.5 ran at medium effort and Opus 4.8 at high. These were the tools’ defaults. Neither model was tested at its highest available setting, and the authors did not measure how increasing effort affected the results. 1Password’s headline also obscures a useful result in its own data. We reanalyzed the patches and recorded test results published with the study, keeping trials where agents could run code and were not instructed to apply the wrong fix. In those trials, 2,634 of 3,067 patches generated by 1Password’s models (86%) blocked the supplied exploit. We excluded runs that the study classified as having consulted the upstream fix. Blocking that exploit does not establish a complete repair, but these results show useful patching capability under reasonable working conditions that the headline fails to convey. The instructions and grading introduce further problems, several of which Davi Ottenheimer has also highlighted: The stopping rule and grading criteria disagree. Agents given a proof-of-concept exploit were instructed to stop once their patch defeated it. The grader then evaluated vulnerable paths that the supplied exploit did not exercise. The grading penalizes intended behavior changes. Agents were told to leave existing tests untouched, even though a correct fix can require updating tests to reflect changed behavior. We found that 8% of ActiveMQ verdicts penalized an intended behavior change as a regression. The automated grades disagree with human review. Models grading their own patches matched human reviewers on the full five-category outcome in 65.9% of reviewed cases. Agreement was 87.7% for whether the original bug was fixed and 70.5% for whether new bugs were introduced. (Table 24) Changing the reviewer changes the result. The two models assigned different outcomes to 36.8% of the same patches. The headline averages their assessments. (Table 20) The Linux reference fix contains a vulnerability. The authors found 248 generated patches that repeated an off-by-one error in the upstream fix. The automated grader caught that new vulnerability in only 24 of them. (Section 4.4) The Chromium grader accepts incomplete repairs. It marked many patches as clean even though they left a use-after-free vulnerability in a callback. (Section 4.9) The grading errors can penalize valid fixes and let vulnerable patches pass. Combined with the handpicked sample and deliberately bad instructions, they leave the report without a credible basis for its headline. Defenders should not take 1Password’s headline rate seriously as a measure of AI patching ability. Developers get one in eight fixes wrong under ideal conditions Understanding agent failures also requires understanding how often developers submit incomplete fixes. Our security consulting work gives us a detailed record of how developers repair vulnerabilities in their own software. We give clients detailed vulnerability reports, then conduct a “fix review” to check whether their proposed patches fully resolve the issues. Our records connect each vulnerability to the developer’s first proposed fix and our assessment of whether it worked. They preserve unsuccessful attempts that developers revise before an issue is considered resolved. We reviewed the first fixes submitted for 2,265 vulnerabilities across 236 Trail of Bits security assessments from 2024 to 2026. The developers maintained the affected software, had detailed reports from our engineers, and knew we would review their patches. Even under those favorable conditions, 283 first fixes failed to fully resolve the reported issue: 12.5%, or one in eight. Figure showing first fix submission outcomes Accounting for multiple fixes from the same assessment, the 95% confidence interval is 10.5% to 14.5%. Sometimes we point out a mistake in a client’s patch during an informal conversation, and they correct it before the formal fix review. Those early failures may never appear in the review record, so our data can undercount failed first attempts. We also excluded cases where the available records did not establish whether the fix worked. A direct comparison with agents would require the same tasks and working conditions. What happened to our patches in real projects Through Patch the Planet, our joint initiative with OpenAI, Trail of Bits has co-authored hundreds of patches for widely used open-source projects. Agents wrote the patches with engineers directing the work and checking the results. Project maintainers then decided whether to merge, revise, or reject each submission. How maintainers reviewed Patch the Planet patches We examined the public review history of every Patch the Planet submission in our dataset that maintainers had merged or closed by September 14, 2026: 186 pull requests. 1Password’s benchmark used six vulnerabilities selected because their fixes were complex. Maintainers merged 126 of our 186 pull requests, an acceptance rate of 67.7%.3 In 91 of those 126 pull requests (72.2%), maintainers accepted the security fix we originally proposed. Review outcome PRs % of merged PRs Total merged 126 100% Accepted with no security-relevant revision observed 91 72.2% Accepted with security-relevant revision observed 33 26.2% Indeterminate 2 1.6% Table 1: Changes requested by maintainers for 126 merged Patch the Planet pull requests. Security-related revisions include repairs to a proposed fix and expansions of its security coverage. Maintainer acceptance does not establish that every patch is correct. Maintainers closed the other 60 submissions without merging them. Most were superseded by other work or declined for policy, process, scope, or maintenance reasons. Four were explicitly rejected on technical grounds. Reason for closure PRs % of closed PRs Total closed without merge 60 100% Superseded, reimplemented, or re-landed elsewhere 36 60.0% Policy, process, scope, or maintenance reasons 14 23.3% Duplicate or convergent with another fix 3 5.0% Explicitly rejected on technical grounds 4 6.7% Other or indeterminate 3 5.0% Table 2: Reasons maintainers closed 60 Patch the Planet pull requests without merging One of those closed submissions was our freenginx patch. A maintainer and an agent introduced the same freenginx crash 1Password’s case study examines a Patch the Planet fix for a memory-safety bug in freenginx’s embedded Perl module. An agent wrote our patch under the direction of a Trail of Bits engineer. It left one vulnerable code path open and introduced a new crash during request cleanup. The paper’s criticism of our patch is correct. The maintainer closed our pull request and committed a separate fix. That fix covered all three vulnerable code paths but introduced the same crash during cleanup. The paper documents the maintainer’s regression too. Both authors encountered the same trap. The original bug allowed Perl to destroy a callback before freenginx used it. Both fixes kept the callback alive so freenginx could use it later. But if the request timed out first, freenginx would make the request unusable and then release the callback. Releasing it could run Perl code that still tried to use the request, crashing the worker. Both authors missed a problem their fix could cause later, during cleanup. Catching it required looking beyond the original bug to what happened when a request ended early. Two authors, one human and one agent, working separately, made the same mistake on the same bug. Readers deciding whether to use agents need to know how their failures compare with those of human developers. Establishing which is more reliable requires measuring both under comparable conditions. We checked what happened after our patches were merged We examined about 33,500 subsequent commits in Patch the Planet projects. When a later commit changed a file our patch had modified, we investigated whether it fixed a problem our patch had introduced. For each suspected regression, an agent attempted to demonstrate its impact with a proof of concept. Other agents and our engineers then challenged the findings. The review found at least ten functional bugs; four build, test, or release automation bugs; and one performance bug. It found no exploitable security vulnerabilities. Two examples illustrate the problems we identified: In go-jose, PR #240 fixed a missing-header crash but exposed an existing validation gap, allowing encrypted messages to succeed even when their key length contradicted the declared algorithm. PR #266 added explicit key-length checks before decryption. In Noble FROST, PR #250 returned cached round-two results without first checking whether a retry contained the same authenticated transcript. Changed or stale retry data could therefore bypass that check. The maintainer corrected the behavior by validating retries against the original transcript before returning cached results. We are extending this investigation to every patch we authored, including patches with maintainer contributions. The findings will help us add checks that catch these failures before we submit future patches. Agent skills for better security patches We are releasing two agent skills alongside this post: post-patch-validation to help agents test security fixes, and review-walkthrough to help engineers review code changes. Post-patch-validation is a new skill we wrote to help agents catch incomplete fixes and regressions before submitting patches for review. It was not used in the Patch the Planet work described above. The skill starts with a vulnerability report and the code before and after the patch. It guides the agent through four tasks: Reproduce the original bug. The agent writes a check that must fail on the vulnerable code and pass on the patched version. A test that passes on both revisions cannot demonstrate a fix. Test another path to the same failure. The skill requires at least one distinct variant based on the bug’s root cause, such as a different caller or a cleanup path. Check for regressions and new vulnerabilities. It compares behavior that should remain unchanged and tests security properties around the modified code. The plan must also include project tests, a sanitizer check, or a bounded fuzzing run. Treat broken test runs as inconclusive. A failed build or missing dependency must not be mistaken for evidence that a vulnerability was reproduced. Failed checks give the agent specific problems to investigate and repair before submitting its patch. The skill saves the tests and results so maintainers can see what was checked. To try post-patch-validation, install the skill and give your agent the vulnerability report and the vulnerable and patched revisions: “Use post-patch-validation to validate the patch in HEAD against <vulnerable-commit>, using the vulnerability report in <report-path>.” Review-walkthrough helps engineers review the patches they are responsible for merging. It turns a branch’s complete diff into an interactive walkthrough that explains the changes in a logical reading order. Review findings appear beside the relevant code, where engineers can inspect them and draft their own comments. The walkthrough can also prepare a GitHub review for submission. Follow the quick start to generate a walkthrough for your branch. These releases join our other public agent skills for improving security patches: variant-analysis helps agents find related defects elsewhere in the codebase. property-based-testing helps them test behavior across generated inputs. mutation-testing helps them determine whether their tests detect incorrect behavior and identify missing assertions. We publish these methods so other teams can use them to examine and improve their own patches. What a useful patching benchmark should measure A useful patching benchmark should measure whether agents help developers produce correct fixes and how much review those fixes require. The principles in our 2018 guide to evaluating fuzzing research apply here: Choose a sample that matches the research question. Explain how the sample was chosen and which repair work it represents. Difficult cases can expose failure modes. General failure rates require a representative sample. Measure the effects of working conditions. Give agents appropriate tools and instructions. Report model configurations and test how reasoning settings affect results. Report misleading prompts and restricted tool access separately. Make correctness verifiable. Check that patches fix the vulnerability beyond the supplied exploit. Test for security, functional, and performance regressions. Validate grades against expert review and publish the tests, configurations, and results. Show how results vary. Report per-vulnerability outcomes and variation across repeated attempts. Repeating trials on the same bugs cannot establish that those bugs represent everyday patching. Measure what agents contribute to the repair process. Compare developers working with and without agents on comparable tasks and under comparable conditions. Measure initial patch quality and the review and revision needed to reach a correct fix. We are optimistic about AI’s usefulness to defenders. Through Patch the Planet, we are committing engineering time to fixing vulnerabilities alongside the people who maintain the affected software. We examine failures so we can improve our methods. We will keep putting agents to work on difficult security problems and making the tools and lessons public. We want other teams to test our conclusions and take these methods further. Our goal is to give maintainers without dedicated security teams the ability to find and fix vulnerabilities that would otherwise go unaddressed. The paper defines a clean fix as fully resolving the vulnerability without materially changing application behavior. ↩︎ The six-target mean has a standard error of about nine percentage points, which the report does not disclose. ↩︎ As of September 14, 2026, our dataset contained 240 public upstream pull requests. We excluded the 54 submissions still open from the outcome analysis. We count pull requests, each of which can contain more than one patch, and exclude maintainer-written replacements from our merged total. ↩︎