The Benchmark Obituary

The Audit

OpenAI's team reviewed 27.6% of the SWE-bench Verified dataset. Specifically, they audited the 138 problems that their most capable model, o3, could not consistently solve across 64 independent runs. Each problem was independently reviewed by at least six experienced software engineers. The findings: 59.4% of the audited problems have flawed test cases that reject functionally correct submissions. 35.5% enforce specific implementation details, like requiring a particular function name, that have nothing to do with whether the code actually works. Another 18.8% test for functionality that was never specified in the problem description. That is 16.4% of the entire benchmark confirmed broken. The remaining 72.4% was not audited. OpenAI also found that every major frontier model, including GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash, could reproduce verbatim gold patches for certain tasks. The models have memorized the answers. Improvements on SWE-bench Verified no longer reflect real coding ability. They reflect how much the model has seen the test.

The Score Everyone Cited

SWE-bench Verified was the industry's default measure of AI coding capability. OpenAI created it. Anthropic set the high score. Google, Meta, and every frontier lab cited it in launch announcements. Anthropic's Claude Mythos Preview holds the record at 93.9%. OpenAI's o3 reached the low 70s. Every model announcement in 2025 and 2026 included a SWE-bench Verified number because that is what investors, journalists, and regulators expected to see. Stanford HAI's 2025 AI Index documented the trajectory: SWE-bench scores jumped from 60% to near 100% in a single year. This acceleration curve was a central pillar of the "AI is improving exponentially" narrative that justified the largest venture capital deployment in history. In Q1 2026 alone, AI venture funding hit an estimated $242 to $300 billion. That is 80% of all global venture funding across all sectors. OpenAI closed a $122 billion round at an $852 billion valuation. Anthropic hit $380 billion. Every investor presentation cited benchmark performance. Every funding round coincided with model announcements touting benchmark improvements. The benchmark those numbers were based on was broken.

The 10-Line Script

In April 2026, researchers at UC Berkeley built an automated exploit agent that achieved near-perfect scores on eight major AI benchmarks without solving a single task. SWE-bench Verified: 100%. The exploit was a 10-line conftest.py pytest hook that intercepted test execution and returned passing results regardless of the actual code. SWE-bench Pro, the replacement OpenAI now recommends: also 100%. The exploit overwrote an in-container parser. Terminal-Bench: 100% via binary wrapper trojans. WebArena: approximately 100% via config leakage and DOM injection. FieldWorkArena: 100% because the validation system never checks whether the answer is correct. The researchers also found a real-world example: IQuest-Coder-V1 claimed 81.4% on SWE-bench Verified. Investigation revealed that 24.4% of its solution trajectories simply ran git log to copy answers from commit history. The measurement system was never validated against ground truth before being adopted industry-wide.

The Replacement Is Already Broken

OpenAI now reports scores on SWE-bench Pro, developed by Scale AI. It is larger (1,865 tasks versus 500), multilingual (Python, Go, TypeScript, JavaScript), and harder (top scores around 45-58% versus 93.9% on the old benchmark). It also includes 276 tasks from proprietary startup codebases that are legally inaccessible to model trainers, and 858 held-out tasks reserved to detect overfitting. Berkeley's researchers broke it anyway. SWE-bench's co-creator, Ofir Press, responded on Hacker News: "SWE-bench Verified is now saturated at 93.9%, but anyone who hasn't reached that number yet still has more room for growth. All benchmarks eventually become saturated." A Hacker News commenter put it differently: "Goodhart's Law in reverse: what can't be gamed gets rejected."

The Mythos Question

Anthropic's 93.9% on SWE-bench Verified, achieved by Claude Mythos Preview, is the highest score ever recorded. It is also now the most suspect. Anthropic argued that memorization does not explain Mythos's performance. Their evidence: even when filtering solutions by memorization probability thresholds, Mythos consistently outperformed Opus 4.6 at every level. A researcher at Philosophical Hacker demonstrated through simulation that this argument contains a statistical flaw. An imperfect memorization detector will always show a memorizing model appearing to outperform a clean model at every detection threshold, even when 100% of the improvement comes from memorization. Without quantifying the detector's false negative rate, consistent improvement patterns across thresholds provide zero evidence that the gains are genuine. As one Hacker News commenter noted: "You can trust that 40% versus 90% is indeed different. You cannot trust that 93% is better than 90%, because at that point it is impossible to distinguish between recall and reasoning."

What Was Actually Measured

A separate analysis found that throughout all of 2025, there was virtually no improvement in the rate at which models produced quality code. They only got better at passing automated tests. The benchmarks measured test-passing ability. The industry reported it as coding ability. The gap between those two things is the size of the problem. Stanford HAI's 2026 report concluded: "We do not have generally reliable AI. We have AI that is superhuman in narrow benchmarked domains and unreliable in others, sometimes within the same conversation." The same models that achieve gold-medal mathematics performance fail to read analog clocks 50% of the time. Stanford HAI documented invalid question rates across major benchmarks ranging from single digits to over 40% on some datasets. Hallucination rates across the 26 top models range from 22% to 94%. This is not a measurement system. This is a press release generator.

The Obituary

SWE-bench Verified is dead. OpenAI killed it. The cause of death was contamination, flawed test design, and the discovery that a trivial exploit could achieve a perfect score. But the benchmark's influence outlives it. The investment decisions it shaped are irreversible. The valuations it supported are baked into cap tables. The regulatory frameworks that referenced it are under construction. The press releases that cited it are still indexed by every search engine. We wrote in March that the AI industry measures itself with benchmarks it designs, administers, and interprets. That the student was grading its own exam. Now the exam maker has admitted the exam was broken. The student's grades do not change. $242 billion in Q1 2026. $852 billion valuations. Regulatory frameworks under construction. All built on an evaluation system that a 10-line Python script can defeat. The benchmark is dead. The money is not coming back.