AI Models Now Verify Code Safety

For decades, the process of mathematically proving that software runs without error has been reserved for the most critical systems in the world, such as aircraft avionics and operating system kernels. Now, artificial intelligence is beginning to handle that rigorous verification work at a scale previously thought impossible.

Anthropic has collaborated with Theorem to use its Claude model for the task of formal verification. Instead of writing new applications, the AI focused on existing code. The project involved analyzing test programs from 123 open-source libraries that are part of Google's OSS-Fuzz security initiative. Over a period of just a few weeks, Claude generated approximately 180 million lines of mathematical proof code. This massive output was designed to demonstrate the correctness of the underlying software.

Historically, this level of verification required significant human effort and time. A team of specialists once spent 20 person-years to formally verify the seL4 operating system kernel. Because of this high cost, the method was rarely used outside of extreme safety-critical environments. The new approach aims to change that by automating the labor-intensive proof generation. An independent verification program called Rocq checks the proofs produced by Claude. Anthropic describes the process as fully autonomous, meaning it requires very few human reviewers, although the foundational tools and formal structures were built by humans at Theorem.

The process yielded immediate practical benefits beyond just theoretical correctness. While generating proofs, Claude identified hundreds of bugs in the target libraries. Many of these defects had survived years of conventional testing because they only appeared under very specific conditions, such as unusually large input files or specific sequences of data access. In one notable instance, the team found as many security bugs in a single week in a widely used PDF reader as are typically reported for that software in an entire year. In another case, a logical error that had gone undetected for over a decade was caught by the AI.

However, the researchers emphasize that formal verification is not a magic bullet that guarantees absolute perfection. The proofs are valid only under strict conditions, including a specific processor architecture, single-threaded programs, and inputs smaller than 64 MB. Since these libraries often run on diverse hardware and configurations, the verification covers a defined subset of possible scenarios. Additionally, the computational cost is significant, with individual proofs consuming hundreds of millions of tokens. Anthropic did not disclose the total cost of the project, but it noted that efficiency is improving. Newer models, such as Opus 5.5, solve half the proofs with about 40 percent fewer tokens than the previous version.

This development shifts the focus of software automation from simple code generation to code validation. If AI can write code and also prove its safety under defined conditions, developers can trust the output more and spend less time on manual review. This could reduce costs across the entire coding process. Anthropic expects that formal verification using AI will soon become as routine as standard testing. Theorem's roadmap already includes plans to apply this method to critical infrastructure components like glibc, OpenSSL, and cURL. The goal is to make high-level mathematical proof a standard part of the software development lifecycle rather than a rare, expensive exception.