Layla Zulfa
The Ethereum Foundation’s Protocol Security team has initiated a groundbreaking program deploying coordinated artificial intelligence (AI) agents to audit the foundational code of the Ethereum network. This innovative approach has successfully identified real-world vulnerabilities, including a significant remotely-triggerable panic in libp2p’s gossipsub, a core component of Ethereum consensus clients, which has been publicly disclosed as CVE-2026-34219. While the agents’ ability to find bugs was anticipated, the team’s primary revelation was the disproportionate effort required to distinguish genuine security flaws from the multitude of confident-sounding, yet ultimately false, positives. This pioneering work signals a transformative shift in blockchain security practices, moving the bottleneck from bug discovery to rigorous validation and human-led triage.
The Imperative for Advanced Blockchain Security
Ethereum, the world’s leading smart contract platform, underpins a vast decentralized ecosystem with hundreds of billions of dollars in total value locked (TVL). Its robust operation relies on the integrity of complex software systems, including core network protocols, cryptographic implementations, and smart contract code. Given the immutable nature of blockchain transactions and the high stakes involved, even minor vulnerabilities can have catastrophic consequences, as evidenced by numerous exploits and hacks in the broader crypto space over the years. This inherent criticality necessitates a proactive and sophisticated security posture, constantly pushing the boundaries of vulnerability detection and mitigation. Traditional auditing methods, while essential, often struggle to keep pace with the sheer volume and complexity of evolving codebases. It is within this context that the Ethereum Foundation began exploring the potential of AI to augment and enhance its security efforts.
Unveiling the Methodology: Coordinated AI Agent Operations
The Ethereum Foundation’s Protocol Security team employs a fleet of AI agents working in parallel against target systems crucial to the network. These targets range from systems software and cryptographic libraries to critical smart contracts that demand absolute correctness. Rather than a centralized orchestrator, the agents coordinate their activities through the shared repository of the target codebase itself, using version control as a shared state mechanism. This decentralized coordination model, inspired by Anthropic’s successful endeavor in building a C compiler with a fleet of agents, eliminates single points of failure and simplifies maintenance. An agent identifies a potential vulnerability, logs its claim in the shared repository, performs the necessary analysis, and commits its findings, making them visible to other agents and human auditors.
This distributed, autonomous approach allows for a significantly broader and deeper search across vast codebases than human teams could achieve alone. The structure of a "candidate finding" is meticulously defined to ensure clarity and testability. Each candidate must include:
- Target: The specific component and entry point an attacker could realistically reach.
- Invariant: The fundamental property that is expected to hold true.
- Mechanism: The precise method by which the invariant might be violated.
- Success: An observable and verifiable proof of failure, such as a panic, a system stall, or the acceptance of invalid input.
- Reproducer: A self-contained artifact, typically a piece of code, that can reliably trigger the identified failure against the real system.
- Dedup Key: A unique identifier to prevent redundant investigations by multiple agents.
This structured schema is not merely for reporting; it is a critical design choice that forces the AI agents to articulate specific, testable claims. By demanding an "observable proof" and a "reproducer," the methodology effectively prevents agents from generating vague or speculative warnings, ensuring that every reported candidate is backed by tangible evidence.
The Crucial Distinction: Reproducibility and Validation
The most paramount rule governing this AI-driven audit process is unequivocal: "Reproducible or it didn’t happen." A candidate vulnerability is elevated to the status of a legitimate finding only when a self-contained artifact can demonstrably reproduce the failure against the actual protocol code. This reproducer must be executable by anyone, irrespective of whether they developed it or possess intimate knowledge of the AI agent’s internal workings. This stringent requirement is the primary bulwark against the inherent noisiness of AI-generated content.
The value of this reproducer-centric approach is most evident in its ability to filter out false positives. The team has observed three recurring categories of misleading results:
- Tests Passing for the Wrong Reason: Similar to a poorly written unit test, an agent might construct a reproducer that appears to run successfully but, in reality, doesn’t actually verify the intended condition or trigger the supposed flaw. The observed "success" is a false positive.
- Agent Hallucinations: AI models are known to "hallucinate" or generate plausible-sounding but factually incorrect information. An agent might confidently describe a non-existent vulnerability or misinterpret code behavior, leading to a reproducer that fails to demonstrate any actual bug.
- Checking the Wrong Property: The agent might correctly identify a behavior but incorrectly frame it as a vulnerability, applying a security invariant that is not relevant or applicable to the specific code segment or context.
These scenarios highlight a fundamental challenge with generative AI in security: its output, while often articulate and convincing, requires independent, objective verification. The volume of such confidently erroneous outputs necessitates an automated and robust validation mechanism. The reproducer serves precisely this purpose, acting as an impartial arbiter that operates independently of the agent’s textual claims.
Navigating the Noise: Prioritizing Real Vulnerabilities
The core of the work, paradoxically, lies not in the initial generation of potential bugs, but in the painstaking process of "signal-to-noise" reduction. A significant majority of candidate findings generated by AI agents are typically wrong, duplicative, or fall outside the defined scope of the audit. This is an expected characteristic of the method, and the operational goal is to swiftly reject invalid claims while rigorously substantiating genuine vulnerabilities with undeniable proof.
Every candidate that survives the initial reproducer validation undergoes two further independent checks by human experts:
- Attacker Reachability: Can a real-world attacker genuinely reach the identified vulnerable component or entry point under normal network configurations? This filters out theoretical flaws that lack practical exploitability.
- Cost-Benefit Analysis: What resources or privileges would an attacker need to exploit the vulnerability, and what would be the corresponding impact on the network if successful? A bug triggerable by any single peer carries a vastly different risk profile than one requiring specialized access or immense computational resources.
Furthermore, a dynamic and continuously updated list of known, fixed, or previously rejected issues is maintained. This prevents agents from repeatedly "rediscovering" and reporting the same closed vulnerabilities, a common issue when dealing with large-scale automated searches. The acceptance rate of candidate findings varies significantly depending on the maturity and prior auditing of the target codebase. For highly scrutinized, mature code, the yield of new, critical bugs might be low, yet this "found nothing significant" result itself is valuable, indicating a high level of security assurance. Conversely, less explored codebases or formally verified code where the deployed bytecode might diverge from the proven model often yield a higher number of valid findings. This observation aligns with similar experiences reported by other leading organizations. Cloudflare, for instance, concluded from their own AI security research that a narrowly defined scope often yields better results than broad, unfocused scanning. Anthropic’s property-based testing agent, while generating thousands of candidate reports, relied heavily on ranking algorithms and expert review to refine these into a high-confidence tier, with approximately 86% proving valid. These parallel findings underscore the universal challenge of taming the prolific, yet often noisy, output of AI in security.
Strengths and Limitations: Understanding the "Jagged Frontier" of AI
The application of AI agents in security audits operates on what Stanislav Fort terms a "jagged frontier" – a landscape where a model might excel at one task on a specific codebase, yet fail at seemingly simpler tasks on another. This inconsistency means that the capabilities of AI in security are not uniformly distributed and require continuous, case-by-case validation. The Ethereum Foundation’s team has identified distinct areas where AI agents demonstrate particular aptitude, as well as significant limitations:
| Good at | Misleading at |
|---|---|
| Reading the specification and code concurrently | Generating call chains that appear reachable but are not |
| Stating and rigorously checking a real invariant | "Gaming" the success check, leading to false positives |
| Rapidly drafting a reproducer from a high-level idea | Inflating severity claims to match dramatic write-ups |
| Suggesting a probable root cause prior to manual investigation | Identifying bugs spanning a complex sequence of valid steps |
The most critical limitation lies in the agents’ current struggle with multi-step vulnerabilities. A single agent session is proficient at one-shot reasoning, identifying issues that manifest within a limited context. However, bugs that emerge from a specific, often subtle, sequence of otherwise valid operations – where the order, rather than individual steps, is the flaw – remain challenging. For such complex, stateful bugs, the AI agent’s role shifts from being the primary search tool to an intelligent guide. Its value lies in suggesting plausible sequences of operations that human security researchers can then feed into a dedicated stateful test harness. Used in this complementary fashion, AI can significantly enhance the discovery of the most insidious and often expensive types of bugs, those that evade simpler static or dynamic analysis.
Best Practices for Trustworthy AI Audits
To ensure the integrity and reliability of AI-driven security findings, the Ethereum Foundation has distilled several foundational habits that are critical for maintaining trust:
- Never Trust the Model Alone: All AI-generated claims, regardless of confidence, must be independently verified.
- Automate Reproducer Execution: The process of running and validating reproducer artifacts should be fully automated to ensure consistent, unbiased checks at scale.
- Separate Agent and Human Responsibilities: Clearly define roles, with agents focusing on generation and initial verification, and humans providing expert judgment, context, and final validation.
- Maintain a Centralized Knowledge Base: A running list of known issues, fixed bugs, and rejected candidates is essential to prevent redundant work and improve efficiency.
- Demand Specificity: Insist on clear, unambiguous claims from agents, backed by concrete evidence rather than vague assertions.
These practices, while seemingly straightforward, form the bedrock of a robust AI-augmented security program. They are not novel inventions but rather adaptations of established principles from areas like fuzzing and traditional software testing, applied to the unique challenges posed by generative AI.
Shifting Paradigms: The Evolving Role of Security Researchers
The advent of AI in security auditing has not rendered human security researchers obsolete; instead, it has fundamentally reshaped their role. The "bottleneck" in vulnerability discovery has shifted. Where researchers once spent considerable time formulating and chasing down hypotheses, their efforts are now redirected towards the high-level, critical tasks of judging AI-generated findings at scale. This includes designing and building robust "oracles" (mechanisms for objective verification), orchestrating the complex triage process, managing comprehensive lists of known issues, and handling the intricate details of vulnerability disclosure.
This shift represents a qualitative upgrade in the work of security professionals. It moves them away from repetitive, labor-intensive search tasks and towards higher-order cognitive functions that demand nuanced human judgment, contextual understanding, and strategic thinking. The value proposition is clear: AI agents enable coverage of significantly more ground than human teams could manage manually. In return, they demand a more sophisticated and vigilant human oversight, capable of navigating a vastly increased volume of information, much of it presented with deceptive confidence.
The speed at which AI capabilities are advancing, as observed by researchers like Nicholas Carlini, suggests that the "generation side" of AI will continue to climb exponentially. If the "judgment side" – the human capacity for verification and triage – does not keep pace, the gap between what AI produces and what can be reliably verified will widen, potentially leading to a new class of security risks where unvalidated AI claims are mistakenly accepted. For critical infrastructure like Ethereum, this balance is paramount. AI agents offer an unprecedented opportunity to enhance network resilience, but only if the human element of careful judgment remains the ultimate product and gatekeeper. The tools are new and rapidly evolving, but the enduring principles of rigorous validation, objective oracles, and meticulous triage remain the immutable foundation of trustworthy security.
