AI in Protocol Security at the Ethereum Foundation, Antoine James | ETHSofia 2025

Everyone uh thank you for having me here and today I'm going to be presenting how we use AI in protocol security. Um but before before we get into the technical details I'm just going to be giving an introduction about myself and uh the team that I work for. So let's just get started. Um, actually this doesn't work. Give me a minute.

No, it doesn't work yet. Oh, is it working now? Perfect. So, uh, I'm Mantoan. I'm 23 years old and I'm a university dropout.

Um prior to working for the Ethereum foundation I was doing web 2 security and uh after that I've done formal verification for a bit uh working for a company based in France uh using language and uh in parallel to that I was also doing web3 security contest on competition platforms such as Sherlock canina uh code arena and stuff like that and so uh who are we and what do we do at the Ethereum Foundation so we're a team of about 10 people. We actually just recruited uh two new members to that team and uh we're also looking for additional members. So if you go on the careers page of the Ethereum Foundation, you'll be able to find I think an AI position right now because we're looking to expand into that field. And things about what we do, we do fuzzing, we do manual reviews and research. So for fuzzing, we do have public tools and we have private tools as well.

Um public tools are public available to everyone and private tools we keep them private for certain reasons. Most of them most of the reasons are we want them to be battle tested internally for about 6 months to 9 months before releasing them out to the public. And the reason for that is we don't want to have people uh finding bugs in Ethereum ecosystem prior to we prior to us. And so if we find them uh and we battle test our fuzzer and then we can release it after six months then we can know for sure that at least 99% of bugs would have been eliminated by then. We also do manual reviews of the EL and CL.

So EL stands for execution layer and CL stands for consensus layer uh clients mainly so such as Basu Nevermind go ethereum ref and for the CL side we also have Lighthouse Prism and stuff like that and we also do reviews of EIPs. So for the next fork of Fusaka we have a few EIPs that will be included in this fork and we had to do some reviews on them. So as well manual review of the changes that have been uh implemented in the clients and we also have a research department. Uh so this is kind of a more uh when we don't have anything specific to work on we can work on whatever we feel like it's interesting uh and that will contribute to the ecosystem. So that includes client diversity.

We have a website that shows the the statistics of clients that being used and their share of the Ethereum ecosystem running on it. We also do ZK and cryptography. Um so this is great but what next? Uh we have AI. So we want to accelerate our security reviews within the ecosystem and for that as you know there's been the rise of LLMs for now a few years with claw leading the the market and chat GPT as well and Gemini and Deepseek as well coming here uh additionally to enhance some specific part of it and I'm going to talk about uh why Gemini for example is useful for specific tasks and why clouds would be useful for somebody from for something else.

Sorry. So I'm just going to be presenting our tools. We have four of them as of now. Uh most of them are public or be going going to be public within the week. Uh we're actually just making a few changes and then we're going to be re releasing them.

Our team lead will also make a tweet this week uh about the uh tool that he has done and it will be available to the public. So the first tool that we have is a spake verifier. uh we had an intern for about a few months at the Ethereum foundation and he has done this as part of his internship. So um the goal of it is to have an equivalent verification. Basically we have specification and we want to make sure that the clients are correctly implementing the specification within the client and that um to make sure that this is correctly implemented we go through different uh methods of verification.

So we have the deterministic and pro probabilistic verification methods. So there's different stages that are uh being done for for this project and this is all done by the intern itself. Uh he has been uh assisted by a senior member of our team for the client side because uh he was lacking knowledge about blockchain and consensus layer in general but he quickly picked up. So that's very great. So the first stage is spec spec in in spec ingestion sorry and we want to parse the specification and load them into a database.

The second stage is client extraction. So we used lighthouse because it's the most used client for the consensus side. Uh it has currently I think a 41% share and so uh we want to do some function extraction from lighthouse and extract all the functions that are being used within the code of this consensus client. The stage C is function matching. So using LLMs again we want to pair uh which functions is corresponding to which specification and the last stage is probabilistic verification.

So using LLM uh we want to ask you know with specific prompts um and detailed prompts is this function matching the specification is it lacking specific edge cases uh and the last feature is advanced features so we can select specific functions mapping database and analysis tools. So just to give you a broad idea and a bit of a scheme uh this is kind of the cycle that this software is using. So first of all injections and it creates uh spec symbols in the database. Then the second space is the second stage is client ingestion as I mentioned. So we want to ingest the client code into the database as well.

The first stage is deterministic matching. So we verified static symbols like constant configurations and stuff like that that are matching the specifications names and we output that as well to the database. And then the last step is LLM matching. So using a variety of LLMs and you can actually choose which LLM you want to use. um it verifies the logic compliance within the function that the uh content of the function is matching the specification and then if if specs are updated then the database are updated as well and the workflow will repeat itself and this is a tool that you can run manually as well if you want to verify a specific function for example but overall we have a a um a cycle that is verifying all clients and starting with lighthouse but we'll deploy with other clients in the future The next the next tool that we have is called e specify and so uh this tool is basically kind of the same as the previous one but this is a manual tool u so the first stages was done by AI and actually a lot of it was done by AI 90% I think um and the idea is to give out specification for the code as well for the developers to uh keep track of changes and make it easier for u third parties and protocol security reviewers as well to verify clients uh and that the specification adhere to what the client has implemented.

So just to show off a few example this is uh especi so it's a command available once you installed the package uh and it lists out the functions that are being used in every um in every fork. So you have for example add flag which is used in alter. You have the new uh add poly polomial cof that is being used in fulu and uh so it just lists out all the specifications that are all the functions that that will be used in the specifications. Uh this is the um this is the the template that we use for uh naming specification and then adding the source. So where is it in the file and the function lines etc.

as well as a reject or a search method to be able to uh find it easily. So you can just control shift f and find it across your project and then the spec as below is the function name and the fork. So just an example of how it is used in teu for example which is a um consensus client for java uh written in java. So we have the function the function max blob per block in electra uh in the previous fork and so it says that it is within the file ethereum spec src main etc to the mainet yl file and so to be able to search that because it is quite a long name is just the name itself we don't have a specific reax for that and the specification mentions that uh the function is max blob per block it is within the Ford electra and it should be u in of nine and if you go within in the file then and search for it then you find obviously uh the line for that. So everything is uh basically written for now I think we have an implementation of that in teu in prism and in lighthouse as well and the idea is yeah to keep uh a track of this and is actually also verified in CI.

So whenever we push any changes if there's a mismatch between the specification that is within the code and the code itself then it will trigger something an error on the CI and then you can adjust the specification. So what is the difference between those two tools? As I mentioned one is used to be one is intended to be manually updated by core devs which is eve specify. So the the baseline the skeleton of it is already written in by AI but then if you want to maintain it uh it requires like manual intervention. uh but it also has better precision rather than the fully automated uh LLM thing that we have discovered privately that has been written by our intern harsh.

So uh the Ethereum code reviewer this is the tool that has been built by our team lead Frederick. Um and the idea behind this is that every time a new PR is pushed we want to make sure that this PR isn't dangerous for the ecosystem. So we have a list of clients that we monitor. Uh and then so we have EL clients and CL clients. And so the reviewer will just review the changes with again different LLMs.

So you have uh one model available or you can also judge the uh choose the multimodel judging option. For photos I'm just going to illustrate the pro the the idea. So this is the landscape that we have when we uh arrive on the web page. So for this I selected the repo blue alloy rvm which is the uh EVM implementation written in rust and that is used by the uh rust ethereum client ref. And so we can see that we have uh two um suspicious PRs that has been detected and so um it is actually by open source contributors.

So obviously people can make mistakes and you know when even when you review it you can also um just miss a few things. Um this is the idea. So when you click on it, you get the repository, the branch, the comment etc. So you can just click on it and view it instantly. On the right side, you can assign it yourself and then also triage if it's a true positive or a false positive and you get a confidence score is been uh built basically by AI and the multi judge analysis is selecting different models.

So we have claude, we have Gemini, we have uh deepseek um chat GPT and we use the latest models for everything. Um this is a description of how it looks like. So basically it says there's a gas error. Uh it used the value 1,200 in the EIP 2537 instead of the 20 uh 12,000 uh mentioned. So it say what's the issue?

So you can obviously like analyze uh the LM just will analyze the EIP and says that this represents a 10x underpricing for the operation and it links that to the correct operation which is BLS. Uh what can happen obviously? Well they can uh mass uh mass compute these transactions because it will cost such minimal fees and how to fix it. It gives the path of the file as well and we need to change the values. uh it actually also gives you a code diff if you want to uh have a look at it and on the right we can see the PR that has made the change uh to just correct this a second example which is kind quite similar um actually we use the value ox something uh instead of 23,000 so we use the decimal instead of the uh the hexodimal instead of instead of the decimal and as you can see uh right below in the code example it says before and after so it gives you like a proper idea on how to correct this uh so these we have a few true positives for now and we keep monitoring this as well uh but is it will be open source this week and the last thing is Evespec Viewer so this is a tool that has been uh built again by one of the seniors uh in our team and the idea is just to review for specification uh it turns out it's been really useful to core devs and as well security guys and the idea is that you have this page and you can search for specific specification filter by fork etc.

and you can look for configuration v variables constant etc that needs to be implemented. So this is really good as well and this is another example. So you have this function that's been uh changed in the electra I suppose uh add valid add validator to registry and so you have the new definition of what should be implemented within the electra fork underneath you can see the alter fork and see the difference for itself and you got all these functions again this is bu this has been built 99% by AI so we just ingested all the code with Gemini because it has a large context window and if you have any questions I'd be happy to to answer I'm just going to give you a few links if you want to have a look at them as well. I don't know if you have access to the presentation, but you should have after the the event. And you can also go on the team link and see what we do and the responsibility link as well.

And for the people, Harsh is the intern that has been interning with us and has built the um spec reviewer tool. Justin is the senior of the team that has been mentoring him and also building two of the tools. And Frederick is our team lead. So, I'd be happy to take any questions if you guys have