How Smart Contract Security is Changing with AI with @sherlockdefi

All right. Hey everyone, my name is Dan Johnson and I'm on the protocol security team here at Coinbase. I'm joined here today with Jack Sanford, the CEO of Sherlock.xyz, a web3 security company that's been working hard to secure onchain systems since 2021. Today, we're serving up our 10th episode of our ongoing Coinbase security series, where every two weeks, our team gets to connect with great builders in the web 3 community and discuss a topic of blockchain security and how to build safely in the space.

As is the case with each of these sessions, please note the disclaimer that's at the bottom of the screen. And with that, we've got a great session for you today. So, I'll let Jack take it away.

Awesome. Thanks so much, Dan. Uh yeah, really excited to be with you guys here today and talk about a really dynamic and exciting topic in web3 security, which is how AI is kind of influencing it and playing a role. Um to give a little bit of background on myself and Sherlock, I'm the CEO and co-founder of Sherlock. Sherlock has been around since 2021.

We do a lot of different security products. We have smart contract insurance. We do audit contests. We do traditional audits. We host a bug bounty platform.

Um, and we work with, you know, AI PhDs in house, some of the top researchers in the world in-house and externally. And so we've had a really interesting vantage point, I think, on seeing how AI has kind of played into a lot of the different aspects of security. Um, and it's really changed a lot over time. We've seen even since 2022, 2023 LLMs um playing a role in bug bounty submissions, audit contest submissions. We've been experimenting with AI since about early 2023 in production in some of the audit contest judging we do.

And so I'm really excited today to give a little bit of an overview of where we've been, where we are today with AI and where we're going to be in the future. So that if you're a developer, if you're an auditor, you know, you can have a somewhat realistic view of what's going on, maybe what tools you should be using as a developer or auditor or or what you should be looking into. And then also what's realistic. You know, if you talk to a an audit firm that tells you they can find all the vulnerabilities in your codebase in 15 minutes with AI, you know, this presentation can hopefully help you get a little bit of context on something like that. So yeah, getting right into it here.

We're going to talk about where Web3 security is today. And then we're going to talk about three specific aspects. Development. So as a developer, how we've seen people using AI in auditing, how we've seen auditors using AI, and then the AI auditing products. Um, so think of it like automated vulnerability finders and how effective those are, what the stages, uh, what stage those are at, what kind of projects are using them right now, and where we think they're going in the future.

Um so really excited to get into this today and hopefully uh it's helpful. So where web3 security stands today um in the past obviously there wasn't a lot of AI going on. There was maybe some machine learning happening in the you know 2015s 2017s but pretty much all of web 3 security until the last year or two was very manual point in time. Basically, if you're an auditor, that means staring at a codebase, understanding it, slowly loading all of the context into your head, um, and trying to think up vulnerabilities on your own, essentially not using any AI. Um, and as a developer, I think this has been even more true where especially with Solidity, you know, 2 years ago, nobody was really using AI to help them write Solidity.

That's changing a little bit now. We'll talk a little bit about that. Um, and so where we are today, we're definitely seeing AI being really useful in certain areas. So I think the the approach for a lot of developers, a lot of auditors has changed from purely manual, no AI usage to something now where AI is a really integral part of their workflow. Um, and this is especially on the auditing side.

So I'm excited to talk a little bit about that and go in deeper detail there. And then in the future, to be honest, no one knows what's going to happen. Um, you know, I'm not going to pretend like I can predict the future in terms of how fast AI is going to come and improve, but I think there are some, you know, pretty obvious things in terms of extrapolating what we're seeing now, um, and how that will affect not only development, but auditing and shipping products to mainet. So, yeah, really looking forward to it. Um, and this quote on the side, Bernhard Mueller, one of the top security researchers in the space, you know, he's been working with models.

I think he built the Hound, which was one of the first uh open-source AI auditing products, and he's seen a lot of really good benefits um through AI as he's looked at the different models as they release. And we just had a new model release a couple of days ago, and every time that happens, it seems like the capabilities and the use cases of AI increases a little bit in the in the auditing space especially. Cool. So past, present, and future, 2015 to 2020, auditing and smart contracts still, you know, somewhat new, especially in the the the the beginning half of that time period. Bug bounty programs starting to take off and so there was no AI really being used, no automated tools being used.

Even the developer tools were were pretty uh minimal on the Solidity side. Then we get to 2021, 2022, we have static analysis tools. We have we have Mithil, we have Akidna, some of these tools that are that are really uh a slither. some of these tools that really add a lot of value. Um, especially when it comes to finding bugs that are not that difficult to find, but having broad coverage is something that the static analysis tools do really well and flagging small things that may not be a critical vulnerability, but could be something that you could string into a critical vulnerability with enough of those small bugs.

Um and then 2023 came and we had kind of the first really useful LLM models at least for writing traditional software uh JavaScript that sort of thing and it was fairly useful for auditing in uh solidity as well. Uh this is the point at which Sherlock and some other audit contest platforms started using AI to be a first pass for judging submissions. So if there are really lowquality submissions or if there are submissions that are the same, so duplicate submissions, this was a really uh useful thing that LLMs could actually do quite well as a first pass and take a lot of time away from human judges so they don't have to go through hundreds of vulnerabilities. Then we get to 2024, we see some of these early AI uh automated AI products, automated AI auditing products entering the market and some early use cases for those. And now in 2025, we have definitely seen some of the major web 3 teams making not only AI and their development flow, but also those AI auditing products part of their an integral part of their workflow.

So that's been a new development over the last few months. Um, and we'll see kind of where things go from there. So in terms of the web 3 security life cycle, everybody has a slightly different version of what exactly the life cycle is and what the right steps are to get to mainet to develop code properly to you know audit the code so that you have the least chance of having an exploit on mainet. Um this is you know something that Sherlock we we reference a lot internally three stages. So that development stage when you are designing your protocol you're starting to implement the protocol in in solidity and some of the things you can do in that stage especially now security consultation so that's working with human auditors and then AI tooling we're seeing this happen a lot lot more which is using some of those automated AI auditing products to find vulnerabilities that are lowhanging fruit so kind of the same use case you might have for static analysis tools AI tooling is coming in being really helpful there so so that you can have a much better setup going to phase two, which is that audit phase.

And this is, you know, the infamous phase where you spend a lot of money, you spend a lot of time, and you really hope that the auditors you're working with are going to find any and all of the vulnerabilities in your codebase. This looks like collaborative audits where you have a small team of auditors, think three or four auditors working together, giving you that audit report of what they find or audit contests. Uh we just saw Ethereum do a $2 million contest for their Fusaka upgrade. And this is another great way to get hundreds of people looking at your codebase. So those are kind of the two main auditing products that we see today.

And then post-launch getting the bug bounty set up. Really, really important piece. Um making sure that white hat hackers have a way to tell you about vulnerabilities. They have a channel that they can go to so that it's not just the black hats who are trying to steal your TVL. Um and then of course having some kind of monitoring solution in place so that you can know when a hack is at least happening.

uh you know I've seen teams where they got hacked for five days in a row and they just didn't know uh they didn't have monitoring set up. So having monitoring set up really great and if you go deep on the monitoring side you can even have threat prevention where your smart contracts are set up to pause or stop transactions when you see something suspicious going on. Um and of course there are deeper pieces in all of these. You can be doing formal verification on the auditing side and variant testing but this is the general life cycle. And in terms of where AI fits into this, we basically see AI fitting into every stage of this.

Like we said before, AI tooling and the development stage, this is becoming much more well-known. I would say in 2025, it was much less common in 2024. So AI is playing a big role in helping not only developers to think of the right way to implement their protocol and to help them see edge cases but using AI auditing products to actually find vulnerabilities before you go to the expensive and timeconuming human audits. You get to those human audits well guess what all those human auditors are using AI extensively now. So, we'll talk about exactly what they're doing and how they're increasing their workflows so you can have an idea of, you know, what the state-of-the-art looks like there and what you can expect when you work with some auditors in terms of how much AI they might be using to find vulnerabilities in your codebase.

And then, of course, on the bug bounty side of things, um there's going to be a huge tsunami of AI submissions to bug bounty programs. And so using AI to combat that, using other uh incentive mechanisms to combat that is already happening and really important. As well as on the monitoring side, being able to understand new exploits as they're happening, have smarter um filtering or or triggering on the monitoring side to know when an exploit is occurring. Um so we already see AI really playing a huge role in all of these areas. So let's talk about AI and development.

You're a smart contract developer. How are you using AI? How should you be using AI potentially? We talked to a handful of developers who we are really close with and asked them basically, hey, what are you using it for right now? What's realistic?

Are you just vibe coding all of your Solidity smart contracts or are you not using AI at all? Um, and so what we've found is people are using AI and they're using a lot of different products and things. It's it's very um you know uh uh not aggregated but it's it's very diverse in terms of how teams are using AI and individual developers really are choosing a lot in terms of what they want to be using. So in that first stage of developing a a protocol you're using AI to understand what what's already out there. So what protocols exist you know what ideas you can take from what what competitors what implementations there are.

Um and you're also using it to generate MVPs. So like hey can this work you know can we get this thing working? Can we get yield flowing through in the way that we want to? Um is there a better way to be doing that potentially that can change our architecture. So we're seeing a lot of that happening in the early stage and then and and people are using you know just classic chat GPT the most expensive chat GPT subscription you can afford using cloud using some of these other tools just to talk um to a base model about your codebase and and understand what's going on there.

Um then we get to the coding and implementation stage and this is the interesting stage because there's a lot of controversy in this stage from the developers we've talked to. Um, a lot of them don't really believe that AI is useful, especially on the Solidity side yet. Um, they've seen a lot of kind of wrong implementations, not knowing how, you know, libraries and imports and different things like that with the Solidity language work. Uh, so definitely be careful here. We have seen some developers using it to um for gas efficiency, like trying to help them find ways to to reduce gas.

Um we've also seen you know everybody uses cursor to some extent for code completion although again very controversial with solidity. It's not quite at the stage that maybe JavaScript is um on the web two side of things. And then even some developers who are using um you know something like lovable to create a really basic front end and just get that going so that they can start you know showing something to their their designer or showing something to uh to their users. Um, so that's a lot of what's going on in the coding and implementation stage. We'll talk a little bit more about that pretty soon here.

And then testing and pre- auditing. Like I said, there's actually been um it sounds like there's a lot of usefulness uh in terms of the AI here. It can write some pretty decent automated tests in solidity and foundry and that sort of thing. It's not perfect but the everybody knows writing unit tests in solidity even though it's gotten a lot better in the last few years still very timeconuming to think up all the edge cases to write all the tests properly and so AI is very helpful there and then using that those AI um automated uh vulnerability finding products before you go to human auditors uh is also something that's becoming much more of a trend and we're seeing um a lot of actually effective vulnerabilities being found in that area. So how what is the value that developers are getting from AI in the ideation stage?

Again, it's finding, you know, it's it's being that kind of thought buddy with you and thinking about, okay, what's already out there? You know, how can I quickly just write something up that can potentially work, hopefully it can work if it's in solidity. Um, it definitely will not be secure or at least check it. Um but we've definitely seen some usefulness especially with people who are doing hackathons or doing really early stage work on the protocol side. They're able to to use um AI pretty effectively in the ideation and planning stage.

Coding and implementation not as strong on this stage for sure. We've definitely seen a lot of people saying that it's not really able to write solidity in a way that is helpful um or even that compiles. But in some cases, you're able to, you know, get um more gas efficient code and you're able to write code a little bit faster. Although again, you will have to spend a lot of money auditing that code. So, a lot of teams are not quite using AI heavily in the coding and implementation stage at this point, but that will potentially change over time as AI gets better at solidity.

And then testing and pre- auditing probably the most useful stage or at least the most adoption we've seen in terms of AI. So writing unit tests and then using some of those AI auditing products or just using you know uh chat GPT or cloud and asking it about certain areas of the codebase asking it to find vulnerabilities. Sometimes you can get results with that as well. Um and it's you know potentially cheaper than using one of these automated AI products. So yeah in terms of could you actually build a a Solidity application on a blockchain with AI?

The answer seems to be yes. um you can do it but it's going to be a little bit ugly and you know would not want any users to be using that at least without a lot of developer oversight and auditing but it is possible to create a solidity DAP using only AI and so this would be an example flow of kind of how you do it I mean using cursor okay maybe that's cheating if you're actually writing some of the code yourself and using cursor to autocomplete um but it is possible to write apps and AI But from what I can tell from talking to developers would not recommend it. Um so yeah the dangers of overusing AI in development. So in terms of this we talked about a lot of these you definitely it will write more code than is necessary. So in terms of using AI to generate full smart contracts we see a lot of functionality that isn't necessary that isn't part of the spec that isn't something that is useful to have in the codebase.

Um, and then of course lack of web 3 specific understanding. Things like the code doesn't even compile or it's not even understanding that it needs to import this this smart contract in order to do what it needs to do. Um, and of course there's even more things uh in the unknown in terms of what's going to happen when people start deploying AI generated smart contracts at scale and getting TVL in them. We're not even sure what that's going to look like. But from what we can tell, a lot of the top teams that Sherlock works with, they're not quite at that stage.

They're actually not even close to that stage. They're very much using probably 90% human effort in terms of generating solidity code and maybe 10% uh in terms of AI. So that's a little bit different than the auditing side of things. Um on the auditing side, we definitely see specific areas of auditing where AI is useful. You might even say it's necessary um or at least saves a ton of time, but overall finding vulnerabilities with AI still something that is being proven out, we'll say in 2025.

Um so how auditors are using AI, this is to be clear, this is about human auditors and how they're using AI, not necessarily the auditing products. We'll talk about those next. But in terms of an auditor that you may work with, if you're a developer and you want to know, hey, how are these auditors going to use AI? are they just going to send me a report that's purely AI generated? Um the the great auditors are not doing that.

Not even close to doing that. Some of the auditors will use it to help them get context on a codebase. I've seen a lot of really positive things in this area in terms of security researchers saying that. But there are also some who say you actually shouldn't do that and you should get the context yourself and you should not trust the AI because the AI is going to potentially gloss over things or or be an abstraction. Uh you know the map is not the territory type of thing.

So relying on the AI can be a little bit tricky there. Documentation. This is almost uh a consensus in terms of being really helpful. So one thing that auditors don't like doing is writing reports. If you're writing reports, you're not finding bugs and you're not creating value for the customer.

So writing reports um using AI to do that or at least to create rough drafts and that sort of thing. We've seen that be really helpful. And I think that's where dozens of hours are being saved right now for auditors writing PC's. this one a little bit, you know, it works pretty well, but not perfectly. Um, so a lot of auditors have seen that it can write PC's and it can do that helpfully, but you have to check the PC's and it can put a lot of random stuff in there and make you think that it's working or make you think the hack is real when it's really not real.

So, you have to be careful there. Um, in terms of idea generation, this is another one that I would say is probably one of the biggest use cases of of AI for security researchers at the moment is, okay, I just spent 7 days on a 10-day contest. I've gone through the codebase three times. I feel pretty comfortable in terms of the bugs that I found and I can't think of any more ways to break the contracts. Talk to an AI, ask it to come up with new ways, you know, go down some rabbit holes with it.

And this apparently has been very useful in terms of being able to find that next incremental bug on day, you know, when you're 80% done with the contest. So, that part is really cool. And then judging validation, um, you can help it. I wouldn't say to positively validate bugs, but maybe negatively um, like disproving bugs. Uh, sounds like it can be helpful in terms of, hey, here's this vulnerability that I just wrote up.

I'm going to send it to the team. Are there any reasons why this might not be valid? and the AI can, you know, be somewhat helpful there in terms of coming up with creative reasons. So, in terms of the the auditor's perspective, uh we've talked to a ton of auditors about this. These are some Twitter posts that that we've taken um that kind of talk through each of those four stages and what people think about them, how they're using them.

Um so, yeah, I think it's really interesting and um we can take a a look at that later if you like to. Um where AI enters the audit process. So in terms of how an auditor actually does an audit. So onboarding this is where you are trying to understand the codebase form that mental model and AI can be really helpful with summarizing at a high level like hey this is what the codebase does um or this is what a specific piece of the codebase does that may not be familiar to you um and this is something that contra a little bit controversial but been helpful in terms of doing that deep review. Um, it's not that helpful for the first 80 or 90% of deep review it sounds like, but once you get to that 80 or 90%, you can't think of more attack cases.

It's very creative. So, it can come up with new exploit paths and then you can manually decide if you like those paths or not. Um, in terms of issue confirmation, so coming up with a potential exploit, this is where it can be somewhat helpful generating PC's um, and finding potential edge cases that you didn't think about in your issue. Uh that's also helpful in the final QA uh section where you are having the AI kind of disprove your issues um and and be like a devil's advocate for you. And then of course report drafting which um is has been a very helpful uh piece.

So in terms of limits, we've talked about a few of them. It's not great at finding vulnerabilities in a way that would be competitive with the best auditor in in an audit contest. Um, a lot of these auditors are still relying on themselves, you know, 80% manual work and then using the AI around the edges to be helpful there. Um, but in terms of where it's really helpful, it's really helpful in summarizing code bases. It's really helpful in validating issues, especially disproving issues.

And then a lot of researchers are using it to generate issue reports, which hopefully they are uh editing themselves as well. So it's not just a purely AI drafted report. uh but in terms of creating a report that is using the English language well and and doing um 80% of the work it's been uh very useful in terms of saving hours there. So yeah, here are some other quotes in terms of how auditors have been using the AI. Okay, in the interest of time moving to the AI audit product section.

So we talked about how developers are using AI, we talked about how auditors are using AI and now we can talk about what these kind of automated AI auditing products are doing in the space. So, in terms of are they able to find real vulnerabilities, 2024 it was highly questionable. 2025 it's a resounding yes. Uh with caveats, false positives are always a question here. Uh depending on the tool, you can get maybe like 50% of issues being false positives.

I think Sherlock AI's hit that a few times. um and and that's generally pretty good and then you have you know 90 95% false positives um from some tools or some code bases and that can be pretty not useful uh to protocol teams because you have to spend hours and hours going through all those issues who uses them. So some of the top teams in the space are using these AI uh automation tools whether it's Sherlock AI or whether it's another um another tool and so this is definitely something that is getting more and more mainstream like 2025 is kind of the breakout year I would say for this and we're seeing a lot of teams a lot of ecosystems really uh pushing hard on this and and using it in their development flows languages purely solidity right now um we've seen a couple implementations of Rust go whether it's Rust smart contracts and and um actually building blockchain code, you know, node code and that sort of thing. Not a lot of great results outside of Solidity smart contracts so far. Um and then how are they used?

Pretty much full life cycle. So you want to be using it during development before you do the audits. During the audit phase, you're probably going to be relying on your auditors to use AI and to combine it with their brain to do well there. And then post launch. This is something that's becoming much more popular especially after some of these attacks like the balancer hack where it's like man this is something where the code was live for 5 years like what can we do to make sure that that kind of code is is secure like is it is it a bug bounty thing is it more auditing costs you know hundreds of thousands of dollars more spent on auditing every year um we think AI can be helpful here especially as new vulnerabilities come to light AI quickly pick up on those and then can apply apply that heristic to your codebase in a way that you know it costs you very little and it's running continuously or or at least a couple times a month so that you can see if anything new is coming up.

So that's a little bit of the broad overview in terms of how AI fits into a a protocol team's life cycle. Um, so this is uh an example protocol uh similar to a protocol that we've worked with directly where during development you can see they are using the AI to find things in individual PRs as they're developing. Then before they go to the audit phase, they're using it to scan their entire codebase and finding real vulnerabilities like real critical vulnerabilities during this phase and they're able to fix those so that when they get to the collaborative audit in stage two, the audit goes more smoothly and you know there's no cost overruns. There's no giant fix review because the AI has found at least some of those vulnerabilities. As you can see, there were more vulnerabilities that only the human auditors could find.

And then you move on to the system runs after that. Maybe it's a fixed review, maybe it's just the the full codebase review. Looks like another additional high was found here. Then you go to an audit contest. And by this stage, you're really hoping that you found everything.

You know, if if the AI is doing a good job, if the auditors are doing a good job, if the developers are doing a good job, then hopefully you are in a position where the audit contest is ideally not finding much. Although this can be a stress test on its own because you've got hundreds of auditors there. Then you launch, you get a bug bounty set up, and then you're using the AI to essentially run because the AI is getting better every month and new exploits are being discovered every month. So maybe not quite every month, but new exploits are being discovered. And so it can be really helpful to use that AI system just to run on the codebase every so often to see if anything comes up.

you know, maybe the AI got better, maybe there's a new vulnerability that got put into the training set and it can potentially save you a $250,000 bug bounty or even worse, you know, a multi-million dollar exploit. So, how AI auditing products work. Um, I'll gloss over this a little bit. I know we're we're running on time here. So, this is one way that it works.

You know, this is very much um Sherlock AI and some of the others that we've talked to. There's lots of ways of doing this though, but generally you're going to be connecting your repo to the AI. You're going to be triggering runs either manually or you're going to have it based on a a PR setup. So every PR that you that you push, um, some the AI will be running on it automatically and telling you if anything is found. Um, and then you're going to basically take a look at the issues.

It's going to look a lot like a traditional audit report. And then you can see all the issues found over all these different PR runs and kind of understand how the codebase has evolved over time, especially from a security standpoint. Um, it's going to produce the same report details that you would expect with a human auditor. Um, and in terms of how the audit system actually works. Uh, you know, this can be a little bit different from from product to product, but generally it's going to be taking in your codebase.

It's going to be taking in hopefully some of the documentation. Obviously, it's going to be trained on any similar code bases, uh, codebases that are similar to your codebase. It will run static analysis tools potentially or something like a static analysis tool, compiling the codebase, and then it will the the meat of it will be the LLM security agents. So, this will be different prompts that are going through and digging deeper and deeper uh in almost like a recursive way to say, okay, is this an interesting path? Let's continue down this path further and try to find attack vectors and then have that judging agent at the end at the end which will basically try to figure out if the vulnerability is real taking into account the whole context of the system.

So, uh yeah, I'm going to skip over this slide a little bit. This is just specialized AI audit products versus general purpose LLMs. You can find a lot of vulnerabilities with general purpose LLMs, but it seems that you can find a lot more with these specialized AI audit products. If you're tied on budget or, you know, you just want a really cursory overview, the general purpose LLMs can be helpful, especially if you give it a specific piece of the code to look at. But if you're like, "Hey, we're about to spend $50,000 or $100,000 on an audit."

Getting a specialized AI audit product and having it really go deep and find as much as possible um can be really useful here. Okay, so in terms of where we're at, we've talked about developers, how they're using AI. We've talked about auditors and how they're using AI, and we've talked about some of these automated AI products. So, how is this, you know, going to change over the next couple of years? Um, I guess before I get to that, let's talk about the connected security life cycle.

Basically, you want to be using AI in all of these phases somewhere. So if you're a developer essentially using AI to come up with test cases for your codebase using AI in the ideation phase if you're in that audit phase either you using automated tools or your auditors are going to use AI to help them come up with new attack paths and creative exploits and then post-launch whether it's using a bug bounty um product that integrates AI to dduplicate or to um filter triage or just using AI continuously over the course of a few months can be really helpful and we've seen teams using this full life cycle essentially. Um and those are the early adopters in 2025. Um but we see this trend only increasing from here. So in terms of what's next again really hard to know right now in 2025 it's more of a nice to have in a lot of these phases.

Uh some of the early adopter teams are using it some of the early adopter teams are seeing value from it. Um, but in terms of being a developer, it's kind of just something that's fun to experiment with. Terms of a researcher, you're going to use it for some of the manual tasks and then the automated AI is going to find some things. Um, but it's definitely not at a stage where it's better than a really great human audit. Once we get to 2026 next year, um, potentially on the developer side, you could be thinking of it as, okay, it's as good as at Solidity as it is at JavaScript, you know, this year or maybe last year.

um which is much more useful than it is for solidity right now. Terms of researchers, we think it's going to be something that is much more like a sidekick that you really want to be talking to it in order to win audit contests in order to find a lot of vulnerabilities in code bases. We think automated AI products will be uh an integral part of the flow by this point as well and have the ability to find some of those broad coverage bugs that human audits can't find. And then once we get to 2027, we think AI is going to be essentially missionritical in some of these phases. Devs need to use it to keep up with development speed.

Researchers need to use it. It's not going to do everything for researchers. It's still going to require a lot of curation, a lot of customization. You're going to have to give it the right context. Um, but it's going to be something that's absolutely critical to doing a top quality audit.

And then on the automated AI side, you're going to want to have one of these on your side, maybe multiple on your side because you can imagine the black hat side of things, you know, the people who you don't want having these tools are going to be having these tools and using them. So, you want some of the good guys fighting on your side and finding those hacks before, you know, you're having big bug bounty payouts or uh or god forbid exploits. So, that's just an idea of what could be possible. Um, and yeah, thank you guys for listening. Hopefully this was helpful to give an overview of what's going on on the AI side of things when it comes to web3 security and it's such an unpredictable space.

So we'll see how things go from here.

Absolutely. Yeah. And that was uh seriously Jack an amazing job. I think uh those those slides are excellent too. I wish our session could be a little bit longer but you know we try to be respectful of everybody's time.

Uh I do want to make sure I have the chance to kind of go through the chat see if there's any questions. I've definitely have a few of my own as well um that I'm curious to hear your response to. So, I'll go ahead and kick that off. Uh maybe we'll we'll have time for maybe two two or three uh and then yeah, we'll we'll wrap after that. But yeah, let's uh I I guess the first thing is, you know, we we've talked a lot about how auditors and developers are using AI, leveraging it now, but you know, every time we think about that, I'm always, I'm sure you are too, thinking about the adversary and uh how they are leveraging AI um and how they're operating, does that change the balance between offense and defense?

Yeah. Yeah, it's a great question. Um it's really hard to know how the adversary is is using AI. I mean the classic adversary you think of in web 3 security that you can put a label on is uh North Korea and some of the groups there and we I don't know if there's evidence that they are investing heavily into building AI auditing products for web 3. I think some of them have been a little discouraged in the past year or two and are potentially you know focused more on fishing and some of these other areas where um it's it's potentially easier to to gain money or steal crypto.

Uh, but you can bet that there are definitely black hats and adversaries who are going to be putting time into this. Obviously going to be using some of the base models in ways that they aren't meant to be used to try to exploit these code bases. So, you know, I think we have an advantage and that we have venture capital money and we have a lot of different teams that are attacking this and it's a very competitive area uh in terms of building a great AI auditing product. Um, but if we don't move quickly and if adoption lags on the protocol side, then I think it's it's very reasonable to expect that, you know, the black hats using AI could potentially do some damage.

Awesome. Yeah. Um, and then I do see another uh in chat here that's that would like to know, what's the most overhyped claim you keep hearing about AI and security?

Yeah, it's a great question. So overhyped claim. Um, you know, I think any there's there's a lot of different kind of marketing strategies and whatnot around finding vulnerabilities and what tools are able to do. Um, you know, we run audit contests and we see a lot of teams boasting about using AI and Sherlock audit contests and that they've, you know, gotten a certain place or gotten whatever. If you look into the details, it's it's often less impressive than uh than than the claim.

So, you know, the the overhyped thing would just be basically saying, "Hey, we find a ton of criticals in the codebase and you don't need to use human audits." Like, you definitely do need to use human audits. We're not at the stage where any AI audit product has even come close to proving that it h it's better than, you know, really top experts doing a human audit. And we'll see if that ever changes. We'll see if that's three years away.

um TBD.

Yeah. Yeah, definitely. It will be a an exciting couple of years ahead uh to see really when that is fully realized. Um yeah. And I guess last question um in terms of you know forward-looking in future generations of auditors and uh yeah just builders in general uh do you think this is there going to be an overreiance on AI?

I mean leveraging it is absolutely necessary as you've pointed out but uh is there you know a risk to creating this generation of auditors that aren't learning the fundamentals because I mean simply they feel that they don't need to.

Yeah that's a great question. I think that is probably the biggest risk on the developer and auditor side is you come into the space you start using the tools and you never learn the fundamentals. So you can't, you know, you can use a lot of AI. Even great auditors and developers, there are some that are using a ton of AI right now. Um, but they have the skills and the fundamentals to be able to course correct it and edit it when it's off.

So I do kind of agree with what what you were uh hinting at which is the biggest risk is going to be people coming into the space and just trusting the AIS entirely um and not having their own fundamental kind of view and knowledge on it.

Yeah, absolutely. I think yeah always important to you know continue learning um you know just keep expanding your you know security mindset if you're a builder in the space no matter the tools that are available and you know frankly that's why we love to uh put on these uh informational educational sessions every two weeks uh here at Coinbase. So, uh, I just once again want to thank you so much for joining us today, Jack. That was, uh, an excellent job. And, uh, yeah, make sure that you follow Build on Bass so that you can stay tuned for our next session coming up.

But thank you very much, Jack. I hope you enjoy the rest of your day. And make sure you thank the rest of the Sherlock team for me as well.

Awesome. Thanks a lot, Dan. Appreciate you having me on.

Yeah, absolutely. My pleasure.