Money Legos in 2025 Securely Integrating DeFi Projects Using AI | Michael Boyle - Quantstamp

[Music] uh my name is Michael Bole I'm a security researcher and auditor at Quan Stam uh web 3 security company and today we're going to be talking about money Legos in 2025 securely integrating dii projects using AI so why are we doing this talk or this is what we're going to cover today first we're going to talk about why we're doing this talk we're going to talk about the current state of integration documentation we're going to look at solutions to improve the current state of the Integrations um and then as well as a proof of concept looking at using llms to help in developers integrate into more protocols so first things first we need to cover defi composability we can think of defi as a bunch of money Lego blocks where each aspect of defi can build on top of each other so in this chart here we have everything on top integrates with the thing beneath it so if we go all the way to the bottom we have an erc20 token called YOLO and then on top of that we have oracles that then pull the price for this YOLO token then we have maybe liquid staking projects which then use those oracles uh and then maybe lending protocols on top of that yield aggregators on top of that so on and so forth and so as we go up the stack we have more and more Integrations which means that we have more risks more consequences and more Trust assump asss that we have to be careful about during the development process so in this example say the erc20 token YOLO is pausable and upgradeable this would mean that every single one of those protocols above them needs to be able to account for the fact that their token might not be transferable at some point it might be upgraded to do something else at some point and because of that they need to be proactive decisions that are made during the dev vment process to be able to handle any situation which can happen for protocols that are you are integrating with so another example say this Oracle has an issue now it has a stale price that would this could potentially affect everything that integrates with it above it so in this case everything on the left side of the call of the chart we're not going to have to worry about it doesn't integrate with that directly and so it's not an issue but we can see that even one thing going wrong part way up the stack is still going to affect a of protocols and this needs to be handled correctly so here we have an example of two protocols that in some ways or where protocol one integrates with protocol 2 uh we're going to say that a user deposits into protocol one and then protocol one does some kind of yield aggregation and then deposits the user funds into protocol 2 and then we'll say for some unknown reason protocol 2 needs to pause uh maybe they found an issue and they needed to fix it or something happened with protocol 2 2 which then creates a unrecoverable denial of service in protocol one so now a user from the users's perspective they see that I deposited into protocol one which interacts with protocol 2 but either way I can't access my funds out of protocol one because of its integration with protocol 2 and it didn't handle it correctly so now not only is protocol one going to be at fault for having a bug in their code but protocol 2 is also going to get some of the blame and have a reputation hit because people didn't correctly integrate with them and so that's mostly what we're going to be looking at today is how to make sure that if that your dii money Lego has the proper considerations for how others should integrate with you um so obviously we're coming back to a topic that grew rapidly in popularity in around 2020 um so we want to come back to it for a few reasons mainly uh we wanted to look at the use of AI and llms to solve help solve this problem uh also security knowledge in general has just become well more well known the space is more mature and so thing the documentation has gotten a lot better um yeah and so since the ecosystems upgraded we want to come back to it and see if we can help out developers streamline this process even more and prevent integration issues going forward so the first thing we did was we wanted to look at the current state of integration documentation and so we have an example here where we're looking at a c category of uh libraries so these are low-level development tools that most solidity developers have used at some point in their lives um we're looking at like open Zeppelin libraries or the soul lady libraries and we can see for both of them that they use documentation that is specific to developers that are going to be using these libraries so they have warnings specific to some of the functions that they may use um and they they indicate proper usage as well as uh common pitf pitfalls that people might run into and so these are very very developer oriented but no user of defi protocols is going to be reading these next we have oracles which again are a very a bedrack Bedrock Foundation of many dii protocols and we can look through all of we look through all of these oracles and we can see both user and Dev focused documentation um which is good because we want to see things where if they can be used by a user it needs to have user facing document but we also need developer and security Focus documentation uh for any di money Lego block finally we looked at the documentation for another very common uh Lego brick which would be lending protocols uh so we looked at a few here like a just Moro and our overall impression of these was that the developer docs were actually lacking and we mostly see user facing documentation assuming that nothing is going to be built on top of these which as we've seen from many yield aggregators and swapping protocols and all kinds of stuff it's just not the case these lending Protocols are used as a base for other projects and therefore should have security protocols and security considerations that are Developer facing and security oriented to give developers the best chance to integrate with these protocols without having a hack so just imagine that something that integrates with one of these lending protocols has an issue and someone sees that they're at assets are technically sitting in a right now but they're completely locked because the pro the project that integrated with a didn't integrate correctly now the user is going to have a bad perception of not only the project that they devel uh deposited into but also a and so it's best practice for any thing in defi to give integration recommendations at all levels um so we want to talk about the importance of composability guidel lines and very specific use cases and how we want to have um like what what do we want the documentation to look like when we're integrating with a project so these are some of the main things that we came up with we need to have an exhaustive and correct and actionable list as possible we need to have every action that can be taken by a user or an integration we need to have it listed we need to have it cons security considerations um we need to know things like is the the contract going to be upgradeable or paused if someone integrates with your contract that is pausable or upgradeable there's a very good chance that if you ever actually do pause or upgrade the contract that you're going to break Integrations and could potentially ruin something else down the line um basically anything that you use as a configuration parameter that is controlled by admins that is not controlled by a typical user can change the interactions with uh Integrations and so therefore they need to be accounted for um again we need clear warnings we need um clear and common pitfalls pitfalls recorded and they need to be easily searchable and llm friendly so in this case that means that we need uh we need sections that are very specifically user focused very specifically security focused and very specifically developer focused so here's kind of a sit situation of how it would look like when an auditor needs to needs to audit a protocol that has Integrations so in this example we have both contracts A and B and we can say that A and B both have different Builders and different Auditors if contract B was built first and then contract a uses contract B if I am an auditor for contract a then I'm hoping that not only did auditor B do a good job but I also need Builder B to have good inte integration documentation so that I can quickly reference how contract a should be correctly imple uh integrating with contract B and how um and how my contract can avoid any of the common pitfalls that might um have come up from integrating with contract B I also unfortunately can't take auditor B's word for it or even the documentation word for it so these are all just general guidelines to get things going but at the end of the day we also need to manually review the code of contract B in order to be certain that contract a is correctly integrating with anything beneath it so now we're going to go into a proof of concept where we kind of talk about using llms to help generate some of these security checkpoint list that we're looking at so we're going to use a project that is a standard lending project and it integrates with chain link as its Oracle um we anonymize the contract it is from one of our actual audits uh but we wanted to keep everything um private and Anonymous so the first part of this um check was we used an llm query with the public integration checklist provided by chain link so we're basically going to take the documentation straight from chain link we're going to take the contract code that we've uh audited and we're going to give them both to an llm and see can it give us security recommendations based off of just the documentation that chain link has uh and then second we're going to give um our own internal auditing checklist and see how that does as well so there's a lot going on here I don't expect you to read this all this is just U kind of a overview of what we actually gave to the llm is context so that it knows um obviously what the project does and how it integrates with uh chain link as well as some of its admin control or um functionality that might affect Integrations so here's what our prompt actually looked like uh we say that for the smart contract above is there any violation of any of the practices described on this page and then we have a link to the chain link documentation about using chain link oracles so what we get back from this is actually pretty uh promising we see that we get concerns about data freshness verification we don't want any stale data coming from uh from the chain link Oracle and so we need to manually validate that Within the contract we see that it brings up the issue of single Oracle dependency so any protocol that uses oracles and needs data at all times should probably have some kind of data availability where we're using multiple oracles uh we look at Price validation mechan mechanisms so sometimes when an oracle has stale or outdated prices uh we could get something that is way off from what's expected or it'll even return zero and we need to correctly handle that situation so that we're not uh adversely affecting users uh we also see that it points out that there are administrative controls which could affect how the Oracle uh how it's receiving information from the Oracle uh as well as decimal Precision handling since not all tokens not all data that is sent on chain is going to have the same number of decimals that we use on chain to represent the data so then our second uh case study here was to use uh internal checklist that we've developed since we've audited so many different protocols and seen so many different dii and Oracle Integrations we have uh plenty exhaustive lists of uh Integrations or for Integrations like this and so we Supply we use our background knowledge of all of our audits to create our own checklist for things that we would look at when a project is integrating with an Oracle and we want to see how this compares to chain Link's actual documentation so again we supplied our checklist along with the code and this is what we got this time um we get more of a question question based answer where it's saying we need to or it's asking kind of repeating the questions that we're asking in the checklist and then trying to answer it for us um so we see it ask things like is the Oracle data up to dat is the Oracle single Source a failure like we talked about before is the Oracle price hardcoded uh is the price feed for a similar asset used to represent another asset which could be similar to say the decimal issue um and then any other or I guess the decimal issue is the next one but any other considerations that are similar to to the previous one so our takea away from this was that you kind of need both um you can't always rely on one protocol to give you a good enough security guidelines that you can just feed it into an llm along with the contract and assume that you actually are going to have all of the information that you want to need for the audit uh you also um your list if you're an auditor even if you've been doing it for a long time is not always going to be completely specific to the protocol that you are auditing so just because it has a chain link integration doesn't mean that it doesn't do something unique or different with it than you've seen um in previous audits so ultimately we always recommend using both and then of course a manual audit to cover in those gaps where you might have unique scenarios so the conclusions are that we have defi security composability is an ecosystem uh level topic we would love to see more protocols create composability guidelines that are security and auditing focused and we want to reduce hacks um for many vendors it's hard to find good security guidelines that are readily available and developer oriented and I think a lot of protocols Overlook the fact that they are adding to the defi money Lego stack they're not necessarily going to be the last brick on the stack even if they are at the time when they create it so maybe right after you publish uh your protocol your next step would be to consider how someone might integrate with your protocol or aggregate your protocol and create a best practices list for others to use u in order to prevent your reputation from being on the line if someone integrates with your protocol incorrectly uh these are some additional resources if you want to take a look and uh thank you for listening I think we have a few extra minutes if we have any questions [Applause] looks like we have a question I'm going to have to ask you to yell at at me and then I'll repeat it so that is an excellent question he asked how do we make sure that the llm is not flawed obviously llms especially at this stage are known to hallucinate and so this is why we still have to recommend that you do a manual audit in in in conjunction with using the llms it's going to give you a good Baseline it's going to give you guidelines to start off of but ultimately you are going to have to double check it and actually do the work yourself it's just going to help hopefully remind you and help kind of create the checklist to start anyone else guess not thank you but