Coinbase Security Series Secure Development Lifecycle of Smart Contracts

All right. Hello everyone. Thank you for joining us today. My name is Emra. Alexis and I are blockchain security engineer at Coinbase.

We both work for we we both work on the protocol security team and which is responsible for securing all onchain components at Coinbase. Um I'm excited to announce the launch of a new educational video series. Um as part of this initiative, every two weeks a different security engineer at Coinbase will present a topic related to blockchain security. So before we start today I'll be kicking off the series with a session on the secure development life scale cycle of smart contracts and before we start please notice a disclaimer at the bottom of the screen and to delve deeper into the talk I will hand it to over Alexis.

All right thank you Emra. What's up? Hey everyone, my name is Alexis Williams. I'm a staff blockchain security engineer at Coinbase. And today I'll be kicking off our first security series talk by discussing the smart contract SDLC with you all.

Okay. So, first a quick overview of what I'll be talking about today. So, first I'll introduce the smart contract SDLC and explain why we need one. And then the next points kind of give it away about what's in the SDLC. But for each of the phases that you see here, I will explain what each of these phases entails and give you some helpful tips for securing your smart contracts uh during each of these phases.

So, as you might have already guessed, the smart contract secure development life cycle, not just a software development life cycle, and I'll discuss why the emphasis on security in a second. There four phases, design, development, and testing, deployment, and post- deployment. Viewing a life cycle with these four phases gives us a more holistic view of smart contract development. Additionally, as the diagram on the right shows, not only is it more holistic, but it is also intentionally designed to be a cyclical life cycle. So most smart contracts not one and done.

So understanding that you re the life cycle for an already deployed smart contract is an important distinction that we want to make. Also within each phase, they're going to be cyclical steps. So this idea of a nonlinear development is going to be thematic throughout this presentation. So why do we need a secure development life cycle for smart contracts? I think this is more of a rhetorical question at this point, but just to make sure that we're all on the same page, there have been hundreds and if not thousands of hacks uh on smart contracts for an industry that's still pretty nent.

That's a crazy number of hacks. If you look at the screenshots on the right, which I've taken from Hexgate, who is our monitoring partner at Coinbase, we can see just the smattering of the hacks that have occurred in just this year alone. So, we know that smart contracts have experienced a high number of hacks. But why is that the case? As you probably already know, smart contracts are extremely sensitive to bug, more so than your typical software that we see in web 2.

And that's primarily due to the unique characteristics of smart contracts and of the blockchains that these contracts reside on. By default, smart contracts are immutable. So if you have a bug in your smart contract and you deploy it without any upgradeable abstractions, that bug is going to reside in your deployed smart contract forever. Next, smart contracts often contain critical complex business. Common products at EC and web 3 like yield aggregators, automated market makers, over and under collateralized lending protocols all have core but extremely complex business logic in their smart contracts.

Finally, we have the concept of atomicity with respect to blockchain transactions. If somebody is attacking your smart contracts and extracting value from it, you're most likely not going to know about it until the transaction is processed and all of your funds is stolen. even if you did know about it beforehand, the only option that you really have is to try to pause operations within your smart contract before the transaction lands. So all of these properties together tend to make smart contracts very fragile and susceptible to bugs of higher severity. Additionally, when we think about smart contract development, we tend to think immediately of these three steps.

Develop, audit. If we break that down even for what we tend to really be saying is develop all of your code in one shot, desperately try to find all the bugs and then cross your fingers and watch the protocol. But if find all of the bugs is just a simple a single step in a single point in time, all that really does is drastically reduce the window of opportunity for finding bugs in smart contracts. So really the point of this cure development life cycle is to provide as many opportunities as possible to identify bugs in your smart contracts and minimize the impact in the eventuality that bugs are found in live contracts. So with that let's move on to the first phase design.

The design phase is one of the easiest to overlook but also simultaneously the phase that can drastically change the security outcome of later phases. So why is that? Well, a bad or systemically flawed design for a smart contract will most likely eventually result in the smart contract being unable to operate safely when it is deployed. And most of the time, if these critical flaws are not found during the design, then they often end up being found after the development and testing phase and right before the protocol wants to deploy their contract. So having this critical flaw stick around for long comes at the high cost of potentially having to rearchitect your system much later in the life cycle which can be very costly.

This is also a good time to talk about the diagram on the right which you'll be seeing quite often throughout this presentation. This diagram exemplifies what I just said. As you traverse the different phases of the secure development life cycle, the more it's going to cost you to secure your smart contract. So since design is at the beginning of the life cycle, it is the least costly phase of security. And this makes sense intuitively.

You haven't spent a lot of time and money yet on a new protocol. So it's easy to redesign the contract, change what you want it to look like, and so forth. But now you might be thinking to yourself, I don't even have any code to review. How can I even secure something that hasn't even been built yet? So this is where things like architecture reviews, risk analysis, and threat modeling are some of the things that you can do even before you have code.

Now let's talk a little bit more about threat model and risk analysis. To put it very succinctly because we could honestly have a whole presentation on just this or a single presentation on any phase within the secure development life cycle. Threat modeling effectively requires you to adopt an adversarial mindset towards your project in its design. So questions like what integrations will my project have? Can these intro integrations introduce bugs and what is the weakest link in my design are good ways to start threat modeling your system.

Looking at the diagram on the right we can see that even without having any code we can begin to threat model. Even this simple diagram of a smart contract that reads external off-chain data and data from protocol A and integrates with an additional external protocol B offers insight into where potential issues can arise. So we can already see even with this simple diagram marked potential points of failures which in our case would be the integration points between these three components in our design. Then moving on to risk analysis, you'll need to holistically view your project for potential issues. So questions like what happens if there is a bug in my life smart contract and how can I design my code to be able to remediate it or minimize impact are all pertinent questions to be self looking again at the diagram what happens if external protocol A is hacked and is no longer producing reliable data what happens if external protocol B just stops working altogether you can use questions like these to inform the requirements for your smart contract which then informs how you're going to code it this process will make your system more robust because you're already thinking about what we would call failure modes in your system.

But if you've never done threat modeling or risk analysis before, it can still be difficult to figure out what kinds of questions you should be asking yourselves. So some helpful resources are past audits and solid, which has a huge repository of audits and bugs that have been found, which can help give you ideas of things to look out for. You can also use AI. Just pick your favorite model and ask it to reason about your design. Looking for weak links and potential security failure modes in your smart contract design using AI can give you a lot of great feedback and insight initially.

Now let's move on to phase two development and testing. As you can see in the diagram, we've now traversed our line a little bit. So now that we're past design, it will be costlier to secure your smart contract, especially given that this is the stage where code and bugs are created together. But still I want to emphasize that development and testing is a nonlinear process. Pivoting or rearchitecting still has a cost but it's still a low one at the start of this phase.

So development and testing is a phase that encompasses many steps. Codewriting documentation testing in its many forms static analysis dynamic analysis audits audit contests and so forth. And yes we include audits in this phase and you'll see why we do that on the next slide. So this slide is a very very highle overview of how you can think about development and testing and what kind of milestones that you should ideally be working towards in this phase. This timeline isn't a one-sizefits-all kind of timeline, but hopefully kind of gives you the general idea of what you should be working towards.

So first in the milestone timeline is your proof of concept, which is the stuff that I would say most people start this phase at. You already have some working code and a general setup for what it's going to do. Next we move on to incremental development which the timeline doesn't show well but within this step is its own cycle of code test code. So adding new features, testing, writing code for bug fixes, rinse and repeat. Then we have the code completion step which is a bit of a misnomer but the idea is that all of your core logic and features are now complete.

You have decent test coverage and everything is fully documented at this stage. Then we move to the greater than 99% code coverage step where you try really really hard to get full coverage with unit testing, integration testing, formal verification, fuzzing, really just trying to max out your testing and pushing out as many edge cases using all of the tooling that is available to you. Then we have the audits and audit contest step which is pretty self-explanatory of all the steps here. Find yourself a reputable auditing for firm or contest firm or both and have them audit your code. Then finally, the last step is audit remediations where you fix all the findings from your audit and then have the code retested and reodited as necessary.

The last thing I want to point out on this slide is that the line between the audit remediation and the incremental development step is to show that this process can be cyclical. Can iterate on one or more steps and then go back to a previous step. You should not view your development process as a fully linear sequence of events. Now, moving on, we'll quickly touch on a few open-source resources for tooling, testing, and security reviewing. First up in tooling, we have Foundry, of course, the all-in-one development framework for smart contracts.

That one is a no-brainer. Next up for invariant testing and formal verification are Foundry again, which has its own invariant testing and also fuzzing tooling suite. Almos, HVM, and Storra whose proofer is now open source. For security tooling, we have a bunch of static analyzers to choose from, including Sliver of course, Analyzer, Adaran, which is Siphon's static analyzer tool written in Rust, and Sgrip, which has beta support for smart contracts. So, when you're thinking about when you should utilize this tooling, just know that this is also a cyclical process.

When you're developing your smart contracts, testing them, and using your security tooling, analyze the code, that is going to be something that happens over and over again. You don't want to wait until the very end to test all of your code and run the security tooling over it. You want that process to be iterative and make sure you're maximizing your testing and running security analysis over each section, one at a time when you have the highest degree of understanding of each section since that's what you just wrote. Lastly, I want to highlight Ciphern's courses as being a really great and free collection of courses on development, security, and best practices and tooling. One of the best ways to help keep your contract secure is leveling yourself up to be able to spot potential security issues before going into an audit.

Becoming your own first line of defense for auditing is a great way to save money and really make sure that you're limiting the amount of bugs that could be in your codebase. seeing as you, the developer, are going to know your contracts the best. All right, moving on to phase three. We have deployment. Now, we're getting really close to this parabolic curve where the cost to secure is going to go way up.

But very important since it's not mentioned on this slide, before you deploy any smart contracts to mainet, you definitely need to deploy it on a test net and run through your full user journey testing for expected actions and actions that are not in your standard flow as well. Okay, with that out of the way, back to mainet deployment. This is where everything for your project is going to be deployed. the smart contracts and libraries, admin multisigs for privileged roles, things like your UI potentially and decentralized storage like IPFS, everything. It's critically important that your smart contracts are not only deployed correctly, i.

e. they were compiled the way you expected and that they contain the smart contracts and libraries you expected, but also that they are configured correctly. Making sure you set all of the constructor values correctly. Initialized your proxy contract if you're using one. Making sure the admin the addresses on your admin multiig are correct and so on are also critically important to validate.

So to that effect I'm going to talk a little bit more about how you can make sure your deployments are successful. Contract validation can be done in many ways with post- deployment validation scripts with transaction simulation and with manual validation from signers. Also note that these validation strategies are not mutually exclusive. You can and should be doing all three tooling you can use to accomplish these different validation strategies include boundary with its fork testing and state variable validations via scripting and then there's tenderly anvil and cast which you can use for simulation and state validation. If we look at the diagram right we have a screenshot of tenderly showing state changes for transaction that occurred successfully on sepoleia test net.

This is a great way to simulate your transactions and inspect the post state which is something that all of your devs and multi-stage signers should be doing. And then we have the various block explorers and source code database which you can use to verify your smart contract source code for others to be able to view it. In addition to using tooling to help identify contract deployments and validate relevant state changes in the event that you're initializing values or upgrading old contracts, you'll also need to set up a process that your devs and signers can follow for validating. It's one thing to be the person who knows exactly what changes are being made. But for other devs who may be the ones deploying the contracts or especially for signers who may or may not be technical, it's critically important to have a streamlined but thorough process that they can follow consistently.

By using this tooling and having a consistent process for deployments and upgrades, you can ensure that any changes that occur are expected and fully validated on your smart contracts. And finally, we are now on phase four, post deployment. So, at this point, your smart contracts are deployed. Lots of people are using your contracts, and maybe your contracts have a high TVL. Mission accomplished, right?

Well, not so fast. There's still work to be done. You can't just sit back at this point and rake in the tokens because now that your smart contracts are deployed and handling real money, malicious actors are going to be scouring your contracts for weak points and vulnerabilities. So you need to have monitoring in place to make sure that you have insight into your smart tracks and alerting if anything is happening that shouldn't be. You should also set up a bug bounty program so that way you incentivize researchers to find and report issues before you get hacked.

And this is where it gets especially costly. So hopefully you've spent time securing your smart contracts during all the other phases. You might be spending a lot on the bounty payouts. Now, the last point I want to make is that the post- deployment phase is really only the beginning of your smart contracts. You're not really ever going to stop securing your smart contracts or spending time and money on security.

The total time spent may taper off at some point, but it will never be flat. This is because there are several elements of security left even in the last phase of this SDLC. First is bug bounty triaging and response. It's not enough to just have a bug bounty program. When security researchers come knocking with findings, it'll be up to you to identify whether they are true issues or not and how to address them.

Addressing these issues may even kickstart the whole SDLC process over again. So, be prepared to spend time on developing fixes and spending money as rewards to researchers who find issues in your protocol rather than exploit them. Then there's incident response. If this isn't something you practice regularly with your team, then you're probably not going to have a good time when you get notified at 3:00 a.m.

with an alert saying your contract is on fire or it's leaking funds like the Titanic. It's not a good time. This also flows into incident training, which beyond getting your team mentally prepared to handle incidents, also includes creating runbooks for how to deal with various issues that could pop up with your project and also creating or running tabletop exercises. Tabletop exercises are a great way to prepare and train for various hacks or situations that could occur. And you don't really need to have any special tooling or security firms help to do something like this.

Even just taking an hour or two and asking questions like how would we respond if X feature on our smart contract stopped working or is a great starting point to larger conversations about security and incident response. Then we have livveness checks which really aren't talked about a whole lot but if you're using a multisig or you have any kind of security mechanisms around keys that hold privileged roles in your systems which you definitely should you really want to make sure that you can access those keys when you need to. So testing that ahead of time I can tell you is a really good idea. Finally there's monitor maintenance. Maybe your monitors don't work as expected or they're too noisy and you want to update the heristics.

But whatever the case is, you will be spending time tuning your monitors. So amongst all these different areas that you can be spending time on, just know that this aligns with your smart contract being perpetual. So long as there is a blockchain, your smart contract will exist forever. So it's your responsibility to make sure that your users are continuously safe using your smart contracts. And with that, that's the end of my presentation.

Thank you all for listening and now we will open it up for questions. Amazing. Alexis, thank you so much for this presentation. So, we already have one question.

Let's see which one is it. Pulling up the comments. Uh, let's see. Do you have any recommended smart contract static analysis tools based on your experience? So, I would say definitely like Slither is like the OG static analysis tool.

It's still to this day, it's being maintained. There's new features. You can add new plugins and stuff like that. It also has a really good corpus of like what do they call them like huristics that are already built in and there's also lots of plugins that exist. So that one's really good as like a starter because there's a lot there's already a really great ecosystem around that tool.

Amazing. While we wait for others type questions, I actually have a question to you. So my question is if you had to choose between audit versus public competition, which one would you choose?

Yeah, so this is a great question. So when I'm thinking about audits and audit competitions, there's really like two things that I'm thinking about. There's opportunities for finding high-end critical bugs and the number of high-end critical bugs that are found and then the level of effort spent by me or any other member of engineering and security that basically has to do things like answer questions, facilitate, judge and stuff like that. So to me, we need to balance these two uh things. We obviously want to have as many opportunities to find the most serious bugs, but we also have to consider that we can't spend forever looking at bugs and we can't spend an overly large amount of funds either.

So, we need balance between the two. And for me, when I'm thinking about audits, audits tend to have more of this balance of like relatively high opportunity to find serious bugs while still being in general like a low medium level of effort. But if you're willing to put in the work, then I think there's an opportunity for audit competitions to give you more bugs and potentially find more serious bugs. But you really, really, really have to put in that work to to be able to get that equivalent or higher outcome.

Awesome. Um, so we have two follow-up questions to the previous question. So the first one is, could you suggest any effective tools for monitoring smart contracts?

Yeah, so at Coinbase, um, we use Hexagate. Hexagate is our monitoring partner. They also we have a agreement with them. So basically any of the base builders get like their their threat monitors for free. So that's like a really great deal to like get started with them.

Um there are also a lot of different other like monitoring tools in the space. I know there's open supplement defender and so between those two I think they kind of have a really good of different like monitoring tooling that you could use.

All right. Um so I have one more question to you actually. So what do you suggest for teams with a limited budget to do to have a secret development?

Yeah. So basically like I mentioned I think in like the development slide right scaling yourself up because there are a lot of free resources to be able to use them. So you as the developer of your tooling like as the developer of your smart contracts have a great opportunity to be your first line of defense when it comes to securing your smart contracts. You don't want to just rely on auditors. That's your last line of defense for smart contracts.

If you're getting to that point and you've not thought about any security and you're going into your audit, you're probably going to end up with a poor outcome with either a lot of like high critical bugs or potentially even like architecture destabilizing issues that you could have found earlier in your design. So, a lot of that is on you as the individual. And so, I find if if you're willing to scale up, you can oftent times kind of take out pieces that would be normally like a build versus buy kind of equation. uh and kind of save yourself a lot of money there.

100% agree with that. I don't see any more questions, but I have one final question before we wrap up today's session, and that will be, how do you use LLMs in your day-to-day work at Coinbase?

Yeah. So, I use LLMs a lot personally for diagram. So, what I found is that LLMs are really, really good at like reading like structured data. So if I'm looking at configuration files for like different like state validations and stuff like that, I find that they're really good at reading stuff like that and also like transaction traces. So if I have like a giant transaction trace that I'm looking at and I'm like it's going to take me forever to like figure out where the pertinent information is, I can honestly run that through an LLM and be like, "Hey, can you find me the section?"

And it like pops it out in like a minute. So that's a really great time saver there. And yeah, and then I also use LLM's obviously for like doing coding and like doing one-off automation scripts. I find it to be really well really good at that as well.

All right. Yeah, I don't see any other question and those were my only questions. So again, thank you so much Alexis for this presentation. For watching this live, we will have another session two weeks from now on where Matt from Protocol security team will give a talk about how to prepare your smart contract for external audit to maximize your return. Thank you so much for joining today.

I will catch you all in the next session.