Coinbase Security Series: dApp Monitoring and Incident Preparedness

Hi everyone, welcome to this week's Coinbase security sessions. My name is Roger Bay, a member of the blockchain security team here at Coinbase, and I'll be the moderator for today. So joined with me is Tom Ryan, a central pillar of our blockchain security team, and he'll talk to us about DApp monitoring and instant preparedness today. Take it away, Tom.

Oh, cool. So, hi everybody. Welcome to the Coinbase security series. As Roger mentioned, my name is Tom Ryan. I'm an onchain security engineer at Coinbase.

I've been with the team for about four years now, spending time on both blockchain security and the security operations teams, and I'll be your speaker today uh for episode 6 of the security series covering DAP monitoring instant preparedness. So, before we get started, I want to lead off with a question. If you had an incident happen tonight, would you be able to detect it and would you feel prepared to respond? I'm hoping that the answer to that is already yes. But if not, by the end of today's session, I want you to feel like you have a plan for detection, feel prepared to respond, and know the steps on how to get there.

What are we going to cover today? First, we're going to talk about why monitoring matters. We'll start by reviewing the Ronin and VNB chain bridge exploits to discuss the value of effective monitoring. We'll then do a quick reality check. I'll walk you through the different levels of preparers you've seen across teams so you can figure out where your project sits today.

After that, we'll move into practical insights. This will be the meat of the talk. I'll share 10 specific insights that you can apply directly to your Dapper protocol in order to feel more prepared. Then we'll ground this within a case study from base talking about how we monitor fault proofs. This is a live complex system that shows how monitoring works in practice.

Next, I'll give you tips on where to start with monitoring if you're just beginning your journey. Finally, I'll close out by showing some free security tools, support, and resources that we offer to teams building on base and then finally open it up for Q&A. Cool. Let's get started. We all should know the value of monitoring and why monitoring matters within security operations.

Let's review the cost of not monitoring versus monitoring via the Ronin bridge exploit and the BNB chain bridge exploit. With the Ronin bridge exploit, attackers managed to compromise validator keys and drained $624 million worth of ETH and USDC. The funds were drained over a 6-day period. 6 days. Users continued to operate normally, depositing, withdrawing while attackers quietly siphoned out funds.

The breach was only discovered when a user tried to withdraw a large amount and couldn't and the postmortem revealed the ugly truth. The team had no monitoring in place to detect the theft. No monitors, no alerting. This outcome could have been a lot different. They had some basic monitoring in place.

6 days could have been 6 minutes. And within this industry, it makes all the difference. On the other side of the coin, we have the BNB bridge exploit. In this case, the attackers managed to mint around two billion two million BNB tokens, but the team was able to detect this unusual minting behavior. The scale of the mint was outside of the normal range uh and thus it triggered an alert.

This allowed the team to execute response procedures, leveraging a pausing mechanism and halting the chain within hours. In the end, the net loss was around, I believe, 100 million. While still painful, it's far less than the full $570 million exposure. So, what lessons do we have here? It's not that monitoring prevents all hacks.

It doesn't. The Ronin bridge still would have been attacked. B&B chain still would have been attacked. But monitoring changes the story, changes the timeline. It changes whether you discover the attack after 6 minutes or after 6 days.

Now that we know why monitoring matters to talk a bit about preparedness. Monitoring alone isn't enough. You could have the best alerts in the world, but if your team doesn't know what to do when they fire, you're still going to be vulnerable. So here's the reality we normally see across teams. On one end, some teams are flying blind.

They don't have any monitors. They run manually checking base scan or ether scan and sometimes hear about issues first from angry users on Twitter. In the middle, we have what I like to call the Bob setup. There's one developer, Bob. Bob's smarter.

Bob knows the system. Bob has some alerts set up that go to his email, but everything lives in Bob's head. There's no documented runbooks, no on call rotation, no backups. If Bob's asleep at 3:00 a.m.

or Bob goes on vacation with Alice, project can be a risk should an issue arise. Finally, on the other end, we have prepared teams. These teams have alerts flowing to monitor channels. They have documented runbooks that explain exactly what to do when something happens. They rotate on call responsibility so someone's always available.

They run tabletop exercises with key stakeholders. They know who has the authority to act, what assets they care about, and even automate responses in certain cases. So the question for you is where does your team fall on the spectrum? Are you flying behind? Do you have the Bob set up?

Or do you feel totally prepared? Here's the thing. It's really hard to ever feel 100% prepared. What matters is that you're continuously taking steps in the right direction to improve your security operations. And with each step that you'll take, you'll increasingly feel more and more prepared over time.

So, let's shift gears into some tips on how to feel more prepared. I want to share 10 insights that you can apply directly to your project today. Uh, but before I do that, I do need to do a quick disclaimer that monitoring incident preparedness is not one-sizefits-all. Security operations will depend on your specific protocol, your organization, your team structure, your tools and tech stack, and the maturity of your processes. So, as I go through these insights, think about your organization, what tools do you have, what processes do you follow, what people are available, and adapt these ideas to fit your context.

Let's jump into it. So, first of all, get your monitoring requirements from your threat models and audits. When you're setting up monitoring, it's easy to get consumed by how am I going to get alerted? What dashboards do I need to build? Where do these things need to go?

Start with your threat models and your auto findings. These documents will tell you where your mo protocol is most at risk. And these risks should drive the answers to what do we need to monitor and why are we monitoring. One tip is to only focus on the how are we monitoring and responding once you've nailed down the what and the why. This will flow naturally once you have those questions answered and make it a little easier when you're not overwhelmed by all the the things necessary to get monitoring set up.

Another tip that I found helpful is segregating duties between the why, what, and how. You have a single person owning the full life cycle. It's easy to kind of get blind spots as you're as you're building kind of think you know the whole thing. But if you have an auditor to inform the what, a security and DRI to define the why and a monitoring engineer to implement the how. The separation creates a healthy friction that helps you uncover blind spots.

Next, test your monitors. Test among tested. That's why we have base, right? It's not enough to send the monitors, but you need to know that they actually work. Simulate realistic scenarios.

Fire your monitors and confirm the alerting the way you expect them. It's okay to have some monitoring overlap. It helps build a more resilient monitoring infrastructure, but just be careful that you're not generating too much noise where your team is going to start ignoring the alerts. When we were testing our faultroof monitoring system, which we talk about a little later, we didn't just assume everything worked. We partnered with Receal 911.

We ran a war game with Optimism. We simulated events. We've got the alerts fired and validated that our monitoring systems actually worked even before we even rolled out the protocols. Next, uh document your runbooks. A lot of people overlook runbooks.

Runbooks are so so important to feeling prepared. When an alert fires, the worst thing that you can do is to scramble to figure out what it means. Every alert should have a runbook and ideally two run books in my opinion. You need one for triage. How do I know this alert's actually valid?

And you need one to response and what to do if it is. One of the main challenges that teams face during incidents is getting the right people in the room. So, runbooks should not only document the actions, but who to call. Do you know how to get in touch with your partner teams, your vendors, the base team, 911, anyone necessary to mitigate an incident? As a general rule of thumb, if the alert is not purelyformational, it should have a runbook.

Next, we want to make sure we have established an on call rotation. You can't just rely on Bob. As great as Bob is, Bob needs to go on vacation sometimes with Alice and you don't want that to leave your protocol unprotected. So if your system depends on a single person, it is fragile. Set up an on call rotation.

If you divine define your severity levels, attach runbooks directly to the alerts, can be really easy to pick up for another team member to pick up where Bob has left off. Next insight is run tabletop exercises. This one I can't stress enough. If you want to feel prepared, tabletop exercises are the way to do it. You'd rather have a messy tabletop than have a messy real incident incident.

So, a few tips for tabletops. Don't reveal the exact scenario in advance. The tabletop won't be as affected as effective if you know what's coming. To that end, have an independent engineer or an outside party help you conduct it. That way, they can kind of offiscate the information and you can start the war game kind of blind to what you'd expect.

There's also various degrees of tabletops that you can run. The simplest to say, "Hey, we just got exploited. What do we do?" Another option, hey, we just received this alert, an actual alert that we have configured and we right, we just got this alert. What do we do?

Take it a step further. Bring the alerts on test net. Simulate your monitors. Make sure that they work. Rest test them or even go a step further and test them end to end on a test chain.

Deploy a conduit test chain, set it up. Um, this can help not only test your monitors, but also test your mitigation procedures and really simulate a real life scenario. The bottom line is any tabletop is better than no tabletop. And it's better to get all the stress and the coordination out of the way early um before having to do it in real time in an adverse scenario. Next one is refine your runbooks from insights from tabletop.

This is also I also say continuously update your runbooks because a lot of times people think runbooks are one and done. Oh, I just deploy this monitor now I'm done. No, runbooks are living and breathing documents. So after a tabletop or a false positive or a change to a monitor, go back and make sure you're updating your runbooks, right? Don't lose the value of a tabletop exercise or all these data points that you're getting uh by not updating them and allowing them to get out of sync.

Conduct a retro assign action items. Make sure that they stay living documents um and make sure that they're aligned with what is in production. Going to rapid fire a couple additional uh insights here. First one, inventory your contracts and vendor dependencies. You can't protect what you don't know exists, right?

they'll keep an updated list of your contracts, your admin rolls, third party dependencies. If an oracle gets hacked and you're unaware that you're even using it, that's going to be a problem. But if you inventory it and you're actively monitoring it, then you're going to be in a much much better position. Number eight, perform key live checks. Make sure your keys are available.

A lot of teams think about key security, but often times overlook key availability. In the event of emergency, you need to know that your keys are accessible. You don't want to wait till the last moment to discover that they're either locked away, not available, or with a previous employee. The best defense is worth is worthless if there's no one can actually invoke these mitigating actions. Number nine, automate responses when possible.

Speed matters in our world, so automated responses like pre-signed pauses can buy you time in the event of an incident. Uh that said, automation does come with some risks. Make sure your automated responses are tied to super high fidelity alerts. Otherwise, you're going to be pausing contracts unnecessary, hurting user experience, or waking Bob up in the middle of the night for no reason. Finally, request help.

You're not in this alone. Base has so many support channels. The community has resources like Seal 911 in case of incidents. Other builders have walked this path. So, don't be afraid to reach out and lean on the community for support.

So, these 10 uh insights might sound simple, but they can help you move from flying blind to relying on Bob to being prepared as a resilient team. All right, let's bring this back to a real world example and talk about monitoring fault proofs on base. Fault proofs are a part of BAS's architecture for decentralized permissionless validation of change state. They allow anyone to propose a state and anyone to challenge it. Fortunately for me, the guy who was working on how to monitor, we had audited the crap out of our fault contracts.

So, we had a really good understanding of what we need to monitor and why. Fault proofs are a good case study because they're made up of several modular components. Um, one of them being the system itself and the other being the actual dispute game that occur dispute games that are created from the factory among among other components that fall into the whole uh fault ecosystem and system. So within fault proofs we ask what are we monitoring right ownership changes privileged actions why are we monitoring to ensure system integrity for dispute games what are we monitoring early withdrawals challenge proposals why are we monitoring to ensure the state integrity once we have the what and the why down it makes it a lot easier to think about the how and in this case for fault proofs we need to monitor each individual dispute game as they're being created so in this case we used uh hexagate uh to build our fault proof monitoring system. I'm not going to go into too deep of the technical details.

We have a blog and a repo with our monitoring scripts if you'd like to learn more. But here you can see a setup of our pure per dispute gave monitoring system that we've built specifically for fault proofs. Um on this next slide here, I also drop dropped in a couple examples of more along the how. On the left side, you can see how we're monitoring individual uh event invocations on specific smart contracts. And on the right side is a screenshot of a gate lang script and custom invariance that we built um for the dispute gate monitors.

So throughout the life cycle of base we're also performing tabletop exercises. One of the ones that we did in collaboration with optimism and seal uh was a war game in which we simulated a fault proof upgrade that led to a vulnerability that allowed withdrawals on layer 1 without a corresponding withdraw corresponding backing on layer 2. So in this war game, the SEAL 911 team set up a test chain. They didn't tell us what they were going to simulate. We replicated our monitors on this test chain, kicked off the war game.

Thankfully got alerted when uh when the exploit was deployed, and then we conduct the war game, uh simulate our mitigation responses in collaboration with optimism. Um and it was really just an enriching and very learning great learning experience all around. So big shout out to Sealat 11 and uh the optimism team for partnering with us on that. If I were to go back and build monitoring from the ground up, right? Where would you start?

Say you have nothing today and you want to get started. How can you kind of take it in manageable bites? Um, so divide into kind of four pieces here. First is function and event monitoring. This is kind of the simplest most rudimentary.

You should be monitoring critical functions and event invocations that could impact your contracts. Right? This is things like upgrades, changes to ownership, pausing, setting different oracles. You can do this on Etherscan, but that's like not a level of transparency. We want to know when these happen.

We want to be able to have be alerted when it happens. Say something pauses and you didn't expect to pause. Right now, we're going to have something we got to investigate there. So, if you have to really start at the base, focus on function and event monitoring. Use your audits to identify what you need to monitor and what functions to alert on and set those ones up.

Next, threat monitoring. So, this is a generic threat monitor with uh AI and ML solutions. This is something that's actually available uh via free hexgate monitoring on base, which we'll talk a little bit more later. This serves a generic as a generic catchall should you not have direct monitoring coverage over a certain aspect. Right?

So, it's going to detect things that maybe you don't have the capabilities to detect or it's you maybe you're just monitoring functions and events. This will give you an added layer on top of that to detect additional threats um that you may not have been privy to. Next, community monitoring. So, I would set up inbound channels. The community can bring issues to your attention.

So, let's say a user sees something weird. They need to know how to reach out to you. They need to know how to communicate this issue. someone on crypto Twitter is, you know, identify that your contract's being attacked, but they can't figure out who owns this contract or if anyone even cares about this contract. You want to make sure that the community knows that this is yours.

They know who to reach out to and they can reach out to you in a timely manner at these channels are monitored and that you can take actions from these channels. So, do things like join the base builder network. You also can uh register your app via the open label initiative. Make sure you plug in all your socials and everything like that. So if something does arise, the community can reach out to you and you can take appropriate action.

One of the more advanced levels of monitoring I'd say is is invariant monitoring. So you can design variance to uh detect when your protocol is not working as intended. These are a little bit trickier. So this be like an unexpected mint over a certain amount. But oftentimes you need to configure like what is that certain amount of the mint that you want to expect.

So you might have some false positives that arise of it. Might require some tweaking. It might require some different scripting languages and things to actually design these invariants um in support of your monitoring systems. Um but don't let that one scare you. They're they're not too hard.

Once you have your odd requirements like that, the how will figure itself out. Um so next I want to talk about some of the perks of building out base. Uh one of the best parts of building out base is you don't have to do it by yourself. We are here to support you. Um I want to plug three programs that we have um that are that are directly relevant to this talk.

So the first one is free monitoring with hexagate. So uh if you apply um which will we'll send out the links for the application after this the hexigate you can get access to the free ecosystem tier on base uh this will provide you with real time MLbased threat modeling for up to 10 contracts plus five custom monitors for the known threats and you can send these alerts to sack telegram or email. We also recently launched uh free security reviews by our protocol security team. Program is currently in beta and offers uh scoped reviews, security recommendations, and hand-on guidance. And then finally, there's also free audits through partnership with security XYZ backed by over a million dollars in security grants.

These are available for builders on base. We want to make sure that all of our builders are supported and have all the resources that they need. So these aren't just perks. These are part of how we think about building a secure ecosystem because if you succeed, the ecosystem succeeds and if you fail due to an avoidable exploit, everyone loses. So, please take advantage of these resources.

You can find them and more at base.org/build. Um, and to close out, uh, monitoring instant fairness, it's not going to eliminate risk, but it's going to change the story. Preparedness isn't about perfection. It's about being being able to respond quickly and effectively.

It's about making sure you're not flying blind. It's ensuring that you're not relying solely on Bob. It's about knowing that if something happens at 3:00 a.m. on a Sunday that you're ready.

So my challenge to you to ask them today, take one step, inventory your contracts, write one run book, test one, monitor, just start. As you keep shipping, you'll feel more and more prepared. Um, and remember, you're not alone. Base is here to support you, and the community is here to support you. So thank you so much for joining today's session and we can move on to Q&A.

Yeah, thanks so much Tom that was very informative.

Uh so we have a user question here from meme market. Are you able to monitor poly market of meme coins? So just for some context par poly market is like a prediction market and I guess for meme coins maybe like can you monitor the price uh of a token or something?

Yeah. Yeah, I'm sure you can. Um, I'll say you can't ne I don't unless it's on base, I don't believe you can do it through the hexagate partnership. Um, but yeah, there's tons of monitoring solutions that are that are evolving. So, uh, yeah, this talk was more focused on how to protect your own DAP protocols rather than monitoring um, poly market coins.

U, but I'm sure there are funny solutions out there um that can that can satisfy that need as well.

Yeah. So one question I had like these uh monitoring we're talking about seems to be a lot we we implement them after creating the smart contracts and the protocol. I was wondering if it's possible to have the idea of monitoring during the design phase. Would that help with creating monitors or anything like that?

Yes. Short answer. Short answer yes. So when you're designing a smart contract and you're well I'll tell you when you're doing designing the monitors for it if it's difficult to look something up or query something or like you don't have an event for it can get quite quite difficult to to develop monitoring solutions for it. So in the design phase if you can build something that's either easy to look up whether it be a contract event or something that can be queried with an invariant it'll make things a lot easier.

uh if it's obscured or hard to recalculate or reperform uh it can be a lot harder. Um and yeah, monitoring it's very important to have security involved in the design phase as well as you design and and launch your DAPs. Um so these are definitely considerations that you want to consider. You don't want to get to the end like oh this is going to be way too hard to monitor. I got to write a bazillion lines of of code to make this happen.

So uh the easier it is to look up or query the easier it would be to monitor.

Yeah, that makes sense. Uh, I was wondering, you know, like Coinbase, we have so many smart contracts. How often do we deal with false positives? Like, how noisy is it for us?

Oh, yes. False positives. False positives are are you never really want false positives, but the thing with false positives is that sometimes they are a natural side effect when you're trying to achieve coverage over a wide range of risks or you're working with a complex set of protocols. So, as much as you don't want them, sometimes they happen. Um, so some of the things that we do to mitigate them is we make sure we test them in advance or making sure that they work as intended.

If we do get a false positive, we always want to make sure that we're going back and we're tweaking the monitor or making a fix or, you know, identifying this is a false positive and communicating it back to our to our monitoring partners. Um, other things we do to combat false positives, um, we when we have certain levels of of confidence in an alert, might just like mark it asformational, right? We're like, we're testing this one out, like we think this one will be effective, but maybe we haven't got the configuration totally down yet. So, we'll just kind of pipe them into like a generic Slack channel uh, until we have full confidence in it. And then we'll make sure that it's going to the devs, you know, going to page of duty, right?

We don't want to be waking up Bob at 3:00 a.m. to all these false positives. So, we want to make sure that that they're working correctly. Um, we also have a team that can support the triage of of alerts.

So, if we have alerts that need an extra set of eyes before they're escalated, we can send to them. But, it's also super important to have that triage runbook because a lot of times they may not have the context of of how to triage it and whether or not this is true true a true positive or false positive. Um but in that case we can kind of have more experimental alerts have a wider range of coverage um continue to tweak and refine until we get it to a point where we feel comfortable you know making sending it directly to the devs and having them respond to it. Hopefully that won't answer the question.

Yeah that makes sense.

Yeah having like a practice stage for monitors before fully implementing it.

Mhm.

Um I was also wondering like you know our monitoring for the fault proof system itself is quite complicated. for always creating new smart contracts like every hour or so. So the monitoring has to be very dynamic like is this like resource intensive? Is this costly? Yeah.

So we uh partnered with Hexigate to build out our faultroof monitoring solutions. Um Hext was super accommodating help us help us build solutions to meet all of our needs. Uh we are you know using all their capabilities in order to to build out our our fallproof modeling solution. Um so to put it simply the cost is is about the cost of of of Hexigate. Um fortunately though for base users uh if you do sign up for the Hexate ecosystem program and the capabilities that are offered there do not support or you know you need more beyond what is offered in the pre-program.

We do uh through the partnership offer a discounted rate for for base builders to get access to the full enterprise tier. you can have the same capabilities um and variant monitors and so on uh that we have for our file proof systems. Yeah, that's good. Yeah. So, I think that's all the time we have for today.

Thanks for attending everyone and we'll see you in the next Coinbase security session and thanks for coming Tom.

Anytime. Thanks everyone.