The War Room Playbook - How to Respond Effectively When You've Been Hacked. Presented by @immunefi

Hey everybody, my name is Dan Johnson and I'm on the protocol security team here at Coinbase. I'm joined here today with Mitchell Amador, the CEO of Immunify, a Web3 security company that's been dedicated to protecting the on-chain ecosystem since 2020. Today we're serving up our 11th episode of our ongoing Coinbase security series where every 2 weeks our team gets together and we connect with the great builders in the Web3 community and discuss the topic of blockchain security and how to build safely in the space. As is the case with each of these sessions, please note the disclaimer that's at the bottom of the screen. Uh and with that, we've got a great session for you today, so I'll let Mitchell take it away.

Okay. Okay, exciting stuff. We're going to talk about something terrifying today. No pressure, doesn't mean you should run away, it means it's going to be even more exciting, but just heads-up on that is first things, hear me loud and clear, we're ready to go, Dan? Yes, sir, sounds great and uh you can go ahead and test uh sharing your screen and getting the presentation up and running as well.

Favorite part. Death. The My turn. Difficulties, right? Right here.

I'm scared already, but you're going to you're going to make the uh you're going to make this topic less scary for everyone by by educating us all. Only a little bit. I suppose it's all contextual at the end of the day. You know, we're going to make some parts less scary, we're going to give you agency, we're going to give you some amount of control, but we're also going to show you the horrors of reality and what can go wrong and I think that's just as important here. Now, uh to kick off, this is going to be an important presentation.

If you are a builder on chain, this could be the presentation, the content that you consume that saves your project from total destruction. If you're building, you probably have all of your net worth plugged into that. This could be the thing that saves everything. I'm going to show you how to survive the hack experience. It's not going to be easy.

It's not going to be quick, but I'm going to give you the tools that have been tested over hundreds of incidents in the industry, many of which I have seen and participated in and helped save first hand to stay safe when you are targeted successfully for exploitation by black hat hackers, no matter where they're from. A little bit about me for context, as Dan said, I'm the founder and CEO of Immunify. We are the leading security services platform in crypto. We protect the vast majority of the industry, around uh 180 or 200 billion dollars worth of assets and contracts and beyond today. Uh we have one goal, which is to keep the industry safe.

That's why we're made in the first place and we're still doing that to this very day. We've made incredible progress in that time and a lot of that progress via on the you know, the bug bounty side, the technical side, security operations, opsec, infrastructure, a lot of those experiences, right, that we those lessons that we were able to learn come from surviving and seeing how bad things can get first hand. Okay? These war rooms are the key to that. But just because something scary is going to happen doesn't mean it has to be a fatal experience.

We're going to show you how. So for starters, the first thing that I want all of you to understand is that war rooms are inevitable. If you're building something big, building something exciting, you're building something meaningful on chain, contributing to this whole new world that we have going for us, then you're going to hit some problems and they're going to be very, very scary and eventually you're going to make a mistake, you're going to get pulled into some incident or another. If you're lucky, the damage will be small and contained. If you're unlucky, the damage can be total and fatal.

We've seen many such cases. A little example of what we're looking at here, right? On the one side of the small and contained, maybe you have, you know, certain back-end infrastructure gets compromised allowing users to spoof or get spoofed or man-in-the-middle attacked in various ways. And if you can just find that piece of infrastructure, it only affects a certain amount of your users, you stem the bleeding, great, you're safe, you made it through. Okay?

Or you can have your front-end hijacked, great. If you move fast enough, you know, most of your users may not be affected by that. On the other hand, you have incidents like compromising of treasury. Okay? So good example of that is something like the Bybit hack.

This has occurred on so many exchanges with hot wallet compromise. You could lose uh all your assets overnight. And of course in in DeFi, the most exciting expression of the on-chain economy, this can become even more total because we typically pool funds in various ways. Not always, there are there are a variety of infrastructures, but you may be pooling your funds in whatever the architecture of the contracts that you've made. And so if someone can compromise that pool, they might be able to steal all of the user funds.

And we've seen a number of such cases. We've defended against a number of such cases. But the important thing to keep in mind, again, is that these war rooms, they can't be avoided. You're no matter how good you are, you're going to make mistakes. Everybody does.

The attackers have infinite time and resources to attack and you only have so much time to defend. So, the solution is not to avoid war rooms, the solution is to be prepared and to know how to carry it out when it inevitably comes and do your best to prevent it in the interim. Now, when you enter a war room, you have two simple objectives. I'll make this real clear for you guys. Your first goal is to stop further losses.

Wherever Think it Think of yourself like a doctor. You have a patient, he comes into the ER, what do you do first? You stem the bleeding, right? You need to stop the thing that will directly contribute to loss of life or problems. You need to stop further losses.

You need to stop the bleeding of users. It is very common for attacks to be carried out over multiple blocks or to be subject to a variety of particular conditions and not be totally executable in a single block or in a single transaction. In such cases, you can have a chance to prevent the majority of damage. In most of the major hacks that we've seen, this has been the case to greater or lesser degrees. Okay?

This is especially so the case for for hacks that are not strictly on-chain hacks. So you need to find the cause and you need to apply that tourniquet, you need to arrest it. Your second goal is to preserve user trust and this is also critical because even if you successfully stop the bleeding and save the users' funds, if you fumble, if you fumble the post-hack communication, people may trust you less and you may end up destroyed anyways. Congratulations, you'll have saved the users, but you will have been burned, you will have been scapegoated because your your inability to maintain legitimacy despite your good actions. And so we need to make sure that we maintain user trust.

And this is a matter of communication, messaging and of course the conduct that we take in the war room. I'm going to show you how to do that, too. But starting from the top, how do we stem the bleeding? What do we do? Step one, we confirm the nature of the vulnerability.

We confirm the nature of the attack. How did this happen the way it did? We identify the root cause in it. How did this happen? Can it happen again?

Very often, uh an exploit is the tip of the iceberg for how it can be exploited again and again and again. So a good example of that is when you have projects with multiple pools, uh which is uh a great many DeFi protocols, for example, if they can exploit one pool, they may be able to exploit many other pools and it's not a given that the attacker will have done the research to identify all of those other vulnerable tool uh pools. In fact, most attacks historically have been different versions of drive-by attacks. The the attackers cannot trivially check all these things. So you need to take action to find that exploit and see what you can do to to stop the bleeding from there.

And that gets us to the next step, right? Which is to pause all affected contracts. And very often, if you can't tell what that means, like which contracts are affected, if you're just not sure, you're having struggles reverse engineering the exploit and seeing what it really is, then you should be pausing all contracts. You don't need to take the risk. Pause them all, save the users, most people won't even notice, to be quite frank.

It's not like they're watching you all the time. But they will be thankful after the fact and you will buy yourself precious hours to identify what really went wrong. The third step that you need to take is you need to lock down all permissions, all admin access whatsoever. You need to know exactly how the state can change from here. And anything you need to eliminate all possible, you know, gray zones or gaps that you might have, anything that that could have been the surprise and make sure all your other infrastructure is taken care of as well.

Cuz you don't necessarily know how far the breach goes. Historically, most on-chain hacks were discrete events in the sense that they happen um just on chain. But what we've begun to see over the last few years with the rise of more state-sponsored hacking groups, most famously the Lazarus Group with North Korea, is that they are compromising a variety of pieces of infrastructure on their road to ultimate theft. Okay? And this is more typical of traditional uh advanced persistent threat behavior in Web2 and the broader internet.

So you got to lock everything down, don't take any chances, you don't know who you're dealing with. They could be very scary people, they could be well-resourced people, they probably are. Be maximally paranoid, lock it all down. With this, your situation will be paused. Hopefully, you will have frozen the state of your contracts, of your protocols as much as you possibly can and you will be ready for the next step.

Hopefully, this is you stemming the bleeding when it comes down to actual thefts. And if you take nothing away from this presentation, then take these two things. First objectives, stop further losses and preserve user trust. Second, these three basic steps, find the exploit, pause everything, lock down all your infrastructure. This gives you time to live another day.

Now, all that is easier said than done. So now we're going to talk a little bit about how that works. We're going to talk about how you should be constructing that war room to get all that done. And at the highest level, there are three basic roles, in my view, that every war room should consist of. These roles, these functions, which can be performed by one or the same people, but are best performed each by individuals, is the operations lead, the security analyst, and the communications lead.

Let's touch a little bit about what each one of them does. First is the operations lead. Your job is to command and to coordinate functions. When I'm in war rooms, this is typically the job I get to do. It's also one of the most painful and most difficult jobs, and you get the unpleasant responsibility of making some very difficult, very hard choices very often.

In this function, your goal is to relieve all unnecessary thought and work from your compatriots, so that they can do what they do best, while you organize overall sequencing and managing of the remediation process. Okay? Your job here is to set up all the workflows, set up tracking, coordination, communication, decisions, and make the best calls that you can, often as a group, at every single point. The ops lead may be a founder or CEO, very often is not, but is someone who will defer ultimately the the the most difficult decisions to uh a more formal business leader, who will make the final call. But the ops lead is responsible for organizing the work in the lead up to that.

I said in another way, your job is to keep the entire context of the hack and the war room in your mind at all times. And what this takes form in practice is you end up creating the documentation, you end up writing and keeping track of what's going on, you keep track of all the participants in the war room, you're identifying and directing them to all the tasks that they should be doing, you're setting the sync cycles, aka how much time you should allow to pass before you come and check in, which I would typically recommend about 15 to 30 minutes. Every 15 to 30 minutes, you want to check back in with your teams and see how they're doing on their different problems. And you get the unfortunate job of time boxing the decision the decisions and being like, "Okay, well, how long do we have to make this call? And what are the trade-offs here?"

Sometimes, you will be deferring that final decision to the formal business leader, sometimes you're going to have to make the call yourself. But again, your job is to have the full context, the command and coordination function of the entire war room. And it's really critical that you get this right, okay? Because in a war room, what happens to everybody is they freeze, they get stuck, they get worried, they panic, they say, "Can I really do this? What am I supposed to do?

My network my network is going up in smoke. Am I going to go to jail? Did I make a mistake? Is this my fault? Am I guilty?"

And as an ops lead, you have to put an end to all that. There's no room for such sentimentality. You need to go and fix the problem. You have to provide leadership, structure, and clarity to the team members and inspire them to do what needs to be done. In the vast majority of war rooms, the ops lead may be the only person who is experienced, who has done it before.

Most people obviously have never done this before. And you need to provide them the certainty and the conviction that they can go forward and succeed and there's going to be a future on the other side of this. So, that's the ops lead function. This role could be folded into the others, but ideally stands alone. The second function is the security analyst.

The security analyst has the critical and difficult job of identifying exactly what the attack vector was and what the exploit was and how big the scope of damage and risk is, so that they can prepare a mitigation path and ultimately apply that and block any other exploitations. This is the person who is doing the hard technical work to fix the situation, right? If if the ops lead is like um a doctor coordinating, you know, a hospital situation, then the security analyst is like a surgeon, who's got the difficult job of actually doing the cutting, making sure he's found exactly the right problem and addressing the wounds and and suturing them up all properly. Now, you might think, "Oh, you know, security analyst is not the most important part. Why don't we just fold the the ops lead role into that?"

No, we don't do that, because the security analyst role is extremely cognitively intense, and they are in a race against time to identify the nature of the attack vector and to mitigate it before it can be exploited further. And no matter what kind of hack you're in, you never know if there is more damage in the beginning. You have to suspect that you're at the beginning of a process, not at the end, that things can get worse. So, you need to take your security analysts, like ideally, it's uh one or or two people working as a team, should be a small group, and you need to strip everything that's not critical away from their mind. Anything they don't have to think about, they shouldn't be thinking about.

Anything they don't have to do, they shouldn't be doing. They got one job, find the source of the bleeding and stem it as rapidly as they can. That's their one job. Now, typically how this works in practice is after you come together in a say a chat room of some kind with your ops lead and your coms lead and your security analysts, and there might be one or multiple of each of those roles, if you're lucky, the security analyst will go off on their own into a private chat to go and do this work and trade ideas directly. You want to allow them to go and do their own thing, because focus is going to enhance the quality of the work and the speed at which they're going to get it done.

If at all possible, you don't want to remain in the same group chat where you can hear everybody else speaking. Instead, the security analyst should be regrouping with the overall war room every 15 to 30 minutes. Uh 30 minutes is a great cadence, in my view, uh to bring back their findings and their recommendations on what to do next. Obviously, if they have some type of breakthrough, then it makes sense to do that even sooner. Like, you do whatever makes sense, but you want to give space to focus, right?

Communication in a war room is expensive, it costs time and attention and stress, and you only want to do as much as you absolutely need to get to the right outcome. It is fine to have multiple security analysts, in fact, generally, uh I think it's a small team of two to three is is is good. Two is probably best, providing people know the protocol One is good, but getting extra eyes on that is better. It's not a case of, you know, more is better. You keep adding more people, you have more people to coordinate, things get more complex, people disagree about things.

Like, you don't want an infinitely large team. Two to three is probably uh where it's ideal to sit. But this is the security analyst role. If you're unsure of who to bring on, then you're you know, I recommend, if you don't have a security analyst in a pinch, you go and you call your security providers and see if they might help. This is how we have done so many war rooms at Immunify.

We have done I've personally done dozens and dozens and dozen like maybe somewhere at least the high dozens of war rooms at this point. And that's because our customers are calling us in, they have a problem, they don't know how to deal with it, nobody else they know knows how to deal with it, they trust us, we come in there, we manage the situation with them to come and deal with it as best we can. You can also call CL 911. They have a great uh free service that you can use that they also get involved with war rooms with experts from around the industry. You can also call an auditor or similar such player, or you can just pick the best dev on your team.

Like, you do you, you're going to have a variety of options available to you. Uh I suggest that you take all of them that you have on access. If you have access to the resource, leverage it, because now is the moment to cash in on that favor. The third function is trust management. And that means your coms lead, your communications leader, who is responsible for coordinating all the messaging that needs to go out.

And believe me, war rooms have tons of messaging. This is rapid-fire PR that you have to get done and avoid making any mistakes, because one wrong mistake, when a user is in a state of fear and panic, is a reputation forever lost and TVL that will never be returning to your contracts. So, you need to focus very, very finely, very, very specifically on sending the right message and inspiring confidence and trust in your users that they can trust you to handle these situations. Okay? War rooms are inevitable, because hacks are inevitable.

And everybody on chain knows that. So, your users need to believe that you can handle these situations. And by the end of this presentation, you will know how to. Now, the coms lead is best suited to be someone else. They can be technical, but typically writing is a specialized skill, in my experience.

But they do need to be focused, they need to be quick on their feet, they need to understand the project as well as the as well as possible. And their job here is to communicate exactly what the community needs to know at every step. In practice, that typically leads to calm, clear, factual updates, where you avoid hyperbole, but also where you avoid uh how do I say, making things rosier than they look. You don't want to speculate, you don't want to project positivity prematurely. Like, you can save that stuff for the end in the moment while these war rooms are happening.

And war rooms can be wild. I mean, it can be a few It can be, you know, 15 or 20 minutes for exceptionally short cases, but it could also be 24 or 48 hours. You know, I've done done some war rooms that ended up being three or four days, because we couldn't really manage the issues over the time frame, which is very painful. Uh I've done several such cases. Like, it can it can get long.

You don't want to say anything that would cause users to lose trust in you later. And so, intuitively, the best approach, the one that inspires the most confidence, is to treat them like an adult. Clarity, calmness, but at the same time, you know, confidence, conviction. And don't say what you don't need to say. People will have the temptation to try and make people feel better by making claims that they should not.

Don't do it. It's a trap. Also, cadence, very important. You should be communicating at regular intervals. Other than the hack itself, one of the most terrifying things is when a project goes silent on you after all these events.

You don't know is that because they freeze? It's a hack. You also have to suspect it was an insider attack. Are they rugging you? Right?

There's lots of reasons to be worried when a project goes silent. So, don't do it. Comms lead needs to come in there. Communicate clearly, communicate regularly, and inspire confidence that they can handle the situation. Typically, if they need to do their work, they will also go into a separate chat to go and coordinate that.

There could be one or two people. You might also bring in a PR firm if you have um access to partners there. Um you will often work with the ops lead and the security analyst to really make sure that the framing and the messaging is correct from a technical perspective and from an operational perspective. But otherwise, you're working come somewhat similar to the analyst. Unlike the analyst whose job is really to to investigate how an exploit occurred, a comms lead role is much more forward-looking.

So, as comms lead, you should be preparing for multiple different timelines. Not just one situation, but multiple paths forward. For example, we stemmed the bleeding, everybody's safe, right? All users are fine. For example, we couldn't stem the bleeding, everybody's unsafe.

Here's what you have to do next. For example, it was never a hack at all. We found the cause of it. It was a user error or something. Great, that's dealt with.

As comms lead, you need to prepare for everything in order of the severity and the danger of the scenario. So, you end up with pages and pages of documentation in that process. In parallel, there's another thing the comms lead should do. The comms lead should work with the ops lead to document what is happening in the war room as the events proceed, creating a living timeline of events. Because the more the comms lead can take over the documentation duty of the war room, which is going to be needed.

All this information about what happened, who was in the war room, what exactly they did, why they made the choices they did, all that's going to come out in the postmortem, which can be an internal doc, but it's best to also an external documentation that is shared with the the broader community so that everybody can learn how to stay safe. All this information should be recorded by the comms lead while the events are ongoing. Yes, you can have the operations lead do it, too, but if you do that, you're going to be distracting them from the most important choices. So, wherever possible, the comms lead should take point on that. Again, in general, every role, every function should be done by a single person and not merged if you have the option.

That's the setup of a war room. Those are the three main roles you need. Those are the three things that they are going to be doing. That's how they're going to be driving the process. And we heard a little bit about the sequencing of events.

Now, the next thing, now that we have this context, the important thing for you to remember, to keep in mind, is to remain calm. Remain very calm. Like I said in the beginning, people mess up. People freeze. People become afraid.

Your ability to remain calm lets you make good decisions about the matter. But more than that, your calmness is contagious. Your reason transmits. The calmer you are, the more centered, the more focused, the more centered, focused calmness you're going to bring to everybody else in the war room, and the better they are going to perform. So, whatever you do, remain calm.

Very important. Now, in terms of how this goes, we walked through a bit of the functions. All these things are actually happening in parallel. Now, you see why we have an operations lead in the first place. Because we're not doing these things one at a time.

No, no, no, no, no. We're doing all of these things at the same time. And we're in a race against time while we do it. Even more difficult, even more stressful. We're in a race against time to identify the source of the breach, of the source of the attack, to block it, and then to preserve the trust of our users and the community at large.

So, while you, the ops lead, or whoever you may be, is preparing the decision flow, mapping out how the the sequence of events could go, and what are the various pathways, and how you're going to respond at those pathways to make the calls, the security analyst needs to be doing the forensic work. Understanding exactly how a fix and mitigation can be implemented, and preparing everything needed to execute that fix. And the comms lead, for their part, they need to be preparing the messaging in parallel for the various scenarios. You don't know how much time you're going to have to execute this. You don't know how much time, you know, emotional contagion can take from you, and how quickly it can spread.

And again, remember, it's not just the hack that can kill you. It's also the loss of user trust and legitimacy that can do the job. So, you need to be ready for it. So, everything needs to be done in parallel, as fast as possible. And the ops lead has the venerable duty of coordinating all this and keeping every decision, the entire state of the war room in their head at all possible times, so that they can maximize the speed of execution.

Having said that, once you're done all that, you can now proceed with postmortem. Okay? Postmortems are typically done 24 to 48 hours after these events are all wrapped up and the fix is applied. And the reason for that is because these war rooms are exhausting. It is not usually feasible to do the postmortem right away afterwards.

People are too tired. People are too stressed. You know, it's just too much to deal with. But the next day, you want to get this postmortem out as soon as you possibly can. The users are waiting for that.

They want to see that you handled the situation correctly. They want to know that they should feel safe. They want to know that their money has not all been stolen from them, and your postmortem is where you do that. Four key components: root cause of the exploit, the attack and postmortem timeline of what exactly happened, the fix that was discovered and applied, and what actions will be taken further into the future to ensure that this type of attack never happens again. Very simple, little bit stressful to write, I'll admit, but this is the outline for every postmortem that you should do.

If you do this, you're not just going to be helping your community members, keeping them engaged, keeping them loyal, keeping your your own project legitimate in their eyes and worthy of trust, you're also going to be supporting the entire future security community on base and in the broader blockchain ecosystem as we all learn the lessons of that difficult experience you had to go through. But sometimes you're not so lucky. Sometimes the black hats win, and they steal the money. Well, luckily for you, that's not necessarily the end of the situation. You have options.

There are things you can do to help with the situation. You can try and negotiate with the black hats. And make no mistake, you should always always always try, although you should also understand that it will very rarely work. Okay? And the reason why we want to do that is because it costs you very little, no matter what the cost, to try and negotiate, to try and get the money back.

Whatever amount a black hat may request of it as payment is probably going to be a good deal for you, if you should take it. And you don't give up any of your legal rights by doing so. The black hat may have won that battle, but if there's ever a chance to get the money back, you should take it. Don't get your hopes up about it. I'm not saying this is going to work.

It usually doesn't work at all. But you should try. In terms of how you do that, there are three things that you should keep in mind. Number one, you need to open comms channels. Luckily for you, the black hat is almost certainly watching what you do next.

Typically, they enjoy getting the jump on you. So, you need to open up a way to communicate. You can start just with your public X or or whatever community infrastructure you're using. Chances are the black hat will read it. Offer a safe, anonymous communication channel that they can use without compromising themselves.

If you offer something that is unsafe, they won't even bother. The classical example is an on-chain message in a transaction. But, you know, some people set up ProtonMail accounts or or or other types of, you know, discrete email services. You know, these things can also work. The second thing that you should do is offer a safe return path.

Make it really really easy. I always recommend people adopt Immunify Safe Harbor, which provides a recovery address at all times. Safe Harbor is a new standard created in conjunction with Seal to make it safer for white hats to protect projects, but we can also use it as a as a clear, vetted place for black hats to return money. Why not? So, if you have Safe Harbor already set up, this is a really great solution for offering that and saying, "Hey, look, we already have a safe place for you to send the money."

And finally, you want to offer them structured incentives. What that basically means is you want to cut them a deal. They keep some amount of money as long as they send the rest back to you. And guess what? No matter what deal they agree to, if they agree to any deal, you should take it.

The reason that you should take it is because you still have every legal right to take action against them if you so wish. Okay? If you think that's really the right call, you have that right. But if you can get the money back, it's always a good deal. Unfortunately, this same dynamic, the fact that we as defensive parties cannot offer realistic legal safe harbor, which is really the thing that black hats need in order to negotiate safely, uh this thing makes it very very difficult for these negotiations to proceed.

Because at the end of the day, governments around the world can always decide they don't care what deals we've made uh with the on-chain community, even if we're hostile to one another, they'll just override it, call it a crime, and prosecute them. That does not actually help the situation here, right? Law enforcement, uh while a necessary part of these types of experiences, is not typically going to be all that helpful in dealing with these situations. Certainly, that's the case if you look at historical DeFi hacks. But that doesn't stop us from trying.

We should still try and come to an arrangement even if the black hats have succeeded. Now, when things have failed, right? When negotiations failed, you still have even more options. And some of these you should do right away. Okay?

High level. First, forensic firms. In my experience, having had to track down black hat hackers a number of times now, the single most useful thing that you can do is post-hack forensics. Because hackers typically leave fingerprints on the crime scene. Most people who can carry out on-chain on-chain attacks do not have the sophistication or the resources to effectively cover their tracks.

And so if you can find them, you can now effectively require you can incentivize them to disclose, right? What they have done and return the money. If they do not, you simply report them to law enforcement. This is typically how most of the forensic firms work in the space. And in my view, when if everything else has failed and you've lost the money, you know, forensic firms are probably your your best bet.

Number two, exchanges. You can reach out to the exchanges to try and freeze all the related addresses. In my view, you should be doing this as soon as possible regardless. Unfortunately though, getting in touch with the network of exchanges is really not easy. So this takes a hell of a lot of work.

But I recommend it regardless. Try and send it to the appropriate contact at the various exchanges and see what can be done to freeze illicit funds. Number three, legal escalation. You can threaten legal action against them. Usually this is also done as part of public warnings.

And I do not recommend jumping to this conclusion if the attacker attackers are smart. Like these are very intelligent people. They'll know that you can prosecute them civilly at the very least and very probably criminally. Okay? They understand this.

You should not jump to threats if you can afford to. You should not jump to uh engage law enforcement if you can afford to because these people cannot necessarily solve your problems. And this is a very counterintuitive thing. It's very non-obvious. You know, if you Let me I'll I'll give a good example.

If you contact, you know, the the Mexican Federal Police. I'm sure there's brilliant investigators there. And you say, "Hey look, a Russian hacker has stolen my money." They have precisely zero influence over the situation. But if you antagonize them in public, you might lose the only chance that you have at getting to some type of deal.

So you should always wait. You should leave public uh threats of legal action with law enforcement until the final step. That doesn't mean that you can't engage law enforcement. You can. You can do that right away if you want.

Just don't tell everybody about it. Because it's not going to help your situation. It's going to make you impossible to negotiate with. Okay? Leave that to the last step.

At the end of the day, we live in a big world where increasingly the the law enforcement and legal agreement networks are breaking down as part of geopolitical competition. And we cannot rely on the traditional legal systems to solve our problems for us here. We have to do our best on our own first and then bring in whatever support we can. I don't like that state of affairs. I'm trying to change that state of affairs every day and make the industry a little bit safer so that we don't have to deal with this type of situation.

But that is how it is for now. And as the operator of a project that just had to go through a war room, you need to be prioritizing what's best for your users and what's best for the broader community. You need to be prioritizing keeping people safe. So be intelligent about it. Okay?

Very important. As a good example of this, I'll go through one little case and uh then we can move towards Q&A and see what would be most helpful for you guys. Uh we had a famous case called Primitive Finance uh many years ago and uh this was a a great team building a very novel derivatives product. Fascinating product. And the key things that proved critical there, well, simply put, you had a a vulnerability that was impossible to fix.

But there was no upgradeability on these contracts. There's nothing that we could do. So what were we supposed to do? Well, we ended up finding three solutions to that issue, right? Here we we risked both a theft of all money in all the derivatives pools.

The team in question was very talented, but these were very difficult uh types of of of exploits to deal with. And they risked losing all their legitimacy. Even though they built some really wonderful technology that was meaningful. Even though they were a great team that had a lot to offer the community, they couldn't deal easily with the situation at hand. And we ended up doing three things that solved that problems.

Number one, they invited trusted experts from the security community that had worked with them to help solve this issue. Specifically that was us at Immunify, yours truly, and our friends over at Dedaub who had done some audits with them. Together, we were able to provide the support that they needed to find the exploits, arrange a fix, and execute. Second, that solution ended up needing a white hat rescue where we had to hack the funds ourselves from each and every vulnerable pool. Pretty scary stuff.

But sometimes that's the right call. We did that. We saved users millions upon millions upon millions of dollars. Okay? It was a great day, but that's what we had to do at the end.

That's an option that you have. Every situation will be unique and particular, but you need to be prioritizing what's best for users. And finally, at the end of that whole process, we didn't hide it. We didn't pretend that it didn't happen. We could have.

We could have lied about it. Like the situation was such that most people would not have known that there were other compromised pools. But we didn't. Instead, we went forward with transparency and that ended up preserving the trust and legitimacy of Primitive for another day and they ended up doing great things with that. This is one example about how these simple actions that you can take on the comms side, on the operations side, on the security side can end up making the big difference.

We executed this entire war room playbook that I just described to you here and it worked phenomenally. And we've all done that now, both me and many other many other security experts in the space have done so dozens and dozens and dozens of times. So this really works, you guys. This can really get to you to the other side. Having said that, I spend every day trying to protect the space as much as I can.

We prevented tens of billions of dollars in hacks so far, but there's a lot more to protect. And I know there are more questions on how to do this than what I've covered here. This is very narrow. So, thank you very much. And uh why don't we bring it back and talk a little bit about what else people need to know about this?

And I'll try to answer the questions as best I can. All right. Yeah, awesome awesome stuff, Mitchell. Um I know you're a busy guy being a CEO and all, but feel like you could also double up as everyone's favorite professor as well. Uh that was that was great stuff, seriously.

Um yeah, I'll scan through our chat here and make sure uh we we capture all of the the questions and feedback. Um I see we have some from like through our channels um and some internal as well. Uh you know, why don't we kick things off with uh That's a good question. You ever you know, with all of this with with all of these war rooms and um yeah, exploits that are going on, do you ever consult with teams just to run dry run table top exercises involving uh threat or exploit response? Um and and at what cadence or how often should teams be conducting these rehearsals internally?

Sure. Absolutely. Absolutely we do so. Now, unfortunately, it's not super exciting to do what we This is what we would call a war game, right? Or a table top exercise where we go and simulate it.

These things are expensive. You're going to spend a day or two just doing this uh for a situation that hasn't happened yet. So you have to be very proactive and forward-thinking to invest in it. But we do do it. There's specifically a a firm that we love to work with on this called Shield Three that has done incredible war games across the industry that we would really recommend to other people.

Uh and this is this is a an approach we very much believe in. But you have to have the bandwidth for it. If you do have the bandwidth for it, I would recommend you do it at least once a year if not twice a year. Cool. Yeah.

But you know, generally only the largest projects get to that stage. When you have like us when you're at a smaller scale and you have one security lead who's responsible for every single problem, you know, one does not easily get permission to go and do a war game. Right. Yeah, and and I mean it might be expensive. Um but I mean even I would say leveraging other builders in the space um that you that you trust um and you don't have to give out sensitive information or anything, but just kind of doing a sort of a dry run with them and bouncing ideas off of each other.

Even if you're a solar solo builder in the space, you know, you you can still take time to plan out the steps that would be involved um with actually triaging a situation like this and it'll just save that precious time in those earlier critical hectic chaotic moments of a of an exploit. You know, out of curiosity, because I think you know the data a lot better than I do. Let's Let's envision maybe the top 10% of on-chain protocols that are out there right now. Like what percentage would you say are adequately prepared to smoothly exec- execute in line with the playbook that you've kind of just laid out here? Sure.

And you know, your your last comment on how wise it is even for solo builders. This is totally true, right? This is the definition of an ounce of prevention is worth more than a pound of cure kind of situation. But you know, everybody's just like, "I just don't want vitamins. I want a pill."

And you'll see this. Like when you look at the numbers, I I would bet that less than uh 10% of projects are well prepared for this. Okay? Interesting. And I think 10% is a glorious number compared to where we used to be.

When I was starting this stuff, I had to deal with a war room every single week. And they almost always happened on the weekends for reasons I'm sure you can guess if you think about it, right? Think about it and you'll be like, "Oh, okay. So, that's who the attackers are." For one thing, and why they're doing their attacks.

Yeah. It almost always happened on the weekends, but now these things have gotten more rare. We've made a lot more progress. So, 10% is way better than things used to be. I can imagine a world over the next few years if we work really hard with, you know, education stuff like this and people really see maybe we can even get to like 20 or 25.

That would be crazy, but everything is possible, right? I think so. Yeah. Yeah, so um, you know, with with sort of in line with this theme of lean teams and, you know, we're not talking the top 10% protocols that do have engineers that are internal, like a a massive team to bounce ideas off of, let's say, for either solo builders or maybe a founder and a co-founder. Um, and that's pretty much all running all of the wearing all of the hats.

Are there any like templates or channels that they can, like in terms of communication templates, things that they resources that are available anywhere online for communicating with impacted users, partners, in some cases media, things like that? Sure. Two places to look. And we have to caveat this with the fact that every hack is unique and every protocol is unique. So, there aren't good cookie-cutter templates for approaching this because they won't match yours.

But, we do have some pretty good things in the form of postmortems themselves and guides on how to deal with war rooms. So, for example, this presentation here that'll be made available to people is a great resource they should check out for how they think about it. I really recommend reading postmortems. We put out a lot of postmortems at Immunefi which we call bug fix reviews for all kind of almost hacks that we managed to prevent. And many projects do put out postmortems these days.

I recommend you read that to understand how things are framed. After you read, you know, five or 10 of those, you kind of understand how it goes and what wiggle room you have. And then I really recommend you you you do some crisis training. If you're lucky enough to have a PR firm or somewhere similar who can provide media training to you, do a little bit of crisis training to understand what it's like and and how you're going to be squeezed and and understand how they respond to things because these are, you know, hacks, on-chain hacks are unique to us and our situation, but crises are not, okay? We know how to communicate and manage these difficult situations.

Your media experts have already gone through this. I mean, but I was prepared for this because I I had already managed dozens of crises before I started running, you know, DeFi war rooms. Like it was all old hat to me at the end of the day. And you already know people who are experts in this. You just have to ask them for advice and for training.

And they can come and help you with that. And they can come and show you how to frame that messaging. And maybe if you're lucky and you bully them enough, they can even come into the war room and do that messaging for you. And that is a much better solution than relying on a template. And speaking of which, right, I'm being prepared, like the best preparation is just to know who you're going to call and have that set up in advance.

Yeah. Yeah, that even in those moments of panic, your brain's not not thinking it's thinking about a thousand different things and even anything that can simplify the immediate response to that is gives you great momentum that is very important to handle things the right way. So, and then I think we'll have time for one last multiple part question. This one, I admit I I I wrote this one, so this one's for me. I'm just curious your take and what your answer is again kind of data-based.

Um, but what's your best guess and the in terms of the percentage breakdown of, you know, white hats, black hats, or gray hat actors in the space? And do you think it's common for, you know, a hacker to change the color of their hat? And I have one follow-up as well. This is a scary question, Dan. See, you know, we were trying to de-risk it, make people feel a little bit more comfortable with the situation.

Now we're going to blow it back up and spook them all. That's how it's going to go. Look, I think that crypto has an incredible community of white hats. Like really the best people in crypto are the white hat security community, in my view. I've been lucky enough to spend years with them.

They're by far my favorite people in crypto. I don't know, like I think what we do in the industry is incredible, but really these people are are are kind of shining lights in so many ways. And unfortunately, on the downside, while we do have these literal Batmans among us, I think they probably make up 10 to 20% of the hacker community. Okay. Sounds about right.

Right? Now, I don't think that the rest of it is black hats. I think, you know, black hats are probably sub 10%. An even smaller Makes me feel better. Right?

But the challenge is almost everybody in between is what we call a gray hat, which means that, you know, they're going to have their own motives and they're going to do what is in typically the primary of their financial self-interest. And the thing with these, you know, hacker types in my experience, and I'm sure, you know, Dan, you've kind of seen the same is these are very smart people. They will all have kind of their own codes of honor or ethics, right? But the challenge is whatever that code is, it may not be the same as yours and it can be very, very gray. And a lot of these guys are very, very gray, right?

So, do I see them changing the color of their hat? Yeah, I've seen it many times. And that's the scary part, right? That's why, you know, at Immunefi we did so much work on bug bounties in in years ago to make those multi-million. It's to deal with this part.

Okay, let's always make sure that the incentives make people go white by default and not black hat. Yeah. Yeah, I guess that you I mean, you kind of answered my my final follow-up with there which is you know, are there situations that you've seen where, you know, someone probably had black hat intentions at the start, but then had that remorse afterwards or the fear of consequences that after having done so, yeah. Many times. Usually after the fear of consequences is, you know, nudged in their way, often via an email to their personal email, that's usually when they're like, "Oh, suddenly I'm very remorseful about my life choices."

Right. Yeah, tell me. Yeah. Yeah, watch your back, look over your shoulder, build your own paranoia because, yeah, I mean, it's it it's it's not going to be worth the the hassle like you said throughout this presentation. It's and it's feels a lot a lot more rewarding I would imagine to be be on the good side of things.

So, and feel like you're actually doing the community, you know, a greater good. So, yeah, I mean, this was this was an awesome chat and and I really appreciate you taking your time out of your busy day to educate the rest of the builders in the space on this super important topic. You know, we have another session coming up in 2 weeks, everybody. We're going to be talking about setting up bug bounty programs and we'll have our special guest Cantina join us for that session. So, make sure you tune in in 2 weeks from now on the 19th.

But thank you so much, Mitchell and the entire Immunefi team for joining us on today's session and educating the community. Happy to help. Be well. All right. Thanks, guys.