Inside DeFi's SEAL 911 Team | Niv Yehezkel , Michael Lewellen - Hexagate, OpenZeppelin

hello everyone this is inside defi seal 911 team with NIV escal and he is the the CTO of hexag gate and we have Michael Luellen head of solutions architect at opland Zeppelin thanks guys all right thank you all right so I'll kick this off and really quickly explain what is the security Alliance and uh seal uh so you might have noticed we're both from different companies uh in the security space uh and there's many other security companies like us we all have our own focuses we work on different commercial projects and many ways we might compete for the same business uh but there's a lot of common problems that this industry has faced for a long time in security that requires more than just individual companies uh trying to fight it so the security Alliance is an initiative that's been spearheaded by Sams who's the head of security at uh Paradigm many of you have probably been following him on Twitter uh he's a well-known Anon who's been fighting for you know and participating in a lot of security incidents over the last several years and he along with many others working with them noticed a common Trend that it was very difficult to create collaboration between different companies between different entities often it was just working through people who you knew um and so over the last uh 6 months nearly year now uh he has been working with many others to develop a nonprofit group called the security Alliance that brings a lot of these companies together to Serve and Protect the web 3 Community as a whole so rather than having to focus just on uh you know my clients or niv's clients or any specific uh you know barriers to helping someone when it's a broader community at risk uh we now have a single group can that can act uh for the benefit of the entire ecosystem and there is plenty to do uh we're going into a new bull market uh lots of security issues continue uh to be coming out we had the the hack with senica that happened very recently uh and we are sure there will be more to come despite our best efforts to try to prevent it and we're going to have to be ready as a community not just as individual companies or contributors so um that's what the initiative is all about uh Sam unfortunately couldn't be here live because he is anonymous uh so we are here in instead to explain what we've been up to and why if you are in the security Community uh you should be getting involved so if you want to learn more go to security Alliance uh.org um but right now we're going to break down the three initiatives and talk a little bit more about that so before I jump into that any further I'll I'll give a quick introduction to myself and then love let N introduce himself um so I'm Michael at open Zeppelin uh as you heard I'm the head of solutions architecture which means I focus on a lot of our our biggest clients uh I do a lot of security work with the compound Dow as their security partner um and then we also do a lot of work with uh layer 2 rollups uh we have a smart contract library that is uh very widely used and sometimes we'll have security issues with people that use that Library such as third web which happened a few months back uh where we have to ourselves be involved in a lot of community outreach um and then we also have a security product that we just revamped this last year Defender um so lots of things and lots of things I might compete with others and so from there I'll hand it over to nive to explain what he does at heate thank you uh so I'm n I'm the CTO of hexag gate uh and co-founder uh similarly to opens up being a security company we're another security security company in the space uh we are providing real-time security uh and detection response for protocols asset managers chains Bridges uh or basically anyone working with smart contracts uh in the space uh we are working with more than uh uh 20 clients with the biggest names in the industry right now protecting over a few billions of dollars at risk and for us as uh Michael mentioned like being part of Ste number one is like very uh uh powerful because also we are connected to this uh a lot of really smart people like Michael and his team and others it also helps help us to uh actually uh protect our customers even in a better way and also like the broader community yep thank you so seal 911 that's the initiative that you've all been hearing about in the panel so let's start with that because there are three big initiatives that the security Alliance works on and seal 911 was the first and still probably one of the biggest impacts that it's had on the community um so if any of you are on telegram there is a bot that you can contact uh it is literally seal 911 Orbot and if you reach out to it you can uh get in contact with a team of security researchers that include Sam uh pachio and then also people from open zeppin from hexag gate and many others all potential competitors but work together in this particular group headed by samsoon to work for common uh security issues that uh people might come to us for so one example was uh in the over the last weekend we had someone that came forward about a bug and a well-known protocol there was no bug Bounty for this protocol there was no contact information this person could find so the only thing they could do was turn to the 911 team and say hey uh I found this bug what should I do uh and what happened is uh luckily some people in the team had the right contacts and like I said in a lot of cases it's who you know um and to a was able to get them in contact with that protocol team get the bug reported get it fixed and then that person even though there was no bug Bounty did end up getting a reward from the protocol for their efforts because seal 911 was able to represent the fact that this person had come to them to report it in good faith um so that's just one example uh there's many others uh ni has been a pretty active participant so love to see examples from you basically as as Michael mentioned like we're constantly getting like tickets and like people from the community like surfacing out like different types of uh things happening uh on the ecosystem so another example uh just recently maybe like a week ago where someone found like some vulnerable comp compound Fork due to some vulnerability that was already li like already like known to anyone but specifically this project was not aware most likely for this well-known issue so this guy has been brought to the c number one hey I was trying to contact the team I couldn't like really reach them uh there is like uh this and that uh funds currently at risk and the SE number one really acted so fast like a few people saw that saw the issue uh specifically it could have been mitigated like without like whad uh uh without any like really wh hat activities but rather like a way to mitigate the issue and I think this could not like this could have turned like for uh for some malicious actor like actually attacking those funds but rather like because this guy acted so fast and because the se11 people are always on then we could really uh mitigate the issue and like save like a few millions of dollars uh just like that U by just communicating like with a with a guy over Telegram and closing Loop is really really fast yep and we all know telegram is the lifeblood for all crypto Communications so uh that's where we uh ended up putting it but I mean the fact is is that we've got over a dozen folks uh this are there's a repo on GitHub that lists all the contributors and the biggest thing is that information sharing is really really uh sensitive if you found a bug uh you have to be really careful about who you share that with so having this common group of researchers who have committed themselves to not share information for their own commercial gain to do exploits to in any way misuse it is a really important commitment to have so that when people turn to the seel 911 team for help they know and can trust that they are going to be careful and trusted with that information and that they are going to be able to help and in a lot of cases uh it's just a fact of teams don't know where to turn protocols don't know where to turn we've had you know protocols that are being exploited just not knowing what to do and need to reach out all the way to hey I think my wallet has been scammed I think my private key was lost in you know maybe the last pass hack or my phone got stolen or all these other things like what do I do next how do I save my funds um because in a lot of these cases these hacks might be ongoing uh they might be there might be fun that could still be saved there might be ways that they could actually report the crime that has been committed to the FBI or to certain fishing sites and actually get some of those funds recovered or at the very least make sure that other people uh who uh could be scammed are not going to be because the word is out that maybe this link or this website is actually a scam and they should avoid it and so there's also been a lot of work on fishing detection where we've uh the team's been able to connect to scam sniffer chain Patrol uh the metam mass team and Tay's also been a very good uh participant in the whole program among many other people that I I couldn't uh possibly list through but they're all on GitHub and uh if you want to give them things then definitely give them a shout out on Twitter so the team's been doing really good work it's honestly one of the areas I haven't been as focused myself but I'm always really impressed whenever I see a success story every other day about someone that uh really was helped that didn't have anywhere else to turn and that's that's the whole point is we have a lot of knowledge to share with the community and it's really important for us to be able to actually provide that when they need it uh and and usually in the speediest way possible because with when hacks happen speed is often the biggest factor and how many funds can be saved um so I think that's a good overall picture of like that initiative and that's one that I think we're going to continue to see a lot of support in we'd love to see anyone that wants to step forward and help again security alliance.org if you want to learn more and get involved uh but the other initiative that has uh been going on almost as long is the uh the seal war games uh which is really fun uh it's more about prevention than response uh and making sure that a lot of protocol teams even the biggest protocol teams are prepared when you have a security incident uh and so compound where which is one of op spp's main security Partnerships uh was one of the first recipients of this so we actually did a simulation where uh Isaac pek who's been leading up this initiative and doing great work set up a main net Fork of compound which basically means let's simulate everything going on in compound but on a separate private Fork let's run all the monitoring infrastructure that you're using to detect security issues and see how that works and then actually introduce a bug and a bot that will exploit this bug and see what happens in this particular case it was an oracle failure so oops the prices are being misreported someone's going to take advantage of that to start making trades to their advantage how quickly does the team detect it how quickly does the team figure out how to respond and then uh how quickly have you resolved the incident and how many funds are you able to save so in that I think we had a we were able to detect the issue within a couple minutes of it occurring or actually within like like less than a minute we're like oh we can clearly see the Oracle reporting bad prices we can clearly see someone starting to exploit that um ironically the thing that took the longest time was not detection but the response um and we knew very quickly what we should do we should pause that protocol that that particular market and compound to prevent the loss um but then we have a a multisig of about four out of seven signers that need to show up and actually sign the transaction deposit so getting all the multisig members to sign very rapidly um in quick success was actually the biggest bottleneck um just a show of hands how many people here have ever used a multisig or a safe all right there's a hand um and I'm going to guess Has anyone used it in an emergency situation where you had to sign it in less than a minute or two with multiple signers yeah it's something that most teams even some of the biggest teams in defi haven't actually done before until they actually have to and then that's the point where the timing really matters so getting to do it in a practice environment where you actually see how quickly can you get to your keys are they on your wallet or are they on a hardware device how quickly can you plug it into the computer uh do you have your computer on you your Hardware wallet are you traveling uh are you in the right time zone uh or will you hear the notification where someone's asking you to sign up uh and sign that transaction that's these are all the things that usually teams don't actually have a chance to think about until it's happening so these simulations are a really good opportunity and we took an overall response time of around 40 minutes and we're able to cut that down to about to less than 10 minutes on average to get multi6 signatures by introducing a pager system ating out some of the multii members uh and generally just practicing hey this is going to be the step that we take to quickly sign and quickly be ready in the future uh and since compound uh they've also done a lot of simulations with both a and year who are very big protocols uh have very good teams with security but still um having an outsider come in and put them in a uh situation where they have to respond and they don't know what's going to hit them uh is really really valuable and it's something that we definitely encourage all teams to do and seal continues to provide this as a service if you go to the website you can sign up for the weit list uh to receive one of these drills for free um and it's generally gauged by how big of a protocol are you how much of an impact on the community would there be if you were hacked and you didn't respond appropriately um but even then a lot of the things that are being learned in re war games are being regularly published on notion and other resources so that other teams can learn how can I run simulations how can I run monitoring um so I know on the monitoring side n this is hexag Gates bread and butter uh just generally what are the things that a lot of teams aren't thinking about with monitoring or are just really important to think about when you might be setting up a war game yeah so uh I think also to add like on what you mentioned like around in response and having like a really uh precise like roles and responsibilities when an emergency happens every now and then also like the C 111 uh members participate in war rooms where really Protocols are not prepared at all whenever like something big hits them and then it's like one of the the most chaotic like things you can really participate in like people just do not know what to do like trying to contact exchanges because like maybe like the attacker would deposit some funds into the exchanges try to contact Circle to to freeze the funds and so on like lots of like you know different kinds of ideas uh which uh in which if you are not prepared up front then you really don't stand a chance uh and uh for uh back to the to the question or like monitoring so basically um like the the mindset that we have when when we um Pro protect the protocol as heate or like part of s one like literally like the same mindset where we uh first thing is like kind of map all the threats in the protocol like understand how uh where exactly like are we expecting some failures to happen for example like an oracle deviation or like some bad Oracle updates whatever or like uh just uh generally like uh in the contract you know audit might have have missed like a couple of uh bugs which is completely uh uh because all it's like are not just not enough uh to cover like any type of bug so when we work with protocols for example we set up monitors for anything really uh that might uh that might fail which is also like a you know monitoring which is um based on machine learning that tries to detect unknown threats and also like the known threats like any Administration changes due to maybe like private key theft any like deviations and oracles and so on once we have all the you know the detections in place then we start working on instant response where we do this tabletop exercise we set up all and responsibilities for every uh Personnel in the team who is in charge of comms who is in charge of like triaging the bug uh who is like the uh the one responsible for sending like or even automating uh some action uh such as pausing the protocol and so on uh so uh all you know like only after you have this full-blown detection and response only then uh we believe that uh then security is really achieved because with just detection uh itself as Michael mentioned you have the best detection but then 40 minutes later you try to assemble all the team from all the time zones to sign some uh transaction obviously it doesn't work so you have them you have to notify them keep it in the loop with it some pager or something and also on the other side of the response like you must have like very clear uh steps on what you're going to do just besides detection so it's kind of a combo of both detection and uh and response 100% and best way to know for sure is to run through as realistic a drill as you can with uh maybe even a few team members not knowing it's a drill uh all the better um so with the time they have left we'll talk about the last initiative that has recently come out from the security Alliance uh which is the white Harbor agreement now I'll say that we've already talked about you know your prevention which is ideally having run through these scenarios and being prepared um when you maybe find a bug or you know are trying to figure how to fix it maybe going to seal 911 um but but there are some scenarios that have come up such as with the Nomad hack uh that happened year or two ago where a security researcher might come across a bug and they might be in a situation where they know that bug is going to imminently be exploited uh and their only options at least in their mind are either I can let it be exploited and let a black hat steal these funds or I can act to exploit the protocol myself to to safeguard the funds and then give them back to the protocol now that a very dicey proposition from a legal ethical just make sure you get it right sort of perspective um however these are things that have come up such as with the Nomad white hat uh exploit where a lot of people in the community were coming forward to help to save funds uh and then in other cases where you've actually had uh maybe a transaction in the menol that you know is about to exploit the protocol so maybe you can front run it to safeguard the funds in that way uh so these are very specific examples they definitely don't apply to every scenario and in many places it's it's probably better to report a bug rather than wh had exploit it but for those scenarios um what uh the security Alliance has put together is a white Harbor legal agreement that allows a protocol to publish uh a legal document that says if you follow these procedures uh We Will consent to you white hacking our protocol to safeguard funds uh and you may send them to this recovery address specifically uh so you know I'm not a lawyer uh no one on here is but this is a legal document that is available and is currently going into open review with the Community to receive feedback to see what protocols might be interested in implementing this and what white hats might be comfortable exploiting protocols in these very specific scenarios um with this ingrit in place um so we're currently collecting feedback on that now um I'm very interested to see what comes but I I think the most important thing there is uh just making sure we have as many tools as possible and that even if we get to a scenario where a white hat is necessary uh we've done everything we can beforehand in terms of getting you know doing good testing on the code getting good audits getting good monitoring getting good instrument response policies all the way down to okay there's now these very specific scenarios where a white hat might be necessary um so in all the the purpose of the security Alliance is to work on all these initiatives simultaneously to better have sec really defense in depth for security across the community um and a lot of this work is coming from different places uh open zeun we feel very confident about our solidity and smart contract auditing skills hex gate is very good at real-time monitoring um in some cases we might be competitors but within the security Alliance we can work together and pull our resources along with dozens of other great security researchers uh that have already put in a lot of effort and uh saved a lot of funds and helped a lot of protocols sleep better at night but at the same time it comes down to everyone's individual effort uh and making sure that we collectively make the best use of it so um that's for the most part the remainder of our time if you are interested in getting involved or if you want to make use of any of these resources or in any way learn more go to security alliance.org uh and that is a great place to get started we're always looking for feedback we're looking for new members to get involved to help contribute their security knowledge um and to just tell us how we're doing and how you know what are problems in the uh industry you want us to be focusing on next so uh thank you all for your time and uh yeah we hope to see you at the next one