Red-Teaming Crypto Wallet Operations From TJ Connolly Fireblocks

[Music] cool all right welcome everyone uh welcome to the captain ethereum stage first thing that came to my mind uh Chad vitalic he might not be so happy about you know what's been happening in the last uh couple weeks in the the crypto ecosystem but uh you know my talk I'm you know quick intro on me I'm a cyber security engineer uh been in cyber security for over a decade been with fire blocks for about 3 years now uh and today we're going to talk about red teaming crypto wallet operations so very timely with everything that's happened in the last week with bit so we're going to talk about the threat landscape why red teaming why this is something that you know if you're an onchain protocol a business that's operating on chain why this is something you should invest in and focus on we're going to talk about the threat model for wallet operations how it's pretty unique uh compared to maybe some other traditional it or cyber secur systems we're going to talk about the latest and greatest hacks radiant capital and bit a lot of good lessons to be learned from them uh and then I'm going to give you uh sort of a highlevel practical framework to take back you know to your teams on questions to ask and allow you to perform your own red team on your wallet operations so what's going on uh external nation state attackers right they they've been busy right uh the human continues to be the weakest link uh with targeted fishing attacks social engineering you know we always have the possibility of like physical attacks like a hammer attack uh and coercion right these are all continuing to be successful um with the rise of nation states like North Korea we even see internal or Rogue actors uh masking themselves as employees and getting into these organizations and so some of the largest breaches uh within you know these are all within the past six months months or so uh these have all been around wallet operations right these weren't smart contract exploits yes there was a smart contract component to it but it was generally taking advantage of the humans involved in wallet operations and taking advantage of their trust so again you know dprk North Korea Lazarus group um you know I think I don't want this to turn into a a a safe bash because it's really not meant to be um but I do want to kind of sound the alarm here right if you are operating a safe uh wallet uh infrastructure you know for your team uh you really need to invest in security now right because dprk has weaponized exploits for this um and they are emboldened because look at how much money they just made right they're going to keep doing this as long as they're making money so this guy's waiting for you so why are we focusing on wallet operations right weakest linol Cy security always will and continue to be the humans involved right again this talk is focused for if you're a DA uh a protocol a foundation or your business operating on chain you have wallets and you know you need to uh do things on chain with those wallets uh you need to have controls around you know who can actually access those and do those privilege uh transactions so multi-party transaction approval and signing right very distinct how I break those two up those should be the primary security layer but the defense and depth of the whole wallet operations can go much further and that's what I'll cover in the talk so you know I I've been working a lot with traditional financial institutions and they have this concept of maker Checker um I think it's a pretty good analogy for uh the type of control uh that you should have for every privileged action that could potentially result in you know either loss of funds or you know somewhere down the tree of like that attack life cycle of of resulting in loss of funds right you need to ingrain your head that every preg operation if there's a maker right that's the person that initiates the privileged operation then there should at least be multiple Checkers for that uh initiator right and this goes throughout the entire life cycle the other issue here is that you know uh in terms of you know things that like happen with radiant you know this is more for like the daps or the protocols uh or the blockchains uh most security testing and audits they're just F focusing solely on the application code they're not really focusing on how that the you know the backend wallet operations is actually going to uh take place and you know typically the weakest link bug bounties don't really Co cover this type of threat either um you know uh bug bouny mostly Black Box really in order to test some of this stuff you need to kind of open up Pandora's Box understand you who are the privileged operators uh in this organization right who are the ones that have signing capabilities who are the ones that have approval capabilities and then create your threat model from there and you know this is top of mine again with radiant uh you know they had pretty good audit reports right their audit reports were pretty clean it was a you know trustworthy relatively trustworthy protocol um but that didn't stop them from you know falling victim to this type of attack so what is teaming uh for those that may not be familiar uh it's really about simulating real world attack conditions to test security defenses as well as responses so red teaming is really you know I like to think of it really as a a sort of practical application of you know what a threat actor would actually do targeting uh you know a wallet operation and it's really around addressing real risks and mitigating real vulnerabilities based on the likelihood of exploitation right so this isn't some academic or uh you know uh compliance framework or something like that this is very much focused on what are the actual threats what are the attackers exploiting and then how do you mitigate those so for types of red team assessments you know uh in traditional cyber security I think blackbox is a little more popular where you know if you're trying to hack you know a Fortune 500 company you'll start from the outside you'll send spear fishing emails to you know the domain I don't think that's really relevant here because you're going to spend a lot of time and a lot of money getting good results from that I think these should all start today as white box and so this is where you know the red team whether or not this is an external you know consulting firm there's a lot of great firms that can do this uh where they come in and basically you know you you guys you know as the organization you open up your your systems your kind of controls um and you know you go through this this process uh these can also be done internally right and this is kind of the key of um you know why I wanted to give this talk is that you don't need an expert to come in you know to do this you really need to own this security process yourself uh and hopefully the the guidance I give you here can help with that so you know what do I think is sort of the best way to do these uh white box tabletop exercises so get everyone that's you know privileged in your organization into a room you know that could be a vulnerability itself um but you know get them in a WebEx whatever a a zoom call um and you know walk through these different threat scenarios and understand you know how would you detect them and how would you respond to them uh and if possible pre prevent them so you know what is the threat model there's a lot right um so I kind of broke it down into you know you have the basic pieces of who's administering the wallet right so this could be a hardware wallet this could be a software wallet doesn't really matter for the purpose of this right you need somebody that effectively is an admin on it right they're the first person in uh that can set up and add other users set up policies rules wh listing if needed you know all these types of things so on that level you want to make sure that you also have makers and checkers so that you don't have an admin who basically has you know unfettered access uh you know and could be a single point of compromise and eliminate the entire security model right so if you had a multisig for example uh but you know it's a three of five multisig but all five of those signers private keys were generated by one person or one system you know that's that that's that's that's a flag um so going down through the rest of the life cycle uh within Acme corpse wallet here transaction initiation right you want to do if anyone initiates a transaction put some Checkers there right are are you sure did you mean to uh initiate this is this exactly what you intended next is transaction approval so this is you know in a multisig this would be like the actual three of five um in other wallets like fire blocks we do this offchain because it's private uh and it also scales a lot better and you can you can enforce a lot more granularity and then you know come actual transaction signing time and this is really where the the safe wallets you know with with radiant and werx and biit uh fell flat transaction signing was performed on an interface separate from transaction approval so you know Bob Alice and Charlie here they saw the transaction approval they said looks good to me right this is a typical transaction that we're used to uh let's go ahead and do it but that approval was not cryptographically tied to the actual signing that then hit the chain of the actual transaction right so this is where blind signing comes up where you don't really know what you're signing because there's a separation between the approval interface and the actual Hardware you know the final point where the actual uh transaction gets signed and submitted on chain so breaking this down for radi and capital uh they they were uh you know they have a a basically routine smart contract operation for updating token emissions uh it sounded like they did this about once a week right but this multisig that did this let's say it's a 30 of5 uh the multisig that you know the three of5 that was responsible for controlling Emissions on a weekly basis had the same exact permissions as transferring ownership of the entire wallet of all the smart contracts for this protocol right so that was a very kind of egregious uh break of you know lease privilege right we shouldn't have like if you're going to be signing things weekly for routine operations that shouldn't also have you know the whole keys to the kingdom tied to it as well they did do a pretty good job and I really appreciate the openness of the team uh when they they wrote these blogs for the incident um you know they did best practices they simulated the transaction payload they copied it out of the safe wallet interface um they you know were using multi- they required a three three signers um but at this point right when they initiate the transaction and they're going through their approvals everything looked legitimate to them it looked like that routine token emissions that they did every week they didn't realize that in reality uh the attacker you know which was likely Lazarus North Korea replac that legitimate transaction with a transfer ownership and again that's the keys to the kingdom here of for radiant Capital uh they lost you know the the whole protocol $50 million later um and again the issue here too is that where the actual signing occurred so at the point where you know you would actually have the opportunity to see what is the transaction getting signed it's transfer ownership that happened on Hardware wallets that you know they didn't have visibility into wasn't decoded blind signing all those issues um so so one of the things that I I took away from this among many but uh you know just again in the the spirit of red teaming and practical guidance here diversification of approval and signing systems right so if you're operating a multisig um and especially in today's day and age where North Korea has weaponized exploits for for safe uh you should probably diversify your signers and your approvers right so I work for fire blocks obviously would love it if you guys use fire blocks but you can use right any other externally owned account as a signer on a multisig fireblocks is one of them there's plenty of other providers uh but just doing that right that is going to greatly uh mitigate your attack surface uh through that diversification the other issue here again I touched on this right it was routine smart contract operation uh for token emissions uh that was the same for transferring ownership of of the entire protocol um the other thing I want to C uh call out here is at the bottom is that I see this a lot with you know big teams that are uh operating shared wallet infrastructure structure they get signing fatigue right if you get a popup notification on your phone multiple times a day of hey I need you to put in your PIN code your Biometrics get out of UB key whatever it is uh you're going to start getting annoyed with that right and I call this signing fatigue um and this is really how you know you lose your level of paranoia and some of these types of attacks uh can get through undetected by bit all right this has been fun uh for the last last two weeks or so um the scariest thing for me you know just being in cyber security for over a decade uh it was one of the scariest supply chain attacks I think in crypto uh due to the fact that it wasn't you know it wasn't byid that was directly compromised it was actually a safe developer or maybe multiple safe developers I don't know there's still more information that needs to come out uh that were actually compromised right so um the I'll kind of table that for now because I know you know I called out down here uh safe's still working on the postmortem and and you know excited to see what they come from that but really again the issue very similar to radiant Capital here is you know bit was using safe wallet interface as a trusted interface they trusted that when they were doing their approvals through safe wallet that's actually what they would sign but the approval interface was compromised so North Korea uh hacked the safe developer they replaced the front-end code of safe uh in an AWS S3 bucket and then that code was only served specifically for buybits wallet address so this was extremely targeted right and this also tells me again I'm not trying to make this a safe bash but safe may be compromised you know safe safe was compromised by dprk uh in this scenario so uh the other issue again want to highlight right uh because um the approval interface is different than the actual signing interface the signing interface was Hardware devices um they were not cryptographically tied so again you looked at the interface the approval interface looked good but that wasn't actually what was sent to being signed right so those need to be tied together uh such that you know you can't be manipulated somewhere within that flow um and again right like there's lots of controls here uh that can be applied on chain uh for something like a multisig for like safe again I don't want this to be a safe bash I love safe uh I think it's an awesome product um but it's tough to have sort of Enterprise cyber security controls when you're you know a dow and an open source protocol and so that's kind of where they fell short here uh but there's there are a ton of tools within safe itself that could have prevented this right so safeguards safe modules those are basically policies uh and different ways to configure this safe uh in a more secure way and then also uh you know decentralized or you know trusted like self-hosted front ends uh that's a that's a big one so red teaming framework all right this is what I'm giving to you all I probably won't spend too much time on this because it's a lot uh but this is what you can take home take back to your teams and kind of go through these checklists and ask yourself uh and start you know uh finding vulnerabilities and and mitigating them um so again first step right you got to identify and prioritize them there's always risks right so we need to prioritize them um I would start with what's been working for the bad guys uh you want to make sure that you know you always have isolation of different admin functionalities and transaction types A dollar transfer shouldn't require the same maker Checker or approval scheme as transferring ownership or or administering the the wallet itself defense and depth right we always want that uh what happens if makeer Checkers are compromised um are you you know using a healthy dosage of paranoia right when you see one of your team members says you know hey I want to you know issue this transaction you know are you sure is this what you meant worst case you know are you being coerced right do we need to get on a video call to you know confirm that you're not uh Under Pressure uh obsc of wallet admins approvers and signers super important right uh um this is maybe more you know North Korean and and groups like that can definitely use this but you know this is really relevant for like Hammer attacks or physical attacks right making sure that you keep the actual approval and signing scheme of your team secure and private uh sensitive software and hardware systems Diversified right uh you know uh safe you can use as the multi- but you can augment that with you know something with fire blocks with MPC uh you can also use onchain controls like those safe modules uh do you have detection and response right can you pause your wallet operations if something goes bad uh can you remove compromise transaction approvers and signers could that be a denial of service or maybe uh uh an uh Avenue for like fatigue as well can you migrate your sensitive devices or Personnel um are you susceptible to signing fatigue and then you know very relevant to to radiant byit and you know blind signing right is what you see is what you s what you see is what you signed should always be the desired outcome here for any wallet operation right so you want to make sure again that the signing the approval interface right that's what you see is actually what you sign right so always ask yourself that and then also private key theft right keys to the kingdom but it doesn't always have to be game over that's where these tools like safe the modules the guards they can actually come in uh and help so this is you know I kind of think of this as like the filter you know fire blocks where we off we operate you know mostly on this offchain security side um but these are all controls that you should take back to your organization and say right are we doing these things uh you know are are we enforcing maker Checkers across all these before transactionally act before a transaction get signed and submitted on chain and then are we also implementing onchain security uh controls as well that's it