Vulnerability management: Zero-Day, Today & Tomorrow | Tim Pechersky, OpenZeppelin | ETHTaipei 2024

hi everyone um June 2016 let's roll back to that day uh that month um when um there was the DOW hack uh happening which ultimately caed very arguable um event of foring ethereum um you know I wasn't uh I didn't start my timer I wasn't uh really a blockchain guy back then but it really took me to the level of thinking wow that's like that's a case uh how you U manage the vulnerability and um that's a lot dictated my uh oops Yeah further uh career in open Zeppelin uh trying to figure out how to basically uh protect the open economy together with amazing folks out there and [Music] um helping other um to stay safe and secure and mostly doing this in the application layer uh in in the uh smart contracts so for those who might not really know about what's up uh about us so opens up in develop smart contracts as a public goods libraries we are famously now for solidity and working on other fields now on other ecosystems uh we've done over like hundreds uh of audits by now and all of this experience led to our actually like more uh software uh a service platform Defender which uh does the code scanning today uh to also detect the vulnerabilities uh and um allows you to do the transaction management uh actually you can get the full audit cycle inside and um it's pretty amazing uh for uh for whole security cycle including monitoring and incident response so famous fir famous law of Murphy or otherwise saying if there is a probability of something going wrong it will go wrong at infinite time span it will go wrong and um some proofs for that last year year we've seen uh great work by you know fire blocks and Hiking MPC wallets and uh people say not your keys not your coins well it's fundamentally realiz us on the fact that it's not really your keys it's just a number someone might know them right um another aggregation of uh cool hiking techniques in our blog um and we we can see here uh really different categories right so it's like compilers protocols uh smart contracts infrastructure and like logic uh so between the contract bug and infrastructure vulnerability which I would call that that you might have like two per two perfectly well uh working contracts no bugs inside included however you combine them together in some pattern and boom you got a vulnerable pattern and uh you know as we are growing and maturing ecosystem I uh would say like we actually going to see more these kind of vulnerabilities over time where um like the contract bugs are eventually like the industry matur will become less uh another topic to outline here is that a lot of these across all of the categories can be uh classified as widespread vulnerabilities that is there is a multiple uh peer to each other uh agents uh affected by the vulnerability over some ecosystem and uh well that like leaves it to the whole Next Level actually when you're trying to manage this problem so in a nutshell like you know how's how the management of vulnerability looks today I'm going to talk tomorrow about tomorrow and what we see in the future later on but here is like today so we basically most of the time we have like maybe pable upgradable contract and well if you lose your keys or your management lost your keys of your keys then you might have like access management model as well uh however mostly like you're really like you're running around like is it possible can you do the upgrade can you upgrade fast like do dow needs to vote for that are there time locks Etc so and what we do is basically we know we set up the disclosures protocol identify access uh you know uh and when really like you know wake up Neo it happens uh you really dream about this magic button about like okay I can just selectively disable without permission some kind of non permissioned way the vulnerability done however really most of the time the case is actually you'll have like a white head rescue I invite you to notice how many arrows are connected in the white Hy rescue block there um and the challenges it's time critical uh ping contracts you really have to evaluate like do you really want to puse the protocol let's say uh well upgradability is a very permission thing to do and uh yeah mitigation is not standard thing you don't have this like selective button so widespread vulnerabilities another topic to cover is that well imagine you have uh some ecosystem and you have Carlos Alis and Bob and they all are kind of pure to each other and they use some dependency now if Carlos suddenly discovers that there is a vulnerability in their common infrastructure it actually super is very important that he has some kind of ability to do the responsible disclosure so that others in the ecosystem are not getting affected by that and you kind of coordinate the whole ecosystem in us uh in in an United manner right now uh the problem is that actually there is some kind of game theory behind because uh most of the time like say if you're having similar patterns in your smart contract infrastructure with your peers it kind of implies that maybe you are like actually competitors and you're not like that uh caring about them maybe even and uh and then also you're actually having vulnerability yourself perhaps so you're like maybe stressed out you're trying to do the media post and you might do something like that you might go and make a media post on uh social media before or the provider or ecosystem which is here like manages the whole thing and helps the ecosystem to solve it together now the moment you do the red any of these red arrows Alice and Bob they are not happy at all because now every hacker out there knows there is a problem and uh perhaps even can see how to do replicate that attack so that being said it's really important to incentive I from the ecosystem the um need for du the responsible disclosures so uh there can be like multiple ways to do that but basic one is just you know we we're community and we're work as community and that's great it's actually I think this is like a this makes a lot in the security so having a community codes code of conduct and using it smart in a wise way say you know so many ecosystems today they allocate the grants to their participants but well you know if you have let's say uh if you package the audits in the grant you're actually building the auditor expert field for your particular ecosystem for the particular notion there must be there and when you know it hits the fan that problems happen you actually have a really like widely uh wide notion about the ecosystem on some team of Auditors who can actually step in and you know help out solving things and you need to support in media and you need to step in when it's needed and uh all of this is quite critical and it comes into like you know defining your ecosystem emergency protocol basically what do you do what happens because of from previous Slide the caros he need a help at that moment and everybody do and the ecosystem is the level where you actually solve that it's a step above them well one practical example is uh recent uh vulnerability management we did with ERC 2771 when combined with multicol contract so for those who are not really aware how it work that um basically if you do the uh multicol through the trusted forwarder and uh you're basically uh can get this uml diagram and oopsy and uh what it allows you actually it allows you to like do the poofing and execute whatever you want uh from uh uh the victim's contract so for our case you know like that was a like first step we had to do is assess a sure and uh work out and understand what's actually happening uh we also at that at that point we already uh had to do some acknowledgement to the community um the next step we did is actually we set up the monitoring and triaging uh in Defender so you could actually go to the defender scan your contracts repository we added quickly the ability to scan for that particular vulnerability and you could see if you are affected or not uh and that moment of course already the uh disclosures had to be made as well and explain how to do the vulnerability management for ecosystem the time was click ticking and uh we were communicating with uh uh some initiatives like seals and security Alliance um to help uh as much uh affected projects out there as possible um you know that being said like okay this is what happens today it's not perfect we know that and uh it's actually quite tricky to manage and uh even in the previous example there was a funds lost on some projects uh not super a lot uh but it's uh something we want to minimize remembering that there was like some some time before like 2016 we did like a hard forks and um today we are like kind of like learning how to mitigate but you know what's what's tomorrow how do we get to the level we want to be like safe and secure for our users so I really like this example with airplane and engine fire basically you have to have a checklist for everything right but what I really like even more and actually I have a lot of background in the more traditional industry uh so I kind of Al know and read a lot of standards in my life so you know you know why the cars really work well for like 10 year and you can actually drive and sometimes even release your steering wheel and trust it's not going to fail why does it work how does it do that right so the answer is standards they have a lot of Standards so you see the Ford GM Chrysler and this is just like you know European American companies they have a lot of Standards there if we take a look these standards are actually being written by most of nonprofit organizations which are collaborations between the industry experts in all of that Fields uh including like you know even like doing the auditing Management Systems uh through the ISO and that's not it it gets like really bigger because like you take like one uh standard for component re reliability and this is like you know how big it is each each each box here it's some separate test with like book to read about really and that we don't really have that right um what we have is something like this and this is what we're are like really relying on the two main pilars today from which we are hopefully are growing to the mature industry and mature uh standards set which will allow users to feel safe um two of those are Community principles and you know the back bound is imuni it works great because there are some Community um principles and uh actually like even like most of the disclosures they do happen actually in quite responsible manner people do like approach us even if they are not directly our clent client asking for help because they want to do the best for the ecosystem because truth is that most of the people are really kind nice people out here like this is why we're all together and I really enjoy speaking to you here because I truly believe in the uh you know good Sparkle inside everyone and then there is also like you know if you take think about like regulatory Frameworks and uh like like funds recovery and like you know let's say criminal and law uh systems they are actually some kind of social rating systems so it's just a matter of we would probably find a lot of user who would put more of this on chain um and it seems that we're going in that way actually now the smart contract security uh it's uh really uh something that still is evolving a lot and U account obstruction changes things uh mainly because not like we have not not because we are getting like transaction batching or pay Masters but rather because we are getting this kind of consensus in our society about that uh smart wallets are the way to go really like you might use gnosis or you might use uh you know da or you might use some new emerging smart wallets but uh it's the way really to go and store your funds on account obstructed wallet software wallet basically and you can do a lot of things there like one of the really like getting exciting for me is that you actually can uh make more user Centric asset ownership models which will ensure more Safety and Security for the end user and uh you can actually plan for disabling vulnerabilities and you can actually build the whole apps and infrastructure infrastructure Registries some of that right uh so so one one one thing with like pre previous example of uh address poofing The Meta transactions was that there was no particular responsible authentication and authorization layers and people do like really miss that point that these are two different things you might want to understand who is sending the message first and then you might want to understand is this person authorized to do that and the classical Security in web 2 as well like it's it's welln things that you do that and you have the intrusion prevention systems which uh you know as today we have a lot of conversations about blobs and uh gas prices going down for l2s it means we can put more Logic for security on chain we you know any method can be anticipated as point of failure basically you can rra your logic like I mean proxy proxy contracts are already doing that basically you have a fallback which sends you to the logic which acts as a wrapper for uh your state now the same way you can do another wrapper there and do like authorization you can do another wer which like authentication and you can do the third and you can a lot do a lot as much as you need and this allows actually to self Sovereign these layers of security uh and The Preserve non Priss Athos of crypto Space by basically saying okay I need like this this this layers to secure my ecosystem and they can be totally orthogonal to each other in terms of who is actually managing which layer and how they are actually decentralized um so secure apps so this is very simple model like which we use today actually like so if you have a d app you have an asset you have a user user will try to do something with a D app right and he will have to approve the asset and the D app will start calling transfer from uh on behalf of the user right um but there is something missing and something very important like when you're going do like say you went for UNIS swap first time in your life or your grandma what will your grandma do she will ask you probably like is it safe to use right so you're on trust attended and the the D app actually is going to do the same if it's a dent app it has to do the qyc actually uh or at least qyt or something to check uh it like in recording previous slid maybe do the intrusion prevention detection something is right um and we could actually you know put this on chain and if we put this on chain we are actually automatically on chain already saying by Oracle which app is secure which app is not once we do this we actually are able to do the automation Cycles inside on on the chain based on some metrics as well so kind of gets amazing from there but right now the trust empties are mostly the um the phys physical or offchain U data um yeah another uh thing looking into the future is more like user Centric uh models so uh you know virtual machine is just another evm is just like another virtual machine virtual machine is just machine it OBS to some same principles and if we recall like how the operational system works right they have a they they operate on machine in and if you imagine like say you're a user and you have a door and the door stores the some asset and the door is Wi-Fi controlled so the person who has control to that Wi-Fi module on your computer we can say he has pretty much control over this um but if you install some app even if you use it it doesn't necessarily mean that that app has direct access to that module there will be an operation system which will manage the permissions and uh well if we have software wallets which I call an age of account abstraction then we can actually treat them as operational system which manages API permissions to external apps and uh that allows us actually to actually if we even if we drop that app inside of the secure registry and we have secure apps in that registry we already at that point can assume that all of the apps in the registry have a same trust assumption between them which means they are like let's say you cannot randomly upgrade it perhaps but uh inside of that EOS system we're quite sure that uh wallet cannot do like something on their like arbitrary it will uh have a very deterministic API which uh apps can trust as well and uh well this allows us to build a security models you know where you could actually split assets from in ERC defined interfaces say erc20 doesn't Define you how you store the data about the assets it just defines you the interface you don't necessarily need to use the same contract it's uh it's up to the implementation which can be changed and then if your user wallet becomes acting as operational system then it can actually talk to the secure apps registry and uh could I even like basically C for like you know I want to install a dii d app what's up with that like which one is good uh and uh it can go like up to the implementation for the applications as well uh where you know you you want to preserve upgradability that's fine that's great we have IP 712 and uh there is like this versioning you can actually in the apps register you can figure out of doing the um semantic versioning right very familiar thing why don't we do that um and yeah an amazing thing about that is also you can actually uh set up the a lot of circuit breakers in different uh trust flow and uh that would allow you to actually really narrow down any vulnerability to particular uh block particular particular onion of security which you can disable um yeah so that's quite it um I think we have some time uh if there is any questions I'm happy to answer and otherwise uh feel free to reach out out uh if you have any questions any questions from the audience hi um thank you for talk and I was wondering that as you have mentioned that um to detect the vulnerability is a like important part um or because I'm a developer so when my contract might be at risk and there always more uh hacks or more vulnerability um like um discover but uh how what are the practices that might um enable us to detect the vulnerability instead of monitoring what is happened already um is there some um something that we can do instead of just monitoring practices for detecting yeah um set up the monitoring uh so and to do that correctly frankly you have to early Define the requirements and criteria for uh efficient monitoring system so for example uh let's say you're operating the bridge which does like mint burn operations is actually quite tricky to monitor for invariance State across chain um so defining cly like what kind of events and what kind of structure for good monitoring you need uh for like reading data off chain is today is useful and uh then installing uh monitoring over uh the like expected uh State operation so expected flow of uh uh data expected value changes uh and uh also we have a forto network which uh is a spin-off from open Zepp in is decentralized monitoring system where you can actually enable um AI attack detection it shows uh it's an AI model so it can like do the false positives but it shows uh a lot of results when it sees real attacks happening uh before they actually happen yeah I um like heard about orta and they mentioned that some maybe some hacks happen there there will be some steps before it happens so they will have a monitoring system to do that to maybe predict that something is going to something bad is going to happen uh I just wondering that like um because uh because open got a Defender so it uh enables us to do the management and also some monitoring and you also mentioned that uh when there's already a hack happen and you can scan like retrieve some um I don't know signature or something that you can know that if your contract has all also has the same vulnerability uh I was wondering that if this is already integrated in your system or you have other plans so in in case of uh code inspector which is our uh scanner tool the way it works is that you integrate it to the GitHub and uh it creates uh reports on your GitHub uh usual on appears and uh in the case which I shown what happened is that we uh the the moment we saw this happening we started adding the new check because in content Spector there are two Fields basically we have static uh rule like set rule set rule set of rules and we have an AI model so for that particular case we added like a just a rule which was checking and that was a manual step we had to do because uh thei model uh yet didn't see that so that would be a like source code level or a bite code level yeah it's a source code so you just plug it into the GitHub repository thank you any more questions well uh first of all awesome presentation I love how you turn the um you know that incident and then like come up with the framework that is really lovely uh I work for quam and I actually like when you guys are doing the the investigation we in parallel also did and it was like very fun and I was I think there's one thing that was very challenging and you know like would would love to pick your brain on how you guys like approached it because um how we did it was like basically looking at our you know clients code Etc and scan proactively uh but then there's the aspect that I absolutely love about your thing is that the community security thing right how do we reach out that is something really hard and I I'm wondering at the time I I know that you guys have acknowledged it I think we we sort of uh I think there was a previous Twitter post from the other one was first acknowledged and we from there we sort of like deduced and then you guys like Ann announced almost at the same time we figured it out and we're like okay but I was wondering whether you know like I don't know like what's the step or what's your experience or learning from that because that was a very very nice uh not nice incident sorry but very interesting incident and I love how you guys handle it as well yeah that's a good question indeed so the reason why we also posted acknowledgement early was because there was this another post and it actually started the race it was a panic so we had to like make some rare assurement for the community um and then the management part was actually uh to to part of this you saw on the slides where we actually were doing the post messages and uh trying to coordinate CommunityWide but also there was a lot of uh work happening just in telegram just reaching out people just finding their contacts just checking like manually like report like who can be affected in this and um this is why I also mentioned the ecosystem uh emergency protocol because I see uh as real need to have a protocol version of doing that uh and this is not something that projects can do themselves they can do the threat modeling they can do incident response drills which like we do the the projects but if they are affected by something that happens from the higher uh level um it has to be uh managed through that protocol there absolutely lovely thank you so much thank you all right and uh great thank you for having me here I really enjoy that