Dmitry & Andrey: Real-time on-chain monitoring | 14th Oct 2023 | Sunway University | ETHKL 2023

everyone uh thanks for coming to listen to us uh so my name is Dmitri um and this is my colleague Andre we work for metlabs uh which develops ZK sync era and overall ZK stack ZK stack yeah and we came today to speak about security operations so um and particularly Real Time on monitoring tools as it is very critical as was shown also by the previous speaker so uh but before we start uh I want to invite some Brilliant Minds uh who are present here um to participate in our code contests uh code for arena uh those are like the ongoing bu Bounty program and um just security contest that we are running right now so feel free to participate not right now please but probably just a bit later if you if you want to win some nice prices so um yeah today we're going to go through the um questions that are listed here and for the first question I'd like to invite my colleague to speak about okay uh how many of you use the monitoring uh in their your organization anyone okay yes no I do I work in security as well awesome yeah so why monitoring is important monitoring is our approach to security it provides us uh the big number of the insights for our activities and uh for our adoption of uh to the threats so just think about that and uh with this idea in mind uh I will provide you some insights related toh basic threats related to the smart chain and smart contracts uh area and blockchains so there are many uh threats related to the blockchain and smart contract ecosystem uh I just remind you about several of them so first of all this is the private private key compromise uh you need to understand that your private key is a master key and if it's compromise uh your funds can be easily withdraw own by the attacker or even the attacker can take the ownership of your wallet and use it for uh any malicious activities that he would like to do reiny so it happens when two contracts uh interact with each other so the malicious contract uh do uh some specific functions uh that are vulnerable in the in another contract so and use it for uh mostly for funds withdrawal uh Oracle manipulation so smart contracts are always uh fed uh with external data so and this data provided by the Oracle so if somebody can manipulate with this data uh so he can manipulate with the contract itself and uh this contract might behave uh in in any unintentional way denial service one of the popular say and probably the easiest implementation uh of the attack when the attacker just overwhelming uh the contracts uh functions or resources so the contract uh functions are blocked or uh just provides the delays with the response front raining this is quite a sneaky uh attack uh where attacker uh using the transparency of the blockchain and he can see when the specific transaction in the Q and he can uh put his own transaction before your transaction if the fee will be higher than uh that transaction that he uh targeted uh the last one this is smart contract vulnerabilities so this is uh flows in the code which can be used by the attackers uh to Huck to hug the smart contracts and drain the uh funds or gain some kind of the authorized access there so what can we do uh to protect again against such threats uh basically uh we have several protection measures so first of all this is the prevention measures this is our initial activities and our first a layer of the defense we using different solutions uh to implement access controls to implement some checks reviews and Etc to reduce uh the common known threats uh next we have a detective detective measures are our eyes so using monitoring and um animal detection features we can uh analyze and find um the malicious or some kind of um unintentional activity in our networks if um the incident happens so in this case the corrective measures kick in so we try to do everything we can to reduce the harm uh of uh the incident to improve our layer of the defensive and to update our approach uh and security response for the future uh B marked incident simulations in response uh in the in the between of the uh protection detective and correction because it's like a bridge because it provides you the ability to simulate the attack without any real harm so you can identify your flows uh you can idite your problems uh based on that you can make your own response and corrective measures for the future so if such attacks uh will happen so and yeah um basically in this presentation uh we mostly uh concentrate on detective and corrective measures um the main idea is to you that one of the measure cannot protect you against every threat it can enhance one part of the security of your uh product but it cannot uh protect it yeah so um the main point here that multi-layer security should be created by Design and uh just imagine uh that uh imagine it as a swish uh cheese so each layer is a slize of the cheese it has holes where the threads can go through so but when you stack everything uh together there are uh not so many possibilities that the threat go through the old holes and uh will be uh will become to as the incident so the idea is to create uh as many layers as possible uh so to achieve that uh you need to think um to start thinking about Security in uh very early even it was it wasn't roll out uh in the production so just um also note that human errors are happen uh the bugs can be found and uh the most interesting part when the bugs and the human errors uh uh happen simultaneously so you need to understand what to do in these cases uh there is no way to avoid it this is the main point that you should notice so um why it is important uh in more than two 10 years already right in crypto industry there are many tools appeared in The Helpful tools that provides us visibility um of the chain that help us to identify and monitor um uh networks products and uh uh projects so but still uh we see that incident are happened and most of the incidents uh alerts happens within the huge DeLay So let's um review several hacks from the previous the one of them are quite recent one is for the coinex uh another one uh related to 2017 it was the parity wallet multis wallet hack so do does everyone knows about that hack so do you read about that or okay so basically coin X hack uh coinex is a crypto exchange uh and this hack started from the private key compromise uh first drain uh around 5,000 hits was done in the middle of the day in 3020 all times uh stems will be uh mentioned in the UTC time zone so you if you read some articles in the internet it might be some different times different with my presentation so I just usec time zone uh basically this drain very uh very large right so and if the company would have the monitoring system in place this drain will be monitored and alerted to them but no was nothing was implemented so and then hackers uh hacker did different uh withdrawn using uh other cryptocurrencies so like BTC Tron Ripple and still those transactions are quite High um after 23 minutes 24 minutes from the first drain uh the crypto uh crypto security company uh cyers they uh tweet an alert um about these transactions and they tag the coin X account official account there and only after 4 hours from the event uh uh coinx responded and uh made the suspension of the withdrawal so the main idea here so if the company would have the propile letting they couldn't uh mitigate or block the first withdraw but they could mitigate the next withdrawals that lead to around uh 70 million lost of the company next uh it's quite a old one uh this is parity multi wallet hack July 2017 so this uh vulnerability was identified in the multi wallet versions 1.5 and later uh the attacker used this vulnerability to drain uh three wallets I don't know I think three wallets right uh so the first one uh was done on July 18th on 20 and 10:30 in the evening uh approximately and other two was done after 13 hours from the first one so uh the main point here that the owner of uh the second wallet identify uh that their wallet was uh drain after 4 hours so they check uh the attackers address and they saw that um this High transactions also hit that account so they did the notification for the ethereum foundation and many developers groups about this issue and about this problem and this group identify the vulnerability of the multi wallet wallet code uh here uh the great example when uh the monitoring or alerting of from one team could help uh the community the crypto community in total uh because uh for for this decentral decentralized attack um because uh to help uh to save the ponds uh the white head group uh did the same hack for more than 590 uh wallets that were V vulnerable uh and they save around uh 81 more than 81 million for the community so yeah they did notification about that after they finalized it so and everyone could claim uh their funds from these saved wallets meanwhile uh par team uh did uh the update of their contract and uh the fix uh the vulnerability so in the end of the day so once again uh as we mentioned right in the beginning of this conference that we are we are making the community so and alerting is not the let's say uh only the problem will the one company alert should be available when monitoring should be available and done by every company so and these feeds and information should be shared between the community so that is how uh we can reduce uh the number of the hacks and reduce the numbers of the stolen uh activities so uh yeah I will provide the mic he will provide you details about what to Monitor and how to monitor and what tools can be used thanks Andre Yeah so basically as was mentioned I I just want to add a couple of words to the previous slide as was mentioned the community effort is very essential and uh it builds the overall trust in ethereum in crypto uh and helps the adoption that's why um building the proper monitoring in in your company in your project you would contribute to the overall security of the community um yeah so now let's speak to uh about what we should actually looking for so basically it's all sort of the anomalies so first of all meets amounts amount of transactions and transactions amounts uh yeah so because um someone can be just drawing um lots of funds from your contract uh by bypassing some security limits with a tiny transactions on gas fees for instance um or for instance you find out that someone just deposited a huge amount yeah so your tvl grows dramatically so maybe it's money laundering um yeah so next one is the transaction times so imagine that you have script um that I don't know equalize the balance between the contracts uh just sends U money from your hot wallet to your cold wallet and this happens let's say every hour and at one moment of time there's a random transaction going to let's say the same cold wallet and attacker just right that the private key he compromised worked um it can be nothing it can be done for a good reason but it can be also a bad reason this anyway have to be noticed and checked um transaction participants and actually governance events participants so let's say you have some um spread um with Shamir cryptography for instance or a multi seek for for voting and um for instance you usually use three three of five yeah and those three people are pretty much permanent yeah so the rest are the backup uh imagine that three of them uh two two backup persons would be hacked uh and and the attacker can can can just use your backup um in against you so if if there's an unusual voter participating the process you should note uh that and just be sure that it's fine um and the transaction details we saw actually a few events uh when the attackers and the code um used a different transaction type like Legacy type type zero and type two and this is the uh technical side of things that is usually missed you still should be very precise so make sure that you determine what is the normal transaction for you um smart contracts themselves um um a bit different so so additionally um that there are some some common approaches right now to uh develop and to to to have um certain governance events uh usually it's um based on open Zepp and team plates for instance uh so um what you should look for is the proposal creation actually key governance events I won't read it um yeah and any significant events for your contract so if someone is minting your tokens um this is like you should know about that right uh and um if you don't have monitoring in place and you didn't do this how you can know only with monitoring um yeah so now let's that that's all cool right but what what what we can do with that um because like building all own tool set is quite complicated but um let's first of all check what we should keep in mind so first of all it should be um separated what it means so um just remember that your infrastructure can be hacked and I when I say infrastructure I mean um your computers your servers your phones um it can be hacked and if you run the monitoring to tool set on the same infrastructure that uh your project is running then um attack can just turn it off it also means that um if you if you are large enough and you start thinking about the inside the threats um or for instance you you want to consider the threat of a person being compromised in your team kidnapped for instance then uh you should also separate the teams your monitoring team the team who configure the monitoring and have access to modify your monitoring should be separate from the team having access to your um normal infrastructure otherwise there will be additional attack Factor uh sorry so it has to be reliable um monitoring is is not a vital part but a critical one so um at least it should not go down without you noticing that so have monitoring for monitoring where T right um remember to document uh why you you implemented some monitoring um have the configuration uh somewhere in git for instance and um test test your monitoring and test it test it again um as mentioned by Andre um there are actually many ways to test it including the simulation scenarios but we will talk about that a bit later um and so what tools can help us to achieve that so I'm um here just talking about our experience so what we use uh in metalabs how we build the monitoring for us so first of all it's um API API of the smart contracts of the chain of our ZK Ina API uh second is the open sappin open sappin is a great tool or we will speak about it a bit later there are some alternatives to them as well um yeah so so these SAS monitoring tool sets allows you to configure um many things in in few clicks yeah um GitHub a free way to store your configuration they have uh actions U you can in many ways just deploy your configuration in few clicks and you can configure it separately uh from from your infr team um Porta Bots Porta Bots is a huge step Port so um this team actually uh built quite good reliable solution for monitoring Unfortunately they don't currently support ZK in Cara but hopefully it's going to come uh soon but for now uh like the the the major um um use case for us was monitoring the L1 part for our ecosystem and um grafana Prometheus and other tools um like pager Duty I don't know um OBS Genie or whatever escalation and Incident Management tool you are using um yeah so about the open sapping as I promised there's a separate slide so um they fully support ZK sync so open Zeppelin overall is the metlabs partner for the Security Solutions and we uh work with them very closely also for the security audits um so what what the product does it monitors basically all the events happening on multiple chains um you can filter out um and put some logic inside it can be a simple one you can also integrate it with the ports if you need some Advanced logic um then it can just escalate to your pager Duty OB it also supports web Hooks and so on and if you are not very techy uh they have GUI which uh can help to configure U the basic configuration just in few clicks so um overall demo of the product is um uh available from this QR code if you don't use them uh I highly recommend you to just uh check it out and uh the the the defender has actually a lot of other um cool security features aside of monitoring like deployment of the contracts blah blah blah so just check it out um yeah so in order to give you a quick start um I've prepared some some basic very basic uh sample of the uh code it includes the git um GitHub actions which can deploy the configuration it incl use the very simple erc20 token and it gives you an alert to your email um so yeah but you can easily um just Kickstart from there uh so okay we detected it yeah so we received an email so what we going to do just imagine your engineer receives a call in the middle of the night okay private key is compromised what I want to sleep yeah uh so be prepared uh it's a very simple uh answer so think about the key attack scenarios and uh possible response that you want to have in a predictable way write it down it's very important because in the middle of the night uh under a huge amount of stress people make mistakes and they do need notes and test it and test it again actually uh the real life exercise with the limited knowledge like simulation of the hex scenario uh improves the incident response a lot but I can tell you for sure you won't make friends like uh in that way a small heart attack is still a heart attack uh and I I can tell you as um I was the participant of one of such exercises in met Labs uh but I I was not aware um and um yeah I was thinking to resign even but but it goes away after after the stress is gone yeah so um and identify document um how you're going to mitigate um the the the the most significant threats like what are the conditions under which you want to pose your contract or you want to implement additional delay um maybe you want to even ultimate part of that uh like if you get the alert a script is executed then publishes the pre-signed transaction to pause your contract this will help uh with these scenarios that Andre demonstr red uh like with hours or like U between the first uh withdraw money and uh the actual uh incident response so uh well so what's next yeah so we are um going to to the decentralized world and um all the tools nearly all the tools that I mentioned are pretty much centralized yeah so to be honest I don't know the answer um and if you know what what to do so so I have some some ideas like uh creating a dow with the cool security people who stake their reputation there and uh actually do the community response to the incidents it can be a case so let me just invite you for some discussion um yeah so if you have ideas please feel free to share yeah we are available somewhere there okay thank you all right thank you very much Dimitri and Andre from metalabs talking about a very important topic about real onchain monitoring it's indeed very important for anyone developing any protocol you never know when hackers might come in and sneak away with your fun so always have to be careful okay I quickly open for any questions anybody have a question please raise your hand just wait for a moment guys anybody anybody feel free to raise your hand no what do you think was the worst hack in crypto so far the most terrible heck oh switch off oh you can switch it on just press the button there is it there the worst what in your opinion was the worst hack I can definitely tell that the one when I had a mini heart attack oh okay so that one was a test right someone was testing your system okay okay that's good so everybody go and test your systems tonight uh just don't tell your developer and uh see how see what happens all right otherwise let's give them a round of applause thank you gentlemen Demitri and Andre