Protecting The Baselayer - from Shanghai to Osaka by Martin Holst Swende (Devcon5)

so my name is Martin holes Linda and I work with the security for the etherium Foundation and I also work with a gift client and I'm going to talk today about basically things that have happened during the last three years so I became the security guy in basically when I was Def Con in Shanghai and we're now in all second I'm going to talk about the base layer the infrastructure of ethereum so what is the theorem infrastructure and [Music] the building blocks of ethereum so etherium is a blockchain and one of the things that can be attacked and it's a vulnerability which can be is the block processing so a client gets blocks and the block consists of headers and transactions and some other things and they need to be validated according to a very particular set of rules it can't be too much validated and it can't be too little validated so there's very intricate rules and it's extremely consensus critical because if clients do it differently we have an immediate chain split there's this EVM which I think is the one thing that everyone knows exists in ethereum and this is where the smart contracts operate and the smart contracts execute on top of this thing called the state which is this global state of where everything is right now and basically they accumulated result of all the transactions in the history of ethereum also highly consensus critical and but aside from that they can also be denial of service attacks that causes slow blocks and this is what happened during the so-called Shanghai attacks we have networking protocols one of them is the eath protocol it's not actually a networking protocol it's it's the ethereum specific application recall so that two nodes can communicate over each other and one can say I need some blocks and they can do announcements of new books etc and this protocol contains the building blocks that clients can use for their syncs there are neighboring protocols for LS there's like science and parodies own custom warp sync this piece is not consensus critical per se but of course it's very important that it works because otherwise we can have problems getting no sync and there can be denial of service and there can be lost blocks and largely orcs and generally bad functioning of ethereum and the eath protocol operates on top of the peer-to-peer protocol the p2p protocol which sits on top of TCP and it has just some basic guarantees and identify no verifies node identities also and not concerns is critical but it's the backbone for the upper layer peer-to-peer protocols so if this were to fail then of oh yeah you'd have the same failures as the each protocol if varium peer-to-peer networking obviously is built upon the premise that I as a node can find another node on the Internet which is not a trivial task but for that we have the discovery protocol which currently is in version 4 there is a version 5 being worked at and this uses academia distributed hash table to announce presence and using this DHT we can find nodes according to a distance metric and they can hone in on each other and the distance does has nothing to do with geographic location or network distance the discovery protocol is not based on these p but UDP so it has no built-in guarantees from the lower layers about for spoofing protection it's trivial to spoof by piece and this protocol can be successful to a couple different types of vulnerabilities one of them is Eclipse attacks where a node is partitioned off from the rest of the network and they only wants to communicate with our nodes controlled by the attacker this protocol can also be used to do distributed denial of service attacks against third parties that are nothing to do with ethereum and amplification attacks which kind of is part of the DDoS and an amplification attack can be said to be an attack where I have a small resource that can be a small network bandwidth and I could sell my and bytes out to the network but the effect that is reached on the victim is amplified by a factor of something else so basically this is one way to develop the attack service surface I omitted some parts there's also the sync protocols but there were only four forth things in this slide template so I had to divvy it up like this so I'm going from the bottom up here and during 2018 and 2019 there was actually quite a lot research into the discovery protocol and there were two papers sent to us before they were released to the general public about the Klipsch attacks and some people also reported by the bouncy bug bounty and vulnerabilities amplification DDoS attacks and Felix and Frank on the guest team found even more issues and one of the eclipse attacks that was found in the first paper is that gif as a client had dedicated half the connection numbers to be incoming and half to be outgoing however if it hadn't already failed done all the outgoing connections it allowed the incoming connections to fill up all the slots so an attacker could simply just over we're overwhelmed I guess no with incoming connection and have all the connections to the thing they could also see that there were other ways to fill out slots by pinging pinging them and looting the identity database and the third attack they found was that if you manage to disturb the timing at the time that a node has other peers will reject him and forget about him and he will reject other peers and only the malicious payers will accept him as for amplification attacks here's one example that Jeff was susceptible to so if a malicious node spoof and spoof pinged the target the target would reach out to the actual victim with the ping with the poems or in the new ping and the malicious node could send in the poem pretending to have seen the first outgoing ping from the target and he's supposed to supply reply token but didn't actually validate the reply token so at that point target believed that malicious node resided on IP address 1 2 3 4 and if the militia of them sends a so-called final request which is only 213 bytes or so it causes a final response called neighbors to be sent to the victim which is 1.5 kilobytes there's an amplification like an amplification of about 7 on these attacks which is not trivial more complex example could be made also where you set up the ball ending with two notes and then had one of them lie about the location of the first one and the first one then do is moved find out requests from that IP address which he'd lied about and ankles the same effect moving upwards in the network stack to ethan peer-to-peer la lair in the beginning of 2017 we saw that the network traffic was really up here so much noise and we started analyzing and looking into what caused that and found that during about two hours and note would typically reject 165,000 transactions because they were invalid so 19 invalid transactions every second just streaming into the client problems were that one of the clients parrot I didn't remember who they'd sent transactions to and another issue was that they didn't actually garbage collect invalid transactions asked state progressed over blocks so we've if they were included they were removed but if they became invalid because of they projected progressional blocks and they were not removed so they were just constantly being recent and after we looked into that and and nail out iron out those issues you could see that the the graphs on these monitoring notes which we put up started levelling out and that the incoming traffic on a node went down by a factor of 20 to 30 during January to March in 2017 and also the only is this protocol level there is the sync protocols as I mentioned and earlier in April this year we saw a new kind of attack which we hadn't seen before which was a target attack or the fasting protocol which was a grieving attack where as soon as a node joined the network the attacker who was scanning the DHT tables a new node would instantly connect to that node and tell him Hey I had the blockchain here I'm full of sinked at block five and the victim having no other source of truth would tell him yeah please send me over the states on the blocks and that of course happens very quickly since what it was a very small chain and the victim would then be fast synched to block five or so and after that switch over to block by block sync effectively doing a full sync to book one the remaining eight point five million blocks so it was an interesting attack because in no way does it benefit the attacker he needs to actively run nodes and run some custom software to scan the DHT and do this attack and gain absolutely nothing other than what's the etherion world burn a bit a bug bounty from junior-m was that there was a vulnerability in the protocol and gap where someone queer state entries with a so-called kind of a null requests were crossing the Tri metal route an instantly crash and appeared so these kinds of bugs if even an attacker has them he can iteratively shut down the etherium Network but just connecting to them mapping the the amount of the DHT and connected them is shut them down iteratively which can be really really bad as for block processing and in 2018 there was a live consensus floor that happened on Roxton Robson is one of the test nets and what happened was that someone had sent one of these EEP 86 transactions on Rob stone now EP 86 and was Anita proposed a new transaction format for account abstraction and it was never actually roll out it was never actually included in any hard work but parity had implemented it and they mistakenly enabled it in the client and not only for test nuts so this was kind of a real critical issue and we were just waiting for someone hacked this someone did this found the floor exploited the test net and we were just waiting for it to hit maintenance fortunately it did not apparently quickly released a fix and we could see eventually that actually there was no attacker per se it was basically the researchers who were playing about a bit with these kind of transactions so not all instances caused by malicious intents and in the block processing area there was also in March a very long-lived consensus bug I think that might have been the most long-lived consensus bug we had found in the block processing engine whereby both gas and parity and did it erroneously so blocks have time stamps a normal time a normal block cannot be too far in the future because the client won't import it however an orphan block or an Omer block doesn't have any restrictions on the time stamp they can be how far much in the future they want and if they exceeded the UN 64 yes would just yeah wrap it around on you in 64 and parity would cap it so both would be incorrect and a calculating correct block cache and it would cause an a consensus failure with the quirk that none of them would be correct this would be kind of hard to exploit because you would need some mind both the quark uncle and a block within a period of 7 blocks and then roll it out to maintenance so you'd have to have quite a lot of mining power to actually do this kind of attack I'm going to have to skip a bit in the EVM portion most of you might know some of these so Shanghai attacks arrived which mentioned a bit caused a bit of problems for us during the about the month there were denial of service attacks because first of all in optimal suboptimal client with limitations and secondly because of gas miss pricings and there were two hard Forks that followed on that and there was a naina chain splits because of how the reversion was implemented and how that whole change that happened we know how it happened we haven't really figured out a good rule to explain how it happened and put it into the yellow paper right now the situation is that all the clients have particular right and DS special rules so you can see here a bit of code from the Goethe areum codebase and get up code from the aleph codebase and they basically just oh we fix this a little bit it's not it's not don't look at it this is ugly we just know that fortunately this is this exact situation will never again arise because the conditions that would enable this to rise can never again happen on the maintenance in January I had to actually remove this slide because each see was still vulnerable last Friday in February 2017 that was really interesting a consensus bug which was a combo consensus bug that could either be used to cause an instant chain split or it could be used to crash all the gif nodes so the stack requirements were misconfigured for the swap top and balance of op codes and the thing is if the stack requirements are off if it leads to I'm not talking about the EVM stack not like an ALICE type if it's an underflow death would immediately panic in all the nodes that were processing this block all the gift notes would immediately drop off drop off the network if it that was like overflow that will lead to a consensus issue where GAF had a different state route and parity and you would split the chain so any attacker which found this could just choose which route to go and it would have been trivial to exploit and trivial to detect if it'd have been past naively so what we did was we read refactor the entire code that that's configured the EVM of codes and two weeks later we reversed the fix reversed the the whole refactoring but left the fix in in 2017 some attackers were hitting the gas jump test analysis so we saw on our live monitoring system that some blocks took like 24 seconds to go through and yeah so we had to rework the whole jump test analysis very quickly bring it down a couple of orders of magnitude in processing time and there were some ideas that this was possibly the Shanghai attacker doing this we don't know I don't have to have time to go through the whole byzantium but in bicenthium when we were approaching that we really started getting payoff from passing and it found all these issues and you can see the last one here happened on October 25 that's actually nine days after buy something for kids and not only that not until February did we uncover the hopefully last bicenthium bug it was Cassidy drew found an issue with our biometric pairing which was we couldn't actually fix it until a couple of months later when we swapped out the whole VM library for an assembly version written by CloudFlare some takeaways from this is that monitoring and metrics and graphing up things really helps to see the overall network health and see what problems there are and what problems there might be in PTC loggers giving a talk later about monitoring in live with her here's a chart that can display very detailed what happens when we process the blocks and exactly where the time is spent and these kind of things really help us to find what's going on similarly for the network traffic's you can pair with a naive one which I showed earlier here's a much more detailed one yeah I mean of course it's really good tooling to to find out what happened when something happened and why it happened and we need good communication clients between teams to quickly coordinate to fix stuff when when goes down on the minute and one thing that's been really important from day one and I think it really has been great is that we have there's no multi culture where we don't just rely on one client to hold up the network and I hope that that node multi-culture will continue on ethereum that's it for me thank you [Applause] [Music] you