Ethereum Security

Hello. My name is Martin Holst Swende and I am the security lead for the Ethereum Foundation. Today I'm going to talk a bit about EVM forensics and managing attacks against the network and how we've been working on that. So, I've been security lead for 1 year. I started just before Devcon 2 in Shanghai last year.

Which started off with the Shanghai attacks roughly 1 day after I started my new role. It kept on for a month. We've also seen during the last year we have done three hard forks. We've had one unintentional consensus split. There was a DOS attack against specifically the Geth client.

There has been thousands of ether stolen in more or less more and less sophisticated attacks both on chain and off chain. We had the the test net totally brought to its knees and then resurrected again. And of course there is the standard IT incidents with leaked databases and someone taking over someone's phone number and their accounts and attacking our GitHub and stuff like that. So, we should all be very clear about where we're all at. This is crypto land and we're all in in land and it's like uh, Australia where anything with a heartbeat will try to kill you.

And if you make a mistake, um, you're probably dead. So, meanwhile for attackers, they have, uh, never had it better. They no longer need to hack, uh, point of sales, uh, computers and trade carding details over, uh, shady forums. They can just hack a computer and or somehow, uh, get some cryptocurrency and immediately turn it into, uh, value. And and it's so it's like wild west in Australia right now.

These are the Shanghai attacks. I'm not going to talk that much about them. The first Shanghai attack is that little blip down there. And then it just kept on going for a month, um, and it was, uh, a lot of different attacks, um, mostly targeted towards gap. But when the dust is settled after, uh, incidents happen, uh, then that's when you can actually do something about them and think about how can we be better prepared next time something similar happens.

And how can we prevent it? So, how can we improve the readiness, uh, and the resiliency? So, for readiness, it's about detecting attacks and performing analysis quickly. So, we started improving that, uh, with the some monitoring, adding up some monitoring nodes that were running in the cloud and adding some graphs. Turns out there were some, uh, inherent issues, uh, which hadn't been noticed before with transaction, uh, propagation, uh, inefficiencies.

Which over the course of a few months in the beginning of from January to 3 to March we managed to bring down the overall network traffic with about an order of magnitude just by removing invalid transaction propagation from the clients. On these monitoring nodes we also added some interface so that we can extract very detailed information about what are the canonical blocks in the chain. And if we see a consensus split, we can get very detailed information about the receipts differences in these and and quickly point out which transaction caused this consensus issue. So here you see as geth master and geth develop and the parity node and the right in this image they're differing on two fields marked in red there and that's because parity RPC interface exposes a few different fields than geth. Now as we're going to analysis going to talk a few words about the EVM.

Because there's there might be a conception that any minor difference in the implementation of the EVM will automatically result in a consensus failure and that's not quite true because there are some things some parts of the EVM which are ephemeral such as the memory and the stack uh and which do not necessarily trigger consensus issues but they're very interesting because they can be used to trigger consensus errors. And in order to really measure EVM side by side and detect implementation differences in EVMs, we need a kind of op by op view of the internal state. So, we pushed kind of hard to get common output format for EVMs so that after each instruction made in EVM, it would output a JSON blob with the internal state as you can see on the left. Um and also a capability to use arbitrary pre-state and genesis configuration with the raw EVMs. So, one problem that can arise is if we're hit by an attack which blows the node out of the water, how can we analyze that?

Uh because our node just died, right? How can I analyze transaction if the transaction crashes my node? Well, if we have a standalone EVM, what we can do is we can just fetch the pre-state about the sender and the receiver, just those two accounts. And we can execute that locally in our EVM and then analyze the trace to find did we miss anything? Was there anything else we should have had here?

Any external references? And fetch those and start over. And if the node crashes, then we've successfully uh reproduced the API the transaction. And for all for this, we only need a web3 standard API without any debug uh specialties. Um Sorry.

So, I'm going to demonstrate uh quickly how we can do analysis of the jump test attack which we were hit by in on June 1. So, I'm running this little reproducer here. Um I pipe in the hash used in the attack, the transaction. Telling it I'm going to use my local EVM, not through Docker. And it basically sets the right fork rules for that particular block.

And executes it. And it has some intermediary traces here. Uh we can take a look at those. Ah. So, we let's go directly instead for the final trace.

I'm showing this in what I call the op viewer or retro mix if you like. It's a remix like debug viewer for the JSON output format that I showed earlier. Um And this is a good start for analyzing what's happening in a transaction. So, you can see this particular transaction. It does an X code copy.

Um and the X code copy fills the memory with 5B. And it does it repeatedly. And as you can see the uh memory is growing. And it keeps doing this for about 600 steps. I'm going to go a bit faster here.

Until it has filled up the memory with 1/2 MB of 5B, which happens to be jump dest. Uh then it puts some more code in there. And this is looks like actual EVM code. 6003565B. And all of you I'm sure recognize that that is the push one, jump, jump dest, and stop.

So it it just executed a create with that code. And as you can see the size of the create is the full half megabytes. Okay, so now we know that the attacker is doing creates and he keeps doing it repeatedly. One part of the memory changes between each invocation. Uh it's a little counter down there and I'll skip forward a bit.

So it's all just creates and it ends on create number 105. It goes out of gas. So by this time uh you can be kind of have an idea so it's it's doing creates lots of times with a large memory segment totally filled with jump tests. It changes one little byte each time so obviously bypassing any caching mechanisms. So by reproducing it and and viewing at the trace in this fashion, we can do a very quick analysis of what happened.

And we can uh benchmark it. Right now it's running at 300 milliseconds. And if I compare that to So this is the EVM uh geth EVM with the patch applied after this attack. I can try it against the EVM without the patch for the jump test analysis. And as you can see it took 9 seconds.

So this tooling makes it possible for us to do quick analysis and then to um um check does this patch work and I can um I can share it with the co-workers and they can try out various patches and see which one is the best. I can also run this in uh uh a web-like format and do the all the same things and investigate other on-chain events. For example, the Sorry, which one did I take? That was the same. The Parity Wallet attack.

And there we have the Parity Wallet attack reproduced. Uh and you can run it locally or you can check annotate the trace of what happened there in the Parity Wallet attack. And for example Yeah, so here is the fated infamous delegate call and the uh Now it's like if you want to if you want to analyze that more in depth. So, the EVM lab which I showed you part of uh makes it possible to do some EVM assembly pythonically uh and investigate these kinds of issues and um yeah, dissect attacks on a really low level. Um We had two hard forks also and in preparation of those we ramped up the testing uh introducing parameterized tests or generalized tests which Demetri talked about yesterday in the breakout room.

And also put it all into Hive. Uh Hive is Peter Szilagyi's super cool framework for running nodes in in a black box fashion. And just synthesized the the environment, the genesis, and the blocks, and everything. And then you can compare the the expected post state after a sequence of blocks. Uh And this makes possible to run It runs about 24,000 test cases against uh Pyethereum, Parity, Geth, and cpp.

Uh and it runs it 24/7 uh 365 uh It removes the dependency of the developers to perform tests as part of the test process. So, now testing can be a totally separate process which doesn't really rely on the developers per se. The fallout, however, after the first hard Sorry, second hard fork is that we had a consensus issue which was um yeah, definitely not what we wanted. Manually crafted tests are great, but there's no way to scale it uh due to the inherent complexity of the EVM. We can't just have enough people know that much about it to to be able to scale it up.

So, we wanted more coverage for Byzantium and started looking into fuzzing. One way of doing that would be to generate test cases randomly, execute them on each EVM, and use this shared output format to compare the internal state after each operation. And just repeat it. And this can be done fairly quickly. Can do a couple of million tests per day if you use raw binaries, and you can use uh these four clients.

The second track is based on libfuzzer where we got in touch with uh Guido Ranken who has done a lot of fuzzing and is a real expert uh on libfuzzer. So, live faster is the core of American fossil up. It's, uh, faster developed by Michal Zalewski. And it's a bit more sophisticated because it uses code paths and instrumented binaries to detect code paths for any given input and then mutating those inputs to maximize the code coverage. Uh, and since everything is instrumented and compiled into one big binary, uh, it is an order of magnitude faster to perform these, uh, tests.

So, I can do about 100 million tests per day. And there was a spectacular and kind of unexpected success in this. Uh, we've had seven or eight consensus issues found. Most of them before the hard fork. Uh, one of them slightly after the hard fork.

Uh, it has been fixed and patched and released. And the clients today, I would say, are more thoroughly tested than they have ever been uh, in the history of Ethereum. And we are still running fuzzers, uh, 24/7. And there's been millions of of tests done on the test net and billions of tests uh, based on live faster. Not sure if there can still be consensus issues or denial of service issues.

And if that's a really, really, uh, concern of you, of yours, then you should run multiple clients and try to detect, uh, mismatches. And uh, you can use the, uh, debug method in geth to find out if geth has, uh, tagged, uh, one of Parity's canonical blocks as bad. And key takeaway here is that all, everyone here are targets for attacker. It's in bold because it's important. So, be paranoid and be proactive and work on improving the security and your resilience and how you can handle attacks.

That's about it for me. Thank you.