New Ethereum talks, every Monday. The week's conference uploads by event, in your inbox.

Loading player…

What Happens if a MEV-Boost Relay Goes Rogue? | Auston Sterling - Aestus Relay, EVMavericks

Ethereum DenverWed, Oct 9, 2024, 12:00 AM

Speaker

Auston Sterling

APPLICATIONS FOR ETHDenver 2024: #YearOfTheSpork ARE OPEN APPLY NOW https://ethdenver.com/apply

Transcript

Hello. All right. Hi, I'm Austin. I work on the Astus MEV Boost Relay. And today I want to talk about relays, the trust we place in them, and what could happen if that trust were abused.

Specifically, we'll look at the worst-case scenario where a relay turns malicious and tries to deliberately cause damage. So, I am going to assume some background knowledge. It's going to be a somewhat technical talk, but as a reminder, relays are trusted entities that sit between block builders and proposers. The proposers trust relays to ensure blocks submitted by validators by builders are valid and will pay the amount they promised in their bid. And once a proposer signs a bid, they also trust the relay to release the execution payload that's been private so far.

Builders also trust relays to keep transactions private until they're included in a block. So, we can already see some pretty clear attack vectors if a relay just decides to not do these things. It's also worth mentioning how much Ethereum as a whole is reliant on MEV Boost. Over 90% of blocks are sourced through relays, and this number has stayed pretty consistent over the last year. Relays tend to self-report their validator registrations, so these numbers do have some issues, but it's clear that there are some individual relays that have a significant percentage of the validator set connected.

And since MEV Boost blocks are five times the value of vanilla blocks, this situation isn't going to change anytime soon. So, we're going to move between a few different topics, first looking at a few past incidents to see how they work, then taking a deeper dive looking at how a compromised relay could amplify the effects of the issues we've seen so far, and finally, we'll consider a timeline of events and what this could look like in practice. So, first, in October 2022, an attacker discovered that the Manifold Relay had disabled block validation, but not secured their block submission endpoint. So, they submitted bids that promised to pay a lot of ETH, but kept the actual rewards for themselves. Since the relay wasn't checking, they went ahead and forwarded each bid to the proposer who took it since it looked like the highest bid.

But each time the proposer got nothing, and the attacker got just an average block's worth of reward. In the end, the attacker got 183 blocks like this through the relay before getting cut off, and they stole about 7.5 ETH. So, the next attack is the Low Carb Crusader attack from April of last year, which was really interesting. It had a lot of moving parts, but the main issue was really one of transaction privacy.

So, here a malicious proposer actually managed to trick the Ultrasound Relay into revealing the transactions in the block without publishing it first. So, this gave the attacker a chance to manipulate those transactions for their own profit. They constructed their own block using the transactions from the MEV Boost block and submitted that to the network instead, ended up making a pretty solid payday. Since the attacker was the block proposer, they knew in advance they were going to propose a block, and they were clever and planned ahead. So, during the bidding for their slot, they submitted some bait transactions to the public mempool.

These were generally trades on DEXes designed to look like easy money for the MEV bots. When the attacker looked inside the block they got from the relay, they found transaction bundles trying to capture MEV from the bait. So, to make the $20 million, what the attacker did was extracted MEV from the MEV bots. Basically, they made sandwich sandwiches. And they could also unbundle the original bundle, moving things around, rearranging as much as they wanted.

So, attacks like this ended up being incredibly profitable for the attacker. And there's more that could be said about this attack, but it's a very quick overview here. And there have been many other attacks, but the key takeaways here are one, proposers can be lied to about block value, and two, builders' blocks are vulnerable to unbundling. So, now we'll look more formally at the kind of attack a compromised relay could carry out, largely building on what we've already seen. So, pretty much every incident we've seen with relays historically has been the result of some bug in the relay software.

So, this necessarily limits the extent to which the bug can be exploited, and as soon as the relay operator realizes what's up, they'll put a stop to it. I put the recent block route attack up here as well. This is pretty normal. Most attacks don't last longer than a couple hours and result in single-digit losses in of ETH for proposers. It's not great, but it's not the end of the world, either.

The Low Carb Crusader attack was much more financially significant, but it only applied to a single slot and required precise conditions to carry out. It's not really something that could be sustained. So, what if this weren't resolved so quickly? It'd be very different if there were an actual deliberate attack causing a relay to become malicious where you couldn't count on its operators to resolve the issue. It doesn't have to be that the operators themselves are malicious.

Um it could be that attackers compromise the relay infrastructure or pull off some DNS attack. Either way, they'd gain complete control over the relay for an extended period of time, and this would be very different than the issues we've seen so far, and it's not as commonly discussed, so that's what we're looking at today. All right. So, here's our friendly neighborhood relay, Astus. And for one of these reasons, it's compromised and it becomes evil Astus.

They like being evil for the sake of being evil. They'll steal money, they'll attack Ethereum, whatever causes damage. It should go without saying that we hope this never actually happens, but we'll use it as the example today. So, let's see what this would look like. For each Ethereum slot, we have a different set of options depending on whether the proposer is connected, whether builders are sending us blocks, and we'll cover a variety of potential attacks.

Um these generally are not novel individually, but we can consider them in the context of current MEV Boost adoption and their role in a larger-scale attack. So, here's our base case. The proposer isn't connected to the relay. We're not receiving any blocks from any builders. The relay isn't in any sort of privileged position.

It can't influence block production. So, from our perspective as the attacker, this is unfortunate. There's nothing we can really do. But it's kind of a nice reminder. Relays only have as much power as builders and proposers give them.

Proposers building blocks locally are the greatest defense from these kinds of attacks. In this case, we have a proposer connected to the relay. So, let's mess with them a little. The proposer trusts the relay to carry out a fair auction, but we don't have to. We can just lie to them.

We saw this earlier with the Manifold incident. The attack there was a third-party block builder, but we can also build a block like that ourselves. We'll tell the proposer we'll pay them 100 ETH, but actually pay them nothing. Even if they're connected to other relays, 100 ETH is going to be the top bid pretty much every time. So, the proposer's MEV Boost client will happily take our bid, and once they're committed, they'll propose our block.

And every slot we have a proposer connected, we can carry this out repeatedly. The planned mitigation for this attack was to have a number of independent relay monitors, which validators would link to. The relay monitors would check each slot after the fact to see if this attack took place. If so, they would send out a message to all listening MEV Boost clients making them immediately, automatically disconnect from the malicious relay. But this ended up not being fully developed.

So, while there are relay monitoring services out there, there's no automated disconnect. So, that means validators would have to hear about the attack through some social channel and then manually modify their MEV Boost relay list to disconnect. Any validators who remain connected are vulnerable on every one of their block proposals. Large staking operators would probably catch on quickly, but I wouldn't be surprised if some smaller operators take days, weeks, months to catch on. We also have the option to just not release the execution payload.

We don't make money directly this way, but we can forcibly cause a missed slot, censor all the transactions, potentially harm consensus health. In the repeated case, the relay could attempt to cause every connected validator to miss every single proposal. The solution, which is actually in place this time, is the MEV Boost circuit breaker. This is additional logic implemented in consensus clients to fall back to local block production if network health is poor. Usually, the circuit breaker condition is some number of consecutive missed slots.

So, if here green is included and black is missed, eventually the circuit breaker kicks in, and the next proposer will locally build a block. The other condition is a larger number of missed slots in a larger sliding window, like this. So, these wouldn't completely stop an attack, but it would mitigate the worst-case scenario and prevent liveness failure, at least. So, for this proposer-connected case, these two attacks take advantage of the ways in which a proposer places trust in a relay. They trust relays to provide accurate information about bids, so we lie to them.

Um they trust relays to release the full block, so we don't. And these could be combined as well. First, you lie to a proposer to get them to accept your block, and then you withhold it. Some fun stuff. All right.

So, this is the opposite scenario. We're receiving block submissions from builders, but we don't have a proposer connected. So, we're mainly manipulating builder blocks. Builders trust relays to keep the transactions in their blocks private and to not exploit them for personal gain. So, the simple option is copying MEV strategies and private order flow.

These are generally the secret sauce that builders use to gain an edge over their competitors and our malicious relay has the ability to not only copy the sauce, but pick and choose the best sauce from each builder submission to create a super valuable block. Of course, if we're doing all this, then we might as well take a note from the low-carb crusader. Submit a bait transaction beforehand, build sandwich sandwiches, and break apart all the bundles. Searchers might have adjusted their strategies since the initial attack took place, but it's probably still pretty profitable. In this case, the proposer isn't connected to our relay, um but if they are connected to other relays, we can submit through them.

We just have to pay enough to win the auction. And if they're not using MEV boost at all, um there's not much we can do this slot, but we might be able to leverage some of this information for the following slot. There's not much a builder can actually do to prevent a massive one-block attack that comes from out of nowhere. Um that would provide the majority of the profit for us and most of the damage for them. But builders tend to be pretty sophisticated actors.

They likely keep a close eye on relay behavior with systems in place to quickly detect these problems and get the heck out to prevent further losses. If a builder continues to submit blocks, um they're placing their funds and those of any connected searchers at risk of more MEV extraction. The final scenario is just a combination of the previous two. A proposer's connected and a builder's sending blocks. This is the normal state of operation for a relay, but it's actually the riskiest.

So, let's put it all together. We examine builder block submissions in order to copy MEV or make sandwich sandwiches. Then, we lie to the proposer promising them a fake high reward. Once they accept the bid and sign the blinded block, they're locked in to proposing our block, where we make bank and they get nothing. Um we release the block, watch the chaos unfold, and repeat as we see fit.

And when you look at all of this, it's pretty significant, right? So, for those of you running Ethereum validators, this is the real implication of adding a relay to your MEV boost config. You're trusting the relay to not be a vector for this attack either deliberately or through some security vulnerability. So, with that, we've covered enough background on the main attacks, so we can use the remaining time to put them into context and really answer the question, what happens if a relay goes rogue? So, we'll draw out a hypothetical timeline of events and see how this would practically play out.

To be clear, this is hypothetical spitballing. Don't take it too seriously. Some of the broad strokes may be generally accurate, but when it comes to the details, I probably don't know what I'm talking about. Anyway, so a day comes when we here gain access to the Aestus relay. We decide to use our power for evil.

So, let's do our best to break Ethereum. So, we're not going to immediately launch the main attack. That's not optimal. We'll start in stealth mode. Um Aestus's builders each have their own private order flow and MEV strategies.

So, let's see if we can sell that information to a third party. We can form a nice partnership with some sketchy searcher or a market maker and coast on that income for a while. At some point, maybe we're getting bored or it seems like someone's catching on. So, it's time to go loud. We can try to time the attack for a holiday or nighttime in the US and Europe.

Maybe open a leveraged ETH/USD short since we know we're going to be causing a stir. Our attack is going to be most effective in its opening moves, so we want to make the most of it. The The proposer block schedule is known ahead of time, so we can wait for a particularly good opportunity to come around where we want to see a lot of consecutive slots of proposers who we know are connected to the relay. At T minus 10 seconds, we're one slot away from the start of the attack. So, we can submit our bait transactions, lay the groundwork for some spicy sandwich sandwiches, uh and begin examining builder blocks and creating our own block.

And now it's time. The proposer sends us a get header request. We give them the info for our big attack block along with a fake 100 ETH bid. They take the bait. They send us a get payload request.

We publish the block and watch the transactions confirm on Etherscan. There's actually a really detailed timeline for the low-carb crusader attack, which it It actually took about 7 hours for the first post about the attack to show up on Twitter and get the ball rolling. So, I'm going to be generous and say that uh people have learned their lessons, hopefully, and will catch our attack sooner. Some entities might have automatic disconnects trigger. Others might have their phone alerts go off and would be confirming 15 minutes in that something is very wrong.

In the meantime, we've had 75 slots to mostly go wild repeating the attack. Aestus has about 2/3 of all validators connected, so that's more realistically 50 slots. Um but we can also expand into some some fun app app-specific attacks, um particularly where we can use our extensive censorship options. So, at 2 hours in, I would hope only a few builders would remain connected to the relay, and the searchers sending to them would either have stopped or maybe their bots would break. Some stakers have also disconnected, but there's probably a long tail of operators who haven't gotten the memo.

Enough that we're stealing a steady stream of cash from the remaining proposers. Social media might catch on. Network activity would spike, which might paradoxically give us more opportunities for MEV. At this point, a war room would likely be thinking this through, but there's not really anything they can do to force the remaining proposers to disconnect from the relay. They just have to spread the message.

There's one potential technical solution, which is maybe have the legitimate relays try to outbid the Aestus relay to limit our win rate. So, we might be able to catch on to that and instead just bid the maximum possible integer. Um if the other relays copy that, then we actually end up uh the tie is broken by which block hash is larger. So, in the end, it taken to the logical extreme, we would actually end up in a proof-of-work-style hashing war to see which relay wins. So, after a whole week, the compromised relay would probably just become a fact of life.

You could imagine Dune dashboards being set up to try to identify and track the remaining connected node operators. RPCs might try to time transaction submissions to avoid those slots. But who knows? Those people might be off the grid, anonymous, whatever. Um I'm sure some very smart people would have found more ways to reduce the chance that a connected proposer would select an Aestus block, but it's hard to imagine them being 100% effective.

So, I wouldn't be surprised if it takes months or until the next hard fork for a stubborn few proposers to finally disconnect. And so, while that was fun, we can step back and say it actually really sucks. There are a few things that stand out to me. Um there are way too many factors to tell what the total losses to all parties would be, but given that the low-carb crusader attack hit double-digit millions, it's probably a reasonable order of magnitude estimate. At times, the attacker would control a massive percentage of Ethereum block space, especially if this happened to one of the larger relays.

There's a lot of censorship potential here and a lot of downstream damage to censorship-sensitive protocols. There's also a chance that this could have really lasting consequences given how long some proposers will remain connected. And as someone operating a relay, I would like to not personally be a point of failure this large. So, here are some possible things we can talk about to help prevent this. In the short term, the MEV boost circuit breaker is actually a great system, and it could potentially be expanded to cover more conditions.

I haven't learned why development of relay monitors stalled, but it could be a good fail-safe as well. There's a recent post on Eth research talking about how to harden external PBS like MEV boost through some changes to the builder specs and consensus behavior. Worth considering. But really, I think we should be prioritizing work on enshrining PBS, inclusion lists, MEV burn, and so on. Things that'll help us eventually get relays out of the picture entirely.

And a bit of a less developed idea, but it certainly seems to me that these attacks wouldn't be so bad if relays didn't have 60% to 90% of all validators connected. But maybe that's just the way it is. So, with that, I'm wrapping it up. Um thank you.

Automatic transcript — names and jargon may be misspelled.