How MEV infrastructure makes Ethereum more brittle | Marius van der Wijden, Geth | ETHTaipei 2024

uh don't let your friends do me uh don't let your friends do meth uh yes yes yes uh there are few puns in this in the in these slides uh thank you for coming out this early um I didn't sleep very well so I'm trying to do my best uh to get through this talk um yeah me infrastructure who doesn't know what me is some people that's really good because that makes my presentation five minutes longer um so me me stands for minor extractable value or recently for maximum extract extractable value and the idea is so for example I don't know we have um some of some of you guys have have coffee right and um if I want to buy 100 cups of coffee from the participants in this room I would need to pay different prices someone would pay I would give me their coffee for like five bucks someone would give me their coffee for like 10 bucks uh someone would give me their coffee for uh 20 bucks so now um I want to buy 100 cups of coffee and uh what what do I do I go to all of the participants and and I look for the 100 cheapest cups of coffee that there are so um this is this is how I would uh I would acquire my coffee now if I know that someone else is going to buy 100 cups of coffee in a minute what can I do I can buy 100 cups of coffee from all of you for the cheaper price and sell it to this other person um who wants to buy the 100 cups of coffee in a in a minute and uh this is basically one form of me where you you know that someone is going to buy some some Goods some currency or whatever and you slip in right in front of them buy it and then sell it to them and you make up a small difference between the price that you bought it and that you sold it to them and um this has a lot of bad externalities um because it means like you as someone who who knows how to do this uh you can extract a lot of value from the ecosystem and um so because um there's a lot of money to be had uh to be made uh doing me and uh extracting uh value from the ecosystem uh a lot of me infrastructure has been built um over the years and um this is roughly an overview of how the me infrastructure looks right looks like right now so if you run a Noe um you you are the validator uh if if you if you run a validator you can also run a node without running a validator but if you are a validator you can either choose to build your own blocks or you can take care uh take advantage of this meev infrastructure and get some of the Kickbacks uh from people extracting value from the trans actions that uh that you're putting into the block um so uh the validator is connected either directly to a relay or connected to MEF boost and MEF boost connects you to multiple relays and uh these multiple relays can be uh are connected connecting you as the validator to multiple Builders and Builders might or might not be Searchers themselves or connect several Searchers so the idea is um Searchers and Builders Builders are building the block um so they are the ones packing the transactions into a block and making sure that the block is kind of valid um The Searchers are searching for these me opportunities so they are looking at the mol they are looking at all the available transactions seeing where is their value to be extracted and um often times Builder and Searchers are actually the same entity uh but sometimes they're not and then the relay is just there to um it's a trust the relays are trusted entities which is kind of bad um but they are here to connect the builders to the validators um because Builders um they don't want to send the block that they've just built to the validator without making sure that the block is actually mined because if you send the block uh to the validator the valid dat could just take the the interesting opportunities that that the builders found and in in in include them in their own block and so steal the me and um the the relay is there to make this this trusted connection um yes so there are there are few so if we talk about um no software so your normal gu or um nethermind this is all here in the in the validator um all of this other software is maintained by people that we don't really know except for MEF boost um uh these things they are still under consensus so they are very important in if something goes wrong here if a block is built in a wrong way it affects the whole network but um some of the software is is not open source most of the software in this part is not open source um some of the software here is open source um but there are uh there advantages of building relays um without open sourcing because you can be the the quickest one the first one um to to send the block to the validator um so we have a bunch of infrastructure that is very important to ethereum that is um not open that is not maintained by by uh all touristic actors in the in the ecosystem but it is maintained by um by actors that are purely in it for the money and in my opinion that's kind of bad and um this kind of shows you so this um this is actually the same view that you have here but turned around so you can see here A bunch of um big validators so for example you have Lio or you have coinbase and then you have uh the relays and the relays are connecting um Builders and Searchers with uh um with the with the validators and there are there are relays that are um that are censoring transactions that will not let transactions through um that interact with certain contracts and there are relays that are not censoring transactions and um because the the relay is kind of a trusted entity it's yeah it's sad that they are they are doing this sensoring so all of these games are extremely um they're extremely dependent on the timing so um you have around three to four seconds to build a block um in a slot and the quicker you are with building a block the better it is and uh the longer you can um you can take like the longer you um or the the the this the if you the the more precise you are uh to get your block up until the cutoff point where it's no longer um uh it's it's too late um the more me you can extract so there are constantly transactions coming in right um and if you wait longer you can uh include more of the good paying transactions um so if you create a block at the beginning of the slot um at Time Zero you only have a few transactions available but if you wait for two more seconds you have a lot more transactions available and you can create a better block out of this so all of this in infrastructure is extremely time critical and what some of the relays have allowed is uh for something that is called optimistic submission um basically uh you as a builder can submit a block to the relay um and the relay will not verify it it will just send this block directly to the validator and um without even checking that is that it's correct in theory this is okay because the block can be verified by the relay after the submission and if the block was bad the person who submitted it will be banned so you can at most submit one bad block the problem is if uh this if this code that bans the proposers is not working correctly um then we have a then we have a bigger problem and we will see this in a few in a few minutes and um you can see here that roughly 20% of all of the blogs uh um are submitted optimistically and 80% still are verified before they are submitted to the validators um which is the the correct way to do it and um another issue that we have is uh private orderflow uh basically some of the builders have opened up private rpcs where you can submit your transactions um so if you say normally if you submit a transtion action you put it into the public men men pool and everyone one can look at the transaction um but if you don't want any anyone to look at it except for um the builders you can submit it privately to them and um you can see that [Music] roughly roughly 10% of all transactions never make it to the public me pool um and most most of these these transactions are actually me extracted transactions but there are also um um there are also uh normal user transactions within there and if you have access to this private order flow you have a huge Advantage because you know transactions you know all of the public transactions but you also know some of these private transactions that no one else knows and uh so you can build way better blocks than everyone else and um so the issue is if there are entities that are better at building something that are better at building blocks they will over time ACR More Money More transactions more value and uh become an outsized part of um of of uh of block building uh which is which is bad so let's look at some of the issues that we had and that that we're seeing um first of timing games as I said you have have up to four seconds kind of to build the block um if you're using a relay it's a bit less because um you cannot uh you have to factor in sending the data sending the blocks between the between the relay and the validator and the Builder and the Builder and the relay and um but you can see in this on the slide on the top you can see the bit select by solo stakers and you can see most of the bits are within the 1 second um range and then you see a um a validator that is playing timing games and they are selecting bits that are way later than what uh the solo stickers are doing and because they can do that um they they can extract more uh me more fees um than the average solo stier and um there's a dis there's a disadvantage to it uh you can only do it up to a point because if you select a bit that is too um that takes too late into the slot um then you might actually miss the slot because you you don't get the the block out in time um the thing is if you're a big validator if you have a bunch of um nodes uh you can play these timing games better than small validators so um it's it's an it's an open issue here and then we have um an incident that happened on February 6 uh this this year um where we had nine Bad blocks proposed by one of the relays and um if you remember in the optimistic uh submission Slide the idea is someone or a builder submits the block optimistically and the block is verified after uh after it was uh put uh sent to the validator the problem here was um and then if if the if the block was bad then the then the then the Builder should be banned the problem here was um the the Banning code didn't really work because they shut it off because PE uh Builders were getting banned for for not the right reasons basically the code was so buggy that It produced so many false positives that they just shut it off so the automatically the automatic thing that would like if the network was actually being attacked would would protect the network was shut off because it was too buggy and um so in the end around roughly one e worth of transaction fees uh was lost during these blocks and the validators have been refunded manually um by the Blox Road relay but this also shows you sure the validators didn't they didn't really lose something um but we've seen uh nine Bad blocks on the network um the network kind of degraded a bit um during this period period and um so it's it's uh it's really bad and there's no recourse for it right like we cannot go to the to the relay and say please don't do it um well we can but like there's nothing stopping them from from from from using this and um then we had another incident uh a month later on March 13th um and this was shortly after and within with with the denune upgrade with fo 44 we introduced these commitments into the block and um one of the one of the blocks route uh relay uh implementations uh had a bug in it and that uh would set all the commitments in the block to the same value basically overwrite all of the commitments with the with the last value and um here 16 slots were missed and uh about 3.2 e one e was lost but those were also bad blocks um so the bad blocks were mined oh I think it was the other way around uh oh yeah the the uh the these invalid blocks they were um already invalid on the consensus layer so they were reor out um so uh kind of again this code to check is this block bad was not triggered because the blocks were not part of the canonical chain because they were reared out um and so yeah they have this optimistic submission um but it doesn't really work and you're not Banning the people that are doing um bad work on the on on on the network and so we're seeing a bunch of bad blocks so please yeah please fix your fix your code and again these for example this relay grpc it's not something um that is that we test it's not something that we test on the on the test Nets it it's not something that is um that is tested on um uh with uh with like unit tests or anything it is made by blocks route um and in this case it it was block rout but um there are a few implementations and um they are yeah they're writing their own software but they're not testing it properly and um and all of ethereum is kind of dependent um on them not not being bad and and not and and writing good software so it's kind of fun um all right uh let's talk a bit about Solutions um one of the solutions that um that is being discussed heavily um also now for inclusion in Prague is called inclusion lists and um inclusion lists basically within within inclusion list the validator um Can Say Okay I want these transactions to be included and the builders have to include them either have to include them or they will miss or they will just not bid on it and um there are two flavors of inclusion lists um something that I call the carrot and the stick uh the carrot one is um is a very intuitive one very simple one um and it is uh it's kind of like the nice way of doing it um basically you allow the validators to specify which transaction they would like to include and then the builders can decide do I want to include them or do I uh just not bid on the Block and um so Builders can still because there are these there there are these censoring relays right and there are these censoring Builders they will not include uh some of the transactions um and uh with inclusion lists uh these Builders can can just say okay I don't want to include this this transaction um um so I'm not going to and I will just not not bid on this block and um they will lose uh kind of lose funds they will lose market share and the people who are willing to bid on the block with the inclusion list will gain market share and um yes and the the the Carro part is blocks that do not include the transactions are still valid so if um if I say Builder please build a block with these five transactions and the Builder sends me a block that doesn't have these transactions um The Block is still valid um so I can still decide okay it didn't include my transactions but the block is so good it it gives me so much uh transaction fees that I'm still going to include it um and now the stick version um again validator specify which transactions need to be included by the builders but they can either be included in the block that is being built right now or the next block and um so the block Builders are forced to build a block with the transactions um or they will lose out on potentially two blocks and um so the idea is I as a validator can force my block Builder but also the block builder of the validator after me to include those transactions um and blocks that do not include these transactions are invalid so um that kind of means the inclusion lists are part of the consensus now and um and and noes need to um not notes need to verify them and it's not possible to build a block anymore that do that that does not include the transactions um this is a very simple solution and it is not part of the consensus it's just part of this validator infrastructure so it can be um done without a hard Fork um this solution requires a hard Fork um but it's also kind of um a bigger incentive um for for Builders to to start or for relays to stop censoring and start including um uh transactions that they would otherwise not like to include and um this is one of the this is one of the solutions that is being worked on and the other one is uh epbs which stands for enshrine uh proposer Builder separation um and the idea is um with with a with this me infrastructure you kind of have this distinction between the proposer of the block so the validator and the builder of the block so the builders and relays and Searchers um and the big problem right now is the intermediary which is the the relay and um so the as I said the the relay is trusted you as a validator trust the relay that it will deliver good blocks to you um because you're signing them without knowing what you're actually signing and the um the the Builder is trusting the relay that the relay is not taking away their their meev their transactions and um EPS is uh are ideas around how to get rid of this trusted intermediary um there are multiple proposals that are being actively discussed um so I I will not go into the proposals that that are there right now um but uh those are kind of the the wish lists um that that we want proposer Builder separation to have so the Builder is sure that if they reveal their transactions they will also be included it and they will get the money and no one else can take the transactions um if roks happen um everything is still kind of working um and uh Builder withholding uh safety means that if someone sends me a blinded beacon block and I sign it um they cannot withhold this information from me um unconditional payment is uh I I either get to um get to build the block or I get refunded so I I can um I I I get paid in the end for the service that I provide both the the the Builder and the validator um and then proposal safety all right I think I'm almost over time so uh practical things to do run your own note you don't need 32 e to run your own no to validate the network you only need 32 e if you want to build your own blocks um if you're running a validator build your own blocks um you don't need to use this me infrastructure your note itself is capable of building blocks um you might miss out on uh on some of the me payments um but you are a good honest note in the network um if you decide to use me uh the me infrastructure use a non- sensoring relay and uh set a minimum bit on your node uh setting in minimum bits kind of says if the relay if the if the block that I'm building so you're always building a block locally um if the block that I'm building locally is only I don't know 20% 30% worse than the block that I that I get from MEF boost then I prefer to build my own block because I know I'm not censoring any transactions um if inclusion lists are available opt into using them opt into uh you note producing inclusion list for other nodes uh for other validators um and then also get involved in specifying and building this as I said with the epbs it's it's still active research um so there's a lot of things going on and um don't let you do let your friends do meth and also don't let your friends do LSD uh liquid staking derivatives is also kind of bad so thank you very much thank you Marius