How Worried Should We Be About Ethereum Client Diversity | Hyperledger Besu

hair I have been here for a bit and I've watched this conference grow and it's become really like kind of a amus thing that I really enjoy happening every year and so for this first panel I think we should do what we do best and talk about diversity um and that's another reason why I'm up here is that there is a lot of discussion about diversity and like is it important is it valuable to the space as a whole and I think this next panel we'll be talking about why or why not we should be F focused on light client diversity for the ethereum ecosystem so I'd like to introduce Margot nikirk of just there's a J in her last name so I want you to know it's huge that I just pronounced that correctly um and she will be introducing her entire panel so sit tight for the next 30 and it's going to be fun all right hi everyone thanks for being with me here today um client diversity timely topic but before we get into all the juicy stuff I'm going to let my panelists introduce themselves just briefly like who are you though I'm sure many people already know who you are but who are you and why are you relevant to this panel hey guys I'm like client I work on the go ethereum team uh my name is Dan faren I'm a maintainer on the hyper Leger basy team one of many hi I am Darren Langley otherwise known as langers and I am a core contributor for Rocket pool I'm Nick I'm the executive director of East Staker and we advocate for solo stakers great thanks um so fli diversity right this is topic that's sort of been in ethereum for for a while Fades into its distance something happens we talk about it again here we are again so just to get everyone up to speed right there was a bug in nethermind took down 8% of the validators um and this reunited the conversation about what that means for GE right what would happened that with with G so maybe let's just start basic like what would that mean in terms of like ethereum's state if that would have happened at that time because it was it was like 85% of of the validators or 80% at that moment like so so just set the stage what does that mean for the health of ethereum yeah I mean from the GU side there's a lot of different types of bugs that you could have in the client you could have something that just crashes the client causes it to go offline that's probably the most you know the preferred error that would be to happen to a majority client because the network stop stops finalizing for a bit we can go and try and understand why is the client why did the client crash and then roll out a fix everybody can update and then we start finalizing the chain again whereas if we have something that causes us to write something invalid into the state produce a block with a header that is not accepted by the rest of the chain then gu as a super majority client would cause the beacon chain to Fork on to another chain so You' have the bugged fork and the um you know canonical Fork the fork that follows the actual specification of ethereum and how to resolve this Fork there's not really you know a clear question basically what we would say is that the canonical Fork of ethereum is the like Fork the Min minority clients are running because there was a bug in the majority client it didn't follow the specification of ethereum and that Minority Fork is the true ethereum chain and eventually all of the uh nodes that were running gu on the minority Fork would have this inactivity leak or they would have some sort of slashing event if they try to come back to the other chain and they would be removed from the validator set and then the minority chain would after one two three weeks would start finalizing again and I kind of want to emphasize that the the huge deal about that part is that you can do like a sort of social recovery of of the chain but it's going to leave a lot of people unhappy and it's going to uh intensely hurt The credibility of ethereum in that S no matter what the outcome is The credibility of ethereum is shaken at that point and the the social upheaval you say I want to get into that a little bit later but for now you bring up some great points and we'll we'll get we'll get there in a second but I want to you know we're talking about nether mind we're talking about G there's a bunch of other minority clients too right so maybe Dano you can talk a little bit about the different Generations like you like to call it in terms of the the execution clients the pros and cons sort of each like why would someone like try you know run a GU or run um Aragon or whatever and um you know with that in mind as well like if you are one of these people that believes that running a majority client is just like more competitive more performative like what do you need to do to sort of convince them to change that mind I don't know if mindset is the right word but that perspective at least right so um on the notion of generations um right now there are five um execution clients that can actually finalize on the uh on the post merg chain but there's been a lot more than five ethereum clients in history um there used to be a C++ client there used to be a py client there used to be another Java client and these were clients that existed I consider that along with GE to be the first generation of ethereum clients those were the clients that were there for um for for the for the challenges of the Dao and for the Shanghai attacks and after that you started getting a second generation of client now I put nether mind and baso in there and I think a lot of those second generation clients um they do more than just main net validation um nether mind's a star connect client and basally was focused on Enterprise ethereum and they've recently moved on to being being the client that is behind the L ZK chain right and um I think post merge we're starting to see a third generation of clients show up I think IID put Aragon in the second generation and so the last client would be reth right and I would put that in the third generation clients that showed up after the merge so the biggest advantage that that gu has in addition to being the most stable is they're also the most battle tested they've been around since the beginning of ethereum they're the only client out there that can claim that credibility so there's a lot of of of um a lot of credit that they have and having run it for so long um you know they weren't always you know they're they had issues at time but they resolved most of them like during the Shanghai attacks parity basically kept the network going and that was an example of where client diversity saved ethereum um so while while while gu was fixing their bugs parody kept the chain moving along and then gu you know they kept fixing these bugs um so that's I think you know what really got into the mind of a lot of the core developers of the value of diversity is that we can recover from these failures whereas there for a single chain um we're going to be down for hours resetting validators like every time salana needs to go down we'll see if fire fired answer fixes that um but that's one of the big advantages um and the issue that some of these clients have is proving that they have the same um battle metal the GU has so like when when um coinbase you know looks at adding a new client they've been running gu since they've been running ethereum they know how to make it work they know how to how to duplicate it how to syn it fast so when they come in with a new client they have no idea what they're getting into and what the issues are so they need to spend engineering time figuring out you know how to make it snc how to recover it what the failure states are how to make it work how to get the reporting out of it so there's a lot of of um uptick to bring in a new client with that um so unless there's you know something above and beyond gu a lot of them it doesn't make economic sense for them to change well this whole issue with um the potential um slash that comes with the over 2/3 you know changes some of that equation and makes it more viable so that's that's kind of you know about you know with the different clients of of how they're trying to fit in and why it's still an issue today from the developers perspective like what are you guys doing to help you know run other minority clients like in the protocol on the protocol level yeah I mean there's a lot of talk about trying to move ethereum to a stateless model which I think addresses many of the biggest problems that home stakers have and that institutions have uh with running these really Behemoth clients that are taking up you know one terab 2 terabytes of dis space and causing a lot of IO thrashing that you have to deal with and think about how to do how to manage from a infrastructure perspective so if we can move ethereum to a place where you run a very lightweight client you get a proof over the network you verify that proof and then you run the execution of the client I think it removes a lot of the complexities of uh being a devops person maintaining a node farm so trying to figure out how to do stateless ethereum that's one big thing uh another thing I think that we can do before we even get stateless ethereum in the protocol is and we've talked about this in the past at other c c of events is trying to just get stateless ethereum working uh locally within your own node and so if you can have one node that's validating the block and able to like pull out all of the data that was executed in the evm we have tooling to then give that as like kind of a stateless block to every other client and have them run it and make sure they get the exact same value so having a way of like multiplexing the execution of a block across many clients can give you confidence that I ran this primarily with gu got the proof back then I fed it to bayu and nether mine they executed it stateless they got the exact same route that gives me very high confidence that block is going to be accepted so I think that those are like you know at the protocol level and then something that we could do be doing do more clients that could really improve how you know people are able to uh still run gu or run like their client of choice but have these like easy backups to avoid finalizing some issue in a majority client you know you so you brought up coinbase about them looking to diversify their execution clients and they just came out yesterday saying that they are going to add nethermind and Aragon to um their client side um and so I wanted to ask first to Darren from the rocket rocket pool perspective and I would love for all of you also just to have your takes afterwards um but I guess is rocket pool also sort of looking into the similar methods of diversifying and uh I also saw when you know nixo you you had commented on that on Twitter on X I should say um slip oops um so I I commented I I saw that you commented on X that you know in terms of the numbers like what what does this mean for the overall health of ethereum and where so I I would like to hear from all you guys like where you see this going um in terms of node operators like looking to diversify their execution clients but yeah let's start let's start with uh with so from a rocket pool perspective uh we're decentralized protocol so uh the team itself and the the Dow itself operate a smart contract protocol all of the node operation is actually done by a permissionless network of node operators we've got about 3 and half node operators in about 137 different Geographic locations so they're kind of all over the place and we we don't actually control them uh so we can't say uh you must run you must not run gu or or you must run gu or what anything uh they run the the client that they want to run in saying that the community are very very uh active uh they're very ethereum aligned they care about staking and care about ethereum so they essentially kind of self organized and self uh you know they're self kind of enforcing so they will actually help other node operators or first of all explain why they shouldn't necessarily run a super majority client um there's transparency they have transparency Tools in showing us what our situation is uh in other words you know what is the percentage of our Network they running gu they educate people on how to to do it and then they've also built tooling around swapping so the within rocket pool there's this thing called the rescue node uh which actually is open to solo stakers as well now uh and what it does is uh the problem with the execution layer swap is that you need to resync uh with a consensus layer it was a lot easier um but with the execution layer you have to resync the client sure so uh with this rescue node essentially you point your validator at this rescue node uh it U you know you can validate and continue on uh without losing um your attestations and and to validate but your node can then resync in the background once that's resynced uh then it can you can swap it back but I want to I just want to sorry I don't mean to hone in but but and and then we'll go to nixo but you mentioned that so the community is sort of in charge of this effort right so what are those ongoing efforts because it does seem like it's widespread now amongst amongst these different you know decentralized or or centralized exchanges like a coinbase or decentralized like rocket pool you know so what I I think specifically like what are those efforts that are happening right now yeah the those three three things there's like transparency on where we actually where are we as a protocol sure uh so like currently we run about 30% gu uh Basu is is very very popular uh and and nethermind as well so we're quite nicely split at the moment okay uh but it's the transparency education and support around helping our node operators to swap uh and then tooling around making it super simple to swap because if as a decentralized protocol if any of those things weren't in place people wouldn't swap you know you need to have transparency you have to have all of that education and support to get them there yeah so nixo with that I saw online that you had um post like sort of done the math in terms of what this means if coinbase or when coinbase diversifies execution clients what does that mean like overall for like the health of ethereum in terms of this like the path towards this diversity so right now the best estimates that we have are from two different um data sets and one data set is a node crawler and that doesn't actually give us what we care about which is validate our client diversity so it gives us good node client diversity but it could for example connect to a node and uh ask it what client it's running and that node could be running 600 validators and it doesn't know that so it weights them equally with a know that's running just one validator the other data set that we have is just survey results so it's somebody um who works at Beacon chain named Sonic who called up a bunch of um operators and said what are you running and he assumes that everybody is telling the truth because right now there's not really any U perverse incentive to lie um and so then that then that is sort of extrapolated to the rest of the network and so uh that got us extrapolating that that survey data to the rest of the network got us to like 85% G um and we saw the best the best indicator that we saw that that was accurate data was during the bugs earlier this month or earlier this year when Basu went offline um the number of nodes that went offline pretty closely matched what we thought Basu represented in the network and then when nether mine went offline it also represented and those are the two clients that follow uh gu most closely so we had a pretty good idea that 85% was accurate with coinbase running 15% of the validator Set uh them switching over to 50% nethermind means that that's uh already by itself a 7 and a half% drop uh kilm also just uh switched to 50% nether mind so that um and they run like 5% of the network uh so that's another two and a half% so just in the last week we have well when coinbase switches um between Kil and coinbase that is a 10% drop in um G which from I think we had actually gotten down to like 75 or 80 um it's like we're being interrupted for Focus being interrupted for a commercial right now we might actually be below the 66% threshold um with that change we are not entirely sure because so much of the network isn't reporting and if you work for or run professional node operators please self-report contact Sonic and self-report it really helps um because there's no way to really get more accurate data at this moment you probably saw this but there's actually a proposal in the engine API that was recently accepted to share the information from the execution client to the consensus client and the consensus client will put in its graffiti by Def fault which execution clients run in isn't the concern about that though that it's really easily spoof it's easy to spoof certainly but like most validators wouldn't you would think so I mean I don't know we'll have to kind of see how it goes the thing is you could also spoof a lot of the other uh signals about what uh client you're running I think the most accurate thing is ATT testation packing that's like the most clear fingerprint about what consensus client you're running all of the other you know ether node things that we've used to see what is the network distribution of execution clients has always just used what's in the status of the peer-to-peer exchange and it's easy to change that if you want to every Fork of gu first thing they do RIP grip through the whole thing change the name of the client so it says something different over peer-to-peer I do like I do like that though because that's basically opting into survey data or like it's it's setting an opt out for survey data so even if it's easily spoof when somebody asks you what client you're running in they just believe you that's far more spoof than than having to change manually change something in your client and if people are just removing it from their graffiti like big operators you can just kind of say why you know what are you afraid of what are you hiding from from the people to be fair that like Rocko uses the graffiti to to track our our node operators and no sorry track what clients they're using um there we have like a special kind of like encoding to to to do that um most people use it it's built by default in so anybody want to go to the Zen Zone yeah I really want to go to the Zen zone I'm really getting sold on this [Laughter] honestly the the issue with the graffiti is it's also delayed because they you have to wait for proposal to to actually so there is a bit of a delay in that but you can get you can kind of get the best gist yeah without being too uh uh privac well by keeping a bit of privacy so I also wanted to ask let's say there is a bug in gu right does the super majority get bailed out like what happens you know like what would happen if so it depends on um how much of it is a GU the time it fails and how much it tests there's two magic numbers to worry about 2/3 and 1/3 if a client under one/ thir fails the chain will finalize and it'll go on just fine without that client but if 2/3 um attest to uh improperly produced block that's when the slashing comes in and where the concerns about um leaking over the over the next couple of weeks does so just getting gu below the 2/3 line um already solves the biggest existential money threat that people might face for attesting to a bad block but that's if gu produces a bad block and remember a lot of the blocks are being produced by block producers now not necessarily the Cent mining so another type of bug you might run into is a block that gu rejects that is otherwise valid and that'll resolve itself you know within a couple of blocks they'll just Fork it do another block and that poor proposer loses their loses their money um so that you know if if there's a bug invalidating a block that was otherwise valid then then then the impact there is is actually quite minimal mhm yeah the the 2/3 thing um as as like K said was you create two forks and in in in they're essenti two universes uh and in both Universe if you have to choose between those two universes and in both cases people are going to lose money so that's where you have a real issue that's where the social upheaval comes from because you're you're essentially creating two separate chains at that point right but the consensus fact says that you're only supposed to go on the correctly validated chain which is where the social which is going to be the chain side where people lose the most money and that's where the social battle comes in where they say hey we're going to lose a lot of money can we go back in time and change the rules so we don't lose billions of dollars and that's where the real drama would happen it's also part of the reason why coinbase switching was such a big deal is because there were there was a very small cohort of people who was saying like I know it's important not to be on gu but the big moneyed parties in the industry are on gu and so if they end up on a part of the chain that is not considered canonical I know that I'll have an advocate to uh advocate for my validator being on the right chain or being recovered and so coinbase and like I I don't think that actually has Merit because coinbase is very lined and I don't think they would um they would advocate for anything that like destroyed the although Circle would is a wherever Circle says the chronical chain is that's probably that's true that would be um so the coinbase switching over does have like implications Beyond just the percentages but you know I gu obviously we're talking about gu here a lot and because it's it's had the most developed right it has the most resources their in Foundation has sort of contributed to it or has contributed to it not sort of um but I I'm wondering too like what is it specific like why are users sticking with gu is it because just like because it has the most resources or is it because like it's just sort of oh everyone's sort of doing it I'm you know I'm I'm joining this I'm going to I'm going to immediately just go into the most Pro like competitive one I guess I don't out what because they already set up their validators and they work sure and it takes it takes an engineering effort to figure out how to safely switch over especially if you're running millions of dollars you don't want downtime you don't want anybody being angry at you and so like you're going to stick with what you know works and you really should have a backup plan regardless um especially in a time where there's not an emergency happening um and so there should be engineering resources really dedicated to setting up fallbacks and um knowing how to switch your primary execution client if necessary M how is the GU team going about all this conversation around this I mean there's I mean I understand that like people like you have spoke here have like triumphed this issue for a long time but you know for the wider crypto ethereum ecosystem and crypto ecosystem this is sort of back in the news right so how do you how do you go about this when um you know your gu is sort of the like the super majority here but at the same time the the ethos around all this is to sort of make it more equal Equitable yeah I mean I think a little bit like we talked about at the very beginning of the panel we're trying to think about how can we make it easier to run other clients and how can we make it so that people can integrate other clients into their workflow even if they continue to run gu I think being able to Multiplex different clients together that way you have higher confidence that the network is going to accept the block as one of the most important things but I mean I have to say in general like it's not something that we're working like day and day out to get people to stop using our software sure we're building our software trying to make a good client and I think that other clients are going to surpass us like guest super majority is not something that's going to be the case in three years I'm almost certain of that and we just have to continue playing our role in this ecosystem as being a robust client and helping to try and lead the protocol forward and other people are going going to continue stipping up I think the best that we can do is just be on a panel like this and tell people that you're probably not going to get bailed out if the chain happens to fork and so you should really consider running another client right and so but what are those magic numbers then around like running the other client like in terms of I mean you have to have under 66% gu okay as long as the chain doesn't finalize an invalid block a block that you know G is accepting and the rest of the network isn't then right but in terms of like the share of the I'm thinking more what we were talking about the second generation of clients like where do we want to see those numbers like creep up to um for for like another mind aesu or or an Aragon is there like a specific number we have in mind or is this just only focus on getting it under the 66 I I mean it's really as long as you get it under the 66% Mark if the other 34% is Basu then that's you know I mean we do we do want to have a lot of clients we do want to have like a good dispersion I think that that is the the best thing for the ecosystem in general that means that every client is like pulling is you know punching at its weight and everybody is like also in agreement that all these clients are doing the things that you expect and so that's where we want to go but reducing gu below 66% is like that's the number one thing that should be the app the most important thing that we're doing right now as a community in an Ideal World with five clients it would be 20% each but it also doesn't matter if it's 20 20% 25% 30% as long as no client is above 6 mhm okay I mean the benefit of having that that 20% split is is that is kind of like liveliness if one if one isn't is has a a bug where it crashes uh it means that you know only that portion of the network is affected or that portion of the blocks that are being produced is affected um if it's a higher percentage then there is a little bit more that are being affected in terms of that a few Miss blocks and all that sort of thing but generally if if we've got even we got a good distribution anyway so uh you know the next block will will come along the next slot will be filled and all that sort of thing you know ethereum is designed to be live um and so it kind of works like that but um yeah you might get some Miss slots some Miss blocks if you if we didn't don't have that split MH so one piece of the puzzle if we were having this discussion last year the numbers would be 25 because there was only four clients last year and now ref's coming along and I think as far as balancing out um and and unfortunately taking Market here zero some game re's gain is someone else's loss but I think reth is really you know in position to really even things out a lot to bring bring it more in line into the the safe zone of less than 66% of the majority client how effective do you think things are this time around in terms of that pressure of like client diversity because I I think like personally I feel like this is and especially in ethereum there's all these conversations like problems that are that the developers are working to solve and if it's not one thing it's another and and then it sort of becomes like a a distant memory um and we've been here before with this client diversity issue so how effective is it this time do you think I think that there's a lot of push and pull because when we did the last um social layer push for client diversity was for the consensus layer client and like the the number of solo the percentage of solo stakers on the network I think was a lot higher um but as more money comes in more of that money is going to large state operators which on one hand means that they're running a lot more money and they should be a lot more concerned but on the other hand it means they're running a lot more money and they are nervous about switching um so there's a lot of push and pull in both directions anyone else have a take no okay yeah I mean risk management um these these large companies they they should be doing risk management a little bit better um you know swapping isn't that isn't n that difficult necessarily but I understand that there's a large engineering effort to do it but yeah there is huge risk in in in this in this particular instance sure so you know we're talking here about how effective this is this time around but when do we think this diversity problem is going to be solved let's say you know you just mentioned that you don't think that gu will be the Super majority client at some point or I don't know what the time frame is for that but that another client might overtake its share um like where are we heading in terms of specifically where like when this will be solved or when this is not a non-issue for ethereum or will this ever be a non- issue for ethereum I think the next big shakeup in client diversity is going to happen when we ship veral proofs whether we ship it with veral trees or not because that's going to have an entirely different nature of client that you can run with it you don't need the terabytes of dis space um in theory you could you know validated on your Smartwatch I mean that's the joke going around but there's so the SmartWatches are powerful enough and the clients are so small and useful um that that's what could literally happen so when it's when the impact of these vertical clients there's going to be a big huge incentive to reduce the amount of um of of software and storage space you're going to need and I think that's really what's going to drive when it's really going to be solved I think that's going to be the big external pressure that's going to solve it I think um until then there'll be minor incremental changes to bring us you know into the safety zone and I don't think it'll be that um I'm not terribly concerned about it I'm not moving you know what I have where I have my stuff staked um I have it across multiple clients um and I'm not you know as it is today I'm willing to take the risk I don't think I I think there is you know a risk but I don't think it's existential to what my investments are mhm great I mean from from from uh so our node operators have always been quite early adopters early adopters in the space uh so they they they took on these these kind of newer clients very very quickly but these clients are now very mature uh and so you know each each of the execution clients are very mature and and great clients great well our time is up but thank you panelists for being here today it was really fruitful conversation and thank you to the audience and enjoy the rest of your time here thank [Applause] you all right all right let's just do another round of applause no one ever no one ever died from getting another round of applause thank you Margo and the client Community anybody have their opinions changed about anything in this last 30 minutes let's get to Pro diversity hands in the air got some yeah okay anti-diversity hands in the air taking a photo of you now um yeah so thank you all of you this is really filled out you guys are very interested in this stuff um the next session we'll have is actually a topic that I'm sure all of you have messed around with but it's uh layer tws and Layer Two Solutions uh basically how do they scale efficiently on top of the blockchains that they're built on and uh the various implementations and I don't know what they're going to talk about the folks from medow but I mean I work with this stuff all the time and there's a lot more to I think the future of layer 2os and rollups and their capacity to bring change to Regions all over the world than we are currently giving credit to and this is to say there are regions where like Layer Two solution with Community currencies backed by like usdc or Bitcoin on the layer one could be a very very viable solution for some communities who experience uh banking interruptions uh internet interruptions all sorts of things so when it comes to you know right now we see layer twos as a solution for just getting more transactions on chain specifically on ethereum right now and in the future I think it might be a way of looking at it as this is now a way for us to get a complete Humanity picture on of our transactions happening on ethereum and I think that you know that's something that's worth worth uh looking into anybody here working on a layer two anyone one guy one guy three guys any women any they thems one and so I'm sure that some of you who are here already know a lot about this stuff um we're just going to get some more chairs set up for them but I'd be curious if anybody who is going to be you know in the Q&A after this get your questions ready now because like I feel as though I watch a lot of layer two conversations just go in circles and I never see like people coming forward and being like hey what about this what about what they're doing over here um and that would be cool another thing I'd like to bring up for everyone because I'm supposed to plug this there's hashtags and various things you're supposed to be doing like going to parties and getting swag um I would remember that information if I was better about this but I'm not so I'm sure at some point there's going to be a voice that interrupts the panel that you're watching and says a bunch of the stuff that I'm supposed to be saying right now um o okay without further Ado it does appear like we're almost there um I'm just going to reintroduce myself cuz I feel like it uh my name is Dr