Client Diversity Panel | Merge & Scalability Summit

[Music] [Applause] [Music] all right with that we are ready for our next talk and that is a incredible panel talking about all things clients for e2 and plant diversity which we know is a really important topic here we have a lot of panelists for this one so we need superfiz for me steak we have carl from ethereum foundation with that line from uh chainsafe and having management the week in ethereum and moderating this will be ben who is the product owner at tecu i'll let ben take over and ask all of our panelists to enter themselves so without further ado let's hear about client diversity welcome everybody thank you cardik and good morning good day good evening wherever you are ethereums uh it's great to be here we are missing one i think lion should be joining us uh shortly but i don't see him yet but let's do some uh intros so uh i'll do myself first i am proprietor of what's new in eth2 which you can find at eth2.news and as cardig said i am also product owner product manager of the teku e2 client which i'm not representing teku here today in this client's diversity uh panel but uh um i hope that gives an interesting perspective uh evan you're first on my screen yes i'm evan van ness i'm most famous for trying to cancel ben for using the youth 2 nomenclature still i also have a newsletter called weekend ethereum news and i've done stuff for a while i guess that's my bio thanks evan carl everyone um i'm uh colbeak i work as a researcher with the ethereum foundation on all things ethereum consensus lately been doing a lot of fork choice stuff and yeah caring a bit about client diversity [Music] thanks carl hey superfiz hey uh so i'm superfiz uh i am probably like an old-school um miner and um with a couple of friends i decided it would be a great idea to co-found a group called eatstaker and through eatstaker we promote uh involvement in the beacon chain by normal people and we work to be welcoming first and knowledgeable second wonderful we love eath steaker uh lion you made it uh say hi and give a quick introduction please we're we're not i'm not hearing lying yeah we can't hear you how about now yes it's good sorry so yeah i've been part of that notes since 2018 ogden helping make running notes easier and then when ethereum came ethereum 2 game so making staking simple to non-technical people has been one of the motors and i'm also called up in lodestar so it's awesome to see this space from both sides of the equation thanks panel uh so i'm going to start by setting the scene and then we'll get into some q a um so the context i want to begin with is the ethereum 2 beacon chain so this is the proof of stake network that we've built uh today we have almost a quarter of a million stakes which is uh eight million ether worth today and this is staggering when i calculated earlier 30 billion dollars worth uh being managed by our eth2 clients um and so these stakes each one of these stakes is managed by a piece of client software from one of five teams my own teku or lions loadstar or lighthouse or nimbus or prism and according to the best statistics that i've seen this stake is quite unevenly distributed between these five clients it looks something like prism having between 60 and 64 almost two-thirds of the stake staked on the prism client lighthouse around 22 percent teku between 14 and 17 and a half percent and nimbus and lodestar each below one percent and these numbers come from a new analysis by michael sproul from the lighthouse team which uses a really really uh ingenious technique to kind of fingerprint uh clients according to the blocks they make so to kind of get to the meat of the client diversity thing so let me ask you first carl uh there's been a lot of hand ringing lately about this imbalance this uneven split between clients on the network in particular this dominance of prism why is this a problem why are we concerned about this so within um any consensus mechanism right you you um you you have a set of people trying to agree on something and um by by going down this this this road which we went um for for for eth2 um we created many we had one specification and from this we both have many clients and these many clients allowed us at least in theory to have a more robust network by having a failure in one client hopefully not affect the network as a whole because they are one one client would hopefully only have a small portion of the network and and and we could contain that failure and hopefully fix that that mistake um in theory this is true and is in practice as well uh but what hap what we've had now is that uh prism has quantity statistics become a really dominant client and this means that the more and more nodes become convinced that what prism says is the truth becomes the truth um and so in a world where we have uh and more than one third of the nodes in the system um so if if even just with more than 33 percent uh prism nodes or any client and if if if they have a bug or uh start disagreeing with the rest of the network uh then we're gonna stop finalizing blocks on eth2 the chain will still make progress but there will be uh there's this nice finality gadget which uh gives you economic security economic guarantees um or past how uh transactions have gone and and that they will forever be a part of the ethereum chain and that will no longer be uh guaranteed until a future date when we can get this one third back online and following the canonical chain so even more than 33 percent is not ideal and so with the between four and five uh main net clients uh that we we have at the moment uh we should ideally have somewhere around which ideally have about twenty five percent um of uh in each of these clients which means it'd be below this threshold and life would be uh all dandy um this this is further accentuated uh with sort of a second threshold we have uh which is what we call a super majority um which is where you have more than two-thirds this is the 66 percent which uh according to the statistics ban was uh digging into a bit earlier uh where prism is hovering just below and what happens in this scenario if if there is a bug in prism or prism does uh for whatever reason start forking off or uh producing bad blocks the prism nodes themselves will think that it's the the rest of the network which looks like one third from the prism nodes perspective that's making the mistake and so the prism nodes will happily go on and build a completely valid chain from their perspective um and this will progress on and the the the result of this is that these prism nodes have this uh the they have their economic finality sort of guaranteeing the wrong thing um which which is not a part of the canonical chain which is really scary um instead of this this failure mode where we have sort of one uh uh sort of degraded quality of the chain but everything still progresses we now have a problem where uh we need to go in at a sort of hard level and correct the these prism nodes so that will require uh database corrections for the prism nodes uh we'll have some serious issues to avoid slashing conditions because the prism nodes will make votes on top of each other which will have certain opinions about the chain but that'll be incompatible with what the the rest of the chain actually did which would be the sort of correct chain um and so this is the sort of like kind of scary scenario where everything's going off in one direction and uh ideally this would never happen and uh all those good things but in practice we never know um and so so i think we we we at this like tipping point where uh in any more and is going to be if there is a fault really hard to correct not not impossible to do so we've seen chains do this in the past um but a correction to do that would would look terrible and would give all the all your favorite maxis that you uh uh non-ethereum maxies that that you disagree with give them a lot of ammunition um for for having to sort of go in and manually hard fork away from any quote-unquote problems so this is really what we're trying to avoid by preventing any node from getting to this two-thirds threshold right um yeah yeah so the the point is that once um a client has finalized its own chain so in this instance prism had made its own chain recovering that and bringing all the clients back together again isn't a matter of a kind of fix the bug and and move on there's a there's this slashing thing going on whereby they fundamentally disagree and for either side to join the other chain means that um a bunch of validators are going to get slashed which is going to be horrifically expensive so this is a kind of new feature under proof of stake which doesn't apply in proof of work right proof of work the chain splits you just kind of fix a bug network comes back together we're back on the longest chain and everything's cool under proof of stake it's really really really complicated to fix yeah and and it's that that's sort of a a feature of this finality that we have right so it's a bug and a feature because we have the system where we can't agree that this will forever be a part of the chain uh this this finality mechanism um if if something goes wrong and we have have something that we think is a part of the chain that's not or there's disagreement about what is a part of the chain and some people bacon incorrect ideas that's really hard to to get out of the system without these uh sort of financial guarantees we have triggering and and slashing a lot of stake cool and just to be clear for everybody we're not dunking on prism here where any client that has this super majority would be a problem in this scenario just so happens that the prism is is the most used client um evan you are on record uh as saying that the long awaited merge to proof of stake could be delayed unless we kind of sort out this client diversity issue unless you have more evenness in the clients can you unpack that for us is that a genuine threat or are you just whipping up some hysteria you want to mute a little bit early so i missed the last part of your question um but look i mean we can't turn off proof of work until we can do it safely and to do it safely we need to have a client at a reasonable distribution chair you know i think carl said in a lot of words um something that i'm doing to tl dr for people who are sticking with prison if you are sticking with prism right now your ease is wildly at risk that has nothing to do with the quality of person's client or any other client it has to do with the fact that the spec incentivizes decentralization and right now you are not contributing to decentralization it's the opposite so let me say that as clearly as possible because i actually get a lot of friends that when i tweet things uh like slide into my dms unhappy with me telling me that like i'm the one that doesn't understand the spec i say like no no no you're the one that doesn't understand uh you chose to stick with prism i understand you did a lot of prism uh test nets and you were comfortable with it but unfortunately the spec is very clear that like decentralization matters and you are not contributing to decentralization and so your eth is at risk so um and you know we've sort of talked about like carl talked about about one-third and two-thirds and those are obviously the important thresholds but it's even it's even there's some thresholds that make a little bit that are important as well which is to say that you know if we lose finality and prism has 49 you're going to get slashed or you're going to lose from the inactivity leak which is which grows exponentially you are going to lose a lot less if prism is under half of the staking clients whereas if prism is 65 you're going to lose i mean depending in on the circumstances a lot maybe half maybe all you know people get upset when i say this but i don't know i'm just stating obvious facts so to do it safely to turn off proof of work safely which is a long time goal of ethereum we need better client diversity anyone want to comment on that or uh we can leave it for uh pick up a bit later confess um you're right about a lot if if we lose finality um prism clients could be slashed absolutely true but um we need to acknowledge that that is currently a low-risk event and i do think it's very important for people who are on prism to consider using minority clients especially nimbus right now um but if we use scare tactics if we make our user base afraid of the chain then they're going to be afraid to interact with the chain rather i think education and promoting the ideas that why we need client diversity using the right tools for the job getting people to say um i may need to change validator clients a few times and i'm okay with that is much better than the scare tactic of saying get away from prison where you're going to get slashed because what we have then is a reflexive action where prism is vilified where people start trashing prism and saying terrible things about it to get other users away from it and what's likely to happen is others will go to lighthouse and then we will have lighthouse with the super majority the only thing we'd the only way to get away from it then is to trash lighthouse so rather than this uh this fear-mongering and no it's not fair evan you're not fear-mongering i think you're you're stating a very a very possible scenario i i really do believe though that education is the best way to get people to make these informed decisions um and rather than reflexively doing everything we can as quickly as possible to get people off of prism um we develop these multi-client tools we devel we develop quick switch tools so that home users aren't intimidated they're not a fear of not afraid of losing their stake but instead they can click two buttons let it happen um and they feel empowered uh they're supporting the network uh and they're they're not operating out of fear that's the i think that our our big difference would probably be that i believe education is the way to reach our goal um and you might be more willing to hit people with what you would call the hard reality um that they're going to be slashed uh and i just i recognize that potential but i think the likelihood is low so you say the fear of them switching clients is that they would lose i mean so i think that's something that we can talk about should people be afraid of that if you take your client offline and you have the slashing protection database for a day and then you come back up with another client you know i haven't done it because i run minority clients but i'm pretty sure that that is like a pretty much 100 no risk scenario yeah we can talk about that later however i mean so if there's that fear okay that's a fear it's it's a you know amorphous fear however there is a very concrete fear which you should which prism stakers do not seem to have currently which they should have which is that if prism has a bug then the network will lose finality and you know we're in a bad spot and the inactivity leak will start melting away their stake and that is something that they you know probably should be afraid of they should at least understand it and be cognizant of the fact that that is a risk that they are taking we have this really neat thing in east staker where we have a lot of of first time validators first time stakers people who it's their first time um engaging with a network in any real way and so um they they found guides they found the coin cashew guide or samarasat's guide and they installed um prism and i'm proud of them they did a great job they are participating with the network um i think it's just so important that we continue to meet them where they are rather than um stand on our high horse and make these these bold demands of what they should do when they are our user base they are our network and so um since you i want to get to that line but since you got the mic fizz uh and this is relevant we you we're focused in this in the recent answers on um the current solo stakers on on the network are they the key to turning around client diversity who is responsible for for making this happen what what's the route from where we are to to where we need to get to uh that's a great question there in my mind there are a couple different ways to look at this um i think that we do need to look at institutional stakers um and um after some conversations with um with leaders from the ef i feel confident that those that outreach is being done but you know i what i can focus on is my scope and my scope is the home users and while i do encourage them to switch clients i really feel like there's a development gap or a software gap there where uh they i don't know if deserve is the right word but what what the community should offer them are these graphical tools these easy switch tools where uh they don't have to to put their their stake out on a limb to hope that they're switching right to support the network because our users generally have two concerns um they have a large investment and they want to support the chain in the community i value both of those things and i don't put either one of them ahead of the other and so what we need to do to reach them and colfax through steakhouse is working on these tools i know there are other efforts working on these uh is to um take the existing implementations and put wrappers around them or interfaces with them so that users can go to a web interface they can see their stats and hopefully in a few months they'll be able to select a drop down that says switch from prism to um teku and yes i did it i saw you there and so when we have that functionality uh i think the users they'll feel supported they're not going to lose their stake um and and supporting the network will be within reach so what we need from the wider community is more support on this effort we need money we need developers we need uh thinkers to to kind of put as much effort into this easy switching as we've put into some other very important things cool yeah there are there are other things to consider um there's um the majority of stakers not on the network yet i mean i think probably we're about a quarter of the way there maybe to the total stake so um if everyone in future adopts the minority clients uh that could correct things and the majority of stake on the network is actually managed by um staking services uh institutionals um stakers uh and again i think you know they they're they will have the most influence um i love the army of solo stakers i am one for the record i run a minority client um but you know i'm not sure you you can guess um but maybe that's the solo stakers it's kind of unfair to pressure them too much because actually you know they're not going to move the needle a huge amount uh and it's really others who need to take responsibility they're professionals right yes i want to be really careful about that that's that disempowers them they are an empowered group and so i i think that we're about 40 of unidentified uh validators and that i know that there are some whales in there but a lot of that are home stakers and so a lot of the responsibility is institutions but let's not disempower those home stakers and say that nothing you do matters because it does matter one person switching is going to move that needle one hair and that matters one person from prison to nimbus uh it's it does matter at least i think we need to believe that and we need to act on that because that's that's the health of the network i call myself the beacon chain health consultant if you don't know that so yeah yeah so i think it exists at multiple levels um so the absolutely the easiest way to sway it is to go to your favorite big institution if they're running a if they're running a majority kind we speak to them and try to get them to diversify but that's moves the middle bunch but everyone has a role to play in this and and contributes to the the overall distribution which is what we care about um we also don't want a scenario where the big institutions run one set of clients and solo or private validators run another set because then if we have faults now we're starting to get like splits in our community between uh the individuals and institutions which is also not a great thing to have um and i also think like your role as a staker part of it is to uh be worried about these these kinds of things right to observe that you have reasonable uptime and that you are picking a a client that's not completely dominating the network these these are important roles and it is literally what you're paid to do right this is the this is the reason you earn uh rewards back right is to do to do this safely and securely to improve the decentralization all these things are important um and part of all of this is to to to choose reasonable clients and so uh you don't have to do it all the time um don't don't panic about it don't change things if you're not not comfortable but at least spend some time thinking about it um and if you do feel like you have the capacity to to to change away from a majority client i really think you should consider doing so great answer yeah a couple of points i haven't thought of love it uh lion you've been very patient uh you're a client dev uh what do we client devs need to do to promote client diversity yeah it's a good point i mean prism has done a fantastic job and that's why they are killing it i mean it's obvious all the clients should see what they are doing why they are killing it and why people love them like they clearly said graphical interface and windows support that's it if everyone does that then it would be much easier and as everyone has said easy switching standardization that's the key another point is that people like def or whoever has access to this tool should see what's the funnel of going into staking why do people choose prism there has been people saying oh because it looked cool okay but where were you looking at um is there any resource that can actually provide this information that hey there are more clients um just go pick those ones because reasons um so yeah i think those are the key things that we should work on yeah i think one of the issues uh up to now is we we haven't really had good data um on client diversity we've had sort of some guesses some bad data it's very hard to get good stats these stats that michael has produced are a game changer i think um yeah but we will probably never ever have actual data that's a worrying thing like in in reality prism could be a minority client and we just don't know well we have so a few things one is this is partly by design right part of having a robust secure anonymous well pseudo-anonymous network um is that it is hard to extract those data right it's not here here's the sign up sheet um put your name and address and we'll find out we'll find out how to do it that's not how it's not like any of this works and it's part of what makes it so hard for us to have this data and as much as we wanted to to help explain things we also don't really want to have to like bake this in um because it exposes individuals um and and so we we work very hard to avoid that as well um and it's not it's not that we're like not looking you can look very hard and there's this tool that everyone keeps talking about um michael sproul's thing is is like really some very ingenious stuff uh to to to to try extract it um you probably can't do too much better um which i think is like a really cool thing because it shows that we have actually built a robust network um so it's like trade-offs that that exist here um we have other data sources there's sort of uh we we have we have no trackers um uh there are a few they're being built out there's uh the the eatstaker survey um with uh which supervisor has been very involved in um and there's some some interesting data uh what watch out for some some interesting alpha that's going to come out of that uh soonish um yeah so there's we there is data that's just not so obvious sometimes we only find out when the things go wrong right i mean we knew back in february when uh what prism share of the network was when all their clients simultaneously stopped producing blocks so there were it was like 70 percent at that time so we're moving in the right direction you unmuted yes i unmuted to say exactly that we actually did know how much how much the prism has in the network as of you know last time there was an issue um well and they've they've we have seen some evidence of of migration it has just been slow evidence slow migration and one of the things one of the reasons i have delayed pushing client diversity harder is because we can't really see the needle when we can see a very clear representation of these proposals and know exactly where we're at then we can measure progress and we can know what initiatives are successful and what aren't can i talk for a minute about prism and how they became successful yes sir so prism is successful um i think this is opinion but it's also based on experience in fact um prism is the only client that has a native windows installer um and a large part of the uterus user base is windows centric they're not ready for linux they don't want it to use they don't want to secure shell into a server um and so that's a big boom another thing is that they have provided um 24 7 support in their discord since like the topaz test net uh and so they were doing these public test nets we were giving poeps out for them we were holding launch calls um and i feel strongly they were doing this work to promote the beacon chain but what ended up happening was rather than simply promoting the beacon chain it created this um prism allegiance which you know it is what it is but i don't think that was intentional on their part they were trying to promote growth of the community um those two things they did and it's not that anyone else didn't do their job it's that uh they hit on two really successful ideas and that's also a great we onboarded a lot of home stakers by helping them feel comfortable through all those test nets and we wouldn't have that we can complain about prison dominance but we also have this this really strong base of of home stakers that might not exist if it weren't for prism's engagement yeah they were first they were first by a long way and i think that really established the name recognition they and they did a terrific job i mean really awesome what has surprised me is how the institutional stakers have just followed exactly the same patterns um yeah i'm wondering then you guys didn't do your job well what happened yeah yeah the background to that okay um first thing just come back on fizz take your runs just fine on windows you have to install java but you know it's not too much of a lift um yeah toku the client aimed at institutional stakers guys come on why are you not using it this would you know a few large institutions adopting adopting tech who would uh really um really uh balance things out more i mean the real interesting questions are around nimbus uh low star yeah where the plans are so so this brings us i mean you know uh lion when i asked you about what client devs can do you framed the response in terms of this effectively i'm going to paraphrase you here in a way you won't like but make all the clients the same um you know same in faces same um experience for users just drop in replacements for each other we've taken a different approach with teku we've said no we are not going to address certain parts of the market we are the client of choice for institutional stakers to an extent nimbus has targeted a a group the low powered constrained device market loadstar itself you're looking at like clients in browser this kind of thing you know devil's advocate here but maybe a different route to diversity is really to double down on specific use cases for each client rather than everybody try to be the same uh any thoughts it hasn't worked out that well right everyone's specializing taku nimbus uh and as we are at the bottom of the list and it's i mean at the end it's a very specific use case so while the specification it makes sense if you are just going to go out for stickers i don't think it makes that much sense you know in our case our libraries are widely used on browsers and other types of applications and we try to make our note um kind of the backbone of flight clients as a hopefully future replacement of impura but that's kind of a parallel effort people just taking right now and the same for nimbus i mean you cannot just put the note on the phone yet and you will still just run it on a machine as you would for anyone else so and you can do both like we we can still follow standards and be special that i mean if you want to migrate somehow you still need the stun point i think some of those definitely comes down to just being early right the the need for light clients is much less until we really have the economic value um attached to to the proof of stake chain right our post merge litecline's gonna be a lot more interesting um having having stuff that compiles well done to a mobile device is gonna be fantastic um and like there are multiple ways to to improve security and diversity of a chain right you can you can be a validator which is what we've been focusing on today uh but you can also just run a node that that helps everyone right and and has you you now have your own source of truth um so uh this this is another way we can help improve the network and then once that's what's happening we may see people starting to care a little bit more about having a super lightweight client they can throw on in the in the background on one of their other machines or something like that and so i think some of these specialization use cases we just sort of haven't really reached that point yet uh which i'm excited to see um and then the other thing i think that ties in a bit to take on what we're talking about earlier um and that's some of the discussions that i've been involved in with these institutions have been rather like scary considering how large they are and that's just that they didn't really do any research they just didn't look hey we see everyone's like we we heard the the word prism a lot must be good cool let's go um there it's honestly been that bad and sort of you'd expect based on the the the haircut they take or for staking rewards that they would care about these things and that they would care about client diversity um but in the cases of some of these these institutions they're just simply not true um they didn't put in the time to do the research and uh that that they're not really worried about uptime it's just like an extra service they happen to provide and pulls in money for them it's not the the approach was sort of money first and uh improving a network second um and like overall they didn't consider the incentives of the network they're participating in which is ultimately like bad for them as well um because as ever evan was saying earlier like you don't want to be running a majority client um in these scenarios it's it's strictly worse for you so i think they need to consider some of these these things a bit more there's a name for that it's called negligence honestly entirely negligent and let me put it down let me put a counter to you about that which is very relevant which is actually you could call it completely rational behavior and the thinking goes like this and i suspect there's a lot of thinking out there we will adopt the majority clients because it's too big to fail no so if prison fails you know 60 70 percent of the network gets into a mess um of course they're going to get bailed out i mean obviously they're going to get bailed out we're not going to let the beacon train go down uh over this um you know and uh so they'll be bailed out if they were running you're like a five percent client or a ten percent client then they're taking a risk because nobody's going to bail them out if if there's a horrible bug right completely rational what's wrong with that reasoning why why are we how are we going to push back against that the network doesn't have to go down and if if you use common sense and exercise that diversity then you ameliorate the risk of that network going down and slashing and needing to be compensated because uh it doesn't exist so in game theory it makes better sense for them to choose a minority client to prevent an outage that they would need to be compensated for or want to be compensated for but kind of carl said something that really really interests me i don't think these institutional stakers care they are not focused on eth they don't care about our ecosystem they don't care about our growth development transactions any of that they're there to make money and the least amount of money that they can put into this to make a return is what they're going to do it's it's one of the reasons i really encourage users not to stake with those institutions they don't care about what we are doing they're there to make money and so the best thing that we can do is uh either solo stake or stake with a trustless open decentralized solution um not give your money to the big exchanges and again this is gross negligence i think this cannot be repeated enough because staking pools especially say public companies in the united states or anybody that has a nexus to the united states given the the united states court system loves to try to enforce its will on the world is if if a staking service loses their clients eath and gets gets a massive inactivity leak or gets slashed i don't think that whatever legal language they have in their contract is going to protect them from the fact that they were willfully grossly negligent and so you know eventually you know they're hopefully going to figure this out the question is whether ethereum loses finality and pulls a mini salana not a real solana because they went completely done don't go i i i think what's there a few interesting things to explore here one is that there's there's a failure spectrum as well we've been talking a lot about the worst case scenario where there's a catastrophic bug and everyone gets slashed that is very very unlikely um these these clients are coded incredibly well they've been battle tested through multiple test nets we've had excellent bug bounties which have found things but we haven't not not not lately um we've had audits for for everyone these and and like it's an open source ecosystem so many many eyes on all of these clients um so that really helps out for all of this but there are other more subtle failures which can cause temporary failures of liveness for so so certain uh the the majority client could stop producing uh attestations uh which is not like a hard fork but it does mean we're now gonna start um having to have to to to leak the uh balances of those those clients and i think if there's a more subtle failure then we may not bail everyone out because that's not so easy to do at leia and i think ethereum like as as as much as we we can have the debate ethereums had had in in the past we have we have made decisions to to to to handle the whole dial situation right um and and we've seen the blowback from from from all of that um in spite of what okay we can get it let's not get into that now but uh relative to that i think we will bias a lot towards not going in that direction even if it causes loss of of of value for a lot of people and i think another interesting thing is a bit to where i was saying is you can end up in the scenario where there's a split between institutions and private individuals and if if we start seeing that where sort of private individuals are running minority clients that have really worked to diversify and institutions are focusing on a majority client because easy and negligent and whatever um it's going to be very hard to convince the entire ecosystem to want to bail out the the institutions because they didn't care about and don't care about uh the the etheric ethereum ecosystem in general and so um why should the the community care about bailing out the eve please let's talk about debate at the scenario it's fascinating i i don't believe we should bail out anyone to be honest and absolutely don't see why it's actually necessary like seriously if there is a back and the network splits the minority network would either leak everyone or expect everyone to get slashed but the network could continue it will still include execution data but it will not be finalized so if people accept the non-finality and the risk of reorgs which is what we actually have today the network could just move on everybody decides who decides which is the right network i mean you got say 70 of stake on um a prism only network that was forked off due to a consensus bug and you got the rest of the stake on the other clients uh which is not yet finalizing but eventually will due to the inactivity leak which is the right network you've got two completely separate networks now we have a spec when it's not the first time that happens right evan's a spec maximalist yeah but we haven't had finality in ethereum before this is new right so two chains that are fundamentally incompatible i i i i agree with evan on this right we have a definition of what what it means to be compliant with with the ethereum chain so i think the clients that have followed the specifications um have by definition that they have the canonical chain whether or not it's finalized via some some something that's outside of the definition of what literally is ethereum um that's that's irrelevant but i do think that there's like the the failure that like terrifying failure mode where uh there's a fork off and the result is everyone's gonna get slashed and it's a super majority right great greater than two-thirds i i think it's unrealistic to expect um that not to be bailed out um as as much as i hate to admit that um it is just so much value that i i i think it would be at least the discussion would be very intense but i i think would ultimately settle on uh merging that back in possibly with financial penalties and we can get into what exactly that looks like but i think it's unrealistic to expect that just to burn all that money um and and pretend we're all good without it to be fair you're you're hypothetical that is not the case of what we are experiencing today where we do not have a super majority we might be somewhat close to having a super majority but we don't have one yeah so it very big it very quickly becomes it very quickly becomes the super majority of them because of the way the chains are constructed so let's say prism's got 63 percent there's a consensus bug they end up on a chain where their their validators are proposing blocks and attestations but nobody else is everybody else will very quickly get their balance leaked away and it will only be a matter of days before prism's got 66 percent and finalizes um the closer they are to that 66 threshold the less time we got to fix it and it takes at least a day maybe two or more to get a fix out so there's a real gun against the head in that scenario the further from a super majority 66 percent the below that the better you know earlier that's why we can't turn it off safely turn off proof of work safely until we are are lower than where we are now and so it's this theoretical two-thirds um you know so in a month we could have prison below 50 prism could easily be below 50 in a month it doesn't it's not it's not an act of congress it really just takes a concerted push by everyone in the community to say we have to do this now we're not messing around anymore everyone work together and i don't mean to be kuba yeah but i'm just saying it it's not that difficult to make it happen if if we just all push in the same direction for a moment in a month this could be an honest i've been trying for over a year you have been trying and i i have been trying buddy we're together but i'm saying it's not that easy every but well because it's it's a few voices it's not it's a few voices shouting in the woods it's not everyone working together and we have that potential as a community to move if we don't bail out and we burn half of the supply that would be like the most ultrasound money thing we could do like the ultimate man [Laughter] like i was iep nine lovers of three thanks so much there's a great comment about incentives um and so i you know one of the reasons i was so excited about michael sprouse data is that we can identify uh by proposal what client someone is running and and that gives us the opportunity to take a a snapshot and say today we know what you're running and if you change from prism to nimbus then we can offer you a poep and those perhaps can be incentivized i i personally i feel like it's still a great idea it didn't uh some of the thought leaders didn't really love it um but it's worth considering or talking about incentives for people who change from not necessarily money directly but incentives for people who change from the majority to a minority client but yeah i'm making approval incentives yeah yeah not getting slashed not getting slashed it's a real thing and nobody cares i mean and frankly like uh nobody believes me when i say it i mean i again i have i have to tell you that like i talked to the staking pools over the last 15 to 18 months you know they all think that we're a data center chain just like every other you know dpos pos basically the same change but um i i think that is like as as much as i was shooting on institutions earlier i i do think we're making like some of the right moves in that direction right we we've opened up some discussions with them and they've acknowledged that that they the approach they took was wrong um and and that they they will try to dig into this and and diversify but they're um i've been really excited to see if if if you go on reddit and like or on on each second be like hey i'm having this problem with prism help set me up you will get sort of ratioed out by people shining at you to pick any other client please just not prison and so i better not shout they better be friendly sorry that's true that's good but in all seriousness i think i i i think we've like reaching a point where it becomes very possible to to see this change happening rather quickly away away from these clients and i i think there's a lot of awareness certainly in the in the ethereum community due to the thought leaders shouting about this for a long time complaining about this for a long time um and working with these institutions to like try help them diversify and see the errors on the ways their ways even if they don't majorly care for our own good right i'm gonna um change tax slightly on an optimistic note have you ever seen optimistic say even i i wanted to say that like my understanding from staking pools is that they have sort of seen the light and that they now spin up non-prism clients but they have not switched all of their their validators from before they saw the light um and i think most of them probably i think there's probably some fear i think a lot of it is also they just don't understand that there's no real loss for taking their entire validators down for a day and you know they're worried about justifying their services to people and you know losing their penalties and i think they really just don't understand the risks that they're taking or they don't care no i think they i mean i think it's like like i said i mean i think that you know if they realize that they could get sued you know for not just possibly like the loss of the entire whatever happens in this you know in a bad scenario but maybe even something punitive like i mean like carl says to make money not to lose i think i think carl hit the nail on the head they they know they believe they'll get bailed out um and they might be right i i don't agree with that even though i i i support home stakers i don't i don't promote a bailout of of anyone who gets slashed when you signed up as a validator and you accepted the terms and you know the terms if you make poor decisions if you get slashed and i hope everyone from e-staker hears this you accepted the risks you need to make better choices if you get slashed it's on you i mean this is important about ethereum this is why it's secure why like we designed the system particularly in this way it's it's necessary and makes it better for everyone i mean personally i have to say that having listened to this panel i all of a sudden am worried about a bailout whereas i never were it was before so i think that like the message isn't even clear from this panel i mean i am 100 against a bailout like i will probably sell on my ethan quit if we bail stakers out who did the wrong thing and we're pushing for centralization same entire time i mean it's absolutely wrong you have a responsibility it's what you're getting paid for you know if you're going to get paid take the take the get the rewards you have to realize the risks you're taking i'm going to move sorry i'm going to move this on um let's say that everyone listening does the right thing and that we get all this sorted out and in six months time we're at like 25 cent on each client it's all rosy uh and then we do the merge um i looked at ethernodes today geth is at like 63 of the network sounds familiar right um is that going to be a problem post merge is there an issue on the execution engine as it will be the eth1 client side is that an issue for client diversity as well and do we need to get that more even uh what do you think yeah i think what's really annoying is that in at least in consensus there has been so much effort and money and talent spent at creating four clients that are almost equal in capabilities and still we don't achieve diversity in ethereum one you can say okay there is only one client that has all the longevity and features and there is another one that's going away and others are coming in but there is no equal to get in a way and still in anything in consensus we don't have the diversity so yeah just wondering how the hell we are going to achieve that in execution yeah so carl i'm interested is it actually a problem um having not the same level of diversity amongst the execution engine side of things or is that uh or is it just as critical as on the beacon chain side of things so i mean ultimately we can run into a a similar diverging situation um if if if certain uh if if we have a super majority and it goes off in one direction we we can run into all these terrifying things we're talking about earlier but i think we've got a few things county against it one is that this code has been running for very very very long um and like all the these clients really have been around the block um and they've been tested by crazy amounts of value locked up on the ethereum system so i think that really helps us handle that the other thing is that you start having these pairings between the the execution client and the uh consensus client and if you you you end up like from all the pairwise combinations you end up with way more possibilities of you can run this execution client in this this consensus client um and the result of this means that uh overall the like the each individual pair sort of counts as a different way to fail in this regard um and so this starts helping us like build up a bit more resilience against these these kind of mistakes um and so i'm a little bit less worried about it on on that side although more diversity is is always better in this regard and i think another interesting thing to like just quickly point out for everyone listening here ethereum like we are unbelievable in terms of diversity even with super majorities no other club no other uh chain comes even freaking close like go go try find an alternative client for any of the top 10 right bitcoin you'll find a few any of the others it's there are no other viable clients no no one else is anything this kind of brings me to my kind of final question uh called perfect segue uh which is which is this and we haven't got time because there's a couple of audience questions um we'd like to cover but um it's a big one was client diversity a mistake and as you say i mean no other chain bitcoin doesn't have it no other chain no other chain i've come across as sweats this at all they're quite happy with their monoculture um why why in ethereum are we bearing this burden which is uh so complex because we care about having the most robust chain and the most decentralization it is strictly better to be more diverse um we if we want to be the the the best chain on pretty much any metric that that this community cares about and uh this is one of the really important ways of achieving that and i think not having a diversity of uh of implementations only serves to weaken you i really don't see the downsides no capture of governance by quarters like say an orange killing that's and that's going to make a comment about another chain decided not to um hey can i shoot one thing in here it's been itching at me just a little bit um and and that is this uh the specialization thing and i know some of you are fans of of client specialization i i really think that we should avoid client specialization in the long run because that means that conglomerates form to kind of tribalize um clients and i while it works for a little while i think in the long run that kind of tribalization specialization uh is is probably not healthy for the network rather we should all look at all of the clients and uh seek to balance rather than picking one that is made for us um just a thought uh lion any closing thought before we move on to audience questions yeah load star is production ready load star is production ready go try it now link in the description subscribe nice one on that note you can run multiple clients right not don't join multiple validators at the same time but you can run multiple beacon nodes you don't you don't only have to run one you can have multiple easy backups all those wonderful things don't run multiple beacon nodes validators good morning all right thank you kartik um thank you panelists um let's move uh cardig if you have some audience questions yeah well that would be great one of the best discussions this is getting people excited i think briefly i think they might still be there the prison team is on the chat um also commenting uh this is a question from danny ryan um and we've kind of talked about the problem and i feel like i was gonna get to sort of statements here uh what distribution of the various projects do you think is actually realistic um like what would satisfy each of you uh to say this is sufficiently diverse and uh and like we can agree that it's just not a super majority i think can you ask that two though can you ask can you ask preferred and launchable because it's two different questions but i mean preferred is everyone at twenty percent and launchable is under fifty percent uh and and frankly like prismatic guys are heroes and you know i think you know as well as the guth team i mean i they get it they understand that diversity matters i mean imagine how you're gonna feel if like bugs are inevitable and like you know and any client look at geth which is a very hardened piece of client software over you know five or six seven years now um and i know the network launched but it existed before the network launched uh and you know we still have had you know security updates that could have caused forks at least twice i want to say three or four times in the last year you know if i'm a prismatic developer and i literally am like living and breathing my client and i've done an amazing job i have a lot of people using it i don't want to see my people get staged because frankly like they might turn on me and i don't want that i mean like how would i feel then it would be awful i mean those those guys are heroes and they should be recognized as such and i would hate to see something happen where you know people turn against them secondly i would go 30 30 20 10 and 10. are you reasoning it's good and it's realistic it's not like 20 20 20 20 that's not realistic i actually align with i align with evan um i i believe for launch well i align with them mostly um we are below 66 for prism i believe that uh we we need to safely be below 50 close to 50 percent uh nearing 50 for launch um and below that i i'm not sure it matters so much what the other diversity is but i i do prison we should be we should have the opportunity to celebrate them so much more than we do but instead it is all reserved because we're trying to increase client diversity uh and so it's just it's uh it's muted like i i we should be singing their praises but instead we have to say well there is prism but you really should choose something else so uh congrats to the prism team congrats to all the clients for developing a valuable product that uh supports our network by the way i have a tweet storm about you know we found two uh bugs in go language in the past year in ethereum right so like even the fact that like prism and gas and aragon are all and go is like you know it's a problem i mean stuff could happen like we could talk about crazy scenarios and you know what like i think bitcoiners are gonna be well just everyone like when when we turn off proof of work they are going to be gunning to find every single little thing just like the dao in 2016 i mean like the amount of people that just want to watch the world burn by finding a bug is huge and if they can find something by like fuzzing the heck out of golang and they find something little and they figure out how to exploit it like they will try so you know and they're not wrong use a boomer language like java can never be a bug in jvm of course everybody knows uh i'll kind of blend a couple other questions in um and it's okay if we go a couple minutes over time because uh the next speakers graciously agreed to enjoy this discussion with all of us um what if we try it as incentives instead of other than not just shouting and truth or being scientists maybe that's probably a better way of phrasing that the only person that shouts is me i don't want to say it directly but uh but yeah yeah so there are incentives right so well the incentives are the risk you you run more risk by running the super majority client um and the problem with risk is that people are not sensitized to risk um then they're not considering the everything goes wrong scenario so um it when people make lazy choices they don't see this the only other incentive that works in my experience is a poep so you know i'm people do anything for a poet um so uh yeah if we can find a way to make that work yeah we we have the mechanics for that uh we what we really need is just community agreement that it's the direction we want to go in and i really think when that happens um hopefully in the next few weeks we'll start seeing a sea change of people like ben said people will do anything for poep um it's it's a it's not it's not a bad idea even cheating yeah well and and so we we've thought about the mechanics to prevent cheating we we have we have proposal data and essentially what you would need to do is have a history of proposing on prism and then a future of proposing using a minority client uh that's extremely what i don't believe it's it's cheatable but we'll have to see go ahead carl for seems like we're gonna say something if not uh i may have another question no dig into your next question um so i think carl you actually talked about one of the reasons um kind of the multi-operating system support kind of helps with diversity and we kind of talked about having a windows client is actually essential but you also vouch for individuals instead of institutions kind of having enough power to make this happen so this question comes from jeff and his question is what do you think specifically we need to maybe get more people excited about having all the other clients been supported on that node so they can kind of run that on their own and maybe increase the pool people can easily run their own instances yeah um yeah yeah i mean i think it's kind of sad that that we haven't seen deafnote supporting supporting other people um there's like promises to change that i i think we'll get there uh but uh we're we're we're not there right now and unfortunately that means that a lot of people have defaulted towards that uh windows support is much less of an issue now uh there multiple clients that support windows um but that's that was an issue in the past so i think all these things sort of like have have changed uh but i think people have been rather sticky in the client like once you set up uh they don't want to touch it um i've spoken to a bunch of people who founded a rather stressful experience and a complicated experience i know it took me many hours and like it's it's my full-time job i wrote a bunch of the specs i know what's up uh it took me many hours still to configure my my personal setup so i know i know the effort that goes into these things but i also want to like stress that you already understand like if you've set up any client you already understand a lot of the things that are going on it's it's it's the same core ideas there are still eco nodes and validator clients if everyone uses this this terminology these these setups work like this you're still going to be looking at opening and closing ports um all those things that you you did in the past if you ever want to switch clients you're not starting from scratch it won't be as bad as as what you you had to do before um and and it's rather easy to set up and i think that the other thing which i don't think we touched on yet a little tangential and that's that all the all the clients are profitable you're not making any like there's no measurable difference really in terms of which clients you're picking into uh in into for you for moving forward uh all the clients seem to be like getting prison's not gonna get your high rewards absolutely not and particularly if you calculate the expected value for the prism failure your prism's probably the worst client to pick in terms of your overall value you can have as a staker yeah and you reminded me in this question i'm not going to talk about rewards i've tweeted about that enough um it reminded me of this question it's not just about client diversity and my bad we we should have covered this there's also stuff like if everyone runs on aws this is equally a problem aws goes down we have exactly the same issues everyone runs on one same operating system uh that has some fatal bug then uh that's going to be a problem so we we need to think about diversity in all things right well we'll uh we'll make the final question hopefully optimistic uh 12 months from now what is everybody's prediction on client diversity and what it will look like we can start with evan i think we're gonna do good i think we're gonna do good i'm not optimistic about it quite honestly i mean i think this is an emergency it's the most important pressing problem in ethereum and i'm the only one that's been talking about it for 15 to 18 months and i'm tired of talking about it because i feel like i'm out on my own shouting into the void and like nobody really really you know amplifies my message and because of that like even my friends that are hardcore and ethereum enthusiastic enthusiasts are in my dms telling me that i'm a footer i'm like it's very frustrating for me personally i feel like our message is very muddled and so like if you ask me honestly i'm not very optimistic about it i'm a lot more optimistic um as i say i think we started to see an inflection point where like we we we've seen people in the community start to to to to to to be louder about this uh in contact with some of these institutions who are promising to make changes um and like i if i i think it's sad that evan feels like he's on his own here i i don't think that's entirely true but like i'll happily give you this you've definitely been beating the drum on this for for a very long time um and like strongly agree it is very important for the safety future uh of of ethereum um and like very much one want this to happen and to have more diversity but like i also don't think like ethereum's not gonna blow up um we we we have really robust clients even if it was 100 prism it would still be we still have a very good system so don't want to over fund it but like it is a very important issue to be concerned one of the biggest in the space i evan you are definitely not alone i have been shouting the same thing you've been shouting from a different perspective um and i know that carl is working the same we we come from different angles um i am an encourager and you are uh using a slightly different tactic but we align um i think that in a year every time i yeah i like being a good guy in a year i think that we're going to have easy switching tools and i think people will identify less with a client and more as a staker um and with that goal achieved people will have those interfaces um looking at steakhouse to develop something like this where they can click a button and say yesterday i was prism and today i am um lighthouse and i think then people will identify as as stakers and uh the ecosystem will be much more vibrant i think that's a perfect note to end this um i want to thank carl maia and fizz evan for being an amazing panelist and ben thank you for facilitating this entire discussion this was absolutely incredible and if in the future if there's ever a bailout conversation uh we'll be sure to live stream it on global and uh make sure you're all on the other side of this uh but thanks again everybody and uh really appreciate you coming on and being part of this discussion thanks kartik thanks everybody bye-bye for sure thank you bye-bye thank you [Music] [Applause]