Ethereum Hard Forks - Danno Ferrin

okay moving on in the schedule we've got one of our resident buffalo corns from colorado janelle dano hello dano of course and of course you're wearing your buff corn mask of course and your and your t-shirt i would expect nothing less from you both in everything if there was a way to do this off my eighth denver bean bag i would have figured out a way to do it but i just don't think i can get the same audio quality my my daughter was actually uh asking me the same thing the other day she's like dad how come you're not doing it for my bean bag and i'm like yeah i don't think it would fit in the studio very well so so we're gonna talk about ethereum hard forks right yep steering hard forks cool so obviously a very important topic um your background is an interesting one i actually remember watching you pitch at ethy or wild hack right before you started at consensus yes and joe actually found you backstage and gave you his business card only to realize that they had already hired you yes that's kind of funny oh is it hilarious it was my last hurrah from google was just like i'm gonna do something fun before i go on board and it's like oh yeah i'm starting in september i'm already doing that like and if it was really funny so anyway that was what 2018 yeah 2018 september 2018 quite a while ago and a proud colorado citizen of course yep and i too have been at every east denver i had to uh get you know some of the worst volunteer slots to even get into the first one because it was so crowded i think i did the overnight watches on the floor that we thank you for your uh your unending service to the community and all the great work that you've done so i'm gonna give you the floor take it away cool so uh yeah i'm gonna be talking about ethereum hard forks and this is mostly um from the perspective of what was what has happened not necessarily how they should happen or how we want them to happen but it's gonna reflect on what really does happen so you know we've gone over who i am a bit of pro forma stuff my name's dan o'farrin my twitter handle is at chemnon it's also my github handle it's all over the place i've been using that name for about 20 years that handle i'm a lead protocol engineer like we said on the protocol group at consensus and i work on hyper ledger basu which is one of the four main um ethereum clients that run currently run mainnet ethereum um and here i am this is a picture of me in my in osaka on a little extended vacation i had after devcon because a hurricane blew through they called them typhoons there but enough with this pro forma stuff you're here to find out how ethereum hard forks work i mean how do they work it's kind of a miracle it's a mystery unless you're on the inside you know what when you're on the inside sometimes even it's still a mystery there so the goal here is to kind of open up the black box and show you exactly what's going on with an ethereum hard fork now the boring definition is to say that hard forks are eips that amend the ethereum protocol eip is of course standing for ethereum improvement uh proposal um but let's take a quick little detour on word choices hard fork isn't necessarily the best description of what's really going on in uh an ethereum hard fork even though you know it is correct in that you know we're making a change that if you don't follow it you're not gonna be able to continue with the chain but the purpose of these is to upgrade the quality of the network to improve things for example byzantium they took out checking the world state on every transaction instead checked in every block had a huge performance improvement they also address security certain concerns at times they address user experience concerns of the uh net gas metering that was going on in constantinople in istanbul that was put in to upgrade the experience of the network so we're not just forking it just to you know create some some place where you have to make decisions what you want to do these changes have a purpose they're network upgrades but i don't know it's kind of a maybe we could come up with a better term on the network upgrades because that's kind of a mouthful so i think we should go with sporks instead of ethereum hard forks we're going to be doing sporks now got my nice iridescent rainbow spork from youth denver so as i said sporks or eeps that amend the ethereum protocol so let's talk about eeps what is an eip ethereum improvement proposal so these are formal definitions of things that the core developers and other people who are interested in improving ethereum right up to describe what's going on and there's a whole lot of different kinds of eeps that are out there they're in three major categories there are standards tracks eeps that are very technical and have a very deterministic conclusion as to what they mean there are informational eeps that um provide information about stuff going on and then there's meta eaves to kind of collect these eaves and in the standards track each there are about four different kinds there that are very useful this talks mostly talking about the first group the core e core eaps are eeps that change the consensus protocol of ethereum when those talk to each other and they say hey here's a transaction i'm going to process the transaction this is what happened all those rules matter at the consensus level in their core these are the most critical heaps to go through and get the most scrutiny and have the most formal definition of what's going on a couple of big examples of good examples of what a quarry would do would be 2200 definition of net gas metering this is amending a couple of other eeps and tweaking some of the details and referencing them to give some very specific definitions of when you use a storage operation how much gas do you charge how much gas do you give back here's the definitions of the test it's very technical um a smaller example of a core ink would be e1344 the chain id op code which added an opcode to the ethereum vm and said what chain id am i on kind of a simply but still needs very specific technical implementation so that's a quarry the next level would be network eeps and this is the level where the the the ethereum clients talk to each other there's a little protocol called devp2p and there's some things going on that aren't critical to consensus but are still necessary for a good well-functioning client to implement so we can talk with each other a couple of examples are eep 706 dev p2p snappy compression saying that hey we should compress all the data that we send to each other and it says you got to set this field here and then use this compression library and it spells it out another example is fork identifier for chain compatibility checks eip 2124 this allows the two nodes when they're talking to each other and say hey i'm running um roxton hey i'm running gourley so before they go deep into connection again say you know what we don't even speak the same language so they won't talk to each other so that one there helps keep the networks flowing smoothly the next layer of eeps is where we start getting into things that affect a broader audience and these are the interface heaps these are the interfaces that a node should expose to things like dats that are using your ethereum clients uh one notable example is 1767 is a graphql interface for ethereum everyone's familiar with json rpc there's a need for that there's also an eip that describes how you can use graphql to conduct all the same queries and use graphql engines to get that data a more recent example is eip-3155 eip trace specification defines a few json rpc methods that go in to the uh to talk about the trace and how things work um the fourth category of standards is ercs and these are the ones that most people know about the classic the big one being erc20 is an eip eip20 another more recent example of an erc is eip-3156 flash loans people are trying to standardize on-chain how flashlights work and the ercs tend to happen quote unquote on chain and within the dapps and the interfaces happen between the dapps and the clients it's probably the separation so the next two larger categories are informational and medievs in informational eeps there's actually a heap telling us that the name of the theory mainnet is mainnet and there's an eat giving an overview of serenity phase zero but not the specification and the last category is the one that directly impacts the network upgrades or hard forks or sporks uh meta eips so meta eip 1716 is the sport meta for petersburg and 2070 is the sport meta for berlin and these are the eeps that say well here are all the other eeps that make up our hard fork or network upgrade or spork that's funny so a quick simplified overview of how we get an e in a production um simplified overview this also though begs a very important question that i'm going to talk about for only three slides governance it's the elephant in the room it's a lot of things that go on in ethereum and one of the things about ethereum governments is the governance itself is decentralized if you could point at a single process in a single person or a single company and they have the final say on eaps then ethereum is then centralized people complain that if we had on chain governance the governance would be centralized on chain so one of the the core things going on with with the way the eeps work is it's all decentralized we eventually come to consensus we come to a conclusion in a decentralized fashion there is no real centralized place where these happen a couple of beeps worth talking about about governance that we need to talk about when we talk about these hard forks there are two eeps that i consider contentious um one of them that actually was implemented and one of them that's being discussed for future that are just contentious eip 779 was the dow it was very contentious it resulted in an actual fork between ethereum and ethereum classic that is the worst case scenario for a contentious split the other scenario is that people accept and they work with one side which is what we're hoping is going to happen in 1559 um it's either going to go in or it's going to go out and the in the community is going to go on one side you know once their once your concerns are voiced i personally favor it but it's not something i'm willing to fork the network over i think we should get it there i think it's valuable and that leads us to the next category of vetoed or rejected eips that were proposed these were ones that wound up being rejected by the community despite being technically sound um eip 999 in 867 were two eeps that talked about locked ether recovery and specific the parity contract um those were rejected by the community um eip1057 praguepao though technically sound in obtaining the technical goals that it stated that wanted to get in the e that too was rejected by the community and was not implemented in a hard fork and these governance issues came up decentralized all across the community there was no one spot that decided and no one spot that came to the conclusion it happened in a decentralized consensus messy manner it's kind of messy it's a lot of work to look at so you know getting some of these in and out is a lot of proof of work and that's all i have to say about governance for this talk so where were we back into the revised and simplified workflow so this is the formal process that we're going to go for go through to get the eips but there's actually i think this is the wrong way to look at it because this is the technical things that check the boxes but you need to think of it like a bill like schoolhouse rock that there's a long process that you can kind of sing over some of the things because you look at the higher level things of the higher level processes that go with it you when we look at just at the key um landmarks that happen along the way and you know this isn't really a pbs until you get that little eni bug down there on the lower right anything i say from this point is educational and informative i am not practicing core development from this point this is for your information and entertainment so you can think of getting an eep as a multi-step process it starts with an idea you write eats you do a lot of tests and finally you roll it out and these are about 12 of the steps that you would go through some steps are skipped some steps are done out of order some steps will overlap but you could also think of it in a perspective of the scope of what you're working on when you have the idea it's just the developer and then when you do your implementation and you grow to your client team that you're going to work with to get the first implementation out and then you work with multiple client teams the four main teams that form all core devs to figure out is this a good idea and what things need to change and how do we do it finally roll it out to a test network and if that all goes well then it can roll it out to production and fuller test networks it's essentially four different tiers of what goes on you have the idea you get buy-in you validate that it's going to work and then you deploy it into a hard fork so the first here is ideation this is pretty simple come up with the idea write a draft maybe prototype the implementation this is the foundation of what all the other eeps come from a lot of things die in the idea stage because as you think it through you realize it's not such a great idea the second stage is where the real work starts where you get the buy-in this is where an eep author might do things they might write blog posts they might start up a discussion on ethereum magicians to get people to accept it they might discuss it in a github pr they might make a pr2 client and have the back and forth there um usually one of the steps you might do a video call or breakout room and the final place that a lot of people try and get buy-in is in the all-core devs call which happens about every other week um this was mentioned about you know about a 45 minutes ago in east denver they went through the process of that but the all-court devs call there's a lot that should happen before it need gets to that point and the eve should be really well along the way of being prepared for a hard fork by the time we get to the alcor devs call now this is where the real work happens and a lot of stuff gets filtered out there you know this is kind of an inverse funnel more of a pyramid that's just the way the word art worked so the third step is validation that's where things get really interesting um you have buy-in from from multiple client teams that say this is a good idea let's try and build it and this is where a lot more of the formal process that can be repeatable and seen happens this is where it starts coalescing around what's really going to wind up in there so you get multiple clients to implement the eep you write reference tests you might start up in a femoral test net and you might run something that has been talked about we'll probably see sometime this year next called a shadow chain network so all apes that go into a hard fork should have a set of reference tests and if you've looked at these it's kind of like looking at the matrix there's probably about 50 people or less that really understand what's going on in those files but it's json-based so it's not too hard to learn if you know the concepts behind what it's testing and it's really the last two groups in that first section that are what are in these reference tests the transaction execution tests and the block execution tests where most of the eags happen and these tests cover multiple sporks um so they have all the test data clear back to the frontier test net to validate that you understand and implement all of the different forks that go on in this they're put in different places they're called legacy tests but if you're going to be able to write a client you need to be able to run all of these tests and almost all time all tests all clients do these reference tests as part of their daily build process um takes you know anywhere from five to ten to fifteen minutes to run um on a good build machine and it's good validation the next step is ephemeral test nets and that's something that started up mostly after constantinople and has gotten a lot of use in berlin these are test nets that we stand up that are um basically they're not meant to live forever and they're intending to try and have clients break these test nets into all sorts of interesting things that you wouldn't want to do in a production test net because we want to break it in something that we're never going to use again before we break into something that's going to stand up long term um one that existed the first one was constantinople sturby because we had some existing issues with the way that the um the net gas metering is working in yolo we've been doing yolo this time with berlin we've actually done it three times which is kind of the wrong use of yolo you only live once worth living three times on this but each of the iterations have had different sets of eeps and finally in the third version of yolo we've settled to say these are the eaves that we're going to put in berlin so we've had our femoral test nets we've had clients hash out issues on these between interpretation of what the specs say and figure out which client is right and which client is wrong and luckily every client has had its turn being right and being wrong so it's really serving its purpose now the more exciting thing that's going to be showing up here shortly is something called a shadow chain that's been discussed on calls i think we're going to see it at the end of the year for the binary trees and possibly for 15.59 and what you do with the shadow chain is you take mainnet and you formulate it in kind of a way that you can run it as a test net you take all the transactions from the mainnet flow you but you set up the things you're changing to use the um to use the new things you're changing the eep the big example here is binary trees the way we calculate the state root is going to be changing dramatically we want to make sure this is stable and runs at scale so we're probably going to set up a shadow chain that the only difference is going to be how we're calculating the state root in these headers and there's some details it's going to invalidate the proof of work we're going to get new proof of work on it or do a proof of authority and that's where the work in the shadow chain comes and why it's not really used in a production network all the input comes from mainnet itself and we just reformulate it kind of a shadow of what it is so once we've done these ephemeral tests now comes time for the real deployment i put the test nets in the deployments here because once stuff goes on a test net uh production test net typically it doesn't come off robson actually has the old constant noble definition on there even though it's been changed to petersburg uh ricoby as well gourley i think uh avoided that because it started with the petersburg um and then you know these four steps basically these five steps basically happened in order you pick blocks for the test nets you watch the test nets activate and then when you're happy with that you pick a main net block number and you watch the mainnet block activate so these are some long-running test nets there's actually five that are probably worth talking about morden was the original test net um that kind of was a casualty of the dow fork testing so instead they started up robson and rink being girly because of you know the chain id kind of ruined the way that morgan was working with his transactions so robson wrinkly and gourley are three of the ones that all clients currently can access coven is still up and running open ethereum runs that and they'll update their blocks when they get it so before we do any of the mainnet production we activate the test nets on these four test networks um and we figure out what's going on with them robson's the best one because that does actual proof of work client mining rink b um won't get block production issues correct unless gets the only one that's correct on it because i only get this mining there gourley is a particularly good one because it has all the clients doing uh click proof of work on it and coven is aura only so right now it's only open ethereum is mining and uh yeah so the one last thing that needs to be mentioned is forks at any time can be stopped and i'm getting close to time luckily this is the next to last slide and it has been done well one famous example is constant ants and opal less than two days from activation a security exploit was discovered and the develop the core developers all gone on call talked about it for an hour we went through the five stages of grief of denial and anger and depression and finally acceptance that we needed to cancel this hard fork within 24 hours and we were actually able to pull it off inferior was able to pull it off all the miners were able to get off constantinople and it was canceled we we huddled and a few months later we shipped petersburg so you know just just because something has advanced to a certain point does not mean it's inevitable things can change so congratulations what's your reward for successful hard fork you gotta do it again in one of 16 months these happen all the time this network upgrade process is key to the lifeblood of ethereum it's not a static thing that never changes now the nature of the changes as we get a bigger thing scope down to become require more and more uh weight to be able to change we're not just going to you know change all the op codes and move them up by one because that sounds like a cool idea it needs to impact security it needs to provide significant user experience and right now those are the things that are going to be the user experience is going to be things that enable us to go to um go to the the proof of a proof of stake network and those are the things that are really going to get the the prep the the support of the all-core devs to to change the ethereum network security will always change things to fix security and these numbers 1 to 16 months these are actual historical numbers there have been there's been one month between hard forks and there's been 16 months between hard forks and a lot of space in between so thank you for your time um i don't think i have time for questions but you can ping me on twitter at chemnon or uh any other place that you can get a hold of me thank you cool thanks dana so uh jonathan can you change his his streaming on the bottom here to ethereum sports really quickly while we do this in real real time dream sports i mean if we're going to make the transition we might as well just make the transition right go for it i love the fork i mean if you're eating stew and you want to sip the stew and pork poke at the the vegetables the same amazing which instruments it's it's why the the sport marmot uses it because it's so versatile ethereum sporks there we go we officially made the transition so in fact i don't know if you know the story about it but that's actually why we we came up with this fork as an idea because we didn't really think that it was a good representation of the ethereum community these contentious hard forks that happened in other places like bitcoin and we just uh felt like a spork was a better description for a really a soft fork which is really kind of a consensus based one anyway uh great presentation thank you so much and great to see you alright thanks