The State of Ethereum State | Danno Ferrin - EOF

hi welcome back today we have Dano faren from E please take it away cool uh yep happy to be here happy to uh deliver the uh state of ethereum State address um I hope you like talking about State because I'm going to be stating the state I'm going to be restating the state and I'm going to tell you about the estate of the state the state is is uh the probably one of the biggest talking points and concerns about ethereum over the past few years um but you know what is this concern with the state um first a little bit about myself though I'm from team ipsilon which is one of the teams at the ethereum foundation I just started there in January I've been a longtime contributor to ethereum I have been to every single eth Denver including the one they did online in the pandemic so I love this place I think this place is great so first we need to talk about what the state is what is the state that we're worried about in ethereum and it turns out it's um it's the data we use and a lot of data about the data we use and that's really where the problem comes in is a lot of the data about the data and a lot of the history so we're going to talk about the chain data and how we bring that data onto your device now this I popping chart yes it's going up to the right this is probably what we want but this is not what we want going up into the right and that is the size of how much data you need to store an archive version of an ethereum node 21 terabytes tracks uh the e the tracks the total size of everything related to the ethereum node in the past nearly 10 years that it's been running um nobody has very few people have 10 to 21 terabytes on that ethereum node this is tracking literally every piece of metadata about it that could possibly be calculated and needed now this is a probably more realistic chart this chart is showing what a full ethereum node is tracking um you'll notice that the black line on the bottom is parody parody stopped working at the time of the merge it's a fairly flat growth and the blue line is gu gu had some pruning issues with some of the data that it was keeping that it didn't but eventually last year it figured it out and now it's a nice slow growth line up on to the right you see right now it's about 1.25 ter gigabytes of data still a lot of data 1.2 terabytes actually uh terabytes of data terabytes was the right word um 100 a th000 gigabytes and that's that's not necessarily a good thing so why do we need all this data what is all this data and what is it all about so if you think about the things that you use in ethereum your account your balance a particular State value that you need think of those as a number with a path to a location of where it stays I am this number in this account and I represent 25 and I live here these things we call leaves and they live at the bottom of a tree so we take all these leaves we arrange them in tree you put them in the bottom of the tree and we can't just transfer however many gigabytes of data say this is the current state we need to reduce it to a smaller number so we do something called a Merkel tree we hash this value together as we go up in levels so the first level will hash the lowest level of trees the lowest level of Leaves we get a branch value that represents that there's few of those we go up the next level there might be a leaf in addition to some of those branches we wind up with fewer values and we keep going up we keep H hashing these BR branches together using a hash function until finally we get one number that represents the state of the ethereum network that is a number with 256 Fs in it it's got about I don't know a whole bunch of zeros it's a really large number and it's unlikely that you're going to find the same number twice which is what makes it a good number and they use a very secure hash number that when you do the math it's going to be really hard to reproduce with anything that is even remotely the same so this is what we do when we when we calculate the hash and we pass the block headers around we reduce it to one single number so how can one single number results in so much data it's because when we change the data we also need to change all the branches above it so in this case if we change two leaves values we then need to recalculate all the branches above it and the root node which isn't that bad if you do it once and if you're a full node you just keep what you need that's relatively current but if you're an archive node you need to copy that data to the side and those some same two values change again you have to store it again and again and again and again and again so you got this big giant pile of tree branches of every time in the history that this value has ever changed in all the metadata around what the actual root node of it is so archive nodes track every single one of these values throughout the entire history of ethereum whereas a full node just keeps a relatively short list of maybe the past few days so this is a lot of data and the next question is well how do we synchronize this data there's principally four methods the first one isn't really used that much it's where each of the client goes around and they take that root node that value and they ask all the other clients hey tell me the children and then they get that Branch value and they go hey tell me the children of that they start at the bottom of the tree and they go all the way down to the bottom of the tree so they go from top to bottom sinking it and this works great in small numbers but when you get to a real full production size main net experience this can take days to calculate and the problem is that clients like to clean up and keep things nice and tidy so that root node that you thought you had around there's going to be less and less clients that are going to be able to share that information so the problem is when you do this fasting and it was faster than some of the alternatives you're eventually going to run out of references to that root node and no one's going to be able to tell you what your trees what your tree leaves are so a number of years ago gu came up with a better idea called Fast sync and and Basu and nether mine also implemented it and the difference is you start with the leaves at the bottom you just say hey just tell me the leaves are give me the first 10 okay give me the next 10 and you keep moving left to right across the tree and as time progresses um you just say give me the current value of that so as you go left or right you don't have the leaves at the same point in time you have them over an increasing point in time but eventually have all of them and once you think you have all of them you add them up you do the hash you get a root value and spoiler alert it's going to be wrong but that's okay because that's when you start the next step which is the healing step you basically do a fasting from this top and you start asking for that number and you ask for the branches below and below and eventually you're going to find an entire series of branches that you already have it's going to save you a bunch of time you don't syn that surprisingly small amounts of the database actually change over the number of hours it takes to sync it so this goes relatively quick another important step while fast sync and full sync are going on is while you're downloading these Branch datas you're also downloading the entire block header in the block history of ethereum the list of all the transactions that anyone has ever executed is being downloaded in parallel to this tree data that's very important to realize the tree data and the transaction list are two separate items we'll come up to a chart on that later now Tre DBS are not the only way to store this data Aragon has its own style of database that that ref is duplicating um so they can't do this fast sync instead they got to do a different kind of synchronization um which is basically got to replay from the beginning and that can take a long time if you calculate all the branches of every single block so rean Aragon optimize it and they don't calculate those values they just download the values run everything to the end and at that point they calculate the root node and if you do everything right the number's going to be just fine because it's tracking to making sure that you've been doing everything correctly um and again at the same time that it's running the this it's got to First download all the blocks and it it it tends to the block download tends to go faster than the actual execution there's one way that nobody really does anymore and it's basically a variant of a database copy par used to do this they take their current database they dump it out to a file and then everyone would download that file and extract that out in their clients it worked great until the blockchain got too big and it got to the point where the amount of time it took to export this data was longer than the amount of time they had to export it before they had a reced again and get started again so so that sort of a database snapshot technique really isn't used before so once you run all this data once you synchronize all this data what are we looking at in the blockchain well if you have an archive node the archive Branch data takes about 20 times as much data as the rest of the chain that little slim Circle in the middle that's the current full node data and all that green circle around it that's an archive node so you look down deeper and what you really get is you get quite a different story the actual data that represents what you're calculating this the account State the storage state in your code takes up only 5% of the data 15% of the data is that metadata of the tree that holds it up so 80% of the data 80% of the data is literally just the list of all the transactions that ethereum has had for the entire history of ethereum um if you're following the rules if you're not following the rules you can chop it off but the rules right now say that you're supposed to download every single transaction for the whole history of ethereum and that's why the blockchain keeps growing that's why people freak out when you say we increase the gas limit we're not going to be able to handle the data the data is not the data onchain that you're using at the moment the data is the entire historical Ledger of ethereum that we have to download if we increase the block size now why would we want to increase the block size and what are some of the things that have been done to deal with this so in that we start talking about layer twos and blobs cuz a lot of the use in ethereum in the past couple years has really exploded on layer twos and layer twos have a vicious appetite for transaction history here is a graph of the number of user operations since the dawn of l2s for a long time there was less transactions that were on Main net and then there was a bit where it crossed over and then over the past two years it's just absolutely exploded and it's just gone Bonkers now one thing to pay attention to that little last little green diamond there on the right uh that is when denune activated and where the most important thing to happen until the L2 ecosystem happened blobs were activated and you can see there's an immediate like tripling of the amount of user transactions that were done in that time because everything became that much cheaper and everyone were just excited to start using it again and it fell off but then it's come back with a vengeance you can see that it is uh 16 times as many transactions happened on Layer Two then happened on layer one this in some ways is the absolute fulfillment of the rollup Centric road map we're using doing more things not on the main chain but on on layer 2 Chains what's interesting is when you look at that growth the gas limit for eum has really remained static for the past 2 years on average there is 15 million gas executed per block it can go up and down there's some floating fee Mark of stuff that goes on with it but on average it's going to be 15 million per block and for over two years since the merge we haven't touched it till about a month ago and last month we updated it to be about 18 million gas per block that little that little uh blue bar at the very end of that is the same so the gas has been flat but you want to talk about the gas let's look at the size of the average block over that time tells an entirely different story two years ago it started out a reasonal level and it kept going up up and up the reason is because layer 2s had to store all of their layer 2 transaction data on the layer one blockchain as call data they had to put all their data in the transaction to be a real Layer Two to have the layer one track all of their data so you can see that that the values went up up and up and it peaked at about you know a quarter megabyte of data in January and then you'll notice that they're about March it absolutely went down down like a rocket slide and went back down to more reasonable levels that we' seen in the past and this is when Canon activated when all the layer 2os stopped posting to layer one and started posting to blobs blobs are different kind of data they're just random data that is associated with a particular transaction that we don't have to keep around as part of the main history blobs have to be kept for 14 days on the consensus client and then the consensus client can keep keep them or dispose of them as they see fit they're only responsible for retrieving them for 14 days and when we target three blobs per block on average what that means is you need to take about 36 gigabyt of data on your consensus client to store these blobs so that means we're transacting in nearly a terabyte of data and blob data now that's for Cancun in about a month or two we're going to fork pectra and we're going to double that number to Six blobs per block is what our Target's going to be what that means is that you need to keep 73 GB of data on on your hard drive and it's going to be nearly 2 terabytes of data per year of blob data that's going to be processed but remember your consensus client only has to keep that 73 gabt of data we don't have to keep the entire history of every blob that's the layer 2's job to do that and we put markers and hashes in the blockchain so they can prove that their blob did exist at that point some really fancy kzg Moon math now in fusaka ideally we're going to launch it this year in the Q3 there's a couple of ideas of what we're going to do one idea the very simple solution that we know we can achieve is just to go up another three blocks Target nine blocks that means we can have about 120 gabes of data takes about 3 terabytes of data the more exciting answer though is something called pieraz and if we do pieraz we can go for a 6X 8X number um 48 blobs now in some of this math you'll notice it looks like I'm doing it for 96 blobs but that's because they do some eraser data um some some magical Moon math to make sure that it works right so that you only have to store about 200 GB of data on your block even though we've gone 8X on it it only doubles and so this provides an average throughput of 30 terabytes per year of blob data that can be stored on the layer 2 blockchain so what is PS that's a 20-minute talk in and of itself but I think this picture really gives you an idea of what it is each client only keeps a subset of the blocks and the subset that they're responsible for keeping they have a little sub network of other peers that they talk to that can get the data and share amongst themselves um so that you know who you're sharing it with and you can make sure that each other is keeping you can challenge it and you can get different pieces for it and they use you know Advanced eraser encoder mechanisms now one of the things to not that this is not the final evolution of blobs on ethereum this is uh step two the Third Step full dank sharding these columns they become simple squares so it's two dimensional and the number that I've heard thrown around I don't know if they're going to be able to achieve this but this is the goal they're going for 200 56 blobs per block that's a lot of data I didn't even do the math for that I mean I might need more than terabytes to track that so what this comes down to on a consensus client is you only need right now 95 gigabyt of data to run a consensus client on your node um the blobs cycle through every 14 days the finalized slots not quite as rigorous as the 14 days but they don't have to keep all of it so the size of the fin L block stays pretty constant it's not going to grow like it does with a block history so what they've done in in consensus is they've got a very consistent Target that this is what you're going to have to keep it's going to be fairly static it's fairly predictable we know what it's going to be it's not going to grow without bound because we can prune the data so so consensus layer really has its act together what can the execution layer learn from what the consensus layer is done and that's going to come up in EIP 4444 now blobs had its own number 4844 and I guess they like fours so much that they went with 4444 um four fours um to Tru emphasize this next it sounds complex but it's a really simple solution to what the execution clients are going to do to maintain their state they're just not going to keep it these two bullet points is basically what the entire specification of EIP 444 is 4444 is if if the data is more than a year old you should not serve it and if you're not serving it you may prune it if you're not going to tell anyone about a transaction back in 2019 you don't have to keep it around uh so so whenever you do a peer-to-peer query if you ask for something older than a year no one's going to give it to you if you ask for um a a a a a block a RPC for something that is you know back in 2022 the RPC is not going to do that for you now you can if you want it's only a should if you deliberately do it you can do it but no one can make you and no one can punish you and no one can Blacklist you as a node if you refuse to serve stuff older than a year that's what 444 allows and what happens if you take all that more than a year old data and you prove R it away you throw away just basically a terabyte of data that you no longer need to keep on your client it takes the amount of data that you need to do and it cuts it by over 2/3 so what used to take 1 and a half terabytes now takes a half a terabyte and what's amazing is if we don't do anything with the block Gast which we will but if we kept block Gast to the same level those numbers would stve relatively con relatively consistent the only real growth we would get is in new accounts New Balances new tokens and those grow at a much smaller rate than just the raw transactional history so it still has a possibility to grow without bound if everyone uses the blockchain without bound and we have to keep all the data there's a plan for State expiry that probably going to have to deal with in a couple of years but as we can see it's still the smaller portion the tree branches to track that data is bigger and the amount of your blockchain history is um it's not even as big as the rest of the data now it's a really great solution it's really subtle and it has powerful impacts but what about all that data does it just go away does it go in the bit bucket how do we get the entire history of transaction of of of ethereum it's kind of important stuff oh before we get to that though when are we going to pull the pin on this we're going to pull the pin on this feature not as part of a fork because it doesn't require a hard Fork we're just going to do it on on an agreed upon day and the day that the client's agreed on in the execution layer is May 1st mayday mayday mayday we're going to get rid of 2third of the data and it's going to be awesome unless you need that data so you need to download and sink it quick actually we we got a better solution to that another thing that deserves its own 20-minute talk is something Piper maran's been working on called the portal Network the portal network is going to keep all of this history for you in a separate peer-to-peer network with separate cryp cryptographic incentives and it's going to make sure that this data is still available and it stores more than just the block history it's going to store the the state history it's going to broadcast transactions and it's designed actually to support light client execution real light client execution the sort of stuff um that that we are looking for for for Witnesses except you're not going to need Witnesses so if you access the portal Network you can actually execute some of your transactions without having to download an entire blockchain it's a whole raft of clients and everything that you can look up if you go to their site eth portal.net but like I said it's an entire 20-minute talk in its own right so I'm here to tell you that the State of the ethereum State is strong it's been a very powerful changing force in the world and it's not going to hold us back we're still going to be able to remember all that we had we're going to be able to maintain and take care of the state that we need in the future and be able to grow responsibly with these changes that we're doing inp 444 and and 4844 with blobs and with restricting the size of the of the blockchain we can increase block space we can increase gas limits and we can make sure that our clients will not run out of dis space unpredictably we'll be able to make sure that everyone can serve the state can can can can make use of the state and can and can um prosper from what the state provides I mean the ethereum state not the government state they're kind of the opposite level on that but again the state is good thank you Deno appreciate your talk it's very informative excellent okay so we're taking a short break thank you guys