CL Data Structures | Michael Sproul

Thank you so much for joining. Awesome. Yeah, very well. Um, yeah, thank you so much for joining everyone. Uh, and especially Michael, it's really great to have you here.

Uh it's um um it's amazing to have somebody who is um expert on the topic because today we are covering the Kosenus layer uh data structures uh how the beacon node handles uh all kinds of data and uh yeah it's really great to have Michael here because he's from Lighthouse and he's been working on the uh three-stage database for Lighthouse. So he knows the in and outs of the um uh of the data structures in in um beacon and the SS and so on. So yeah, thank you so much for being here and um yeah uh the stage is yours. Uh please uh uh share the presentation and go ahead. Yeah, thank you Mario.

Yeah. Um it's a pleasure to be here. Um yeah, I'll um get started right away. Here we go. Um, I don't want that.

Oops. What did I do? Okay. No, that's good. Um, yeah.

Okay. So, today we're talking about um consensus layer data structures. Actually, can I make this full screen? I feel like there we go. That's better.

Um, yeah. So, consensus data structures. Um, as Mario said, I'm from Sigma Prime and I work on the Lighthouse client. Um, I've been working on Lighthouse for about 6 years now, which feels kind of crazy to say out loud, but um, yeah, it's been a while and I've worked on lots of different things in Lighthouse. Um, but lots of the consensus code and the database stuff um, has been yeah, stuff that I've sort of worked on um, or come up with ideas and then working on with other people.

So, um, yeah, we're going to go into some of that today. I'm going to cover sort of three main topics. Uh we're going to talk about how beacon state gets rec uh represented in memory. Um and I've got a quick overview of what beacon state is um just as a refresher. Um and then we're going to also talk about how beacon states are stored on disk.

And that's the tree states um work that Mario alluded to. And then we've also got um a little short um topic on how Lighthouse specifically stores blocks and blobs um mainly on disk. Um yeah, so let's start with the in-memory stuff. So the beacon state, just to remind you, um is the state that's specific to the consensus layer, and it just includes things um that the consensus layer needs. So it's all of the new stuff that was added in Ethereum 2.

0 know um or added during the merge um and and none of the stuff like the smart contract state um or the account balances or anything like this and as a result it means that it's relatively compact. So on um Ethereum mainet we have maybe I think it's probably 1.8 million um validators now and the size of the beacon state is pretty much just proportional to those 1.8 million validators. So each each validator gets maybeund and something bytes in this state where they have their public key, their withdrawal credentials, their activation epoch, their current balance, um things like this and then you just have 1.

8 million of them. So it's it's like yeah it's around yeah it's around like you know the order of hundreds of megabytes and not like hundreds of gigabytes like the um like the execution state. Um so specifically I checked yesterday and the mainet state um at the head is 250 megabytes and that's when it's serialized as SSZ which is SS is this encoding format that we use in the consensus layer. Um it's basically just it's a binary format where you can um represent um any of the data that is in the consensus layer in that format. So you just encode it and you get a string of bytes and that string of bytes is 250 megabytes for the um for the for the current mainet state.

Um so when it comes to storing the state, we want to think about what sort of things we want to do with that state in memory. Um and there's kind of three main ones that affect our um our choice of a state representation in memory. Um the main one that we do is block processing. So this is the the act of um getting a new block from the network maybe via gossip or RPC um and then having to process that block by loading its parent block and the and the state for the parent block and then processing the block on top of that state to get a new state. So it's sort of like you have the old state, you get a block, you get the new state and that's this that's the consensus um state transition.

So that's an important thing that we have to do. Um hashing the state is also something that we have to do. So um Ethereum uses this system where we um we hash the the whole state as a um as a binary tree in the case of the consensus layer or as a Patricia Merkel tree in the case of the um execution layer. But in the consensus layer world, it's all binary Merkel trees and um we have to just compute the state route. So we get one singular hash256 value uh which represents the hash of the whole beacon state.

So we'd start with the state before block processing. We'd apply the block, we get a new state, and then we compute the um state route for that post state and make sure that it matches the state route that was included in the block. because if it doesn't match, then that means we've probably got a consensus inconsistency with the proposer of the block. And that does happen sometimes when um when there's a bug. Uh but hopefully not too often.

Um and then the last thing that we want to do with states that's kind of significant is reading and writing them from disk. Um and this kind of ties into the second topic that we'll cover which is how we represent state on disk. Um but our in-memory representation ideally sort of has to be compatible with that or it has to not um mess us up too much. And in the case of Lighthouse, we actually we reading and writing from disk is actually still really slow for our state representation that we use. But I'll go into why that's okay in a little while.

Um but yeah, basically we want to choose a representation um that balances these concerns. Um, and we want to keep memory usage low because we don't want to just throw all this data around in memory and then suddenly the bacon node is using 25 GB of RAM, which is still happens sometimes unfortunately, but usually it's um fixable. So yeah, that's the that's the idea. Um, I'm going to go into a bit of detail about block processing specifically and and how this motivated our choice of um state representation. So um most of the time on the chain you've got a very orderly system where we've got the rectangle here is uh B1 that's a block and after applying B1 the state of the system is sigma 1.

This sigma is the same sigma that sigma prime is named after the state um which is a nice cute coincidence. Um and then we have a new block B2. So this is what I was talking about where we we have the existing state of the system. and we have a new block and we want to apply that block using the state transition to get a new state. So most of the time that's exactly what happens.

Um maybe we had sigma 1 in memory. So green is in memory. Um we apply the block and then we have the new sigma 2 and we just we take the exist existing state we mutate it when we apply beta uh block two and then we get a new state sigma 2 which does mean that sigma 1 uh is no longer held in memory because it is uh it has been mutated into sigma 2 right so this is a this is how lighthouse used to work before 2024 um yeah I think that's Uh so the problem with this is what happens if oops you get a reorg or an attempted reorg where suddenly there's a new block that doesn't know about block two um and it wants to build on the state from block one. So it wants to build on sigma 1. U this is this would be a cage miss.

Um so you're you have sigma 2 in memory but you don't have sigma 1. So then you have to rebuild sigma 1 and this is maybe going to be slow um because you might have to load some data from disk. you might have to redo some block processing um to get to that point. And at at this point, this was a major source of um performance headaches for us in Lighthouse because it would be we were trying to come up with these heristics about well maybe if it's you know if this block is late then we then we copy this state rather than moving it. But then the copying itself was slow.

So we're sort of paying a performance penalty all the time to um to account for this scenario that could happen. Um and you don't know that this is going to happen until after you've already processed B2. So you don't know whether you needed to keep signal one until this happens. And this happens quite a lot on mainet. Um, I mean, not not too much, but it it probably happens every day, multiple times every day, because all it takes is one proposer to say propose their block B2 late and then someone else is going to build B3 to to orphan B2 because it was built late and no one saw it and no one voted for it.

So, it's kind of a lowquality um block. So, this this chain, it's not bad that we reorg this out. It's probably actually good that we reorg this out. Um but the beacon node needs to be able to handle these reorgs and um needs to be able to quickly process this block so that it's not then late or processed late because of the cage miss. Um so what we want to do is we kind of want to have our cake and eat it too.

We want to have access to lots and lots of bacon states or at least you know a handful more than one definitely. Um, but we also want to keep uh memory usage low. So, we don't want to just say, "Oh, well, each beacon state is 250 megabytes, so I'll just keep 100 of them." Because that's going to then use what? 100 times 250 is like um 25 GB of RAM, right?

So, we don't want to you we don't want to just do the naive thing of um of like having a million beacon states in memory. Um so we thought about well what if we used something a bit more clever to store them. So there's a there's a paradigm called copy on write uh which is where whenever you change a data structure you copy um all of the parts of the data structure that you're changing change those um copies and then you still have the original data structure fully intact. Um, and this is very popular in functional programming languages where this concept is called persistent data structures. They're also called immutable data structures or things like this.

So, but the idea is that it fits very well with the blockchain paradigm because you have the old thing. It's immutable. It's got a state route, you know, the old state, it's completely um like sort of sealed and then when you when you apply a block, you get a new state and that new state doesn't go and mutate any of the old state. It just makes new um new parts that reference the old state. So, um this is sort of what it looks like.

The main data structure we use in the consensus layer um that's persistent is this um it's it's like a it's a vector with um leaves that are the entries of the vector. So v1 v 0 v1 v2 etc along here but it's a vector represented as a tree um because we want to be able to hash this is a binary merkel tree even we want to be able to hash these leaves together to get the internal hash and then hash all the way up to get the root. So uh if this if this list were part of the beacon state um we would have all of the leaves which would be the values. So these could be like the list of validators um as the leaves and then the internal nodes are going to be hashes of those pairs of of um children nodes until we get to the root. And then on the beacon state as a whole we're going to have a whole bunch of these um lists.

So we'll have the list of validators, the list of balances, the list of something else. And then we'll put those in a binary Merkel tree and hash them up to get the state route. Um so yeah this is this is the main data structure we use and I've used a sort of hasll like syntax here but um that's just to illustrate really that you know a tree is either a leaf which has the hash of the leaf and the value or it's a node which has the hash of the node and two children which are both trees or it's this is a kind of special case um it's a zero node. So for the parts of the tree that we haven't added any leaves in yet, we don't actually have to um put anything there. We can just know that okay, there's a there's basically a an empty um space at at this particular height and we can we know we can premputee basically the hash of this um subree and just know that there's a we know the hash of that when we're calculating the hash of its its parent.

So these zero nodes are just sort of like um error an optimization that just just fills in um the the unexpanded parts of the tree. Okay. Um that Yeah. Does anyone have any questions at this stage? Because I feel like the first time you see this it's probably a little bit um confusing.

Any questions? Feel free to raise your hand or write in the comments. Oh, I don't think I can actually see the comments, but if there is one. Yeah, let me know. Okay, cool.

Um, I guess I'll I'll keep going. Uh, but we can also we can I can answer some questions at the end. So, this is the idea of the um immutable or persistent um data structure that we use. And this is what it looks like um when we apply a mutation. So it's uh the yellow nodes are the ones that are being updated and the green ones are the updated versions and the white ones are the ones that are shared between both versions.

So this is we have a list uh v 0 to v4 and we're pushing a new element v5 and the result of that push is that we we add we we create v5 as a leaf node and then we make a new parent node for v5 which has v4 and v5 as children which means that we are no longer referring to this previous parent node or this zero node that we're in the that were in the old tree. So they're kind they're sort of um they still exist, but they are no longer referenced in the new tree. Um and then we need a new parent for that node. Um so as its left child, it has this new child that we created, but as its right child, we can reuse the right child of this of this node that already existed. So we get to reuse this zero node and share it um between the the new tray and the old tray.

Um, and then we create a new route as well, which is the same sort of thing where we have the the new right child that we just created. And then we get to reuse the old left child, which includes all of the values v 0 to v4 or v 0 to v3 at least um that were in the previous tree. So we have this new data structure which shares most of its um data with the old one um and just adds a few little nodes in order to um represent this update. And this is vastly vastly less memory than if we had copied the whole tree and then added the V5 which is essentially how it behaves because we get to have both copies. We get to have a full we get to have a full version of the the old tree and we get to have a full version of the new tree and we can treat those as completely independent values if we want to.

Um but they end up under the hood they use um basically the same amount of memory as V5 would use or with a little bit of overhead for these extra nodes um which were added. So that's that's pretty cool. Um when we implemented this I was pretty excited. Um there are some um performance differences between this tree representation and like an array um that you would usually use to represent a vector. Um so because it's a tree most of the operations like get set and append are log n because you have to traverse um down to the leaf where n is either the um the like maximum length of the um list which is what we use quite a lot in SS or it's the number of things in the list.

It's yeah uh they end up being fairly similar. Um and yeah in an array um like a vec in in Rust uh you can you can append to the list in 01 uh amortize time because sometimes most of the time you get to just push into spare capacity and then sometimes you have to double the capacity of the list like that's a very standard thing that vectors do uh but we have to pay a login every time. The one where we win and this is um this is the main reason to do this is cloning the data structure. So if you clone an array you have to iterate through and clone every element. If you clone the tree then you just copy a reference to the root and then you have a new tree which is exactly the same as the old one.

And then any mutations that happen on that tree will automatically um only update the nodes of the new tree and not touch the old one. So cloning like that is safe and it's very fast because we just we use a reference counted pointer for the for the root note and then we once once we clone that it's it's basically it's basically free. Um and then iteration is interesting. Um they're both O so they're both proportional to the number of elements in the list. Um but in the tree case the constant factor is quite a lot larger because you have to do this big tree traversal where you go all the way down to index zero and then you go sort of up the tree to index one and then you know you go up and down basically a lot and you have to maintain this stack of nodes that's growing and shrinking and stuff.

Whereas in an array you have all your all your entries just contiguous in memory and you just go you just increment a pointer to jump along between them. So um in practice iteration even though it has the same um bigo complexity ends up being quite a bit slower for the tree vector and it's something that we want to um we we we end up having to avoid doing too much of. Um yeah so in Lighthouse the way this is implemented is a uh library called Milhouse. Um, we implemented this library several years ago and then sort of went through a process of stabilizing it and getting it merged and getting it tested and um, there's there's fuzzing and there's property tests and stuff to make sure that it's really solid and that it could that it could get into the core of the client and replace what we were using before. Um, that replacement went pretty smoothly I think.

Um and then yeah we can fit around 128 states beacon states in memory um in a similar amount of RAM to three to four like full states uh which is pretty cool. So our previous cing state cing system could only keep around three or four states before it would um run out of memory. So we we'd have to limit it to three three or four states. Um and then the new system you can say okay I want to I'll keep up to 128 states in memory. Um and that will be um that will be a similar amount of RAM.

So it means we can process any sort of weird side chain that some proposer might throw at us. Um and particularly the one slot rig case is now trivial because we can just um I've got a diagram of that. um we can just we'll almost always have sigma one still in memory when we get the the next block and then we just we just clone it which is really cheap and then apply the block and mutate it and then we get the new the new state here and then then we can have all three states in our memory and then whatever happens next we're prepared for it because we have um everything in memory. Cool. I'm just going to have a sip of water.

Um there are some challenges to uh using the using the tree state uh repres representation. Um this is actually where the name tree states comes from as well. So um this is the the origin. Um although it's yeah it's kind of overloaded to mean other things now. But yeah there there are some challenges.

Um the iteration is slower as I was talking about before. uh memory fragmentation can be a thing like compared to just having data in a vector where it's all in one vector and it's continuous. We're now allocating all of these nodes everywhere and the the memory allocator has to deal with all of these tiny allocations. Um, and then the the diffs in memory, they're not diffs per se, but this these new nodes that you get when you mutate data structure, pardon me, are um they can still be large. And we this is something that sort of caught us off guard with Hleski because um we had the state C size set to 128 which was very um aggressive I guess.

Um and then in Hleski we ended up with all of these during the non-finality incident we ended up with all of these um very different states where the where the diffs were very large and I think they were up to 200 megabytes which is almost the size of a full beacon state. Um so yeah that that diff was really large because of um the inactivity scores and the balances changing a lot in every epoch boundary. basically all the validators were getting penalized and we we're generating these this like a lot of churn in the state. Um and so that's something that uh we've been working to improve. Um but yeah, there are also huge benefits to this um to this paradigm and I think on the whole it's been worth the trade-off.

Um so Lighthouse is now much more resilient to processing forks and rigs. Um block production is faster because you have to do this thing in block production where you um you want to get the current state. You want to produce your block on that state. You want to compute what the post state is and then you have to basically throw that away and broadcast your block to the network. And then when you get the block from the network, even though it's your own block, you have to then reprocess it.

At least that's how it works in Lighthouse. Maybe some other clients do a thing where they they keep the post state after block processing, but that's not something that we do because it's yeah, you have to yeah, you have to reexecute the execution payload and stuff. So yeah, um and then yeah, this has also helped us have uh slightly faster API queries because um anything that requests state that's cached and we can c so much more state is going to be fast. So that's um that's been really useful for like people running API um servers and stuff off Lighthouse. Um yeah, inspired by the Hleski non-finality incident, we have a few improvements coming up.

Um and I'm working on a blog post to cover these. Um they're not actually ready yet, but hopefully soon. Um the changes and the blog post will be ready. Um the main one that helped for Hleski is intra rebasing. Um which is the idea of reusing um identical entries of a list to share memory.

And I wish I had a diagram here, but you can imagine if you have a list that's all zeros, you don't actually need to have multiple copies of um the zero leaf nodes. you can have this sort of circular thing where you have you have basically one zero and then you have um a parent node where both its left and its right child point to that zero um and then you have another parent node where both its left and its right point to that same um child node. So you you can basically sort of collapse the whole tree into a um pretty much a linked list um where the left and the right children are the same sometimes. Um or you can do things where you you have if two sub trees of nodes are the same then you just reuse the the previous subree. Um and this is something that um yeah we have a we have a function that you you give it a list and then it looks at similarities like sort of self similarities within that list and then it will um it will reduce the memory usage by by taking out these subre.

Um and this I think this dropped the this dropped the memory usage for um the hleski um states from about 200 megabytes to about I think 100 because we eliminated like a lot of duplication in the um in the inactivity scores which was cool. Um and then the other thing that we're working on which is a bit trickier is um pruning the size of the cache um or pruning the cache based on how much memory it's currently using. So if we can measure how much RAM a particular tree or state or list is using um then when we're pruning we can we can say okay well give me the least useful um and most memory using thing and I will delete that first. Um, and I think this is a really cool idea because it means when you're running a node, particularly if you're running on like um constrained hardware, say a Raspberry Pi or something, you can say, "Okay, I do not want to use more than 4 GB of RAM. Um, and I want you to prune the cache um to stay within that within that limit."

So, that's something that we're quite excited for. It's just um yeah, there's a bit of overhead to working out how much RAM one of these data structures actually uses in memory. You have to sort of explore the tree and calcate stuff. If anyone's interested in this stuff and wants to like get involved, um there's some pull requests on the Milhouse repo that you can um that you can look at. Cool.

Um maybe maybe that's a good spot to pause for a quick question. Does anyone have any have any questions about the in-memory um stuff? Any thoughts? Any questions? Yeah, Katya.

Um, I wonder what happens when uh the memor is overloaded. Yeah. Um, so actually that's a very good question. Um, at the moment it's uh it's a bit of a risk. What we do is we just hope we set the size of the cache to a specific number of states.

So say 32 or 128 states and we just hope that it won't run out of memory. Um because what happens if it runs out of memory at the moment is um Lighthouse will probably just get um killed by the operating system for using too much memory. And that was happening quite a bit on um on holes and that was one of the reasons we we actually lowered the the cage size from 128 to 32 in order to limit that. Um but yeah, this this size braced um pruning where where we can measure our own memory usage and keep it under control is a much better um solution than just set some limit and then hoping that um we don't run out of RAM because yeah, at the moment the limit is just set to 32 for everyone regardless of um how much memory they have. So yeah, running out of memory is very bad.

It basically just it means you have to you know you have to crash and then you have to start again. Does that answer your question? Yeah. And will you be slashed in this case for No, you So, um, no, you won't you won't get slashed um because you will just you will lose a little bit of um revenue if you're running a validator. Um, but slashing is really only for um like misbehavior where you're signing two messages that conflict.

Um, so it wouldn't cause you to do that. you would just uh drop offline momentarily um and you might miss a couple of addistations and you would not get paid for those addistations. So you might get penalized a tiny bit for those add a stations. Um but if you could yeah if you crash and then you come back up and you're online and you're testing again then you're fine. Um the problem with running out of memory is that often um it'll just go in a cycle where you'll you'll build up memory and then you'll run out you'll crash and then you'll start again.

So you might test for five minutes and then crash for five minutes and then a test for five minutes and then crash for five minutes. So yeah, validators can go through these cycles sometimes where they're having performance issues um where they're not really um particularly if there's non-finality, they're not really making money. Uh they're not earning um any rewards. They're mostly getting penalized because they're mostly offline. Um, and the penalties are higher uh when the um when the chain is not finalizing because it's like that's the incentive to try and kind of get back online.

So yeah. Um and if I've been bower so I they will just have a missed block. Yeah. Exactly. Yeah.

If you have a proposal like maybe your node crashes you know a minute before you get the proposal and then by the time you're back online you you've missed it. Yeah. you could you could miss a block like that. And yeah, a lot of the people running on um Hleski were running uh like hundreds of thousands of validators each. So if you have a 100 thousand validators on a beacon node and then the beacon goes offline, there's a good chance that you'll miss something good.

Um but yeah, it's also just a test net. So we're glad that that happened on a test net and not on mainet. Um, and in general, yeah, test nets are always a bit bumpier than mainet because um, although people are trying to keep their stuff online, they're not they're not actually gaining and losing real money by doing so. So, um, it's always a bit um, lower in terms of participation and stuff. Thank you.

No worries. Um, any other questions? Yeah, I think we're good for now. Okay, cool. Yeah, let's go on to um the disk stuff.

So, how do we store the state on disk because we've covered how we store it in memory. Um, but if the node crashes or something and comes back, we want to still be able to pick up where we left off basically. Um and in the beginning when everyone started working on the beacon chain six seven years ago um we you know the beacon state at that time was very very small and people were thinking about strategies like oh what if we just you know we just write the beacon state every slot and I think lighthouse used to do this very very early on like before any of the real main test nets these days that would be a very very expensive way of storing state because every state is about 250 megabytes. So, you have to write 250 megabytes to your SSD every 12 seconds, which is going to wear out your SSD pretty quickly. Um, and you would also end up storing about 657 terabytes in a year um doing this.

So, that's that's like that's crazy. That's like larger than any disc that anyone has. um it's larger than any execution node, archive node or anything that anyone complains about having to store. So we definitely can't do this. Um the next simplest approach is to um okay what if we don't store a beacon state every slot?

What if we store one every epoch? And an epoch is this, you know, this 32 slot period. So this is a lot better. Um but it's really only 32 times better. So, um it's 20 terabytes per year instead of 660.

Um and it's still 250 megabytes of IO that just happens in a big burst um once per epoch. Um and then yeah, this is still important to think about because we do sort of end up using an idea that's kind of like this um because we we still use this concept of block replay. So, um, if we only have every state, if we, sorry, if we only have a state from each one state from each epoch, uh, the question is, you know, well, how do you get to those states that are that are in the middle of the epoch? You know, you might have stored the one from slot 32, but the current chain is at slot 36. So, how do I get back to slot 36 after I um shut down my node and start up again?

And the answer to that is that you can replay the blocks. So if you've got this the state at slot 32, you would apply the blocks from 33, 34, 35, 36, and then you would have the state at slot 36. And um this block replay um doesn't use any memory because or it doesn't use any disk space because you already have those blocks stored. Um but it does use some processing time to to reapply those blocks. So there's a trade-off there between how much you store on disk.

you know, do you store 657 terabytes or do you just store the blocks and then try and replay the blocks? Um, but if you have to replay too many blocks, like if you have to play replay 100,000 of them, that's going to take a really really long time. Um, like multiple probably multiple tens of minutes or hours even. Um, so that's not always feasible, but for short distances, block replay is really useful. So if you're only replaying one epoch at a time, you can do that really quickly in probably less than 100 milliseconds.

Um yeah, so most um beacon nodes uh Lighthouse included use um a couple more optimizations on top of just replaying blocks. The main one that helps a lot a lot a lot is to just not store historic states. Um so when the chain is finalizing there's usually only um two around two or three epochs worth of states um that have not already been finalized and any blocks that come in are going to have to reference those states and can't reference things that are already finalized because the things things that are finalized are already fixed. You you're not you can't build a block that builds on an old finalized state. That's just not valid.

Um the the network will reject it. So if we only have to store two epochs worth of states, then actually either of these um strategies are fine. You could write the beacon state every slot and it wouldn't be that many. Or you could write the beacon state every epoch and it also wouldn't be that many. You know, you have between maybe um two two epoch boundary states, which would be 500 megabytes, or you could have 64 states and that would be, you know, 32 times larger.

So it' be like 16 gigabytes. But that's yeah, that's still within um within reasonable limits, right? If your node uses 16 gigabytes of disk, you're not too worried. Um it's better than using 16 GB of RAM. Pardon me.

Um but then there's also other options. So um yeah, for historic states, um maybe you do want to keep them around sometimes. So there's there's certain use cases where people want to go back and look at what the state of the chain was um in the past particularly for validated rewards analysis and things like this. Um so what we used to do in Lighthouse was we'd have a system where you could store one state every 8192 slots. This just it's it's this number for a very specific reason, but 8192 slots is about a day and a half or something, maybe a little less.

Um, so you would have one state every day roughly and you could store all the states back to 2020 um without using too much disc space. This is like I guess 8192 is yeah it's it's several times um smaller than the 20 terabytes per year. It would probably end up being maybe a few hundred um gigabytes max 200 GB or something. Um the problem with this is that replaying 8192 blocks to get um to any specific state takes quite a long time. So if you have a snapshot from yesterday at um 12 p.

m. and then you want the state from today at 11:59 a.m. you would have to replay all of the blocks in between probably 8 8,191 or 8192. Um, and that would take potentially like up to I think we saw it take up to like 10 minutes um to get one historic state.

So yeah, although you had access to all of those states, it wasn't necessarily quick access. Um, and that was pretty useless because usually people that are doing this sort of analysis, they want to run a whole bunch of queries against the historic state. They don't want to just load one one time. Um, so having it take 10 minutes is just not really um, uh, practical. So this is where we cooked up a feature called hierarchical state diffs.

Um and this for a long time this is the sort of technically correct name. Um but for a long time it was bundled with the in-memory tree states feature. So this came to be known as tree states because we had a couple of experimental like test releases that we were calling tree states that had this and had the memory change and that people didn't really notice the memory change as much as they noticed this because people that were running um archive nodes were like wow this database is much smaller and and um you know the queries are much quicker. So yeah, this is the this is the tree states and it is a tree in a way, but really the original tree states that the name is drawn from is the the in-memory one. So this is the this is the hierarchical state diff diagram that I'm going to show you.

Um and it looks a bit crazy, but I'll um I'll explain the idea and um hopefully it makes sense. So similar to the idea of storing a beacon state and then doing block replay to get to a different state, we thought what if we store a snapshot very very very infrequently. So every 2 to the 21 slots that's about once per year. So if you have a snapshot that's once per year and then you have a um a diff from that snapshot to each month of the year say. So you have a January, February, March, April, May diff.

Um then you can just jump to any month um by applying that diff on top of the yearly snapshot. So you to get the to get the um state for say you know April 1st or whatever you would load the snapshot for that year for this year. Then you would load the April diff. you to apply the April diff on top of the snapshot and that would take you to probably April 1st because it's the first day but say if you wanted April 4th then we have another layer in the tree um which you could think of as like the the days of the month and these are diffs with respect to the start of the month. So to jump to April 15, um you would, you know, grab the the one for April 15 and apply that diff on top of the April diff um which you'd already applied on top of the snapshot.

And then you have the state for April 15. And yeah, that's that's kind of the that's the concept be behind hierarchical state divs. um we call them hierarchical because they are sort of organized in this like top down or bottom up sort of chronological um structure and um yeah we we actually I think we have six or seven layers by default so the layers that we have don't quite correspond to years but it's more or less that kind of thing. Um, so yeah, there's if you want a specific state, you'll probably um jump back in time along a couple of these like um these diff edges um to to load your state and then um we also still support block um replay at the at the leaf level. So I think the default yeah the default configuration um it will use diffs to get a a state that's on an epoch boundary.

Um but then for any states that are um more granular than that. So like in the middle of an epoch somewhere um we will just replay the blocks to get to there. So we still have this property that we're only ever replaying um at most 32 blocks. Um but yeah, we have this very quick um this diff thing to load um you know, however many layers there are. In this case, there's four.

You would load at most four diffs and um and apply those. So yeah, that's hierarchical state divs. Um we released that in Lighthouse in version 6, which went out in December 2024, which is just a couple of months ago. Um, this results in us being able to store all the states since 2020 in 163 GB. I checked my um I checked my node yesterday and that's how much it was using.

Um, and the the cool thing about this is this 163 GB is with the default configuration of the layers, but you can also completely configure um how dense you want your diffs to be. So, if you want to have less layers of diffs um where each diff is a little bit chunkier, uh you can do that. If you want to have um more layers of diffs, you can do that. If you want to have less frequent snapshots, you can do that. If you want to have, you know, really frequent snapshots so that you can jump to anywhere quite quickly, you can also do that.

And you can really trade off um disk usage against performance. There's like there's a huge there's a huge state space that we haven't even fully explored um of these of these like diff um configurations. We've found one that we think is pretty good. Um but there could be a better one out there. Like if if if we had a lot of time, we could um we could sync a thousand nodes with a thousand different configurations and then see which one is the best.

But that would use like a lot of um a lot of disk space and a lot of time to sync them. Um yeah uh the specifics of the diff algorithm that we're using at the moment are um that we split up the diffs for the balances and the inactivity scores because they're often repetitive and often changing um in very similar ways. Um and we split them off from the rest of the state. Mainly the validator registry is the thing that uses um a lot of space. Um so for these balances and inactivity scores because they're just vectors of numbers um what we do is we just go along the vector and we subtract the old value from the current value.

So it's just like the new value minus the old value. So if your balance changes by one um plus one then the diff will be plus one. If it changes by minus one then the diff will be minus one. Um and then that gives us this vector of um uh of differences basically um which are probably quite repetitive because B uh balances change in similar ways for for validators. Like if everyone's getting the same reward then this vector is just going to be the same number over and over again.

Um and then we run that through a compression algorithm um Z standard. This is one I think Facebook developed it or something. But it's kind of like it's a really good when we did benchmarks it's a really good um compression algorithm. compresses data like quite a lot and also very quickly. Um so that's really cool.

And then we have um this other one is less well known. Um the the diff algorithm that we use for the rest of the state is called X delta 3. Um it's just a C library that was written by some guy as far as I can tell. Um, but it's kind of it's it's used in some Unix projects and it's it's not like completely um wild west sort of um code. So, it's it's relatively stable.

It's just not being updated anymore. Um but it's it was the best when we um when we benchmarked um the both the compression ratio and the speed of compressing and decompressing. It was it was by far the um it it met the sort of sweet spot for being um quite compact but also quite quick to to compress and decompress. Um so yeah, that's what we've we've used at the moment for the validators. Um but yeah, we are open to changing this.

So the the the diffs are are versioned on disk. So in the future we could potentially change algorithms and it would be completely um backwards compatible because we could we could continue using the old diff algorithm for the old discs that people already have diffs that people already have stored while um while storing new data um with some with some new algorithm if we wanted to. Um, we've actually recently worked out a small change that we want to make to to the diff algorithm and we're going to do that using a like a schema upgrade um that that yeah that will go through and then hopefully just be completely transparent for um for users. Yeah. Um so the the the freezer state um hierarchical state diffs are very useful um for the chain when it's running normally and if you you know if you want to keep all of those historical states it's it's a it's really great.

I think it's probably um yeah it's probably the best way to store archive states. Um I think there's a couple other clients that are working on this now as well. Um, I know that the the um the Aragon consensus client called Kaplan have something that they they claim is even quicker than our thing. And I think they're probably right, but um yeah, there's there's maybe a bit of a bit of more optimization to do there. Um but the main the main downside is that um we can't use this um hierarchical state diff system at the moment for uh the non-finalized states because um it only works for the those uh historic states in the freezer database.

So um during the non-finality incident on Hleski or and we knew this was a problem. So this was something that we were preparing for but we got caught off guard basically. Um but yeah, during that non-finality incident, we um we ended up using a lot of disk space because Lighthouse's strategy for the um the non-finalized part of the chain is still to just store a state every epoch. And when um when finality stops, you end up storing, you know, I think we had three weeks of non-finality. So we had um you know, whatever 225 times 3 weeks.

So it's 225 epochs in a day um times 21 times 250 megabytes per state. So the disk usage was growing at a really alarming rate. Um and then also we have every side chain um also has states stored on it. Um and they were also using a lot of dis space. So I think Lighthouse on Hleski was using upwards of 700 GB, 800 GB, more than a terabyte sometimes um which is like really really really too much um and can potentially lead to you know this sort of death spiral where the network um it like runs out of space and then people get knocked offline and then um because they're offline more people run out of space because the chain is still not final.

izing it's kind of this like once the chain stops finalizing it can be really hard to get back. Um we did manage to get it back on Hleski but through sort of sheer um willpower and like getting people to coordinate in a really um I guess an unusual way that they probably wouldn't coordinate on mainet. People basically agreed to slash themselves which is uh yeah would not happen on mainet. Um but yeah we do have a plan to fix this and this was something that we were working on even before the hleski non-sality. Um, and it's to use the hierarchical state diffs for the non finalized states, too.

Um, this seems to work really well. Um, it's just in the process of being tested and, um, we're ironing out a couple of bugs and making sure that it's really like watertight before we um before we ship it on on mainet. Yeah. Um, behind the scenes, something that I've sort of glossed over is um, how is this data actually getting stored in the database? Um, most clients use an embedded database of sorts.

So either Level DB, Rock DB, MBBX, Red DB, sometimes SQLite. I think Nimbus might be the only one using SQite. Um, and yeah, these these databases are are basically used as like a key value database. So you have a key that is some bytes and then you you map it to a value which is some other bytes and the database just stores those for you. Um, and you can then flip them up later.

All of these databases support some form of like transaction or um an atomic write. The one that we use in Lighthouse mostly is Level DB and it's looking a bit old at this point because it doesn't really have proper transactions but it gets the job done for us. Um and we will probably move to Red D as it becomes more mature. Red DB is like a Rust. It's a Rust embedded database.

Yeah. Um this is the last section and then maybe we'll do some more questions. Um so there's a bunch of other data apart from the state that we have to store. Um namely the blocks and the blobs are very important. Um and the way that we store them in Lighthouse is actually pretty straightforward.

Um and we probably want to change both of these strategies in future. But um at the moment the way blocks are stored uh we just use the block route as the key and then we have the SSZ um block as the um as the value. So this is just the block route maps onto the bytes of the block and then when you want to look it up you you give the database the block rate and you say hey give me the give me the data for this block and the database just gives you the block straight away. It's very simple. Um the downside of that is that we actually store all of the blocks um that we have in the database in one big bucket.

So the things in there are moving around a lot because they're just ordered by block route and then the database will like uh rebalance those every now and then. So an improvement in the future that we want to do is to actually store the finalized blocks by their um by their slot as the key. Um because then that allows us to store them sort of in a continuous contiguous um in a block. So you know in in sorry not in a block in like a uh in a sequence. So if you want to load all of the blocks between slot zero and slot a th00and you can just go to the database and go okay 0 to a th00and they're all sitting there next to each other and I can just load them all up really quickly and there's no like fragmentation um happening.

Whereas at the moment if you want to load zero to a th00and you have to go okay well zero has this block rate and then one has this block rate and you're like jumping around everywhere. There's a lot of um non-local accesses. Um and then yeah for blobs it's very similar. We um we use the block route as the key because blobs are fundamentally always tied to some block route. Um and then we have an SSD list of blobs as the value.

And honestly, this was a mistake. We should never have stored the full list of blobs as the value. What we should have done was use the block root as the key and the blob index with it also as the key and then have a single blob as the value. So then that would allow us to say, okay, I want um blob one from this block route. So you'd say the block route and and then one as the index and then you would get a single blob.

Um, and you would still be able to iterate all of the blobs for a block route by just looking up the block route and index zero and then looking in the database because everything in the database is stored in like um a key increasing order. So if you have block route zero, block route 1, block route two, block route three, block root four, they are all right next to each other and you can always get the list um from the um from the uh from the from the sort of keyed form whereas you can't go the other way. So the advantage of the the system that we want to move to is that it would allow us to store um just a subset of the blobs for a specific block. Um, a lot of the layer 2s uh are not really interested in all of the blobs ever. They only want to know about the ones that are related to their layer 2 um sequencer.

And um yeah, this is something that we want to support long term where you can have a node that just knows a subset of the blobs. Um yeah, I've put some I've put some links here. Um, these are kind of adapted from a previous talk I did on that was more on the functional data structure side. But if you're interested in digging into the ideas behind Milhouse, I've got some links for um, functional data structures and functional programming generally. Um, there's a library that we do also use that I didn't really cover called Rapids.

Um, which is persistent data structures in Rust. If you're familiar with Rust, that might be interesting. Um, Proto Lambda, who works at Optimism now, but used to work on L1, um, also has a Python library, which was a bit of an inspiration for Milhouse, which is our library. Uh, this works in a very similar way, um, but just adapted to Python. Um, and then, yeah, then we've got the Milhouse library and, um, if you wanted to learn more about the Lighthouse tree states um, on disk, then we have docs about that in the Lighthouse book.

Thank you. Any um any questions? Awesome. Yeah, thank you very much, Michael. Uh yeah, we have a hand up from Kia.

Yeah. Um yeah, uh I wonder um do you have any improvements these strategies for for storing blobs? And I mean the same as for beacon state. Um Oh yeah. So, so blobs are hard, right?

Because um they don't have a lot of um they don't have a lot of structure that we can exploit. Um like the beacon state, the diff thing works very well because it's it's always like it's always like evolving from a previous version of itself. Um whereas blobs are kind of just random, right? like it's or maybe some of them could be related to each other, but often from the um from the baking nodes point of view, it's just like here's some arbitrary data that you have to store. Um yeah, a while ago we were thinking about trying to use compression just on a on an individual blob um basis.

Um just if there's repetitive parts within a blob, you could at least compress it a little bit. Um or I think sometimes blobs will have a lot of zeros because they're like filling out to the size of the um the blob because blobs are blobs are fixed size, right? Um so we yeah, we could potentially do compression that might that might be something that we do um when we do this um transition. Um but yeah, I think with Peras when we have data columns, I think because they're encoded in a way, um they use like a redundancy coding. Um I don't think they will compress very well.

Um so yeah, I I'm not sure. Do you have any ideas for blobs? No. Yeah, it's it's hard. I think I think no one no one really has any ideas for for how to um how to compress blobs.

I think one of the things that maybe helps for blobs is having um just better tooling for for controlling which blobs you want to store and being able to import and export them from nodes and stuff. Um so this is something we're working on at the moment. Um, and we're planning to host like a um we're going to try and host an archive of all the blobs on mainet because this is something that no one else really seems to be doing. There's maybe there's a few people that have this um but we want to host that and then encourage other people to um seed that data. Um and then from there maybe um people could build specific snapshots like okay here's all the all the optimism um blobs or here's all of the um you know blobs for a different L2 um and then yeah that could be that could be a way of like keeping the storage um manageable.

Yeah. Yeah. Thank you. You should reach out to Mario. He likes archive notes.

Yeah. If you want to host one let me know. Well, I'm running my Aragon archive node right here like next to me and uh I ran out of space with Lighthouse couple times already, so I switched to Kaplan for now. Um Oh, interesting. Okay.

Okay. It was yeah it was actually getting like I was it was it was always getting too big and I cared more about the execution archive so I uh just you know resting it from checkpoint every time and uh yeah now I now I don't have any lighthouse archive data anymore but um yeah maybe if I get another disc. Yeah. Yeah. Were you you so you were running it with um with the tree states database and it was still too big.

Yeah. So there was the the after the update it got better. Uh but it was still it was still running out of space like it was just growing so fast like uh I'm not sure. Yeah, even if I even if I um uh recync from uh from uh checkpoint after a couple of weeks it was like a terabyte or something of Oh, really? Yeah.

Yeah. Um yeah, maybe it was just my settings or something, but uh so actually now the uh new version of Aragon is smaller. I need to resync the Aragon because they have the Aragon 3 which is like one terabyte smaller. So I might be able to try it again. Uh so I'll give it I'll give it a try with the uh yeah settings as well.

Yeah, I don't think I don't think Lighthouse should ever hit a terabyte anymore um with the default config anyway, but we can we can chat about this. Yeah, totally. Yeah. Um yeah, need to uh need to try that again. Like I would like to have more both archives, but yeah, so far I could fit only one.

Um yeah. Um cool. Uh there are some questions in comments. Uh maybe if you can see them uh in the Jity, you can you can read it. Uh there's a question of SS um in the Jity.

I don't see it yet. Can you can you see it in comments or I also the comments aren't really working for me either. Oh, that's weird. Um like right after I'll grab coffee. Yeah.

Yeah. Yeah. I had some of them disappear. It's weird. Uh but there is a question.

Uh can you talk more about SS and how the Oh, I'm going to pronounce it. Merkel is this macation. Mralization works. Is it similar to immutable vectors you talked about before? Yeah.

So um yeah, so SSZ and Merkelization are two things that we do with the data structure once we have it in memory. So um the SSZ representation is is basically a um it's like a it's like a flat representation of all of the data in the data structure. Um and to encode as SSZ, we have a um we have a a function that can basically encode a list by just iterating along the list and just writing out the um writing out each entry of the list. So when we're doing SSZ, we um we iterate along the leaves of the tree and we leave out all of the internal nodes which are only related to localization. Um, and then that gives us basically a um it's it's it's fairly similar to a flat flat vector of the of the leaves.

And then obviously if the if the leaf nodes themselves implement SS, they will get serialized inside the inside the leaves. And SS has some stuff. The quirkiest thing about SS is it has some stuff about what to do for things that are variable sized versus fixed size. So when things are fixed size, it's all fairly simple. They just get written out.

But when they're variable sized, they'll get written in a different place with an offset that tells you where where they're written. Um, so yeah, SS um yeah, we have in Lighthouse, we have a derive macro that implements um SS encode and decode for any strct. Um and then it's just the basic types like numbers and um lists that we have to implement um SS said manually for and then the derived macro if you have a bunch of you know like the beacon state has a bunch of fields that are U64s or lists or whatever the the sort of primitive types the derived macro completely takes care of that and just says okay serialize each of these fields in order and then when you're deserializing same thing just deserialize each and then but that we'll use under the hood the the basic implementations. I hope that answers the question. Um SS is one of those things that sort of it has a bunch of subtlety but you just look at the spec and you just try and do what the spec is saying and run some tests and make sure that it's working.

Um it's not I would say it's not especially um complicated like on a conceptual level. It's just it's just a binary format. Um for merkealization we actually leaned more into that with the um immutable list thing. So meralization is the um the process of hashing all of the leaves into the internal nodes into the internal nodes and all the way up to the root. And we um we are storing all of the hashes uh inside the immutable um vector tree.

So we are cing all of the hashes that we that we um that we want to have. So um this means that hashing is very fast for um the Milhouse types because we have a full cache of um all of the hashes for the previous nodes. And then when we apply a mutation, we might get some new nodes that aren't hashed yet. And then when we rehash the tree, we just rehash the the parts that are new and we reuse all of the hashes that we have for the for the parts that are already hashed. Um, so yeah, our benchmarks for um like hashing the beacon state got a lot faster when we switched to the Milhouse um type because we were not um we were not redoing as much work and we were able to um take advantage of the the the similarity between um each successive state.

Very well. Thank you so much. Um awesome. Um any more questions for feel free to raise your hand. We have some uh that we missed in the comments before um so not sure if it's still um if it's still was answered but uh do these data structures apply to every client?

Um yeah so every every client um will be using some variant of these data structures. Um the ones that I've described are somewhat lighthouse specific. So the hierarchical state diffs are pretty lighthouse specific for the um for the database on disk. Think anyone else is doing it exactly like we're doing. Kaplan is probably the closest, but I don't actually exactly know what they're doing.

Um and then the in-memory one. Yeah, that the in-memory tree states um are used by I think at least loadar um and branding maybe but I actually don't think do u yeah so load yeah loadar has a very similar system to us where they have mutable copy um as far as so Yeah. And then a lot of the other clients will have similar um similar sort of tree structures like what we had in Lighthouse before we had this was a um a sort of more customized um c that was just for the tree hashing uh the localization um part of the of the calculation and then the actual data was stored in a vector. Um, so a lot of your clients be using something like that where they have they do have a tree but they keep the tree separate from the data whereas we've decided to put the data and the tree together in one thing so that it's all uh unified. And both of these have their advantages and disadvantages.

We've we've leaned into being able to copy the data um is a useful thing. So being able to clone it and and make variants of it really cheaply, we think that's a good thing. Um, but if you just want to have really really fast updates, then it's actually going to be quicker if you have a just a regular vector that you can update and then a tree that you keep separately. Um, so there's pros and there's pros and cons to each and different clients um do slightly different different things. Interesting.

Thank you so much. Yeah, we do. Well, um, all right, folks. Any more questions, feel free to ask. Raise your hand.

Uh, we have some that we missed in the chat because of the mess, so feel free to repeat them. Uh, yeah. Um, yeah. Hey, I was wondering how you guys are thinking about um implementing the pruning feature. Oh yeah.

Um it dep it depends what you mean by pruning. So there's several types of pruning. Do you mean the um like EIP 444 where we drop history? I I'm I'm thinking of if if you could open up the slides. Yeah.

Hang on. Give me a sec. Uh um can you see my screen now? Yeah. Um sorry.

You still there? Can you hear me? Yes, I'm here. Sorry. Um, which which slide?

Go on. Go on. Next. Next. Next.

It's about the state. Yes. Up. Uh, again. I think this one.

Uh, no. No. This one. Next. Next.

Next. Uh, next. Am I am I sure about what? Oh, you mean you mean the one you mean the one about um storing states during non-fality? Exactly.

Yeah. Yeah. Yeah. Um, so what what we're doing for this is um very similar to the hierarchical state diff um algorithm. The the thing that's different is um for historic states, it's all very simple because there's only ever one state at a given slot.

Um so the the the sort of all the indexing and all the math that we do is just related to the slot for the for the historic states. So that's that's like quite simple and easy to understand. In the case where the chain is not finalized, you have to deal with um multiple forks like during the Hleski non-finality incident at one point there were there were more than a thousand different forks, a thousand different potential head blocks. Um and ideally we want to be able to store all of the state for all of those forks, which is kind of a lot of a lot of state. Um so we still use the same idea of the hierarchical um state diff.

So we have like the the yearly snapshot, the monthly snapshot, daily snapshot, the daily diff. Sorry. So so yearly snapshot and then monthly diff, daily diff, whatever. Um but we have those tied to a specific change. So we might have multiple diffs for say April 3rd.

Um, and then you know the the the two diffs for April 3rd might point to the same um diff for April 1st or they might point to different ones again and then they might point to the same snapshot for um the start of the year or they might point to different snapshots for the start of the year depending on how long non-fality has been going. Um but most of the time most of the time non non-fality is relatively short. So the the length of these um diff chains will be relatively small. Um and there's just some uh there's some complexity around how we store the finalized state itself. Uh because the finalized state is a sort of natural snapshot for the for the diff algorithm, but it's always moving.

So you you don't want to diff um you don't want to diff with respect to something and then your your thing that you're diff with respect to moves forward and then the thing that you were diffing from is now deleted. So um we are dealing with that um by also keeping some of the older state that is um an ancestor of the finalized state. So we kind of keep we kind of have like another diff snapshot in the database um say from the start of the year which acts as the basis for the finalized state and everything after it. Um and then that yeah that snapshot um gets used um yeah despite being older than the actual finalized state if that makes sense. It probably doesn't make sense.

I think I think I'm not explaining that particularly well. Um but yeah, BA the to summarize the idea is that we use the hierarchical state diff thing. Um and then we just we tag each of those chains of diffs by the um by the state route of the of the chain. So um we can have multiple multiple diffs at the same at the same slot. Um wouldn't that like have a big overhead for like memory and storage as well?

Yes. Yes, it does. it it's definitely more than the than the freezer database. Um but it's still a lot less than um storing the state every epoch because the diffs are relatively small. Like a lot of the diffs are um you know a megabyte or less.

Um, so even if you have uh a thousand dips for slot um you know 10 10 10 slots after the finalized state um that's still a lot better than having a thousand times full states at that at that height if that makes sense. Like it's you'd have you'd have 1 megabyte time a thousand rather than 250 megabytes times a th00and which would be the current system. Makes perfect. sense. Thanks.

All right. Yeah. Very well. Um, any other questions, thoughts? We still have a few minutes to go.

If you guys want to use them, please go ahead. Um, or again, if we missed any of your questions in the comments before, I'm very sorry. Feel free to repeat them. Okay. Um, yeah.

I wonder how you use cash for for for this strategy like Yeah, actually that's a that's something I didn't cover, but um we have a um so for the um do you mean do you mean more for the hot um for the non-finalized states or more for the freezer states? Uh for Yeah. for freezer. Yeah. So, I can cover both if you like, but I'll cover phraser first.

Um, so yeah, we have um we have quite a complicated or let's say complex c um because there's two ways of storing um those historic states. They can either be as like the bytes that you get out of the diff algorithm or they can be as like the deserialized um tree basically in memory like a like a like a Milhouse a Milhouse tree. Um so the C actually uses um both of these representations um and it will convert between them um like sort of on demand. So if um say if you load the state at slot 1,023 um how that will happen is you'll load the snapshot you'll apply a bunch of diffs you'll get the slot 123 state and you'll convert it from bytes into a full state and then you that full state will get served back to the application layer. Um and then if you then if you request the state at slot 1024 um we will look at the cache and we'll say okay well we have the diff bytes for 1023 and we also have the full state for 1023 and we could use either of these to get to 1024.

So what we'll do is sort of opportunistically if we have the state there then we might just replay one block to get to 1024 and then we have the state the state for 1024 or if it's like a longer distance say we had the state for 1023 and then we load like um 2048 and 2048 is diffed off it would probably be more likely to be diffed off 1024. So say we have the diff bytes for 1024 in memory and we load 2048. Um then we would have the diff bytes for 1024 which is just a bunch of bytes a vector. Um and we would load the diff for 1024 uh so sorry diff for 2048 and apply it on top of those bytes to get the bytes for 2048. Um so we do this sort of combination of um keeping the bytes around or keeping the um keeping the state around or sometimes both.

Um, and there are probably probably still improvements that we could make to that to that cache. Um, because but yeah, the the one that we have at the moment is potentially um yeah, could be a bit inefficient. Um cuz sometimes it will miss an opportunity to you know um or it might just store too much I think because it would be maybe storing the bytes and the and the diff or it would miss an opportunity to like um to use some some information that it already has loaded. Um yeah, does that answer your question? Yeah.

And for non-fality and then for non-fality it's a little bit simpler. um we just store um a bunch of states in memory and then we rely on the um we rely on the on the sharing between the nodes um of those of those states. So, um, we have like the finalized state and then usually with 32, um, states ced, you'll have the finalized state, you'll have, um, and all of the all of the states in the most recent epoch, and they're all sort of sharing a lot of their data in common because of the the fact that they're based on each other. Um we also have an operation called rebase where you can take a state and you can make it um make it share as much memory as possible with another state by um you know wherever they are wherever they have identical data you use the data from the from the base state. So something that we do with the state ces we often if we get a new state we'll rebase it on the finalized state so that it's sharing as much as possible with that finalized state and then we'll put it in the cach.

But yeah, that's the cach that state cach in memory is the one that we want to implement the um size based pruning for. So the one where we want to say okay I only have 8 GB of RAM or I only have 4 GB of RAM. Do not use more than this. And then it will prune states from there based on um how big they are and how and try to stay within that limit. Thank you.

Thank you. Very well. Um, yeah, I think uh we are uh close to wrapping up. Um, folks, anything else you would like to ask before we finish? Any last thoughts?

We covered a lot today. Yeah, that that actually it went quick from my perspective, but I Yeah, I feel like we cover a lot. Thank you so much for um having looked up. Yeah, thank you so much for joining. Yeah, I really appreciate it.

I think we have uh great content here that we can share and uh hopefully people will enjoy the recording as much. Yeah, awesome. Um yeah, thank you so much, Michael. And yeah, it was great. I mean, almost 90 minutes, so uh uh uh strong start in the morning for you.

Yeah, very well. Yep. Cool. Thank you so much. Awesome.

All right. And have a great day. Bye, everyone. You, too.