Database Architecture Improvements, Layer 1 R&D - Day 2

[Music] hello my name is andrew i am from aragon and i'm going to talk about uh database models for ethereum specifically for the execution layer and more specifically for full nodes so that that is related to guillaume stock and with the vocal tree you have blocks with witnesses but still there should be some nodes the builder nodes perhaps who will have all the data so that will mostly be about those nodes so how was the how we used to think about the the database model the state model in the old days it was nice and simple you have the marco patricia try and it was used for two purposes so it was used to calculate the state route and it was also like evm which uh has access to the state it also used to go through the miracle patricia try uh and what are the benefits of this data model that it is simple uh conceptually because it's only like one data structure you have to deal with and also there is another big advantage is that when you have this knuckle patricia try it's really easy to evolve it through time so you make some changes with every block and most of your leaves leaves being accounts and and storage slots they don't change so this the route will change for sure and a lot of intermediary intermediate nodes but most of the leaves won't change so you can kind of combine a lot of tries in one one one database that will that will you give you the easily a view of multiple states you can do side chains you could do what not using [Music] using the same data structure so it will be a forest essentially uh what are the drawbacks of this data model well it's it's basically it's slow why is it slow uh because well consider the evm access the vm access to the state you from the evm point of view you what you are interested it you are interested in the leaves but in this model you access all the leaves through the entire tree so you go from the root and through the intermediate nodes and so on so it's very slow from that point of view also from the point of from uh that this is conceptually how fast sync used to work and you get all the nodes you start from from the root and you download everything why why is fasting slow because essentially the the all the the nodes on top of the leaves they take a lot of space uh and also there are some like more technical issues you could only download only like a little limited number of nodes perhaps that could be improved but the main reason is that that you had you were downloading a lot of uh derivative data but then when you start thinking about speed and performance of course there are many many conflicting uh it's a multi-factor thing thing and you have to think about what are you optimizing for are you optimizing for performance at the tip of the chain or you may be like you're interested in a building a full archive no to you and you are interested as some kind of historical transaction tracing do you optimize for sync time disk footprint uh footprint etc et cetera so uh that those are important questions and uh of course we want to optimize for everything but sometimes you have to make trade-offs so here is a more modern data model i say so it corresponds to the early state of aragon and also how i understand it conceptually at least it corresponds to the snap sink in gas and you like conceptually you start thinking about the hash state as your primary data model so what i what i mean by by by the state by the state i mean the accounts the storage and and also the code and essentially on a very high at a very high level you can think about it as uh as the associative array also known as as map or key value pairs or dictionary and it can be stored like that that associative array like how it is store stored in your in your ethereum client that's another question it's it can be uh b plus trees uh in say lmdb or it can be some other uh database storage mechanism but it's for if you consider accounts it's basically a table you have and the key which is the the hash of the account address and the value is is the account value somehow serialized might be rlp might be another way of civilization uh but then so when so hash state is the the leaves the actual state and from the evm point of view you don't need access to any intermediate nodes you you are you load a particular address like an account from a particular address you load a particular storage slot uh you read a particular code on top of that hash state you can build and you you you have you have to build the the the the commitment which is the miracle patricia tri state route and for that at least in aragon we have this another database table with all the intermediate nodes or like maybe not all but the upper part of the knuckle patricia try and those are completely derivative from the data so once when you have the state you can like that that's all you need then you can rebuild the entire meko patricia try from scratch and you might not do it with every block you might you might have a clever algorithm to update the knockout patricia try with uh with the changes but conceptually state the hash status is the primitive um what is what what is uh what are the pros of of this model the it's it's it's reasonably fast it's reasonably simple and the fact is that that you have this mirko patricia try um it helps you to verify during the during [Music] sync it helps you to verify that the data that you download that it matches the statute so you can download a range of uh of accounts and then you can ask for all the like i prove essentially approve um from that uh range of leaves that all the intermedia with all the intermediate nodes you can verify that those leaves belong to the state route right so it's a nice property you you you utilize this commitment scheme which is the miracle patricia try to help you with sync verification so you don't have a malicious agent feeding your rubbish you're in sync that's nice uh so yeah by this by state we mean uh accounts storage and and code and with easy in a a a as a map as an ss as an associ associative array so the the keys are hashed account addresses and hashed storage slots and why hashtag the reason why they are hashed is because when uh ethereum was designed it was uh there was uh originally the keys were not hashed but if you don't hash for the purpose of uh miracle patricia try uh root calculation but if you don't do the hashing then there is a danger of the the tree being imbalanced and that is a potential attack so this extra hashing is a means to keep the miracle patricia uh try balanced and then but with the with this model at least how it's uh in this simple form hash state now it's it's not easy to do to to have the the multi-state so hash state is only the current state and uh corresponding to a particular block to a particular state like a point in time and to do like just in order to support uh like multiple states multiple blocks to support three orgs and and and so on you need some kind of deltas that would move you from a particular state to another state that that's what we do in in aragon we have uh we have backwards deltas that you can so if you need to do a reorg you use backwards deltas to unwind to the fork point and then you kind of re-execute blocks again but you you i guess if you you can do like you you can be creative with deltas but you in this simple model you need something clever to handle the multi-state and yeah and also there are questions of course how do you update those intermediate intermediate nodes and so on uh which i'm i i i'm happy to go into during the longer session uh now enter state expiry so state expiry is this new vision combined with the vocal uh tree uh so which will uh which will achieve statelessness and end state expiring so i'm using uh your mentioned that there are like uh there are a number of uh quite a few approaches but basically i'm using this uh approach from vitalik's post and the you have periods uh and uh each all the pre all the periods apart from the most recent one uh are frozen and and you only update like data from from the period from the very last from the latest period and you can copy uh some if you access state from the previous period you you copy data from from that so we kind of for our purposes we are only concerned with the um most with the two most recent periods uh and the the commitment scheme changes to instead of being the americo patricia tri route to the vocal trial route so by commitment like you commit to a certain state so that's the future data model uh uh i'm proposing and the the there are two changes like the one change is that you move from the head state to the plane state um so the keys are now not not hashed anymore the the for account it will be the account address rather than the hash of the account address and why why do you need that you need that because for vocal tree uh the how the vocal tree is specified there is no that like there is no extra hashing uh in the velcro tree uh and there are al also other benefits if you move to the uh unheshed state you can have range queries so you can arrange like give me slots from from 0 to 120 you can do that easily when your base data model is the is the hash state you cannot do that easily also for from the avm's point of view it's this plain enhanced state is even better because from the edm's point of view you access account by its address but not by the by the hash of the address um so that that that's kind of one fundamental change and then on top of that like adjusting uh the scheme to the new world where to the state expiry world we can have uh states corresponding to period zero and period one both have the same format so that's the the plain state format and e and the period one the period zero acts as a backup to period one but period zero is is frozen it does not change it's it's static which leads to interesting implications uh and then with the same the same underlying state format but you can have with this like multiple commitment schemes you can have the marco patricia try uh for uh for period zero and the the the development tree for period one then and you you but then you you have to here you have to be clever on uh in how you go from your state which is like simple a simple map of an enhanced keys uh to accounts to to to the miracle patricia try route or to the local uh vocal route so it it can be quite tricky and complicated but there are like certain upsides that you you you uh now it's more flexible you decouple your state from a particular commitment scheme maybe if we don't like local trees in the future it will be easier to migrate to a new commitment scheme but there are of course many questions with with this data model how uh how do you think securely how does the transition work uh where do we get pre-images because we have a population of snap sync death notes that don't have pre-images and and so on so uh yeah i'm i'm happy we need more time to discuss those questions i'm happy to talk about it uh during the longest session uh then basically like that at a sneak peak uh in aragon 2 it's still under uh active r d we kind of we utilized this idea from from state expiry that you can like you you can split uh the state along the the time axis and uh like your say you have blocks from uh 1 to 10 million uh the the state corresponding to like two to as of block 10 million and that is fixed like for uh and you can compose uh uh later changes um uh which will be more in flux uh but bigger historical changes there those are static and and they don't change you can put them on on torrent uh those like snacks historical snapshots uh and uh use some kind of uh social commitment to to those data files that because they don't have a change so it's somewhat similar conceptually to the locks structured uh merged three databases a little bit again we can go into into more detail at the longer session and that's still under active development for aragon too and but we when you have a static uh state aesthetic associative array you can do all kinds of clever things like we are we are toying with the idea of perfect of the perfect hash function right so and the takeaway is for all el clients migrate to unhap state in aragon it's like took us quite a while it's a huge uh undertaking but then you'll be able to do velcro you will be able to do the gas reform in velcro your evm will be fast lots of benefits [Applause] [Music] you