L1s Untangled Stateless Validation and the Path to Hyper Efficiency | Bowen Wang - NEAR

[Music] hey everyone I'm Bowen from near and uh today I want to talk about uh stud validation and how it solves the uh scalability problem so let's start with a problem that uh maybe many of you here are familiar with uh which is the state growth problem uh and it's a huge obstacle to blockchain scating uh the basically the problem is as follows as a blockchain get used uh more and more over time uh the state would ACR through uh usage of transactions people calling smart contract uh more people creating accounts on the blockchain so on so forth uh and if we take a serus example uh I think the state size has been growing at about um tens of gigabytes every year uh and obviously this depends on how exactly you count the size of the State uh because with merization uh or with the MPT structure uh the C is overhead uh with many hashes uh in between uh but anyways the uh state is estimated to be several hundred gigabytes today uh and it's keep growing uh the problem with that is uh uh become it becomes more and more expensive for people uh to operate a node uh whether that's a validator node uh or just people want to operate a full node and at some point uh this problem would become so severe that it becomes uh very difficult for people to keep up uh with the operation of the network so what are some of the solutions we have to this problem uh since it is not new uh even I think at the beginning of the term people have been start thinking about uh the future uh and how they would address this problem uh so I think there are generally two directions people are exploring uh one is charting uh so basically the idea is that uh instead of having one Global State uh we divide the state into uh many uh states of smaller shorts and uh none of the no in the network actually need to store the entire State um and the other direction is State pruning uh so this is idea that even though the state itself can be very large let's say on the order of uh sever hundred gbt or even larger uh you don't actually need all that state all the time um for example when you execute a block uh there is a gas limit associated with that and it naturally implies that there is a limited set of state that you can touch during the execution of the block uh so what you really need is that uh it's uh you really need uh the state that's needed for execution of a specific block so the idea is that uh can we find some way to store only the state that you need uh for the execution that that you need to do and this is where the stus validation idea comes in uh the idea was originally proposed by vitalic back in 2017 uh and roughly the idea is as follows uh instead of every single node in the network having to maintain the entire State uh in order to execute and verify the execution um the networks divided into uh two rowes uh there is the validators uh who do not maintain any state locally and they rely on state witness uh to execute the blocks on the other hand there are block producers uh who is responsible for producing blocks uh obviously they maintain the state locally and they also produce the state witness so the state witness here it just a glorified word uh for saying that uh the all the state pieces that are touched during the execution and also the proof that it indeed belong to the correct State Route uh so if if EV validator receive St witness then they can uh verify that the state witness indeed are referring to the state uh that that belongs to uh the state that specified by the state rout that they know so this idea is called weak Steelers uh basically there's still some entity in the network who's maintaining the state uh but the vast majority of uh NOS in the network do not actually need to store the state locally so the advantage here is that because uh the the validators or like majority of them don't need to stay uh store any of the state locally uh it relieves this huge uh storage pressure from them and also it uh eliminates this uh Downstream problem where like uh the validators don't really need to think about what happens when the state grows large uh because they themselves don't need to uh store any state locally anyways uh so for each block they would receive the state witness required to execute that block and they just do the execution based on that and verify whether the execution is accurate or not so on the other hand uh the block producers they do need to store the state uh but roughly the idea is that there only need to be a few of them uh in their entire network and because of this uh they can afford to operate more expensive hardware and potentially get more reward as a result so this asymmetry of row uh may feel strange at the very beginning but it's actually a very arogant design trying to reduce uh the overhead of the network and improve decentralization uh so this idea is roughly that a small portion of the network uh need to operate more expensive Hardware um but at the same time they're not gaining disproportional power uh to control the network so the block producers here yes they do run more expensive Hardware they do more stuff uh like not just producing the blocks but also um producing the state witness and distributing the state Witness uh but there's not much they can do uh more than what they can already do today for example they can decide what transactions they want to include they can decide what transaction not to include but beyond that there's no way for them to uh crop the network by themselves uh because validators would actually validate all the execution uh and the state witness as well uh verifying that they actually belongs to the correct State uh also the execution uh is also verified by running the block through the St witness provided by the Bro producers so on the other hand uh validators themselves are pretty cheap to operate uh and because of that there could be um a lot of them and that's definitely beneficial for the decentralization of the network now I would argue that this is great uh but it's not a full solution uh for the scaling problem so this the main problem here is that uh while the this introduction of this asymmetry of row uh greatly reduces the pressure um the hardware pressure on most of the nodes in the network uh we still cannot just simply ignore uh the uh block producers so basically the state grow problem still exist for them and if we don't deal with this problem it means that at some point let's say the state size which um on the order of several terabytes uh it will still become difficult uh for Brock producers to uh to actually store them and operate them uh because it's not not just about storage uh they also need to be retrieved um in a relatively fast way when you actually execute the block and the failure to do so may cause degradation of the performance of the network and also just to actually operate those uh Brock producers it should not be so prohibitively expensive that only uh one or two people are able to do it uh still there should be a handful uh of of those block producers otherwise the network can easily suffer from a liveness failure uh if those prod producers go offline and that's not what we want so what Solutions do we have uh to address this problem uh in one of the earlier slides I talked about sharting uh so uh to reiterate this is basically the idea that um the global State can be divided into sh state so that no one no valer needs to maintain the state of all the sharts uh meaning that they don't need to store them locally and that the networks generally scales which network shards uh obviously there's overhead associated with the scating uh for example uh the network needs to coordinate uh which vs get assigned with Shard and how the cross Shard communication actually works uh but in general it scales with the number of shards in the network so the the idea is that we can combine those two different approaches together and produce an arrogant design that's um that would solve the scaling problem for L1 blockchains so roughly the the idea is as follows we still have two rows in the network there's the trunk producers um so that means the Shar block producers and the validators uh so the chunk producers uh would be responsible for producing um chunks or sharp blocks for each chart and uh also the state witness associated with those um chunks so the validators they do not actually need to maintain any state locally in order to do the execution uh and the Val on the other hand would get rotated on every single block uh into into validating a different Shard um and uh as you will see this is for security reasons so in uh in a lot of the earlier designs for sharding uh the security assumption is uh big problems uh because with the more shards that you have uh there is this um problem that cannot be ignored that it's possible for some shards to get corrupted even though the global assumption the global bft Assumption of no more than one s malicious vors may still hold uh meaning that there could be the case that globally there's only one third of vators but this assignment to Shard uh means that some viors uh some Shard get a uh a lot of malicious viors uh and there's also this adaptive corruption problem uh which is that even though um the assignment may be fine but over time uh the um Valar of A Single Shard can get corrupted and uh they would be they would conspire to produce something like invis State transition and can m a lot of token out of s air uh and that's definitely not great so the state the in the stus validation design because the valid actually rotate on every single block um and the the assignment to sharts are dictated by the randomness Beacon uh provided by the protocol and it's uh not possible to bias that uh there's a security assumption obviously depending on the system parameter but basically we can make it such that um the probability of of uh some sh getting corrupted is uh extremely low and and won't happen in our lifetime uh so that is kind of what it makes it secure and avoid the need to having to design this uh fraud proof which is uh very difficult uh to implement and also to test uh so that's not the uh not the only benefit uh obviously as we kind of already talked about regarding stus validation uh the network is more decentralized over time and and because the validators uh they don't need to operate expensive hard where they don't need to store any of the state locally and uh the network become more decentralized because of the lower cost of operating of our it and there's also an interesting benefit as a result of this uh design change uh which is that uh we can actually try to hold the entire State uh in memory for execution uh so let's say because of this charting design uh we can't have many shards in the network and as a result uh it's possible to limit the state uh the state set the size of the state for each chart uh to something let's say 50 GB um and for trunk producer they can actually operate uh somewhat expensive machine let's say a machine with 64 GB of RAM uh which is actually not that expensive today and actually hold the entire state in memory during the execution and on the other hand the VOR themselves when they uh receive the city witness and do the execution uh the state is naturally held in memory uh so basically during the entire execution on different type of a the state is always being held in memory which means we can decrease the gas cost uh for storage related operations uh and make the super much faster so this is also a very future phasing design um because today in this design uh the broad the trunk producers would generate State witness uh and then they disseminate that to the validators the vators would verify them uh would verify the trunk execution using the Ste witness uh but there is a obviously non neble overhead created by communicating the stck witness to the validators and also validator having essentially to do the same execution as the chunk producers uh with their knowledge proofs uh we can actually uh replace that uh for example the chunk producers instead of generating uh St witness they can generate a zero knowledge proof uh which is much much much smaller on the order of uh sever hundred bytes or sever uh several kilobytes and the those uh zon TRS are communicated to the uh validators the validators on the other hand uh would verify the uh uh zge proof which is also much cheaper than uh doing a verification of Chunk execution today um so this is overall would make the system even more uh decentralized and uh um even more scalable because uh it's going to be way cheaper to operate aide uh under this assumption so this this uh design that I just talk about uh we already implemented um already almost done with implementation and near uh so uh if you're familiar with year you probably know that we had this nice Street Shing design uh back in 2019 and we recently updated design to include the serus validation change um and we're currently running a incentivized test net uh so if you're interested in learn more about that you can find it at uh n.org Stak War uh and also you can scan this QR code uh if you want to learn more about the technical details uh of the design uh so this would link to the updated white paper that talks about all the uh uh details of how how it works on a much deeper level uh and we're aiming to launch main net uh at the end of Q2 uh and hopefully that would bring a lot more scalability transaction throughput uh to the network and yeah that's it for my talk thanks