"Sharding for Scalability & Development Experience" - Illia Polosukhin

on stage Ilia from near protocol he has just um finished um the the near protocol guys have just finished their hack the rainbow hackathon that was a huge success um but Ilia is going to chat to us about sharding for scalability and development experience Ilia can you hear us yeah thanks Simona for introduction uh well it's great to be here and welcome everyone I hope your hacking has been starting well um yeah let me let me kind of talk through some of the sharting and uh I'm sure with all the news around these two there's a lot of questions uh so I'll try to kind of cover some of the topics and uh surface some of interesting questions but if you have uh anything any other questions that I didn't cover please uh ping them in the chat and and we'll we'll respond at the end so first I wanted to just mention like uh near itself so near is a pro stake layer one that's specifically build with experience in mind so which covers user experience and developer experience and scaling if you think of it is part of this uh kind of experience because as your users as your uh like user based growth as your application gets more usage you should not be dealing with you know scaling your application and also your user should not be dealing with you know High fees and figuring out you know what to do with with all that right like this stuff should be completely transparent and just work and so kind of the general idea right is to build the experience that matches kind of the web 2 right the existing internet and in such a way that you know you can bring in non- crypto people and they can just use applications and it's been pretty important to obviously interoperate with ethereum because this is where kind of most of the existing developers and users are uh but we also want to on board lots of developers uh who are coming new so let's talk about scalability so uh let's start with how you cannot scale right and so there's been a lot of kind of uh new consensus algorithms where people would be you know providing an idea how to scale the uh how to scale the blockchains with you know some amazing consensus algorithm and you know being able to do stuff stuff that is uh faster there's been a lot of also like you know changing from blocks to trees from to dags to some data structures which are you know presumably allowed to to do something faster or switch into different Pro proof of right um and to be clear obviously proof of work it has inherent like limitation and that limitation actually is in the fact that you first need to organize a block and then you need to mine it and that mining time is actually adds on top of your uh kind of execution time and then then actually that is possible to pipeline So in theory it's possible to like optimize that and actually uh have throughput faster uh with Pro work but it's still not the bottleneck the actual bottleneck is not data structures it's not consensus it's not a proof of it's actually just executing and the reality is we we run we want to run a smart contract platform right we want to run generic code code and this code needs time to execute and so this is actually just syncing the uh EV ethereum blockchain with fully download it right snapshot of all the blocks and so there's no network there's no consensus there's no you know data structures it's just purely evm execution and as you see there's a limit how much gas you can push through in in a second and so evm itself uh with on like a kind of somewhat default Hardware right maxes out at like 200 300 transactions per second of like normal smart contracts this is like real real smart contracts on Main net and uh this is the bottleneck right like it doesn't matter if you switch to you know proof of something right like proof of authority it will be the same way uh it doesn't matter if you have like you know the magical pipelining and stuff like this so the only other option like the way to to scale right is either significantly optimize the transaction processing so so this way is includes you know changing from evm to another uh kind of execution environment right so web assembly is faster but it has its own challenges there is uh kind of berkly Network that Berkeley uh system that salana for example uses which is faster you can put execution on for example gpus and stuff like this the problem is all of that really works if you have a homogeneous load because uh if you only need to do payments for example you can scale you know pretty dramatically because you know you can parallelize that you can have like lots of processing power uh like on one machine going going really quickly and doing things in parallel but at the end actually one of core limitations it's not an interpreter it's actually hard drives right just being able to randomly access hard drives is hard so and again like you can put more hard drives but like there's limit and so the other option is just split the work right I mean that's an option that the whole internet is using right Google is not running on one machine Facebook is not running on one machine right there's lots of machines running in parallel processing things in parallel and so this is what what's called sharding right uh I mean this is not a New Concept right it's been around for you probably two decades if not more and the idea like we need we need to scale up like we cannot have Network pretty much everybody in the network doing exactly the same work they need to paralyze the problem is this is very hard and this is hard because there's a lot of kind of technical challenges on how to make sure that everybody's doing the right thing how to make sure that the chain is correct when you don't actually run all of the things so uh I'll I'll just cover kind of like a little bit of of a difference but uh between kind of default sharting approach and also what we've been working on but so the general idea right is you paralyze the work which means you kind of need to do a bunch of stuff in parallel which the kind of naive approach or like the approach everybody starts to take including us is to paralyze the chains itself run multiple chains in parallel and then in some way to kind of link them together and and do this um like use this as a way to communicate between them and so this is approach that you know originally uh like if2 took it's approach originally we took it's approach that polkadot implemented it's approach that kind of if you think of Cosmos as a sharded chain which it's not and it's a little bit different but like in general the concept is the same is by scaling by a lot of chains and but like one of the core issues with this uh and we will cover like a lot of other uh interesting interesting issues it's just like it's inherently complex and has a lot of kind of repercussions and so the we taking is really uh paralyzing the blocks and I also draw some parallels with with other approaches as well so what are the kind of technical challenges with sharding well one of the big challenges is if you think of any proof of stake system uh usually if you have a onethird malicious and network split you can actually take over the network right uh so either stall it if you have one3 plus one or of stake or if you have a uh like Network control you can actually do some uh like Fork of the network and one sort of Stak you know usually like you can you know you can incentivize growing the stake uh it's still sadly in current proof stake systems we still have a pretty big concentration but uh hopefully over time this will grow um and uh you also need to kind of track like you're you're pretty much saying that because we assume that there's less than onethird of uh malicious actors been on the network then uh pretty much the whole like by induction right the whole chain is valid because like every other block is is every next set of validators is selected by the validators that were correct um and you need to actually like um have some some uh intersection between when you when the valat is are rotating because they need to actually download the state so that's how the this works uh the sorry the problem is that if you if you think when when we doing charting right what happens is that in each chart you have uh you need an assumption of f malicious and so this leads to a problem that if you sample uh kind of for a long term right you just sampled it and just run with that uh this valers can get corrupted right uh and and sampling you know like let's say you have you know 10 sharts 100 charts sampling into that will uh will pretty much lead to like probability if only if less than one third like if oneir or like even 20% is malicious like sampling into one of the shards can be higher but also what's more important is that uh people can get adaptively corrupted right over time so you need to actually rotate validators across across the stack and and make sure that uh they are pretty much cannot um like cannot know which Shard they validating too much in advance uh so that's where like this thing come in is like you pretty much rotate validators uh but you need to give them some heads up to to download the state of the next shart because compared to a normal uh like what single chain right where all the nodes have the full state of the whole network now you only have a partial State and if you are responsible for validating next chart you need to go and download the state of that chart before you are able to validate otherwise you'll be lagging and uh and I'll mention data availability in a second so the other problem is what happens if somebody actually did attack right the let's say there is a more than oneir like two3 of uh malicious validators in one of the charts because like in general the numbers are smaller of stake and uh the problem is that Z A Shard cannot detect us uh very easily right it's not validating that Shard it's not actually validating State transition it only kind of keeps track of that chart so uh and like the kind of the attack would be to send let's say inv valid State like for example say well all the money are mine on this Shard and then send them to another Shard to exchange and exit right and so maybe like Network like everybody will detect this but it will be too late and people already stole the money uh so the kind of the general approach to this is called fisherman and this is where uh somebody can pretty much tell the other Shar that hey something is wrong uh in this chart and they provide a fraud proof right and so this is similar to like what we've just heard in kind of Plasma's approach as well where if something is wrong in a plasma right then uh somebody can provide a fraud proof to the ethereum and and exit pretty much from this and so the the the problem here is and similar to plasma is data availability right so you need to somebody needs to actually have a uh fraud proof generated in the first place to be able to uh to prove that something went wrong right and so this malicious actors did not produce a block right if they did not post a block to the network right then this uh let's say honest actor in that chart has no way to really uh create a fraud proof and proof it to the other chain the other side is obviously finality right so like if you send a cross transaction and there's a fisherman right the fisherman needs some time right so in case of plasma it's seven days big part of it obviously is because of gas issues and you know transaction sensoring and bu of other stuff but uh this change period needs to exist I mean it can be shorter but like this is the finality of this cross shut transaction right we cannot execute it until we make sure that there's no fra proof coming in and there's no like no no like there's no way to not prove there's not going to be a fra proof right so there's no way to like speed it up and so this creates a lot of issues on like just kind of creating a fast cross transactions between chains uh the other thing is like if if you for example do speed up then you need to actually like roll back a bunch of stuff right you need to Fork off like if if you allow transaction to to go in and you just kind of revert it you need to revert like kind of cascading effect of all the shards that get affected and uh this also can be very complicated and then the data availability part is what I mentioned right if validators did not post invalid block right there's no way to prove that's that uh that there is an invalid State transition and like next validators cannot do anything about it um so so important is obviously to have a mechanism to provide data availability to network and so one of the main solutions that pretty much everybody uses uh so it's Us bulot and other folks um is to pretty much use eraser codes and so this is idea that when uh when somebody produces a block uh or in our case a part of the block they need to send out parts right so they need to send out eraser coded parts of this block uh to other participants and only if those other participants saw this Parts they will include this block as valid like they don't need to validate it they don't need to uh make sure the state transition is correct they just need to know that there's data availability so uh so what you do you VI in into consensus right you actually include this into consensus that you can only accept uh those things that you saw parts of and if you have enough of these parts so enough people are tested to having these parts then it means that actually from this people you will be able to rec reconstruct this uh this blocket the that that was transmitted right so even the original Val original block producer or in our case chunk producers disappear uh you'll be able to reconstruct just from this Parts uh the full information and so just just for context raure coded means that uh it's a special encoding that even if you only have um like in this case you have six parts right to split the data into six parts and any two parts is enough to reconstruct the original data right so it's kind of a 3X duplication of information that's like any of the two parts is sufficient and so this mechan Mech ism is kind of at the core of like our approach and P as well uses this uh to pretty much U make sure the data is available across across the network and so I mentioned before right so we transition from this idea of having multiple chains running in parallel to having actually one chain and sharding blocks right so and so this actually allowed us to do a lot of things including because we have now one blockchain uh this allows to really uh simp simpy the whole challenging things right because if there is a issue that H have happened at some block right we can just roll back the whole chain within like that would uh that contains all the malicious T right so we don't need to like find you know a bunch of chains and figure out where where they went wrong and roll roll them back you just have this kind of a one finality for all of them and so this this provides kind of a better experience it's also just easier to implement like that that been one of Thee core things because obviously like this things this stuff is very complicated and it's very hard to test so uh the other thing is obviously it provides a better cross transaction uh uh functionality and so we can move into this I think like uh few things important to mention here which um like if you think of it what rollups are and what plasmas are in a way is kind of a specific versions of uh like of this as well right it's it's all a way to run some kind of computation outside and then settle into the main chain where you main chain is providing data availability and it's providing the kind of finality security right the challenging system right so in in in this case like in in our case we're using kind of the our like full chain to to do this but you can imagine this like full chain would be ethereum and then all the shards are just running as a parallel chains and that's kind of what like plasmas and optimistic rollups are uh and then like they have all the same problems that I mentioned right they have a sday challenging periods to to do this if it's optimistic rollups for ZK rollups right they can prove the full State transition so they don't have this problem and uh this is kind of how it all links right in in reality all of this methods really just uh like different ways of handling these problems that I that I mentioned now the important piece is how to communicate between right so if you have a multichain environment or multi- Shar environment or like multi rollup environment right uh it's actually becomes really complicated how to how to uh connect them together right and specifically there's a lot of kind of issues around rollbacks right so if z a chain rolled back what do you do right uh like you know if ethereum rolled back if you are optimistic rollup you actually need to roll back as well right because you're tracking ethereum uh if you're a separate chain and there's a roll back uh like what do you do that right and there's gas cost issues right it's because like if you trying to send a message from one chain to another chain if the gas is different there it's in a different token right uh like who's going to pay for it how you're going to calculate when you're sending in transactions that you prepaid the right amounts and stuff like this right you now I need to attach new like an as a token or something else um the critical State Recovery is is is around rollups is is around uh roll backs right if something happened right if if a rollup producer stopped right uh what do you do like like it's a mass exit it's a uh kind of you know like you roll back some stuff you prove things you make sure that you have all the data availability but like what happens with all the transactions that are in Flight right what happens with all the cross chart like you were calling from here to here like what happened with them and it's important to figure out like will it like what's the order of execution that will happen on the other side if that happens right if like optimistic rollup like producer stopped everybody's Max Max exting but you actually had a proper exits as well and they'll not get executed at the proper time because they all like waiting for seven day it's like what what's going to be happening in in all of these cases um and then important part is compatibility right if you have like two rollups or two shards or two chains right if uh the communication between them is complicated it has gas it has all all this like s-day waiting periods uh or even like a minute waiting periods like composing contracts composing app become super complicated and especially if you want to also receive a call back right and and hear back about what happened on zarp so with near like with near approach right we have pretty much been focusing on fixing all those issues and making the experience really really uh like really smooth on how this works and uh think of it right because we have one blockchain like from a developer perspective you actually don't need to think about uh kind of shards and sharding in general like if you go to our Explorer right now you actually will not see we running one Shard we have another test network with four shards but you will not see any difference because the sharding is completely hidden from the developer and the kind of main thing that we we uh propagating is that the all the contracts working asynchronously so the communication between them will not happen within one Atomic transaction it will happen within next block right so you pretty much creating message and you're sending it to another contract and another contract will execute it in the next block and uh this is similar to like message passing in erlang or any other like similar language and first of all it removes any kind of reeny problems uh second if you know if there's any roll backs this can be handled cleanly because all those receipts are kind of staying in the block and then like you roll back you restart it and you have all the receipts just coming in and you need to process them uh they are mandatory to process before any other transactions coming in that are like all the receipts uh like this messages pass from one contract to another uh kind of coming in in the beginning of the of the next block um it solves gas costs because we kind of uh pretty much like one of the core ideas is uh dynamically balancing contracts between shards and so this would allow to kind of even out the gas cost over time so it will not be like immediate uh like you know within within a like few hours you may have spikes but like over long term right if there is some contracts that are more used than others it will pretty much even the amount and split them between different charts evenly um maintaining kind of the balance um but importantly it kind of has this compatibility property right because now you can realize that contract will be delivered right and you'll get a call back and so uh and like the only thing that you need to handle is the fact that it's not Atomic right so so you do need to handle kind of state reversals somewhat manually uh there's like few tool tools to do that on developer side but you do need to kind of handle like a a failure of the execution of other contract the benefit is now you can handle the failure of execution of Z contracts which you actually couldn't on ethereum so you can actually call contract see if it fails or not and execute some logic based on that uh so think of it try catch and yeah so this Callis kind of allows you to like attach uh obviously near as as a resource it allows you to attach information and receive it so here's an example how how this works uh so you pretty much this is in Rust uh for web assembly contracts you can create you can call is white listed based on some interface uh you provided uh the kind of contract that you want to call uh some arguments uh no deposit in this case and some of gas and then you have a way to attach a callback to this execution uh on on one of the methods on the contract itself so pretty much will call it it will execute something thing and then return and actually that contract can call something else and then process something get call back get get call back right so you can actually have like pretty long chain of of of calls uh that processing uh now so this is actually like this is very useful you can actually have like a very interesting like multi- Shard contracts now built on across this you can also Shard your existing contract so for the users like for example for a token we actually have some proposals for sharded tokens because tokens actually don't need to have all the accounts in one Shard right you can actually Shard them uh as well and there's like few other applications that kind of need the scale of multiple shards so going Beyond even one uh one Shard but uh there's also a lot of like the the like the defi specifically right has some issues with with async calls and like this this will be very clear because one of the kind of benefits of of Defi and like I usually call defi is just a like a prog programmable exchange right so if you think of it you deposit money into exchange right and it gives you a bunch of tools I'm talking about like NASDAQ or you know one of those exchanges they have a bunch of tools already pre-built on on that exchange you can trade you can do stuff and within within that exchange right everything is atomic everything very fast nobody can front R you hopefully and and then you can go to another exchange right so so and so what etherum with def5 provides in a way is ability to deploy new contracts new code into this one kind of uh one exchange one place and so this is provided by like ability to execute kind of atomic transactions Atomic across multiple calls so it it does become complicated to do that for async calls and one of the main issues is that um because if you think of it like if you have two contracts that have a sync and you have some kind of logic flying between them uh people can try to front on you to go after this contract earlier on right and so there's ways to mitigate that in this kind of multi isync environment but obviously this like as as this system grows right as you have more and more pieces working together this becomes more and more complicated so there is a value in sync calls and so we are actually uh just launching the evm what we call a volts so in a way it's like a one Shard of a zvm and I mean it can obviously scale with you know but goes up to one chart and within that evm uh we have syn synchronous calls right so within that evm it it behaves like ethereum so you can think of it just launching ethereum shards inside near actually originally it was done as a uh just a smart contract so we literally compiled the VM code and just launched it as a smart contract on near and had like original initial testing but now we pre-compiled it and kind of speed up so that gives us like you know what whatever the raw performance you can get uh from the from evm you get that the interesting thing is you can have many of them so you can have multiple EVMS in parallel running and then have async calls between them so you can think of it like can have multiple exchanges or some other contracts maybe that are not exchanges that dealing with like you know in game items or whatever they don't care about like specific defi stuff uh and so you can still async communicate with them within one block so you have like all the same functionality and you can communicate with all other contracts and in a way this actually where as like you can call it realistic rollups right because what real what optimistic rollups is is you execute something off chain and then you post it on chain as a data and then you don't execute it but then you wait for anybody like opt like pretty much optimistically it's there but then somebody needs to wants to challenge you need to wait for period here you actually execute this state transition of evm right away and it's like valid right away right so like I call it realistic rollups I mean even even better was if if there's evm that's running like offchain and that just post post all the data and gets valed onchain but because we have one second blocks you don't even need to do that it pretty much get the same performance so like each evm volt will be like as fast as like optimism or any other uh kind of evm engine running as a separate thing but it will work within this whole uh environment and this like kind of going forward is some ideas how to scale this up but uh this is kind of future work uh so before kind of ending I have one one last thing I want to mention so someone mentioned about our rainbow hackathon and so this idea is like in a way um like from perspective of of etherium near is you know is kind of a multi sharted environment which you can run contracts and with EVMS it's like many EVMS and there's no kind of like with this bridge we actually connecting those two environments and from etherum side evm is from ethereum side near is side chain right with like many side chains connected together with evm and with web assembly from near side evm is ethereum is just another Shard right because you can actually call into it and get responses as callbacks and kind of have all the same uh functionality just with a little bit of lag on on the finality and all timing and so this is pretty much facilitated by like a light client smart contracts on both sides uh that are providing you pretty much this trustless decentralized way of connecting two chains and and uh and it's done in a very generic way where you have like multiple uh like levels of smart contracts that pretty much provide connectivity and as developer you can link in any way and prove any fact about Zaza chain uh just to finish up uh we like one of the main things we've been do we've been doing is really focusing on education in the space so we actually have two series one on YouTube and one as a podcast uh one on YouTube is called whiteboard series and it's about um kind of talking about protocols and talking about kind of how they work on whiteboard in in depth it has you know eum 2 Cosmos polad do optimism Matic and and I think there's like over 40 protocols now and then the uh the open web Collective is building the open web podcast is a podcast about entrepreneurs and Builders and and investors actually coming together and sharing sharing information sharing kind of insights on how to build businesses and how to build uh in this open web so keep building join us at near. chat I mean if if you are interested in kind of expanding your business we have a protocol agnostic open web Collective which is really a incubator and accelerator for uh for companies in open VB and if you just want to learn more about near there's open chat and I'm open for questions awesome thank you so much Ilia I think my favorite piece was realistic rollups versus optimistic