Sharding Implementation vs Research by Raul Jordan

Raoul is the co-founder of an organization called prismatic labs which has been doing absolutely massive work on aetherium research and development he's going to give you an update on the work that prismatic has been doing on sharding and Casper better known as Jasper hopefully that's a thing now as well yeah and I mean I just had been on the sidelines watching some of the work these guys have been doing over the past few months it's it's enormous they've they've done as much to push aetherium research forward as anyone in the community so great thank you very much and join me in welcoming Raoul [Applause] you have the clicker all right testing testing all good to go can I start ok awesome hey guys nice to meet you all my name is Raul I'm the co-founder of prismatic as we mentioned and here to talk to you guys about what's going on with a theorem 2.0 all the sh asper related work and what is to come so just a little bit about us so who is prismatic labs right we're a group of open source a theorem developers are just assembled organically towards the end of 2017 we're basically all you know just wondering the same question like hey there's a sharding spec out there nobody's working on it like why don't we just get together and build a repo and build a team and make it awesome so we just started we found each other all on like reddit threads on git er on github super cool how like things happen nowadays and then we were able to just start hacking away on what vitalik have put out there as an initial sharding faq so if you guys have been following the latest kind of research and maybe have seen this post going around on twitter and social media there's recently been some talks about how alley r1 development versus layer 2 development should progress throughout the years and you know the idea is that we've had protocol so just like you know HTTP we'd had SMTP we'd have these things that are public goods for the internet that have been around for decades and you know like they don't need extensive innovation extensive maintenance going into them they just work as expected they allow us to build more complex like more complex functionality on top of what we have as primitives of the modern Internet and these things you know are basically established and are there of course we can upgrade them there's HTTP 2 coming along there are better versions more secure versions of things but in general these things work and at the end of the day like when we're building applications on top of protocols we just want these things to work at a low level and be able to build from there without having to worry about like these constraints right so you know our job at prismatic labs working on like sh asper and all the latest stuff is to build a robust and scalable system that works is expected basically and then we want all your layer 2 folks that are working on interesting stuff to take the lead from there building out either like execution engines working on like other side chains like maybe like shard specific applications in the future which we're gonna go into get into in a bit but so what does it mean to work on a theorem 2.0 and what is the mental framework that we need to think about when we're going into this and the general idea is that working on this means that you're operating under constraints right from the very GetGo like when you're looking at the initial spec you see all these cool ideas about oh look like you know we're gonna have all these shards we're gonna have like super quadratic scalability we're gonna have amazing like crash our execution but at the end of the day you have to ask the hard questions like how is the current state of a theory I'm gonna migrate to a theorem 2.0 how is gas going to work in this system in particular like what is the role of the proof of work chain in this example and how can we just mesh all these things together it seems like a massive hodgepodge of interesting ideas but no no book yet written down that cohesively ties all these things together in a way that's easy to digest to digest and easy for developers to start working and implementing right so how did all this begin so all this began where the tell a kind of shorting FAQ that's been out there for a long time and basically it explains like hey how can we get to this like next stage of aetherium this like you know next this utopia topic vision of having like super quadratic scalability and like maintaining security right it's a really good overview if you guys haven't checked it out highly recommend this but that's essentially how we started you know there was a decently put together spec it seemed like it was thought out pretty well so you know as an implementation team how can we turn this into something tangible how can we make sure that we work around what the current system needs and build something that's actually useful and can be deployed to production in the future right and a prismatic lives we care about you know building something that eventually everyone in this community will use and that it's it's actually going to be an upgrade to the system rather than just like you know a toy implementation or just playing around with us so earlier this year everyone that was working on like sharding and the theorem 2.0 assembled in the sharding research gathering in taipei where we basically talked about all this stuff and you know i'd say that we probably came out with more questions at the end of the conferencing came in with mostly because there is so much to figure out it's so difficult for a single team to be able to take on the entire initiative every theorem 2.

0 all the way from like the consensus layer the protocol upgrades cross shard execution state migration security you know we need as many minds as possible working on the sort of stuff so how did the spec evolve you know I'm gonna go through a little bit of history here and then I'm gonna go into some of the current details of the spec and what we've done so far so initially the idea was that you know to maintain compatibility with the proof of work chain we could actually deploy sharding as a contract so the idea is that we would have this smart contract that we would deploy in the proof work chain that would manage the entire system of the shards so it is you would have like shards that exist us like these separate entities like kind of like as little side chains of sorts and essentially people called proposers would submit lobs of data basically merkel routes to this charting contract and because the transaction is mined on the proof of work block you basically get you know the proof of work probabilistic finality that you obtain right now of an ethereal transaction right so this is a really interesting meadow way of getting to where we want we're basically leveraging what we currently have to build up this even more complex system on top of it and that leads you to the question of ok how can we go from this into something like you know like a super quadratic system how can we deprecate the main chain how can we do cross charge transactions here right what about miners can they like try to front runners can I try to like sensor things can they mess things up in the sharding system and the answer to those questions are yes those are valid concerns you know if you try to leverage aetherium this way like you'll run into some issues additionally like casper and proof of stake was originally going to be done through a mechanism like this so we were going to deploy a Casper contract on the main chain that would manage the entire proof of stake system and do you know the whole checkpoint EPOC system where you would get finality on the main chain so what are the issues here as well you know like people are wondering am I going to have to pay gas costs for these like simple Casper transactions right I'm gonna have to pick a Scots for sharding transactions how does Casper and sharding inter that are the two contracts and they talk to each other right so we get into this huge mess and we start to wonder the natural question wait can Casper and sharding kind of come together if we're doing them is this kind of this you know this hybrid system where we have a contract deployed and then we manage things through here why don't we just kind of merge the two in some interesting way that avoids all these constraints of dealing with gas costs and like EVM compatibility right you know so as I mentioned before like if we try to do these complex systems as contracts on the main chain we run into the issue of gas we run into the issue that like whenever we want to upgrade the system we have to do a hard fork on the system and you know this can be manipulated at the whims of the minors so we're actually better off building etherium from the ground up and the good thing here is that we're not entirely starting from scratch in terms of how we're going to deploy this we already have a maintain that we can leverage for finality reasons and for security that we already have and from here came the idea of a random beacon sidechain I'm sure you guys have probably seen this around like you know around Twitter around latest III switch posts but what does this really mean right like what is the motivation for doing this as a side chain and like what is the idea of a random beacon right so as I mentioned before instead of having this contract deployed on the proof work chain we would instead create this side chain that's going to be a foolproof state chain like no BS it's going to be synchronized with a current probe work chain and it's going to be the heartbeat of the entire charting system right so we saw the the poor guys talking earlier in the panel about how sightings are so powerful and we believe the same we understand that we can actually can actually leverage what's already there in the main chain we can leverage the proof-of-work algorithm in the beginning to onboard people into this side chain and then we have basically a system we can start from scratch so this means we can define it we can define an entirely new chain for a theorem 2.0 that has full proof stake with cash for a 50 manages the entire sharding system has its own notion of like state execution and has its own notion of built-in pseudo randomness that we'll get into a bit later ideally here we have we have a we eliminate all those design constraints that existed before right we don't have to limit ourselves to being worried about what's gas gonna look like what's this what's gonna happen with this how can we do this without breaking backwards compatibility what we're saying screw it we're only gonna use the approved work chain for synchronization and for people to deposit stuff and join this network but if the room 2.0 is going to be its own new thing from scratch and that's I think really exciting gives us a whole realm of possibilities and here's this amazing diagram by she'll away from the etherion Research Foundation so the idea here is that you have this maintain that we currently have and kind of link to it in parallel you have a beacon chain that basically stores maintain approve work hashes I know this diagram was kind of crazy but you know it'll get a little bit simpler and from here we allow the ability for basically shard chains which are these you know these constructs that split the state into partitions to coordinate their data and be able to reach if you have the beacon chain be able to reach consensus on what canonical shard chain data should be right so philosophically one can think of this one can think of shard chains as a coordination device for essentially how a beacon chain is gonna reconcile them and determine what is canonical or not one way I like to think about sharding and the reason why we can leverage block chain in such a way is in terms of compressibility of information right so imagine that inside of a block instead of storing like the entire you know the entire entire set of transactions like storing like it's a certain transaction route storing like a State Route instead of that what if you could store a simple message that says like hey this is what happened on shard 5 these are the validators that like sign these blocks here are the merkel roots of the latest transactions and you can compress all of this into basically a short form message that summarizes the entire state of a single shard and a point in time right and this is a very small amount of information that would need to be put inside of a block so realistically you're able to compress all this information that's occurring and do so in a way that's secure while leveraging something like you know the finality of a proof stake system and the security of the main chain system so let's go step by step so like how does how does this room 2.0 how is it going to work how can you guys join how are we really going to leverage this main chain right so the first thing we're gonna do is we're gonna be deploying a contract called a validator registration contract on the main chain so it's a simple way where you guys can deposit 32 eath and what's going to happen then is that the beacon chain that's going to be constantly listening for logs on the main chain it's gonna be like hey look somebody registered and put in 32 eath so now we're gonna queue them up as a validator for the beacon chain cool so we can verify it we can verify you know verify you're a person using your public key and from there we basically cue you in so what happens is if the beacon teen has this notion of cycles and cycles are basically you know number of blocks a number of slots of where blocks can happen so once you reach a new cycle that's when you become an active validator and the beacon chain so you can do some interesting stuff like you can start actually like connecting to shards you can start packaging transactions and data into blocks and start proposing them and doing interesting things here so what's an important constraint here for those unfamiliar a big constraint that we haven't sharding regardless of the of the whole sharding manager contract idea is that we can't allow people to know what shards are gonna be assigned to in advance and by people I mean validators so if you're the person that's going to be proposing a block on a shard ideally you know we don't want you to know that hey you're gonna be part of shard five always you know that allows you to some interesting things like you can do some interesting attacks you can do cartel attacks you can do some interesting look-ahead situations here so you know when you have when you if you split basically a security of the etherium network into n partitions you're gonna get you're gonna get a certain certain decrease in security proportional to that and we want to prevent that that's why we do this pseudo random shuffling such that you only know you only you're only going to know very very little very few moments in advance what start you're gonna be assigned to and which one you're gonna have to be performing your you know your block proposals right so that's why we call this a random beacon chain it has it has a built in notion of pseudo randomness in it and we're leveraging interesting tools such as like random mechanisms which are a cool way to like generate randomness through a committee of people you know we're doing using BLS signatures for like signature aggregation and what's interesting here is that you know we have this enshrined notion of randomness that we're then able to use for aetherium 2.0 and that can be used inside of dabs that can be used inside of like the contracts that are gonna exist on this chain and it's in particular it's gonna be used to select validators right so say that a new cycle begins so say it's like you know it's like slot 60 64 and the beacon change so all the people that deposit at 32 ether are now gonna be selected and say oh you're gonna be on like shard 1 you're gonna be on like shark 5 and here's one you have to propose a block so you get a very limited time for him to do so right and here's what happens so what do validators do in a theorem 2.

0 so validators are committees that essentially like you know they they reconcile what happened inside of a shard and they produce a block for the beacon node and that block is gonna be signed by other validator known as the testers so for example you can see here that a validator is gonna package transactions into a beacon block they're gonna put in you know all the block they're gonna put the block hash if what happened in a shard and possibly other fields and they're gonna be reconciling ok who signed who signed this right one of the major bottlenecks that existed in like casper f of g and like basically all the proof stake systems came down to like signature verification and the bottlenecks like you know you're gonna have to verify so many signatures in such a short span of time and like how can we really fix that constraint so one of the things that we're leveraging is like something called BLS signatures for those that are unfamiliar it's basically a way of like doing signature aggregation into a single a single signature that then somebody can verify right and this is something that's being used by Definity and it's something you know it's it was created by damn Bonet I'm professor Lin and Xiao Han so here is an example of what I was just talking about right so you're gonna have all these people signing beacon blocks how can we make sure that like the verification of those signatures is efficient so that's why you know if you guys follow the research threads we talk a lot about blf signatures like signature aggregation and here's what it really means it basically means that like I have all these signatures from these validators I want to be able to aggregate them into something that like Bob up here is only gonna verify once and that's very effective that's very nice that allows us to stay within the constraint of like people being shuffled very frequently across shards and having the time between like block proposals on the beacon chain be very small right so you guys might be wondering okay like okay this is interesting this is cool but I still don't see how this really gets us to scalability and I'm gonna be talking about scalability and like three particular ways one of the ways is horizontal scaling so if you guys know about database sharding it's a basic basic idea that you can you know partitioned database into like and shards and automatically you get some for a linear factor of being able to scale like that what's interesting here aside from that is paralyze ability of transactions so in a theorem in particular I know it has to process like if you have ten transactions coming in and so the network you know you basically have to process all ten of them every single node and network is gonna have to process all ten of them in a certain order and what we can do with this is allow for a greater split allow for paralyze ability and allow people to not have to download the entire state of the network to be able to be in sync that's why it's really cool that if the room 2.0 one of the major goals is that you're gonna be able to run it on your smart phone you're gonna be able to run a validator note or a beat or even a full beacon note on your smart phone an interesting fact like the beacon state is actually a very small in size relative to like what the a theorem state would be so that's going to be something really interesting and that's how we think about scalability and what the constraints are so we're very excited about this it's not very clear so how do we ensure that data is kept around right so you know I mentioned before that validator need to be basically assigned to shards and they get reshuffled every so often so for example okay I got assigned to shard five but boom now I have to switch and now I'm like in shard six like what incentive do I have to not just like you know wipe my wipe my um my shard chain data folder and like start a new one from scratch right well the way we do this is we unfortunately called a proof of custody which is basically a crypto economic scheme where we enforce these validators to maintain the shard data that they download for a certain amount of time that's called a proof custody challenge period and in that time they can be challenged by other validators they can be like hey hey you're supposed to be holding on for a shark - shark - right and like you know if you respond like no I don't have anything then obviously you're gonna get deposits laughs in some way or another so it's very very counterproductive to not do this and that's still something that has not been well defined in the spec so there's some interesting constraints here like how can we keep the proof custody challenge period like small enough such that people don't have to store that much data but like also not big enough such that you know like so we have the other problem basically we don't want we don't want the alligators to be storing so much information so what does the research not cover and that's something that we've been like really you know dealing with a prismatic lab since we started and the most interesting things are like what's gonna happen to existing state what's gonna happen to crush our transactions and obviously I get this question a lot and you know the reason the way I think about it is that look like crush our transactions or something that come at a higher level than the consensus level of the beacon chain and that's something that you're gonna be able to like reason more about once you have a better idea of how the beacon chain is going to function and how you're gonna be able to like coordinator cross chards right I'll be getting to questions in a bit yeah so there's no way any team could do this alone you know aside from us there's the other teams the theorem research team is working on a Python implementation and there are other teams as well like working in rust Java and us of course so there at theorem 2.0 bi-weekly implementers calls that I highly recommend you guys to like check out or at least check out the notes and you know add prismatic labs we're building a production ready client so what does this mean basically you know we're choosing a battle-tested like build system that's in production at Google we're using a top notch like maintain p2p library from protocol labs which is I which is like used by IP FS which is let Peter P and we're using the best-in-class RPC framework which is Google's G RPC so we also want to maintain a very welcoming and open source contribution ethos so I want to make sure that like like like knowledge for theorem 2.0 and scalability is not restricted to like two people on our team that basically newcomers that you that basically want to help they have all the tools needed to be able to contribute as much as we do and get excited about this as much as we do right so what have we done so far and we'll have a lot more updates for you guys by Devcon so get excited so we basically have a right now a full be contained that synchronizes with approve or a chain node we have beacon node p2p via like local DNS with gossip sub we're able to interact between a validator node in a beacon node we have all the validator shuffling and registration contract functionality done and here's some like you know some code running so yeah so we have a beacon block simulator we implemented all the Casper FFG functionality and we have state transitions going on and we also have an end-to-end system basically for advancing the beacon chain from Genesis so the idea is that you know in a few weeks or so like - things mature more it'll be easier for you guys to be able to spin up a beacon node get a validator node running have it you know generating some test blocks and advancing the chain from Genesis just like you would a death note in dev mode right now so yeah so this is our team if you guys want to get in touch check out a repo all that stuff is there just like a prismatic labs we have a biweekly update that goes out on medium so I highly recommend yeah thank you guys I went a little bit over time [Applause] Thank You Raoul that was awesome yeah I don't think we have time for questions unfortunately but if people want to find you and communicate with you what's the best channel to do that um yeah I'll be walking around so I want to meet all of you like if I can and yeah get in touch via you know Twitter contact us on prismatic labs website there are lots of ways but yeah sorry about that them didn't leave time for questions