EthCC 2: Rob Habermeier

[Applause] so we are now you're gonna hear our last talk before I be before the lunch break and this is gonna be from Rob Amaya and he's gonna be giving a talk on substrate and specifically what he's been working on lately so Rob's the the co-founder of of poccadot so he's he he does a lot of the implementation of like the grandpa consensus algorithm which some people presented here which is the novel consensus algorithm for polka dot and then also making that consensus algorithm into a sub substrate runtime module so that anyone who's building a chain can use it so he's gonna be talking about some of that work and a bunch of other stuff that I don't even know about yet so give it up for Rob [Applause] thanks jack hello everyone is there a clicker yes here it is so yeah I've been doing a lot of the implementation stuff for polka dot and that spans quite a range of projects at this point so there's the grandpa consensus implementation I won't be talking about that today but on the seventh on Thursday I have a big talk where I'm going to dive super deep into all the consensus stuff and polka dot but that's that's its own talk so what this one is about is about substrate we we at technologies we we built this framework called substrate it's for building block chains basically making that whole process easier this talk is focused on sort of an extension to substrate called cumulus it's some libraries for writing runtimes and integrating with polka dot basically to write para chains with substrate so polka dot first of all I guess I'll start with a quick introduction to what polka dot is so polka dot is a it's a network of heterogeneous chains you have chains that do different specialized things that are all secured under one consensus mechanism by the same group of validators they can pass messages amongst themselves and it's governing Bowl it's extensible and it aims to be scalable as well so one of the core philosophies that we've had is this idea that specialization breeds optimization the idea is that if you have two meter for very fine-grained units of computation things like add a number multiply a number shift left shift right then you have to stop after each of these small operations and subtract a gas fee essentially so this gets very hard if you've tried to do things like implement complex cryptographic primitives or or just particularly complex calls so if you have a rough idea of what your chain needs to do things like for example file storage or some specific kind of zero knowledge proving you can package that up into a much coarser grained operation and only bill fees for each of those and that's basically what I mean by specialization breeds optimization when you specialize into a specific type of transaction you can optimize the implementation and as a result do more computation so what a blockchain is composed of I like to think about this this is not 100% accurate because there is some cross feed between them but I like to think of it as a separation between the consensus algorithm so some kind of process that's providing security on the blockchain there are some blocks being created it doesn't really matter what's in them and there's consensus over that so everybody eventually can come to an agreement which is secured economically that things are finalized so that's what consensus is and then there's a state machine so state machine being what do those transactions do what's actually inside the block what kind of computation is happening could be UT exo's it could be EVM it could be wise them it doesn't really matter this is what we package up into that state machine abstraction so what we try to do with polka-dot is sort of separate those abstractions basically when you're building a blockchain if you have an idea for a blockchain most likely the main thing that you're trying to do is you've come up with a unique state machine some kind of unique form of transaction that's actually the novel part census is often not innovated on and speaking as somebody who's actually implemented a number of BFG concerns it's actually really hard and it's a lot of development work so the fact that you can ignore building consensus algorithms you know the security is provided from another level that's very very valuable it means that you can spend all your time focusing on building that state machine so a 10,000 foot view of the way that this works is we have something called the relay chain at the top level so the relay chain is the main chain of poccadot where you have validators and staking and governance going on and this is where para chains are registered so era chains are essentially just pieces of web assembly code I'm gonna get into that in a moment which tell you how to validate the block so the blocks from polka dots perspective are just raw bytes and these para chains are distinct specialized heterogeneous Paris tape machines in one form of para chain which I'm not going to talk about in this talk because they're sort of a whole different breed is bridges so what I'm going to talk about is what you would call maybe more native pair of chains pair of chains that have polka dot do all of their consensus and computation for them when a bridge is is something distinct where you actually have another chain which has its own consensus process does its own computational work and that those proofs of finality of that other chain are bridged over to polka dot and vice versa so that's that's sort of a different matter so I won't be talking about that today so the anatomy of a pair of chain let's go into basically what a pair of chain is but this talk I'd like to aim more at developers so this is not just a high-level view of okay this is what the abstraction is it's really more meant to be this is what the abstraction is but also how you apply it to your own work when you're writing that code so the the biggest part is this validation function that I mentioned before it's a piece of web assembly code validators don't know any pair of chain specific logic beyond what is encoded in this validation function so that's web assembly code that a signature that looks like this that's sort of pseudo rust but I'll walk through it so we have this function validate we have the last header from polka dots perspective that's just some bytes right so that ampersand brackets you ate that just means some bytes we have some information about the state of the polka dot relay chain so that would be maybe the last block header that's in polka dot the last polka dot block head or not the pair of chain block header and the incoming messages to that pair of chain so this is a state that's actually changing for every relay chain block this does mean that pair of chain blocks are localized to relay chain blocks so if you would submit a candidate para chain candidate and it was not included you would not be able to submit that exact same pair of chain candidate to the next relay chain block when that consensus process happens so it's sort of a one-off thing either it gets in or it doesn't and then the last part is this block data so this is meant to be it's something like a proof of validity it's not just the block body it's not just a transactions it's something more than that and I'll revisit that once I've set the ground for an example the other half of writing a pair of chain is to write the collator node so writing the validation function is actually relatively easy because it's just a piece of deterministic code pure function it's it takes some inputs and it gives an output either it's yes here's the new block header or no that out that input was garbage so that's pretty easy to write if you have the right tools the harder part is maybe writing the collator note because you have to do things like gossip transactions you have to come up what a call later node is supposed to do is come up with the inputs to that validation function the block data specifically such that that validation function returns true so it has to have the logic for actually handling those messages that are incoming to the parrot chain it has to have that logic for actually propagating pair of chain blocks to other nodes in that pair of chains network now you don't necessarily need those things because as it happens pair of chains don't actually have to be blockchains but within substrate they are like you can write specialized pair of chains that actually aren't block chains at all and then this sort of doesn't really apply like for example if you would have a federated pair of chain where it's just actually here's just Bank of America issuing blocks to polka dot and they have their own private thing attached to it they wouldn't need to go through all this hassle but if you're writing a decentralized pair of chain using substrate or some other framework right in the call later node is actually quite difficult so the remainder of this talk is really going to be going into how do we make that difficult task of writing a call later node much easier so pair of chains are simple abstraction but like all block chains are pretty complicated to implement so I put together like a very short list of all some of the things that you have to do when you're writing a block chain so consensus algorithm database are pcs that sort of thing a lot of optimization goes in there this is just a very short list that I made for you guys we basically have done that before at parity we've we've done that a few times we wrote in the theorem client which I'm sure many of you have at least tried out we wrote Bitcoin client we're writing the Z cash one right now we've written the polka-dot node as well so basically Donald this before and we figured well okay we've done it all before we know how all that works why don't we just package that up into a set of tools and we've been talking about that probably for the last six or eight months more intensely but we started work on it about a year ago and that was called parity substrate so it's just a set of tools for writing block chains but critically the block chains that we've supported thus far have been what I would call sovereign chains so there's a framework for pluggable consensus and things like that but it's it's been as of yet more focused for writing chains that are free-standing that provide their own consensus and finality and aren't networked with other chains as part of their first-class validation so among other things substrate does provide this framework for pluggable consensus and some I guess what I would call composable runtime modules so when I talked about the state transition but or the way that you do that in substrate is to write what I would what we call a runtime so runtime is essentially made of specific components that define transaction types and call types and ways that you can interact with the chain and these are actually just modules that you could write yourself or put together from repositories that others have published on github there's a set of modules that we've put together called the SR ml the substrate runtime module library but of course you can write your own as well so typically a chain that you write with with substrate looks something like pre-built components for things like balances and transfers perhaps staking and governance combined with those special bits that make your state transition novel whatever your research team has come up with that nobody else has so what cumulus is is an instantiation of that pluggable consensus framework to provide polka-dot consensus so it's an instantiation of pluggable consensus that follows the relay chain it knows how to actually run sorry about that it knows how to run another blockchain node internally which is typically a light client and follow the relay chain and the other thing that cumulus provides is essentially a way of extending without substantial modifications actually with quite minimal minimal modifications extending the substrate runtime you've already written into a polka dot validation function compatible run time because typical block authorship and typical block to ship have some key differences to the validation of polka dot parrot chains I feel like this clicker is not working I'm going old-school so cumulus is meant to turn and already written substrate chain that's sort of our idea right because obviously we've put substrate out there and a lot of people have been using it already have been building on top of it so it'd be kind of crazy if we would just put out these new libraries okay you have to rewrite everything everything you did before is gone so we're not doing that instead essentially you just add an extra module and extra piece of functionality to your runtime and then it can become come with the polka dot relay chain so a node using cumulus as I said before embeds a polka dot light client this is so that it can follow the consensus and finality of the relay chain so that's sort of what you would have to do I guess if you think about any sharded or partition blockchain system is that when you have these root chains from which you're drawing security you have to essentially run one one extra or maybe multiple extra nodes so that you can actually process that consensus so nothing I say after this point applies to bridge chains if you would write for example a Bitcoin or aetherium bridge it doesn't really fall into these categories this is strictly for writing native pair of chains right so how do para chains differ from sovereign chain substrate I'm sure you guys are asking this question because I sort of led up to it so the part one is block authorship so para chains draw their finality and consensus from the relay chain whereas sovereign chains are providing their own finality and consensus so this can be done in a number of ways for a sovereign chain you can have agreement over every single block I love Pete pbft you can have something like albatross with rato which was just mentioning where you have more progressive consensus you can have grandpa which we've designed and implemented which is sort of a chain based consensus system where you're voting on chains in a pbft style round and a common denominator is selected but in in parrot in consensus you really only have one the option which is to follow whatever the relay chain is doing the relay chain in fact does use grandpa and block authorship in the para chain space depends on information derived from the religion for instance whatever the block hash of the best relay chain block was whatever the message is coming from other pair of chains are whereas block authorship on a sovereign chain is completely isolated so you don't have to worry about this stuff and these two factors actually add quite a substantial amount of computation so revisiting this function right so we have this validate function that we need to come up with a way that's what block authorship is in the context of a parent chain we need to come up with block data that fulfills the requirements of this function such that this function returns okay with the new header so pokets block data is just meant to be some kind of proof it's supposed to maybe include transactions some subset of the state it could be a zero knowledge proof the verizon validation function doesn't really care but as an example this is this is how substrate structures things for now so we have the state organized into a tree there's a route stored in the header it's just as aetherium doesn't so I mean for what you're really doing is transforming a light client into a deterministic small piece of code that's compiled down to web assembly so when you would write an aetherium light client you would do the same thing if you want to prove a transactions execution on a theorem you would get the header where that block was executed you would check what that state root is and then you would request all of the tree no it's necessary to execute that transaction the tree nodes encode different pieces of account State for example if I want to read account X I go down into the leaf where the key is X and I read out the account state and then I can start to work with that so substrate has a similar model for state although it's it's not account based because it's a bit more abstract and than that you could write something that's account based but the way that it actually works is that it's just key value based so you arbitrary keys which are bytes mapping to arbitrary values which are also bytes so let's say if we would have one transaction the yellow transaction and it needs these tree notes to execute and then we would have this blue transaction which needs these other tree notes to execute what we would want to pass to this validation function is just these tree notes so I mean in practice this tree is humongous right the tree node sizes all in all add up to gigabytes and gigabytes and gigabytes and we have huge problems with that with state bloat people are talking about storage rent and ways to clear this out and how to how to make it not so massive so of course you want to omit all of those parts which you aren't using so what a block data proof for a typical substrate chain would look like is just these this a subset of all tree nodes which you need to execute the block transactions so what cumulus collator does is exactly that so this is essentially the pre-built binary which is generic over a substrate chain that's been tweaked with cumulus runtime so that's a binary that you can deploy which just knows how to build has invalidation functions and it knows how to build that block data that satisfies them and the pipeline for doing that is something like this so we're essentially combining two big pieces of information so one of those pieces of information is the state of the relay chain right so we're watching the relay chain we're seeing what's the latest head of the relay chain according to our fortress rule what messages are in coming from other pair of chains we're fetching those from our other peers we have that data on the other side we have the transactions data for our pair of chain so we have some kind of queueing gossiping collation happening where we're observing which transactions are coming in from our other peers on the para chains sub network so we want to combine these two pieces of information come up with all of the state tree nodes which are necessary to execute those incoming messages which it's up to the pair of chain really to decide how those are executed they could be perceived as just regular transactions they could be perceived as some other kinds of operations entirely so we need to figure out which state tree notes are necessary to execute those incoming messages and we need to know how to execute those incoming transactions so the way that we do this in substrate is we have this special polka dot inherent data that's passed to the substrate runtime so we we execute this special inherent transaction that's not actually authored by anyone it's a special polka dot inherent transaction which contains the messages and some other data about polka dot and crucially this data is emitted from the block that we send to the block data that we send to the validation function because it can be read arrived from the inputs to the validation function that's already a parameter so it would be completely redundant to pass it again so what you're actually passing to the validation function has blocked data is the transactions all the tree nodes you need to execute the messages and transactions and that's it so none of the messages none of that other stuff because it's already a parameter to that function right so the other way that pair of chains differ from sovereign substrate is chain synchronization so what we do is we're following the relay chain we have information from our embedded light node about what's been finalized what block ash in the polka dot relay chains state we can find out what's the latest block header that's been finalized from each pair of chain so if we're following para chain X we can get the latest finalized block of polka dot we can look in that block state and we can see oh this header was finalized so for light clients if you were building a pair of chain light client you'd already be done right you've established finality on this block hash the problem is for full nodes for full nodes you want to download and execute all the transactions in all the blocks ever so the problem is there is essentially state pruning so you can't if you have if you know block 1 million with some is finalized on your pair of chain and all you have is block 100 how do you find out what's the canonical hash of block 100 when nodes on the polka dot relay chain have actually pruned out their state so you I mean first of all you would have to do some kind of binary search probably you'd have to do something clever to actually figure out what which pair a chain block number corresponds to which relay chain block number and second of all even if you do do that there's a chance that that data just isn't available on the network anymore so you could just download in Reverse right because you have a small cache why not just download all the blocks in Reverse and then import them once you've done that so that does work you could do a headers only sync first then download block block bodies but this is a bit more inefficient because when you've got a chain that spans hundreds of thousands of blocks or millions of blogs that is still quite a lot of work to do before you actually begin executing transactions and really synchronizing with the state of the network so the way that we do this is so finding the canonical hashman older header is not easy the solution to this that we are employing in kumo right now is log to ancestors the idea is that so log two is basically all powers of two right so you store this table with like 20 entries or 30 entries and this is in every single pair of chain blocks State it's not in the block bodies it's in the state and then this for each entry like the 0th entry the 1 entry the the second entry the third entry you store the canonical pair of chain block hash of the last block whose number mod two to the I was equal to zero so this lets you do basically you can always update this very easily as long as you have the parent hash in the block that you're actually building because you can just see okay how many tables of entries of the table does this last parent hash overwrite so it's easy to update and it does often let you skip back very so you can string together pretty good compact proofs of ancestor using this technique with an exception to the state pruning rule so it doesn't mean that notes keep around a little bit more data it's really almost nothing but you Dean do need to encode this special exception to the state pruning rule that allows you to essentially figure out what the canonical block hashes were every few thousand blocks in the parrot chain so what you do then is you just sink forward blindly but in the meantime you're figuring out what these canonical block hashes were at various intervals and then as you encounter the blocks that you've downloaded at those block Heights you make sure that they actually are canonical and you filter your peer set accordingly so you could be like eclipsed for a tiny bit of time and then you just kick those guys out so it's completely fine so just as an example of what this table might look like I've done sort of a tweaked bit of the table it normally would go like 1 2 3 4 5 6 and all in all in sequential order but I skipped some of the entries because they just weren't interesting but if like the the last block number was 1 million and 123 you would have these tables these entries in the table so the 16th entry of the table we would have number 980 3040 and for the 18th entry of the table we would have 780 8432 and we could see what these canonical block caches were for the pair of chains so we can easily just skip back to those those blocks and do another few jumps and then figure out what was happening in really ancient history yeah so the synchronization strategy is essentially as I as I described so essentially I mean this is this diagram looks a lot similar to the last one the reason being that you're combining pieces of information from the chain the pair of chain and from the real a chain so you're following the last finalized pair of chain head you're using log to ancestors in the pair of chain State and you're downloading all blocks for the pair of chain sequentially and then combining these three pieces of information in the middle seeing what checks out rejecting what doesn't and you can saying successfully using that strategy so I only have about 45 seconds left so let's tie it all together so just to sum it up what cumulus gives you is a consensus and chain synchronization algorithm but you don't have to implement yourself that lets you turn your substrate chain into a pair of chain without having to worry about those things so we'll do the engineering and implementation and testing of that in cumulus and it will be generalized essentially to all substrate chains that embed cumulus in their runtime as well it gives you a call later node which is using this consensus and synchronization algorithm to sync up to the head of the chain and then continue to extend it with new pair of chain blocks again you don't have to write this either and it gives you some tools for writing substrate runtimes that know how to handle incoming messages and to send them out as well and those api's are super super simple so it's not even necessary to discuss them here so how to get started with cumulus so the truth is right now you can't because we just started writing it we've just designed basically it out but the way that it's intended to work is that you know it's a minimal modification of whatever substrate chain you've already written so if you would have a desire to write a pair of chain on substrate you can just start writing that chains runtime and logic and substrate already and then cumulus as it's ready can be used to port it over with basically no effort so that about sums it up thank you very much [Applause] we do have time for a question or two yes you in the back Jack's gonna make bring you a microphone sorry about that can you so I was saying you're speaking about your pair of chains versus iron chain substrates and how institution making come from Relay chains um in my head I'm trying to visualize this but doesn't this increase like potential latency when you're doing a v8 view change with pbft okay so if I understand the question it's the fact that you're following finality of the relay chain may increase latency yeah yeah so that is true essentially you have to go through this extra step of build the block submit it for inclusion in polkadot and then see how long it takes for that polkadot block to be finalized that's true the fact is though that we've we've done quite a lot of work on the polkadot consensus algorithm just to make sure that blocks can be finalized as soon as they are actually readily available so the grandpa consensus paper is super cool you should check it out Alastair Stewart from the research team did an awesome job on that so essentially latency is still more or less just bounded on the propagation time as long as you know you're not experiencing some kind of major adversarial attack the blocks are going to propagate out to that all the Grandpa consensus nodes and then as soon as 2/3 of them all have that block in their chain it'll be finalized just like that other questions thanks to that point basically if I got it correctly it doesn't practically doesn't have to delay the power chain advancement because you you basically submit checkpoints every and blocks to the polka dot or you submit or you need to finalize every block on the relay chain yeah you can do both ways it really depends on how the block times line up so one thing that would be quite cool is like very very fast chains on substrate right that are happening with blocks once every 100 milliseconds or 50 milliseconds or something what we're looking at for the polka dot block time is more in the range of like 15 to 30 seconds so if your block time is much slower than that on your substrate chain if it could be five minutes then obviously you would probably just submit every single block if you have a much faster chain or even if you just don't want to pin yourself to the relay chain that much you can intersperse it it's really flexible as long as your validation function can support that but that's actually a good ticket for cumulus this hawk has really just been focused on the idea that you would send all the blocks over to polka dot really chain you don't have to do that yeah yeah other questions for Rob okay great thank you all right we're taking a 30 minute break here at this track to get lunch and so forth so you can come back here around 1:30 and we'll get started thanks for coming