Adrian Manning | The anatomy of a basic Ethereum 2.0 client | CrossLink Taipei 2019

hi everyone my name is Adrian I'm sigma prime which among other things is building an aetherium 2.0 client in rust called lighthouse so the talk that I want to give today is titled the anatomy of an aetherium 2.0 client which some of you may know is to be a little bit ambiguous because there's not just one aetherium 2.0 client there's many in fact that there there's all of these ones so lighthouse is the client that I'm working on it's a rust based one and it's because I'm working on it it's the one that I'm most familiar with but there's many others there's Jasper which is another rust point maintained by parity there's Trinity which is written in Python there's harmony which is written in Java Artemis is also in Java I believe those two clients are going to be merging into one there's a lodestar which is written in JavaScript prism written in go and Nimbus is written in nem so as you can see there's a there's quite a range of aetherium 2.0 clients and so what I'm going to try to do in this talk is not specifically talk about one but more about all of them in general how roughly how they work what's what's their internals and I'll try and be as general and abstract as I can some parts of this are Lighthouse specific because that's the one that I work in with some of the examples but I will try and be as generic as again so another thing to note is that all of these clients are written by individual teams over a range of different programming languages so they are fundamentally different even though they're implementing the same specification okay so the structure of this talk I've split it up into three main sections and the three sections are designed to try and give information to a wide range of audience so the first section that I want to talk about is for people that are interested in participating in aetherium - that is the end users so if you're sitting here today when you've got some aetherium and you're trying to figure out I want to get more aetherium and I want to contribute to aetherium 2.

0 then this part is for you this is a very high-level overview of what you need to do to be able to participate in aetherium - how to use the clients and essentially you know make more aetherium on aetherium - the second section of this talk is more focused towards students and developers so this is people that are interested in the underlying technology so I'll be talking about like the client architecture some of the core technologies and in the spirit of yesterday's panel if there's any students or developers that are interested in anything that I talked about pretty much everything that I talked about is under research and development and we always want extra help so feel free to chat with me afterwards the third section of the talk is going to be kind of focused towards more commercial users that towards more commercial users and I'll be focusing on some of the API is that it was associated with a theorem to cut 2.0 clients so that is how if you're building applications on top of etherium - or if you want to how do you actually connect to these clients so these are the api's that do that ok so let me begin with the end users so in aetherium 2.0 there's where we're transitioning to a proof of stake so in aetherium one we have miners so the miners you know produce proof of work but they do proof of work and contribute to the system as a response they get a as a as a result they get a reward for doing this in aetherium - we have proof of stake which means you have to contribute some stake and perform some actions on the ecosystem in order to get rewarded so the analog of a minor in aetherium 2 is a validator so you need to be a validator in order to participate in the system and by being a validator you can you will be rewarded for contributing to the system so some of the so I just want to briefly go over the validator life cycle so what you have to do to participate in a theorem - and this is a very high-level overview if you want a more in-depth discussion of the validator life cycle I recommend looking at shell ways slides who spend a bit of time and a lot more detail in the validator life cycle so anyway to begin with the first thing you need to do is you need to generate key so key generation is the first step to become a validator in aetherium - we're using the BLS signature scheme which is different - aetherium one in a theory and one we use skip to 5/6 k1 so what that means is you can't use your aetherium 1 keys in a theorem - you need a new set of keys and you need to generate 2 keys not just one so the two keys that you Ede are a validate assigning key and a withdraw or key we have two separate keys because as a validator when you're participating on the ecosystem you need to do constant signing and so that means you'll be using a key quite regularly so if that gets stolen or your software gets hacked they can't immediately withdraw all of your stake right so the withdrawal key is something you keep prior you keep separate and that is the key that you need to withdraw your stake from the system once you've done key generation you need to then deposit some original aetherium one into a deposit address on the etherion one chain and you need to supply your validator key and a hash of your withdrawal or key and you provide a signature approving the ownership of the validator key the signatures required because there's a cryptographic attack in segregate signature aggregation and BLS called the rogue public key attack if you're interested but this is just one way of mitigating that so these are the the main things you need to contribute to the to the etherion one deposit address to become a validator once you've done that deposit you've deposited your 32f you then need to wait a period of time the waiting time is essentially some time for the etherium to chain to recognize your deposit you get put into a queue and then after a period of time you become activated once you're activated as a validator you then have to actually run some aetherium to clients so to participate in aetherium to and get rewarded for your stake you need to do active contributions and what that means is that you need to be running and aetherium to client more specifically you need to run a validated client and a beacon node and I'll talk about the differences between those two things but fundamentally what you're participating in the system you're voting on shards and you're voting on blocks at a network layer what the software actually produces are things called you produce blocks and you produce outer stations so this is your contribution to aetherium - and by doing so you get rewarded with with more aetherium provided you don't modify the client and misbehave then you will get not rewarded or the opposite of rewarded you'll get you will lose money so after a period of time you can if you don't want to be a validator anymore you've decided you don't want to run the client you can exit you don't have to be a validator forever there is a number of ways you can be you can exit being being a validator in the etherion shoe system the first one is a voluntary exit so you can you can opt out when you opt out you're no longer a validator you can get removed due to excess penalties so if you are inactive for for a long period of time your balance will slowly decrease until it reaches a threshold at that point you will get ejected from being a validator because you don't have enough stake in the system anymore and the third one is being slashed so if you have done something bad or you have modified the software to try and do something a little bit malicious you can be slashed what that means is that you've done something that breaks the rules of the system and we immediately eject you you get kicked out so this is roughly what you need to do to be a validator in aetherium - and how you can participate so I just want to give you a very high-level overview of how you use the software to do all of these things so the first thing I should mention this is in their section of generating a key and and and and putting a theorem - into the deposit address is staking pools so staking pools are similar to mining pools in a theory and one which allows you to just send a theorem to essentially a staking pool you don't have to send the requisite thirty-two after you can send less than that and the staking pools will pull that together they manage a whole a whole network of beacon nodes beacon validations they control the software for you and they will they'll pretty much do all of the heavy lifting and hard work and in return you you will receive some stake as you were in it so the benefits of staking pools is that you don't need to manage the validated keys yourself they generate the keys they will do they'll interact with the deposit address you don't need to manage any of the beacon nodes or validated class you need to manage the software you don't need to interact with the deposit contract you don't need to stake for multiples of 32 F you can do less and these things are typically you I focus so one example is rocket pool which I've got here which is a slide a screenshot from their beta program this pretty much abstracts a lot of the a lot of the work of running to clients but it's it's an easy way to participate and join the system so I figured I would mention this so if you are not going to join a staking pool and you want to participate yourself you're going to have to do the do the steps that I mentioned earlier manually generate keys and interact with the deposit address so from a client perspective currently I think most clients are interacting this is hopefully this is small and not too dark it is slightly dark but the idea is currently we have CL eyes command line interface tooling that allows you to run a single command to generate some validated keys as we move closer towards maintenance I think all of the client teams will be focusing more towards UI things so that you are allowed to you'll be able to just click and point to generate a key but currently if you if you want to try and test these things out I think all of the clients this is an example of lighthouse if you want to figure out how to use our CLI we have documentation but you can essentially run a single command which will generate a validated key boy I think we'll update this so that it also helps with interacting with the deposit address so after you've run your single command and you've generated a key and you've deposited onto the deposit address in aetherium one you need to then run a beacon node and a validator client so each participant has to run a beacon odor and validator client the beacon node I haven't quite talked about this yet but the beacon node is the main piece of software similar to death or parody in etherion one it has all of the the core logic I guess the validated client is kind of a you can think of it as a like a hardware wallet it manages the keys and does the signing and it's separate from the beacon node they're currently at the moment is some development going to around UI that wraps these theorem two clients one example is Mitch Matic which is working with prism and here are some two screenshots that they have of this UI so this is the idea is that you have a UI that wraps the beacon node you can click on it create a creator validator you can see other valid A's on the network you know what your balance is you know how much is coming and it gives you a state of the current state of the network so again as we move towards phase zero there will be more and more work going on towards building UI focused a theorem to clients but at the moment you can still run everything from the command line so again an example of running theorem to from the command line is with lighthouse so with lighthouse you right you can run a single command that's the the top window is a beacon node the bottom window is a validator client the validator client automatically connects to the beacon node and does it its duties I think Danny Ryan calls those duties but this is essentially an example of how of how to run a beacon note and a validated client once you have this on the manor you just let it go you sit there and you should accumulate rewards for participating in the network ok so that hopefully is a high-level overview of how you could participate using aetherium to clients on the network I now want to focus a little bit more on the core technology some of the the the technologies that we've decided that underpin that underpinned these are theorem to clients so hopefully if there's any students or developers that interested this section pretty much has everything that's under research and development so let me begin with client architectures so most I'm pretty sure all aetherium to clients have segregated validated client from a beacon node and as I was saying a beacon node contains the core logic similar to Gathol parity in aetherium one in particular it's has the state transition logic it has its own database it is the thing that can change the network stack so it interacts with other peers on the network and it has an RPC server or HTTP RPC server to interact with other other applications on top of it so this is the the main piece of software that allows you to interact and participate in the etherium to network if you're going to be a validator then you also need to run the separate by a software called a validator client so the validated client manages your keys it you don't need to have one validated client per key you can put multiple keys into this piece of software and it will look after all of your validated keys for you and it keeps track of the tasks that the validator is required to do so when you need to propose a block when you need to do in a test hey the validator client will keep track of that and perform the signing when it needs to it also should depending on the client infrastructures a lot as I said there's a number of different teams building each of these pieces of software so they're going to be different across different implementations but they should have inbuilt slashing mechanisms which means it keeps track of what you've signed before so that you don't accidentally make a bad signature and then and then lose all of your stake so this should have a preventive mechanism inside of it to just stop you from accidentally performing bad signatures on the network okay so more specifically in the validator client they're the way that they're fundamentally the way that it works is it connects to a beacon node it checks for specific tasks from the beacon node so given it set a validator validator private keys that you've given it it knows it knows about all the validators that it needs to perform tasks for it asks the beacon node given all of these validators when am I supposed to propose a block or perform an attestation when that time comes then the validator client can ask the beak you know to either give me a block that I'm supposed to sign or give me an attestation it will perform the signing and it will send it back some important things that you should know is that the the idea is that the validator clients in the beacon note even though they're built across a number of different client implementations that the the communication between them is standardized so what that means is that you if you have a favorite beacon node client you can you and a favorite validator client which is written by two different companies or two different two different languages they should be cross compatible so you can have a lighthouse validator client connected to a Nimbus beacon node if you prefer you can also have a validated client connected to many weaker nodes for redundancy so if one of those goes down you don't want to actually start losing money so you can have backup beacon nodes okay so a little bit more in depth into the beacon node some of the core components that we have fundamentally it looks a beacon node looks very similar to a standard blockchain client that you would expect it has HTTP API that allows you extra applications to connect to it it has state state execution and syncing logic so it when you get a block how to how to transition from one state to the next and has its own internal networking the networking is significantly different from etherion one that warrants a little bit more detail in this slide so in aetherium 2 as rails spoken about we're using live p2p in particular we're using these protocols here in lib p2p the first one is gossip sub which is pub sub rail quickly went over this this is we're using this protocol to propagate blocks across the network so this is a an epidemic kind of protocol which allows us when we when we get a block we propagate to the network get propagates to all the other peers and get sent across the network we have a custom protocol called ëthe to request response this is a direct peer-to-peer communication protocol like an RPC so when an upir joins on our network and we find that they have potentially a lot more blocks than us we want to sync with them we use these protocols to perform syncing to download blocks immediately from them we're using discovery v5 which is an upgrade of discovery v4 as that you can imagine from their name it's fundamentally a modified CAD emiliya that works under UDP and it's currently in in development we I have an extra box here which has extra protocols because Lippitt appease modular it allows you to add and remove protocols and different clients and different teams can add their own protocols that allow their clients to interact and talk to each other one example is Liberty offers a protocol called identify which allows you to acknowledge or find out they you know what protocols your peer supports the version it's using and among other things so different client implementations can actually introduce and use different different protocols and you will see that when when we hit main it I imagine yep ok so everything I've talked about so far actually everything even on this slide is still hunted development as we all said gossip sub is kind of transitioning there they're trying to build a new one called episode so the first protocols under development discover v5 has only just kind of been specified there's still other more specifications going on so everything here if you're interested in networking or some of the state execution this is all still under under quite under development and and some research and development so the next thing is everything I've spoken about so far is kind of general across all aetherium to clients so now I want to talk about some of the discrepancies that you might be interested in across the clients so we have an aetherium to specification and that tells us roughly how the rules about how the system should work but it doesn't tell you about how you should implement them a simple example is that the there's no optimizations really in the etherium to specification so the implementers and the client teams will find their own implementations and you'll have different performance trade-offs for memory speed or size so some clients that are trying to build in a theorem to client that goes on a very small embedded device we might make decisions that that optimize on size but but but it has a trade-off versus time so different client implementations will have these kinds of trade-offs so I just want to even though there's a whole heap of these I just want to want to talk about four major components that you might see differences between a theorem to clients the first one is the database so straight off the bat if um two clients can have different database types there the layout of databases will be different which or will also affect the optimization whether people favor speed on read or write access so the number of different optimizations are going to happen in the database alone so the next one is syncing we have a networking specification that that tells us how we can download blocks in a and and state from from our peers and it implies a general form of syncing strategy but the implementations there are different optimizations and different strategies that in some cases can be faster and in some cases can be slower depending on the state of the of the chain and the network you can you can have optimization guesses which if they don't work out can be slower so different different syncing strategies will you likely see across different clients and the way that we do peer load balancing so if want to download blocks from a hundred different pairs that's also subjective so different clients will have different syncing strategies I imagine the next interesting one especially for people that are doing this to make profit which I imagine everyone this is the fundamental incentive is attestation selection strategies so in aetherium one where if you're a miner and you're getting transactions you get a lot of transactions so many transactions that you can't fit them all into a block how do you decide which transactions do you put in your block in a theory one it's easy because we just we want to we want to maximize the amount of money we make when we when we solve this block so what we do is we just choose the etherium transaction that has the highest gas price and we order them and then any ones that the lowest ones we just removed from the block in a theorem - it's not so obvious we can't we can't do that same strategy not obvious which which ones that you add into the into the block that you have will maximize your profit so there's a number of different attestation inclusion strategies that different clients can take ideally you want to maximize your profit but there are going to be inherent trade-offs in which strategy you use so you might find you make you will make a different amount of money depending on your client implementations in principle in theory for choice so for choice we have so we have an algorithm so that if chain Forks or diverges we need to run an algorithm to determine which from given our subjective view of the current state of the block chain which one is the longest which is the the canonical change the specification has a fork choice in it which is prohibitively slow so none of the etherium to clients should use it they need to use an optimized optimization of the original specification and there's a range of these different potential algorithms that clients can use some of them have different trade-offs as well pretty much everything I'm talking about here has some trade-offs so implementations depending on what they're focusing on will take one trade-off over another so you might find that that different clients faster and some some regards and slower and others another thing I should mention is most strm two clients have their own metrics internally built I think most most of the implementers at the moment have these have these built in mainly for profiling testing performance and just looking at the current state of the network lighthouse has has a docker if you are interested in running this thing and it gives you it this thing spins up for nodes and looks at how those four nodes work on the network but this is kind of interesting if you want to play and decide which client kind of works for you you can run separate clients look at their metrics you so hours and I'm pretty sure a lot of the others gives out information such as database read rights how many resources are being used some of the networking statistics so you can depending on your use case on which aetherium to client to use you can look at some of the metrics that they have built in which is another interesting fact so the last part this talk I just really briefly want to go over commercial users and developers so this is just specifically the data the data model and the the API is that we're planning on using for aetherium too so if you're if you're a big data scraper or an exchange and you want to interact with aetherium too you're going to need to interact with one of these clients and typically through one of these these api's so the first API I want to talk about is the validator client to be connote API so this is the we're trying to standardize this API so that as I was mentioning earlier the that you can have a range of different validated clients from a number of different teams interact with a range of different beaker nodes from a number of different teams so you can have one validated client from one team interact with a beaker node from a number of other teams so by doing this yeah you can clients can can connection are two arbitrary beacon nodes and it also works the other way around a variety of you can use a variety of beacon nodes as backups so if one if one client has a very catastrophic bug such that all of those type of clients go down you can have a backup of a different kind and you can continue running on the network without losing any stake [Music] yeah so this repository up here the etherium api's repository is currently I guess currently under development this is a this is a screenshot from the from the validated clients at beacon node API which how pretty much outlines what we're currently thinking of using as the the API between those two things so if you're interested in building your own validated client because you want to add extra features or if you want to make any kind of self improvements and this is the API that you should conform to this allow you to connect to most of the etherium to bigger nodes the next API is the beacon node API so this is you can think of this as like the like the JSON RPC API of Gathol parity we we want to standardize this as well across all of the clients so that you can build applications on top and and the underlying client that pins the applications it doesn't matter which which implementation does this we the current plan is to use a HTTP API this is from lessons learned from aetherium one and the other important thing is that this is currently under discussion we haven't specified or come to agreement on exactly what this HTTP API should look like so in I saw in Mel's talk yesterday from monolith he he said that they had to rewrite the JSON RPC forget because it wasn't sufficient so I encourage anyone that's looking at building the applications on this honor theorem to to get involved in these discussions because we can modify this now before we we all start and a programming this so the there's a pull request for this for this rough page HTTP API up on the top right there prismatic labs also have a version as well so we need to kind of standardize the beacon own HTTP API and that's it that's hopefully an overview of some of the etherium to clients and what's being worked so thank you for your time thank you listen they're also sharing and do we have any questions from the audience raise your hand if you have any questions my oh you look like you have a question to ask I have to speak afterwards okay you sure okay anyone else okay great no that's against a woman Adrienne thanks again you