Fork choice and Gasper | Ben Edgington | Lecture 16

recording is on welcome for first session and CH and I believe he most famous stud thank you so much Mario um I appreciate the uh warm introduction it's a pleasure to meet with everybody here the uh Proto cevs is one wonderful um this is probably the best looking slide in the in the presentation not just because it's got my face on it but the rest of it is just a dog's breakfast so don't uh uh criticize me for that please um just brief intro um I uh discovered ethereum somewhere in early 2016 and that marked the transition from what I refer to as my suit and tie wearing years to my hoodie wearing years uh when I joined consensus in late 2017 uh did a bunch of stuff there um including as Mario says um overseeing the teu team I didn't write a lot of the code but uh I found fabulous people who have uh turned it into one of the leading consensus clients now uh I left about a year ago just over a year ago joined op little sojurn in L2 um at optimism uh and I recently left again and I'll tell you why um the green stuff down here is a kind of um hobby weekend evening work that I've been doing over the years um for four years I publish what's new and E two a sort of um bi-weekly newsletter on uh the consensus protocol as it developed um but the main thing Mario's already referred to this is this uh upgrading ethereum book that's the official title not the E2 book but you can find it at eth2 book. info and it's a bit stale I'm nearly two hard Forks behind uh I it's the bad year and a half out of date uh I had I haven't done any work on it whilst at optimism so the idea is now to pick up that work and spend at least the next year just bringing it up to date with denb Electra fulu maybe and fleshing out the content in there so it's complete I will be linking a lot in this presentation to um the material in there um there's just too much to cover in in one presentation like this so if you want to go deeper it's all there right let's get on with uh easy agenda uh quick intro to Fork Choice stuff uh we're going to look at lmd ghost we're going to look at capser FFG we're going to look at how they're combined into Gasper and then I'm really hoping to leave some good time at the end for some uh AMA either about Fork choice or anything else about cev life um yep so let's get into introduction to Fork Choice uh so I think you've had Alex Stokes or there's been a presentation U maybe you've watched by Alex on basic consensus stuff uh I'm haven't really got time to recap all of that so I'm going to skip through this but you know you've got the presentation you can follow the links um and they'll take you to the relevant Parts in the book um but useful Concepts to know for this session safety so colloquially nothing bad ever happens so we have a distributed system there are nodes all over the world uh all communicating via this unreliable Network some of them may be adversarial uh some of them may be faulty or you know resource constrained or slow or whatever um but uh they this network is trying to come to agreement and if it's a safe Network then nothing bad ever happens which means a reorg maybe or a double spend or uh inability to progress the chain or uh or inability to finalize the chain I should say something like that um counterpart to safety is liveness which is also known as availability and this is the idea that something good eventually happens so um the idea is that we can always add blocks to the end of the chain uh for example or we can always continue to finalize the chain uh something good eventually happens and there's a theorem called the uh cap theorem consistently consistency availability partition tolerance theorem which basically says says uh you can only have two of these three um and partition tolerance isn't really um reasonable on a real Network like the internet there are always going to be partitions where some part of the network can't talk to another part of the network um so in those circumstances we have to choose whether we're going to favor consistency so our system is always in a um well-defined State and every node agrees with every other node or we're going to choose availability which means that our system can um continue working uh even in the presence of of partitions like as as happens on the Internet it's a very simple theorem um and uh quite easy to understand um where where that comes from and the point is that ethereum prioritizes liveness so uh we have this lmd Ghost Protocol and the idea is that you could it just even if there's a big partition of the network or half the nodes go down or something like that the what remains what you what you can see locally will still continue to to make progress and add blocks to the chain um and that that's a designed decision not not all chains do do that so Cosmos chains for example prioritize safety uh but on top of that we add this kind of best efforts finality Gadget so we'll we'll come to that uh uh in much greater detail so we kind of guarantee liveness and we have best efforts safety and in combination we call that gas asper so that sets a scene we'll be unpacking all of that uh as we progress now the challenge that we have before us is that networks that prioritize liveness tend to be forkful um if they weren't they would be safe um because everybody would agree with each other all the time but because nodes disagree because of late messages or um because an attacker is feeding different messages to different nodes or um because of faulty software or something like that they have different views of the network at any given time and this this results in what we call Forks so we end up with a block tree rather than a Block Chain uh and here we have a block tree depicted we have a root which is it's lying on its side um this poor tree uh it has its root at block a and then it has branches which uh come off that so the branch that ends in Block C branch that ends in Block F branch that ends in Block e and where there are multiple branches coming out of a block um so it has multiple descendants we call that a fork uh and I our task here is to choose between forks so that's the purpose of the fork Choice rule uh it's a method by which the network converges on a single branch and therefore a single shared history so what we're doing is turning this block tree into a Block Chain which is just got a linear history uh going back and we prune out the blocks which are not canonical not included in that um that branch and how do we do this well there's a kind of horis I mean uh the the goal is based on the information that we have about the network which we receive through blocks which we receive through um uh peer-to-peer information which perhaps we receive uh carry pigeons whatever it is based on all the information we can gather about the network we try to build a block on the head of the chain that is least likely to be orphaned by other nodes so we build a block on the head of the chain that other that we believe other nodes in the network also think is the head of the chain and if we all agree then we've converged we have a unified view of the network so we have to somehow guess BAS on this potentially faulty potentially late potentially incomplete information we have about the network we have to guess which block most of the network thinks is the best is the one that uh they would build on if they were in our shoes and then we we have convergence onto a single view right uh so far so simple I mean that's uh that's the sort of basics of fork choice and we you know we just make this Choice um hereby dragons um this slide is a partial list of issues with ethereum's proof of stake Fork Choice uh going back in time it's not going to go into all detail now though we will revisit one or two of these um in slides to come and although it can look superficially quite simple and the concepts are not too hard it turns out to be fishlyn edge cases and it's very easy to get stuff wrong um and this kind of ignominious uh history is Testament to that so there are links to all sorts of exciting attacks on the chain that people have discovered now we think we fixed all of these right uh now um but yeah nonetheless I wouldn't bet on there not being more issues in there and the other exciting thing about Fork choice is it's not um operating in a nice clean isolated environment it um has to operate in conjunction with the real world uh so two weeks ago um the whole ski test net was upgraded with pectra um and uh it was shall we say disaster um and I've linked here to the start of the conversation on Discord from uh that evening two weeks ago uh and if you can follow along so 40 hours in the life of FK choice if you can follow along that conversation that the devs current devs are having or had about how to get the network to converge onto one chain after it fragmented um in hsky then you know you're 90% of the way to being a cordev I mean it's uh goes into everything and the fork Choice was fighting against you know faulty clients were the cause of the initial incident you know they disagreed on the um block validity this L to kind of peer-to-peer meltdown where nodes couldn't find each other couldn't find other nodes on the on the right same chain as them so blocks weren't getting propagated attestations weren't getting propagated very well uh and then there was all sorts of interference from slashing protection cut kicking in low levels of block production um clients were offline trying to get back online they hampered because there weren't snack sync servers checkpoint sync was difficult because it relies on finalized checkpoints optimistic sync um also caused problems because it uh of the window of availability of it clients were going out of memory and falling over it was extraordinary and amidst all of this this nice platonic Fork Choice rule that we have has to handle all of this and come to agreement about the state of the network and um yeah rescuing hsky has been uh educational so moving on from the kind of challenges we get kind of into the specs here um I will show some code uh in the course of the presentation um we've got um this object this attestation data object and this is how um nodes communicate with each other they they communicate with each other in two ways one through attestations um and there are hundreds of thousands of these every slot and also through blocks of which there is one per slot ideally um and so the main way nodes communicate with each other is via attestations uh and so each validator on the network broadcasts its view of the network VI an attestation it does this once per ebo so every validator um that's got a stake uh has the opportunity once per Epoch to uh broadcast its view of the network and primarily via peer-to-peer gossip but at a station's also packed into blogs by blog proposers so the the record is there what the contents of an attestation um well you have a slot number just so you can anchor it in time the committee index is historical uh after the Electra upgrade uh this will be set to zero uh all the time so don't need to worry about that and then there are two kinds of vote and this is what we're going to focus on now so this is the uh what feeds Fork choice this is the raw material for running Fork Choice two kinds of vote lmd ghost vote and FF FG which is a Casper FFG vote um and lmd ghost votes are for a block so you vote for the root hash of a of a particular block and um FFG votes are for two checkpoints and we'll look into those more later and the two consensus mechanisms that we have um lmd ghost and Casper FFG uh operate on different time scales and and Via different mechanisms mostly so um lmd ghost largely depends on attestations receive by gossip it's a kind of realtime protocol it has to make quick decisions on the order of seconds uh and doesn't have time for to wait for attestations to get into blocks so in the specs this occupies its own um document which is called Fork joints. MD uh and um Casper FFG depends only on attestations that have being packed into blocks um so uh handling of those is part of block and Epoch processing so you'll find all the stuff about that in in Beacon chain. MD document uh we're going to focus on Fork choice. MD uh in this session but yeah that's just to help you kind of Orient yourself in in the specs and the other thing um that matters is the store so this is the nodes view of the network um what's classically called a view in In classical consensus Theory this is all the information that um and you know absolutely everything that a node needs to know about the state of the network outside this is all the best information it has about the state of the entire network uh and it's called the store um it has started life very simple so these lines without a red asterisk were the original ones um in the uh early versions of the spec over time the lines with asterisks and these objects and structures have been added solely to fix issues that have been found over the years so that's just another sort of hint that everything is more complex than you would hope it would be um it has grown into a bit of a monster there so that's the store and the third interesting thing about kind of fork choice is lmd Ghost Fork choices it's event driven um so there are handlers um which you can find in that fork choice.

MD document um and there are four handlers effectively um on tick so the fork Choice store the store is regularly updated it updates its current time which is uh at the top of this store object here um uh every time a tick happens I'll tell you what a tick is in a moment and it does some light housekeeping uh as well it's a very lightweight function uh a tick is just a regular um call I mean it could happen like twice per slot in teu we did it every uh twice every second so every half second um we perform a tick and the fork joice would update its time and do some housekeeping more interesting is on block so this is what happens when when my node receives a block uh on the wire from anywhere then it uh this event is triggered on block um and the Handler uh adds blocks to the store uh as and when the node receiv them uh on attestation similarly anytime an attestation is received via any means whether it's in a block or on the wire um then uh it updates the validor validator latest messages uh in the store so we have this uh latest messes field at the bottom and um on attestation updates that field and then we have this on a test of slashing which was added uh later just uh to avoid um an edge case where validators could be slashed but still count within Fork choice and so we need to know um if they've been slashed and discount their votes it avoids some attacks where um validators can uh vote um they can equivocate they can contradict themselves and make multiple votes um and there's some latency in in in the slashing so that deals with that and so because it's event driven um the the the FK choice is always ready to Output a best head block anytime I want to call it if I need to make an attestation or I need to build a block and I need to know what to attest to or where to build I just call the get ahead function on Fork choice and it will spit out a block and say there you go attest to this build on this and uh and I'm done right in that's Whistle Stop intro stuff uh let let me just pause as promised for uh any questions um we will be looking lmd ghost next and then Cas for FFG um anyone want me to clarify and I that hello I wanted to ask about the job and the work that proposers do in collecting the attestations and packing them into blocks so are there any incentives either Rewards or punishments for doing that or not doing it and like what if I'm a proposer who wants to save a little time and and not do that can you help me that a little yeah um there are no direct incentives to pack attestations into blocks uh however there is a minor um incentive to do so which is I cover it in in the book a little is that my reward as an attester um is proportional to the number of attestations they get on chain so if 100% of attestations get on chain I get a full reward if only half attestations get on chain only get half an attestation reward and so does everybody else so if I want to make sure I maximize my attestation reward then I need to make sure that as many attestations as possible get on chain um which is my sort of indirect inent incentive to to pack attestations into blocks does that make sense yeah definitely thank you yeah quick one for um the ATT station the beacon block route are you voting on what is the previous block that everyone else is building on for this round in the slot or is that you saying like this is what I think now will be at the current head of the block like the new hat yeah yeah so Alex's presentation covered this a little bit um but uh so a block is supposed to be broadcast within the first four seconds of a slot and we'll look at that a little and then at the 4 second Mark in a slot I'm supposed to attest to um that block whether I've seen a block in which case I attest to it or um if I haven't seen a block I attest to the the the previous block so it's I I declare it's an empty slot um so that's uh the the the timing and what I test do so uh at 0er seconds the Block's supposed to be published 4 seconds for it to propagate at 4 seconds I make an attestation and broadcast it and at 8 seconds um the aggregator start aggregating attestations ready for the next block proposer at at 0 seconds you know 12 seconds to to propose the next block does that answer your question yeah it'll be this spot yeah 4 seconds you s out the side testation data and it'll be the what you're saying is now the new beacon block rout yep y I been talking on mute okay so what is the optimal way for validators to send the latest messages whether it's blocks or in block or VI the P2P which is the optimal for them uh yeah so attestations need to be broadcast by P P2P um because uh they I I am not likely to be the proposer of the next block so I need to broadcast to all my peers on the network my attestation they circulate uh uh around the network they will later be packed into a block but that's that's much slower process so uh yeah it needs needs to be broadcast okay so if validator is selected as proposer uh for that particular slot then they can uh send in the blocks also is it I'm right or wrong uh yeah I mean you include your own attestation for sure yeah um okay okay thank you thank you V yeah all right let me let me press on uh let me how we doing on time all right okay need to get a move on um right let's talk about lmd ghost so this is the main um Fork Choice uh rule that we have on on ethereum proof of stake so the naming is insightful lmd stands for uh the MD stands for message driven so it relies only on attestations um and validators attest this is what we just said validators attest what they believe to be the best head in the current slot so if they've seen a block in the current slot that's it if they haven't uh at that 4C Mark then they attest to the the the previous block from from the previous slot so it's message driven that's MD and it's l latest so only the most recent attestation from each validator counts so um there have been other variants uh mediate message driven fresh message driven recent latest message driven uh I'm not going to go into those they're mostly passed into history um so just it's just to say there are other ways that this could be done but we use late message driven so whatever the most recent message from validator 10,000 is that's its um that's its vote in lmd ghost which is his beacon block rout uh the ghost is more interesting so this stands for greedy heaviest observe subtree algorithm ghost uh and actually dates back to a Bitcoin paper for by S palinsky and zoha and um it was really about what what what would happen if they decreased the block time in Bitcoin is 10 minutes right targeted um and made bit Bitcoin a bit more forkful uh than it than it currently is um would it be uhco us as the heaviest chain rule could they improve on that and the answer is yes um ethereum considered implementing proof of work version of this but never did so um but we now have it under proof of stake and we're going to look into it in detail so that's the naming and this is sort of the the key points about lmd ghost so his time scale is short it's this 12 second slot-based um protocol so we're making a decision every 12 seconds about what the head of the chain is um and so it's relatively rapid for a global network uh that's why we have to broadcast dat stations via the via the P P2P um as as swiftly as we can and and the goal is um uh it's used for two things so I want to find out what the head of the chain is for two reasons one is if I'm proposing the next block I want to know which block to build on that's least likely to be Reed out so that my block survives and makes it into the canonical chain uh and it's used if I'm going to make an attestation in a slot I I need to know which um is the best um block to to be attesting to uh in that slot and the goal is that all of the nodes that share enough information will come to the same decision about about this and the network will Converge on a single canonical chain so that's theistic that we've spoken about which branch um in the block tree is least likely to be offened in in future um uh that's that's the one to be building on and what we do is we weigh the votes that we receive in attestations and it I call it weighing because they're weighted with the stake that each validator manages the effective balance of each validator uh and it's important to note that in any given slot so I've told you this is slot based it's happening every 12 seconds in any given slot only a maximum of 3 and 1/ 18% of the votes are fresh so 132nd uh of the slots so every slot only uh one out of 32 validators is making an attestation uh because there are 32 slots in an Epoch so our new information in every slot is quite limited we're only seeing uh the world viiew of just over 3% of validators uh in each slot and it's quite feasible that the other 97% of validators have a different view of the world uh and we could be led astray by that so worth worth bearing in mind what are the properties of lmd ghost well it's uh live it will always output a viable headlock on which we can build uh and there's there's a proof that exists of this under synchrony assumptions um under asynchrony there's a proof in the Gasper paper of it that it's probabilistically live but needn't worry about these These are details it's basically live it guarantees that that you can always continue building the chain uh as for safety it doesn't give any useful guarantee so lmd coost doesn't guarantee you any kind of um any particular kind of safety uh however there is some ongoing work on a confirmation rule which if you assume that you have less than onethird of the validators are adverse material uh which we assume anyway uh basically otherwise many things would break and you assume that the network is synchronous in some sort or eventually synchronous then um you can say that uh after you've collected enough votes for a block it will not be Reed out of the chain So within a couple of slots you can say that this uh under these assumptions that there is a block that won't be reared out um so that becomes some sort of safety guarantee but we're not currently using that that's sort of for implementing it in future um there's a link there if you want to find out more about that but assume for the purpose of this presentation that that there are no safety guarantees in lmd ghost so what are we doing um here is a block tree I have uh we've got blocks ABCDE e FG um why has c not built on B well maybe C her uh didn't hear about B in time so it it it built on the best head it you of which is a um maybe it thought that B's block was invalid for some time it received it in time but thought it was invalid for some reason um maybe uh C saw some juicy me in B's block and decided it want to to steal it and so reor it out of the chain and steal its me for whatever reason C didn't buildt on B it built on a um and you know likewise for various reasons other blocks have have forked off um I've labeled each block with a number and this is a sort of a number of votes stake weighted number of votes is sort of completely arbitrary in this example um so that's the the the latest BL votes for Block f are 20 for Block g a 10 for block e25 and so on and in pure lmd ghost a on the left the root here is would be the Genesis block of the chain so we go all the way back to Genesis Block in this algorithm now this is the same diagram but updated and what I've done is on each line I have put the weight of the branch so each block has a weight and if you sum the weights so D and G add up to 20 so the um Branch leading to D has a weight of 20 b d and g add up to 40 so the branch leading to B has a weight of 40 and we can see down the bottom you know f is 20 e is 25 if we uh the branch going from a to c is worth 60 because c e and f add up to 60 so that's um critical for for how the algorithm Works um because we're looking for the we're we're greedily looking for the heaviest observed sub tree so this 60 between a and C is the weight of the whole subtree descended from it so the heaviest observe subtree is what we're after and here's how the algorithm works it starts at the root which is a and it follows it it finds the weight of the branches coming from a there are two branches and it follows the one that has the maximum weight the heaviest Branch the heaviest sub tree so in this case we go from a not to B but to C and when we're at C we just do the same thing again we look at the weight of the subt trees coming off C and we see that the one that goes to e is heavier 25 rather than 20 and so we end up at e and so we say this is our head block according to this algorithm it's a different head block than if we applied the longest chain rule that would give us block G so we're applying the ghost rule so we end up at block e and that's uh that's it really that's basically how lmd ghost works very simple um here's some code which instantiates that um so we start with it's called head but it's a root at this point we start with a root of the tree uh which in the code is Justified checkpoint root but if as I said in pure lmd goes that would be the Genesis block we'll come to this justifi checkpoint thing later and then we have this this annoys me for reasons which I can't quite pin down but this this is a recursive function but it's been written iteratively which I yeah me but anyway what we are doing is um iterating uh over the branches that come out of this uh block so we start with the root uh we find the children of the root block and we get the weight of each subtree using this uh get weight function and my greatest contribution ever to the ethereum uh consensus specs is naming this function so thank thank you very much um and we choose the maximum amongst the weight of these so we get the weight of each of the children branches each of the observe sub trees and we uh find the maximum and that becomes our new head and then we go around the loop again now nobody would ever implement it like this this is horrifically um inefficient because you're just repeating so many calculations so either you need to do some quite aggressive caching memo memorization uh or you need a different algorithm um protol Lambda has done a bunch of implementations of this uh a few years ago um one of them became Proto array which became popular amongst client implementations prism used it for a while they moved away from it I think teu is still using it um but if you really want to dive into how this is implemented in practice then you can follow the link uh there to that but yeah that's um oh yeah and then we are the As for the get weight function how do we weigh these sub trees um well we look for all validators that were eligible to vote uh for it and we add up uh eligible to all validators that are eligible to vote at this particular time and we add up their effective balance so votes awaited by the effective balance of the Val valid dator as long as they are unlashed and active but only if the vote is for a block that descends uh from the the the root of the chain so when we're at the Genesis then uh we the the weight is you every block in the tree and as we go along the branches then it's just the weight of the sub tree so it's the weight of all the blocks that haven't for which the root block is the ancestor you know the particular block is the ancestor um is this and then we return that ignore propos a boost for a moment we'll get to that so better in a diagram you know we we just add up all the blocks in the subtree uh here and that's the weight um and that's lmd ghost uh very simple um there are a couple of wrinkles to it here uh we want to ignore undesirable blocks so there are certain blocks we don't want in the block tree um we we want to make sure the block is valid that it was correctly signed by the expected proposer this is actually checked Upstream of fork Choice uh I think in the gossip ler if I remember correctly uh we want to make sure that all of the ancestor blocks of the block that we're looking at are valid we can't you know if there's an invalid block then all of its children are invalid as well we can't recover from that so we exclude those you want to make sure that the block uh postate hash is correct that it did it respect the state transition function that's the execution layer State transition and it's a consensus layer State transition um so we have to rerun that and check it's correct so those are basic validity checks uh new in denb last year this time last year um we added blobs and we have a another check uh we only include in the fork Choice blocks whose data is available if we cannot get the attached blobs for a block then we uh we ignore it until we can so we can't make the head of a chain a block for which we don't have the data or we can't get the data uh in future potentially uh we want to might want to make sure that the block is non- sensoring so there's this um fossil uh work going on and that would exclude from Fork Choice any block that uh for which there's evidence that it was sensory that it hadn't included the transactions which it was supposed to have included or if ethereum becomes a government chain maybe they want to only include blocks that do sensor uh for example um you know sensor a tornado cache or something but uh let's not go there um and in future we might want to look at um more sophisticated data availability for example um appeared as uh rather than just checking the attached blobs there are more sophisticated checks that we can do uh and there's some already some work being done on that for fulu upgrade uh yeah and this is handled in the on block Handler we didn't go into all the detail but you know we check whether all its ancestors are there check it's not from the future check it's not too old this is an optimization check the data is available uh the next bit of code uh checks that the state transition function is correct and so on butth this is this is where we we we filter out the undesirable blocks before they even get into the store so we don't have to filter them later um blocks for which data is not available may be held back and then become valid later may be resubmitted so that's this comment here it may be cued and subsequently considered um in which case on block would have to be called again right nearly there on lmd ghost just want to talk about two more things proposal boost and late blocks um so uh in 2020 a balancing attack was identified um and under some rather extreme circumstances a very small number of validators could Fork the chain into two equally weighted branches um and balance the votes for them and maintain this split indefinitely um propos a boost is the fix for that so blocks uh the way it works basically is are blocks that are received in a timely Way by an node within 4 seconds of start of the slot get a huge extra weight in the get weight calculation and this extra weight also uh almost certainly makes the timely block the the head of the chain so that that's the one that that that I I'll build on I'll declare the head so this makes honest proposals Almost sure to build on the timely block that they received um and and this kind of principle is that a block that is timely should not expect to be reorg you know if I'm proposing a block in good faith I should not expect my block to be reared um I've got a link at the bottom to much deeper discussion uh of this but how does it change our um diagram this is the same block tree that we had before uh and how it changes things is that block G which is the most recent block you know time goes from left to right here uh if I received it in a timely way then when I run the fork Choice it gets another 40 votes 40% of the total uh testing weight in the slot uh added to it which in this case would make when we run our lmd ghost Fork Choice it would make G the head of the chain rather than e which was previously based on um attestation weight only so if you include propos a boost weight because G was timely it gets an extra 40 points and that makes it the head of the chain and that's the one I'm going to build on next um y 40 is great question uh it's had values started off life at 25 then it went to 70 then it went to 30 3 and that now it's 40 you can look it all up in the book uh and this is how it works in code I'm going to skip this because time is is pressing on but it's handled in get to wait um and uh in get proposer score this is where propos a boost is handled um then this is you get this extra 40 points if the block was timely uh and yeah ontick resets that so when I move from the current to the next slot then proposal boost is is reset um and so it no longer applies so just for the the duration of one slot if I receive the block for that slot it gets boosted and is much more likely to be the head of the chain and so this um the point of this was to mitigate the balancing attack basically cure that balancing attack and it's uh effective at doing that but it also turns out it has another good use case uh we can use proposed boost to Fork out late blocks um there was a time when it looked like well in fact nodes were demonstrably publishing blocks later and later the publish time for blocks was creeping even beyond the 4 second Mark you know five five and a half seconds into a slot um they weren't gaing many attestations but because they were received by the next proposer in time they were often built on uh as as the head of the block a head of the chain why would people publish blocks late well it's all about meev um and it's just having the maximum time to uh gather sweet order flow and um pack it into into a block um now late blocks destabilize the network um they don't gain many attestations and um this leads to confusion and forking and reduces the reward for honest validators who attest to an empty slot that turns out later not to be empty because the block was was late so we want we want to discourage late block publishing um proposer boost allows an honest proposer to build on the parent of a late block um and the honest proposers block becomes the head so as long as I propose My Block in a timely way uh then I can build on the head of the previous or I can build on the parent of the previous block uh and my block won't get orphaned and that's a um and there are lots of avat there and I I I won't go into it in in detail but the idea is we can use this extra power that a timely proposer has in order to Fork out late blocks and punish them uh and it's been great um it's pretty much cured the problem so uh Hallelujah uh last topic I think on lmd ghost um slashing so early proof of stake designs had a real problem um because nothing was at stake I mean it's stake was used as an anti-il mechanism but wasn't used for anything else um on there so um it was basically just to limit the number of participants on the network um because if there's but however nothing was really at stake there were no sort of punishments uh there and that meant that equivocation which is changing your mind or supporting two Fork simultaneously was Costless it's Costless for proposers to publish multiple blocks so why not build on every head I mean why why do I need to choose one I can just build on the all and I'm guaranteed to be in the canonical chain um and also it's Costless for validators to vote for everything they see um which is a rich source of attacks so the solution to this is slashing um vitalic proposed this uh in unbelievably January 2014 um with slasher we don't really use any of the slasher mechanisms now but the idea that you can punish stake for bad behavior solves the nothing at stake problem and actually made proof of stake a viable consensus Mech it's not a consensus mechanism but a viable approach to um uh Fork choice so yay we have a consensus mechanism um this is the summary I showed you before lmd ghost uh perfectly good consensus mechanism and you know if we were just using that ethereum would probably run uh fine um um and we could just uh crank out um blogs via lmd ghost and be happy with the world um but the idea is we can do better uh because we have two features under proof of stake we have a well defined set of participants we know who the stakers are effectively you know they they they have an onchain identity uh it's it's a closed set um unlike improv of work where anybody can be miny on the chain and we can do slashing we can apply economic penalties for bad behavior so uh how about we add we use these properties to add some stronger safety properties to our chain and that's where we get to Casper FFG but I am going to pause for breath and take questions questions on lmd ghost yeah so you mentioned that lmd ghost doesn't give us finality which is about what I think what you're going to talk about now but you also mentioned this confirmation rule that maybe people were working on where you could get stronger from lmd Ghost and it really sparked this memory of um like Vlad zamir and CBC Casper and I wondered like if that's too off topic no problem but I wondered like are those similar or is this confirmation rule that you talked about like a different in some way yeah I don't think it um owes anything to vlad's work and you can take a look um you may be more familiar with CBC Casper than than I am but uh uh you can take a look at the link um that I I put there and see how it is constructed um from my rudimentary knowledge of CBC it is not similar um to to that but uh you know there's a lot of information around on the network um to to guide us and so if we make some simple assumptions we ought to be able to do better than way 12 minutes for finality yeah that's the idea okay thanks uh hi Ben thank you for the talk and especially for the book uh could you please briefly explain uh justification why the protocol need needs it yeah we're going to do that that's next oh okay thanks yeah uh so in G head function we are using the Justified checkpoint to fetch the head of the chain so uh is it because lmd ghost doesn't ensure the finality and uh justification is happening on every block every Epoch not and final is happening on every twoo uh it's we come to that in the Gasper bit when when we get there remind me if I don't answer that question there perfect yeah thank you right let's press on still time is going quick more quickly than I expected as usual um we'll look at Casper FFG okay so lmd ghost is a perfectly good consensus protocol but we've decided we can do better so we have this Casper FFG thing um and the namey again is instructive I mean the Casper is a bit obscure um and random to be honest I mean it's just got in there for historical reasons I won't go into um FFG is a friendly finality Gadget so the friendly is Cas with the Friendly Ghost um but it's a friendly finality Gadget a gadget is what vitalic calls a kind of self-contained enhancement to another process so Casper FFG is a gadget for adding finality to an existing blockchain consensus me mechanism so uh it was originally actually planned for adding finality to proof of work as part of the proof of State transition that's EIP 1011 which Danny Ryan was heavily involved in um and it was so proof of work doesn't have finality and the idea was that Casper FFG could be overlaid on top of it with uh uh some bringing some elements of proof of stake to bring finality to the proof of work um uh mechanism however we have adopted it uh instead for um to overlay on lmd ghost and uh oh yeah the original paper is really nice actually it's it's worth a read I do recommend it so what are we doing with this time scales of Casper FFG it operates on the time scale of minutes Epoch so an Epoch is 32 slots um and uh 6.4 minutes so the time scale is you know six or 13 minutes um the goal is to confer finality on the Chain so this means that the chain will have a checkpoint and we I'll tell you what a checkpoint is later a checkpoint that will never be reverted so it will always exist in the canonical chain and all of its ancestors of course except at enormous cost so we can rever a checkpoint but if we do so then we have an enormous quantifiable cost uh which some Atta will have had to pay in order to do that and it uses a two-phase commit uh Alex talked about this I think in his presentation which is a classical sort of um consensus mechanism approach um so uh among validators having at least twoth thirds of the stake so we need twoth thirds of the of the stake to be honest as long as that's the case we can achieve finality um and we'll talk about that a little more uh and we waigh source and Target votes received in attestations but we only consider attestations uh that are contained in blocks for for this because it makes changes on chain um you know finality is updated on chain we can only consider uh attestations that are in blocks because attestations on the wire on P PTP or feral they they can be lost forever um blocks are forever though and it has properties which are analogous to liveness and safety um because it's an overlay it's not it doesn't have its own liveness in a way it's not producing blocks um but it has a sort of plausible liveness which means that Casper FFG can never get into a stuck state that is unable to finalize anything so in other words Casper FFG can always have a go at finalizing another checkpoint without having to uh Slash participants or um uh unnecessarily and it has uh an accountable safety property so if what less than onethird of your stake is honest then it's fully safe it it offers you um uh um classical consistency um but if more than one you can never know if less than or more than oneir your stake is uh adversarial or not uh but even in the case of more than oneir being um adversarial then uh it has its accountable SA safety uh finalizing conflicting checkpoints comes at enormous cost so we call this economic finality so th those are the properties of Casper FFG and we add them layer them on top of lmd ghost Casper FFG works on checkpoints so it relies on being able to see votes from the whole validator set um and that's too expensive you can't have every validator testing every slot uh that would be you know currently a million attestations every slots going around the network it would just overwhelm things we only have 3% going around the network every slot for that reason so what we do is we actually accumulate votes across an Epoch so we have 32 slots and we do you know 3% or three and an eigh per each slot and then by the end of the epoch we've got all the votes for that Epoch because every validator has voted once uh and then we do the accounting for uh C for FFG at the Epoch transition so this is a solid line um on the right and at the beginning the epoch transition to the next Epoch we we add up all the votes and we figure out uh have we got Justified and finalized checkpoints so a checkpoint is the first is the Block in the first slot of an Epoch so we can see at the bottom the checkpoint object it contains an Epoch number and it contains the root of a of a block so it refers to a block in the first slot of anoch if there's no block there then it will be the block that's most recent on that Branch um is the checkpoint so every Epoch has one checkpoint um in there uh unless it's branching unless it's forking in which case it might have multiple different checkpoints but it's always the first slot of the epoch and validators vote so this we're going back to our attestation data uh We've looked at the lmd ghost vote uh and at the same time validators make an FFG vote and they vote for a source and they vote for a Target and these are checkpoints uh that data structure that we we have here they vote for um Epoch and a block route so the source is the highest Justified checkpoint I know of so in my store I've got the best Justified checkpoint I know the most recent one uh and so when I make an attestation that's my source vote is for that checkpoint there this is Justified and the target is whatever so assuming we're in in Epoch n plus1 the target checkpoint is the checkpoint of um the current Epoch so um I vote from source to Target and we can imagine it as a um as an arrow sort of curved Arrow here is is is my vote and if I see more than two 2/3 of validators by stake weight making the same Source Target vote then I have what's called a super majority link so a vote is a link between source and Target and a super majority link is when more than two thirds of validators have made the same uh vote uh for that so this this gives us this sort of two-phase commit uh and this is a bit tricky and I go into it in much more detail in the book and we haven't got time really to go uh into a lot of detail but basically what we're saying is um justification so my target vote says my preferred branch has current checkpoint X you know whatever the checkpoint of my current Epoch is then if I hear that two-thirds of the validators agree with me so they I get votes I receive attestations with Target for the same checkpoint from two-thirds of validators I can then mark x that checkpoint is Justified so I'm saying uh when I make an attestation I'm saying I like X as the target vote uh and I'm receiving from other validators uh their attestation their target vote and they're saying they like X as well so as long as I get more than two-thirds of them agree with me then I justify that checkpoint X my source vote is saying I heard that so it's it's repeating reflecting back to the network what the network has just told me and this is why we have two phases and it's you know um classic part of the the mechanism and and very hard to explain but uh um you can you can read more about that um my source vote is saying I heard that more than two-thirds of you like X so I Justified it and you broadcast that to the network this is my Justified checkpoint if I hear the two3 the validators also said that they Justified checkpoint X I can I can mark it as finalized now now the point is that when I justify a checkpoint that is my commitment never to revert it I effectively it's locally finalized for me now it's not fi finalized globally because I don't I don't have a global view but it's it's finalized Loc for me because um I've heard that two-thirds of validators out there um like that and so I commit to never reverting that justified checkpoint now they might be lying to me which is why I need to do the second round um and finalize it if I hear from two-thirds of validators that they have also Justified it then we have this kind of global property that that checkpoint then becomes finalized um in in the protocol and it's a little more subtle and I'll show that in a moment but um effectively justification is my local version of finalization I will never revert justification as a node because basically I'll get slashed for it so uh you can think of it as finalization is just a kind of cherry on top it's not a big deal uh as a note the key thing is justification and um that might answer uh I think it was Raj's question uh earlier now there are Casper Commandments uh I'm not allowed to make a double vote so I can't vote for the um same Target uh in two different attestations so if there's a li chain I can't uh vote twice for the same Target in Epoch 3 here that's not allowed um and if there's a fork in the chain I can't vote you know for the same Target uh there so same Target from two different Justified um checkpoints on on the linear chain or different targets at the same height on a forked chain you know with the with the same Source vote there those are no no uh and the other commandment is no surround vote so that's what it is you know technically at the top the technical statement but surround votes easy to see you know if um my source and Target vote surrounds another vote that I have made or is surrounded by another vote that I have made then I also um that that's also against the Commandments that's also forbidden um so on a linear chain it looks looks like uh the diagram on the left uh the diagram on the right is interesting because this is actually what happened on hsky uh after the upgrade uh briefly the bottom Fork was the majority Fork which was faulty they they had the wrong um deposit contract address or un configure deposit contract address but enough of them forked off with that that they were able to justify a checkpoint uh on the faulty chain um however they are not allowed to rewind justification because that would result in the making a surround vote because the chain has moved on uh so all of the validators if they want to try to rejoin the correct Fork which is the minority top Fork um they will get slashed and we we'll come to slashing in a minute they will appr bro can this commandment shall we say for now no surround vote so there there's a nice juicy real life example uh of of this rule in action um and I've spoken of the properties of Casper FFG um there are pro of accountable safety there proofs of plausible livess they're very nice proofs actually quite simple um and um I've linked to them here um in in my writings though you can also see them in the original Casper paper um and there is one further rule that is required that uh we only finalize a checkpoint that is direct child of a Justified checkpoint so in the top diagram uh Epoch n checkpoint is justify Epoch n plus 2 because we have a super majority link from n to n plus2 but Epoch n plus2 checkpoint don't finalize in this circumstance and if you work through the proof of accountable safety you you'll be able to see why it just becomes um glaringly obvious um so we only justify we only finalize a uh checkpoint when it's direct descendant the the the checkpoint in the next eoch is Justified and finalization but justification is the most important property finalization is just a kind of nice cherry on top and of course everything in ethereum is more comp licated than you hope it is um so actually we're doing something more sophisticated than this called K finality but you can read about it there but this is the original Casper Vision uh slashing um what we do is that if either of the Commandments are broken by um uh a validat one or more validators we slash them uh and this guar guarantees that if more than onethird of validators are faulty or adversarial so they finalize a conflicting checkpoint um then at least onethird of them must have broken a commandment and we can detect it and we can prove it on chain uh the code is is very simple you can check it there um and the aggregate cost to an attacker if it's one attacker doing this would be at least a third of the total Stak uh ether which is what um 10 million eth or something uh at the moment um which is you know vast even today's prices uh so we have this economic finality property a quantifiable cost of messing with finality even if the attack has more than a third of the validators and this is this is an amazing breakthrough um if under classical consensus um mechanisms more than onethird of the replicas validators are adversarial um all Bs are off I mean generally they they they halt or go Haywire so um being able to put a cost to attacking the chain uh is um very powerful okay quick pause there that's Casper FFG we're going to look in the final section it's only a few slides uh at how they're combined into Gasper but I will as promised just pause there for questions about C Casper FFG and I'm I'm aware that I have gone through it swiftly so apologies for that yeah let me press on just a few more slides um Okay so we've got these two we've got this gadget and we've got this consensus mechanism lmd ghost Fork Choice R we've got this finality Gadget how do we put them together this is gper uh um inelegantly named um and basically uh it looks like this we have um our usual blockchain block tree going from left to right uh the first part of it is finalized by Casper efford we have a finalized checkpoint everything before that checkpoint is guaranteed to be linear we have one canonical chain and a very nice clean history any branches off that chain have been pruned out and are um lost in the midst of time uh but in the more recent Past after the most recent finalized checkpoint and you know there's an argument to say Justified here I won't go into that but think finalized for now uh this is the domain of lmd ghost uh and it operates um for more recently last couple of epochs since we last had a finalized checkpoint so everything up to about two epochs ago is nice and clean and then from twoo ago we have lmd ghost uh operating and how do we we combine them together practically well we adopt uh we apply Casper ffg's Fork Choice Criterion we follow the chain containing the Justified checkpoint of the greatest height so the highest Justified check so LM pure lmd go starts at the Genesis block a would be Genesis but if we apply the Casper FFG Fork Choice then the root of our lmd consideration is uh the highest Justified checkpoint that we know about um the highest finalized would be earlier than this but we've already committed as a node never to reverting block a because it's Justified so we can start there there won't be any other branches that are interesting that we need to explore so that's it that's basically it when we run lmd ghost we just start from the highest Justified checkpoint um and that explains where this comes from we looked at get head function previously um and this is the route where we start uh our search when we when we run the get head algorithm we apply the cast refuge for Choice fruit and there it is the world if that were the end of the story which it isn't unfortunately this is what it looks like in reality um how lmd ghost and C Refugee actually bolted together is ugly um vitalic agrees um um and you can look at the quote from him uh later um I skipped over this get head has this mysterious function get filtered block tree where does that come from that's not in the Casper FFG paper anywhere um what it's doing is removing from Fork Choice consideration by removing from lmd Ghost consideration any branches that had nonviable heads what is a non-viable head so this is um so here we are if we start with a Justified block this is the second block from the left here we've got our block tree which we're feeding to lmd ghost but before we run the getthe head algorithm we're filtering out some branches with nonviable tips um and removing them from consideration uh a uh block is nonviable if the state that's associated with that block does not agree with my store about the current Justified and finalized checkpoints so the the block is associated with state it was built on a particular State and it contains the data that transitions to the Post State when you run the block um and that assumes a certain Justified and finalized checkpoint um but it may be that that state associated with that block doesn't agree with the state that I have have the view that I have of the world uh and so we have to filter it out of the lmd ghost Fork choice and this resolves a finalization deadlock issue that was found which is a failure in plausible liveness one of the properties of Casper FFG and basically lmd ghost and FFG have a sort of conflict um that they're hard to kind of wrestle into conformance with with each other uh there so we filter out um is with unviable tips uh from the uh from consideration by lmd Ghost and so that was all very nice for a while and then we discovered another problem uh which is unrealized justification and finalization so where this stems from is that what we do the way we accumulate Casper FFG votes is gradual we accumulate them slot by slot and it could be that by 2third of the way through an Epoch we already have enough votes to justify uh this checkpoint uh checkpoint n so um but we won't actually mark it justified in the state until we get to the end of the epoch when we run the state transition uh the epoch transition so we call this an unrealized justification we've got enough votes to justify it but we haven't actually Justified it yet because that's only done at the end of the epoch um so that can happen you know when when you've you're two-thirds of the way through the epoch slot but around slot 21 and this led leads to an unrealized justification re attack which means that the proposer of the first block of the next Epoch can Fork out up to nine blocks from the end of the previous Epoch details are in the link um then another attack was found whereby an adversary can reorg arbitrary numbers of blocks from the start of an Epoch um and the issue with this is that the filter block tree function was being too aggressive in filtering out potential heads of the chain um because it didn't account for this unrealized justification so um blocks that would match my checkpoints if they had gone through eoch processing but hadn't so it was unrealized um were marked as non-viable whilst actually near an Epoch boundary they were viable and so and they were removed from Fork choice and this lead reog and so on so this is really by saying uh you need to follow the links to really kind of dive into this and it's very hard to reason about um but really by way of saying that this night's neat combination of of Casper FFG and lmd ghost is actually a mess um the answer is single slot finality yay and that's the end of the presentation Let's uh you got about 10 minutes if you want to ask me questions [Applause] hi um am I a yeah we hear you all right nice uh so I like to understand like because so far the upgrad have been very I guess similar I it sort of BS in together one on top of another but in this single stock finality I like to understand like practically how to uh the protocol actually um does a I guess a consensus for because the execution f is quite simple you just drop off the transaction ultimately but I think the consensus fult is something is very hard to implement uh uh practically yeah yeah it's good question and I I don't think we've we've really thought through the the mechanics of it yet I suspect that it will get punted to Justin Drake's effort on what used to be called The Beam chain which is changing name now so just as we built the beacon chain alongside proof of work and then we shifted the execution layer over to the beacon chain um Justin and teams are working on what used to be called The Beam chain um I don't know if it has a new name yet uh which is really the next upgrade in the consensus layer next massive upgrade and uses a lot of ZK proves and so on with a maybe five-year time Horizon potentially that could deliver single slot finality um and we'd do the same we'd do another merge we would take the execution layer off the beacon chain and then put it onto the beam chain um and run it from there so rather than upgrading the live consensus mechanism um we would kind of lift and shift the execution layer again we don't once we can do it again I see thank you well I'm amazed you got a lot of stamina people um feel feel free to ask me about any uh non-consensus stuff if you want to B A Light Relief as well having to talk about uh teq or uh other things yeah I'll catch you um yeah I wonder if Gasper um has been changed through the hard Forks or or not or it's stable and solid yeah it's good question so there are some aspects that we um so Fork choice you don't have to upgrade through a hard Fork generally because it doesn't affect the state transition so um you can more or less roll out changes um whenever you like so when we implemented propos a boost for example clients rolled it out kind of when they were ready and um just generally um you know within a few weeks of each other um it didn't affect it didn't break the chain until it did uh and I've got a section on this in the book it did actually lead to a very long reorg because some clients were using proposal boost and some weren't uh which doesn't technically break the chain it's not a hard Fork um but it um you know caused a certain amount of of chaos because long long reorgs are not desirable so in principle we can do uncoordinated upgrades for most of this Fork Choice stuff but in practice um we uh try to coordinate them not as hard fors but as um you know very tightly um coordinated upgrades between the clients just so we don't risk having another long Rog does that answer your question yeah so it means that uh phase zero uh is the basic spec right for fork Choice yeah and it phase zero Fork Choice spec kind of keeps getting updated all the time so um it it even though it says phase zero it's not the fork choice that we started with is more or less the current Fork Choice um and it and changes to uh Fork Choice like the filtering function and so on get kind of back ported into the phase zero spec I'm not sure what the rationale is for that presumably because they're not associated with hard Forks they've been done outside of hard Forks Maybe thank you yeah so in one slide I saw that the init Epoch was justified and the in in two start Epoch was justified so what happens to the N plus v like can it be justified later what happens to an unjustified Epoch um when um when an Epoch becomes finalized when a checkpoint becomes finalized and implicitly everything prior to it becomes finalized um uh as well so it's a bit of a technicality really I mean in principle when an Epoch becomes Justified everything before it is effectively Justified we just don't call it that um because we have to track whether the direct descendant of a Justified Epoch is Justified in order to declare it finalized so really what you're looking at is the finalization and that that just effectively finalizes everything before that that that finalized Epoch um but yeah it's not really a useful to think about the state of um um EPO before the last Justified one they just they are what they are no I don't like I didn't get it like what happens to that imp plus unjustified yeah I mean it just doesn't matter it just sits there I mean you know the chain moves on and we try to justify new uh new epochs and eventually we'll finalize something and everything prior is is is marks has finalized because we're always dealing with the highest Justified you know our source vote always comes from the highest Justified Epoch that we know about so we don't care if the previous Epoch was justified or not justified it's just irrelevant to us um so we're only interested in ever interested in the highest Justified Epoch that we we know about hi uh I like to ask uh base throwups versus sequencer base throwups which will you prefer and why oh man that's a good question um yeah I'm too fresh from my optimism experience to be objective about this I need to uh I I I need to spend more time uh looking into the whole Bas rollups and Native rollups and all of that scenario so I'm going to dodge that question um yeah [Music] sorry thank you yeah I saw that there are many implementations of the consensus client but what they differ mainly is the language that they're implemented so is there any other difference in them or it's just that yeah I mean it's quite interesting being I was like product owner for teu for years right and it's quite interesting being product owner of a of a product which effectively has to be identical to all the other products you know the the behaviors cannot differ in any meaningful way from between prism teu Lighthouse um you know load style whatever because you um otherwise it would be a fork so it's kind of a bit of a weird role um they tend to differentiate on things like you know amount of dis space used um teu we work really hard on the RPC interfaces so that it could be used by infura and others um so they were super uh responsive um so I tend to differentiate on features but the the the base base product the state transition function and all of that you know the gossip and um uh all of those things uh basically identical between the clients um so yeah um reliability up time and so on um you're always looking for that little Performance Edge but uh fundamentally there's not that much to choose between them these days so hello uh hi yeah just I wanted to ask about uh like uh there is this two round of voting uh in the slide number 40 like you said like uh the proposal gets like I like X I like X then he says I heard your like X then like uh they all reverted and then it's finalized so this makes it two stage process so like there is some material or like you could discuss about it like yeah uh so this is a two-phase commit so you can look at the original pbft paper um if you want the sort of more academic view I've written about this at length in in the book um uh you you can look at it it only matters in the most adversarial environments it's really really hard to come up with an example where justification is not the same as finalization and I've got one in the book but it's um very contrived you know is uh in reality justification and finalization are almost always going to be equivalent unless you're in an extremely hostile network but in order to prove things in order to guarantee finality you do need to do this two- phases you need to hear from everybody and broadcast that then you need to hear from everybody that they heard what you said and and double check uh everything uh you can look up the bisontine generals problem that covers this um which is the original work by Leslie Lamport and if you really want to test your reasoning about this stuff I put a link a cheeky Link in the EPS um um GitHub announcement of this call to the blue eyes problem uh right at the bottom of that page uh if you spend a couple of hours thinking really really hard about that problem uh if you manage to solve it then you pretty well understand consensus mechanisms I would say you know you will get deep insight into this two-phase commit thing um so recommend that if you if you want to kind of get deep knowledge on on what's going on here yeah that's great thank you [Applause] yeah thanks all uh good luck with uh with the rest of your studies and I look forward to seeing you on PO death calls in years to come it's been been pleasure thanks Mary thanks SP bye record