Danksharding | Dapplion, ChainSafe | ETHTaipei 2023

me so I'm Dublin cordep at lodestar and today we're going to talk about scaling so yeah as you know we are putting a lot of effort in scaling I think it's the most important thing we have to work on and today I'm going to talk about proton charting what's it about why is that important and is Proto dang sharding really only a meme or not we're doing this for real just stay tuned so yeah why brother why are we putting so much energy and scaling why shadowing at all why Roll-Ups why is this all important so the key thought here is that we all want ethereum to be a maximally useful with the maximum throughput possible and secure chain to do that we want to have a base as large as possible that can run nodes so we limit ourselves the requirements to run a node in these fixed issues Red Line what this means is if you want to serve the entire population say people buying coffees this doesn't scale we cannot have everyone processing everything so that's where shadowing comes along and okay trade-off I'm just speaking here an icon doesn't have any meaning but we could just keep increasing the requirements of running a note that would mean that only a tiny majority of well-capitalized individuals will be able to run nodes and everyone else would have to trust the majority in case there is a disagreement you have two options you can either just drop the security of the chain and follow the majority or stop and manually resolve probably you just go to crypto Twitter there will be some huge arguments someone claiming this is right this is right which is not a not not great it doesn't scale so what we want is something on that unicorn area where we have high throughput but still keep reasonable security by having everyone run nodes so going back to this example the key intuition here is we won High throughput chains we want scaling but what if we automate this step what if we automate the Stop and Recovery and that's where rollaps come from that's the key framework so optimistic rollups they automate fraud proofs you can this you can automate dispute resolution and then with validity Roll-Ups like ZK there is no dispute resolution because correctness is asserted immediately so that's fantastic because we have a one of n trans assumption which is quasi equivalent to mainnet but we can go crazy with throughput but there is a catch if data isn't available the whole system collapses and I'm gonna explain why but it's the most important takeaway I want you to get here for optimistic rollups if data isn't available they can steal your funds not great for validity Roll-Ups if data isn't available you are locked out of the rollup that's pretty bad and why does that happen so for optimistic rollups the key assumption is that whenever there is an update on the network you must assume that someone is watching and will submit a fraud proof in case of misbehavior if there is no knowledge of the transaction input list no one got submitted no one can challenge it and then the security breaks for validity roll ups if you don't submit so if the transaction list is not submitted you know that the current state is correct but you have no idea what the state is that means that you cannot create more blogs you can also mini transactions you are stuck on the roll up so key thought here if no data availability no roll up this whole thing collapses cool so who needs to perform this duty this task of ensuring that data is available like a naive thought would be if we are offloading computation to roll up participants the same thing we can do the same for for data right like we can have some specialized actor or something that keeps the data and proves to everyone that they data is there but no everyone has to check for data availability um and the the thing here is the data availability problem that the point is you cannot prove that not publishing data is a uniquely attributable fault so in this example if the publisher is small issues and holds the data and then someone else raises the alarm about the data and then the publisher releases the full block or the publisher is benevolent and then the policeman raises the alarm falsely you have no way to tell these two scenarios apart at T3 and this breaks any design that we could build to have like something like fraud proofs or data so everyone has to check for data cool so what we do today we we have data availability in ethereum that's what Roll-Ups are using they just take the data and throw it on the Block and we are here at idea one just everyone downloads everything it works it's safe it doesn't scale so next step naive knife assumption cool okay let's make everyone those action of the thing so uh you do probabilistic sampling you take all this data and you just check some some parts uh the problem is that this isn't safe if you do it naively because even if you hold withhold like a tiny fraction of the data it breaks all the assumptions you have to be sure that a hundred percent of the data was made available in order to guarantee roll-up safety so this is where crypto comes to the rescue and what we do is we check the data is available but not not data but the extended data that means that now you only have to prove to yourself that 50 of the data is available if that's true you can just recover everything later and you're safe and now if you do like 20 samples you have I think it's like one in a million probability that the data would be available and that's secure enough so where do we put all this data this thing Roll-Ups They Don't Really Care about this data and the VM they just want this data to be available somewhere for a fraction of time so blocks are really inefficient because we keep them forever they are available to DVM so we need something else something new that is temporary so we keep for weeks not available to the ABM so you really only pay for bandwidth and that's where blob space come from and we have this new thing this new The Blob data that the ethereum doesn't care to interpret or what it means it just like smashed into the block with a commitment so it can be referred now why are we doing all these kcg crazy crypto stuff the thing is we are using Merkel trees today in ethereum but they do not allow to do this data recovery things and if you remember to do scaling safely we need this Azure encoding extending data magic thing otherwise we are unsafe so we have to use polynomial commitments because they very very nicely allow you to do these days they're recovery and the best network we have today is kcg so that's what we are doing cool so the final Vision um two to three years maybe a bit optimistic I hope that would be awesome but we need scalable Network datability sampling and blobs with kcg problem this thing is very hard this is really a really really difficult problem that researchers are working on but it's not solved yet and roll labs are crying they want scaling now or at least something so we could we could do the blob thing like this thing of happening having some data that it's cheaper because we don't keep it it's not on the VM we could do that today with Merkel routes or some other simple commitment scheme the problem is we would spend like a year developing this and then just throw it away because we need the kcg eventually so why don't we just do the kcg now it's it's technically stupid like we don't need it we're just gonna spend a bunch of time building Something That We're Not Gonna use but we are saving ourselves from having to throw something and that's exactly what proto-dan sharing is about we're just doing the blob thing first with expensive crypto because we don't want to have to do it later cool so what prototype sharing really asks the network to do is to download an extra bytes per block it doesn't sound like a lot but it is a lot just keep in mind that everyone has to have this thing this blog best blob before for seconds in order for the network to be stable since the merge the slot goes pretty fast you at zero seconds are expected to produce a block then you have to propagate this block to the network then your peers get this block have to validate it both consensus and execution and then you set it as head if you do that later than the four second Mark you consider that that block is missed that's what we call orphan and then you vote for the previous block that has all sorts of consequences uh it it's unstable it allows for yours to happen more easily and more importantly if you vote for the wrong thing you get less rewards so say that if propagating the blog or validating the blog happens faster for some actors like big operators they will get more rewards and that will be a centralizing factor because we will the incentivize solo stickers and it's even worse now because now we have Meb boost we introduced that after the merge and it's already pretty bad so this little thing here like producing a blob that when I did this chart I thought it's like I don't know half a second now it's like a full second or even two which is bad so we are already at the at the break of the four seconds so the key here is how can we safely transmit these blocks over the network in the most efficient way possible so we can package the things in different strategies we could just the whole thing together like Block Plus blobs we could partially couple all the blobs together the block separates or just full decouple so the the the thought here on the signing to couple or not it has two different sides so we have the historical sync which is not that important for today and then the real-time gossip which is the one I was talking about we want to have that before the first second and the initial spec what we did some months ago when all this started we just we just didn't had enough data so we just went for the simple way no complexity couple everything together we don't have to modify the evaluator client or that signatures it was nice and yeah some very smart people are doing experiments which I cannot disclose yet but it's bad it's not ideal uh it's it will very likely lead to network degradation so we went to the mountains in Austria and this handsome man just said okay let's do it let's flip the blobs like we have to do it so yeah sad it's there they are separate now and the the intuition here is you like again 512 kilobytes it doesn't sound like a lot but it is a lot let's assume on a solo sneaker you don't have a lot of bandwidth and this bandwidth is shared with Netflix your TB uh Facebook whatever also the note is doing other stuff it's handling attestations and other messages so assume you have 10 megabits only for blobs and you have to stream one megabyte which is a reasonable Target that we will go four in one year that takes a full second and if you have to do that for eight years it stretches your bandwidth even more thin so if you couple you have to send the whole thing and then when it gets the next Pier then it validates the thing and then you have to do it again so the the metric of time to Total propagation is very bad but if we can split and have a way that we can send a chunk and then validate this chunk and then forward it's much more efficient from an outload bandwidth usage so what this is what the the chart below and that's why we are freeing the Bluffs so Anton from Techo did some great simulations uh and yeah the results are fantastic so if with the couple we shorten like by half and even for a more realistic scenario where we have like Fast nodes and slows notes you cannot see it but this is full of blue and these people are wrecked but if with the couple then they are kind Okay so tolerable also interesting he found that flood publishing is not good and so when when you are in in Gossip in the current protocol we usually only forward the things to a small societosphere but when you publish we just send to everyone and that's thought to be better but it's actually not and the reasoning here is that if you have a thin bandwidth and you have to spread that message the first hop if you broadcast to everyone is awfully slow so the exponentiality of networking kicks too late but if you just send to eight then you have the exponential kicking in faster and that's why it's better so going to today's spec that's the topics that we're going for we have one Topic in Gossip where we send individual blobs they are signed and then for request that's what you use to recover when gossip fails or either you want to sync so here in Gossip I'm using the notation we do for gossip sub 1.1 spec if if it says reject it means that you do not propagate and you penalize the beer ignore you do not penalize but you actually you don't penalize the beer so here we send wall form timely and finalized from the known chain blobs the third point is important we now check that they have a correct proposal signature so that means that now we have to change the validator client to sign not only the block but every single blob individually that allows to have content integrity and not have to do any kcg verification on every hop so we spin up propagation you have to do that eventually but you do that in the four choice and then the fourth one is that you only propagate one single block comma index Tuple for blob that's spicy because it means we don't have any slashing condition for blobs that means that a proposal that is malicious it could sign conflicting blobs and attempts to split the network because it would send say to the left blob one which is the correct to the right blob two which is incorrect and then they too could not come into agreement what's correct because the blob is missing however with the request we can recover you will not get it from gossip but when you get the block you would realize oh I'm missing the blob number two so you would use the buy root to recover and then you are fine so that would cause to add latency in the network but it's thought to be safe because really it's in the blog producer interest to broadcast everything as fast as possible otherwise the block is missed and in reality if they want to they can just completely miss the block today uh and the penalties that they don't have any reward so for for recrest you can't request by root you use the Temple Block Road index you are supposed to use that to recover from these problems in Gossip when you're missing blogs as a backup and then you use the range to sync from like when you are offline so an interesting point here that I think it's not talked that much is that blobs are only exposed up to the retention window this is again from the very beginning if you remember we don't want to keep blobs forever we only kept them for like two weeks something like that so if you today stop your node you go on holiday and then you turn it back again after two weeks you will not be able to sync because you are from behind the retention window and now you are trying to sync blocks that have blobs and you don't know if those blobs exist or were not available and the P2P layer has no mechanism to give that to you so now you will be forced to use a weeks of GTV checkpoint sync to kind of tell the note I have manually go somewhere to Twitter or Discord and checked that data is available so trust me it is yeah you can jump into the green area and continue doing your thing and that also applies in case of reworks if we have a deep reorg that goes beyond the retention window your node will get stuck but this is very unlikely and also this is not really a problem today because the weeks of gt3d window is kind of on the range of this but something we want to do in the future maybe is like shorten this window to relieve the notes of having to store so much data if you want to go like Beyond one Meg of blobs to Mega blobs and this can be a problem so so yeah it's been an awesome progress we have come from very far this all these things started early last year now we are on definite 5 where we are actually implementing all of these I'm talking today the three develops initiative and yeah the spec is easy but now we are shifting all the complexity to the implementation so it will be about just fixing all the details but I think it's worth it because ultimately this is gonna remove a potential centralization Factor on the network by putting everyone on the level fields and yeah thank you so much for your attention [Applause] [Music] oh and uh okay so it says for a Dev dying give us a amazing talk about Den shouting so anyone here it's on to um have some questions then we can I think we can accept two or three questions right sure okay maybe I will pause first one so yeah and last thing so I'm the lead at lodestar we are a consensus client we are the last one to join the the launch pad so if you care about nine diversity please run load star join the Discord we will help you it's awesome and yeah thank you so much so since 10 shouting is a it's a puncture of a research so the learning curve is pretty high so do you have any uh suggest reading or a video that you can recommend to our audience yeah with with any Topic in ethereum if you want to learn about it just put vitalik space the name of the topic and that's usually the best material uh yeah we just do that he has great presentations on YouTube and everywhere uh very well explained and yeah I mean just take it with time it's com it's complex but it's not yeah it's not that bad and also since the eip4844 is coming so in uh your perspective so what is uh progress if um or how long have we achieved to the attention in the future so I would say now we have the two splits the consensus layer and execution layer the consensus layer it's I would say quasi complete we are now finishing the P2P layer like what I'm talking today about the free develops the execution folks still have a bit of catch up so I'm optimistic end of year hopefully could be a nice Target anyone has a question foreign the uh the the Thanksgiving uh upgrade happened like uh how what should I change about my my Hardware or or like what should I pay attention to yeah so with Thanksgiving you have to expect that you would be needing more bandwidth and a constant uh amount of of this extra the disk doesn't have to be SSD it could be a hard drive and I think it would be about 200 gigabytes extra during it and then bandwidth we'll see but yeah maybe like between one and five megabits when we're starting but besides that should be good thank you very much who else want to ask questions so just for understanding is these changes are they only going to be consensus layer relevant like adding these uh blobs there or is it also going to be an update on the execution layer yeah so the the EIP itself affects all the layers and literally everything but the blobs itself they would not be exposed to execution the execution layer only gets the hash of the commitment and that's all they need any questions all right okay stands for the amazing talk thank you thank you