PeerDAS, Lion DappLion: Scaling Ethereum Today | ETHSofia 2024

[Music] thank you so much for having me it's a pleasure to be here to speak in my now Hometown I know it's after lunch so thanks for attending uh I'm a core developer and I love to do these presentations to expose what are uh ceps actually doing on ethereum and how basically we make everything possible for you guys the ones actually driving value to ethereum possible today's topic I'm going to face it for all levels if you are a beginner I'll try to get on board with you and today we're going to speak about scaling so just to get everyone on the same page scaling refers to increasing the capacity of the chain to have more transactions the way you do that is by uh achieving to lower their price so the chain becomes more democratic right now ethereum as you know it's a pretty expensive chain maybe not exactly now but through its history so only those types of activity that are highly valuable like defi are feasible to happen at the chain if you maned to scale ethereum to this uh meme dream of 10K TPS then any type of activity would be possible including uh social stuff uh or other other types of applications that don't have such a high transactional value so why scalability has been also always a challenge is for this trilemma I'm sure you have seen this chart uh you you cannot have the three of these and ethereum always prioritizes security and decentralization over scalability so to visualize why this triangle is so complicated let's run through the trade-off that the most decentralized networks as of today choose so we have Bitcoin and ethereum who purposely limit the cost of running a note that's to make hopefully as many solo stakers as possible run noes at home we limit how much bandwidth you're going to spend how much RAM you have to have how much dis you have to have and that puts a limit on how many transactions a chain can process at a moment of time if we want to scale we can choose a strategy of a chain I'm not going to name and just make the cost of running a node higher that allows to increase the capacity of the chain but at the cost that running nodes becomes more and more invisible if you extend that to the Limit only a small cartel of highly specialized individuals run noes and then if they want to do something nasty like 51% attack you basically have no recourse the only thing you have left is go to Twitter yell and be like hey these people are trying to steal our money we need to do something so you have to Hal and do a social recovery kind of like the Dow fork with it an ethereum so this is uh pretty expensive and ideally we want to automate that step on ethereum and that's exact L what rollups do rollups give you the certainty that the that the rules of the chain are not violated optimistic rollups they do that with these dispute resolution and economic games and validated rollups do it by construction with crazy math so rollups achieve scalability in computation but they still rely on data and I'm going to talk about what data is and why later but the key concept is that for a rollup if data so transactions or state diff becomes unavailable they basically they are fued so optimistic rollups you can steal everything because you can submit a transition that no one knows the step so this dispute resolution game is invisible for validity rollups you can advance the state of the rollup into a place that no one knows uh exactly what what what it is so you cannot construct a valid transition into something else so you can do a ransom attack like you would have fans in this ZK rollup and the only way to get them out is for you to give me half so it's not as bad but still pretty terrible so the key concept is ethereum's job now becomes to guarantee data availability and who has to do those checks unfortunately everyone and so why that's the case let's look at this other diagram which is this probably is the most important one I want you to take away from this presentation and this is to understand why data availability is such a difficult problem so let's run to two cases we have a system where we have proposers that that publish data and then we have slashers that raise an alarm if this data isn't available so on case one the proposer is the malicious one it hides data then the slasher raises the alarm and immediately after the proposer releases the rest of the data then in uh case two it's the other way around the proposer is honest but the slasher falsely raises the alarm then you as an observer at T3 you have no way to attribute fault and that's the problem data availability is not a uniquely attributable fault we cannot run math on this concept so we cannot have rules like we do with computation therefore that's why everyone has to check data availability because you cannot delegate these checks on anyone so execution we scale rops and data availability we cannot scale it in this uh immediate way of Delegation so what we do is to shart so we split the responsibility of who has to check which section of the data so the most basic thing is exactly what we do now which is everyone downloads everything if you run a note you would just download all the blocks and by default by construction you achieve these data ability checks the problem is that it doesn't scale that's the first image that we saw so the second idea the naive approach would be let's just check some of the data we'll randomly sample and check this bite of this block this bite of this block this bite of this block that's not safe because those terrible consequences of the rollups breaking happen only if one bite goes missing so this approach is not safe instead what we do and that's basically the the strategy of ethereum Celestia and all these strategies is to extend the data with what's called an aure code extension such that now you only need to have half of the data in order to recover all the data now when you your checks you just have to convince yourself that at least 50% of the data is available and then you can have certainty that everything is so that's that's basically the genius idea that unlocks everything and just to summarize uh like the way it's structure you can do really really small checks and due to the probabilities tending really down to zero we can sample tiny amounts of data and have the certainty that everything is available so the road M so far this is uh this Z years is maybe like 2023 we had this uh realization that we have to go into the rollup Centric road map but rabs wanted something now they need scalability now so the plan of the last hardfork was to introduce this cryptography as soon as possible while we figure out the actual Das which is the network component it's really tricky I'm going to talk about it now so RS wanton something now and that's why we gave them Proto sharding which came on the fork this March and it just uh the unlock was to introduce a separate market for data independent of gas and what that has allowed is that because utilization of blob data is under the target fees has been uh mostly zero for now and if you have used l2s you have noticed that transactions now are practically free and that's because of this a funny side effect I want to talk about is if you have been following the uh ultrasound meme that if is deflationary now because rollups are essentially not paying for anything because fees are zero now we have inflation again and I personally think this is a good thing like ethereum is not here to make e holders rich is here to make a useful system that everyone can can use so I don't know my take is that this is this is perfectly fine and we eventually want to go to a place of equilibrium this is the road map basically in a in a technical sense now we have no sharding so you can see that uh it's it's a one-on-one relation everything that the do Costo this is everything that the network produces what we want to achieve is a downward strend where the the actual load for the note the data that you have to custody is not a one-on-one on what the network requires and even try to Trend it down so that nodes have to custody less and less and less data so Pras is the first step that we are developing right now and it has a modest sharding coefficient the higher you go in the sharding coefficient the harder things get because you have to in a very timely manner figure out who has what retrieve it and verify it and do this thing called reconstruction and this is a lot of Concepts that are new to blockchain and new to ethereum so we are still trying to figure out how how can we do that in a way that's resilient safe and not exploitable today again sharding we have these blobs that's the the data that rups would post and what we do now with pias is extend them with the Ure and code that we talked before and we chop them vertically in this units that we called columns these columns are both the unit of custody and the unit of sampling so in PR the the key of this protocol is that we are not introducing any crazy new networking structure we are reusing components that we know and love and we have used for a long time so let's run down how it's going to work the proposer as today chooses a set of transactions some of which would be blob transactions these are the ones that include data from rabs then it does this uh extension this math then it chops them into columns and it's some of these columns are sent by gossip sa to the custodians that need to have them and then the second step is the sampling that's what actually give us the the sharding the Samplers would do checks random checks on some of these columns and again if 50% of these columns exist or they see them and are valid they consider that the whole thing is available and that's the key to scaling they only check some things and convince themselves that everything is available but there is a problem so the attesting deadline is a 4 second window today that a testers have to meet before they cast a boat and then decide if the block is available or not and this deadline is pretty tight what we have seen in testing since we have developed this this this feature is that some testers miss the deadline doing this sampling is not uh an easy task it involves retries there is network issues that that corise so some people get left behind and unfortunately those that left behind tend to be those that have not the best bandwidth or not the fastest computer so the solo stakers so this is kind of annoying because now we are introducing a calization Factor where we are incentivizing people to have good bandwidth and good Connections in order to have like not a penalty when doing sampling so what what Francesco who's an amazing brilliant researcher at DF realizes that maybe we don't need to do sampling and the realization for that is to distinguish these two concepts so Global safety is the key property that we all want to achieve which is is the block available do rollups are are rollups safe that's the most important we want to guarantee under all conditions and the second one is individual safety so for example you can you be convinced that the block is available when in reality is not that's individual safety so achieving individual safety is actually pretty hard and that's what we would get with sampling but it turns out that if we just have enough custodians coding enough data we can achieve Global safety with no sampling and that's that's the realization we're going to move the attesting deadline from sampling into cost so we only do the initial Distribution on the first four seconds and then sampling would be an optional task for anyone that needs additional safety so if you're rolling running like an exchange or some high value application then you can do sampling to get this extra level of safety on top of it so the the issue with this approach actually is the more data that we distribute on the first mechanism of custody is with gossip app and gosip sub is very efficient it's very safe but it's very expensive it has a 4X penalty due to the way that the mesh distribution Works So eventually we want to move as much load as possible from the first distribution to the second so we can achieve this scaling factor that we need otherwise in practice the bandwidth that nodes perceive would not be much more less than the status quo so that's the current design and it's technically not pieras that's just being very uh subtle but this is what we call subnet D because we are doing s like sampling but instead of through peers through the subnets of goip sub and I mean it's great because for me as a client implementor this is a massive simplification like I'm very happy with this design so Pier progress I updated this slide uh today because unfortunately um as of a month ago Pi do have been uh descheduled from from the next Fork which is spectra into the next next one which is fusaka so hopefully by Q4 2025 we could see this I think in the in the meanwhile for pectra we will probably just increase the number of blobs in the network without actually introducing sampling to have a little bit more scalability in the meanwhile um since then we did the initial interrupt and proof of concept in neota which was an amazing event held in Kenya and then we have been doing Deets um each one dead and active later uh again I think this has been proven to be much more challenging than we all expected devet just keep failing for uh no reason until we figure out um so so yeah hopefully by fusaka 20254 we could see Pi in mayet and even though we're still in definite this this this stuff is working we have working prototypes I think all clients are now catch up to the latest spec and yeah we're R DEET so here are some fun metrics to look at and that's until today so P again Pi is the first iteration but we want to do so much more when we to go to these uh very high levels of sharding like 4X 64x we need to do something even better and that's what's called the 2D um DK sharding this is the the final road map what we're going to go to but doing this in practice it's going to be quite complicated to to achieve it we need to have some new networking constructions that we haven't find a design that should suitable yet there is active research and we have a couple ideas that are pretty solid on how to achieve it but we're getting there the key for you to take away from this is if you remember from the original design we had vertical columns so a column is quite big in practice but these red squares that you see here they are tiny tiny tiny I think about 500 bytes so in the future with this if you want to get a data availability ass St you just choose maybe like 20 of these red squares download it from the network and then you can prove yourself that data is available so this is so cheap and so tiny data that you could even do it from a smartwatch and that's the vision of the future that you could have a smartwatch which is doing these availability checks it runs a snar uh verifier and then boom you have full nodes guarantee on a tiny device on your wrist so that's all thank you so much uh so first thanks for the talk it's really clear I'm more on the defi side of things uh you know for me it's like kind of the the wiard in the tows are doing the thing to the magic to scale etherum I'm at the application Level and it was really nice to uh to understand better what's happening at the network level uh my question was more about um transaction fin it and how it would be impacted by this uh as far as I get it from from your presentation essentially at least you need um the the P sampling to occur to have some form of finality so at least six minutes or can you can you explain on that yeah so finality is actually independent to this mechanism because what what can affect finality is if due to the availability checks the tester boots start to diverge and they are not useful then that can affect but when you vot finality you are doing FFG links which are lagging from the head so it's it's much more unlikely that they would be delayed or affected by the cost of of sampling this would be more of an issue of uh the tip of the head that would be unstable and we could have more reorgs but it shouldn't affect finality oh all right and so how in a major P do with all the implementation how what is finality looking like in terms of uh delays yeah so it's in inas finality would be exactly the same voting from a checkpoint to the other and uh when when you have justification and then finalization then yeah you have the same economic guarantee that you had before all right thanks yeah thank you hello uh you have an interest interesting concept but I was wondering since you're so young how how much did it take you to De to develop this concept so uh the concept of of sharding has been on on research for a long time and this final design came from uh danar Dan rat he's a a brilliant researcher from the theum foundation but I think many of these things um come from iterative research like the theum foundation have is thinking about scaling since since Genesis since 2014 so it has been years and years and years of pondering and going in circles and seeing what what can be done um even at the beginning the original roadmap was to do uh full sharding not only of data but also execution so you have independent block producers but that whole shenanigan uh it was it was attempted but it turned out to be so complicated that um yeah people started to look okay what can we do what can we simplify and then the key realization was that if we give up uh sharding proposing so we'll have one proposer that has to do everything which in a way it's centralizing but we can take that sacrifice then we can massively simplify the design and only shart data uhuh and and for me personally I I didn't come up with any of this um like all all of us try to contribute where we can but my role specifically is is a a client implementor so we basically take this design and implement it in practice and try to find as many optimizations as possible to to make it feasible thank you sure about data availability and I understand that this approach uh you know by having access to very small bits of the data gives you assurance that the rest of the data will be there so I understand that for this you're using erasor coding and based on that I was wondering what is the percentage of the data that have to be there so that the rest can be like reconstructed or you know like reproduced so sorry I I don't hear the audio super good so you're asking what what fraction of the data has to be there for if you have maybe if I close it it's better yeah so I understand that with rasor coding we you just need to have a small amount small percent can you hear me better now I'll try I'll try thank you thank you so with rasure coding I understand that you only need to have available a small amount of data small percentage and I was wondering what is this percentage so it be like 10% 50% 70 so in in the current construction it's 50% 50% okay and how based on you will picking very small bits of the data anyway to make sure uh that you have access to the rest of the data what happens how can these very few small bits give you Assurance for the rest of the data because it's just like 50% the one you have so again you are playing with a chances if I understand correctly that you will get the correct Parts mhm okay so you're asking basically how reconstruction works I just want to if I understand correctly you are only playing with a chances that you need to hit the correct parts of the file to make sure that the rest of the file will will be available yeah so in the in the 2D wall for example uh if you have exactly 50% you can do reconstruction but it's going to be really complicated because you have to solve a Sudoku so I would say the closer you are to exactly 50% uh the more difficult would be like even though it's mathematically possible it can be pretty complicated and we always work in the assumption that there are some actors in the network that have everything or have a large part of the of of the of all the data like in pieras we have a concept that's called super nodes where they have absolutely everything so for them doing reconstruction is Trivial because they always have everything for for 2D uh the goal is to implement distributed reconstruction so it's going to be this complicated procedure where someone has a column and then reconstructs this other column and then these people do the row and you have like this soku and eventually recover it mhm um but yeah like mathematically if you have half have the data you technically can do it what's then specified is how many rounds of iteration you need to achieve that exactly okay cool thank you very much thanks a lot [Applause] [Music] I'm going