Danksharding, Layer 1 R&D - Day 1

[Music] i'm going to give a quick overview of denk sharding there's a more in-depth session this afternoon for those who want to learn more in detail about it so data charts this is the idea of data charts is to provide data availability for rollups the meaning of the data is no longer defined by the consensus layer but instead will be defined in the application layer and ethereum based layer consensus in the future will have no responsibility other than ensuring that this data is available and why have we changed this from the original plan several years ago which was execution charting well we know that roll ups are about 100 times more efficient than native execution so it's hard to see that execution charts would ever be in high demand so our goal is to provide about 1.3 megabytes per second of data availability with the full charting rollout and this is about 10x the current max capacity or like 200 times the typical capacity of the current ethereum base layer and this has been the ethereum target since um roughly late 2019. um how do we construct such a data availability layer um well the basis is a technique called data availability sampling because in order to scale we need some way of ensuring that data is available that scales that doesn't mean like that means that full nodes do not need to download all the data so in some way we need to be sure that o of n data is available using just of one work and the idea is behind this as we distribute the data into n chunks and each download each node only downloads k randomly selected chunks so this is the idea of data availability sampling the client picks a few random chunks and downloads those instead of donating the full data but we need a way to deal with the case where some of the chunks are missing and those might not be the ones you're sampling so if you just do this that doesn't give you good security because one chunk missing could still be very important data so the only way that this works securely is um if we employ a technique that's called erasure coding so what we do is we have to extend the data using a so-called read solomon code and that's just a fancy word for polynomial interpolation and so what you do is you take the data you put a polynomial through the original data which is indicated by the blue boxes here [Music] and you would you evaluate that polynomial at the same number position of positions again and then like using elementary polynomial maths we know like with any um uh like for example in this case it would be polynomial of degree three because we have four data points and then with any four of these data points no matter which ones they are we know that we can always reconstruct 100 percent of um of all the data so we can know what exactly the polynomial was and this means we have uh we have changed the problem from from ensuring that all these chunks are available to the problem that knowing that the 50 at least 50 of the chunks are available um and this is the basis of why the sampling becomes efficient because for example now acquiring 30 random blocks is enough to make sure that the data is uh the the probability of the data being unavailable is less than 2 to the minus 30. so that's that's really nice um but yeah one of the problems this poses is now we need to know that this encoding is correct and um through the rescue comes what ben just introduced kcd commitments so the way kcg pro commerce must solve this is because kcg commitments can only commit to polynomials um so uh if we did the same thing if we tried to do this with merkle trees we'd always have this question while now like you have a merkle tree of uh some data but what's what if someone just didn't encode a polynomial but instead just encoded random garbage and um and now yeah you can you cannot reconstruct it right so everyone everyone has different samples and um what they reconstruct is not the same data so you wouldn't you wouldn't have consents on what the data is um and so uh you can solve that with polynomial commitments because then you know that everyone will always get the same polynomial and everyone will always be able to reconstruct the data um so what what has changed with what's now called dank sharding so the um the original sharding proposal as we had it previously was based on having lots of parallel running shard chains and that would all have independent proposals so each chart has a has a proposal and then there's a committee for each of those charts that um does sort of a first pass check on uh on whether that chart data that someone proposed was actually available and um the problem with that construction is that well ideally you want to know in the next beacon block um as in like the next for example that would also contain the next um ethereum l1 execution block um you would want to know whether that data was available or not so that you can start building it for example like the roll-up can include the new state route um however um with uh with all these different proposals um this is hard to achieve because you would have to rely on being able to get all these messages all these words on whether data was available within one slot and that would impose very tight constraints that we are not confident that can always be fulfilled on the network um so with the new design that is now called dank sharding what we do instead is we have one proposer who proposes the beacon block and all the shard blocks and the reason that we can do this is that we nowadays we use a technique called proposal builder separation where not well basically normal consensus participants like for example current eth2 validators do not have to be able to actually construct blocks they can outsource this they can basically there's a bidding process where some entities that are capable of producing these blocks will make bets and uh um and the the is true validator's role is only uh to select the highest bidding among these and uh and select their block basically uh and the nice thing is now we have a single entity that's responsible for creating this whole block that includes the the l1 execution block the beacon block and all the shard data and thus we can verify whether it was available or not and aggregate like that wasn't possible before because here they're different proposals so we cannot simply uh rely like if we said oh if one of them is not available we just say everything's not available then that would be unfair to those who've actually done their work properly and actually made it available where versus if there's someone responsible for making everything available then that's fine like we can just punish them and aggregate if they fail to do any of their work um yeah and this is an illustration how this would work for proposal builder separation so what happens is you have a proposal block which selects the beacon block and that would be small and then like this actual main block with all the payload and and the chart data would be created by a builder in a separate block so one of the essential components of this construction is the two-dimensional scheme so we've had before ben introduced kcg commitments and and we we can use them to commit to polynomials and one possible approach would be to just commit to all the data that we want to avail make available 16 to 32 megabytes per block in one kcd commitment however that has certain disadvantages in particular if not all of the data is available then someone would have to interpolate a polynomial um of this big size like and that would be like a very very big polynomial and that's actually a lot of work and so in order to avoid this problem what we instead do is we we split the data into into a square of data and we commit to each row using one kcg commitment and then we interpolate we extend this in the rows but we also extend the columns so we compute um we compute again the same number of kcg commitments and now the whole thing is basically a two-dimensional polynomial on a square um and uh and that has certain advantages in particular it means that each row and each column individually can be reconstructed so like this in order to reconstruct the data you do not need one actor who is able to download all the data you only need actors who are able to um download full rows and full columns and so that's that's much better like some the the assumptions that we need to make on who is present on the network are now much lower um and um and this also has another nice advantage which is that um we we have a much better properties in terms of validation so like um what's uh what we intend to do is that each validator is responsible for checking uh two rows and two columns and we have the property with this 2d square that basically if at least 50 of the rows or 50 of the columns are fully available then everything is available like then you can reconstruct everything that's the property of this polynomial reconstruction um so basically um with this like let's say that um less than 50 of the rows and less than 50 percent of the columns are unavailable are available which that that would be required in order to like fail the data availability then uh then the probability that some that a random validator votes for this is actually really low it's uh two to the uh minus four so it's once 116. so any unavailable block can get at most 1 16 of the votes so it's under normal conditions it cannot pass it cannot become um the the canonical chain um and the second nice property of this is that we can use the validator set um for for reconstructing unavailable data so because we have overlap between rows and columns validators can can use what they sample in order to fill in the gaps so basically like if uh so if if you if any everyone only downloads rows for example and only 50 of the rows are available then basically we would have no way of filling in the remaining 50 um like basically they would always remain unavailable but because every validator has compute some like download some rows and some columns there would always be some overlap so basically if 50 of the rows are available then we can we can actually get all of the columns like basically like um it would be enough to reconstruct the columns and then from the columns we can reconstruct the missing rows so that's nice that provides um distributed reconstruction and finally what it also provides directly is data availability sampling which is the thing i started with so that's our safety against malicious uh majorities so even in case the validator set became entirely or like well a majority of it became evil and they tried to publish a um a chart block that's unavailable um they would not be able to get through with it because what full nodes do is they they pick these random samples and and check that they're available and in this case so it's changed before it was uh in on in a one-dimensional polynomial you need to check about 30 samples um uh because 50 is enough to reconstruct however um for a two-dimensional poem it's a bit different because you need to be sure that at least three quarters of the data is available so you now need to do 75 random samples um on this whole square um but that's still very a very scalable technique so for example if you compute the bandwidth required to do this each sample is 512 bytes so if you do 75 samples um in 16 seconds that's only an average bandwidth of 2.

5 kilobytes per second so that's a very scalable technique and very powerful yeah and yeah that's it that's a rough overview of what comes with tank sharding and yeah hope to see some of you on the in the session later today [Applause] [Music]