Devcon 5: Ethereum Swarm Breakout Sessions - Ethereum State on Swarm

already hello everybody well I'm talking to you today about essentially fulfilling what was a promise from the very early days with like hearing was launching about the idea of let's use warm to stormy dream state my name is Piper Miriam I work for the dairy foundation I mean a team of Python developers and one of the main things we work on is a client for both the eath one and two and ever since Kennedy's written in Python we have pythons like the second best language for everything um and we have had a bunch of challenges because of that we're not as fast ago where it was fastest rust and so we've had to come in thinking the different angles which is then hard at first but now it's actually kind of cool because it's caused us to have to innovate in areas that other clients didn't necessarily have to earlier this year I went to the swarm summit with the express intent of getting this effort started and this one team was incredibly receptive it's been a really cool it's been but first I want to tell you a little bit of base knowledge that may or may not be things that you already know um so let's first look at the data structures that the theory of la chaîne uses we've got chant I'll get these kind of split into two categories there's the actual chain data itself this is the data that you need to rebuild higher states of the hearing watching and the state data this is the data that has used during the EBM execution the chain data looks roughly like this there is a chain of eight point six roughly million blocks there are five hundred million transactions and receipts it comes out to about one hundred four hundred gigabytes in total process similarly we've got accountant contract data seventy six million accounts about ten percent of these are contracts this comes out to a total of three hundred and seventy five million three nodes and this actually accounts for about even though it's only 40 gigs of data when you sync any theory and this tends to account for more than half of the time that you spend syncing the main reason for this is that one the protocol has some inefficiencies in it we have to actually request these all individually and there are really small pieces of data so these requests are very inefficient talk about this a little more all of this data is content address this is the structure of it in general you've got the header chain this traces back eight point six million headers that all reference their parent each header has these four tri references and Nutrisse transactions receipts and uncles and the state route which represents the entire either an estate itself the contract storage things like that on all and all of this comes out to about 1.5 billion things I'm trying to get a sense for like what any theory in note has to do to give you a picture of sort of like where the challenges are in serving this data if we take a naive content addressed approach we will have to request 1.5 billion things by their content address and that's a lot guest sinks in one to two days which means that in order to do that we would have to do approximately 10 to 20,000 things a second the idea here is that there is a ton of individual pieces of data that we need we need to get all of it and if we go at it very naively we have to do a lot of stuff per second to actually be able to sync this here's a little bit about tries they look something like this this is how we store the states this is how we store a count data this is how we store contract storage and the important part here is that the only important parts really are in statement at the top and these green minutes at the bottom all the stuff in the middle can actually be rebuilt from the nodes on the bottom so we use the top as as like the reference point for what is the state of this but all we really want is the bottom of stuff those inefficiencies I was talking about are that in order to get at this bottom stuff currently we have to work our way down through all of this middle junk which adds up pretty heavily to a lot of extra data and a lot of extra processing has to it's also really unfriendly to your hard drive it's a lot of very small reads and writes again 1.5 billion things that we have to get and one more thing this was talked about a little bit more a little bit earlier but I think this isn't one of the important parts here is that swarm nodes like data that is close to them the swarm network has a topology chunks hatched to some area of this network and if you a swarm node are in a specific area shrim caches - you will prefer to hold on to that one but you will also prefer to hold on to things that are being frequently requested so if something is popular this one notes will tend to want to hold on to it well so I'm to the good stuff we are working on building a bridge between the eath and this one network we call this VCC eath and our goal was to used it swarm as this like alternate data source this word alternate highlighted here because it's important to note that it doesn't matter if the data is is guaranteed we have a backup source to get this wrong which is the current etherium network the idea is use this one network for what it's good for so as long as most of the data is there are the data is there to some reliability threshold that's great if some of it's missing not a big deal you can always fall back to kind of the current paradigm we came up with a plan earlier this year looks roughly like this we tried to come up with something that was like minimal and sort of delivered value as fast as possible so figure out a way for them to talk to each other we did that over the Deaf p2p protocol we have a new bzz eath protocol that lets swarm notes make their in notes communicate with each other the two things that we knew that we could do right away that are so essentially one-to-one parodied with with what the etherium network does which is serving block headers by their hat and serving tree data by its hash and there's some unsolved things that we need to figure out which are more efficient ways to communicate transaction receipts and huncle's as well as just an entire tribes of data I was really excited to see this last week that 0.5 release includes this I'm a little embarrassed that we don't have a Trinity release that actually supports this but this should be very sooner in the corner um so I spent a lot of time trying to figure out how to explain like the concepts in this and this is one way that I came about it which I wanted to highlight that at first glance fetching this data over swarm is worse than fetching it over the ethernet work and that isn't a hit on swarm swarm does what it does very well the reason that the Ethernet work can do better is that it has contextual awareness of changes so you can ask an eighth node give me the block at this number and each node to know that but for a swarm note to know that it doesn't make sense for swarmed out a concept of canonical block numbers this weren't all about content addressed by its hash so we have to come at this from sort of a different angle than we do in the eath context similarly you can ask an eighth-note for a set of transactions for a given block you can prove that data once you get it and the reason that an eighth-note can serve it to you again is that it has concepts that keeps this extra metadata information about the chain that is syncing but it doesn't make sense for swarm to know that information swarms should be sort of serving data by a Content hash so I I really like this graphic this is what it's like to come online as a new 8th note most full notes have a have a maximum capacity then they can serve a total number of teens I guess and most of the time all of those slots are full and you the said kid over here on the right who doesn't get any data and this is sort of the general state of the network you stumble around you tried to find some eighth notes that aren't completely filled up and maybe you get to sing from them the thing is that not all eighth notes are alike Carol comes on line and she wants to sink maybe she connects to Alice maybe she connects to Bob however if Bob is partially synced then Bob is only likely to be able to respond to about 50% of Carol's data requests Alice police ain't likely responds to about a hundred percent of the idea here is that not all these notes can be like swapped for one another maybe Bob was broken and bob is like stuck at some weird like four can never got off of it and it's just holding like a peer slot you keep asking it for information he may be served 50% of it but there's a bunch of that that he's not going to be able to give you this all builds to the concept that were interested in here is that if alice is a fully synced node analysis piping her data about the etherium watch-chain into the bcz network then Carol can get that data from Alice like normal but Carol can also request that data from any bzz node and mmm it doesn't matter if the one that she's requesting from is the one that alice is piping the data into that node will then be fetch the data from inside of a swore networking in about what this allows for is Alice just a single fully synced Peter piping her data entity it's one network to serve thousands of peers where she previously she might only be able to serve twenty thirty I don't know but this actions automatic can afford on the scaling because of these kind of natural properties of the swarm network that are amazing in that popular content will naturally spread itself around will naturally be of and so the idea here is that's one doesn't do what it does best well it may not have this contextual awareness of the blockchain of the state of the chain of this kind of like deep information it can do this one thing but probably it can do a bunch of things really well but it does this thing really well which is storage and retrieval of stuff by its content address and the powerful part here is that we don't have sort of like a chicken-and-egg situation where we need a bunch of nodes typing data in to get a bunch of Dan out one single ëthere typing their data into the swarm network can serve thousands of other peers as long as there is adequate capacity and they swarm network handled so this allows the eath network or the concept of an ephah peer that comes online to sync it allows the network to handle a much higher capacity we could have thousands and thousands of nodes come online all at the same time and under the current network they would struggle to sync they would connect to eath peers and be partially synced and then other ones that are trying to sync we connect to them and they would only know mr.

small percentages of the data but if they're connecting into the swarm network like we're tinkering around with all it takes is one fleecing community to serve all of these new peers to be online this is where we got to so far we have a a communication channel that eath peers and swarm eath notes in swarm nodes can communicate over we have to wait transmission of block headers so a system where a swarm note that wants to we can get notified of essentially like the tip of any aetherium blockchain and then trace their way back to origin fetching all of the headers as well as ways for the etherium node to fetch an arbitrary headers from them up next we're looking at - a transmission of tree data again this is kind of the inefficient version of tree data retrieval where we retrieve it node by node again these are like easy things that are just straightforward content addressing and then we have experiments that we need to figure out those solve problems for how to efficiently retrieve transactions in their seats and so that's what comes next we're looking at how we do blocks block data transactions and uncles and receipts the header again remember it's stores a reference to a try and that try is is essentially all of the transactions and given a set of transactions and a statement for that try you can verify that they are the same thing but in order to actually request this data and naively it's way less efficient and even though you say give me the transactions for this block and it just hands them to you and they make some kind of up to you to verify that they are legit what you wanted but if we request these via their contents address over bcz node then we're looking at kind of om complexity instead of a 1 because we have to walk down the tree fetch every single one of those and then rebuild that list of transactions from that data so there's an inefficiency here that we have to work out and this comes back to that 1.5 billion things again no doesn't actually process 1.5 billion things it uses these efficiencies of being able to collapse what would be 300 requests out of a single request in order to not have to process 10 to 20 thousand things per second its actual needs are much lower because it uses sort of these shortcuts and things and we have to figure out how to do that well within the context of even Network talking this one method so one of the first things here is ways to it more efficiently retrieve tree data everybody told you earlier that the way that we currently do it is get the state route to get this thing get this thing get these things and then finally we get to the data that we need one of the things that's actively being worked on are protocols and ways for us to fetch these things in bulk and only fetch the data that we need we've called these prefix queries as one way to think about it or range queries which are give me all of the nodes between these two values here or another way to think of it is just like give me the thing at the bottom I know what I want I don't want to trace my way down the tree just give me them give me the leaf node at this key and so one of the ways that we're looking at doing this is giving swarms and like Native understanding of the tree structure this isn't native context of the blockchain but this is more of a way to have it natively understand this tree structure and thus be able to return information in a more rich way another thing that that we're looking at is just screaming retrieval of all of the leaf nodes for a tree so rather than my giving me part of it it's hey just give me the whole thing I want all in it or give the pieces from it that I don't have yet but it's ways to efficiently retrieve all of the data out of a tree without having to request each of the notes individually another thing that we'd like to do is leverage this feature of swarm called feeds show of hands swarm feeds cool I am potentially going to bastardize this a little bit but the general concept of swarm feeds is that it is a time series database for a key value store so you had a key topic you've got a public key whoever it is that is publishing to this feed and you've got a value that's updating over time and the idea is that you can retrieve this value give me the current value because you can also work your way back through history and say give me what it was before give me the booking for that or give me it back at this point in history so the idea is that a eath node could feed this sort of subjective time series data into a swarm field which but then lets nodes retrieve data that isn't exactly content addressed but they can consume the speed and say oh okay I already synced from this node earlier I know that they gave me good data I see that they have a feed of all of their blocks that they've received I don't even need to connect to Alice anymore I can just consume her feed out of the swarm Network now to sync the watching so this lets a a foe node a good like reliable phone mode not even need to connect to appears comes online and still let that Peter hold data that they're essentially feeding into these fees we've been working on this thing called being sync in Trinity I'm sort of like a fundamentally different way to sync the chain remember I told you that we had to innovate to figure out a way to do things because and the general concept Jason may hate me for some emphasis I'm bastardizing it but the general concept is that EF has this thing called fasting they're like fasting the sort of occurrence the current way that you sink and etherium mode remember there's 375 million tree nodes and the idea is in fasting you download all of this state and once you're done with all of it then you start executing blocks on top of it this takes a long time this is a lot of data at home ya think is that a block may only touch a handful of nodes in the tree our experiments show that it's somewhere between one and three thousand three nodes they get touched in a given block and the idea is that we download this data on demand so we spin up the node we sync the header chain and once we know what the tip looks like and we know what the statement is we get the next block and we just start executing it we'd run into a place in our state database where we don't have the data that we need we fetch it on demand for the network that we've executed the blog just like we remained at that just like we're a full node but we only had to download something like a hundred thousand times less data so what we've validated here is that we can have a syncing strategy that takes a completely empty database and takes you to essentially full neuro status in about one to two maybe five minutes so that takes the sink time from two days down to five minutes which is huge there's a lot of asterisks in there about what I mean when I say a whole note and there's this entire backfill thing that has to happen but the general concept is that this that we can go from zero to 60 or whatever you want to call it it way faster than you can in the fascinating world what we're looking at is ways to make this even faster um block witnesses are this concept of what are all of the things that we need to execute lock and block in we touch all these red ones and block unless when we touch all these red ones and essentially if you just have this data and a little bit of extra metadata sitting in Bukit it's the right data then you can execute it that block if you package all that up we call that a witness in current Bheem Singh we actually have to spend some time working our way down through this to fetch all this we make like something like actually 3,000 requests yeah I mean and witnesses is it turns that image don't want that if somebody else is packaging up a witness for a block for us and maybe putting it in a swarm feed then we can consume their stream of witnesses come online ad and they start pulling witnesses from them immediately and start executing blocks immediately on download one couple things per block so so stateless clients are another term for this we have not been calling this like stateless because the idea is that we do want to sync to a full note we don't want to throw the state away at the end we want to bring more full nodes onto the network after which we can do some of the more fun things like yeah so the idea here is yes a stream of witnesses we can use swarm feeds a reliable know that you synced from in the past you can jump online you don't even need to connect to them you connect to any bzz node and you can pull this data out another thing that doesn't exist very much as archive notes are five minutes are heavy you need terabytes not terabytes a terabyte plus a bit of hard drive space in order to sync an archive node and archive Noda is in the etherium no that doesn't throw away old state it means that you can go back to block two or ten or a million and execute things at that block and it holds on to the state there the standard mode of nodes is to throw away state that's older than maybe 200 blocks or so and so the idea is that an archive node has the entire historical States that could be you could possibly have a theory another what we want to do is go sync an archive node as we do it pipe every single one of those tree nodes into the swarm network then we have all of that data there we know that some of it over time is going to like disappear off the network because people aren't asking for it but the nice part is we can just work our way back until we find a block that does have a consistent full state and then recompute everything forward from there to refill that history so using swarm as sort of like the global art it makes a lot of sense because the data access patterns for all of this old information you don't need to get at it super fast it's not something that people use regularly but it is valuable to be there so that you can occasionally when you have special needs rollback time and see what the hell was going on is a lot 2 million cool another thing that we're looking at is how we can store some of this extra metadata so when you sync anything in node there's actually remember earlier we have about a hundred gigs of chain data about 40 gigs of state data but if you look at the footprint on your hard drive it'll be about 200 gigs there's a bunch of extra metadata remember there's 500 million transactions and your note stores these mappings of this sort of concepts data transaction hash tag lock hash things like that this alone is like 20 gigs of just index needed 20 gigs of metadata and every single node computes this every single node stores this on their machine and it would be ideal if there was a nice way for us to fetch this data over the swarm Network it is one of the problems that we're still working on figuring out how to solve a pretty standard thing is to ask your aetherium node give me this transaction give me the transaction for this hash until you've downloaded the entire chain state as a nerium node you don't know this and there is nothing in the etherium protocol to retrieve a transaction by its hash which is why nodes have to store this metadata but it makes sense for us to try to figure out ways to store this in Network itself it gives us a way to sort of like share these indexes or contribute to them as a group things like that another thing that III in the notes have started doing recently is sort of considering certain chain data as ancient blocks like one through a million are like not going to change really blocks like anywhere it heads is a couple hundred are very unlikely to change and we can think of these sort of adds just like a pinned only files we can package them up into chunks store them as no sort of ancient chain data and then again if you have a mode that you've synced from in the past you know they were reliable which really only means that it's low likelihood that they're going to be briefing you if they actually published hashes or their ancient chain data we can download these things in bulk all of this data is still verifiable you're not trusting them in any way the worst thing that they can do is give you garbage data that doesn't validate locally and then you throw it away and you go somewhere else to get it but we could speed up a ton of stuff by just downloading I mean how long does it take to download a hundred gigs of data on a decent internet connection it doesn't take a whole day so if we packaged this stuff up into chunks of bulk data we can download the entire header chain and the entire ancient Shane data a couple of minutes rather than what currently takes hours or days another thing is actually using swarm for scalable by client infrastructure in the current model there's no bzz in here and all that LDS peers rely on phone nodes to get their data if we have all of the state data in the swarm that way and if we have all of this information in this worm Network then now all of a sudden Elias like client nodes we would be able to get that data from other Eve peers from bzz peers and potentially from each other and a lot of this is just neat ongoing research advancing the like my protocol frontier iam clients but the majority of data should be retrievable over you know yeah over the bcz network equal pipe it's been really cool working with this warren team on this i've wanted to collaborate with them for a while and and they were so receptive to doing this and it's really exciting to like see these networks bridged and talking to each other something that will be actively working on over the coming months and excited to have other people join