Devcon 5: Ethereum Swarm Breakout Sessions - Persistency in Swarm

so my dog will have a little bit of water left with the in septic shock that was presented by in the morning so there you will you will probably recognize some of the themes what I would like to approach this set of problems from a slightly different time first I'm going to outline what the challenges of persistency are so basically in a network of freely participating nodes where every single node is operated not only by a different entity following its own interests what are why why does it how does it become possible to nevertheless ensure that the data that we upload to this very very diverse and constantly reorganizing Network how why white can still stay there and remain achievable then we're going I'm going to talk about swap incentives so this is the part that is a little bit overlapping with the morning presentation but I will more full on the emergent behavior so we'll talk about what incentives we have in place and what we can expect to happen because of those incentives then I'm going to talk about production I hate spam and I use this word spam in the group that's impossible that's it so basically it's any condom that you don't want it's not necessarily advertisement it and everything that is that takes up space takes about and so on and then in the last part of my presentation I will go through a few use cases possible use cases or use cases that people are currently already working on and see how they play out with them with the kind of tools that we have or that we're going to have so let's start with the challenges so the challenges are basically come from four different sources the most important one is teacher so nodes are coming and going they think some data with them when you notes join the network than some content which they don't yet have is being searched asked of them and so on so the fact that nodes are leaving and joining the network it basically does put a stress upon our continent and it does make it does make it more challenging to retain it and to locate it and I want to talk about how we address this the second one is spam so what spam is in this context it's content that it potentially crowds out other content so the even though we expect that swarm will have many many nodes but no matter how many nodes swarm has its resources in terms of bandwidth will always be limited and if there's no bound on the content that is being uploaded and there are some problems with the incentivization of that then we can expect spam crowd out useful content and I'm going to talk in detail about then there's opportunity costs so again because of the limited resources in this case of one particular node so they have some storage they have some bandwidth and they are making decisions individually each node for each nodes operator is making decisions of how to use these limited resources and by not using them for one purpose or sorry by using them for one purpose therefore going the opportunity of using it for some other purpose and then finally we have incentive misalignments which are again the well-known problems in game theory and economics so I will shortly go through the they affected swarm in particular so jerk when notes leave it's obvious how that is a problem because they reduce the redundancy we have the same information in the network stored on fewer notes notes joining swarm they required time to what we call things so in in swarm lingo syncing means obtaining all the data that will be eventually surged at your note so in swarm the addresses of nodes and the addresses of data comes from the same address space it's a distributed hash table that's what it means which means that your address kind of determines what kind of data you should store what kind of data is going to be requested from your particular node and what we call Cynthia is the process of obtaining that data and if nodes join it it takes time where they sync and during that time they are still they cannot actively participate in Internet work they cannot they cannot serve requests so from these two things it's apparent that any changes in in membership beat notes joining were notes leaving unveils some kind of data movement something needs to happen because notes so as notes come and go make the data needs to be shifted around and that is that that particular that particular process is what we call pool syncing and so on so when you're reading sworn documentation pool thinking is this but is this special kind of thinking that happens because the there are changes in the in the membership of the network and also we have we have this concept of the radius of responsibility which is the kind of data the kind of distance of data for which a single knowledge is responsible so if Sauron consisted of a single node that is radius of responsibility will be infinite so basically to the compacts the whole data space but as more and more nodes join the network this radius of responsibility is getting smaller and smaller and and there's no sleep it's getting bigger and those changes of those changes are the ones that need to be handled by by the posting process spam so that there are two kinds of stuff there's malicious spam and there's accidental spam so in case of malicious spam and I will talk about when some nodes might engage in malicious spamming so it's an attack it's a it's an attack it's performed by an attacker who has a specific selfish goal of obtaining some benefit at some other nodes expense and because of that they are pushing content to swarm they either want to to do it for the purposes of censorship or they want to to get paid for something and so on so there can be malicious and there's accidental spam so basically if uploading doesn't cost anything that people might not be expected to be you know sparing with their uploaded tivities and they just might casually afford a few that's not caring about the consequences and the load that they placed on the rest of the network and one of the interesting challenges that we faced and I think we did a pretty good job of resolving it that it's really difficult to tell uploading from sinking so in both cases data is being moved but in one of the cases when you're uploading data you're uploading content into the network that has not been there initially in the other case it is of sinking you're just moving content around as as necessary and of course the difference between the two is that thinking is something that is necessary for the continued retract retrieve ability of data from swarm whereas uploading is a service that in which for which the network which will take more resources from the network and therefore somehow the network needs to become infected forever but from a single notes point of view in a single node as a very limited a very limited view of the whole network it is only connected to a logarithmic number of peers allegory a logarithm of the total number of peers from a single nodes point of view it is really difficult to tell whether a chunk that is coming in is sinking or an appeal and as you can see from the arguments that I just highlighted by misleading the node about the nature of the incoming chunk you can really influence there and so if uploads our freedom they might crowd out important data and also censorship resistance as spam resistance they are kind of at odds with each other to some extent because if it's easy so if if there's some sperm production mechanism then one Avenue of attack a censorship at that is to somehow classify the content that you want remove or suppress inside the network is to classify the spam so in on one hand the network needs to be like it needs to be difficult to to remove data that is that is uploaded legitimately on the other hand it should be easy to get rid of spam and there's kind of a trade-off here hunt I'm going to talk about how so now I'm just highlighting the challenges but then I will talk about how we are actually making these challenges so there's opportunity costs I already talked about them so basically your there are two kinds of opportunity costs one is storing swarmed a diverse with something else so you have a limited disk space and you want to allocate some of it for swarm and other parts of it for other things or even inside the swarm inside inside the space that you allocate it or swarm there's a choice whether you store this data versus other data and these all imply some opportunity of something there should be a there should be a clear way for each node to make these decisions in such a way that if they're follow their own rational self-interest the emergent behavior will be that they will the work and the interest of the whole network so the misaligned incentives are so the problems caused by misaligned incentives are selfish storage which is kind of an analogy with selfish mining where basically by maximizing your profitability you somehow are doing unnecessary work so it's selfish mining if you recall is for example things like mining empty blocks and also there's a tragedy of comma there might be tragedy of common situations when there's some behavior that's in everybody's interest so the benefits are shared for the whole network but the costs are kind of private to the node that is engaging in this behavior and in this case we can expect that this kind of behavior will be under provided so this these are the incentive misalignments that we want to avoid so the one incentive system the most basic incentive system that we already have implemented and tested and you have already listened to a very detailed presentation about how it works and now I'm going to talk about how to swap all by itself addresses some of these concerns so just a quick recap what swap does on a very high level so in each case if one node wants something from another node they eventually pay that's kind of the bird's-eye view of the thing and of course the option number one is that they pay by reciprocal service so typically they don't pay by by something checks or make go check payments the typical preferred way of of paying for services reciprocal service if the about the accounting of mutual services gets dis balanced then it can still be settled by a payment and if it's gets this balance and there's no payment coming forth then eventually this sanction is to disconnect to denote that is running in the rap so I don't want to talk too much about it since you've already heard a presentation in the morning but the interesting part which I would like to spend a little bit more time on is what these things in some device like once we have swap what can we expect to happen first of all nodes will try to retain the most profitable chunks so they want to retain the chance they want to store the chances and their best interest in terms of swap to keep no keep chunks which are requested from them very often typically these are the chance that are nearest to them which is wonderful because that's what we want them to do so swap all by itself incentivizes that you kind of follow the DHD protocol correctly because if it if you're storing the chances are far away from you then they're not going to get requested from you and that's that wait for you the other thing that swap incentivizes in capturing so if there's some popular content that is being requested from you all the time and even though it's outside of the radius of the responsibility but somehow it gets requested very often then you can earn money by caching because you kind of buy it once you store it and each time it's requested of you you're selected so actually swap does in some is caching so we have some caching codes were not and it's not worse for somebody to take that code out because that caching behavior of the swarm node is actually in the interest of that node through you know through being paid for it via swap so one one consequence of sort of swap being in place is that we can expect popular content to be replicated very broadly and persistent so if there's some very popular that something like I don't know like wallets like my crypto or some other tablet everybody uses then we can count on this content being available forever because swap actually some devices both in storage and discussion now there's one thing that we encountered observing what happens which what we call the donut effect it is that if someone that is really popular and it's requesting all the time then it's always searched from cache and it's not actually served from within its radius of responsibility which means that selfish nodes that basically keep a tab on every little chunk of count how much that chunk earns them they might decide that even though I am the closest node to this chant i will still garbage collect this chunk because it doesn't make me any profit but that's okay it's okay because while while the content is popular it's served from cache anyway so the users of the network will not experience any and the loss of content and if the you know if the popularity comes down then the syncing process and of gravitate this continent back to back to where it belongs so so the content will still persist so the dormant effect does not affect persistency even though it's an observable behavior however sinking itself the fact that content is being pushed towards where it belongs is incentivized only weakly by swap basically there are two reasons why you want to sink current one is that you want to make room for other content so instead of just deleting something you rather sink it before you delete it because there's this common good thingy that if we all do this then the network will work better and we will kind of all benefit from it and it doesn't cost much more or it shouldn't cost I will talk about this so it shouldn't cost you much more to sing something and then delete it not just simply delete it so the fact that you know you're earning money by swap therefore you want the network to function correctly yet you want to make room for stuff that is profitable for you these two things together we clean some device that you push content that is far away from you to where it really belongs but it's it's a weak in something it's not a strong conservative and garbage collection again notes can be expected to optimize their garbage collection strategy to maximum profit and there's this heuristic that we have implemented it's the garbage collection tube so garbage collection to you is consumed from the bottom whatever is at the bottom of this Q gets garbage collected and chunks that are have been charged that have been accessed recently they're moved to the top of the queue which means that if something gets requested from me so people actually paid to me I don't want it to garbage collect soon so I move it from wherever it isn't the queue I move it to the top that's that's one strategy however it's very important that sinking does not insert into to the top of this queue because what's at the top of this queue is kind of proven content that has already been requested from you the stuff that is being synced towards you it's improving on and you're not sure you cannot be sure that it will ever be requested all of it can be spam and you don't want sinking to crowd out profitable content so what happens in practice and the way it is what notes are currently implemented is that sinking actually sink somewhere towards the middle of this cube and what would really help and we're going to have that and I will talk about it shortly is some credible signals some signals that are really costly to fake that somehow marks important content that distinguishes between garbage and spam and on one hand and important valuable information on the other hand because if we have something like this it would actually advise the garbage collection strategy in order to to be profitable for so we have arrived to the third part of the presentation so how do we distinguish spam from important information and for that we have come up with what we call a most digital model so it's been inspired by the international mail delivery how like paper mail is being delivered across the globe for like at least two or three centuries man - for sure so upload should cost like each there should for upload somebody needs to pay for uploading because uploading does impose new costs on the whole of the network but it's important that only the uploader pays for it because otherwise if if those that are sending we uploaded content also would also need to pay for anything they just won't do it so somehow we need to mark chunks okay this chunk is bad word and that that should be recognizable by looking at that chart and so the unit of accounting here is so the dimensionality is value over information over time so that's one example int age or / second but it could be anything it can be like dollar per kilobyte per year or something like that so the dimensionality of accounting is is this a very important insight is that because the hash function distributes champion distributes chunk uniformly across all the nodes actually the local information that they note has is actually sufficient it's surprising so basically you don't need to see all the chunks marked with the same type of postage stamp and yet you can have a pretty good guess of how many there are because you know how large the network is because you have a logarithmic number of beers so up to a factor of two do you know how many nodes there are anymore and you know how many how many chunks with that particular postage stamps you observe and you simply multiply that by the number of nodes in the network and that gives you an accurate estimate an accurate enough estimate of of harmony like how many chunks have been marked with that particular post system moreover the inaccuracy that you have stamps from the inaccuracy of your estimate from the number of nodes in the network but that's the same for all types of for all types of postage postage stamps meaning that in order to compare two different postage stamps your accuracy of estimate will be still will be better still because the error that you make with the number of estimate will be the same for both of them and it will kind of cancel out so the most disciplined the simplest implementation of the postage stamp and where that that's the one that we are going to implement first is the so called proof of birth so proof of birth means that in order to put a postage stamp on a number of chunks you need to prove that you have spent either so proof of word wouldn't work proof of work is a very natural candidate and we could just take a page from whisper which uses proof of word for this very purpose but the problem is that the delay the amount of time that you need to spend to generate that proof of work would be proportional to the file size so it would be a real drag on the performance of the network and the other problem is that proof of work has a huge specialized attacker advantage so if some attacker is specializing in attacking this particular this particular feature then they have a giant advantage over the regular users who are using general-purpose hardware for just using it so specialized attackers with special purpose machines or special purpose software have a huge advantage and that would actually become a denial-of-service mobility because they would span the network at a much lower cost to themselves that what the legitimate users would have to expand so proof-of-work is as attractive as it is unfortunately it's unworkable so instead of that we have proof of birth which is May which is basically a digital signature so proof of burn is a digital signature on the chunk with a public key which is registered with a smart contract in exchange for actual payment which is locked up in a smart contract enough released so you burn some ether in exchange for that your public key with a expiration date is registered by that smart contract that anybody can check with an access and therefore they can check whether you have a pink burned enough hitter in order to do that signature on that particular child so you pay once on chained to this contract and then you can sign a whole bunch of chunks but the individual nodes and swarm will know how many chunks you have signed with that particular with that particular key so they can divide it so they can tell how much you actually burned in order to sign that one chunk which is nice because you do only one on chain operation you can do millions of of chain operations so we are using the objects bearing with you of course it raises some privacy concerns the privacy concern here is that if you're doing an upload and you're signing every chunk with the same key then it kind of indicates that all these chunks belong together so even if they are encrypted you don't know the structure you don't know in what order they follow after one another but still you know that these chunks they originate from the same upload and that can be a vulnerability for example for some section so if some of these chunks turn out to be whatever offensive condom that some powerful entity wants of this one then they can say that okay remove everything that's been signed with with a skinny perhaps and we want to avoid that so we want we somehow want to solve this problem and we actually do have a solution I will not go into details so we will have this ask me anything part in which I will in this in detail it involves a cryptographic primitive called blind signatures I will talk about them later during the ask me anything session and the next stage after the proof of birth is the storage lottery which is even better than proof or burn because it actually allows storing nodes to use the data that they're storing as lottery tickets so each chunk that has been signed because of a payment a smart contract automatically becomes a raffle ticket in this lottery in which the postage that was paid for uploading all those chunks will be raffled off to those who are actually storing these chunks and this way you can actually struggling something like thinking so at this point if we implement storage lottery same thing will no longer be weekly in some device there will be very strong profit motive in storing what you need to store because we can have these very easily enforced rules that whoever submits a chunk to claim there to claim they're winning in the lottery if somebody who has an address that is closer to this chunk also submits it they will actually win so they can challenge that and when the price of it the closer you are as an uploader then so the good the closest uploader is the one that wins and because of that a lottery ticket is suddenly worth much less for the notes further away than for the notes that are closer which means that the note that discloser is willing to pay for it is willing to bite for a price for which the salary willing to sell because the value for the two is different so there's a mutually a mutually beneficial exchange transaction that can happen over swap so you can use swap to pay for these and suddenly you don't have that problem that to base for sinking because note actually want to want to buy the the lottery tickets which give them a expected expected that and calculable so unexpected women that they can actually calculate its idiots they can statistically forecast it and therefore they know at what price what price they are willing to pay and because of the way it is arranged the notes that are further away they're willing to sell for that price which creates a market for actually sinking the chunks and also it solves a large part of the unpopular content issue so that's the one that has been addressed in several talks and I will also talk a little bit more detail about it in the last part of my presentation so one popular content is the one that is not asked for in from you and therefore swap does not in some device its storage but this lottery to some extent does so in the last part in the concluding part of my presentation I will go through a few use cases two minutes okay that's you know so the first and easiest thing is sending large files so it's nice because if the persistency that it requires is on your magnitude of days you can get if you definitely get feedback on the beginning of the availability because that's the the progress of our feature that has been talked about already possibly the end of the availability so you can have some estimates of the costs involved and therefore people at data front have already implemented fare drop for which the kinds of incentives that we have been placed in entirely adequate the next slightly more involved use case is hosting static web content and most importantly that's web based apps so here for popular content swap is enough we already discussed that for pop for contemn that is not yet popular so you have a startup you have published that but nobody knows about it so you're exerting marketing efforts to make it popular but before it gets populated still needs to be available it still needs to be persistent so what can you do well the easiest thing that you can do is you can pay for it by swap you can pretend that it's popular you need to download it yourself and keep paying for it and therefore those who are in some device by almost well will host it that's actually the like the step 0 of pinning so this is something that we're going to have very soon when you can store stuff that you want so on to host and you just keep reuploading it and that would get unpaid and also pay all the notes swap to do that and because of the garbage collected away the garbage collection key works it will provide the other what incentives work so our post it then there can be it can be delegate sit so you pay someone else to do it for you or eventually there might be smart contracts around which that was organized that are doing this work for a person so this is again something that is very well within reach and we're hoping to get here this year persistency against censorship so here you want to force sensors to play and lose whack-a-mole beings so basically you want it for the publisher to be easy to reload and for census to be difficult question oh sorry ok 7 I have one more slide ok so ok ask me anything storing in Syrian state on sworn we have already discuss this storing encrypted personal they does so this is like this is the holy grail of persistency this is the hardest kind of persistency so imagine you have your own encrypted backup nobody will ever download it except you and probably you will not download it ever so for these we have these things which are a little bit further down there so this is the functionality that probably will not have this year and the very last thing that I would like to just mention and because this is a that I very often get like how can people how do we delete from Swan and we have this fundamental issue that we have now we can do it the web 2.0 way so what happens if you delete something from Facebook if get smart as deleted well we can do that too so basically what what delete it means that it's no longer accessible from a given UI or a given API also it might become undiscoverable from the current ethereal State now since ethereal state is is a eternal record it might still be discoverable from a classic ethereal state but also as we have incentives for garbage collection if something is not being paid for the incentive structure that we have in place will mean that swarm will slowly but surely forget it so basically these are the things that we can do so that's it