File pinning in Swarm

[Music] today I'm going to talk about a feature called pinning in swamp this is a very recent feature there is a blog which will release two days before I highly recommend that you go through the blog to understand more about this feature and more upcoming features in our blog so let's go through some history behind what is swamp and why we need this feature and how useful this feature is if you look at the very traditional distributed systems they are designed using what is called this twine server master agent architecture so there will be one or more masters and multiple agents must decide what agents needs to do and agents execute the things blindly what the master says this is a lot of issues which I'm not going to discuss the main the main the main class of systems that we've already discussed today is peer-to-peer systems peer-to-peer systems don't have any centralized entities here if you look at this every box represents a node a computer which has certain qualities like certain this certain CPU certain network ie well so these are self-organizing networks that means you can bring down the node anytime up it will try to find out its neighboring nodes and it will connect to only a few nodes not they not with all the nodes because it will exhaust the connection and all these things that will be scalability issues so it will just connect to few nodes which are closest to it depending upon certain parameters to to just help these nodes get a boot start there are a few loads that will be designated as bootstrap nodes which are marked this gray in color here these two nodes so if any new nodes comes up they by default will contact this node get the neighbors of this node and will try to see whether that node is near closest to me otherwise get the neighbors of the that node and so on until until the until it reaches to a point where it has found a group of neighbors which are closest to it and that is what that will that that that's that's the neighbors that this this particular node will use to communicate to and from the network right suam uses something like this it uses its uses in academia based system to get connected to the network and once it is connected you know you can basically do all the functions of the network in small suam is basically a distributed data storage system so every node will have to give some peace some portion of its disk to the network so that the network can send information to store there and retrieve whenever it's actually needed here is an example of a file being uploaded from suam node if we consider this color-block as a full file when you upload it to Assam note what some notice is that it splits the file into smaller pieces called chunks which are 4k in size and sense those chunks - it scatters these chunks nodes around the network closest to each address right so these nodes will get these chunks and store these chunks in there in the disk right so now let's look at one such now where it has told a chunk this node will keep on receiving chunks from the network to store and at one point so this this this this structure here is least recently accessed so as we get more chunks or chunks get access these gets pushed up or down and if our chunk is not getting accessed and more chunks come from the system our chunk will get pushed down until at one point the chanko vanishes this chunk will be garbage collected because you you just you just in a promise to swarm that I will I'll expend only this much disk space and if if it goes beyond that so on garbage collection kicks in and it just picks up garbage collects that know that particular chunk now because of this what happens so when you try to access the file this particular swarm notes try to contact these nodes took you back the chunks and one of the nodes or one or a few of the nodes will not have the chunk so you won't get back the file right this is the missing chunk scenario right there are two long-term there are two actually solutions today one is a long-term solution and another is a short-term solution long-term solution is to add incentivization to swamp this is to is to you know when somebody uploads a file if he agrees to pay for the file then the node that keeps the chunk has an incentive to not garbage collect the particular chunk so that it can receive incentives from the you know from the uploader right so that chunk remains there in the swamp network there is a there is a whole bunch of protocol called swaps where and swindle in swamp that is defined there is Papers written and yeah but the issue is the implementation is very complex and it is ongoing until then what will actually happen is that anybody who wants to upload a file in swarm or you know a dab website in swoon when he actually comes after some time the tabs will be garbage collected so he has issue that he cannot use swamp right now to do his that development and you know to test his death and so on for this we introduced a feature called pinning pinning is of two types local and global I will talk about them so this is the sort of expanded view of what will happen when you say you know when you upload a chunk you can say hey I mean when you upload a file I'm sorry when you upload a file you can say hey pin this file also so what actually happens is inside this warm node you know the chunker which actually splits the file stream in small chunks and keep the chance the chunk index and send dad's to the network store also keeps that in something called as the pin index as so that local copy of the file is always available right so when somebody comes and asks for this file I'm sorry when somebody comes and asked for this file even though chew chunks are missing in this small network the entire chunk is already visible is already available in the upload in the pinned swarm known so a client can come to the pinned swarm note and can ask for the file and it he will he will get the local copy of the file but it has editor it has a disadvantage if you go to another suam node and ask for the same file it doesn't have a clue that this is actually pinned so this will go look into the swarm network and it will find that few chunks are missing and it will say file not found right so this is the first step in you know in actually pinning this is implemented using this this is still useful because you I mean if you are adapt developer you can host your own swarm gateway and host your own app here and still connect to the network and can build your own app out of swarm and you know test it and run it like it is like a like it is as incentives are enabled right until okay so we still have this problem where you cannot access this from any other storm known users have to go to that particular so node where this DEP is pinned and that is still a problem and for that there's another feature which is an extension of local pinning is called global pinning which is coming in near future what happen is that if you access the file from another small node it will try to look into the network if the network is not giving enough chunks it will find the pin er that has pinned the node and get back the chunks from the pin ER and served them as if it has got it from this one network right so with these two solutions swarm will be will be ready to deploy gaps and Verte for access from anywhere until incentivization is ready even after incentivization is ready pinning is extremely useful because spinning provides printing exposes lot of use cases like storage insurance and all these things so this is all about the pinning feature if you want to know more about pinning I would suggest you to look at the pinning epoch in the penny in the swamp github which has more information and yeah enjoy have a look at this form documentation for to to see how to use spinning there are examples on how to pin and unpin and list files which are already pinned and things like that I would strongly recommend people who are interested in swarm to go and try this feature you I want to show you a little example of how to use spinning if you see you when you start Swami you have to use enable pinning flack and on the other terminal like this wanna show to file small file and one big file let's upload the big file and when uploading you have to have this X form pin is equal to true header which means that I wanna I wanna actually pin the file so the file is uploaded now once the file is uploaded you want to actually look at look whether the file is pinned and this is the command to see the file spin the the hash we got in the hash we got here is the same the pin counter is one now and let's say we want to pin the same file again with somebody else now if you do the same if you actually do the pin list command the pin counter is to know that means two people are interested in this file right so now let's let's let's pin this more fun and using the same exponent pin header equal to true and let's see if that final spin yes that file is pinned and the pin count of that file is 1 and yeah you can see the other file we'd be pinned also now let's try to actually delete file the over one of the old files which is being twice so once we deleted and we have to a list command and now the pin counter has been read is to one since one of the interested parties unpinned it if you have been again that will go off so only the small file now remains a spin so it's like a TC counter so the number of number of people interested in it the pin count will keep on increasing if all of them aren't very uninterested it will unpin the file this is a small demo of the [Music]