Storage & Retrieval on Filecoin Part 2 - Longfei Wang

foreign [Music] and welcome to the hack FS file coin uh storage and retrieval on falcoin part 2 session joining us today is longfei Wong who will be taking us through this session and with that I'll pass it over to Longford to get the session started thank you say you'll be able to see my slides right so if that's uh it works fine and I'm gonna start today's session hey everyone and good morning good afternoon good evening whenever you add and welcome to listen to this talk about the storage unit table part two I hope you guys have spent some time and listen to what Matt has talked about last time and to be about like making storage deal on Falcon through smart contract either through like a client contract or through the aggregation so this topic will be part two and I have the what Mass talk and we'll talk a little bit more about what what afterwards if you make a storage deal on Falcon what it can do to replicate to renew or maybe to repair on the spark one virtual machine through the small content so uh let's uh maybe introduce myself a little bit and my name is longfei and I work with the Falcon developer Advocate and now because fbm is super important on that so we're gonna I'm gonna mainly hopping on a VM team as well so if you guys uh work with hackathons and you guys stay in the community either through slack or Discord you might see my name popping around everywhere I'm trying to help developers in the community and hopefully I'm being helpful and be able to bridge you guys with the team product team or our engineer team to be able to fix your problem and to provide a better and more stable uh infrastructure for building things on falcoin cool and oh so for for guys have been listening and following all the filecoin workshop and you might have ever seen this slides uh thousands of time but I think that's a great size to be able to tell what's the file coin plan is and why we're doing things at a certain stage of the development phase if you're not and the first time jump into this workshop and I will give you a brief talk about why what the Falcon are doing and why we bring fcm on board at this period of time so obviously uh Falcon is building a decentralized network decentralized storage to provide the storage for all the like web two or three words uh decentralized storage obviously the storage is most valuable things for us starting from like 2000 2017 or 220 220 will launched in a minute uh we are putting a lot of effort trying to providing the decentralized capacity into the network trying to onboard the storage provider trying to onboard storage biot to provide a more stable uh the storage to the global that's the first that we've been doing and we've been doing a great job since 2020 and then the network has wrote a lot uh you know like if you look at our blockchain Explorer or any static file aesthetic website you will find out that the storage capacity regardless of raw storage capacity or that just the storage capacity through farcoin Plus have grown rapidly until like 12 12 or 13 gigabit right now and after that obviously we want to onboard data of outgoing we want to onboard the data like valuable data set and Humanity data set on filecoin either through web 2 applications or collaborations or providing the storage for all sort of the web 3 applications platform like ethereum Solana and near or any blockchain to onboard the data on filecoin and that's going uh very great as well and uh the the third step is what we're doing right now we are trying to bring the service on filecoin and harvest the real value stored on filecoin which is data and then providing the computation through this data through the either students data or through the state status of the data stage of this data on filecoin and to be able to build another web web scale applications on PowerPoint and that's where we bring file Queen virtual machine into the picture to be able to provide the on-chain programmability through the data through the state of data on filecoin as well as all the the computations like allow things like that and this is the brief idea of what we're doing and uh why biocon virtual machine is important and how it will bring the program will be into the Falcon Network I think now we've been talking about filecoin and we've been talking about how Falcon is providing the data for for uh web 2 web 3. now because we are at a hack fs and we're trying to get a developers interested to be on for altcoin and so the most important part is how are we gonna build our coin and what kind of a service and what kind of tools that we can use I think for developers we want to be clear that uh you we want to be clear to developers there based on what you're building and you need to find the right Builders paths to go to follow to use the right tools to build and we don't want to develops to bump into the road that they just completely within the wrong tool and the building things that would not expect them to do so this diagram prepared by Sarah one of uh one of our team members on fvmt is great it's very clear and guided developed through the process of picking the right tool and then thinking from the perspective of what the product needs so we can look at this diagram from two side on the left side the green each part is if you want to build something more programmably on starcoin which means if you want to use in the programmable storage on filecoin and then the right side is if you don't and for the applications that you're building you just want to store your data in a decentralized manner that is that you just want to put your data in the file coin or ipfs network and you don't need to handle or like making storage deal and monitor the storage data or doing the renew or replication that kind of a job and you will go this way for example if you're building an FDA project if you're building a GameFly project like AI or social yeah you just want to store your data in your decentralized manner out of the control of any Central party and you go this way all you need to do is use this storage service web3 Storage nft storage Lighthouse or S3 now they change to evaporator tools so and then all you need to do is just sort either and then all the those storage service will take care of the storage deal making process so actually all those Storage storage service are the builders for the left side of this diagram this diagram so that they are building something on the valvecoin and those storage service will build on file coins storage we need to make storage the on Falcon Network and monitor the file the storage status and be able to renew and repair and then regard and also if the if the data are small they also need to do the aggregation they are if you are building some data store service like web3 storage or asteroid and you're mainly gonna link on the left side of the picture or if you're building something like we've been talking about data dial Perpetual storage and storage automation then you also need to go all the with the programmable storage inside and if you if you're storing some data on smaller size or less than four gigabit which is what what would the extinguish of smaller size at the small data large data of our coin and you probably need to go through the aggregation or you if you're storing data loss then 4 gigabit you can go on with your clients all this have been covered by Matt from the previous session and then the one the thing that I would hover mainly today is if I made a deal on filecoin regardless to aggregation or without aggregation there's a deal landed on Falcon we just need to keep Monitor and then be able to replicate as many copies as we want and be able to renew or maybe essentially will be able to repair if there's something wrong with the storage deal or filecoin so so today I think uh I always want to be clear if you are in this workshop and if you're interested into building Perpetual storage or renewal repair and you're going to mainly land it out here but if you're building that that's how it is that's the part of the develop so we want to chat we want to try to developers who are interested in this site and pay more attention to this Workshop if you're building an ft or D5 and the storage service might be the best options for you okay so let's get into it because everything I'm going to talk about today is for the storage deals I want to go back one step and to refresh everybody's mind on how far coin storage works I think and Matt has also covered that in the previous talk but I'm going to bring it again because that's will be important for you to understand how we'll make some of those process on filecoin and on the filecoin virtual machine through small contract and what what while we are doing certain work like while we're doing replication while we're doing renew and uh while we're doing things through the aggregation all right let's uh first go through this storage deal making process very quickly and I will I will see what wheel can put on Falcon Virtual Machine and what we can do is renew so if you want it's starting with the client spot if you as the client of Falcon and you want to store data on filecoin so the first step we need to do is yeah you can find your wallet and you'll be able you need to have certain field token uh in your wallet so you'll be able to pay the storage service fee and maybe potentially pay our cash flow from from your storage or what you can use a data cap at to to fund to to store your verified data on filecoin and that's the essential part of the using uh filecoin and then once you have that you want to store data the first step you need to do is doing data preparation so you need to prepare your data from a regular file into the call file which is the data format used on filecoin and ipfs based on IP Audi standard and you'll be able to retrieve them by the CID so those the call file can be either saved on the file server or ipfs eventually the storage provider will retrieve this data and then store for you once you have that ready the second step is you need to make a storage deal proposal to the storage providers on filecoin based on what the storage provider request based on the geographic of the storage Brighton so if the storage provider sees that a proposal coming and that's okay for them they will accept this deal and then eventually request the data you want to store from the fire server on ipfs and once they get it done they will publish this storage deals on chain on developer Network and after going forward you'll just keep proving the storage there have stores through the proof of replication or prefers FaceTime so from that there was something stay on chain which is deal storage deal and then storage deal will have all this information recorded on chain I won't bring this up because that's the most important value as the best importance program on that deal is starting ebook and ending Epoch which indicates that how long the storage deal will leave on Falcon Network as well as if this storage deal is active on falcon or not that's very important for renewable storage or repair storage or making sure that we're stored enough replications for these clients as they reached and as you know this whole process happens on channel obviously and from this picture you can see data data preparation storage your proposal data transfer all this happen off chain and then do you publish proof of the deal and install HD also on chain so now with fvm coming in we are trying to move some of the component on chain through the programmability through on the smart contract so now if you imagine that you can start using like a field token to fund your storage and maybe you can have your own token to fund the storage trying to incentive update a Sunday incentive for clients to use trying to incentive storage provider maybe like geographically to send to and then the second part we can make it on chain is making storage deal like match the presentation last time for this is the first part of the process that we can make deals through the client contract which means you propose deal in the contract and contract will immediate events on chain and a storage provider boost will be able to pick it up and publish deal all the orange side the orange product I put it here will be able to make on chain through small contract and another part really important part of today is we can also make the deal renew still repair and do replication on chain as well and on chain will be able to monitor all this process and made it eventually on chain before we go into all this detail of the renew and replication I want to bring another concept up is degradation aggregation the deal aggregation and Matt also talked about last time so I want to emphasize a little bit more because that's also something that is very important in the Falcon ecosystem and I think it's a super interesting idea for developers to build on top of it as well and you can see there's there's a QR code down here if you're interested in building aggregator you can just go scan and look at the specs that our product can prepare for you and you break through and see if it makes sense if it's something interesting for you to build and then go photo if you have any questions just talk to us to talk to product Kingdom of our coin but essentially what is a good question why is the need you probably heard I talked about data size over and over again like all because there are some status eyes that we call sector size which is on Falcon is how big the data USC it will restore and then in the fall coin this exercise is a 32 gigabyte or 60 to give it imagine that if you store like an FD data which is like a few gigabit or maybe at one megabit how much data you need to store to pack to like 32 gigabit that's a lot of data you need to pack into there and uh it's not economic friendly for the storage provider to pick up some more data and install imagine like 32 gigabit and you put thousands of files into one deal and it will generate tons of messages to generate a tons of proof for storage provider to do so it's like expensive execution also very very expensive on a gas fee and imagine all those deals will come a different turn like how much how long they want to store how much they wanted to pay and I was touched by the Indie it's very difficult to handle so and that's why we bring aggregation into the picture so uh that what aggregation do is the aggregator will listen all the files and they'll pack them into uh one deal and the one one deal or maybe a call about two and also at the same time generates what we call per side it's inclusion proof that you can improve that your file is included in that car into that sector and then be able to verify that as well imagine that if you're putting all your files into a black plastic bag you will not be able to verify and parsite is the standard that we're bringing in to be able to help to verify in aggregation and then the process of aggregating the files and generating the proof all could happen on chain and option as well unchain means to small contract option means what uh web3 storage or uh Falcon data to start doing and all they are building by some of the collaborators in our system as well like Falcon data tools Lighthouse and seafron they are building this if you're interested building your own generator option applicator go scan this code and you'll be able to uh just learn if you want to see some example maybe you can see the how far one data tools and Lighthouse are building their solution so keep in mind aggregation aggregator is very important to handle small deals and that's what we're going to take as an example for our deal uh aggregation now the renew and the application we are using we're using the aggregator in that example but of course if you're storing some data with larger size you can directly use a client contract to do it so let's look at this picture so now we're talking about uh renew what do we call a rest Ross uh it's a you could do replication in the service renew as a service or essentially repair their service so what it will do and we can look at briefly on this side list if I'm a client and I want to store my data on filecoin so if this data are small first I need to store that with aggregator so aggregator will have a small contract which will be able to record all the data come in and there's going to be aggregation aggregator part aggregator function will be aggregate all the files together and they generate the proof essentially standard approval back to the clients or recorded the proof in the smart contract and that's a lot of once you successfully make a storage deal and then storage Geo will leave on Falcon Network regardless it's too aggregated or not aggregation aggregation and this deal will be staying on file coin network with a certain time which is the lifetime and now minimum is 180 days and possibly something might go wrong with the sector sector might go down because the storage provider has some issue maintaining that and so after you making a deal successfully on filecoin we need to constantly monitor this deals so first making sure all the sector are healthy enough and then the deal isn't expired or if the clients want to store multiple replica and want to make sure that that's enough applications exist on Falcon now all this has to be monitored through small contact or through uh what we call rust node as well so if anything goes wrong we need to make sure that away we kick off another storage deal to repair or to renew based on whatever Service uh what what the the developers are building on and then if you are interested if you want to know more here's the QR code as well you can scan and then go to the stack page that will be preparing I've got these shows this is aggregation uh spec page and also the storage deal renew a process as well and keep in mind this is just the starting point of repairing a spec and we are providing the minimum features required to be able to make the storage renew or storage application works and obviously there are a lot of more detail could be added in there and if you're interested uh go find it out if I find out and then do expanding that knowledge all right so let's talk a little bit more into the detail on each part of this how does it work and I'm not going to spend too much time talking about how to make a storage deals through aggregated aggregator because Matt has covered it last time so uh I'm just gonna go over very quick and the example here making storage deals to aggregate the the important part comparing with what Matt talked about is we bring the the application part which is involving updating data I'm uploading data to aggregator and then be able to submit the CID to get applicator as well at the same time it might need to register the renew job or replication job or repair job with what we call the Ross notes in the future and then Russell will be able to monitor those and that's what we expect developers or we expect Builders to build for this uh thing maybe it's going to be a part of the Perpetual storage part of the uh storage service that they are working on so and then the prices are quite simple and if you're making a storage deal with aggregator you're going to submit your data and some media CR CID to aggregator and then aggregator smart contract will be able to monitor that if they received a request of storing data and they're going to emit the events of receive a requests and once the requests come in and have have enough data to pack into one sector the aggregator will do the job and then aggregate all the deal generate the past I and then be able to uh call back the complete function so the complete function will record the the proof and then the the Deals Deals including the scid in aggregated smart contract and I will be able to be incredited by uh like by by the uh I mean the the notes as well as the small content which I will cover next and once you have this is the process of making storage deal once the store is your is is finished and then the aggregated small contract will be able to record all the deals have been made for that CID and included seid into the video process and that's the the first step of making storage GEOS to go greater uh keep in mind uh that we're taking a falcon data tool Edge as the example here and if you're interested you can also use them as well and there's a small contract have been provided by uh by the aggregator and it has the submitted function a complete function that I've demonstrated in this I talked about in this diagram and I also have a get audio function will be eventually used by uh the steel stator smart contract as well and once that process once the deal is mixed successfully on chain The Next Step will be we'll keep monitoring those deals and if something goes wrong the last note will be able to take actions uh here I'm going to take replication and renew as example and repair will be coming later as well so if we if you remember earlier I said when application make a storage deal and you can also register a renew job and replication job waste Los node with this note So eventually this note uh I will work with the small contract and monitor those deals on filecoin through to the aggregator so how that works is and there's a deal status smart contract in a small contract and you're going to keep uh you're gonna get uh all the active deals through the aggregator smart contract so we'll send the requests to the aggregated small contract and then get a list of the deals that including the CID have made on chains and then they will check with on Falcon Network calling the building actors to to look if those deals are still active or not and once they get data back and it will emit the events and tell you meet the events about which deal is active or not and eventually the roster will be able to listen to those events and then calculate if there are enough activities exist for this CID so if not for example if the clients want to have 10 replications on falcoin but we can only find nine active videos which means this is one missing and uh it could be a repair and replication as well so the rust node has to take action to start a replication work or start a repair work kick off not a new proposal with aggregator and activate idea for for these coins this is the brave process which involved a lot of monitoring a lot of customer logic into the replication as well and that works the same for the renew job the only little bit difference is on the deal status contract we're going to look at is there any deal expires within certain Epoch and this those list of ideas are also request from the aggregate decimal contract and you'll be able to get a list of videos that have been made to include that CID Inc and for each of the deals we'll look at the building action and to look at the deal term to make sure if it's expired or not if it's close to expire we're going to emit events and those expired deals events will be listened by the Ross note and Rosner will be decide if if it is a new deal needs to kick off to do the renew and all these are the brief process of uh what's a deals Monitor and what Ross note should do so like I mentioned earlier this is a very very minimal features that we have put into this bag to make sure that this renew as a service or replication as service works fine end to end and but obviously there are more that we can add into this future for example if we look at the deal status small contract currently what we designed it for is for what the client has store for the seid and this contract will be able to get a list of the deals and keep monitor those deals periodically so there are two functions that we provide one is get active deals which we'll use to determine if there are enough replications for this CID on chain and then the second one is get expiring deals which is used to determine if it is there's any deal expiring that we need to renew for it so this is a very basic function but like if you think about more and you will think you probably will get oh but like when exactly should I monitor those uh contracts when should I when exactly should I monitor the deal do I need to get Oracle into the picture how do I do a chrome job to make sure that I don't run that uh static check too often and maybe like if if there's new renewing repair company uh what's the logic should I should I do right should I should I repair should I renew based on what the clients ask for or maybe I can renew uh in a different location and pick the different uh storage provider to do that and how to pay how the clients will pay the renew service or repair service those are a lot of more detail you can add and I think for this kind of a renew and repair rust service there and I root for the Ross node as well and we're designed what's known as the point of be able to monitor this deals make we have made through the aggregator and also be able to uh more have more into the expiring deals or active videos and taking actually kicking off the job uh of a new idea or replica idea or repair deal with aggregator or not aggregator on filecoin and so all this all very detailed spec or included into this QR code if you scan and you'll be able to find it and that's that's the part we are pulling it out and uh if if there's any developers are interested into building a replication renew as a service and you can go follow and go through this note and read this spec and welcome anyone to build it for it and if you're building a like a Perpetual storage or if you're building a storage service you're also interested to build this and also very welcome to do so if you feel free to reach out to us if you have any question and all this document has been puted out uh into our LinkedIn tree if you are interested you can scan and get all the information we're looking for like aggregator specs aggregate at respect or this uh renew repair service back and also if you can follow us on Twitter then we have a twitch happen every month to talk about all this indeed to detail I think I I first uh think in my I might don't need 30 minutes but I after I'm going through I feels like a lot of information to digest and I hope after this session you'll be able to go back and have a the the picture of how the the deal process is on Falcon looks like and uh maybe have the aggregation in my why it's needed and having uh have a clear picture of why we need a renew I need to repair when I need the replication you need that picture and if you're interested in building it just keep following us and if you have any question maybe you can ask now or maybe talk ask us on slack or Disco okay that's all my content for the day thank you yes if anybody has any questions please feel free to type them in the chat now or take yourself off mute and ask and as long as I said um if you don't have anything now go please definitely take advantage of the Discord partner channels as well to reach out to longfei and other partners via that route okay if there are no other questions or no questions thank you long favor for the great presentation and thank you everybody for attending and yeah we will see you all uh later cheers have a good one thank you everyone