Filecoin 🛠️ Filecoin data onboarding with Delta - Elijah Seed-Arita

foreign [Music] hello everyone and welcome to the hack FS Workshop while coin data onboarding with Delta joining us today is Elijah cedarita who will be taking us through this session and with that I'll pass it over to Elijah to get the session started okay thank you hello everyone um my name is Elijah I am uh on the outer core engineering team at protocol labs and we've been working on a set of Cool Tools called file client data tools and it's a collection of tools to help with onboarding data onto file client and we'll get more into it very soon so I want to start off the talk just talking about uh the current process today of onboarding data to file coin yourself so if you haven't done it before um it could be a bit of a tedious process if you have done it before you probably already know that um there are a lot of steps involved and uh of course that first step involves finding your falcoin client um that could be Lotus Venus it could be something else like Phil client there are a lot of options but you're going to be um you gotta download your client read the instructions for it set it up fund it with your wallet configure it connect it to the filecoin network Etc um and of course uh you will probably only be doing this once to be uh Fair however just another uh stepping stone in the whole uh just process of getting your data onto filecoin um part two is beta preparation so you're going to be processing your data into the correct format and that's just being car files you got to split your data up into 32 gigabyte segments so they can fit into pieces on your file coin storage provider um following that once you've prepared all your data um you're gonna have to find your storage provider so you're going to search for providers providers you're going to get on information on chain listed um they're going to advertise their price um their peace size Etc you might also want to take into consideration some sort of um off chain indexes such as availability or reputation indexes and things like that um so once you find your storage provider you can move on to part four which is deal submission um and this is going to involve generating your deal parameters so you're going to have to specify your size of your deal your duration the max price that you're willing to prep to pay you got to prepare the PC ID the payload CID Etc um so once you've got all these deal parameters put together you've got to submit the deal to the storage provider the storage provider can then accept or reject your deal proposal um you can uh then you gotta check if it's been accepted if it has been accepted you can go ahead if it hasn't been accepted you might want to retrieve with a different storage provider a different deal parameters Etc um and then once your deal has been accepted you can prepare to transfer it to the storage provider that might be overload P2P it might be over um boost uh if you're doing an offline deal then you might want to set this up yourself with a storage provider and of course after you make your deal that is not all you need to keep track of the deal going forward if it expires if it fails you're going to want to renew your deal with a different storage provider or with the same storage provider um and just keep track of the deal and its health over time Etc so people do this obviously this is how it's done this is how it works and um you can do it it's just it's just a you know a lot it's um it's a bit tedious this mainly comes from the issue of a fragmented tooling ecosystem a lot of tools are split up between different teams you're going to be often cobbling together different pieces of Technology you're going to have like uh what one tool for preparing your data another tool for uploading it another tool you might have to be putting together your own scripts and often there's just a lot of knowledge involved and especially this becomes really prevalent as you're starting to get into uploading increasingly large amounts of data and this is specifically a place where we are looking to address the problem overall it works but it could be simpler so that is where we're looking to come in with file coin data tools falcoin data tools it's autocor engineering's solution to this data storage complexity I wouldn't say problem but something that can be improved um it's a unified and coordinated set of falcoin tooling so it's multiple tools and they've all been developed in a standardized Way by our team outer core engineering um so it's got an opinionated architecture and since we have been building up multiple layers of this technology um that means that we've been able to work on optimizations that go really deep through the stack through multiple layers of the stack um so on top of that we're a dedicated engineering organization so this is not a contract it's not like a Consulting engagement so it means that we're going to be continuing to update this continuing to support this it's not just a one-off one-off project that we're going to have to move on from later so we're going to keep supporting this and we're going to keep pushing out updates and improving things as we go so we do have also a real SP on this on the team that's going to be Json and slack or Jason chalica I hope I pronounced that name correctly I've never actually asked him about that but he's representing our storage provider requirements and it gives us um we've been able to have a very grounded view on the requirements on the storage provider side of making deals um and we've also got Solutions Architects namely uh Bill schreckenstein and they're able to help with white glove data onboarding and we can help you out with getting your data on and just facilitating that process so again filecoin data tools is a collection of tools and that includes sqa it's it's Delta Ptolemy Delta is the main focus of this presentation but there's a lot um and you can actually uh we've got great docs at uh docs.falcon data.tools and uh visit that link very soon um but yeah so the uh the actual component that's going to be making deals which is the focus of this presentation is Delta so Delta is file coin data tools steel engine microservice um and so it's API oriented it's controlled through API it's a Daemon and um it wraps up all the complicated filecoin deal making process into a single Service uh one of our big focuses for this development of this application has been supporting large data onboarding uh requirements and right here you can see we've got our GitHub link uh to the open source project you can download it build it run it yourself github.com application.research slash Delta um I will mention that so we've got a hosted option and we've got um your the open source version same version you can run it yourself if you want or you can use our hosted version um so Delta's features that involves card generation comp P generation piece commitment generation deal making we do both online and offline deals so if you're going to do an online deal we'll transfer the data to the storage provider for you if it's an offline deal then we'll just set up the whole deal uh uh we'll set up the deal basically and then you can handle transferring the data yourself we also do deal repair which is where if your data goes offline or if it's coming close to expiring we can repair it uh it we've got Bill status checking very straightforward statistics tracking of your deals uh wallet management so you can choose which wallet you're going to be using from your deal and storage provider selection of course so the architecture of Delta pretty straightforward we've got a rest API in the front um the rest API uh is just gonna listen for uh user requests to make deals and of course next to that we've got an ipfs node so if you want to transfer your data to Delta over ipfs instead of over HTTP that is an option behind that we've got the dispatcher job framework and this takes care of all the different types of jobs that Delta can do as I listed in the previous slide so that's going to be peace commitment data transfer deal proposal generation Etc and then behind that we've got a database which I think is pretty straightforward so when you upload a file to Delta first thing that's going to be done is it's going to compute a com P for your file then the um you have the option to provide a user provided wallet otherwise if you don't provide that then the default wallet will be used for Delta you've got a suitable SP will be identified after that and then a deal proposal will be created and submitted to that storage provider then the data will be transferred to the storage provider if the deal is end-to-end or online if it's an offline deal then we'll of course skip that step and you can handle it yourself so to note there is a one gigabyte minimum size on files uploaded with Delta however that's not a hard limit because we have other tools in our stack are built to address this problem so I'll get to that very soon so again I mentioned um we're able to do some unique optimizations based on having a more unified and uh like coherent stack and uh just two of those that I'm going to go over right now um the first one is going to be com P generation so back in Estuary V1 um we had this the coffee generation step was taking three to five minutes which is not great um to be fair however we have been able to bring that down to about two seconds on our dedicated Hardware which is a pretty substantial Improvement of course more optimization lies in the future we're looking towards hopefully getting to um uh sub seconds uh compute generation and of course uh we're also a team within protocol Labs as previously stated so we do have direct access to the cryptographers that build the follow coin protocol we can exchange information requests um Etc so here we've just got a graph um we've got Delta uh on over here on the right and gray and this is just com P generation times for different sizes of data up to 16 gigabytes over here on the right um so we've we've been able to make a great Improvement and this is uh mainly to help facilitate in uploading large amounts of data the other optimization that I'll go over and of course there's more but this is going to be our new filecoin data infrastructure or FBI FDI is our hosted infrastructure platform that's been developed by our infrastructure expertise on the team so we've got infrastructure and kubernetes expertise uh and we're looking to build effective horizontal Auto scaling to match our load um the componentized architecture of course also of Delta allows the independent scaling of services so we can match our load very precisely we have also learned from our previous project our previous offering Estuary V1 to be ready for the scale that's going to be involved so that's why we've been putting a lot of focus on building up this dedicated infrastructure so um that out of the way I'm going to go and give a quick demo of using Delta Delta is very easy to get set up for yourself I'm going to start with a demo of setting up a Delta Daemon instance so again we've got the hosted instance and then we have the demon uh the the local instance that you can clone for yourself if you want to use it so let me just open up my terminal so um of course this starts with cloning Delta that's going to be github.

com application Dash research slash Delta as previously stated I'm going to go into it make it I've already built it because I don't want to wait for it building the application on stream right now um but the configuration is really simple you're just going to be copying the example environment file into the [Music] um uh actual environment file dot ends and then after that you're going to have to get an auth token you can do that really easily by just running crl um to the auth.sqa dot Tech slash register new token endpoint and this will just give you a sure-off token so you can copy that go into your end file and I have my old token here but we can just put in this new one and there we go and then you can run the demon what's the demon so it's going to start up here and there we go running on Port 1414 you can start making API requests to it and it's good to go so for this presentation I'm actually going to be using the we have a special hack FS hosted um Delta running right now and it's running on filecoin calibnet so your deals aren't going to be actually on um you mean that let's get to that so making a deal with Delta very straightforward as well it's a single API request um so we're going to be doing is uh you're going to be making a request to the slash API V1 deal end to endpoint and you just gotta pass in your authentication token and um your data as well as your metadata so there's two options here you can send your data over HTTP or you can also choose to send your data over libp2p if you want and that'll involve um giving yourself or passing in a CID and a host to the metadata but we're just going to be using HTTP right now because that is simple so let's just make this VR crl curl request like that um so actually CSH has already um remembered all this um so we're going to be making this request to Delta slash API V1 Steel to end um we're gonna put our authorization header this is going to be our new um token that we've just copied and after that um we're going to be passing in the form data form cool okay done equals and I prepared a file here called um test file it's a one gigabyte file because again uh Delta itself requires a one gigabyte minimum uh file size so this is a um uh just a file of random bytes that I prepared with DD and the random you random Source anyway um after that I can just pass in the metadata you don't have to fill out the metadata but you do have to pass this parameter so we can run this now and uh so it's going to take a bit to upload because this is a gigabyte of data but again um so if you want to use lib P2P to transfer your data instead of using HTTP and the form data like this here you just pass it the CID and the host in the metadata there's other options available for your metadata as well you can choose which provider specifically you want to upload with um you can choose a lot more of the deal parameters here and that sort of thing Etc so let's just wait for this to finish after that we're going to be able to check status here status checking is really easy it's on the open endpoint or the uh the open API instead of the V1 API and you do not need an authorization token to access this all right so here we go we've got our response here and it's inside of our response we're going to get a CID this is our our content CID I think do it requested yeah CID this is our content CID and then here we've got our or sorry this is yeah the contents the 80 and then the content ID is local to the Delta instance it's 878. we can check the uh status of this content so girl a get request to um access coeus that is trade attack slash uh slash open slash stats content slash 878 so it looks like there's been four content updated since I last random command and here we go um yes so moving on this is if we want to do one gigabyte oh yeah and here is the um here's that uh API address for the hack FS delt instance so if you want to do one gigabyte file this is great however what if you want to do smaller files what if you've got like I don't know a bunch of little tiny files and so that's where Edge Ur comes in eduar is built on top of data Delta and it's a Content aggregator so what it does is it'll take multiple different files that have been uploaded and basically pack them together and then make them all upload them all to storage providers and batch deals um so those files are going to be aggregated into this bucket and then they're going to be yes submitted to the storage provider so again um this allows us to upload much smaller uh deals or much smaller uh pieces of data and it also will immediately make the data available over lib P2P if you want to access it immediately since the batch deal is not going to be made immediately since uh it's going to be it's going to wait for multiple pieces of content to be added so um making it this deal make a deal with Ed you are let's do that um I've got my deals with eduar example here so we've got a hack FS instance for Edge as well it's going to be hack FS Co usray.text Edge instead of Delta so making a deal with edit first let's switch to Edge and make a request uh or it's just LS here I'm just going to make a deal for some little tiny file here let's just make it for a Docker compose file of the edge repository so I'm just going to curl um host https hacker pass 0us.xray.

tech slash Edge looks like we've got an autocomplete here already as well um slash content slash add again add our authorization header the lost my authorization header that I just got I'm just going to reuse this one here yep our data that's going to be let's we're going to do the docker composer Lionel um and then the last um because of the request the form uh metadata that's going to be nothing because we're not selecting anything specific so let's run this that happened really fast for now uploading a gigabyte this time so that was much easier again we're going to get the ID um Content ID is going to be this year and we can use that again to look up status now we can curl to I could pass straight.sec slash and then this time we're going to query to Edge slash open slash status content and then we're going to paste in what was that ID here one six eight one seven and we've got our status here so our status success kind of cut off right there but yeah so this is going to be pinned now and you'll be able to access it over ipfs from the edge ipfs Gateway now um so this is great this lets us upload a very small pieces of data now on the other side of things what about large data sets so large data sets is something that we've really been focusing on because we want to be able to unload um or sorry on board very large pieces of data and we're talking like very large data sets on the terabyte scale um and so the way that we are doing this is with Delta DM ltdm is um stands for Delta Data set manager and it's basically a tool again built on top of Delta for managing uh massive data sets with potentially long-running upload duration so it's not something that you're going to be wanting to do in one HTTP request so this is able to replicate data to multiple providers you can manage your upload configurations you can filter which providers you want to be getting which data sets which wallets you want to be using for which data sets you've got a whole record of your actual deals that have been created and the best part about it I think is that we have an accessible UI which is very exciting so the repository for this is github.com application that's research slash Delta DASH DM again can build this yourself check it out yourself do whatever you want with it as it's open source so the data set upload process with Delta DM starts with creating a data set then you're going to attach contents contents are going to be you're going to have a file generated in an external program this is going to be Ptolemy or some other external program Ptolemy is still in the works but um this is going to be integrated into Delta DM so this is going to be a completely end-to-end process in this one UI which is going to be very nice um so you'll attach your contents that lets you upload the actual data that you want to send to storage providers possibly multiple um then you're going to register your wallet and Associate it with that data set because you're usually going to be having if you have a fill plus grant that Grant is going to be for that specific data set um so you're going to want to associate that wallet with the data set that's designed for after that you can add whatever providers you want to be able to make deals with and then you can set up your replication profiles which will start automatically replicating these data sets to the providers that you've specified so let's just do a quick demo of this Qi I think I already have it running locally there we go so this is our UI very nice very beautiful um so again the first thing that you are going to be wanting to do is make a new data set you can call it whatever you want so hack effects for example you can set your application count your deal duration um and setting it up and then so now that's going to show up right here then you can go in and attach content I don't have any content to attach right now but this is where you would attach a Json file describing the content that's been generated in Ptolemy and again pathology is going to be integrated into this UI so that'll be very easy after that you can go in and add a wallet um I've already got a wallet right here but normally you would add a wallet via the CLI so that you're not potentially um transmitting your wallet Secrets through HTTP unsecured so that'll be done through the CLI but after that once you've got a wallet you can go and manage and you can add which data sets you'd like to make deals with using this wallet so if I wanted to add my hack FS data set we can have that here save it and now that's going to allow hack FS to be done with this address after that we can go into our providers we can add whatever provider we want if I want to add F0 blah blah blah blah um called hack fs and go there manage it change the whether we want it to allow self-service which allows storage providers to come in and request data for themselves and after that point we can go into replication profiles and this here you can choose like stated before which data sets you want to be replicated to which storage providers so if I want to replicate for example to the hack of s search provider with the actifes data set we specify that and after that the Daemon will start automatically making deals to this hacker Fest provider so yeah that's that's our solution for making very large uh uh deals with filecoin data tools and with that I think that is actually all I have to present so yeah more info go to Falcon data.tools for great docs docs really well prepared I'll actually pull that up right now Alpine data dot tools that's going to be the subset here feel free to go check it out um yeah I think that's it so thank you very much for coming thank you for listening uh if there's any questions um we can answer that but if not then that'll be it yeah thank you yeah cheers thank you Elijah yeah if anybody has any questions uh please feel free to type them in the chat or take yourself off mute thank you and if not uh go please remember that you can always go to the data tools a partner Discord Channel where you can uh reach out to Elijah innovators from the team and just the partner channels in general if you have any questions uh specifically around the technology uh thanks Elijah for the great presentation and thank you all for for joining today we'll be back tomorrow with a few more sessions so cheers thank you all have a great rest of your day thank you have a good one bye