Filecoin 🛠 Saturn & IPNI: Finding and fetching Web3 Content - Patrick Woodhead

foreign [Music] coin Workshop Saturn and ipni finding and fetching web 3 content and joining us today is Patrick Woodhead who will be taking us through the session and without pass it over to Patrick to get the session started hi there thanks very much and I hope everyone can see my screen my name is Patsy Woodhead and I'm a technical program manager at protocol Labs at the program which I manage is recruitable markets so that is how do we retrieve content out of filecoin and out of ipfs then how do we uh how do we accelerate that around the world to clients wherever they might be I've just seen something up in the chat yeah please please post any questions um and if they're really relevant I can I can address them immediately and if not yeah we can have a few questions at the end um great so the challenge statement for this bounty is as follows retrieval via distributed CDN build the most impactful application leveraging the Saturn Network and or ipni the data Discovery and verified retrieval and we're going to go into a bit more detail about the Saturn Network and about ipni and then what we really mean by we're discovering there and also what it means to have a verified retrieval um obviously we're working in a trustless environment and so when you when you retrieve content you do have to verify that you've got the content that you have uh that you've asked for so it's important that you're able to verify on the client's side uh so as I said the Saturn and ipni these are two projects which are being worked on our protocol Labs we're going to begin with Saturn and we'll go into ipni uh after that and then we'll also see how the two are connected a bit of motivation for the Saturn Network here is a statistic and I can give people the the figures behind this if they'd like to and I can share Links at the end yeah we buy 2025 the global CDN Market is expected to be twice as large as the cloud object storage market so that's really saying that retrieving files and accelerating files to clients wherever they might be it's actually a bigger Market than storing those files in the first place and actually when you think about this it doesn't make a lot of sense it's probably more straightforward just to Simply store a file somewhere uh in the cloud um or let's say a bit web3 equivalent here on filecoin um but it's probably more complex task to actually make sure that those files can be retrieved by anyone around the world in a performant amount of time um and economically and also reliably and people do want their their app said the apps to work very well and to be seamless and snappy so with that in mind uh we then look at what full coin and ipfs uh offer and uh where there's some problems uh with achieving that goal of having a really Snappy experience on the retrieval side so ipfs retrievals through the public ipfs gateways there are a few of these um there's one run by critical Labs which is ipfs.io and also dweb.link um firstly there's a centralized um the ones from my political labs and everyone's run by by other companies as well um and they aren't the fastest um they sometimes do take a bit of time for you to retrieve your files and also as they essentialize you're you're relying or you're trusting um that gateway to provide you with the file you don't have to do any verification on the client side they're your trust anchor um and you're just assuming that they're gonna return you the right files um so and that's really speaking about ipfs and then the filecoin side there isn't a cache layer to accelerate retrievables from Five Point Storage providers uh there's been a huge amount of work recently uh on how you can retrieve files from Falcon storage providers um it was a originally only available over graph sync which we'll talk about a bit more later and there's been a lot of progress recently in HTTP retrievals which uh which is obviously great because HTTP tooling is everywhere um but it'll be great if we can then start to see the emergence of these cdns these cache layers in front of filecoin so that you're not having to go back to the origin server of the data each time the third thing I can mention here is that uh historically ipfs and Falcon different networks they haven't really interoperated much um and going forward we'd like to have the whole content dressed Universe be able to be content addressed and not have sort of lock into particular networks um we'd like to see ipfs network and filecoin interoperate so that's the sort of the the background um that brings us on to what is Statin Saturn is a decentralized CDN accelerating retrievals from the ipfs and farcoin networks foreign Network launched in November 2022 and anyone can join to be a point of presence uh there are now over 2 000 points of presence around the world um a large network capacity of 11 over 11 uh tpps and a whole bunch of daily requests 158 million plus um so there's a lot of traffic and a lot of points of presence around the world serving that traffic uh oops go on to the road map um as I said like November we had the launch and by December we wanted anyone to be able to run an L1 node and so anyone can now go to the GitHub repo of the Saturn L1 node and get themselves set up uh by March we wanted to actually have some performance guarantees in this network and we started to see a really nice time to First Bite for ipfs Content flowing through the Saturn Network um in April we started to open kind of a beta testing Force for starting customers and in a program in the process of that at the moment and then just recently in May June we have essentially the new operators these points of presence they they receive payouts on a monthly basis um and since the launch of the fvm which is the Smart contract layer on the filecoin network since the launch of that we've been able to move a lot of the payout logic onto your smart contract and so Saturn node operators have recently getting paid by an fvm smart contract um and then go forward we're looking to link up the satin L1 nodes to Falcon ipfs and how about interoperability that I've already spoken about the so now got an architecture diagram um this is Saturn and also I'm bringing in ipni now we'll talk about that a bit more in a second uh but we'll focus a bit more on just the Saturn uh request flows and before we can motivate ip9 in a sec uh so on the left hand side if you can see my mouse we've got a client who is looking to get hold of a file I've got a CID and they want to get hold of the file so they can make a request to the Saturn Network um if they're a browser client this can be done over DNS to the theft network if they are on the server they can actually use this sort of SDK thing called Caboose which allows them to connect to a nearby Saturn node um and they make the request and at the stat one of the Saturn nodes picks up this request if it's in the Assassin node cache it's able to serve it back very very quickly oops excuse me um if not then the status nodes need to fall further back into cache Miss and get that data from another layer and that's where ipni comes in how does assassin know find where that content is um as you can see here and we'll come back to this diagram after I've introduced ipni um there are two ways that statino can get hold of it they can make requests to the ipfs network or they can make request to the Falcon Network and with different protocols bitter graph sync and more recently with HTTP now as well they're able to get hold of blocks and then return a car file to the client foreign Network there's browser retrievals uh server referrals and Gateway retrievals um the browser retrievals they may request straight to a Saturn 01 but as you're getting back a file which is you have to be able to verify it because it's from a satin L1 node that you don't particularly trust uh Saturn also has developed a service worker which incrementally verifies the files because they come back into your device and then streams it into uh the browser the browser page seven retrievals uh as I mentioned they would they do not use the DNS layer and they do not have the service worker because it's not browser but then you have a similar logic of incrementally verifying as fast as they come back uh they also have several tools give you a little bit more flexibility about parallelizing our retrievals from a bunch of different Saturn nodes and also have a kind of fall back if certain Saturn on one nodes are going down so just a little bit more of available Machinery if you're making retrievals from the server and this can allow people to build their their businesses uh around Saturn they can they can fetch stuff and fasten and then pass it on to their clients in turn um and then Gateway retrievals Saturn will soon have a Saturn Gateway where you if you don't want to do any of this uh apply inside verification you can just trust a gateways system that we're working with the ipfs.io team to to sort of turn the current ipfs gateways into um a more of a global gateway for any content on filecoin and ipfs okay gonna just do a quick little demo here of the browser retrieval um we've got a few steps here uh so the first thing is that we're going to add a script tag into a very simple HTML page um which allows us to get hold of this Saturn widget and we also fetching this that in the service worker and putting that into our repo um you then add that service workers to your domain which is just the replay I'm about to show you and then we can stream um NHS video so I will escape from here go across to here well that seems to be a bit broken but okay this is a bigger so this repo is on GitHub if anyone wants to uh um start from here or just play around with it make this a bit bigger as well okay so as I said we're just adding this this script tag here um adding the the file that you want to fetch and we have already downloaded the Saturn service worker and it's in the the root domain of this website and then very simply just going to run an MPX serve and other demo Gods on my side thank you they're taking a little bit longer than usual okay so yeah the video is coming through we can have a look at some developer tools um we start to see different chunks the HCL s file coming in from uh from starting to Maine and putting in different chunks of thought and this includes the ability to skip through to different parts of the video as well we'll stop there all right uh that's just one example of stuff you can do with Saturn um so I'm going to go towards the end to create you some more ideas like of all the different use cases that we've been talking about and give everyone a little bit of a taste of things they could uh they could hack on something links if you want to go to the homepage static head to saturn.tech at docs.

tech tell you how to set yourself up as a Saturn L1 node uh or give more information about how you can fetch from Saturn explorer.tech will be out hopefully within the next few days and it gives a world map of all of the different Saturn nodes around the world and it's kind of summary statistics about how much bandwidth each of those nodes are sharing you can head to the GitHub which is Falcon Saturn and that Statin demo just seem literally very simple just a few files um you can get it from here and you can recreate what I just what I just showed you head to Falcon file man slack and then there are YouTube videos for microvesting which go into a bit more depth on this as well ha okay now gonna I didn't mention ipni much when I went through this diagram the last time but just to come back to motivate ip9 when a satin L1 node doesn't have a file in its cache uh with if people know the ipfs network well you sort of get like an element of content Discovery from bit swap um bit swap is it asks the networks sorry the the kind of Neighbors in in the ipfest networks that you have whether they have a particular block which which corresponds to a CID and if not then you can speak to What's called the DHT and you can find out um which which nodes do have that CID but with with graph sync it's different graphs and you have to know who you want to speak to before you start so you sort of need to know which file coin storage provider has a file before you can actually fetch it and that's what ipni comes in IP now which stands for interplanetary network indexer is a way for you to discover which filecoin story providers and even ipfs nodes uh have the file that you're looking for so yeah as I suppose I've sort of already mentioned quite a lot of this uh bits on DHT do provide that content Discovery um one of the issues here is DHT is unable to keep up with the sheer number of Records or what we call advertisements um that people want to put up to show where their data is um and it can be a bit slow there's a few hops involved in going through the DHT to discover content um and graphs Inc and HTTP for that matter you need to know who you want to speak to before you before you start speaking to them so it doesn't have sort of content Discovery built into those protocols um so ipn9 um is a routing system designed for providing cids by the billion with millisecond lookup latency um so Ike and I in a similar that ipfs is a pro school ipni is a protocol and it's about the whole flow of you having a CID wanting to know uh where you should get hold of the file which you want so to look up and then actually uh providing the the CND user content behind it so that's the whole the whole flow so you have it you have a CID and you make a request to um a network indexer which is part of the ipni protocol and the network indexer will return to you um which provider which storage provider or even which I Professor node has that content and over which protocols it's available so uh bits for graph sync HTTP and more and more the stuff on on filecoin is available over HTTP so start to see the advertisements coming from the the network index there including HTTP as well as graph sync so how does the network indexer then know where stuff is stored over here we have a professional nodes or Lotus nodes or boost and when a storage deal is made um on filecoin or when something is stored in ipfs node they are able to advertise uh the which cids they're storing and they can either gossip that um or they can even just send HTTP requests to um Network indexer and that then stores a huge number of records of just a mapping of CID to provider um then once it's got that in a system it's pretty simple retrieval clients are able to make requests for a particular CID um or particularly multi-hash this is this a multi-hash is sort of made into the CID that's kind of a layer below um and find out who you've got hold of them and then this next step is once you know who's got it you just uh you just make that request and able to retrieve it from from the source I was very careful to say ipn is a protocol and um not a instance is anything um so if you come across different terms while you're looking into this I think this diagram can help iconize the overall protocol and these other things are sort of members of of the API protocol network indexer is this specific um a specific service that actually in Jesse records and offers to look up store the index is an implementation which was made by political Labs of a network indexer and then cid.contact which is also the domain name so hexcl.contact to find out more this is actually an instance of this store the index implementation of the network indexer spec um and an index provider is just a really yeah client of the network indexer who's able to provide um the ad resistance ipni links um github.com specs or go lib ipni as a Blog series on the falcon.

io blog um and if these slides are shared you're able to get this link here cid.contact uh docs.cr equal dot contacts head to the ipni channel on Falcon slack and again on YouTube if you go to ipfest thing and search for iconi um you will be able to find some amazing Talks by the engineers who worked on this Okay so we've now gone through Saturn and we've gone to ipni the next step is uh what could we actually what what sort of applications or use cases could we look into and actually I'm going to go to this next slide first um because I think this could help everyone figure out what they might want to work on um the Saturn team is still working very hard at linking up all of these different networks ipfs typing I filecoin and now nft storage um and it's right at The Cutting Edge speeding edge of what protocol apps is working on and so some there are some things which we have got to work like some of that HSS video streamings you've seen But there are other things which we haven't got to work so much so there's one one way of looking at this would be to just use the the apis of that and IPI ipni um and build something on top of them I think a sort of a more interesting approach would be to get under the hood and see if there's any way of adding functionality of features or having a go at interfacing some of these networks and finding optimizations or interesting ways in which we can get them to talk and interoperate uh more efficiently when it comes to all these different verticals um we saw an example of video uh but there are so many different things which we would like to to use a CDN for um so first years can we get the ipfs gateways not too centralized and decentralize those then there's huge amounts of nft um media files which are content addressed or on filecoin uh what's what's the best way that we can serve those um how do we index those in that in their Millions um in ipni how how can we optimize the flows around around these sort of files public data sets uh if we if someone's interested looking in like storage as well as retrieval can we get any public data sets on and then try and make them retrievable and just get that end-to-end flow going and the same goes for DSI a really interesting one is around blockchain transaction databases and also blockchain State can we make snapshots and store them on filecoin and then make them available through uh Saturn and ipni for fast retrieval and can we even address that the database availability problems uh by by using this can we help people to get hold of indexes or around blockchain State um so that they could their damps can and get hold of all the relevant stuff to to work seamlessly and and sort of have the metadata available and information available to make future transactions and know that you're up to date with the latest transcoding um this this feeds into into video and music and but what ways can we can we compute over data as it's moving to these retrieval networks uh add little functions um like for example live here it's got a whole bunch of of tools in the toolbox which could be integrated with Saturn um to create a node that's able to do some some really powerful stuff and that leads into decentralized compute uh could we sort of combine retrieval and compute networks so we've got sort of edge Computing or like almost like Edge lambdas um they're able to do little jobs around the place um and return the results to the client search and Discovery um we've looked at indexing but we haven't looked so much at sort of content searching so uh what are these files actually contain uh can I discover um music can I discover videos can I discover the blockchain state that includes certain things uh a more General kind of developer tooling around how we can feed stuff back into into Saturn or sorry back into the app from satin or from ipni and then yeah mobile music video um databases and also uh gaming which just different verticals will have different challenges and different opportunities foreign thank you very much I guess there's sort of five minutes left um if anyone's got any questions yeah if anybody has any questions please feel free to type them in the chat or take yourself off mute and ask her and if maybe if people are thinking um what would you Patrick what would you think uh I guess um maybe some project ideas you would like to see folks do or any sort of thoughts you have around maybe getting some creative juices so and if people are still struggling on what to what to what to work on yeah I think if people are starting what to work on I would focus on um this like some of the blockchain stuff so can we get other web 3D data sets and um get them into filecoin or into ipfs and then find a way to to retrieve certain indexes over that that blockchain state um and make that flow end-to-end work um we're speaking to a bunch of uh people interested in in kind of backups or snapshots of blockchain state so the validated nodes can get themselves up themselves set up very quickly but I think what's interesting there's another like a okay yeah a smaller project would be can we just get small amounts of blockchain state where you can you can create little kind of yes little indexes over overstate and and try and get that into the client as quickly as possible I think that'd be a very interesting project if for a first run yeah for embarrassing questions now's the time before we ended ended ah we have a question I want to create some data streams where should I start bubbles um that is a good question um I think the best place to start is just hitting up the um yeah satin GitHub or getting in touch with the team on Five Point slack um because we have got some um streaming capabilities particularly with the video stuff but yeah I think we're big just to get you in touch with our engineers and um and see where we can go from there so so I don't know anything other than about uh actually file Cohen ipfs or any of the rest so this is all very very new to me but one of the things I'm thinking about is um where would I be looking if um so I understand the idea that you pay for file storage and you want to held stuff but if people are consuming you know gonna want you want to publish files that people use um is there which of these Technologies if you wanted to get paid for the use of the data not just the hosting and distribution yeah so are you saying if you if you wanted to publish data and also be paid for who's gonna who's gonna consume that data right um well so if you want to you you can run an assassin uh point of presence you can get paid for serving the content um you can run a Fargo storage provider you can be paid for storing the content if you're a Content publisher it kind of depends on the business model there's because kind of in the traditional web 2 space there's actually the content publisher who pays to have their their content accelerated they want to provide a good experience to their users and most of the use cases we're looking at is sort of following that business model whereby content publisher is the one who's going to be footing the bill in the end um and then it will be sort of an out whoa that'll be an interesting thing to look at is there a way that you could have a trustless protocol which would allow your clients to then pay to access that content rather than it being sort of out of band your clients paying you through another mechanism so yeah that would be a very interesting project to look into okay but there's there isn't part of this file coin and ipfs and Saturn apni stack of things where it's really address tries to address that yet there there is we have got some work streams um where we're looking at paying directly for files so that's where the client actually transfers um makes micro payments for bytes of a file and then the team called magmo um who are looking into like a state Channel approach to that um which is super interesting there's a whole different set of use cases which will be unlocked by that um Saturn goes to the approach that the content publisher was the one paying which means that it's actually not the clients we're paying and actually when you across the board at the moment clients don't really like paying um it'll take like a paradigm shift in how we use the internet if we sort of have everyone making migrate payments so it's kind of two different work streams the micro payment and the supplements the payment right yeah you mean web three laughs thank you thank you hey are there oh yeah I have another question is there any way to manage access to token gated content using ipfs and leverage Saturn L1 to speed up the Authentication um great question there is there's there is some research going into how we could have sort of Access Control onto ipfs um nothing I believe has been actually shipped with any of the major ipfs notes um the Saturn L1 to speed up Authentication that again is not a feature that would be a really interesting to look into like access control and authentication um on yeah through this network one of the problems you face though is that the Assassin l1s can be run by anyone so uh if your or the authentication layer is across a bunch of 2000 nodes which you can't trust then it's I suppose a bit of a challenge like how are you setting up your authentication layer we've got some ideas around using um kind of dense and you can uh to manage kind of how a client can uh connect can have an account with a particular content provider and get access to their content as well as their metering how much usage each each client has got to the network um so yeah there is some research in that space nothing too concrete but we're looking at ucans as a way to um to measure and and have sort of identity around people accessing assassin Network thank you awesome well cheers well thank you Patrick uh we're currently at the top of the hour thanks for the great presentation and thank you everybody for joining