Web3 Storage and NFT Storage - Alan Shaw | HackFS 2023

foreign [Music] our next presentation uh comes from Alan he needs no introduction but I'll give him one anyway he's a software engineer working with protocol Labs uh currently working to make the distributed web more accessible to developers Alan if you wanna get started I will hand it over to you all right um welcome to this presentation I'm Alan um and this is a mashup of two talks about storing data on ipfest and filecoin using web free.storage and nft Dot storage so let's do it and first of all I'm going to talk about nfts um quite a lot the the kind of off-chain side of them uh how they can be stored decentralized on ipfs and filecoin for the Long Haul um so um I'm kind of new to nfts most people are they have been around for that long um but when I found out about them I was uh I was super excited to see that um the the ipfs was already being used uh to to store them um so for example if I go to like any kind of uh website like this one for instance this is the this is the game it uses nfts they have like a um a kind of marketplace uh and they have like nfts that you can buy and trade and stuff uh let's see let's have a look at I don't know if this one for instance um I happen to know that this um this particular game uses the marketplace of marketplaces um called open c open C there we go um and I can just paste in hopefully the um the uh token ID of that this particular nft hopefully I click on it oh did that come on there we go is it working hello there we are there it is those come on internet no open C is not having a good time at the moment all right um let's try one more time okay right anyway this is a cool car uh but yeah I can click through and I can see that that its metadata is here I should just be able to click on this this talk is like fraught with uh with danger so um we shall see how it how it goes it doesn't look like things are going to work out so well today for me um anyway this is going to igfest.io um uh the these this uh you can see in the bottom left there that it's a CID on uh ipms.io and the metadata for this particular nft here it is it's just come up uh it's stored on ipfs so hooray for that that's great um and uh hang on here we go back to the notes there we go um and but why why is data for nfts stored on Ike FS anyway well it's because it's kind of prohibitively expensive to store data on chain on the ethereum blockchain at least and ipfs actually has some really nice properties for off-chain data that um we'll go into in a little bit um and so um as I learned a little bit about nfts I came to realize that um the the developers that are enabling nft creators so like the the open c for example like Foundation like Zora they all really cared about users being able to access their data should they ever just like disappear off this off the face of this earth like if they if they fade away like that they still want people to access these nfts that they've bought and and created and created um so they don't want the nfts to just disappear also and so um ipfs uh this this comes into play now and this this thing called ipfs can solve this problem it actually fits the bill really well um and um from my point of view at least it's not really surprising that it's being used but uh so let me just rewind a little bit um and explain um some basics of ipfs in hopefully not too much technical detail um but I've chosen a uh a little um nft here this is a uh little one-eyed squid thing that's on actually on Sandbox that Sandbox game um and uh anyway so if I put him on itfs firstly what what do you get you get AC ID it's called a Content identifier and cids are a kind of cryptographic hash and they address the content not this location they address that squid with the one eye not where it is um and um and that means that any given CID will always refer to the same piece of content so if that um if that squid were to grow another eye for instance then the CID for that squid would actually change um and what typically happens is those cids are stored uh on chain and they're linked to from chain and if that wasn't the case if if we used like a regular URL to like Google drive or like a Dropbox URL or whatever that could be problematic why could that be problematic well um it the content at the end of that URL could be deleted by the owner the domain could could uh could expire all the hosting like it could be not be paid and then and then just go away the the image could or video or whatever it is could be altered in some way so I could have my squid turned into a cute cat or something which which would be disappointing um or for instance the URL just might not be accessible from within your country or maybe you're in like a some sort of corporate firewall or something where you actually need to get hold of that data from um a pair that's maybe closer to you than the internet location where it where it actually is so ipfs how does it work anyway well um so if someone who's using ipfs um requests that data by a CID then um the person who has it will send it back to you and they'll receive it you'll receive it so um and now two nodes have that same content and they can serve it to anyone else who comes along so uh like this if someone else comes on and goes do you have this CID the content for this CID and uh and uh either one of those notes can help to transfer the bytes to that uh to that node we call it uh bit swapping uh because why not uh and and so that that happens and now free of them have it and the cool thing about this is that the original node who had the content can actually go away like on gone um and the data will still be retrievable from other nodes uh in the network and this is the awesome power of peer-to-peer hooray um caveats there's two caveats the first one um the first one is that if all of the nodes uh that have the content um go away then you're not gonna be able to retrieve it it's kind of kind of obvious worth stating if there's no one on the internet who has the content you're not gonna be able to get the content that's caveat number one caveat number two is uh slightly less obvious it is garbage collection and so depending on your um configured kind of GC uh configuration it might run at a particular time it might run when your configured repost storage limit is hit so you might be fetching things around ipfs it all goes into your repo and eventually you'll use enough data that ipfs will go okay we're kind of full now uh let's have a let's do a GCE run uh and uh and the third other um time that it might be run is it could be run manually so you can actually invoke GC from the command line um or you can uh if you've got the HTTP API exposed you can call it from like that so what you need to do is tell like the test this content is not garbage um because um when you obtain content from the network that you didn't actually add yourself that content is eligible for garbage collection so if I got this picture of the squid from someone else then um it could be garbage collected in the future so what you need to do is kind of tell your ipfs that this uh this uh piece of content is a rare nft and it's not trash don't throw it in the bin please um and in ipfs that's referred to as pinning and if you don't pin your data then it will eventually be garbage collected so um even if the content is super popular you can still eventually um exit the network and so these two caveats are why uh developers generally use a third-party service like pinata or textile Fleek to store their content um because they have nodes that are publicly available and they stay online like 24 7 and obviously they pin the content so it doesn't disappear cool so nft little storage uh Molly just mentioned this earlier so you kind of already know a little bit about it it's something we've always been working on for a while now it's a place where you can store your nft data decentralized on ipfs and filecoin safe in the knowledge that'll be there tomorrow um and it hopefully goes some way to addressing that permanence problem um and uh it's free it's free as long as possibly uh as following us we can possibly make it free um and the one of the founding kind of principles rules is that it should be easy to store data using this service so how does it work well and you upload your nft data to nft. storage um and nft storage stores that data on ipfs nodes in the network that's right we actually have a cluster and we store it in on uh free um free nodes at least um so okay so far so good but how's that different from any other like pinning service well um behind the scenes or nft.

storage is doing is it's negotiating deals to store your data with storage providers on the filecoin network and so the filecoin network you might already know incentivizes the storage providers to store data quite heavily they get paid to store data they also get typically get paid when data is retrieved and they also get Block rewards when uh for for providing capacity to the network uh so why why is that good well if NFC that storage were just to happen to just go away like it's gone then um and and also every Miner on the uh sorry every um ipfs node on the network went away as well then um then the file coin storage providers who had have the content will still have to we'll still have it and they will store it for the duration of of the deal that's been made for them and so you can think of these storage providers as just another set of ipfs peers with really strong incentives to constant continue storing that data uh and this is this is hopefully a really good level of redundancy basically and should uh like I said go some way to solving that permanence problem so um who it is again NFL search free decentralized storage for nfts on ipfs and Falcon um and uh the URL is nft.storage I've said it about a thousand times you should go take a look um we have a simple API it's Express doesn't open API schema if you don't know what that means it means we can generate docs like this we can generate like actually generate clients from this schema uh which is which is kind of rad in different languages um that's that's really cool um the API is real small um you're probably going to need even smaller subset of it like these two like if you want to upload a file just post to slash upload but what what I really want is to draw your attention to here is this um this check API and and so with this API you can actually check uh the status of any uh the CID of any nft that's been uploaded to nft storage and the output for that kind of looks like this and so you can see here like the uh the like pinning status of this particular CID but also uh the deals with uh file coin storage providers that have been made or are being made currently and so you can see their status um like time date stamps for when things have happened chain that the on chain deal IDs so you can go and look it up you can see the the address of the storage provider who's who's actually storing it or going to be storing it and what's interesting what's really interesting here is that there's two things there's um there's the batch root CID here and and also the data model selector they're the kind of important ones because what happens is that as data comes into NFC that's already gets uh kind of aggregated together into this big uh big pile of data and that gets that gets sent to Miners and um and then um if you ever want to get your piece of data out of that big aggregate then you're going to need the data model selector and that allows you to pinpoint your particular bit within uh within the aggregate uh so anyway this is this is a cool API that you can you can use to check stuff um yeah we also have the same thing in web free.storage which I'm going to talk about in a sec um JS client we have an official JavaScript client this one isn't generated from open API we've lovingly handcrafted this one for you uh it's got typescript types it's got a nice API docs um we've got a metadata API as well and this metadata API is designed uh to make minting nfts really easy for you by um guiding you through creating like ERC 721 and 1155 compatible metadata um uh and uh we will take a look at that um let's take a look at that now all right here's a here's an actual dinner demo so let's see how it goes there my goods are um hopefully gonna help me out here here we go here's the whole whole desktop okay um so what I'm gonna do is I'm going to use this metadata API I've already installed the client from nft.storage using npm I'm importing it like this and I've just created a um a version here I've created my token already on the website just go to nfc.storage and create it I've exported it in my uh terminal so that should just work um and what I'm going to do is I'm going to try and store this this image called pinpai uh because it's appropriate about pinning and stuff so uh pinpai let's try and install this using the metadata API so what I need to do is uh call client.

store and then we give it an object that looks really similar to something that you'd get in when you click through from your um token URI uh which is minted on track on chain uh two and you this what is stored on ipfs essentially the metadata for the nft where the actual uh image file or video or whatever it is can be stored et cetera Etc but it looks uh it's really simple it looks something like like this I'm going to call it pinpai it has a description uh and record we're gonna put it and buy a comic because it's a comic thing and then what's kind of cool here is that um we put the the the properties called image and this has to be essentially the URL to an image where your nft which is your nft essentially um but what we put in here is actually just a file and what happens behind the scenes is that um the nftl storage client will automatically upload that image for you and replace that image with a UR URL to the um the the ipfs asset which is stored on ipfs so uh let's just quickly do that create a file I'm going to use fs.reath file sync I'm going to get the pin by jpeg [Music] you have to give it a file name which could be different from the actual file we're reading from I'm just going to give it a mine type oh well slash bug uh cool uh so we this returns a promise so I need to await on it and I get back this object uh I get a subject this metadata object and I'm just going to log out the metadata set Dot URL and this is the this is the URL that you'd actually use in your minting process and that would be stored on chain essentially uh logs now today I'm just going to print out the data here just to show that this file gets transformed into a URL to this um this particular asset uh I think that's all I need to do so we'll see if this works uh store I mean the wrong directories okay sorry about that right here we go so let's just take a look here uh so yeah so we printed out this this locked out the uh URL this is the URL that we're gonna put in uh on chain uh and also um here is the data and you can see this image got transformed into a URL as well uh so that's super cool um here we go and what I should just be able to do is head on back over to uh to here and uh if I put this into my browser uh uh yes there we go and I should just be able to fetch it from any ipfs HTTP Gateway um and this is the data that I uploaded to ipfs so super cool all right um that's that demo successfully done uh stop that sharing [Music] right let's get back to the slides because slides sides are here here we go and cool all right so what what do we do there well um we actually uploaded data um and then we got back a CID from nft.storage um and nft.store which is doing the work of um generating that CID from the data that we sent it um and there's a couple of problems with that um firstly we have to trust nft dot storage is doing the right thing it's not giving us the CID of some malicious virus in some way uh it wouldn't do that but you know you don't know in this trust us peer-to-peer World we're trying to inhabit um so it's better to to have generated that CID ourselves why can't we do that why don't we do that um well um the other thing before we get onto that the other thing is that um that that we have to um wait for that CID to come back to us we have to upload that data and um uh and wait for that crtd to come back when actually it would be really useful to have that CID before we uploaded because then we could mint our nft on chain uh whilst it's uploading and if it's something big then it might take a while to upload so actually having that Cod um generated on on the client side before it gets uploaded is really useful because then I can use it somewhere else I can maybe even store it in a database in my in my app or whatever um so so it means that I can I don't have to trust nft storage and um I also get to get to solve this ux issue basically um so yeah why can't I just generate that's the ID before uploading the data Well turns out you can uh new is storing Cars and Cars are not automobiles in this world cars are content addressable archives and it's kind of like tar for dags and dags are directed a cyclic graphs and they are the things that the ipfs generates when you import data uh to it so yeah tough dags if that's a if that's the thing anyway um the so it allows you to know that's the ID before you upload um and in the client um yeah well it's so on the client side we can do the work to generate that CID um and it means assembling that data into the graph but then we have that CID we have that group file but once we have the graph we pack it into a car and send the car instead don't send the data send the car it's got the dag and then that can just be stored by nft.storage you know you know what's being stored is the the thing that you added because you generated the CID on the client side so there we go um and it also means that we can split it up into multiple cards like you can send you can send partial partial dags like bits of it and that's cool because if you've got like a loads of data you can split up into smaller bits and it also allows us to send data that's more more than 100 megabytes in size one of the um kind of quirks of the service is that we just decided to put it on like cloudflare workers and they are cool because they put uh put you know your worker close to whoever's requesting that API or using that API but they do have a limit on like how much data you can send them in in one chunk uh but it's okay because we can chunk up things so we don't have to we don't have to abide by this 100 megabyte rule anymore so you typically end up this is completely contrived uh diagram but you might end up with like something like this a yellow line it uh kind of rings around the nodes in the graph that you'd maybe be sending in one car so you might send this set of nodes in one car you might send this set in another car and you might send this set in the third car and we know that all of these cars are related because they have the same root CID that's pretty cool um and that that allows like failure and retry without sending everything again we can just send any any car in any order uh because it doesn't matter um and um and that allows us to also report on and progress after we finish sending a chunk so we can say we've done and then you get like your UI feedback so that's cool all right let's try a real quick storing the card demo I know I feel like this is going way over time I'm sorry uh all right so here we go and stop the share let's share my whole screen okay cool um great so this is very similar to what we had before I've had to I've imported this extra ipfs car module this is going to help us pack a car from our content um the path I'm going to use is I'm going to use pinpai again uh uh simple Jeff and the content is good the content is actually so that's just the uh the file name essentially uh read file read the file then but actually I'm not going to use tin file at all because what I want to demonstrate is actual uh squirting and chunking I've got my skate movie here it's 24 megabytes that's just uh let's just make sure that here we go here's my Escape movies 24 megabytes that's great um I'm gonna I'm gonna use that um what happens is that the cars get split into 10 Meg chunks so we should see um two uh free chunks uh two 10 Meg size and one format uh so let's get Ruby here it is hiking here we go uh and and so the cool thing is at this point here we've packed or this uh this video into a a car file uh and we've also got the CID before we've even sent it um to our clients so we can we've got the CID here and we can when we get the CID returned from the client we can verify that the CID we created is a CID that has been returned to us so I'm calling this method called store car pass at the car and um and the cool thing is that we've got this um nice callback which gives us like the size of the chunk that was the car chunk that was stored so we can log out uh we can log out the uh how many bytes we've stored uh for each chunk that gets sent hopefully that's good to go still um there we are uh npm node node index let's try again here we go so we got so you can see here we've got that CID like straight away um and so now what's happening is uh well the CH there we go we've got one chunk got sent that's about that's like four Meg and so these other two here's a 10 Meg chunk just come through another 10 Meg so we've got three chunks got sent they all got sent in parallel we send up to three in parallel um to not starve browser browsers but um that's that's the kind of cool thing about it and the four megabyte uh chunk even though it was maybe at the end of the data it it won because it's smaller uh so there you go uh and the CID we've got back is the same as the CID that we um that we uh we calculated that's pretty cool back to the slides all right uh we are getting on for finishing off but at this point I've talked a whole lot about nft storage um but web free dot storage is the brand new thing it's just just launched almost almost literally uh and um and with free dot storage is built for any data so it's not just nfts you know if you remember this slide from earlier then um web free dot storage that's exactly the same thing uh so uh but it's just not not it's not just for nfts uh it's for any data so you can put uh put images videos uh you can put uh documents uh whatever you like even ipld data as well so um you can pick what works for you the best so if you're going to be minting nfts then you're probably going to want to go with nft.

storage it gives you the tools as you saw to create and store nft metadata um and uh and that that and that metadata will comply to ethereum standards um it also follows best practices for linking to data stored on ipfs um and um and for everything else then web3. storage has you covered so you should uh check that out the other the other difference right now um is that um cars are baked into the web free dot storage client that we've we've made this is a nice nice JS client we've done for web free.storage as well um we'll catch up the nft. storage client at some point but at the moment you can see for for storing files at least um you just call like client dot put and then it does uh there's no mention of cars here like it's just it just does that for you it um it will create the dag it will split the car up it will um upload it will do um uh retrying if it should fail for for example so um so hopefully uh even easier uh than out of T dot storage all right so let's really quickly dive into the architecture um this is essentially a fork of nft. storage um in the future nfc.

surers might the back end to nft.org might be where three dot storage um we'll see how that goes but um they are very similar at the moment very similar and and so what happens is that users interact with the HTTP API they send car files and we put stuff into our ipfs cluster that's cool and we have like a six node cluster we have we put stuff on at least three of those nodes in the in the cluster so that's kind of cool uh and we back up everything to pinata pinata uh provide a pinning Service as as you probably have um come across um if you've done any kind of ibfs stuff um and uh yeah that's there um the storage broker service this is where all of the interesting stuff happens basically and what what it does is it grabs all of the cids from our database a database is called Corner um no idea why uh and anyway so it batches them up into uh presence so they're not really batches there are Aggregates of data essentially now presence for Miners and they are the presents are actually cars as well everything is cars um and um and what happens is the storage providers they come and eat them up uh not really literally eat them but they will take them because it's a self-serve kind of thing they're like give me give me the deals and store that data cool okay um last last bit uh free when we when we say free we often get questions like why like how how does it how's this impossible well um Let me let me tell you a little bit about how it works this is this is essentially um that that bit in the bottom right hand corner of uh um our diagram here just uh expand it out this is one particular storage provider who has some hard disks uh anyway so what happens is that storage providers commit their hard disk space to the file core Network they they basically say I've got they've got this capacity uh and they get like significant block rewards for doing so and that's this this kind of circular thing because that happens uh essentially as as the block progresses they get they get rewards and this translates into real world profits profits are down here these money stacks at the bottom left um and uh and yeah so so that's that that happens and that that means that they're really like incentivized to keep uh committing additional hard disk capacity to network so they keep doing that like more and more um just so they can they can make more profit and of course why wouldn't you do that um however when uh the storage providers are storing data from users on the network their likelihood of winning block rewards goes up by a big factor of 10x so um wow look at those fits they just roof raced up um so you can see the potential profits to be made here by storing real user data and this this is such a powerful incentive for final coin storage providers um to to store data such as the data that is coming from a credit storage um or nft. storage that they're they're willing to offer um free storage in order to get this block reward multiple and in some cases storage providers and want to store data so bad either they've actually paid users to store their data so anyway so as long as these block rewards continue to be really powerful incentive um which will be true for a long long time and for example like Bitcoin block rewards are still an incentive for Bitcoin miners today um storage should be free so all verified users on the filecoin network right now um and so if it's free for us to store data on filecoin and that's that's free for you to store um to store data with us so uh yeah if you want to learn more about the the mechanics behind that and how to be a verified user then if you go to the filecoin docs um and then click on this there'll be a link there for filecoin plus it's docs.filecoin.io um and um and you can you can find out uh how to do it and that is uh mostly all I have to say uh but thank you so much for listening and um hopefully I haven't gone too far over