Web3 Storage and NFT Storage - Alan Shaw

as as was stated our next presentation uh comes from alan uh he needs no introduction but i'll give him one anyway he's a software engineer working with protocol labs uh currently working to make the distributed web more accessible to developers alan if you wanna get started i will hand it over to you sure thank you okay let me just share this with y'all all right um okay if you don't see the screen then let me know it's gonna be boring um all right um welcome to uh this presentation uh i'm alan um and this is a mashup of two talks uh about storing data on ipfs and filecoin uh using web3.storage and nft.storage so let's do it and first of all i'm going to talk about nfts um quite a lot the the kind of off-chain side of them uh how they can be stored decentralized on ipfs and filecoin for the long haul so um i'm kind of new to nfts most people are they haven't been around for that long um but when i found out about them i was uh i was super excited to see that um the the ipfs was already being used uh to store them um so for example if i go to like any kind of uh website like this one for instance this is a this is a game it uses nfts they have like a a kind of marketplace uh and they have like nfts that you can buy and trade and stuff uh so let's have a look at i don't know this one for instance um i happen to know that this um this particular game uses the marketplace of marketplaces um called open c c there we go um and i can just paste in hopefully the um the uh token id of that this particular nft hopefully i can click on it did that come up there we go is it working oh hello there we are there it is oh come on internet no open c is not having a good time at the moment all right let's try one more time ah okay right anyway this is a cool car uh but yeah i can click through and i can see that its metadata is here i should just be able to click on this this talk is like fraught with uh with danger so um we shall see how it goes it doesn't look like things are going to work out so well today for me um anyway this is going to ipfs.io um uh the this uh you can see in the bottom left there that it's a cid on ipfs.io and the metadata for this particular nft here it is it's just come up is stored on ipfs so hooray for that that's great um and uh hang on here we go back to the notes there we go um and but why why is data for nft stored on icfs anyway well it's because it's kind of prohibitively expensive to store data on chain on the ethereum blockchain at least and ipfs actually has some really nice properties for off-chain data that um we'll go into in a little bit um and so um as i learned a little bit more about nfts i came to realize that um the the uh developers that are enabling nft creators so like the the open seat for example like foundation like zoro they all really cared about users being able to access their data should they ever just like disappear off this off the face of this earth like if they if they fade away like that they still want people to access these nfts that they've bought and uh and created and traded um so they don't want the nfts to just disappear also and so um ipfs uh this this comes into play now this this thing called ipfs can solve this problem it actually fits the bill really well um and um from my point of view at least it's not really surprising that it's being used but uh so let me just rewind a little bit and explain um some basics of ipfs in hopefully not too much technical detail um but i've chosen a uh a little um nft here that he's a uh little one-eyed squid thing that's on actually on sandbox that sandbox game um and uh anyway so if i put him on ipfs firstly what do you get you get a c id it's called a content identifier and cods are a kind of cryptographic hash and they address the content not its location they address that squid with the one eye not where it is and that means that any given cid will always refer to the same piece of content so if that um if that squid were to grow another i for instance then the cid for that squid would actually change um and what typically happens is those cids are stored uh on chain and they're linked to from chain and if that wasn't the case if if we used like a regular url to like google drive or like a dropbox url or whatever that could be problematic why could that be problematic well um the content at the end of that url could be deleted by the owner the domain could could uh could expire or the hosting like it could be not be paid and then and then just go away the the image could or video or whatever it is could be altered in some way so i could have my squid turned into a cute cat or something which which would be disappointing um or for instance the url just might not be accessible from within your country or maybe you're in like some sort of corporate firewall or something where you actually need to get hold of that data from um a peer that's maybe closer to you than the internet location where it where it actually is so ipfs how does it work anyway well um so if someone who's using ipfs um requests that data by a cid then um the person who has it will send it back to you and they'll receive it you'll receive it so um and now two nodes have that same content and they can serve it to anyone else who comes along so like this if someone else comes on goes do you have this cid the content for the cid and uh and uh either one of those nodes can help to transfer the bytes to that uh to that node we call it uh bit swapping uh because why not uh and and so that that happens and now three of them have it and the cool thing about this is that the original node who had the content can actually go away like gone gone and the data will still be retrievable from other nodes uh in the network and this is the awesome power of peer-to-peer hooray um caveats there's two caveats the first one um the first one is that if all of the nodes uh that have the content um go away then you're not gonna be able to retrieve it it's kind of kind of obvious worth stating if there's no one on the internet who has the content you're not gonna be able to get the content that's caveat number one caveat number two is uh slightly less obvious it is garbage collection and so depending on your um configured kind of gc uh configuration um it might run at a particular time uh it might one run when your configured repo storage limit is hit so you might be fetching things from ipfs it all goes into your repo and eventually you'll use uh enough data that ipfs will go okay we're kind of full now uh let's have a let's do a gce ram uh and uh and the third other um time that it might be run is it could be run manually so you can actually invoke gc from the command line um or you can uh if you've got the http api exposed you can call it from like that so what you need to do is tell i give us this content is not garbage um because when you obtain content from the network that you didn't actually add yourself that content is eligible for garbage collection so if i got this picture of the squid from someone else then um it could be garbage collective in the future so what you need to do is kind of tell your ipfs that this uh this piece of content is a rare nft and it's not trash don't throw in the bin please um and in ipfs that's referred to as pinning and if you don't pin your data then it will eventually be garbage collected so even if the content is super popular you can still eventually um exit the network and so these two caveats are why uh developers generally use a third-party service like pinata or textile fleek to store their content um because they have nodes that are publicly available and they stay online like 24 7 and obviously they pin the content so it doesn't disappear cool so nft storage uh molly just mentioned this earlier so you kind of already know a little bit about it it's something that we've been working on for a while now it's a place where you can store your nft data decentralized on ipfs and filecoin safe in the knowledge that will be there tomorrow um and it hopefully goes some way to addressing that permanence problem um and uh it's free it's free as long as possibly uh as long as we can possibly make it free um and the one of the founding kind of principles is that it should be easy to store data using this service so how does it work well um you upload your nft data to nfl.

storage um and nft.storage stores that data on ipfs nodes in the network that's right we actually have a cluster and we store it in on free free nodes at least um so okay so far so good but how's that different from any other like pinning service well um behind the scenes what nft storage is doing is it's negotiating deals to store your data with storage providers on the file coin network um and so the file coin network you might already know incentivizes the storage providers to store data quite heavily they get paid to store data they also get typically get paid when um data is retrieved uh and they also get block rewards when uh for providing capacity to the network uh so why why is that good well if anything that storage would just uh happen to just go away like poof it's gone then um and and also every miner on the net sorry every um ipfs node on the network went away as well then um then the file coin storage providers who have the content will still have that will still have it and they will store it for the duration of the deal that's been made for them and so you can think of these storage providers as just another set of ipfs peers with really strong incentives to contin continue storing that data and this is this is hopefully a really good level of redundancy basically and should uh like i said go some way to solving that permanence problem so um here it is again nfl storage free decentralized storage for nfts on ipfs and falcon um and uh the url is nft storage i've said it about a thousand times you should go take a look um we have a simple api it's expressed as an open api schema if you don't know what that means it means we can generate docs like this we can also generate like actually generate clients from this schema which is which is kind of rad in different languages um that's that's really cool um the api is re real small um you're probably going to need even smaller subset of it like these two like if you want to upload a file just post to slash upload but what what i really wanted to draw your attention to here is this um this check api and so with this api you can actually check uh the status of any uh the cid of any nft that has been uploaded to nft storage and the output for that kind of looks like this and so you can see here like the uh the like pinning status of this particular cid but also uh the deals with uh file coin storage providers that have been made or are being made currently and so you can see their status um like time date stamps for when things have happened change the on-chain deal ids so you can go and look it up you can see the the address of the storage provider who's who's actually storing it or going to be storing it and what's interesting what's really interesting here is that there's two things there's um there's the batch root cid here and and also the data model selection that they're they're kind of important ones because what happens is that as data comes into nfc that storage it gets uh kind of aggregated together into this big uh big pile of data and that gets that gets sent to to miners and um and then um if you ever want to get your piece of data out of that big uh aggregate then you're gonna need the data model selector and it allows you to pinpoint your particular bit within uh within the aggregate uh so anyway this is this is a cool api that you can you can use to check stuff um yeah we also have the same thing in web3.storage which i'm going to talk about in a sec um js client we have an official javascript client this one isn't generated from open api we've lovingly handcrafted this one for you it's got typescript types it's got nice api docs um we've got a metadata api as well and this metadata api is designed to make minting nfts really easy for you by um guiding you through creating like erc 721 and 1155 compatible metadata um and we will take a look at that um let's take a look at that now all right here's a here's an actual demo so let's see how it goes there my gods are um hopefully gonna help me out here here we go here's the whole whole desktop okay um so what i'm going to do is i'm going to use this metadata api i've already installed the client from nft.storage using npm i'm importing it like this and i've just created a um a version here i've created my token already on the website just go to nfc dot storage and create it i've exported it in my terminal so that should just work um and what i'm going to do is i'm going to try and store this this image called pinpai because it's appropriate about pinning and stuff so uh pimpy let's try and store this using the metadata api so what i need to do is uh call client.store and then we give it an object that looks really similar to something that you'd uh get in when you click through from your token uri which is minted on trent on chain uh to and what is stored on ipfs essentially the metadata for the nft uh where the actual uh image file or video or whatever it is can be stored et cetera et cetera but it looks uh it's really simple it looks something like like this i'm going to call it buy it has a description uh and we call we're gonna call it and buy because it's a comic thing um and then what's kind of cool here is that um we put the the the property is called image and this has to be essentially the url to an image where your nft which is your nft essentially um but what we put in here is actually just a file and what happens behind the scenes is that um the nft storage client will automatically upload that image for you and replace that image with a ur url to the um the the ipfs asset which is stored on ipfs so uh let's just quickly do that create a file i'm going to use fs.

read file think i'm gonna get the pin by jpeg here we go uh you have to give it a file name which could be different from the actual file we're reading from i'm just gonna give it a mind type of image slash uh cool so we this returns a promise so i need to await on it and i get back this object i get this object this metadata object and i'm just going to log out the metadata dot url and this is the this is the url that you'd actually use in your minting process and that would be stored on chain uh we've essentially some log metadata i'm just going to print out the data here just to show that this file gets transformed into a url to this um this particular asset uh i think that's all i need to do so we'll see if this works i'm in the wrong directory okay sorry about that right here we go so let's just take a look here uh so yeah so we've printed out this this logged out the url this is the url that we're gonna put in uh on chain uh and also um here is the data and you can see this image got transformed into a url as well so that's super cool [Music] here we go and what i should just be able to do is head on back over to to here and uh if i put this into my browser uh yes there we go and i should just be able to fetch it from any ipfs http gateway um and this is the data that i uploaded to ipfs so super cool all right um that's that demo successfully done stop that sharing okay right let's get back to the slides because slides besides here here we go um cool all right so what what do we do there well um we actually uploaded data um and then we got back a cid from nfc dot storage um and nft.storage is doing the work of um generating that cid from the data that we sent it and there's a couple of problems with that um firstly we have to trust nft.storage is doing the right thing it's not giving us the cid of some malicious virus in some way uh it wouldn't do that but you know you don't know in this trust us peer-to-peer world we're trying to inhabit um so it's better to have generated that cid ourselves why can't we do that why don't we do that um well um the other thing before we get on to that the other thing that is the um that we have to um wait for that cid to come back to us we have to upload that data and um and wait for that co2d to come back when actually it would be really useful to have that cid before we uploaded because then we could mint our nft on chain uh whilst it's uploading and if it's something big then it might take a while to upload so actually having that cid um generated on on the client side before it gets uploaded is really useful because then i can use it somewhere else i could maybe even store it in a database in my in my app or whatever um so so it means that i can i don't have to trust nfl storage and um i also get to get to solve this ux issue basically um so yeah why can't i just generate that cid before uploading the data well turns out you can uh new is storing cars and cars are not automobiles in this world cars are content addressable archives and it's kind of like tar for dags and dags are directed acyclic graphs and they are the things that the ipfs generates when you import data uh to it so yeah tafford dags if that's a if that's a thing anyway um so it allows you to know that cid before you upload um and in the client um or so on the client side we can do the work to generate that cid um and it means assembling that data into the graph but then we have that cid we have that root file but once we have the graph we pack it into a car and send the car instead don't send the data send the car it's got the tag and then that can just be stored by nft storage you know you know what's being stored is is the the thing that you added because you generated the cid on the client side so there we go um and it also means that we can split it up into multiple cards like you can send you can send partial partial dags like bits of it and that's cool because if you've got like loads of data you can split up into smaller bits and it also allows us to send data that's more more than 100 megabytes in size one of the kind of quirks of this service is that we decided to put it on like cloudflare workers and they are cool because they put uh put you know your worker close to whoever's requesting that api or using that api um but they do have a limit on like how much data you can send them in one in one chunk but uh but it's okay because we can chunk up things uh so we don't have to we don't have to abide by this 100 megabyte rule anymore so you typically end up this is completely contrived uh diagram but you might end up with like something like this a yellow line it uh kind of rings around the nodes in the graph that you'd maybe be sending in one car so you might send this set of nodes in one car you might send this set in another car and you might send this set in the third car and we know that all of these cars are related because they have the same route cid that's pretty cool um and that that allows like failure and retry without sending everything again we can just send any any car in any order uh because it doesn't matter um and um and that allows us to also report on progress after we finish sending a chunk so we can say you know we've done enough and you get that your ui feedback so that's cool all right let's try a real quick storing the card demo i know i feel like this is going way over time i'm sorry uh all right so here we go uh let's stop the share let's share my whole screen okay okay cool um great so this is very similar to what we had before i've had to i've imported this extra ipfs car module this is going to help us pack a car from our content um the path i'm going to use is i'm going to use pinpai again the content is good the content is actually so that's just the the file name essentially uh read file read the file actually i'm not going to use pinpoi at all because what i want to demonstrate is actual splitting and chunking i've got my skate movie here it's 24 megabytes let's just uh let's just make sure that uh yeah here we go here's my skating movie it's 24 megabytes that's great um i'm gonna i'm gonna use that um what happens is that the cars get split into 10 meg chunks so we should see um two uh three chunks uh two ten megs size and one formic uh so escape movie here it is oh typing here we go uh and and so the cool thing is at this point here we've packed this this video into a a car file and we've also got the cid before we've even sent it to our client so we can we've got the cid here and we and when we get the cid return from the client we can verify that the cid we created is a cid that has been returned to us so i'm calling this method could store car pass it the car and um and the cool thing is that we've got this um nice call back which gives us the size of the chunk that was the car chunk that was stored so we can log out we can log out the uh how many bytes we stored for each chunk that gets sent hopefully that's good to go so um [Music] there we are uh npm no node node index let's try again here we go we've got so you can see here we got that cid like straight away um and so now what's happening is uh well there we go we've got one chunk got sent that's about that's like four meg and so these other two here's a 10 meg chunk just come through another 10 meg so we've got three chunks got sent they all got sent in parallel we send up to free in parallel um to not starve browser browsers but um that's that's the kind of cool thing about it and the four megabyte uh chunk even though it was maybe at the end of the data it it won because it's smaller uh so there you go and the cid we've got back is the same as the cid that we um that we uh we calculated that's pretty cool back to the slides all right uh we are getting on for finishing off but at this point i've talked to a whole lot about energetic storage um but web3.storage is the brand new thing it's just just launched almost almost literally uh and um and with three dot storage is built for any data so it's not just nfts you know if you remember this slide from earlier then um web3.story does exactly the same thing uh so uh but it's just not not it's not just for nfts uh it's for any data so you can put uh images videos uh you can put uh documents uh whatever you like even ipld data as well so um you can pick what works for you the best so if you're to be minting nfts then you're probably going to want to go with nft.

storage it gives you the tools as you saw to create and store nft metadata and that that and that metadata will comply to ethereum standards um it also follows best practices for linking to data stored on ipfs um and um and for everything else then web3.storage has you covered so you should uh check that out the other the other difference right now um is that um cars are baked into the web3.storage client that we've we've made this is a nice nice js client we've done for web3.storage as well um we'll catch up the nft.storage client at some point but at the moment you can see for for storing files at least um you just call like client.

put and then it does it there's no mention of cars here like it's just it just does that for you it um it will create the dag it will split the car up it will upload it will do retrying if it should fail for example so um so hopefully uh even easier uh than lft.storage all right so let's really quickly dive into the architecture um this is essentially a fork of nfc dot storage um in the future nfc dot storage might the back end to nft storage might be where free dot store isn't um we'll see how that goes but um they are very similar at the moment very similar um and and so what happens is that users interact with the http api they send car files uh and we put stuff into our ipfs cluster that's cool we have like a six node cluster we have we put stuff on at least three of those nodes in the in the cluster so that's kind of cool and we back up everything to pinata pinata uh provide a pinning service as you probably have um come across um if you've done any kind of ibfse stuff um and uh yeah that's the um the storage broker service this is where all of the interesting stuff happens basically and what what it does is it grabs all of the cids from our database our database is called fauna um no idea why uh and anyway so it batches them up into uh presents so then not really batches there are aggregates of data essentially now presents for miners and they are the presents are actually cars as well everything is cars um and um and what happens is the storage providers they come and eat them up uh not really literally eat them but they will take them because it's the self-serve kind of thing they're like give me give me the deals and i'll store that data cool okay um last last bit uh three when we when we say free we often get questions like why like how how does it how's this even possible well um let me let me tell you a little bit about how it works this is this is essentially um that bit in the bottom right hand corner of our um our diagram here just uh expand it out a bit this is one particular storage provider who has some hard disks uh anyway so what happens is that storage providers commit their hard disk space to the file coin network they they basically say i've got they've got this capacity uh and they get like significant block rewards for doing so and that's this this kind of circular thing because that happens uh essentially every as as the block progresses they get they get rewards and this translates into real world profits profits are down here these money stacks at the bottom left um and uh and yeah so so that's that that happens and that that means that they're really like incentivized to keep uh committing additional hard disk capacity to the network so they keep doing that like more and more um just so they can they can make more profit and of course why wouldn't you do that um however when uh the storage providers are storing data from users on the network their likelihood of winning block rewards goes up by a big factor 10x so wow look at those profits they just ruff raced up um so you can see the potential profits to be made here by storing real user data and this this is such a powerful incentive for file coin storage providers um to to store data such as the data that is coming from a freedom storage um or nft.storage that they're they're willing to offer um free storage in order to get this block reward multiple in some cases storage providers want to store data so badly that they'll actually pay debt users to store their data so anyway so as long as these block rewards continue to be really powerful incentive um which will be true for a long long time and for example like bitcoin block rewards are still an incentive for bitcoin miners today um storage should be free for all verified users on the file coin network right now um and so if it's free for us to store data on far queen and it's it's free for you to store um to store data with us so uh yeah if you want to learn more about the the mechanics behind that and how to be a verified user then if you go to the file coin docs um and then uh click on this there'll be a link there for file coin plus it's docs.filecoin.io um and um and you can you can find out uh how to do it and that is mostly all i have to say but thank you so much for listening and hopefully i haven't gone too far over