Web3 Storage and NFT Storage - Alan Shaw

foreign [Applause] [Music] our next presentation uh comes from Alan he needs no introduction but I'll give him one anyway he's a software engineer working with protocol Labs uh currently working to make the distributed web more accessible to developers Ellen if you wanna get started I will hand it over to you all right um welcome to this presentation I'm Alan um and this is a mashup of two talks about storing data on ipfs and filecoin uh using web free dot storage and nft Dot storage so let's do it and first of all I'm going to talk about nfts um quite a lot the the kind of off-change side of them uh how they can be stored decentralized on ipfs and filecoin for the Long Haul um so um I'm kind of new to nfts uh most people are they haven't been around for that long um but when I found out about them I was uh I was super excited to see that um the the ipfs was already being used uh to to store them um so for example if I go to like any kind of uh website like this one for instance this is a this is a game it uses nfts they have like a um a kind of marketplace uh and they have like nfts that you can buy and trade and stuff uh so let's have a look at I don't know if this one for instance um I happen to know that this um this particular game uses the marketplace of marketplaces um called open c open C there we go um and I can just paste in hopefully the um the uh token ID of that this particular nft hopefully I click on it oh did that come up there we go um is it working hello there we are there it is oh come on internet no open C is not having a good time at the moment all right let's try one more okay right anyway this is a cool car uh but yeah I can click through and I can see that it's metadata is here I should just be able to click on this this talk is like fraught with uh with danger so um we shall see how how it goes it doesn't look like things are going to work out so well today for me um anyway this is going to ipfs.io um uh the these this uh you can see in the bottom left there that it's a CID on uh ipms.io and the metadata for this particular nft here it is it's just come up uh it's stored on ipfs so hooray for that that's great um and uh hang on here we go back to the next there we go um and but why why is data for nfts stored on Ike FS anyway well it's because it's kind of prohibitively expensive to store data on chain on the ethereum blockchain at least um and ipfs actually has some really nice properties for off-chain data that um will go into in a little bit um and so um as I learned a little bit more about nfts I came to realize that um the the uh developers that are enabling nft creators so like the the openc for example like Foundation like Zora they all really cared about users being able to access their data should they ever just like disappear off this off the face of this earth like if they if they fade away like that they still want people to access these nfts that they've bought and uh and created and traded um so they don't want the nfts to just disappear also and so um ipfs uh this this comes into play now and this this thing called ipfs can solve this problem it actually fits the bill really well um and um from my point of view at least it's not really surprising that it's being used but uh so let me just rewind a little bit um and explain um some basics of ipfs in hopefully not too much technical detail um but I've chosen a uh a little um nft here that he's a uh little one-eyed squid thing that's on actually on Sandbox that Sandbox game um and uh anyway so if I put him on ipfs firstly what what do you get you get AC ID it's called a Content identifier and cids are a kind of cryptographic hash and they address the content not this location they address that squid with the one eye not where it is um and um and that means that any given CID will always refer to the same piece of content so if that um if that squid were to grow another eye for instance then the CID for that squid would actually change um and what typically happens is those cids are stored uh on chain and they're they're linked to from chain and if that wasn't the case if if we used like a regular URL to like Google drive or like a Dropbox URL or whatever that could be problematic why could that be problematic well um it the content at the end of that URL could be deleted by the owner the domain could could uh could expire all the hosting like it could be not be paid and then and then just go away the the image could or video or whatever it is could be altered in some way so I could have my squid turned into a cute cat or something which which would be disappointing um or for instance the URL just might not be accessible from within your country or maybe you're in like some sort of corporate firewall or something where you actually need to get hold of that data from um a peer that's maybe closer to you than the internet location where it where it actually is so ipfs how does it work anyway well um so if someone who's using ipfs um requests that data by CID then um the person who has it will send it back to you and they'll receive it you'll receive it so um and now two nodes have that same content and they can serve it to anyone else who comes along so uh like this if someone else comes on goes do you have this CID the content for the CID and uh and uh either one of those nodes can help to transfer the bytes to that uh to that node we call it uh bit swapping uh because why not uh and and so that that happens and now three of them have it and the cool thing about this is that the original node who had the content can actually go away like Gone Gone um and the data will still be retrievable from other nodes uh in the network and this is the awesome power of peer-to-peer hooray um caveats there's two caveats the first one um the first one is that if all of the nodes uh that have the content um go away then you're not going to be able to retrieve it it's kind of kind of obvious worth stating if there's no one on the internet who has the content you're not gonna be able to get the content that's caveat number one caveat number two is uh slightly less obvious it is garbage collection so depending on your configured kind of GC uh configuration um it might run at a particular time uh it might run when your configured repost storage limit is hit so you might be fetching things from ipfs it all goes into your repo and eventually you'll use enough data that ipfs will go okay we're kind of full now uh let's have a let's do a GCE Ram uh and uh and the third other um time that it might be run is it could be run manually so you can actually invoke GC from the command line um or you can uh if you've got the HTTP API exposed you can call it from like that so what you need to do is tell like if this content is not garbage um because um when you obtain content from the network that you didn't actually add yourself that content is eligible for garbage collection so if I got this picture of the squid from someone else then um it could be garbage collected in the future so what you need to do is kind of tell your ipfs that this uh this uh piece of content is a rare nft and it's not trash don't throw it in the bin please um and in ipfs that's referred to as pinning and if you don't pin your data then it will eventually be garbage collected so even if the content is super popular you can still eventually um exit the network and so these two caveats are why uh developers generally use a third-party service like pinata or textile Fleek to store their content um because they have nodes that are publicly available and they stay online like 24 7 and obviously they pin the content so it doesn't disappear cool so nft.storage uh Molly just mentioned this earlier so you kind of already know a little bit about it it's something that we've been working on for a while now it's a place where you can store your nft data decentralized on ipfs and filecoin safe in the knowledge that'll be there tomorrow um and it hopefully goes some way to addressing that permanence problem um and uh it's free it's free as long as possibly as long as we can possibly make it free um and the one of the founding kind of principles rules is that it should be easy to store data using this service so how does it work well um you upload your nft data to nft.storage um and nft storage stores that data on ipfs nodes in the network that's right we actually have a cluster and we store it in on uh free um free nodes at least um so okay so far so good but how's that different from any other like pinning service well um behind the scenes what nft.

storage is doing is it's negotiating deals to store your data with storage providers on the filecoin network and so the filecoin network you might already know incentivizes the storage providers to store data quite heavily they get paid to store data they also get typically get paid when um data is retrieved and they also get Block rewards when uh for for providing capacity to the network uh so why why is that good well if anything that stories were just to happen to just go away like it's gone then um and and also every Miner on the net I'm sorry every um ipfs node on the network went away as well then um then the file coin storage providers who have the content will still have the we'll still have it and they will store it for the duration of of the deal that's been made for them and so you can think of these storage providers as just another set of ipfs peers with really strong incentives to constant continue storing that data uh and this is this is hopefully a really good level of redundancy basically and should uh like I said go somewhere to solving that permanence problem so um who it is again NFL search free decentralized storage for nfts on ipfs and Falcon um and uh the URL is nft.storage I've said it about a thousand times you should go take a look um we have a simple API it's expressed as an open API schema if you don't know what that means it means we can generate docs like this we can also generate like actually generate clients from this schema uh which is which is kind of rad in different languages um that's that's really cool um the API is real small um you're probably going to need even smaller subset of it like these two like if you want to upload a file just post to slash upload but what what I really wanted to draw your attention to here is this um this check API and so with this API you can actually check uh the status of any uh the CID of any nft that has been uploaded to nft storage and the output for that kind of looks like this and so you can see here like the uh the like pinning status of this particular CID but also the deals with uh file coin storage providers that have been made or are being made currently and so you can see their status um like time date stamps for when things have happened change the on-chain deal IDs so you can go and look it up you can see the the address of the storage provider who's who's actually storing it or going to be storing it and what's interesting what's really interesting here is that there's two things there's um there's the batch root CID here and and also the data model selection that they're the kind of important ones because what happens is that as data comes into nft that's the original gets uh kind of aggregated together into this big uh big pile of data and that gets that gets sent to Miners and and then um if you ever want to get your piece of data out of that big uh aggregate then you're gonna need the data model selector and that allows you to pinpoint your particular bit within uh within the aggregate uh so anyway this is this is a cool API that you can you can use to check stuff um yeah we also have the same thing in web free.storage which I'm going to talk about in a sec um JS client we have an official JavaScript client this one isn't generated from open API with lovingly handcrafted this one for you uh it's got typescript types it's got a nice API docs um we've got a metadata API as well and this metadata API is designed uh to make minting nfts really easy for you by um guiding you through creating like ERC 721 and 1155 compatible metadata um uh and uh we will take a look at that um let's take a look at that now all right here's a here's an actual demo so let's see how it goes there my gods are um hopefully gonna help me out here here we go here's the whole whole desktop okay um so what I'm going to do is I'm going to use this metadata API I've already installed the client from nft.storage using npm I'm importing it like this and I've just created a um a version here I've created my token already on the website just go to nfc.storage and create it I've exported it in my uh terminals that actually just work um and what I'm going to do is I'm going to try and store this this image called pinpai uh because it's appropriate about pinning and stuff so uh pen pie let's try and install this using the metadata API so what I need to do is uh call client.

store and then give it an object that looks really similar to something that you'd uh get in when you click through from uh your um token URI which is minted on track on chain uh what is stored on ipfs essentially the metadata for the nft where the actual uh image file or video or whatever it is can be stored et cetera Etc but it looks uh it's really simple it looks something like like this I'm going to call it pin buy it has a description uh and record we're gonna call it and buy from it because it's a comic thing um and then what's kind of cool here is that um we put the the the property is called image and this has to be essentially the URL to an image where your nft which is your nft essentially um but what we put in here is actually just a file and what happens behind the scenes is that um the nft. storage client will automatically upload that image for you and replace that image with a UR URL to the um the the ipfs asset which is stored on ipfs so uh let's just quickly do that create a file I'm going to use fs.reath files and I'm going to get the pin by jpeg um here we go uh you have to give it a file name which could be different from the actual file we're reading from I'm just going to give it a mine type uh image slash uh cool uh so we this returns a promise so I need to await on it and I get back this object this metadata object and I'm just going to log out the metadata set Dot URL and this is the this is the URL that you'd actually use in your minting process and that would be stored on chain essentially uh that's a log metadata I'm just going to print out the data here just to show that this file gets transformed into a URL to this um this particular asset uh I think that's all I need to do so we'll see if this works store Mid s score I'm in the wrong directory okay sorry about that right here we go so let's just take a look here uh so yeah so we printed out this this logged out the URL this is the URL that we're gonna put in on chain and also um here is the data and you can see this image got transformed into a URL as well uh so that's super cool um here we go and what I should just be able to do is head on back over to uh to here and uh if I put this into my browser uh yes there we go and I should just be able to fetch it from any ipfs HTTP Gateway um and this is the data that I uploaded to ipfs so super cool all right um that's that demo successfully done uh stop that sharing okay right let's get back to the slides because slides uh sides are here here we go um cool all right so what what do we do there well um we actually uploaded data um and then we got back a CID from nft.storage um and nft.storage is doing the work of um generating that CID from the data that we sent it and there's a couple of problems with that um firstly we have to trust nft.

story just doing the right thing it's not giving us the CID of some malicious virus in some way uh it wouldn't do that but you know you don't know in this trust is peer-to-peer World we're trying to inhabit um so it's better to have generated that CID ourselves why can't we do that why don't we do that um well um the other thing before we get on to that the other thing that is that um that that we have to um wait for that CID to come back to us we have to upload that data and um uh and wait for that co2d to come back when actually it would be really useful to have that CID before we upload it because then we could mint our nft on chain uh whilst it's uploading and if it's something big then it might take a while to upload so actually having that CID um generated on on the client side before it gets uploaded is really useful because then I can use it somewhere else I could maybe even store it in a database in my in my app or whatever um so so it means that I can I don't have to trust in a few lots of storage and um I also get to get to solve this ux issue basically um so yeah why can't I just generate that CID before uploading the data Well turns out you can uh new is storing Cars and Cars are not automobiles in this world cars are content addressable archives and it's kind of like tar for dags and dags a directed acyclic graphs and they are the things that the ipfs generates when you import data uh to it so yeah tar for tags if that's a if that's the thing anyway um so it allows you to know that's the ID before you upload um and in the clients um yeah well it's so on the client side we can do the work to generate that CID um and it means assembling that data into the graph but then we have that CID we have that root file but once we have the graph we pack it into a car and send the car instead don't send the data send the car it's got the tag and then that can just be stored by nft.storage you know you know what's being stored is is the the thing that you added because you generated the CID on the client side so there we go um and it also means that we can split it up into multiple cards like you can send you can send partial partial dags like bits of it and that's cool because if you've got like a loads of data you can split up into smaller bits and it also allows us to send data that's more more than 100 megabytes in size one of the um kind of quirks of this service is that we decided to put it on black cloudflare workers and they are cool because they put uh put you know your worker close to whoever's requesting that API or using that API um but they do have a limit on like how much data you can send them in line in one chunk but uh but it's okay because we can chunk up things so we don't have to we don't have to abide by this 100 megabyte rule in anymore so you'd typically end up this is completely contrived a diagram but you might end up with like something like this a yellow line it uh kind of rings around the nodes in the graph that you'd maybe be sending in one car so you might send this set of nodes in one car you might send this set in another car and you might send this set in the third car and we know that all of these cars are related because they have the same root CID that's pretty cool um and that that allows like failure and retry without sending everything again we can just send any any car in any order uh because it doesn't matter um and um and that allows us to also report on progress after we finish sending a chunk so we can say you know we've done enough and you get that your UI feedback so that's cool all right let's try a real quick storing the card demo I know I feel like this is going way over time I'm sorry uh all right so here we go okay [Music] cool um great so this is very similar to what we had before I've had to I've imported this extra ipfs car module this is going to help us pack a car from our content um the path I'm going to use is I'm going to use pinpai again the content is good the content is actually so that's just the uh the file name essentially uh read file read the file then but I don't actually I'm not going to use pinpai at all because what I want to demonstrate is actual uh splitting and chunking I've got my skate movie here it's 24 megabytes that's just uh let's just make sure that uh here we go here's my skating movie is 24 megabytes that's great um I'm gonna I'm gonna use that um what happens is that the cars get split into 10 Meg chunks so we should see um two uh free chunks uh two 10 Meg size and one format so let's get it movie here it is oh typing here we go uh and and so the cool thing is at this point here we've packed all this uh this video into a a car file uh and we've also got the CID before we've even sent it um to our client so we can we've got the CID here and we and when we get the CID return from the client we can verify that the CID we created is a CID that has been returned to us so I'm calling this method called store car pass it the car and um and the cool thing is that we've got this um nice callback which gives us like the size of the chunk that was the car chunk that was stored so we can log out uh uh we can log out the uh how many bytes we've stored uh for each chunk that gets sent hopefully that's good to go okay so um stalker there we are uh and here no node node index let's try again here we go so we got so you can see here we've got that CID like straight away um and so now what's happening is uh we'll be there we go we've got one chunk got sent that's about that's like four Meg and so these other two here's a 10 Meg chunk just come through another 10 Meg so we've got three chunks got sent they all got sent in parallel we send up to three in parallel um to not starve browser browsers but um that's that's the kind of cool thing about it and the four megabyte uh chunk even though it was maybe at the end of the data it it won because it's smaller uh so there you go uh and the CID we've got back is the same as the CID that we um that we uh we encapsulated that's pretty cool back to the slides all right uh we are getting on for finishing up but at this point I've talked to a whole lot about inner student storage um but web free dot storage is the brand new thing it's just just launched almost almost literally uh and um and with free dot storage is built for any data so it's not just nfts you know if you remember this slide from earlier then um web free dot storage that's exactly the same thing uh so uh but it's just not not uh it's not just for nfts uh it's for any data so you can put uh put images videos uh you can put uh documents uh whatever you like even ipld data as well so um you can pick what works for you the best so if you're going to be minting nfts then you're probably going to want to go with nft.storage it gives you the tools as you saw to create and store in ftms data um and and that that and that metadata will comply to ethereum standards um it also follows best practices for linking to data stored on ipfs um and um and for everything else then web free.storage has you covered so you should uh check that out the other the other difference right now um is that um cars are baked into the web free dot storage client that we've we've made this is a nice nice JS client we've done for web free.storage as well um we'll catch up the nft.

storage client at some point but at the moment you can see for for storing files at least um you just call like client dot put and then it does uh there's no mention of cars here like it's just it just does that for you it um it will create the dag it will split the car up it will um upload it will do um uh retrying if it should fail for for example so um so hopefully uh even easier uh and then out of T dot storage all right so let's really quickly dive into the architecture um this is essentially a fork of nft. storage um in the future NFC storage might the back end to nft.org might be where three dot storage um we'll see how that goes but um they are very similar at the moment very similar um and and so what happens is that users interact with the HTTP API they send car files and we put stuff into our ipfs cluster that's cool and we have like a six node cluster we have we put stuff on at least three of those nodes in the in the cluster so that's kind of cool and we back up everything to pinata pinata uh provide a pinning Service as you probably have um come across um if you've done any kind of ibfs stuff um and uh yeah that's there um the storage broker service this is where all of the interesting stuff happens basically and what what it does is it grabs all of the cids from our database a database is called fauna um no idea why uh and anyway so it batches them up into uh presence so then not really batches there are Aggregates of data essentially now presence for Miners and they are the presents are actually cars as well everything is cars um and um and what happens is that storage providers they come and eat them up uh not really literally eat them but they will take them because it's a self-serve kind of thing they're like give me give me the deals and how to store that data cool okay um last last bit uh free when we when we say free we often get questions like why like how how does it how's this even possible well um Let me let me tell you a little bit about how it works this is this is essentially um that that bit in the bottom right hand corner of uh um our diagram here just expand it out a bit this is one particular storage provider who has some hard disks uh anyway so what happens is that storage providers commit their hard disk space to the file coin Network they they basically say I've got they've got this capacity uh and they get like significant block rewards for doing so and that's this this kind of circular thing because that happens uh essentially every as as the block progresses they get they get rewards and this translates into real world profits profits are down here these money stacks at the bottom left um and uh and yeah so so that's that that happens and that that means that they're really like incentivized to keep uh committing additional hard disk capacity to network so they keep doing that like more and more um just so they can they can make more profit and of course why wouldn't you do that um however when uh the storage providers are storing data from users on the network their likelihood of winning block rewards goes up by a big factor like 10x so um wow look at those Prof office they just roof raced up so you can see the potential profits to be made here by storing real user data and this this is such a powerful incentive for file coin storage providers um to to store data such as the data that is coming from a freedom storage um or nft. storage that they're they're willing to offer um free storage in order to get this block reward multiple and in some cases storage providers um want to store data so badly that they've actually paid users to store their data so anyway so as long as these block rewards continue to be really powerful incentive um which will be true for a long long time and for example like Bitcoin block rewards are still an incentive for Bitcoin miners today um storage should be free for all verified users on the Falcon Network right now um and so if it's free for us to store data on farquin and it's that's free for you to store um to store data with us so uh yeah if you want to learn more about the the mechanics behind that and how to be a verified user then if you go to the filecoin docs um and then click on this there'll be a link there for filecoin plus it's docs.com.

io um and um and you can you can find out uh how to do it and that is uh mostly all I have to say uh but thank you so much for listening and um hopefully I haven't gone too far over