Simple off chain Data Storage Using Web3 Storage & NFT Storage. Store all Things on IPFS & Filecoin

So, our next speaker is Youssef, and he will talk about blockchain data storage using web3.storage and nft.storage. So without further ado, let's see how we can create nft.storage.

Welcome. It's great to be here. Thank you. I will share my screenshot here. Steeply.

So, I'm Youssef, and I work at Protocol Labs, where we build IPFS and Filecoin. These are two really interesting technologies that we think can help everyone in this decentralized application ecosystem take advantage of this huge network of storage that is now available. So let me move on to the next one. This is kind of a combination of the two conversations about Web 3.0 storage and NFT storage.

These are both services that we built to make using IPFS and Filecoin really simple and accessible, especially for Web 2.0 especially for Web 2.0 developers or anyone in general anyone in general who wants to find a way to this low- friction storage network. So, first I want to talk a little bit about NFTs. So, we got really interested in NFTs.

Um, you know, we've been doing this for a while now, but apparently this year they're kind of gaining traction in the public consciousness. That's when I personally started to take an interest in them. Um, I was somewhat aware of them before, but mostly it was really encouraging for us to see IPFS being used to store a large number of NFTs. Um, and we think it's a great fit for this type of application because application because , first of all, storing data on most blockchains is extremely expensive. And so to have an NFT that represents something outside of itself, which is the case for many, there needs to be a connection between the NFT and data that is stored outside of the blockchain.

And IPFS has some really great features for creating that kind of connection. So we'll look into this a little bit. Um, but we started to realize that developers like OpenSea, Foundation, Zora, etc. really care about ensuring that users have access to data if it ever disappears. And that's kind of a common trait in such circles: no one wants to trust any single party to stick around forever.

But if we come together, if we all enjoy playing together, we can hopefully develop a system that will outlast any of us individually, right? And it's individually, right? And it's , um, yeah, it's good for us to build systems with that in mind, even if, hopefully, we're going to be around for a while longer . But this is something that users are concerned about, especially when they realize that they... realize that they...

I don't know, maybe people already know about this, but I think people who are new to NFTs don't always understand what's on the blockchain and what's off, and what the connection is between them. So this is kind of an educational piece where we're trying to tell people what this connection is and what's a reliable way to make it. So we think IPFS fits into this mix fits into this mix . Um, and if you're new to IPFS, it's an interplanetary file system. It's been around for a few years now, and the gist of it is that you first host your data on an IPFS node, and that node participates in this large peer-to-peer content sharing network.

Where, essentially, if someone needs content, they ask for it, and if I have it , I provide it. And everyone does it, and it's actually a pretty efficient way to distribute data across the network in a variety of ways. But, first, so to speak, you first place your data in the IPFS node, and you get back this string, the content identifier. And this is a cryptographic hash. The default is SHA-2 256.

But there you can use many different hash many different hash functions and everything else. This is a highly customizable software. But the fundamental point is point is that this CID will always identify one specific piece of content. And any changes to the content would result in a different CID. So, with CID, you can say, "Hey, network, please give me this data."

And when they come back from anyone, you can always verify their accuracy because you can compare them to the CID you used to retrieve them. So that means you don't have to worry about who's providing the data about who's providing the data , where it's coming from, or, you know, anything you need to know about it. As long as the protocol works, as long as someone has the data and sees your request, they can provide it and you will receive it. And that's a pretty cool feature for linking to things that we'll talk about in a second. But here's the gist of the request process.

You request data by CID, the original node responds, and now they both have a copy. And if a third node shows up with a request, they can both respond in parallel and say, "Oh, I have it. Here are a few fragments." This guy on the left gives some fragments, the guy on the right gives some fragments, and now you get the file potentially faster than you could download it completely from one. But there are a lot of variables here regarding latency and so on.

Therefore, it is very difficult to make absolute statements about performance characteristics compared to other systems, but overall there is potential for efficiency. Hmm, yes, now this third node has it, and as a result they can also participate in the provisioning process. Oh, and that's great. The starting node disappears, but the content remains. And so we got closer to this dream of having content outlive us and our, you know, organizational guidelines and so on.

Um guidelines and so on. Um , the data can still be retrieved from other nodes, and, you know, that's kind of the power of a peer-to-peer network. They believe that no single knot is blessed or special. Um, but there are some caveats to this approach. With IPFS, if all the nodes that had content disappeared from the network, there would be no one there to provide it.

It's like an active process where your node has to be online to respond to requests, and if no node is online with that content, it simply cannot be decrypted. Although, I should point out that the same goes for, you know, HTTP servers, like, for example, facebook.com suddenly became unintelligible, because then no one would be able to access it. But with IPFS you have an interesting property where if someone later adds the same content to the network, provided they use the same hashing algorithm and everything else hashing algorithm and everything else , then they will get the same CID, and the same CID, and any references to that original content will be decryptable. So, I think a third party can fix a broken link in a way they can't with HTTP.

So, that's pretty cool. But the second caveat is that there is an automatic garbage collection process for these temporary copies that are created as content is distributed over the network. Each node will keep a small copy for a minute, but eventually they may experience disk congestion disk congestion and have to clean up and discard the data. So to prevent this, there is a thing called pinning, which is signaling to your IPFS node, "Please support this." Hmm, don't collect garbage from your data.

And, um, that's one of the main ways for people to practically get their data into IPFS and make it available is to use a pinning service that will provide that data on your behalf data on your behalf . So you don't need to manage your own infrastructure, etc. infrastructure, etc. And you know, when we were looking at the NFT space and thinking about what we could do, we saw that we could simplify that process a little bit for people by actually becoming a staking service staking service , but using some other technologies, including Filecoin, which we'll talk about in a second. So, we represent this as nft.

storage. This is a website. It is free to use. You log in, get an API token, and then you can upload your materials through us, where we host them on IPFS where we host them on IPFS . And also, for long-term storage, it's all sent to Filecoin.

So, it works like this: yes, you take your NFT, it's a very simple download interface. It's just an HTTP request. There are several different variations, depending on your needs and the endpoint you want to use. But then, anyway, we get your data via a push request, and then the NFT repository will store it on an IPFS cluster that we manage. And then, uh, we also back them up with a third-party provider, just for safekeeping.

And behind the scenes, however, we are also in talks with Filecoin storage providers for long-term data storage. And they actually get paid by the network in the form of block rewards, which, uh, is a pretty interesting, uh, economic perspective that we can talk about in a second. But for now, the advantage of Filecoin is that it fits this property, allowing network participants to disappear while still giving you some solid guarantees about your data. So, if NFT.storage were to completely disappear, and all IPFS nodes disappear, and all IPFS nodes that we or others run would lose all content, the data would still remain on Filecoin, and the protocol would constantly verify its existence.

So this is, um, basically, if you had a worst-case scenario where we just completely closed shop and couldn't pay the bills anymore, um, everyone has this kind of publicly verified emergency hatch, a seed vault, so to speak, maybe another way to think of it, where the data is stored and it's right there. And we provide all the information to retrieve them in our API, etc. So, um, yeah, to use it, you go to nft.storage, which is now a top- level domain, so that's cool. And, um, it's really easy and free.

Hmm, and you should delete it. This is very...This is an HTTP API. It is not very big. Hmm, uh, the main endpoint is the upload endpoint for storing things, but there's also, um, yeah, things for getting information about the status of your upload.

There are, you know, people who store on Filecoin and IPFS and so on. So, you can get that detailed information if you need it. Um, and if you're working with JavaScript, we have a JavaScript client that simplifies things a bit and helps format data according to ERC 1155 standards. So ERC 1155 standards. So , if you're creating an ERC 1155 contract or one that's compatible with it, use the client.

store method and it will do some useful things for you. You can give it a file object and it will add the file to IPFS and create a hash create a hash link for you in your JSON metadata, and then wrap it all in this thing called a DAG-CBOR graph. It doesn't really matter what DAG-CBOR is, but it's essentially just a compact binary JSON object. But it's a nice way of addressing hash- addressing hash- link data that we like to play around with. Um, this is part of the IP LD project, if you're wondering.

Um, so I'll quickly show you how to use NFT storage, and then we'll talk a little bit about our other storage service. So, here's the site itself, you log in via GitHub or email. You will be sent a magic link that you need to click on. If I go into my profile, you'll see a few things I've saved. Mostly it's just testing and such .

And we can go in here, do a little digging, and get an API token that we can use to authenticate. Um, I should also mention that my home internet is very slow. So things that are slow for me may not be slow for you when you use them. Especially when I'm loading things. This is a bit sad.

Um, but anyway, here we are with API keys we are with API keys , and I created this one a few minutes ago. You can create and delete them here. And you know, here's my key for later. I'll probably be back here in a minute. Uh, I have an empty project and just an empty folder with some images that maybe we could save to NFT save to NFT storage, and let's create an NPM project to write code.

So if I do an NPM initialization and I just accept all the default values . Now I have a package.json file and I can say NPM install, uh, nft.storage. And that gives me a JavaScript client library.

Uh, I'm also going to hook up a few more libraries once this is done to make it easier to create a simple little example program. So, minimist is for extracting command line arguments, which is quite handy. And then, uh, MIME types to determine the type of file you received, because the store function wants you to give it a MIME type. And this method, which formats according to ERC-1155, expects it to be an image, uh, file type when you give it an image field. Well, you can actually give it an MP4 for compatibility with platforms that use video in the image field.

So, we've installed some dependencies, and I'm going to go to the editor and create myself a file. I'll call it index.mjs. Something relatively new in Node-land: if you name your file with the .mjs extension or if you add the type module to your package.

json, you can use the import syntax. So, import NFT.Storage from NFT.Storage. I'll also include the file object because the API uses the file object that you see in the context of a web the context of a web browser, and then it provides an implementation for Node if you're working on Node.

js. So, I can import this here. And then to read a file from my local disk, I import the fs module from node. So, import fs from fs/promises. I can use async await.

And we will involve the minimist that we installed earlier. And let's make my channels. So, that's all we need for a second. And let's create a function for storing NFTs. So, let's just call this asynchronous function store NFT.

It will accept the image file name image file name . And then maybe a title and descriptive metadata that we'll, uh, label our NFT with. description description So, to do this, we will create a client for the NFT repository. And for that, we'll need a token, which I'll be a naughty boy and embed in my source code. Uh, yes.

And obviously, this is actually, uh, an area that will be improved very soon. Now, uh, the best practice for securing your API for securing your API token for NFT.Storage is probably to protect its interaction with NFT.Storage from the backend NFT.Storage from the backend service, since bundling it into a purely decentralized frontend application would expose it to the client.

But we're working on solutions for that, which will include direct authentication through the wallet and using decentralized identity, and also, uh, some kind of user scope tokens so you can manage authorization on behalf of your user. I mention this because, um, it's just a common problem that we're working on. So stay tuned for updates. In the meantime, I'll take the token from here and insert it here. So, I figured it out.

And now I can create a client. I'll say: const, um, client—this is a new NFTStorage, and you give it a token argument. And now I can do something with it. But first I'll need to read my file. So I can say that the file data is equal to fs.

readFile. Sorry, I need to wait. This is asynchronous. And, my God. the image file name , and the type will be mime.

the file name of the search image. Because it simply extracts the file type from the file extension. It's not very difficult. And then we say, " then we say, " Okay, I have the data and the file type." So now I can create the file.

Uh, this is a new file object. It accepts an array of blob parts, which are essentially just things that can be converted to binary data. So, this stuff that we've extracted here from the image file name will work just fine. Actually, I...Let's see .

No. Let me quickly zoom in on this so everyone can. This is a little easier to read. That's it. So, I create a new file and pass the file data to it.

Filename... file data to it. Filename... Maybe we'll shorten it to the last part of the file path. So, import basename.

And then you can say: This is basename image file name. Okay, and then the last part of the file constructor file constructor is the additional arguments where you can pass the MIME type. So, we have tested it before and can pass it on . And now we have a file object, which I can call image, right? Because that's what the store method expects.

Yes. Image. Now this is a proper file that we can pass on to the client. So now I'll say, uh, I'll just return client {dot} store image name, description. Uh, and there are a few others.

You can add additional properties, you know, if you want, you can have properties with properties with any key values ​​for your own use case, which if you're working on ERC if you're working on ERC -1155, we recommend, for example, adding your own properties to the properties if you can, just because it simplifies the process. We're trying to beat the standardization drum a little bit. Um, because being on the receiving end, we also, uh, you know, parse Ethereum chains and find all sorts of NFTs in the wild and see what people are doing. There are many interesting variations of metadata. But anyway, now we have it, and we can say that we can cause it.

Let's create a main function. main. And here we can take our, uh , structure arguments. Yes. We will need this.

complete. Come here, dirty Come here, dirty . Good. So, uh, the arguments are equal to the minimalist function. And for this you pass arguments to process or process.

argv. Extract the first two from the node considerations. And now the minimizer—it's pretty simple. It takes any arguments, parses them, and gives you a dictionary or JSON object right away. So, I'm just stating that you gave me the correct data.

So, there is nothing. We want there to be an image, a name, and a description. So if there 's no image, complain. This can be a bit tricky for live coding. But still, I mean, error handling is important.

We'll let the description be optional, I think. I don't know what to say. Let's extract them from this object and I'll add them to the path in the store NFT function. So image here is the name of the image file name of the image file , and we can just throw that in here and say that the metadata is equal to await and store NFT, and then give it that image file name , name, and description. Now let's enter the console and see what happens.

Therefore. Good. So if I call the main function, we should be able to call this thing as a node script and see what we see. So, let's see. Let's begin.

Uh, if I use node index.js and then say dash dash image, I can give it, uh, images and call it Better. And yes, I probably should have added a little console log saying, "Hey, we're doing something right now." But, here it is. It's already done.

So what happens with the result you get is that you have an IPFS URL that has the IPFS colon/slash format. If you want, you can convert this to an HTTP gateway URL quite easily. So, uh, this part here is the CID, and then there's the file path. So if I want to make this a gateway URL, I can say CID Sorry. Uh, I accidentally copied something here.

CID.ipfs.dweb.link. There are many other gateways There are many other gateways , but this is the one that sticks out to me.

And then, um, add the file path at the end. So, I'll quickly show you what it looks like. Uh, I'm doing cid.ipfs.w-web.

link /metadata.json. Yes, I spelled that wrong. Metadata. Here.

So, so , uh, yeah, there's a little package that creates a little graph structure for you. And then, um, you can walk through this graph using this path syntax or something. But, basically, it's just Yes. Yes, let's begin. So there's metadata that we've saved, and you can see here if, um, it's big enough that it's an image field that has the same ipfs://w-uh to the image itself.

So, you create another link to the gateway and extract it gateway and extract it . Uh, I, um, did it. If you have Brave installed, it will automatically follow these links, but for some reason I didn't think to use it to use it , um, for this demonstration. Oh, well. link and So, this is the image itself that will be banished.

Um, and I... So if you want to get a machine- get a machine- readable, uh, or a traversable form of this, uh, without having to parse links and so on, this and so on, this wrapper object, like the root object of this graph, where if you just have the CID, uh, without the metadata.json, it's actually... with that DAG CBOR thing that I talked about earlier, where you can get that object and then use, uh, our IP LD tools to go into the metadata and find the image itself, and put it all together into this graph. So not sure why this is taking so long to resolve, but I blame my terrible internet.

So we'll see. Uh, let's maybe take a break from all this and, uh, talk about Web3.storage. We'll be back and hopefully it will be loaded by then . Good.

So . Good. So , listen, there's something I want to mention before we get to Web3.storage, which is, we have this thing called. There is a format called CAR, which is a content archive, and you can send us content archives.

And if you want to pre- format your IPFS data, for example if you have a specific layout, or if you want to send large files over 100 megabytes, I should also mention that we will soon be doing this automatically for you. So we don't have to do it ourselves, but it's nice to understand what's going on "behind the scenes" so you have a mental model of what's working. So the advantage of using a car is that you can know the CID before loading. So you don't have to wait for our server to tell you what the ID is, and you can check if it's correct, that we store exactly what you give us. And as I said, it's fragmented, and it has this, and you can divide the cars into fragments here.

So, I have one big graph, and I can split it into small graphs, all with the same root, and our server will put them all together for you. And yes, that's pretty cool. So I'll talk a little more about cars in a second because we have another service and I'm just short on time. So I should go into the details of the Web3 repository into the details of the Web3 repository . It works very similar to an NFT repository, especially “under the hood.

” They are very similar. And this... So if you remember this slide from the past, this is the same thing, but for data that is not NFT. So, it's really any data you want to store on IPFS and Filecoin, you can just upload it through Web3.storage and we'll store it for you.

Um, and it has built-in CARs. So this is coming soon to NFT storage, but right now Web3.storage already has this in a JavaScript client where you basically give it a file of any size and it will create a CAR file for you create a CAR file for you , do the fragmentation, and send everything. And this gives you access to the root CID once we calculate it on the client. So you can display in your UI and say, “Oh, here’s the thing.

Now send this to Web3.storage in parallel and it will be stored for you. And you'll get progress callbacks and stuff like that. Hmm, very briefly, this is what the architecture looks like. Hmm, I'm not going to talk much about this, but there are some cloud workers.

We maintain some state in the database. But then the interesting part then the interesting part is the storage broker service . It will take your data and bundle it into chunks, usually I think we usually make chunks of 32GB, um, and store each of them on the Filecoin provider. Because it is the most efficient format for getting data directly into the Filecoin pipeline. Hmm, and in parallel with that, we've already added it to our IPFS cluster and to our redundant Pinata cluster.

So it is available for download almost immediately after downloading. Hmm, but then maybe I think we already have about an 18 hour delay to get it on Filecoin. Okay, sorry. I'm not sure if it's eight, but I've actually been asking people what the real number is, sorry. But it's been a few hours now because we have, um, this aggregation process going on .

Um, and as a result of that, by meeting the miners where they are, sorry, storage providers, as they called it, we, um, can offer the service for free. So here's something that often comes up when we talk about a service, like, how is it free? Um, and that's mainly because , um, the main economic driver for Filecoin providers right now is the block reward. So, they initially receive a reward for allocating network capacity. So if you connect disks to the network and prove that you did, um, then you get a potential block reward relative to your total storage capacity, the capacity that you add.

Um, but, um, if you store real data, boom, you get a 10x multiplier, essentially. The way it works is that there is a whole system called Filecoin Plus in which a notary distributes this resource, which is called a data limit. And the data cap is essentially a kind of voucher that says we believe this is real data, not just some random garbage that someone created to inflate their numbers, I guess. So the idea is that by passing your data through Web3 storage and NFT.Storage, we have sufficient confidence sufficient confidence that you are a real person or that there is a real social need behind this data.

I mean, that's what we're achieving, because essentially our mission is data preservation and all that. So data preservation and all that. So , verified data increases the probability of a block reward by 10 times. And as a result, this is the actual cost of storing the data, and there is some collateral that the providers provide upfront to guarantee that they will store it. All of this is essentially insignificant compared to the increased reward.

So it's in their best interest to provide long-term storage for free, so we don't charge users. Anything for data storage. Um data storage. Um , we have minimal operational costs to keep IPFS running, you know, but in the grand scheme of things, it's not a big deal, and we're very, you know, committed to providing that to make people's lives easier. So, this is what I got.

Um, thank you very much, and if anyone has any questions, please let me know. It was wonderful. Um, also a great recovery. I feel like we're going to go overboard with time, but it worked out well. Yes, yes, absolutely.

I got down to business. Um, I think there are two common questions, one of which I might have already addressed, is the interaction between web3 and NFT the interaction between web3 and NFT , what does storage do, i.e. formatting and car archive formats . Um, a common question that common question that I think we get, especially during the hackathon, is what are the best practices you recommend when people want to think about storing metadata or even updating it?

For example, how should we deal with this? And how can an NFT repository be useful for such things? Um, and I want to ask a question about how you actually...What's next after that? Like, is this done?

Like, is this a great service that you offer? He is working. Like how are you actually going to improve this thing, or what are the future features? Um, actually, I mean there are some reliability things that we want to implement, and right now, for example, we're doing a database migration that should make everything faster and, you know, more reliable. But in terms of features, I think those two issues are somewhat related, especially with Michael and Gazelle, I don't know if you've met them yet , but they're trying to find some kind of roadmap for scalable NFTs, or, no, not exactly mutating, but tracking, you know, NFT claims over time.

And so what we see happening now is using events to send, essentially, you know, entity attribute triples, where, you could say, the entity is an identifier for the NFT itself, and then the attribute is some kind of statement about it, like, you know, here's an alternate representation, or like this , you know, it's been remixed into something. What do you imagine? Many scenarios. But then eventually you stack them up, essentially compress a big deck, and then you have, I mean, an idea of ​​the evolution of the NFT over time. But, and this is all speculative for now; we're still trying to figure out how practically we can do this, and how, and also, you know, is this something that requires a lot of support from people at the beginning, or is there a way to apply this to NFTs that already exist?

And I think there is. Hmm, but there are a lot of details. So, I think that's where our focus is right now with NFTs for NFT focus is right now with NFTs for NFT storage. Trying to figure out, I'm not sure if it's actually going to come directly through the NFT repository, but I mean we're going to support it there, but I think it's a broader effort to try to figure out what the data model should look like. Hmm, in terms of the actual usage experience, we're now...

So, we have two separate services now, and we're going to make the NFT going to make the NFT storage a Web3.storage client. So, just so we have less stuff to manage. And token management token management is a major feature that will soon appear on both platforms, I think. This would be a way to have something like an administrative token that you could then create a user token for, and then users would be able to manage and delete their own data without having access to anyone else's.

But we'll also have a completely decentralized way of using a decentralized ID to prove ownership of a key, and then we just want to know that we can identify you in some way. We don't need to know who you are. So we're developing it, and it should be out. I mean, I can't promise a timeline, but great...well, the IDs here sound pretty incredible.

This is great incredible. This is great . Well, I think that's... . Well, I think that's...

all the time we have today. Thank you very much for this presentation, as well. Yes, thank you, that was great. But perfect, so with that we're ready to move on to