IPFS & Filecoin 🛠 101 Introduction to IPFS & Filecoin

[Applause] [Music] okay well yeah let's get started then um so hi everyone my name is cree um i'm with the youth global team and joining us today for the ipfs and file coin workshop is discord um he is going to give us a like 101 introduction to ipfs filecoin um super excited to watch them myself uh and yeah i think we'll do a q a towards the like last 10 minutes um so feel free to like use the raise your hand emoji for that um once we start q a or if you're a little bit shy like i am um you can just pop your uh questions into the chat and we'll answer them at the end um so yeah discordant feel free to take it away all right thanks uh hello everyone um i am discordion and welcome to an introduction to ipfs and vialcoin i'm a community engineer with protocol labs and this talk is about 30 minutes long but i'll try to cut it just a little bit shorter so we have more time for q a so first a quick breakdown protocol labs is an open source research and development lab building protocols tools and services to improve the internet we try to operate in the public and be as open and accessible as possible coin is a decentralized storage network to store humanity's most important information all right what's on the agenda today we're going to learn about the core concepts of ipfs and filecoin and why they're important talk about what tools exist today to help you get started building then we'll talk about some possibilities throw out some ideas that might be fun to build with the tech finally near the end i'll tell you some resources you can explore into to learn more about ipfs and filecoin as well as where you can go to find our communities and connect with fellow ipfs and filecoin enthusiasts data information and knowledge are a couple of the most important assets of our connected era they are critical to human development that's why we believe they must be safeguarded open and in the hands of the people but they aren't in the current model let's start with that problem currently the web 2.0 model is centralization amazon google fastly if any of these fail which they do occasionally entire services can go down as you get decentralized and further and further distributed your services resilience increases as users themselves are what are powering the service so what's the solution how do we build a web that is distributed well a key part of ipfs and filocoin is that they are distributed ipfs and filocoin are not the answer to all the problems of the web but they are the answer to some particularly when it comes to removing borders and silos for data over the course of this workshop we'll see the ipfs a peer-to-peer hypermedia protocol for content addressing and filecoin the world's largest distributed storage network are important building blocks of this new web3 generation files and folders might sound boring but really they're not so ipfs is the interplanetary file system what is a file system files and folders any file with any content why interplanetary because it was conceived as a way to upgrade the web in a way that would still work when the network stretches across planets the idea being that if you're sitting on mars it may take one hour for a request to go to and come back from earth but if that content was already fetched by someone else on mars then the request should not travel all the way back to earth for it and that feature should be provided transparently by the network protocol rather than by something on top as we will see ipfs is distributed by design no central authoritative servers are storing content and no central server needs to be contacted in order to obtain the content typically you would upload the content to a server a location anyone wanting to download that content will have to get it from that location with ipfs things are slightly different in ipfs you run an application called the ipfs peer and with it you become a server also known as a node in the network you don't upload content anywhere but instead add it to your node and make it discoverable by everyone else any content that is published on ipfs uses content addressing which is a way of referencing a piece of data by its hash to do this we will need to know how to obtain a content identifier so ipfs addresses content by what it is instead of where it is it enables you to refer to a file or folder via its content id regardless of its location let's think of the process we follow when we save data locally it just goes on our drive and is addressed there by its path the web is no different when we open a website we're just opening some files the difference is just that they need to be downloaded from a remote location and then we do that with a browser which will print pretty things on our screens with ipfs we're also obtaining files from a remote location however the key difference here is that we don't need to know the location of the content but instead what is called a content identifier ultimately the content can be at one or several locations but as we will see it does not matter anymore where it is we switch from a location based identifier to content addressing by using a cid next let's look at the key technologies making this possible in order to have content addressing we need to create content identifiers for each piece of information that we want to put on the network think of this as a kind of cryptographic fingerprint of that piece of content every piece of content produces a different fingerprint all the fingerprints are of the same size regardless of the amount of content that they represent this fingerprint which we named content id can be reproduced anytime from the original content by hashing it this means that if we obtain a piece of content after requesting a content identifier we can verify that we were given exactly what we asked for but what about folders well folders are really just special types of files which have a list of files in that folder as its content that list provides the names of those files and in the case of ipfs they're cids since a folder is a type of file a cid can be obtained in exactly the same way as for any other type of file this means that we can represent a folder or even a full file system using a content address structure let's bring it to life as you can see here on the left side our top level folder has a root content id it has two entries corresponding to two folders and those two folders have other entries corresponding to files each entry has a different fingerprint that's visualized with the colors here this content address type of graph is what we call merkle dags directed a cyclic graph these merkle dags used by ipfs allow us to move from location-based addressing to content addressing in a single step we're just replacing locations with the root c id of their content and the sub paths stay the same one characteristic of content addressing that i mentioned previously is that the fingerprints are unique for any type of content what would happen if we wanted to copy file.txt to the folder of the second user well it means two things firstly we don't actually have to copy the file we just have to modify the folder to reference the content two copies of the same content have the same identifier so to ipfs they are the same thing there is no notion that the same content oh sorry uh there's no notion that the same content can be duplicated in different folders in ipfs because it is actually the same thing with the same cid we call this property deduplication secondly since we changed a folder the fingerprint of that folder changed so we had to update the folder too to reference the new fingerprint this means that the fingerprint from that folder changed resulting in a new root cid the moment something changes you get a completely different cid your previous cid still references your previous version of the content which has not been changed the fact that a cid will always represent exactly the same piece of information unlike a location unlocks the capacity of doing verification on any piece of data this matters because if the cid is guaranteed to give you the same content you don't have to get that content from a trusted centralized server you can ask anyone in the network for that cid regardless of whether you trust them the first step when adding content to the ipfs network is to obtain the content identifiers for that content this can happen offline or online at any moment once we have those cids the next step is to announce them to the ipfs network so they can be retrieved by other ipfs peers but what is appear and what does it do peers in a peer-to-peer system are nodes that are connected to other peers to form a network or swarm forget for a moment about ipfs and think about a group of people if i want to address a person and communicate it helps if i can identify them for example by their name if we share a common language so that we can communicate and if we have ways to verify that we are who we claim to be same with ipfs and the peers in the network each peer has a unique identifier their peer id this identifier is linked to a cryptographic identity which allows each peer to communicate securely through an encrypted channel the peers in the ipfs network are peers which support a set of services or protocols one of them serves to request certain content by its cid from a different peer and download it actually i see a pretty easy to answer question here does ipfs prohibit publishing of duplicate data actually as i mentioned previously if you add the same data you'll get the same content identifier so technically you can add the same data to your node and if somebody else added that exact same data you'll end up with the exact same content identifier so if a third person wants to retrieve that data and they have that content identifier both of your nodes will serve it if that makes sense we can go into it a bit more in depth later but in order to get there it means that peers need to be able to discover each other the network transports they support tcp bluetooth and the services they can understand this step is called content and peer routing and is achieved using something called a distributed hash table or a dht um i'll answer uh juan's question real quick too uh how can a legal content be removed from the network um a good way to think of this is think of like ipfs is like a distributed http so if you see a website that's hosting illegal content um i guess what you'd probably do is report it to uh like a government authority of some sort or however you prefer to get uh content taken off the internet with ipfs it'd be the same way you could blacklist the node so your node doesn't talk to them but ultimately at the end of the day you would need the nodes hosting the illegal content to shut down if you host an ipfs node it won't just automatically download other data so you'd have to actually put it on yourself so hopefully that answers that question cool all right so the dht or the distributed hash table is a critical service that each peer runs you can think of a distributed hash table um think like a phone book except everybody helps maintain it is effectively a series of keys paired to values kind of like names paired to phone numbers in order to get the value associated to a key for example a content id a peer will need to have or find some other peer that is storing it the process consists of asking other peers in the network who has the value for this key but not in a random fashion remember that not every peer is connected to every other peer peers with certain names or peer ids are more likely to store certain key value pairs thus a peer knows which of the peers it is connected to is most likely to have the value for the key that it is looking for and if it doesn't have it it will request information about peers known by the other peer which are more likely to have it um to answer another question is the way to protect uh data using password and ipfs um sure i guess two ways you could do it you could encrypt the data before you upload it like you would on the regular web um another way if you wanted to use ipfs in like a private swarm way so it's just your own nodes and they're communicating in an encrypted fashion uh we do support that um it's an experimental feature called a private network and everything's encrypted through i believe like an aes key or something and oh any questions that i don't get to i'm gonna try to get to all of them at the end so now that we have a database which is maintained collectively by all the peers in the system how does ipfs use it simple when the content is added to ipfs a peer will insert one row in the dht keys are either content ids or peer ids peers can add rows to the dht declaring a their peer id provides specific cids and b how to connect with their peer id peers can then query the dht to find which peer id is providing which content ids peers can also query the dht to find information on how to connect to that peer id publishing this row involves finding a peer id similar enough to the content id to store the row in short any peer that wants to download some content will have to do two things first well first query the dht to find which peer id is providing the content they're looking for second query the dht to find information on how to connect to that peer id at the end of the dht queries appear knows who has the content it wants and where to get it now that we've gone over obtaining a cid advertising it on the network and locating content let's take a brief look at transferring content the last step to retrieve content in ipfs is of course to download it because we know how to contact the peer holding the content and we know what content we want we can connect receive and verify this means that any content received through ipfs is authentic and matches the request once content is downloaded the peer id will or sorry the local peer will advertise itself as the provider for that content on the dht making it available to other peers for download the protocol we use to download files is called bitswap and it also works for quick discovery by asking all neighbors whether they have the content ipfs really is useful to anyone who wants to serve or retrieve content on the web which is almost everyone as certain pieces of content get more popular the data becomes more readily available and resilient but why would random people help store my digital content well the answer is they likely wouldn't unless you pay them or they like you or the data you create also if you pay for just a single provider you're kind of losing some of the beauty of decentralization and trustlessness because you now have to trust that single provider to make your data available if their servers go down your data may be temporarily or even permanently inaccessible and another question any free ipfs storage networks except for pinata uh absolutely if you want to store general uh data um there is web3.storage and if you want to store nfts there is nft.storage um and then i guess i'll sneak another question uh can we stream video content from ipfs into our uh the app uh yeah you absolutely can uh but the it's it might be an interesting challenge and it would actually be an awesome um an awesome thing for hackathon uh yeah i'm just gonna continue um so there is so is there a decentralized and reliable alternative the answer is yes and that's where we start talking about filecoin oh i really like this question is there any gas fee concept in ipfs writing data if yes how do we calculate gas um ipf i actually don't mention this in the presentation so um ipfs actually doesn't use a blockchain at all so there are no gas fees associated to anything um also when you add data on as i mentioned previously it's just getting added to your own local node and it's not being automatically replicated to other providers this is why i've mentioned like web3.

storage nft.storage if you wanted like a simple way to bootstrap uploading data and currently both of those services are free for up to one terabyte and then uh yeah can we deploy a react app or any front-end application ipfs you absolutely can if it's a static website but as there's no like server back-end if you rely on a like a server you're still gonna need to include that in some way so i'm just gonna continue um filecoin is storage designed for web3 from the ground up it's compatible with ipfs and other existing web3 technologies such as near and many others it's cryptographically verifiable so you can store cids on any blockchain really and reliably retrieve the same data every time provided you have a storage deal made over filecoin over 18 million terabytes as of february 2022 that's a capacity which could store almost three percent of the world's data and the network is still growing at a very rapid pace this is enough capacity for almost any use cases and there are already many many organizations and people using filecoin to store data let's take a moment to talk about the anatomy of a follow coin storage deal first we have proof of replication this proves the storage provider is storing a unique copy of the original data there's proof of space time which proves data is stored continuously over time storage providers must take filecoin fill as collateral to dissuade malicious activity storage providers by fill put it up as collateral and must have internet connection or yeah connectivity and disk space they get paid in and earn fill to store and retrieve data storage providers the users providing the content and retrieval services compete on storage reliability and speed in an open market based on supply and demand for permanence any storage deal can be renewed an infinite number of times by anyone or anything for redundancy there can be an infinite number of copies of the same storage tool ipfs itself is storage layer agnostic and that's because of the magic of ipfs content ids they're a property of the data itself which makes them storage layer agnostic allowing for full flexibility and modularity the fact that it's storage layer agnostic means that you can combine it with the storage layer of your choice that could be filecoin which we think is a great option but you may also store your data with a centr with centralized cloud providers or any other decentralized storage solution ipfs and filecoin are perfect complements for each other ipfs is great for fast flexible retrieval over the ipfs network and can even be accessed through http gateways for users not yet equipped with ipfs though ipfs needs somewhere to persist the data filecoin answers this problem by offering excellent persistence and verifiability through distributed storage deals which any storage provider can accept all right i promised to tell you about some of the tools to help you build your app so let's go over a few of those now but right before that let's take a second to stock talk about a story storing and providing data or sorry storing and retrieving data we highly recommend choosing a storage helper they simplify deals and workflows making it fast and easy to build oh um another simple question it seems how to retrieve the latest version of the data a cid will always refer to so when you generate a cid off a piece of data think of it kind of like a hash uh there is like uh some chunking involved but an easy way to think of it is like a hash or a fingerprint that will always refer to that version of the data so when you share that cid it will always refer to that version and then we have a new version of the data you'd get a new cid when you share that it'll refer to the new version and hopefully that answered that all right so let's take a second to talk about starting retrieving data they simplify uh yeah so storage helpers simplify deal making in general finding providers negotiating prices waiting for confirmations verifying storage providers handle all of that for you you can store to both ipfs and filecoin with a single call ipfs for fast flexible retrieval and filecoin for persistence and verifiability a storage provider with filecoin integration will ensure that you never need to think about these things so what's out there first up is fleek's ipfs hosting it's designed for fast modern censorship-proof websites and web apps on the open web and works with most modern frameworks it works by connecting up your github repository then you add some build settings from there you can deploy your website to ipfs which is backed up to filepoint you can find more information about the services that fleeq offers over at fleeq.co web3.storage is a powerful solution for hosting your data for free backed up over filecoin and available over ipfs seamlessly data is immediately uploaded to a hosted estuary node um and now elastic ipfs and i believe there's a little bit more magic working in the background but it automatically stores the data with up to six providers around the world it can be used a variety of ways including the http api and web interface nft.storage is a powerful solution for hosting your nfts for free they're also backed up over filecoin available over ipfs seamlessly nft.

storage is designed for nft developers who want free decentralized storage you simply upload your data and get back an ipfs cid that can be used in on-chain nft data as a pointer to your content storage and retrieval is free and you get properly formatted metadata back immediately after uploading you can fetch it back via ipfs and it's backed up to filecoin via free storage deals nft.storage is easily used via its javascript javascript client library their http api or can even be used as a remote pinning service and finally we have lighthouse lighthouse is designed to be the ultimate solution to your permanent storage or sorry your permanent data storage needs for a fee you can pay to store your data forever lighthouse is a fairly new technology to the family of storage providers so they're looking for people to try their stock out and provide feedback i've listed only four storage providers here but from ceramic to asteroid there are many many others to discover so by now you might be wondering what can i build with ipfs and filecoin well we've been brainstorming some ideas to help you get started but first let's briefly talk about web3 enabled architecture and what that looks like there are several layers to any application and web3 offers no exception you have storage helpers such as web3.storage nft.storage fleek infura ceramic and more there are many there are so many tools out there for storage providing that finding the perfect fit for your project should almost certainly be possible we have the logic layer as well through smart contracts they can point to cids served over ipfs stored via filecoin a pinning service or maybe even persisted through sheer popularity the logic layer offers endless possibilities of so many technologies including ethereum near and many many more the logo on the far right in the orange box is the logo of the filecoin virtual machine that's launching around summer this year which is very soon right now uh fully evm compatible smart contracts will and will enable co-locating storage and compute um and i believe with the fem we're going to be calling those uh actors and they're going to be wasn't based or webassembly finally we have the front end this is what you see and can include typical web 2 methods like throwing your page on a web server and having some server side code do some work but also web 3 where the website is served over the distributed web often with integrations like crypto wallets and nfts serving static websites over ipfs is quite easy so i hope to see more and more web3 type frontends as we move forward into the future all right so usually i read this entire slide but in the interest of time and i hope getting to all the questions and i love all the questions by the way everybody um it's really awesome um i'm just gonna leave this up for a couple seconds and then as soon as i stop talking i'm just gonna go to the next slide but there will be a recording and you'll be able to pause and look over these ideas so there are loads of opportunities for you to build as you're building also consider that not all of you have to build dx feel free to dive deeper into some of these other layers falcoin creates a whole new economy with lots of opportunities to create new ventures at every layer um yes carl i can i can provide a link to the slideshow after um yeah and i'll try to remember do that uh i like this visual because you can see um you can kind of see some exemplary business models and how you may build moats around them thank you um more than an exhaustive list this might be a helpful mental model to think about where you see the most potential do you want to build a great interface that interacts with our tech the top or are you interested in building the dev tooling that makes smart contract coding easier maybe you're excited about the hardware and have ideas on how to optimize a web3 data center please join our community everyone is welcome and we offer tons of support programs beyond hackathons to help you get started so we've reached the end of this presentation so now i'll point you to some resources that you can use to continue learning and hopefully engage with our awesome community so come join our ecosystem to learn and build we have an awesome community comprised of builders who help us foster a positive and productive environment i highly encourage you to check out a resource we've compiled of several tutorials currently available at tinyurl.com learn.

ipfs.filecoin as well i hope to see you around i'm available at many hackathons so please ping me or send me a dm even if you're not in a hackathon i'd love to hear from you i'm discordion related discord servers and the filecoin slack or discordian or pound 3926 that's all for me i sincerely hope i helped inspire or teach you something and i hope you all have lovely days and if we have time for question answering i'm totally happy to run over yeah no you're good for another five minutes go ahead awesome um okay so uh i know one question that i didn't quite get to uh was administrator uh followcoin ipfs is different uh yes they're totally different so um filocoin uses a uh like a blockchain um to use proofs of space time it's more of a storage solution where ipfs is more of a data transfer solution if that makes sense um and then amman actually has their hand rates so i would like to hear your question sure thank you um thanks for the presentation by the way this is super helpful to contextualize everything so my question is something that happens fairly common with kind of what two things is we know like the address and file and we have two entities one edits that file and we've hard coded kind of the you know that file link um in both of them so in this situation because the civ depends on the content itself if you know personnel entity a edits a file we want entity b to view the updated version of that file and like read data off that how do you kind of handle that from it like implementation and yeah encode what does that look like yeah um so i guess to make sure i understand the question correct so if you share a link to a file once you'd like uh so if a user views that link later and you've updated it for to see the new version of the content right um so we actually handle that using ipns which is the interplanetary name system and you can even attach a domain name to that uh using dns link so using a combination of those two technologies um you can get something that feels a lot more like traditional websites that makes sense but i think the user example also falls within the question i was thinking more like programmatically like if you have two different subsystems kind of looking at the same file but i'd imagine the same solution works in that case as well yes yeah um i i guess i'm not fully understanding like the question but if you type it out to me in like uh discord or something i'll definitely or on the forums i'll definitely try to like get to you and hopefully you can get through that so uh next question i know i missed a lot of questions um in final coin if i try to sort of follow it's bigger than the biggest disk of all storage providers will follow coin automatically split my data storage um so i guess uh the easy way to answer that is the data is already split up um but that gets into much more complicated like inner workings of uh file coin but you're you're talking about servers that have like like hundreds of terabytes of storage space uh okay uh so what's the speed like for data transfer um the speed is quite fast um i don't know if it's it if it's exactly as fast as like http but it's paralyzed um for most uh use cases it should be fairly fast but that would be something that i guess you'd have to test out and see if each stage works exactly how you want it um i personally find it quite fast does ipfs have something like an sdk yes we go to docs.ipfs.io you can go through um there's like apis um and uh i guess i'm not entirely sure like what you would need specifically for like like a software development kit but um we do have like libraries available for javascript and go uh and then there's http client libraries which can interact with like an already existing daemon um and yes definitely please post any questions of discord if i can't get to it what if the storage provider who has part of my data shuts down his pc so if you're talking like over filecoin uh they actually incur harsh network penalties um so unfortunately like while your data will be gone um they will also lose like their staked file coin so they have like like very high incentives to not do something like that um but in the question of will you be able to access your file if you've only stored it with that one storage provider then um and and nobody else has a copy of it then no you won't be able to access your file all right let's see when i upload the same image nfcs web through the search they both give different cities i zoom up through the storage cid of the folder is that correct um yeah that is likely correct also nft.search will generate nft metadata so i believe they're both giving you the cid of a directory but the nft.

storage1 will also contain um like a metadata file uh of the nft metadata so that's why the root directory cid is different um so i can specify the redundancy with my use of ipfs yeah actually one thing i find really cool about ipfs is if um so what i like to do is i publish my website onto my own node and then i can just take that cid and upload it to other services like web3.storage or whatever like service i want and then that kind of just offers me redundancy immediately um it's a lot easier than http where you'd have to like build up your own like cluster of like uh http servers um yeah hopefully that answered that question uh minimum file coin to stake to become a node validator sorry i don't actually know the answer to that question off the top of my head um that is a really good question though uh i'm unfortunately more of an ipfs expert than a file coin expert but that's a good question um what is the storage provider who has product okay i already did that one all right i think we are now at our five minutes let's see uh there was one question how does ipfs handle versioning of files that was the first question um asked and hopefully by the end of the presentation it makes a bit more sense but um it's as simple as like each new version of the file has a new um new cid cool awesome thanks so much to scordian um yeah if he wasn't able to get to your question because of time or just because there's a lot which is super cool um but things can definitely get lost uh just pop into the discord um scordian and the team will be in there to answer all your questions and yeah thanks so much for like the super yeah impressive and informative uh workshop um and yeah happy hacking everyone thanks for having me happy hacking you