IPFS & Filecoin 🛠 Introduction to IPFS & Filecoin

[Applause] [Music] hello everyone and welcome to an introduction to ipfs and filecoin my name is discordian a community engineer with protocol labs this talk is about 40 minutes long so brew your favorite tea relax and let's dive in first a quick breakdown protocol labs is an open source research and development lab building protocols tools and services to improve the internet we try to operate in the public and be as open and accessible as possible ipfs is a peer-to-peer protocol making the web upgradeable resilient and more open filecoin is a decentralized storage network to store humanity's most important information all right what's on the agenda today we're going to learn about the core concepts of ipfs and filecoin and why they're important talk about what tools exist today to help you get started building then we'll talk about some possibilities throw some ideas that might be fun to build at the tech finally near the end i'll tell you some resources you can explore into to learn more about ipfs and filecoin as well as where you can go to find our communities and connect with fellow ipfs and filecoin enthusiasts data information and knowledge are some of the most important assets of our connected era they are critical to human development that's why we believe they must be safeguarded open and in the hands of the people but they aren't in the current model let's start with that problem currently the web 2.0 model is centralization amazon google fastly if any of these fail which they do occasionally entire services can go down as you get further and further distributed though your services resilience increases as the users themselves are what are powering the service so what's the solution how do we build a web that's distributed well a key part of ipfs and filecoin is that they're distributed ipfs and filecoin are not the answer to all of the problems but they are the answer to some particularly when it comes to removing borders and silos for data over the course of the next 40 minutes we'll see that ipfs a peer-to-peer hypermedia protocol for content addressing and filecoin the world's largest distributed storage network are important building blocks of this new web3 generation files and folders might sound boring but really they're not ipfs is the interplanetary file system what is a file system files and folders any file with any content why interplanetary because it was conceived as a way to upgrade the web in a way that would still work when the network stretches across planets the idea being that if you are sitting in mars it may take one hour for a request to go to and come back from earth but if that content is already fetched by someone else on mars then it should not travel all the way back to earth for it and that feature should be provided transparently by the network protocol rather than by something on top as we will see ipfs is distributed by design no central authoritative servers are storing content and no central server needs to be contacted in order to obtain the content typically you would upload the content to a server a location anyone wanting to download that content will have to get it from that location with ipfs things are slightly different in ipfs you run an application called the ipfs peer and with it you become the server you don't upload content anywhere but instead add it to your peer and make it discoverable by everyone else any content that is published on ipfs uses content addressing which is a way of referencing a piece of data by its hash to do this we'll need to know how to obtain a contact identifier ipfs addresses content by what it is instead of where it is it enables you to refer to a file or folder via a content id regardless of its location let's think of the process we follow when we save data locally it just goes on our drive and is addressed by its path the web is no different when we open a website we are just opening some files the difference is just that they need to be downloaded from a remote location and we do that using a browser which will print pretty things on our screens with ipfs we are also obtaining files from remote location however the key difference here is that we don't need to know the location of the content but instead what is called a content identifier ultimately the content can be at one or several locations but as we will see it does not matter anymore where it is we switch from a location-based identifier to content addressing by using a cid next let's look at the key technologies making this possible in order to be able to have content addressing we need to create content identifiers for each piece of information that we want to put on the network think of this as kind of a cryptographic fingerprint of that piece of content every piece of content produces a different fingerprint all the fingerprints are of the same size regardless of the amount of content that they represent this fingerprint which we named content id can be reproduced anytime from the original content by hashing it this means that if we obtain a piece of content after requesting a content id we can verify that we were given exactly what we asked for but what about folders folders are really just special types of files which have a list of files in that folder as content that list provides the names of those files and in the case of ipfs they're cids since a folder is a type of file a cid can be obtained in exactly the same way as for any other type of file this means that we can represent a folder or even a full file system using a content addressed structure let's bring it to life as you see here on the left side our top level folder has a root content id it has two entries corresponding to two folders and those folders have other entries corresponding to files each entry has a different fingerprint that's visualized with the colors here this content addressed type of graph is what we call merkle daggs directed acrylic graph these merkle dags used by ipfs allow us to move from location-based addressing to content addressing in a single step we are just replacing locations with the root cid of their content the sub paths stay the same one characteristic of content addressing that i mentioned is that fingerprints are unique for any type of content what would happen if we wanted to copy abc.doc to the folder of the second user it means two things first we don't actually have to copy the file we just have to modify the folder to reference the content two copies of the same content have the same identifier so to ipfs they are the same thing there is no notion that the same content can be duplicated in different folders in ipfs because it is actually the same thing with the same cid we call this property deduplication secondly since we changed a folder the fingerprint of that folder changed so we had to update the upper folder too to reference the new fingerprint this means the fingerprint from that folder changed resulting in a new root cid the moment something changes you get a completely different cid your previous cid still references your previous version of the content which has not been changed the fact that a cid will always represent exactly the same piece of information unlike a location unlocks the capacity of doing verification on any piece of data this matters because if the cid is guaranteed to give you the same content you don't have to get that content from a trusted centralized server you can ask anyone in the network for that cid regardless of whether you trust them so the first step when adding content to the ipfs network is to obtain the content identifiers for that content this can happen online or offline at any moment once we have those cids the next step is to announce them to the ipfs network so that they can be retrieved by other ipfs peers but what is appear what does it do peers in peer-to-peer systems are nodes that are connected to other peers to form a network or swarm forget for a moment about ipfs and think about a group of people if i want to address a person and communicate it helps if i can identify them for example by their name if we share a common language so we can communicate and if we have ways to verify that we are who we claim to be same with ipfs and the peers and the network each peer has a unique identifier their peer id this identifier is linked to a cryptographic identity which allows each peer to communicate securely through an encrypted channel the peers in the ipfs network are peers which support a set of services or protocols one of them serves to request certain content by its cid from a different peer and download it but in order to get there it means that peers need to be able to discover each other the network transports that they support tcp bluetooth and the services they can understand this step is called content and peer routing and it is achieved using something called a distributed hash table or a dht the dht or the distributed hash table is a critical service that each peer runs you can think of a distributed hash table like a phone book except everybody helps maintain it it's effectively a series of keys paired to values kind of like names paired to phone numbers in order to get the value associated to a key for example a content id a peer will need to find some other peer that is storing it the process consists of asking other peers in the network who has the value for this key but not in a random fashion remember that not every peer is connected to every other peer peers with certain names or peer ids are more likely to store certain key value pairs thus a peer knows which of the peers it is connected to is most likely to have the value for the key that it is looking for and if it doesn't have it it will request information about peers known by the other peer which are more likely to have it now that we have a database which is maintained collectively by all peers in the system how does ipfs use it simple when the content is added to ipfs a peer will insert one row into the dht keys are either content ids or peer ids peers can add rows to the dht declaring a their peer id provides specific cids and b how to connect with their peer id peers can then query the dht to find which peer id is providing which content ids peers can also query the dht to find information on how to connect to that peer id publishing this row involves finding a peer id similar enough to the content id to store the row in short any peer that wants to download some content will have to do two things first query the dht to find which peer id is providing the content they're looking for second query the dht to find information on how to connect to that peer id at the end of the dht queries a peer knows who has the content it wants and where to get it now that we've gone over obtaining a cid advertising it on the network and locating the content let's take a brief look at transferring the last step to retrieve content in ipfs is to download it because we know how to contact the peer holding the content and we know what content we want we can connect receive and verify it this means that any content received through ipfs is authentic and matches the request once content is downloaded the local peer will advertise itself as a provider for that content on the dht making it available to other peers for download the protocol we use to download files is called bitswap and it works for quick discovery by asking all neighbors whether they have the content ipfs really is useful to anyone who wants to serve or retrieve content on the web which is almost everyone as certain pieces of content get more popular the data becomes more readily available and resilient but why would random people help store my digital content the answer is well they likely wouldn't unless you pay them or they like you or they like the data you create also if you pay just a single provider you're kind of losing out on the beauty of the decentralization and trustlessness of everything because you now have to trust that single provider to make your data available and if their servers go down your data may be temporarily or maybe even permanently inaccessible so is there a decentralized and reliable alternative the answer is yes and this is where we start talking about file coin filecoin is storage designed for web3 from the ground up it's compatible with ipfs and other existing web3 technologies it's cryptographically verifiable so you can store cids on any blockchain really and reliably retrieve the same data every time provided you have a storage deal made over filecoin at over 18 million terabytes as of february 2022 that's a capacity which could store almost three percent of the world's data and the network is still growing at a very rapid pace there is enough capacity for almost any use case and there are already many many organizations and people using file coin to store data all right let's take a moment to talk about the anatomy of a filecoin storage deal proof of replication proves the storage provider is storing a unique copy of the original data proof of space time proves data is being stored continuously over time storage providers must take file coin fill as collateral to dissuade malicious activity storage providers by fill put it up as collateral must have internet connectivity and disk space they get paid in and earn fill to store and retrieve data storage providers the users providing storage and retrieval services compete on storage reliability and speed in an open market based on supply and demand for permanence any storage deal can be renewed an infinite number of times by anyone or anything think like a smart contract for redundancy there can be an infinite number of copies of the same storage deal ipfs is storage layer agnostic and that's because the magic of ipfs content ids is that they're a property of the data itself which makes them storage layer agnostic allowing for full flexibility and modularity the fact that it's storage layer agnostic means that you can combine it with the storage layer of your choice that could be filecoin which we think is a great option but you may also store your data with a centralized cloud storage provider or other decentralized or distributed storage solutions ipfs and filecoin are perfect complements for each other ipfs is great for fast flexible retrieval over the ipfs network and can even be accessed through http gateways for users not yet equipped with ipfs though ipfs needs somewhere to persist that data filecoin answers this problem by offering excellent persistence and verifiability through distributed storage deals which any storage provider can accept all right i promised i would talk about some of the tools to help you build your app so i'll go over a few of those now but first let's take a let's take a second to talk about storing and retrieving data we highly recommend choosing a storage helper they simplify deals and workflows making it fast and easy to build they simplify deal making in general finding providers negotiating prices waiting for confirmations verifying storage storage helpers handle all of that for you you can store to both ipfs and filecoin with a single call ipfs for fast flexible retrieval and filecoin for persistence and verifiability a storage provider with filecoin integration will ensure you never need to think about these things so what's out there first up is fleeq's ipfs hosting it's designed for fast modern censorship-proof websites and web apps on the open web and works with most modern frameworks it works by connecting your github repository you add some build settings and from there you can deploy your website to ipfs which is also backed up to filecoin you can find more about the services fleek offers over at fleeq.com web3.storage is a powerful solution for hosting your data for free backed up over filecoin and available over ipfs seamlessly data is immediately uploaded to a hosted estuary node and automatically stored with six providers around the world it can be used a variety of ways including the http api and web interface with a new web3 solution coming soon as well nft.

storage is a powerful solution for hosting your nfts for free backed up over filecoin and available over ipfs seamlessly nft.storage is designed for nft developers who want free decentralized storage you simply upload your data and get back an ipfs cid that can be used in on-chain nft data as a pointer to your content storage and retrieval is free you get properly formatted metadata back immediately after uploading you can fetch it back via ipfs and it's backed up to filecoin via free storage deals nft.storage is easily used via its javascript client library the http api or can even be used as a remote pinning service finally we have lighthouse lighthouse is designed to be the ultimate solution to your permanent data storage needs for a fee you can pay to store data forever lighthouse is a fairly new technology to the family of storage providers so they're looking for people to try their stock out and provide feedback i've listed only four storage providers here but from ceramic to estuary there are many more to discover by now you might be wondering what can i build with ipfs and filecoin well we've been brainstorming some ideas to help you get started but first let's briefly talk about web3 enabled architecture and what that looks like there are several layers to any application and web3 offers no exception you have storage helpers such as web3.storage nft.storage fleek estuary infera ceramic and more there are so many tools out there for storage providing that finding the perfect fit for your project should almost certainly be possible we have the logic layer as well through smart contracts they can point to cids served over ipfs stored via filecoin appending service or maybe even persisted through sheer popularity the logic layer offers endless possibilities with so many technologies including ethereum near and many many more the logo to the far right of the orange box is the logo of the filecoin virtual machine aka the fvm that's launching around summer this year fully evm compatible smart contracts will enable co-locating storage and compute finally we have the front end this is what you see and can include typical web 2 methods like throwing your page on a web server and having some server side code do some work but also web 3 where the website is served over the distributed web often with integrations like crypto wallets and nfts serving static websites over ipfs is quite easy so i hope to see more and more web3 type frontends as we move forward into the future here are 11 ideas of varying vagueness difficulty and complexity i'm going to try to go over them briefly so you can create a social media website where data is shared directly peer-to-peer timelines decided organically by the data you literally receive in a stream filtered by the client maybe you'd want to build a p2p chess game where the two players stake an nft and the winner takes all a distributed video streaming service could be nice users could monetize their videos through a token maybe requiring some form of payment for certain content or extras you could help cache the planet help store popular javascript libraries over the distributed universe an rss emulator could be fun users could subscribe to a pub sub topic which is a published subscribe system you can use through your ipfs note this is enabled by lib p2p peers could subscribe to a topic they're interested in and another peer could publish updates to it decentralized achievements could be fun create a system where web3 games can easily integrate achievements into their games adding a whole new layer of fun to new and existing web3 games you could make an app to simply snapshot a website and provide the immutable cid representing that website build the distributed metaverse where any user can host their own planet allowing users to colonize entire new worlds in an inexpensive and distributed way you could build a decentralized service that assists you in purchasing the correct temperature of coffee based on the weather that day a distributed git platform could be cool bonus points if you can integrate an issue tracker into it too a wrapper for youtube to help cache videos onto the decentralized web could be absolutely invaluable in preserving certain types of content which are completely lost to time well those were 11 ideas and i really hope at least one of them helps inspire you to build something if you want even more inspiration though find our team at an event and or check out hackathons.

filecoin.io for some past winners and see what they build as you're building also consider that not all of you have to build d-apps feel free to dive deeper into some of these layers biocoin creates a whole new economy with lots of opportunities to create new ventures at every layer i like this visual because you can kind of see some exemplary business models and how you may build moats around them more than an exhaustive list this might be a helpful mental model to think about where you see the most potential do you want to build a great interface that interacts with our tech which would be the top or are you interested in building dev tooling that makes smart contract coding easier maybe you're excited about the hardware and have some ideas on how to optimize a web3 data center please join our community everyone is welcome and we offer tons of support programs beyond hackathons to help get you started all right we've reached the end of this presentation so now i'll point you to some resources you can use to continue learning and hopefully engage with our awesome community come join our ecosystem to learn and build we have an awesome community comprised of builders who help us foster a positive and productive environment i highly encourage you to check out a resource we compiled of several tutorials currently available at tinyurl.com learn dash ipfs dash filecoin as well i genuinely hope to see you around and i'm available at many hackathons so please ping me or send me a dm even if you're not in a hackathon i'd love to hear from you i'm discordian on related discord servers and the filecoin slack but that's all from me today i sincerely hope i helped inspire or teach you something and i really hope that you have a lovely day goodbye to help inspire your projects i'm going to leave you with some clips i've cut together of sergey nazarov from chain link and juan bennett from protocol labs i hope you enjoy um both the chainlink community and the falcon community have um you know many thousands of developers probably tens of thousands of developers together those developers uh experimenting with uh with the tools and the tech and so on coming up with new use cases coming up with new applications and so in experimenting is what ends up building uh the new kinds of applications that are gonna flesh out and improve improve the world and one of the the coolest things that that's happened is uh you know the the these chain-link spring hackathon uh earlier earlier in the year um had like this record-breaking uh developer turnout with you know around 4000 sign-ups 140 project submissions participants from 170 countries it was a huge uh huge success and and you know one of the really cool things that that happened out of that was many of those of the participants in that in that hackathon went on to develop their their hacks into um applications and businesses and uh went into um the the uh the one of the several accelerator programs and i think nine of them uh at least i know of at least nine uh participants in the in the chain-link spring hackathon that went on to um to the tachyon cohort uh uh accelerator so you know cool ones that i'll mention include um you know or these nine are like weather exam which is a weather station oracle network um fractio tomago boto skill wallet nifty royale xerox essentials blockbed and chester so i'm mentioning these because they're you know they're example builders and example applications that uh came out of of that uh hackathon and is gonna and they are now uh off to the races building uh one of these applications um helping improve uh improved technology is that over 500 000 in prizes now it i i don't even think it's just the prizes i think it's the fact that there's a lot of mentorship there's people from google there's people from a lot of different large uh and small organizations that can that can help both very early developers and later stage developers and you're you're absolutely right that i've been really impressed um at the amount of hackathon projects that actually continue to live and continue to exist as not only participants in various incubators but end up actually raising substantial multi-million dollar rounds