IPFS & Filecoin 🛠 Introduction to IPFS and Filecoin & How they shape the future of web3

[Music] [Applause] i think we are shortly ready to start so i would just love to welcome everyone who is here today who is joining us from the stream as well and uh today we'll be talking about introduction to ipfs and filecoin and how they shape the future of webf3 there will be multiple topics we are covering during the session so at the end of it i will also give some time to answer all the questions so feel free to stay tuned and we'll be here also during the hackathon so let us know about all your questions and ideas about building projects during the hack to start with uh there are multiple projects which we have uh inside the protocol labs uh myself i'm a startup operator protocol labs but we work as a community of people and organizations that are building better future through the technology there's protocol apps which is open source r d organization there is a file coin foundation which is a steward of file coin community which is aspiring to put the power of humanity most important information back into the hands of everyone and there are also multiple core solutions such as ipfs which is appear peer-to-peer protocol that making web upgradable resilient and more open you can think of it as a peer-to-peer version of an http and there is a file coin which is a decentralized storage network to store humanity is most important information you can also think about it as an airbnb for storage and those are multiple components which come together in one place to be the web free layer foundation for the data storage and much much more so we'll go through a agenda today which first will be the core concepts of ipfs and filecoin then i will tell you more about the tools of web3 which will be particularly handy for you during the hackathon to build straight away on top of ipfs and filecoin i'll also highlight you the possibilities of webfree right now to give you a bit more ideas of what is possible what could be done and maybe that will be helpful for some of you to brainstorm your ideas further and there will be also some learning resources and discussion q a in the end so starting with the concepts of ipfs and file coin i will not go into too much details what exactly web free is but trust me that we are on somewhere transitioning from web 2 to web free where web 2 was read and write so you could write an information on the internet and you can read it from and now we're transferring to web free which is read write and trust and trust here as being an operative thing where you can verify some information on the internet directly without having a prior knowledge to it it also being supported by a bunch of underlying primitives and technology that builds up verifiability directly in web free moreover web free stack is for web free stack verifiable data is essential for a truly distributed system this image is already quite outdated but because the web3 is exploding like hell and there are more and more projects appearing each and every second but at the same time uh the file storage is still a core fundamental piece of it moreover you can place now now us not only in file storage as it is with ipfs and filecoin but you can also definitely move us to the cold storage solutions and uh with soon updates of the file coin where we're developing smart contracts and filecoin virtual machine you could also easily place us to the layer one blockchains with smart logic and smart contracts so once again we are some way in transition between web 2 and web 3 where web 2 is being decentralized solution where the data knowledge and information is being kept at the centralized points and we barely have the ownership of that data which is being centralized it's a critical point to the human development and that's why we at protocol labs believe that that this distribution of the knowledge and distribution of the power should be safeguarded and the knowledge should be open and in the hands of people but at the moment data information knowledge is not publicly open and publicly owned and there are a few companies which you familiar which is google amazon apple which are being the silos of the data and limiting uh the ownership of that and if some of this solution fails due to the system being centralized it fails all together and you basically lose access to your knowledge what you can do instead instead of having a centralized solution where you have single points of failure when where your data is monetized by the data monoliths you can have a decentralized or even better distributed system where the users have the power to manage their data where the users power the service themselves and at the same time the infrastructure which is being developed in a peer-to-peer manner it's also been censorship resistant you it ensures the privacy of your data you can self-verify all the content which you're having in the web and the data is being non-siloed meaning that at any point of time you can discover the information that you are really interested in without any limitations and i will tell you in the next 15 minutes or in the next half an hour what is ipfs and filecoin and how they contribute to this vision of distributed decentralized web-free internet and how to build on top of those protocols to make sure that your really going to the future in the web free and moreover i will tell you that files and folders is not a boring thing so first of all ipfs is an interplanetary file system which where uh files and systems is of the file system is obviously files and folders you are familiar with them already you work with computers all the time so there's just the same thing while interdis interplanetary thing is way way more interesting imagine that you have to send uh imagine at some point we reach mars and we'll have to send information from master earth if we maintain these centralized solutions to for any call or any update or to watch a video you'll have to make a call from mars to earth and from earth to mars backwards and this will at least for one call take you an hour so that's not very scalable solution when your teams think about expanding to other planets or generally just expanding your network and that's where ipfs is coming handy into the place because of distributed solution in the first place it allows you to download the files not from the central place somewhere on earth but rather have if someone has it already cached on the mars you can directly retrieve it from them therefore connecting and creating a network of peer-to-peer data exchange where everyone can cache and store the data and it's available not from only one source but from multiple sources creating this resilient and distributed solution that can scale across the planets and that's where resilient and offline first solutions such as apfs is coming very very handy to develop a network which is independent of centralized solutions moreover in order to achieve so it's very very complex if you try to address the data as you commonly do it in web 2. for example if you go if you're looking for a book and your friend tells you oh you can find this book in the library on the second floor on the third bookshelf and on the third row from the top then you and it will be the second book from the right when you come to this place and get the book suddenly you realize that it's the same book that which was in your backpack but you just didn't know that it was there and you didn't know that you were referring to this book exactly the same thing is happening right now with computers or web 2 way of addressing information where in order to find the file you actually go to the location of it instead of that or it's actually happening the same in the browser where to find an object in the internet you go to an http domain you go to the path to the index of it and then you find a file there it's a very very uh weird way to structure finding information and why not instead to actually look for information itself so instead of going to the library to a certain place you just tell a person which book to find and essentially you are looking for a content and not for the address where it's located and that's where ipfs brings the most value because it allows you to address the content by what it is instead of where it is you essentially replace a folder or a file location with a content id which is a unique fingerprint which allows you to discover the object's internet regardless of their location and i will tell you more how the ipfs works specifically for example you have a content which is hello world text document sorry and you encrypt you create a cryptographic hash function out of it which is a sha 256 standard algorithm and you create a content identifier for that so once you created a content identifier this is a unique fingerprint which points to exact content of the file which you created so the same content will always resolve in the same cid and content cad can be reproduced anytime from the original content so imagine that it's independent of who creates the cid rather it's always a computer which is dependent on the file you upload so if you have the same content any computer in the world or any solution will create you the same cid therefore you can easily discover the object by what it is rather than by where it is and it also adds us this trust function of the web3 where once you received an object from the ipfs or from any other solution you can validate through the cid if the content you received matching the original cad you requested if something is flawed then you'll have a different cd and then you know that it's not the content you are looking for the best part of it is that content is not only files itself folders could be the same content and essentially a folder is a special kind of a file which lists other files inside of it so all the core principles which apply to the uh files in ipfs work the same way in the work the same way in the folder structure so you can also use files and folders in ipfs without any limitations and to give you a brief idea of how exactly it works here is a small diagram where we have four objects in the bottom and each of it has produced its unique cid so you can always trace an object by a cid or each and every and these images are grouped into two folders first one containing the third three uh first three and you see by different colors there is a fingerprint so these are different cids in the uh on the first place then all these two folders are then connected in one folder and you get the root cad of the first initial folder so when you have the root cad you can discover which objects and files are inside of it and which cads continuously are inside of it and that's what exactly merkel dag is or merkel direct acrylic graph which is a graph data structure and where each node is content addressed by itself so cads and ipfs allows you to not only address the files but also the structures in the direct acrylic graphs and the beauty of it it allows you to really duplicate the internet in a sense that imagine there is a case where a new folder is created but the content of this folder is based on already existing solutions in ipfs and on already uploaded content so here for example i create a new folder which contains two right images and even though i create a new folder with a new list apparently it gives me the cads of the previously added objects because again cads are referring to the object content and not to the where they are located therefore we get the ability to duplicate the amount of files we store and essentially instead of storing hundreds and thousands of meme on memes on the internet you can just store five redundantly and always point to the same objects instead of just storing them again and again and again this also gives us verifiability if some of the contents inside the folders change then the new cid will be as a result and therefore you can always trace whether the data which is served to you is reliably what you have requested in order for these to work in the network in a peer-to-peer network there is also some functionalities which needed for you to have as a network so first of all the network of ipfs is a swarm of peers where peer is a node in a peer-to-peer system in order for the system to work there are many many peers across the world which are serving parts of the content therefore if something changes or something happens to any of the peers this whole system still became resilient and you you can just retrace the content or find the content on any other peer that is available and still online so in order for the system to work you have uh you need to impose several requirements on the peer-to-peer system therefore each and every node in the network must be discoverable so you have to be able to discover where the node is located and which content contains and it must be rootable so you need to be able to any node to have a trace for other peer-to-peer nodes so that at any point of time you can discover both the content and where it is located and that's where a distributed hash table functionality which is available in ibfs is coming very handy imagine distributed hash table as something like a phone book where you have uh a name of a person and a phone number of a person but instead of having just a phone book by yourself you are sharing it with your friends or your peers so you maintain it jointly and you all adding new names and new numbers to it and that's exactly how the distributed hash table works in ipfs so there is a group of peers or a group of nodes which are maintaining the same distributed hash table and adding the content to it on upon uploading to a specific peer or note and it's maintained by multiple peers but only a small number of peers so you never have a distributed hash table of the whole ipfs because it will be a tremendous size and very hard to maintain rather if the content is not served in this particular dht or distributed hash table the peer itself can refer and ask neighborhood peers if you if they have content in there dht tables and it's very fast and very simple to trace whether there is a content available on ipfs network in general and in which particular dhc table it is located because all the dhts contain the cads of the content in a manner that they are discoverable and in manner that they are grouped together by similarity of the cad compared to the content and that's where it's becoming really easy to manage and operate the whole network because every peer can add rows to the dht table declare their peer id and provide specific cads based on how they connect to the peer id and how they are available through the network moreover peers can query the dht to find peer id which is providing the uh which content id so making every note available to trace and reach any content in the internet and that being a self-serve system where it's also independent of protocol apps or development solutions where you can it's an open protocol on the first place so whenever you have this solution it's available for you at any point of time regardless of limitations or centralization which i talked to you beforehand the the very nice piece about it is that once you have all these concepts ready and technically working which they are already for quite some years uh you need to start thinking how to build a network out of that and how to serve the content which you particularly have or you own through the network so it's available to you over time and it's available to you whenever you need it so there are multiple ways of using ipfs in general you say though you need to understand that if there is no nodes there is no retrievability so you need to think about who actually runs the nodes in peer-to-peer network so obviously first of all you can run your own node and store all your content just on it so but then it's becoming the same single point of failure and if something happens with your content then it's basically lost uh then you can also run your own network of nodes but at the same time it comes at the cost of infrastructure running and the team to manage it you can also pay pinion services that are available already right now on the network such as pinata temporal infuria and many more but this is a payment solution so you have to pay someone in order for them to provide you the content from peer-to-peer network and at the same time you may just hope that content becomes popular enough and that you can that other nodes will be in it instead of on your behalf just for free and that's that system where you have to rely on a solution or on someone's good will to pimp your data in order to be able to retrieve it sounds reliable and decentralized for you for me personally not and that's exactly where we can talk about file coin and how filecoin incentivizes persistence and verifiability of storage on web3 filecoin is a solution which allows you to make a deal with a storage provider which we have more than 4000 at the moment with more than 18 exabytes of storage capacity already available on the network and it allows you to create a deal where a storage provider confirms that he has your data for the duration of the deal and at the same time it's available and we can prove uh verifiably prove that he stores your data continuously through the whole duration of the deal the duration of the deal and the provider has to run cryptographic proofs on the data which he stores for you every 24 hours so we're making 100 sure that the data is available and the beauty of it is that it's also being replicated multiple times or sort redundantly on the network so that even if something happens with one single storage provider it's always available to you regardless of what an additional benefit or the way we designed it to actually serve the multi-purpose is that it also utilizes the same cad solutions as i just described you into inside of the ipfs so essentially you can have your ipfs cad being stored and verified over a file coin deal and it's stored redundantly multiple times so that's how you achieve this decentralized verifiability storage on web3 for your lifetime of your project your lifetime of solution and just long term having preserving your knowledge which is essential for human development and that's where you can also run multiple times the dealing on the file coin that's where you see if you first receive the fundamental reliability or resilience of the structure the beauty of it that you can also run it in parallel with cloud or with any other solution because we are essentially agnostic to what are the other solutions you are using to store the data you can also backup it with the cloud you can use it different solutions from different storage providers but it doesn't stop you or we do not lock you specifically on file coin for you to retrieve your data and that's where i would say that ipfs is really loving filecoin and they really designed to work well together where ipfs is designed for the fast and flexible retrieval where the gateways local nodes and browsers are available the beauty is that right now we are also natively supported by multiple browsers the brave is already fully supporting it and running an own ipfs node on the background for you so you can directly watch in the browser all the content from ipfs and there are multiple more browsers that are coming soon with the implementation of ipfs reading where whether they're using just a gateway for ipfs or they're directly installing the nodes they're more are coming from firefox and then we're also working on implementing it into the chromium so all chromium-based devices will be also natively able to read ipfs cads and content inside of it also in on top of the ipfs there is a file coin which provides you these needed persistence and verifiability of the data on chain and reliable storage of the data for the duration of your content or duration of the lifetime of the deal which you have made on top of ipfs and filecoin and that's where we can slowly transition to the tools of webfree where you can i will give you in the next uh 10 minutes or the details how you can build on top of ipfs and how filecoin and how to easily kickstart your project during this hackathon so that there is no problems in building your own nodes and finding the solution that you need to develop on top of ibfs and filecoin so the very first and very very easy to use solution is nft.storage where uh it's completely a free solution to store your nfts with simply two two lines of code you can register on the nfc the storage website you can get your api keys and when and whenever you make a call with an api uh your data will be stored on ipfs and filecoin and it will be pinned redundantly five times across the network the beauty of it that cads are again completely uh computed locally so there is no dependency on any of the storage solutions which you are using and at the same time you can still discover it over the traditional ipfs network so all the content could be stored and served on the web directly through nft storage also the beauty of it is that from protocol apps perspective we are treating nft.storage as a public good therefore it's designed to be completely free for use for any nfts we preserve the nfts as a common knowledge and we believe that that's the way how to build the web free so that people really have the access to the data they create at any point of time also if you even think about but what's the incurring cost maybe we'll run it for price later on and then charge everyone because they're locked to the system uh apparently we are also at least 100 times cheaper than avs and cloud solutions so it's run on behalf of protocol labs and whatever happens to the solution it still could be pinned on your local nodes and be available and retrievable at any point of time when you want to scale your solution not just for nfts but for any kind of storage that's where the web3.storage is coming the most handy it's provide it's the solution for general ipfs and file coin storage and it provides you a simple and familiar interface also with only two lines of code to implement to store it reliably on ipfs and filecoin and it provides you one terabyte of data for free so you can use very simple javascript or go client implementations and it has an http api for remote pinning service and a web interface to upload your data or to retrieve your data on your browsing solutions when you're also thinking about developing your websites that's where flick hosting is coming into place it's flic.

co and it allows you to build your websites on top of ipfs and filecoin and store them directly into ipfs network which uh makes you independent of any cloud storage provider and it's reliably accessible through the peer-to-peer network as well it is very easily built on top of github repositories so you can just connect it uh set up your build settings and it will be deployed directly uh onto a pfs and backed with uh the file coin solution so just uh simple and amazing tool to run your solutions on top of a really decentralized network and be ready for web free whenever you're building your project there's also an advanced solutions or additional solutions where developers who have uh powergate if they want to have a powerful ways to connect and extend protocols such as ipfs file coin and lip b2p it provides you the opportunity to have a docker container wrapped around ipfs node and filecoin node and it gives you the flexibility of staging storing and retrieving data and have the default configuration for the miner selection and one more topic which i want to cover in this presentation and which is the more and more upcoming and important effort of filecoin in particular is filecoin green it provides you now the opportunity to measure the energy which is used by storing your data and our whole effort of protocol apps and ipfs and filecoin in particular is to build the reliability on the web where you can specifically verify that the data is being stored in sustainable way and it's uh provable that the sustainability aspect is not faked or is not uh just a number on the on the dashboard but rather it's something which is uh cryptographically or verifiably proved that the solution is totally green it also provides you the consensus mechanism that uh is used by filecoin which requires each node to prove that resources contributes to the network therefore you can easily calculate uh the amount of the energy which is being used to store your data and then later on scale it to more projects on the green aspect of the sustainability of the blockchain i think that is one of the interesting topics to work also on in the scope of the hackathons because the green aspect was not that uh in the raiders of developers for quite some time right now but we have seen more and more the problems or the general sustainability issues on the world around and i think as vitalik was mentioned it's really a matter and great effort to build the sustainability of the blockchains in the first place making sure that the tools which we're building are scalable to the future also from the sustainable perspective and that's where i'm coming to the amazing possibilities of webfree or what to build right now how to build it and generally it's an amazing network which we are running and the ecosystem which accounts for many many people building on top of ipfs and filecoin it's quite an outdated uh view already but there are all types of uh solutions from flick from orbit from uh technical solutions there are also many many teams that are building on top of ipfs and filecoin from other chains there are practical technical implementations and i encourage you to look through the hackathons.file.io website where most of the hackathon participants from the past year were listing their price projects so you have all the access and all the knowledge which we maintain across our ecosystem so you can have your kickoff of the project for the hackathon for if amsterdam as soon as possible and build on top of reliable solutions rather than developing something new from scratch also there is a list of projects i really like in particular which were using ipfs and filecoin first of all we are also backing up the internet archive which contains the most important human data right now we are also collaborating with starling lab which preserves which creates a framework for data integrity and preserves uh testimonials of the holocaust survivors that's where we're also looking to contribute to the public goods and creative commons they're an effort joined together with ethereum foundation and ethereum networks where we're collaborating on funding retroactive public goods funding and generally finding solution for the public goods to be reliable and accessible topic over the web free as well there are also nice practical applications such as audios which is a new version of music tool where you can listen music podcasts and everything there is live peer solution which gives you the opportunity to do video streaming and nft meeting of the videos straight away with their development we also have an amazing projects with [Music] teams or organizations outside of the web free for example we're developing the new york city open data with file coin and ipfs and if you're familiar for with opencl wearable or you're generally interested with nft's topic then you i can also highlight that in the back end of openc they're also storing all the nft content through ipfs and filecoin so in order to summarize that webfree is essentially a decentralized web where the blockchain is a fundamental layer which allows you to make uh verifiability on chain and provides you the independency of a centralized state rather than ensuring that the content is being available it's been linked and being resilient to the future proof solutions and applications and just to sum it up apparently the best way to predict the future is to create it so i really encourage you to do all the best on this hackathon having an amazing 36 hours of hack build your projects build it on top of ipfs and filecoin let us know if you have any questions about how to build us will be available on the booth we'll be also giving a talk tomorrow uh our ceo juan manuel will be providing a talk on uh public goods and crypto economics so there is a lot a lot of opportunities and once you join the ecosystem they're more than just building on top of ipfs and filecoin we are also running a huge grant program so for us and for you hackathon is just the first step you can scale further with our grant application with our micro grants development grams and then launching further your project with pre-seed seed rounding and startup as a part of an awesome community and last but not least i will also highlight for you some learning resources there are discord and slack available and specifically for this hackathon i believe we are running through the eve global discord but feel free also to join our discordant slack from ipfs and filecoin you can find it at file cos filecoin.io.slack and you'll have all the answers there we are working as a public company so everything is available from our development directly on slack you can reach any of our team members on slack and that's where we believe that working open source is not only building up so open source solutions but actually leaving open source making sure that we are collaborating instead of hiding information and where everyone can contribute to the scale of the greater good and the greater ecosystem with that being said i would just encourage and say good luck to you all on developing your hackathon projects it's definitely an amazing time and i was a hacker myself a few years ago i love this time and i'm will be here together with my team to support all your questions and making sure that uh an experience for you to build on top of ipfs filecoin and any other solution from protocol apps is an amazing experience and you want to stay with our ecosystem further thank you a lot and that would be it from my side and i guess we are running greatly in time so we have at least 10 minutes for the questions but i'll also be happy to stay further and answer all your questions afterwards so who want to be the first yes please storing data with like access control or is everything always public and if you have to do your own solution yeah that's a very good question oh yeah uh so the question was if the content on the ipfs and file coin is public or you can somehow encrypt the information first or what what are the solutions to work with the private data essentially so ipfs and filecoin is deliberately an open uh network so all your content which is stored through ipfs and filecoin is public you are completely right that you can first encrypt the information and then decrypt it when you download it but it will still be available on the public network as a peer-to-peer uh storage solution uh i would say that there are some teams who are working on encrypting the solution uh on encrypting protocols on top of ipfs and filecoin but you are more than welcome to develop your own i think that's a hot topic right now and if you're working with some private data it's very interesting to also think where which parts of data is actually sensitive and should be private because quite often it's only a small fraction of the actual content you want to store which is private and you can split it and still store most of the content in a completely open system there was one more question yes please well whenever we speak about public uh goods of all like foundational human knowledge is very important topic i believe the like the content about learning about uh just general education is the something which should be preserved and should be available regardless of the pay walls regardless of proprietary information that's something which all the society should build on top of there's also public goods which could be the data sets which provides you the knowledge on for example medical data sets or something which you can build on top of in order to create a more resilient and more efficiently working system at the same time public goods could be some uh i mean there is common projects which we are building from retroactive public goods funding with ethereum foundation where it's also being a focus of ensuring that the developments which are happening in the public goods field is uh being well funded and receiving sufficient support from the teams and at the same time apparently just storing the data for someone could be already a public good whenever the information which is being stored is socially valuable or uh just generally valuable as a human knowledge does that answer your question yeah yeah thank you are there any other latency issues questions yes so uh if we are speaking about web applications like hosting the website through flic.

co i uh i believe they're also hosting it on their own nodes so you get quite uh good retrieval speeds though yes you have definitely keep in mind uh on the real retrieval speeds of the data uh ipfs file coin is now uh not always as fast as the traditional solutions though we are right now building the super hot cdn solutions where in one two months it will be production scale ready and the same speed as the cdn on of a traditional cloud so that's the feature we are also aware of and we are developing uh towards making it uh large scale and available as a retrieval uh storage market but that's uh a topic which we're also discovering right now but there is already like you can already build on top of ipfs and filecoin and get proper user experience though it takes us an additional time to make sure it's all fluent and available storage we're actually operating that's extremely fast already which is like specifically fine but yeah there's more and more but there's a couple of retrievers firecoin eventually will have um will incentivize that with the extra crypto economics where you have a market for retrieval notes and then basically anybody could be a retriever at which point you have an extremely atomized like cdn that's more distributed than any other cdn could ever be right because anybody is rich perfect any other questions then what does it mean for me to run an ipf note let's say if i want faster dual speeds can i clean the content of my node yeah that's exactly how it works so if you are for example if you're sitting in the same room and someone from the room already have your content he can directly serve it to you instead of going to a centralized node and downloading uploading it directly so that's the beauty of it where the more peers are serving the same content the faster the retrieval you will get and if you are working from the same network that's the ideal case where the peer can just directly streamline you the content which is which he has and that's where the more knows the better is the approach but at the same time uh having this retro ability speed is also dependent on like the size of content and some of the delays which may occur on like peer-to-peer network which are like which are traditional to it what is it internally is it the object storage traditional unix file storage uh can you elaborate a little bit more on the question how exactly the file is stored inside the inside the node you know is it like object storage or do you use some technologies like i don't know how hdfs stores the files yeah so it's a it's a file distribution so it's uh any object could be uploaded independent of the file format and it will be served uh from the node to the node uh not all the nodes will contain the file fully so some of the nodes will be only serving it as like transmission nodes so they do not need to store your file directly but they will be only transmitting through them because essentially there is uh connectivity could happen through each and every node on the network any other questions just please i have a slightly more technical question so as far as i understand the way the cid hashing function works on images divides the image up into separate chunks yes essentially if you change any pixel of the content you'll always receive a different cd i think that's a very interesting corner case and i would love to know if you explore that further during your hackathon but i would assume that if the content or if the file size also changes then it will also affect the final cid because essentially you store it in a different format than it was originally so the idea of the cid encryption is that every unique content creates its own unique cid and in downscaling i believe you still change in the back in the background of the file how it's been recorded so it doesn't take your image directly in a sense it takes the way machine explains it and then transcodes it into a sha function yeah any other questions if not i would love to thank you all for your attention it was an amazing time and uh pinning service i would use directly web free dot storage or nft storage they also pin the content on your behalf so it will be pinned five times redundantly and available for the file coin deal for the file coin network and it's also free to use yeah then thank you all and have an amazing hack [Applause] [Music] [Applause] [Music] you