Making IPFS Work For You – Matt Ober

hey everybody this talk is called making ipfs work for you if you can't see anything sorry apparently the bulbs a little iffy for the present here check out the live stream with the full colors so my name is Matt Ober i am the CTO of pinata and we've learned a lot from running our platform over the past couple years and we're excited to share that knowledge with you to start off with a little bit about me or the why I'm qualified to give this talk slide I started off building aetherium applications for about two years a little custom custom enterprise stuff and we noticed a lot of then we're using IP FS which kind of led us into pinata I've been working with ipfs for about two years since then and this is my third eath denver so I'm excited to be back and speaking this time so what is pinata pinata is the solution for ipfs DevOps as we like to say if you don't want to manage all the ipfs infrastructure or if you want added we're done and see your speed for your IP FS base product come hit us up it's easy it's fast and you know we make sure it's stable through our network of nodes around the world so we also have a $500 bounty this weekend for any hackers that are working with ipfs super easy all you really need to use do is just use our API looking forward to submissions all right onto the talk so blockchains have a problem they're really good at time stamping data but they're really bad at time stamping a lot of data it costs way too much to store any significant amount of data on Shane and for example for something like aetherium it would cost thousands of dollars to soar as a store one single megabyte on Shane and that's clearly infeasible for many applications so this has led to many projects storing data off Shane and then referencing where that data is stored on Shane oh yeah so a common thing that we see a lot an example that we would actually encourage people to avoid is this pattern of referencing basically modern server URL based infrastructure so let's use our example here say you have an NF T or a non fungible token on chain you want to store that data off chain but then you reference it with like a modern URL so you know my server comm slash the grayscale pinata alright we can reference that back on chain but the problem with this paradigm is that we can change what that underlying image is without the end-user knowing this isn't very immutable this isn't tamper proof and it's not very web 3 something that we're kind of looking to move away from so if we don't to use the cloud to store off chain data than what should we use if anybody has a wild guess the title of this talk is probably going to give this one away for you so enter the interplanetary file system or ipfs for short ipfs is what's called a content addressable file system and it's what many blockchain projects are turning to for they're off chain data storage so how does ipfs actually solve our blockchain data storage problems pretty simply so how it works is you send data to i/o PFS and ipfs runs it through some hashing algorithms and gives you back a unique hash or what's called a Content identifier and then at any point in the future you can ask the ipfs network and say hey can I get this content for this content identifier and if somebody's still hosting that content you're 100% guaranteed to get back what was initially uploaded this is tamper proof and immutable is exactly what we want when dealing with option data so to give it a little bit an example of how this would work in real kind of application base level let's imagine we have a document and we sent it to ipfs ipfs gives us back that hash and then we put that hash on something like say the etherium blockchain now a user in the future they want to grab that content so they see on Shane okay here's the hash for the content that I'm looking at retrieve they take that hash and then they asked the IP FS network hey do you have the data that I'm looking for from this hash ipfs says yep here you go and it gives it back to the user the user is happy so many of you are likely coming from a blockchain background and it's why you're here today at Denver when working with ipfs it's very important to know and remember that while ipfs is used by many blockchain products it itself is not a blockchain and this is most apparent and how the two technologies store their data so with blockchain networks when a user adds data that data stays on Shane forever that's because all of the nodes in the network are storing that data and then if that user goes offline users in the future can still pull that data from the network regardless of the weather the original user is still online even if that original user is no longer connect to the network the data is still there it can be retrieved so with ipfs this is not the case in order for a user to retrieve data from my PFS there needs to be a node that's online and hosting that data so if the node hosting the data goes offline the user that's wanting to retrieve that data cannot get that data because there's nobody to serve it so the key thing to remember here is that the ipfs network does not actually store your data for you it's not a magic cloud where everybody's kind of you know working together to store your data for you because that costs money and that's not really there's no incentive model to store data on ipfs so if you want users to be able to retrieve your data you have to have somebody storing that data and it helps to think of IP FS in this regard to modern data storage solutions so in a modern application let's say you have your phone you send data to your company's server and your server will send that to your database that you own somewhere with ipfs same sort of thing happens except instead of sending it to a server you're going to send it to either a company owned ipfs node or you could send it to a service that you to host that content for you cough-cough pinata so the key takeaway here is that a node must always be hosting your content gonna drive that point home if you want your users to retrieve it you got to have somebody hosting it and they got to be online so let's dive a little bit deeper for those of you utilizing ipfs and your own projects I'm curious if you ever run into this scenario your user uploads data you stored on your company ipfs node the user attempts to retrieve that data and the Gateway times out and then now the user is sad because they can't find the data they're looking for so this experience is relatively common with companies when they're first starting to work with ipfs to understand why this is happening we first need to understand how content is found on the ipfs network like we mentioned earlier the ipfs protocol acts as one big routing layer for servers that are hosting content on the network it's helping nodes announce and find content so if you've seen the diagram we have here we have one node with the content and it's constantly announcing to the network and saying hey I have this data I have this data this happens all the time and then there's other nodes in the network this is a common peer let's say the announcing node and the requesting node are both connected to the common peer knows that our our host node has the data so then when the requesting node asks all of its friends hey does anybody know who has this data our common friend here knows who has it knows to route the requesting node to the host node so this is out content propagates throughout the network however ipfs has grown a lot lately there's hundreds of thousands of nodes that are currently on the network and currently content discovery has started to feel a little bit more like this where you have the host nodes or announcing their content and you're shouting into the sea nobody knows where anything is because there's just too much noise so this this isn't great for user experience so how can we avoid this congestion our first fix for this problem extends from we're calling swarm connections so ipfs nodes naturally connect and disconnect to other peers is to try to find nodes the hosting content they're looking for however if a requesting node asks their friends if they know who has this content and the host node just happens to be one of those friends or this appears that they're asking it's going to immediately respond back to the requesting node hey I have this content it's basically shortcuts around this lengthy lookup process that many nodes are going through on the public network in practical terms this means that the potentially long network discovery process is skipped and the content can be served quickly so key takeaway number two here connected nodes means fast content discovery this is super important for any project that's building with ipfs it's taking a while for your content to be found you need to make sure that the host node is connected to who's ever requested it there's a few strategies we can use for this so how can we use this to our advantage two main strategies I talked about the first one a little bit which is directly swarm connecting your nodes this is like the easiest kind of simplest way the manual way to do things and then the second method that we talked about is private or dedicated networks and I'll explain what these are in a little bit each of these solutions has trade-offs and one may be good for you one may not be good for you this kind of depends on what you as an application or looking to build so we talked about swarm connecting let's talk about how this works node a let's we'll manually connect to node B and at this point note a won't note disconnect from node B but note node B doesn't care node B doesn't care about node a because it hasn't explicitly been told I care about node a so as node B as garbage collecting it may garbage collect the connection that node a established with it so in order to prevent this both nodes need to agree to connect to each other perpetually so we want to do is after we connect node a to node B we need to tell node B to connect to node a so it's this mutual process that avoids kind of nodes being taken advantage of maliciously it ensures that if you want to stay connected parties have to agree so with this connection it lets applications create this fast lane between nodes that want to retrieve content and those that are serving content so you have your requesting node and your host node and they're kind of able to bypass this whole sea of nodes that are all shouting at each other and again fast content discovery so let's talk a little bit about private networks for some of you that are building on public platforms this may not be the best solution for you but for those of you that don't necessarily care if your content is retrievable via public gateways it's a good option for performance so how this works is that ipfs lets you make these things called private networks where basically all the nodes of the network have to have a private key that they had used to announce hey I have the permission to join this network and in these networks all nodes are only going to be connected to other nodes in that network so you're not going to connect to the main ipfs network and what this typically means is that there's going to be less nodes in the network you're not gonna unless you have a huge application you're probably not going to have hundreds of thousands of nodes that are all utilizing said private network maybe a couple hundred maybe a couple thousand even still that's very easily manageable by where ipfs is at right now which means a lot less noise and a lot quicker content discovery so you don't actually need to manually swarm connect your nodes when you utilize the private network all the nodes are just by nature are going to be connected to most other nodes in the network similar regard we're going to talk about what's called dedicated networks this is kind of a marketing term that we came up with for private ipfs networks that you've exposed the pub the private key for publicly we didn't want to call them private open networks that didn't seem fun so we got dedicated networks so similar in technical setup and benefits as the private network set up but like I said we publicly announce the key to get into this network and what this means is that we kind of have a network shard so if you are an application that is only dealing with up data for your dau or your scientific network or your music platform whatever it is you probably only care about data that's within your network you don't want to deal with the Dow's data if you're trying to serve music files or content files but you know you want to keep things a little sharted so what we can do is announce that that private key and then just by nature nodes aren't going to join that network unless they care about that data so you kind of self select your network into little tiny network shards and the cool thing about this is that nodes can switch back to the public network at any time IP FS is constantly getting better at content routing it's constantly getting better it's you know finding where and how to serve content so and if at any point in the future you want to switch back to the public network you can all you need to do is just get rid of that private key flip a flag and you're back on the public network and all of your data that you've accumulated on this dedicated network is still going to be there you haven't lost anything the data is the same the hashing algorithm is the same it's pretty compatible pretty easy to switch so it's kind of a temporary stopgap solution that we recommend to some companies that are really experiencing load when using IP FS so what this looks like in practice is we'll have something like the public IP FS to network with all the gateways that are connected to it and then you have little network shards that are technically private networks but anybody can join them so you know one for scientific data that you know a dhow maybe add X and then you can run multiple gateways on these networks as well the cool thing about IP FS is anybody can run a node anybody can run a gateway and that's what makes it be centralized you know you choose the data that you want to host which is inherently centralized but if multiple people are doing that and the access points for that data are decentralized it's what helps kind of distribute the content and that's you know what we encourage people to do so in summary let's compare swarm connect in it's the simplest set up your nodes are still participating in the global IP FS network but the speed improvements are only going to be realized by nodes that you manually connect to it's it's a pretty manual process to set up and it's less scalable Private Networks more complex of a setup and the network is only for his private it's only accessible by nodes in the network so this is good if you don't want other people seeing your data and you also want speed improvements it's a smaller network which means faster content discovery for all Network participants and then lastly dedicated networks again a more complex setup is this is basically the same thing as a private network but like I said we've exposed our private key so anybody can technically join it but only people that really care about the data in that dedicated network are likely to join it so it cuts down on the traffic and then again smaller network faster content discovery for everybody involved so thanks for listening to my talk if you guys have any questions feel free to reach out to us at pinata we'd be loved we'd love to help you with anything your building [Applause] [Music] you [Music]