Protocol Berg: Dennis Trautwein - Best of Both Worlds: Exploring the Role of Centralisation in IPFS

thank you so much and thanks everyone for joining my talk about yeah exploring the role of centralization in ipfs first a few words about me so I'm a research engineer at protocol labs and I'm at a in a team that's called probab and we're doing protocol benchmarking and proposing yeah doing measurements and using these measurements to uh propose protocol optimizations uh you can find me in all these with all these handles um all over the web and um me and my team we work closely with external collaborators on um the centralization aspects on ipfs and that's what I want to share today with you um to set the base uh I I will start with a brief overview of ipfs I believe many of you already have heard at least of ipfs but I will explain um just a few bits that we need for the later Parts um then I will go into the challenges of decentralizing ation ipfs so we identified three three challenges that I want to talk about then I want to propose or like I I want to present the designs that we developed at protocol apps to um tackle these challenges these are the hybrid components and finally I will yeah present the measurements that we did to actually assess if these hybrid components solved the challenges that we identified previously and then I will give give an Outlook of What's Happen happening um so first of all what is ipfs ipfs is a decentralized stor and delivery Network which Builds on peer-to-peer networking and content based addressing and ipfs is actually much more much more than you probably think so if you look at the specification of ipfs ipfs the ipfs stack is a a suite of specifications and tools that share two key characteristics which is content based addressing and transport agnostic so it's not just Kubo that probably most of you are familiar with but it's also Helia which is the JavaScript implementation of the ipfs protocol there's elastic ipfs Lotus and and many more that you can find uh on the documentation site there um what is ipfs in in stats so I will just talk about the Kubo Network that ipfs will bootstrap into um as I said there are also more networks like the Falcon Network and from that vantage point we identified 31,000 nodes um continuously participating in the public DHT 230 or like all of these are estimates around 230,000 unique DHD clients uh 2.8 million unique Gateway clients per day and 120 million requests that we see with our um D um with in our HTTP gateways which will also get into a little bit later and I want to emphasize that all of these numbers are just from our vantage point at protocol app so these are probably much higher so there's plenty of usage um of ipfs here um to understand the um components that I will go into in a second we briefly need to revisit how ipfs actually works so simplistically ipfs uses the hash of the content as its content identifier so you can see an example there and as opposed to http a location based protocol requests are for the actual content using that hash and we're not using the address of the host storing it this creates a problem most notably so where is the content for that hash and Trad there's like the traditional decentralized uh indexing option which is the cademia DHT and the cademia DHT basically indexes two types of Records one is the provider record which Maps a Content ID that you see up there to a peer identifier which looks roughly the same this doesn't bring us much and then it indexes also Pier records which Maps these perer IDs to actual Network addresses so we need to do two lookups in the DHT to actually find the hosts that are actually storing the content to retrieve it and when we want to make content available to the network we need to store these provider records in the DHT and for that we do it uh we need to contact 20 peers this is also relevant in in a few seconds so to make one piece of content available we need to at least store this provider record with 20 peers conversely when we want to look up that CID we need to find at least one of these peers to actually um figure out the network address where it's hosted and this is also the first challenge that we identified so if you're a content provider with a lot of data massive content publication becomes a big challenge so as I said we need to First identify the 20 closest peers in this cademia hash space and then store that record with them and because we need to do it for with 20 peers and do this lookup all over again when you're a content provider with a lot of Records um you you get into trouble very quickly as we can see here so this shows the publication time of such a record on the x-axis and then this CDF a cumulative fraction on the y- AIS and what we can see here that around 20% of Publications take more than 10 seconds so let's imagine you have terabytes of data you want to announce it to the DHT this is just taking ages and you need to refresh those records every 24 hours um to to keep the content alive and available in the network and this just doesn't stick so the first challenge is this massive content publication part then the other challenge is the the flip side so the content retrieval performance um again this has the uh lookup time for these records on the x-axis and again a cumulative fraction on the y-axis and uh we can see that at least in Europe and the US this these are the uh two lines on the left um from these three and uh we can see that most of the requests are actually resolved within half a second but as you see recall that we have provider and P record so we need to actually do two of these lookups which adds up and we are still in order at least one order of magnitude away of HTTP content retrieval times so content R Performance is our Challenge number two and challenge challenge number three is adoption um so what we what we found um with this uh at this Source at at the bottom here that um 50 of 58% of website traffic traffic actually originate from mobile clients and putting ipfs on mobile clients is not quite uh so doesn't work right now because of the resource footprint of ipfs so this is a big Challenge and as we can see here over 90% of Internet users actually access the internet uh using a mobile phone so how can we solve that problem this adoption challenge here and so prolapse explored some hybrid options so how can we so let's sacrific the decentralization aspect to like just a bit to gain uh massive performance improvements and at procps we developed interplanetary Network indexes on the top right that addresses the massive content publication challenge then for Content retrieval performance we developed Hydra boosters and for adoption especially on mobile clients uh we have deployed HTTP gateways which are all semi centralized uh components of the of the network and yeah I want to I want to briefly explain how each of them work so these interplanetary Network indexers are mainly designed to accelerate this massive content publication um it is a complementary component to the DHT it is in it also indexes provider records and you can think of it as a high performance key value store and the provider records are a bit different recall provider records in the DHT map pids sorry CS to P IDs but in this network indexer they also contain the network addresses already and the retrieval protocol in the case of ipfs this is bit swap but it could also be for example graph sync so you do one request in the case of um the pl operated c. contact indexer UM to/ c/ your CID that you want to look up and then you receive the record which includes the network addresses where the CID is hosted you can connect to that host and retrieve it and retrieve the data um the talk right after me from masi will get will get into much more detail in how uh the ne indexes work and the way forward and so on so I would highly recommend that you stick around after this talk then Hydra boosters are a new type of DHD server that is designed to accelerate content routing um so one Hydra booster consists of thousands of virtual Hydra heads where all of these Hydra heads are connected to the same the same common underlying database that stores all the provider of gods and the trick here is that these Hydro heads are strategically placed into the DHT in the um in the DHT key space so that each provider record storage request so I recall we are storing it with 20 peers that there's they placed strategically such that at least one of these requests one of these 20 peers will always be one of these these Hydra heads which means that the Hydra knows all of the provider records of the network and then when later in the game someone wants to look up that provider record and uh because there are so many hydr heads it is very likely that you hit one of these Hydra heads right away and even if it's far away in the key space it can immediately share this or serve this provider record because it has the common database underneath and the HTTP gateways um it's designed to improve adoption of ipfs and basically meant as a bridge between uh H the HTTP world and the ipfs world and it is bundled within Kubo so if you ever run the Kubo demon you will find uh at the end of the lock lock output that there's a Gateway running on Port 880 I believe and this has a similarly simple um HTTP API as you can see here so you just uh go to in this in the case of proc collapse we are hosting one of these gateways at IP s iio you go to ipfs io/ ipfs SL your C ID but instead of being served the provider record you get served the actual content so the the the Gateway will go out into the network and fetch it and this has a couple of advantages for example the aggregation of demand in these centralized uh gateways allows and the the concept of content addressing allows for very efficient and aggressive uh caching of the content and there are actually plenty of of um gateways deployed and you can see it on the right there's the uh a list of public gateways that you can use but I think IVF siio is the uh most important one so these are the three components for these three challenges um so how does it uh how do they actually perform so do they actually solve what they were set out to solve and for this we did a measurement campaign and we measured all of these different components and I want to share some some numbers here um for for example the indexers store around 100 times or two two ERS of magnitudes more provider records than the DHT and the top providers the top single providers in in each uh in the indexer actually um provide or offer more than 100 or a thousand times more records um so you can you could argue okay so the index are basically dwarfs the the DHT in terms of indexing capability but it's actually a little more tricky because um the the indexer itself only uh has indexed content from around 600 to 700 providers I'm not sure this is the most up to-date number but this is the number I have so 10 10 to the two probably and the DHT hosts um provider records from over 50,000 providers so it's like we have more providers that serve or offer less content in the DHT and in ipni we have less providers that uh serve more content so I would it serves so it it solves the problem for these massive content uh content hosters very efficiently here and each advertisement which is a concept of uh of the of ipni that probably Mai will go into later as well each advertisement con contains around 10 to the four cids and so you can do the maths and uh each like indexing each provider record takes around 500 mill micros seconds and compare this with the 10 plus seconds of a single provider record in the DHT so the this is like a massive speed up for these types of uh content hosters um on the content retrieval side um the so I mapped the content retrieval performance challenge to Hydra boosters but actually ipni also helps with that because of uh the the centralized nature um here we have like an example of content retrial times compare compared in different region between the DHT and this network indexers and here we can see that in for for example in the uh AP Southeast 2 case um DHT lookups take around 1 and a half seconds the uncashed ipni I case is around 300 milliseconds and if it's cached at the edge with um I don't know with the CDN or I think it's cloudfront in this case you can get it down to less than 10 10 millisecond lookup times for for provider record so a massive speed up here as well um in the case of Hydra boosters we did an experiment uh so we did a bunch of DHT lookups from different Vantage Point points all around the world and in one case we had this common Hydra database enabled and in the other case we have um we have ignored Hydra boosters uh at all um and we found that the the lookup times uh uh increased obviously um but not as much as we expected you need to you need to think about when these Hydro boosters were designed the lookups times were not around 1 seconds or slightly less than one second the lookup times were in the order of multiple seconds so uh maybe maybe tens of seconds um but now we find okay Hydro boosters actually uh they increase the performance or improve the performance but not so much anymore then in terms of adoption um for us the the Gateway are the main mechanism to improve adoption and we looked at um at the Gateway adoption um from a client client perspective and what we find what's also on the right here um this shows the traffic requests um and and and users of of our gateways in different different metrics we find that 32% of the total traffic and 90% of total requests come from mobile devices which indicate great usage on mobile which was previously just cut out of the ipfs ecosystem and 33 33% of total traffic and 30% of total requests originating from desktops and we also have like net tools Cate category which includes things like w get which is also surprisingly High uh in this case um in terms of web developer adoption we find that um three times fewer requests um originate to or like reach the HTP gateways without a referral which means or which we interpret as this request doesn't originate from from from a website so this request is not so so this request doesn't come from a website which means that three times more requests come from websites that have actually embedded HTTP crl or use ipfs in their uh yeah in in their website setup so it it seems like web developer make use of H these HTTP gateways to ease adoption of the ipfs network but at the same time we also find that the top 23% of website referers make up over 70% of the requests so a huge uh like yeah some kind of centralization here you could argue um obviously with the deployment of these tools there come a couple of trade-offs and I want to just highlight a few um for for example for the interplanetary network indexers um a failure of these indexes could render sign significant volumes of the provider records inaccessible which is a problem um however there are seven indexes available right now and protoc claps is working on a federation protocol so that these ipn can actually share these advertisements between each other which will also Mari explain I I believe in the in his next talk then on the hydroa side um because we are deploying like thousands of virtual hats in the DHT um it has massive influence over the routing which could re um enable result poisoning so it could just serve bogus records to to to users and it's also poses the risk of eclipse attacks um but we figured that this is actually quite difficult to and the worst thing that can happen is that um you have slow retrievals but it cannot compromise availability of Records so it's actually not that bad on the HTTP Gateway side we s by using it we sacrifice endtoend verification of cids so when your browser requests a CID from a an from an HTTP Gateway you're not doing the CID verification of this content that actually got served to you so it's not matched against the the CID but we are also working on like a on service workers that inter accept these HTP requests and uh that do the verification on the client side as well but um for for me personally the the biggest risk is the compromised privacy so um operators that deploy these kinds of systems have actually a big view into usage patterns and so on and this asymmetry of power is like a key concern when deploying and operating uh these kinds of systems and there are actually like this the centralization versus decentralization discuss is like it it's a spectrum and we haven't touched on all these different other parameters here as well so this is just a glimpse in in in what uh what we can actually uh yeah so so the problem space is is huge actually and this is just the the technical side here that that I discussed but actually there's one more thing because we have actually shut down the hyra boosters you remember that graph on the bottom left I mentioned that we figured the Improvement that Hydro boosters bring are actually not that big anymore for the monetary for one for the monetary cost that they incured for protocol apps but also with all these different um privacy concerns that I that I mentioned so at the end of November last year that you can see on the bottom right which shows the retrieval times as a rolling average um we cut or we shut down the Hydra database which stored the provider records it still participated in peer routing so serving peer records but it didn't serve any provider records anymore more and what we saw is this massive spike in latency and then it leveled up in a uh so so then then we found another plateau and what we uh what you can think of is This difference here between these two lines is roughly the performance hit that we incure by shutting down Hydra boosters so in my opinion this is like a this is exactly the success story so we're building is is like we're helping a centralized as we're building and deploying a centralized semi centralized system the um ecosystem evolves and is at in at a later point in time in a stage where we can actually shut down and remove the centralized system so that the decentralized system can stand on its own so but what happened here this is a different story and I highly recommend reading this blog post from my colleague yanis and uh it's called what happens when half of the network is down so right at the same time when we shut down the hydro boosters we identified that actually over 60% of the nodes in the ipfs network were unreachable and we haven't even noticed so this is like a huge advantage and like a great success story for decentralization because imagine any centralized servers if 60% of the infrastructure is down you would definitely notice service interruptions and so on and the only thing that we noticed is it degraded retrieval performance and uh yeah this this blog post gets goes into much more detail and uh highly recommend reading reading that so let's conclude um in my opinion these measurements demonstrate Effectiveness in addressing these decentralization challenges it's still up to the developers to um balance these trade-offs because all of these components are optional um the Hydra boosters are proof for a viable path that we are taking we compromise decentralization to drive adoption and revert as soon as the ecosystem is ready for um living without decentralized components and yeah you can find more of this data and performance information and so on um in our weekly reports which we publish at stats. i. Network um more comprehensive information at pro.

our website where we publish um daily uh measurement results all the code for the tools are available at PL probab organization at GitHub and yeah that's it thank you so much cool yeah we have time for maybe one maybe two small questions um doesn't the fact that the indexers have uh orders of magnitude more uh Records uh despite having uh uh two orders of magnitude fewer participating nodes uh doesn't that mean that DHT is essentially not capable of supporting this data set especially we're talking about indexing individual blocks from filecoin right yeah that's I I would say so that the orders of magnitude that these massive content providers um have of more data um this right now cannot be indexed in the DHT and so I would consider these indexes again as a bridge until something something better better comes along and uh yeah so I yes the answer is yes yeah okay we can take one more question over there all the way in the book thank you um I'm curious Dennis I'm not a planted question I promise um if say I'm some other network other than ipfs that's building on top of some of the similar protocols like other lib P2P um based networks like ethereum polka dot Etc can I use any of the interesting analytics that you guys are creating at Pro blab um or there any particular areas that I should be paying attention or following along to analyze overall node distribution or um you know other transaction volume that might be useful I mean we are um so one of the tools that we we've built is like like a network crwler which um doesn't doesn't only support ipf like the amino DHD that how we call it right now the ipfs DHD but also um supports crawling the the po. network which could give insights into agent versions distributions and and uh uptake of new agent versions and so on which extends to also kusama and other substrate networks um so this is one one area um so if you're interested in that data and these and probably tailored measurements for your network just uh yeah reach out to us and we're happy to deploy these systems yeah to your network cool thanks so much yeah thank you