Raúl Kripalani | libp2p: Modular peer-to-peer networking stack | CrossLink Taipei 2019

thank you for being here my name is Rob Rob Lonnie for the Libby TB project from protocol labs it's super exciting to see so many people in the room that are actually interested in peer-to-peer networking this is as I said yesterday in the panel I don't know if you heard it it's traditionally been kind of like a niche area of Technology which as we continue building new layers of decentralized systems and creating creating stacks up on stacks upon stacks we are realizing that the foundation of everything is really the core the networking if we don't have a strong rock-solid and networking layer that is resistant to all kinds of attacks that is that is extensible that is modular that is future-proof and so on we really have nothing right so this is what I'm gonna talk about today I'm gonna talk about the Libby to be project who here has has actually used the Libby to be project have you who is a user of Libby to be alright we have what this is amazing so this is a testament of what I was saying right it's it's kind of like peer-to-peer networking and I know that I have more people over here who haven't today but this is kind of like a testament of what I was saying right it is peer-to-peer networking tends to be so invisible to the naked eye that very few people actually dig their hands deep into it who has heard about Libby to be before okay that's all right so let's get started with this I have a lot of information to share with you today so I'm gonna do like a very high bandwidth high throughput delivery of this if you if something is not clear feel free to find me later so let me tip it so what I'm what we're gonna go through today this is the agenda for today I'm gonna do a quick introduction to Libby to be that we're gonna then I'm gonna walk you through the different elements of the stack the different layers of the Libby to be stack so that's addressing an identity transports connection bootstrapping secure channels multiplexing protocol negotiation peer discovery content routing absorb connectivity and we're gonna talk about the future actually we're gonna talk about the future all throughout so whenever you see this little crystal ball that means that it's a feature that is projected for the future and hopefully the future will come soon for many of these things all right so what is Libby to be so Libby to be is a modular peer-to-peer networking stack I like to explain it to people as a set of composable building blocks that application developers platform developers design entire systems developers can assemble to create the right stack for the right networking step for for what they need it is really put to focus on this composability aspect on the extensibility aspect and of course on future proofing because one problem with current networks is in the current networking system and and in general the current networking architecture of the Internet is that it's unfortunately very ossified so this you this means that you know decisions taken 20 years ago without the understanding of what the internet would turn out to be in the future and how important it would be to humanity are actually affecting us very greatly today so Libby topi puts a focus on future proofing it runs on many runtimes server browser mobile environments where there have been experiments and soon there was a lipid 2p implementation that is going to be very suited for embedded devices Libby to be originally originated in the IB FS project who has heard about a BFS here fantastic okay so Libby 2p was the networking stack of the ipfs project that protocol labs decided to extract for my PFS and turn it into a general-purpose networking stack for other projects to build upon it's implemented in seven languages and has Karelia bustling and a flurry of activity that's just that just keeps increasing in the community it was created and stewarded by by by protocol labs but it's really we think of it as a public good it's owned by the community it's currently licensed under the MIT license but it's soon moving to the permissive license tag which is a combination of the Apache Software license and the MIT license to avoid any any uncertainty with regards with patents and so on so as I said I like to explain let P 2p as a set of composable building blocks and these are the ones I'm not going to go over those again because I already did in the agenda but you can see that you know it's like kind of like a Lego passel of different pieces that fit together to build the right networking stack so who's actually using Libby to be of course ipfs polka dot aetherium 2.0 you've probably heard about this in several presentations already is using is using Limpy 2p file coin metamath scope and bizarre 0x and so on and there are a number of projects that are considering Libby to be as well like cosmos the coda protocol and matrix who is experimenting wouldn't it be to be so I'm pretty sure that everybody in this room would have heard about several of these projects what's cool about Libby 2p is that especially about having these many adopters is that these are at these are active people who are getting involved actively in building the future of Libby 2p so really when you look at the networking stacks of all of these projects separately and you put like a magnifying glass on on each of these they look a lot alike right and we'll talk about this in a second by going through what is actually needed to make peer-to-peer networking happen right so it's great that you know all of these projects are unifying efforts and really building a foundation that works for everybody right and pulling resources there's no point reinventing the wheel everywhere right so this is what Libby to be aspires to be as well so as I said Libby to be has currently seven native implementations we've got Jas go Russ pilot b2b I think we might have some of the pilot b2b implementers in the room if Alex Stokes is here and Kevin and she'll way they all have contributed to pilot p2p we've got JVM Lapita P by the way Russ let be to me we've got one of the contributors to Russ maybe to be here as well so you know it's a bustling community that just keeps growing and what I'm really excited about is particularly several of these implementations here the JVM the p2p implementation is going to be the base for Android and for Android specialized chassis for mobile special specialization of libido P and is gonna be specially workable for for embedded devices so much of the traction of the BtoB is coming from a theorem 2.0 adopting will it be to be for the networking layer so I don't know if you've if you've seen this back before but here is in case you wanna you want to search for it it's under the earth 2.0 specs repo you'll get to see how their b2b is actually being used in a three-room 2.0 and as I said so if hyrum 2.

0 is a multi-client effort where these plans are implemented in different languages and all of the teams that are implementing eath 2.0 are also contributing actively to the lip Adobe ecosystem which is amazing right so one of the philosophies of lebih to be is this thing that is called decentralized process address right so if you think about the current internet the it revolves around the concept of location addressing right so it's always an IP address that's always a port there's always a protocol and you always hit a process that's sitting on a machine of which you have the locator a decentralized process addressing kind of turns that around and the the reason to be of Lapita P is really or kind of like the guiding star the North Star of Libby 2p is the ability to locate connect authentically negotiate and interact efficiently with any process in the world no matter what the runtime is right so it could be server browser IT and so on so long as its identity is cryptographically derived from its public key and Libby to be really in strives to make all of this happen in a seamless in a seamless manner this is not the current case right now there but this is what we want to get to by making all of not real a packet switching and a number of things transparent to you such that as those process relocate and roam around the world there you'll still be able to address them by process ID now how does this so let's start getting into into the technical details of maybe to be the first thing kind of like the block the building block like one of the main building blocks of everything is addressing how do we actually address these process in the world there is a concept of multi adders which are essentially impossible future-proof i don't know if you if you get to see the coloring on these slides unfortunately it might be a bit difficult to see but a multi adder is a composable future-proof upgradable transport agnostic address so if you think about how we address processes in the world we usually have an IP address support you need to know if that the thing that's running behind is using TCP it's using UDP right if in the future you wanted to communicate with the same process over a different transport like WebSockets or Bluetooth or you know you name it you would have to modify your protocol if because bluetooth doesn't have IP addresses right it has Mac MAC addresses the WebSockets for example has ports right so it's very has context buts right and a number of things so you would need to modify your protocol how do you avoid this multi adders is kind of one way of doing this basically what it does is it creates a binary so you replace all of these IP addresses ports and all of these like granular things with just one byte string right and that byte string is extensible and it is it represents exactly the it encodes the different aspects of how you address that particular process in the world so the IP address the TCP what protocol what protocols hurry what network protocol you're gonna speak with that where that process the port number its identity and so on and multi adders are normally represented in this string notation but when they are part of messages and protocols they are really byte packed so really hinging on this in this idea of multi adders and the hierarchical system for a naming process in the world in the world who here has heard about plan nine as an operating system cool a few people mostly the protocol developers that screen so plan nine is is an operating system that was developed by by Bell Labs and I had this really cool concept of and this core notion that everything is a file so you can imagine the hierarchical nature of multi adders being able like if you were able to mount the hiya world right that is addressable via multi adders onto your file system and have kind of like a demon such that whenever you open a file descriptor or you write or you read from a file descriptor behind the scenes is actually using lip p2p to make that connection happen that is kind of like a very cool idea it says this is really the inspiration behind you know the nature of of multi iris ideally I would love to be able to mount the entire peer-to-peer worlds Network on my on my laptop and be able to interact with it via simplify descriptors so a key concept within the Libbie to be networking stack is pure identity and pure identity and peer IDs are derived from public keys and Libby to be supports for different types of public keys there is nothing preventing you from adding a fifth of sixth seventh type for of cryptographic keys of key types but of course you will need to do that across all implementations because otherwise they might you break into her ability now the way that a peer ID is forum it does essentially there is a little protobuf struct where you which you inform with a key type and the in the and the pop key and you hash that by a something that's called multi hashes which are future-proof hashes that basically prepends the type of the hash how that what the hash function was when was a digest function to generate the the the value was just to make it self describing and normally we we indicate peer IDs and multi adders so whenever you address a process in the world you also indicate what the pier ID is of that process now when I talked I talked about you know making connections actually happen right so how does this how does the Libby 2p stack really create those connections it has the concept of transports and transports are basically a core abstraction and maybe to be you should really think about them as connection factories right so they are capable of dialing an address which is the term that we use to connect outbound to appear and listening for inbound connections right so a transport actually produces real connections it's kind of like the layer four of of the BTB and currently we have four transports with different levels of maturity TCP quake WebSockets WebRTC and in the future coming to a node and here you would be having a really stable stable and ubiquitous quick which is important especially when you combine so has anybody heard of HTTP three oh so if you've heard of it you will know that HTTP 3 basically uses quick as the underlying transport and uses TLS 1.3 as the security Channel so by making by making quick and TLS 1.3 pretty predominant in the lipid open Network and potentially making those processes run on port 443 we essentially get a pretty good means of censorship protection right because all traffic essentially looks like HTTP 3 I mean censorship protection in terms of censorship resistance in terms of in terms of deep packet inspection now transports other transports that are coming in the future bluetooth by our friends at both Bertie Bertie is a company that's developing an offline first chat platform and they've created a Bluetooth transport for Golda p2p which is amazing UTP we're working on a message orientation as well and what's cool about the multi adder concept is that as you create and as you add support for new transports into a Libby to be stack they just get represented as one token in the multi adder if the peer supports it if the current implementation supports it it would be able to dial and listen if it doesn't well it probably has a lot more transports that they can use anyway so how does so imagine you've got a multi adder right and this represents appear somewhere in the world that you want to interact with how does the dialing process actually work so first thing that happens is so if you look at this multi adder it's basically saying hey I want to contact the process in IP address 1 point 1 2 3 4 at this port number I'm using TCP and this is their peer identity this is their this is the pure ID now the way that this happens is we first use a transport the TCB transport to create that connection the TCP transport gives us a raw connection which we then upgrade into what we call a capable connection right so a capable connection is one that's authenticated that's encrypted and multiplexed now we're gonna talk about all of these things in a second there might be like very foreign terms for you if you haven't heard them before but we're gonna go through this so the first thing that we do is negotiate the signal we negotiate on what secure channel we're gonna use with that peer and then we negotiate a multiplexer dynamically this is important because it makes the whole system future-proof if we find a bug if we find an issue if you know a particular security protocol is broken if you hard code the usage of a particular security protocol a secure or a security handshake then you've also had the protocol already right so making that change and appear to be your network is super difficult appear to be your network you don't control the network right so by definition so like 101 deny the founder of vertical labs once said you know peer-to-peer network is one not only not only a network that you don't control but it's gonna actively try to attack you in all possible matters right so you gotta make sure that every aspect every step of the way is dynamic enough so that it can be change in the future if it needs to change right so what's important to say here as well is that some transports don't actually need upgrading and quick as an example of this if you've heard of quake then you will know that it's multiplexing encrypted by default so the we can avoid making this upgrade because the transport already gives it to us this is kind of like a diagram of sequence diagram of how this happens in reality with multi stream select 1.0 I'm gonna talk about the future now if you want to read stuff like this and you're interested in like understanding the nitty-gritty details of how this happens then go to the Lippe to be specs repo and you'll find all of all of that information there I'm proud to say that it is in a much better state than four or five months ago where it was pretty in some cases on document it so yeah so ok one of the one of the things in this process is the security channels right so a security channel lmp2 P provides two things it provides pure authentication and it provides transport encryption right and well other things like transmission transmission integrity non-repudiation and so on a different security guarantees it depends on the it depends on the protocol all implementations supported as something that's called se Cayo and some implementations supported TLS 1.

3 and there is also a secure channel called plaintext for testing well you should never run a plaintext channel in production so but this is useful if you combine it with something like Wireshark and you want to see what's actually happening on the wire as your Libby to be notice running okay so those who those are the current security channels and what's coming down the pike are two things that I'm super excited about we've got wide support for TLS 1.3 there are some roadblocks and some implementations like rust and Jas have a few roadblocks TLS 1.3 is available 100% angola p2p still hasn't been rolled out to a live network so still needs to undergo considerable testing and another one that I'm very excited about is noise handshakes which who here has heard about the noise protocol framework protocol differs again excellent so the way that we've adopted noise is pretty interesting because we were using something called noise bytes I'm not gonna go into it but I will say that noise handshakes are a requirement for aetherium 2.0 maintenance so we're definitely collaborating and there's other actors in the space like cosmos who are also interested in this alright so I mentioned this term that might have sounded a bit alien to some people which is multiplexing okay so the first thing that you need to know about a peer-to-peer network is that establishing a connection with a peer is difficult expensive and negotiation negotiating what that peer is it's not is not easy it so we wonder we want to amortize the cost of establishing a connection right which means that if I've got a connection I want to talk different protocols without peer different different applications for example or different different I want to have different conversations with that peer parallel conversations I don't want to establish a connection for each because I don't want to undergo the cost of each so what I do is use something called multiplexing which basically here's the really nice diagram of how multiplexing actually works so you can imagine a connection a single connection opens up and becomes a strand of multiple parallel conversations right which are isolated from one another and this is what multiplexing gives you we've got four implementations of multiple of the multiplexing layer yam exam Plex and speedy stream and quicks native multiplexing and in the future we are considering adding support for something called multiplexing for those applications that don't actually need the multiplexing support we will not we don't need to negotiate a multiplexer and I don't know what happened there okay all right so another thing that's interesting as well that I wanted to discuss was protocol negotiation so protocol negotiation and so once you establish a stream will appear once you establish one of these conversations with a peer you want to agree with them what is the thing that you're gonna talk about right and that is what we call a protocol you negotiate a protocol would appear right now we use multi stream select 1.0 which is a interactive chatty speculative and we're going to be moving soon to multi-select 2.

0 which gives us a lot of cool features fancy features such as one round-trip connection establishment right now establishing a connection with a peer it's kind of a bit interactive and chatty so would be would be moving to the scheme in probably a few months from now it's gonna it's gonna give us as well cheap repetitive protocol selection upfront negotiation session resumption and a number of things that we're specifying in this in this issue and this board request if you want to if you're interested in in digging deep into that so these each of these conversations that peers have with each other are identified by protocol IDs so there's protocols like flop sub floods a bit swap and so on normally they have a protocol ID is just a straight right it doesn't actually need to follow a particular format and you can think about it as breast endpoints right so when I establish a connection would appear I want to address something some functionality in that peer and that functionality is identified by by a protocol idea that color choice was really bad for for this particular room right so we've talked about up until now we've talked about two peers establishing a connection to to one another but you have to realize that and appear to be your network there is no central authority there is no central directory of peers you don't know who's out there right so you need to discover who's out there ellaby to P provides an abstraction called peer discovery and it's basically a very simple interface that can be backed by many implementations so one implementation of the center face is a DHD which is you know a very widely used concept in in peer-to-peer networking for peer discovery we can also it can also be backed by rendezvous nodes it can be backed by things like mdns for local peer discovery and in the future we'll be able to use bluetooth and fs and even hybrids of these systems because one of the problems of Discovery Networks has always a recurring problem is bootstrapping right so how do I get the first seed of peers to actually be able to interact with the rest of the network and there are some interesting approaches that can be done on that layer by using hybrid by using hybrid approaches cool so I won't go very deep into this content routing is something that the Libby 2p provide the Libby to be stack also provides is also a pluggable interface which is normally normally implemented by a DHD right so content routing gives you the ability essentially of locating content address pieces of data in the world another another functionality of the Lapita P stack is as pops up so pops up is a message oriented communication pattern which is in contrast to request reply communication patterns in pub/sub there are publishers of messages that are sending messages pertaining to a particular subject or a particular matter and that matter is normally called a topic so publishers are publishing messages on topics and there are subscribers around the world that are wanting to receive those messages and it's basically an m2n communication pattern and it is inherently a synchronous and decoupled in nature there are no timing conditions when a publisher sends a message it doesn't know who is listening for that message who it's gonna arrive to it just knows that it submitted a piece of information right so we have a number of property so peer-to-peer pops up is a difficult problem you can imagine in centralized systems and an enterprise systems pops are based a very widely used approach so if you view something like ActiveMQ Kafka or whatever then you you will have been familiar with what pops up is but in peer-to-peer networking it is orders of magnitude more difficult to create a attack resistance buffs of network and there are a number of characteristics I won't go into it but if you're really interested in this then come find me the Liberty to be stack provides two different routers for now for pops up we've got floods hub which is a very naive router and we've got gossip server which is the thing that lived at aetherium 2.0 is building on and gossip sub provides a layer base layer in primitives to build adaptive pops of networks adaptive and self optimizing your Brazilian networks it is not a solution by itself it is a building block it is a primitive to build intelligent routing mechanisms like episode so episode is one that I'm very excited about it's a combination of three different approaches that are part of the that exists in the state of the art but it's a very particular combination that I think it's going to give very good characteristics as we go forward right so another and the thing that the Libby to be stack provides is relaying so in many cases you will have two peers around the world will not be able to immediately interact with one another because one of them is sitting behind a nap right so circuit relaying is a feature that allows two peers to interact with one another by a third peer in a secure fashion right and this is a picture from my BFS camp in Barcelona a few months ago and that's kumoi v's and people from the from the meta mas team probably have seen them before and they basically got this vibe around the neck and kind of like portraying what relaying looks like it was a very cool exercise now relaying by itself is costly it's ineffective and you don't want all your traffic to go through a third party provider even if that third party is not able to see the contents of those messages right it's very inefficient and it creates centralized pockets of communication in the in the network so really what we want to what we want to get to is something called decentralized hole punching I'll talk about this in a second but before that just wanted to say that Libby 2p provides two features alternate in order to relay that basically alternate sensors weather pier weather or node is running behind a nut and if it does it activates a the relay sub system automatically so the relay subsystem will look for peers in the network that are acting as relays it will connect to those beers bind with those beers and then it will start publishing related addresses for the peer that is sitting behind a NAT so that will essentially make that peer connectible from the outside world now I talked about decentralized hole punching the whole idea and I have one minute and a half left I think I think I'll make it I did say that it was gonna be high speed and high battery than high throughput so there you got it so hole punching is very exciting this is currently going undergoing active implementation hole punching is the idea that there are certain mechanisms that piers can use to pierce through now to burst through nuts right to create port openings in knots relaying traffic relaying all of the traffic between two piers is very inefficient so instead what we want to do is when we establish that real a connection between those between two piers we want to use that as an embryonic connection to start signaling for hole punching so when you establish a relay connection both piers will start a synchronization procedure to decide when is the right time to start sending messages very like appropriately crafted messages to try to break through their corresponding nuts to establish a direct connection and if that thing succeeds if establishing a direct connection succeeds well it b2b does well the b2b is gonna do is transplant the connection state into that new underlying direct connection between those piers okay so let's talk a little bit more about the future of already discussed I've already provided nuggets about what's coming down the pike but I just wanted to make a list of stuff that I'm very excited about so one of our main focuses right now is distributed reproduce herbal testing and we're aiming we're shooting for the stars here and we're aiming to test workloads on clusters of up to 100,000 nodes right this is super difficult it is a very difficult problem that from what I understand hasn't really been attempted or hasn't been really had like long-term success in terms of being used actively and integrated in a continuous integration and development process so this is something that we're focusing on we're focusing on things like conformance test kits better specs network visualization improving the DHD as I said not whole punching episode and many more things I'm just wanted to provide you a bit of insight into the test ground effort which is the testing platform that I discussed this is just an architectural overview we're now gonna go into this but if you're interested in in this afraid and what's happening you can check out I BFS test ground the a BFS that's one gripe oh this is probably going to move to the Libby to be rep to their Libby to be organization yeah and another thing that we're working on is also instrumentation so we want to provide users with the ability to x-ray into what Libby teepees doing under the hood because a networking stack is very abstract is very difficult to understand what's going on under the hood and giving visibility into that and observability is crucial at this point especially as you know more and more projects are relying on Libby to be for their maintenance so the way that we're going about this is first we're creating a standard introspection protocol that can be used that can be implemented by any Libby to be implementation we're taking a data for the story until approach here so that means instead of like this is these are mock-ups of what we ideally want you know this this this console or the platform the the web console to look like but we're taking a data first oriented approach with me which means hey the first step is to make Libby to be implementations actually emit fire hoses of data of everything that's happening under the hood right once once we got that we are gonna be we are creating a set of libraries that can consume that firehose of data on a browser and populate a local state a reactive state such that d3 react based d3 visualizations will be able to hook up to it and be driven by all of that firehose of activity that's happening within an Libby to be node of course another thing that we'll be able to do is record Libby to be activity a firehose of avila p2p activity and then reproduce it and play it over and over again in a browser as well so I think that's pretty much everything that I had for today yeah I just wanted to say that will it be to be you everybody here and if you want to get involved in shaping the future hacking on meaty things or contributing then you have all of these venues and that's also a def grants repo for stuff that we want to do but we don't have the capacity to do and then the p2p team and we're a we're willing to incentivize the community to work on these things these are my contact details so if you want to hit me find me around or follow me on Twitter or email me thank you very much [Applause] you