devp2p Network Monitoring - Eleazar (E.G.) Galano, Infura - GoerliCon 0|9

all right what's up what's up hey how's it going I mean you look like you don't need any introduction because you already everyone is just watching you so why don't you take it over well but big round of applause for him though oh look at him he's blushing all right one second let me share my screen there we go hey can you guys see my screen yes all right so everybody I'm eg within Fira really awesome that Gourley khan's going on I wish I could be there in person but actually yeah so today I wanted to talk to you guys a bit about dev p2p monitoring I'm really excited about Gourley Khan really excited that this is going on a quick word about obviously I worked for in fear and when they're the co-founders of in Fira and when I first found out about Gourley Khan at eath Berlin I thought it was a fantastic project and I wanted to be involved so first of all thank you to everybody that created Gourley and is involved in pushing it forward I think one of the things that we've found over the last few months is the value of tests nets so knowing that with the Constantinople fork and the issues that we had on the test net on Rosten we saw the value of a cross client test net and we saw the value that developers needed with having a stable environment in which to test their applications and how that's also distinct from the test environment where protocol developers need to be able to test consensus changes so I'm really excited about the Gourley initiative and really happy to be presenting today so today I wanted to talk a bit about our dev p2p Network there's there's so much attention that's paid in our space to you you the economic incentives and laertes scaling and the stuff that's going on with ET point now that we really take for granted what what I like to consider later zero of all of this and that's the Deaf p2p layer the block chains are distributed networks and peer-to-peer networks and so it really deserves a little bit more attention so today I want to talk about the current state of our deaf p2p Network and a tool that I developed in my spare time that I I'm open sourcing today to help people understand the insights that I've found over the last couple of months playing around with this so let's start with a quick recap of deaf p2p for anybody that's not familiar it's the peer-to-peer layer that allows all of our etherium nodes to connect without there being a central point of failure it's a distributed hash table condemned only a based routing table so this is pretty much the way that you know most here the current peer-to-peer networks work for discovering other appears to connect to you and sharing information and it's EDP based which will be relevant a little bit later in my talk and but it's not just EDP the deaf p2p protocol is a mix of both UDP and TCP so to connect to the network you need to connect to a blue tab these blued nodes are hard-coded into the client implementations so if you go looking the source code of parody of death of pantheon of any of the other clients that are out there you're you're going to find a hard-coded list and this hard-coded list is the list the initial was maintained by the foundation and later on other people started to add some nodes to this list of these known points from which you can discover other peers on the network you don't need to use these nodes as long as you know a note that's on this network you can connect to that one and start the peer discovery the the nearby peers to connect to you messages the that's going to give you additional peers to try to get information from so in depth p2p that's the fine neighbors message and that's something that I ended up using a lot for basically crawling this network and trying to understand the health of the peers that are out there discovery is distinct from the eath sub protocol so that's what I was talking about earlier with the UDP part of the protocol and the TCP part of the protocol UDP is only used for discovery so that's the cadonia based discovery his ping pong like we bond our connection together after that and then we're allowed to ask each other about our are different peers that were aware of that's all udp-based and then as soon as you discover this peering list you switch over to a TCP based protocol which is actually the etherium sub protocol that's the way that you sync information from nodes the way that you transfer information about transactions and the way that you propagate block information so if we're talking about what we currently know about the network the the things that come up pretty often are how many nodes are there on the network that's generally been seen as a metric for the health of a peer-to-peer network is how large is it so typically our network has ranged anywhere from you know a few thousand five thousand eight thousand all the way up to over thirty thousand last year right now I think we're sitting around somewhere in the ten thousand range how many of them are running a specific version of a client so before people were just curious about how many people were running Jeff verses parody where people still running a theory MJ and and then it became more recently what versions are people running after he put out the great tool earlier to see how many people were connected to the or how many people were updated in time for the fork or the attempted for how many people are using white clients and how many white clients servers are out there one of the things that I discovered recently is that there's a it's it's still difficult to get certain like client implementations connected because they there aren't enough peers that are serving like client data I'll get into that a little bit more later so one of the reasons that we should care is that this is this is something that could potentially help us discover attack vectors such as eclipse attacks if you're not familiar with the Eclipse attack that was reported a yeah it was a couple years ago now it was a way of basically surrounding appear with malicious peers and in feeding it false information in a way to eclipse it from from the rest of the network and give it its own view or malicious view of information knowing the the size of the network and the number of peers can can potentially help discover things like this denial of service mitigation so recently another attack vector was discovered where it wasn't a direct attack vector it was more of ulnar ability where the EDP based protocol could be used to create a an IP spoofing attack so I I can craft a a UDP packet that says give me all of your you know peer information and this is my IP address and instead it's an IP address of you know a crypto currency exchange or something like that and and now I've weaponized the peer-to-peer network to deliver tons of data to you a target of my choosing right so the having a collection of this peer-to-peer data will allow us to see okay what version of the client was found that will is vulnerable to this how many of them are still running right so that we can get an idea of the severity of a specific will Nergal II that's recorded so the current view into the network is primarily ether nodes Ethernet has been a great service that that's been running longer than I've been part of eek system over three years and it's been the way that you could quickly get a glimpse of you know what's the client breakdown the client versions the operating systems I I wanted to try the run one myself and maybe I just didn't know the right place to look but I couldn't find the source code for it for Ethernet and so I said well let's let's try writing one myself so so I started out trying to build a dev p2p indexer I decided to build it on top of a theory MJS dev p2p trying to look at the implementations of the dev p2p protocol across Python going rust a few other ones this seemed like the the easiest one to take and hack on pretty quickly just because of the modularity if you know node modules and JavaScript so took this and started building on top of it and the first thing I needed to know is how do I discover these nodes I start with the blue nodes and then I'm gonna get some Kear's and then what so I decided okay let's start with that we're going to write a simple client to you as these new peers come in I'm going to ask them for their peers and then just use traditional distribute the hash table crawling techniques and so that's you randomize your fine neighbor lookups so the way that be lookup calls work is that you basically give it an address and you're going to be looking for peers within a range of this address the the specific serving that specifics of that is an XOR of the enid address so your your publicly public key that represents the address of your aetherium node it is basically the start of where you're going to try to discover neighbors and if you randomized that when you hit these peers you're going to get different lists of information and then and then you can wait that's pretty much the way that you have to go with a peer-to-peer network there's um there's only so much that you can do and if I run this crawler do you run this crawler if ten different people run this crawler we're all gonna get ten slightly different fees on the network and that's just how this distributed system works but getting as much information as possible will give us a better idea of the whole picture so once we send out these messages we discover these peers we ask them for their health information this is part of the etherium sub protocol is when they first report backing they say these are the capabilities I support this is my client you know basic information but then the what I really wanted to know what why I started this was working on in fira there's a lot of misinformation out there that that we come across on Twitter of in fira runs all the nodes on the etherium network which I definitely know it's not true we were the only ones that can keep nodes in sync that most of us here know is not true and I always resisted the urge to respond to these things because they're coming from trolls where if I just respond I have no way of backing up you know what I say without actual information and that this isn't information that's on chain it's not verifiable unless you can verify it yourself by running it and so so that's what I started out with this is all right let's let's let's create a tool where other people can verify the size of the network in the health of the network and so in addition to trying to find the number of peers I was trying to find are they healthy and if they're healthy are they are they mostly like clients are they full nodes you're not able to see things like are they archived nodes that's something that's outside the space of the peer-to-peer protocol but knowing whether or not they're they're full nodes I think is sufficient so as this data came in I decided to throw it into a document store I initially started with Postgres and then moved it into manga or sorry manga and then moved it into Postgres instead I needed to enrich the data because I wanted to create visualizations something that was better than the simple v's that ether scan sorry even nodes was had which was the basic pie charts like we we have all of this data we should be able to see you know what's the number of healthy nodes over time over the last year what's the number of nodes that are keeping in sync if say you know god forbid there's another spam attack or something on the network and nodes are struggling to stay in sync what's the effect that that has and so one of the pieces of key pieces of information but you need there is the block number in the etherium sub protocol what you get back is their best hash so this is like the the best hash that I'm aware of and for you to be able to determine what number that is you need to look up that actual block information the DEF p2p client that I built us on top of was not a full client it was just like a very thin definitely the peak client and so I ended up clearing my local node that was running to be able to you say give me this block number and then I would index that as well so that's what I'm talking about with enriching the data additionally it was also the Geo IP information so that we could see what's the the client distribution across the globe like are most of them running in the Amazon data centers are we pretty evenly distributed the block numbers like I discussed and then since we're storing this in a document store with the intent of visualizing it wanted to make sure that we were indexing on pertinent fields so that we can quickly query this information we're going to have a lot of information in here you know hundreds of documents per day potentially if we're querying a specific node for their health every couple of hours and then indexing it wanted to make sure we can query on like give me all of the nodes around this block number give me all the nodes that oh actually an interesting thing that I was thinking about was our people cycling their Enid's you know like are people concerned about their privacy I'm saying you know I'm running a cryptocurrency exchange I don't want somebody to be you know detecting where my trans where transactions are originating from if they can in that if the network's small enough you potentially could so maybe they're cycling addresses maybe they're cycling Eno's and so that was something else and and the last thing that I thought was valuable was capturing the capabilities of a node so how many of them are supporting light clients or enabling whisper enabling swarm all of these things are captured there and not exposed currently in any of the sources but sources of information that we have like like ether nodes so all of this is very useful to understand the historical state of the network and besides that I'm a little bit sentimental about this and in thinking it's it's great and surreal to see the growth of our ecosystem over time the this information is not stored on chain we talk about the blockchain being Amida belen this you know information being stored for posterity but part that's only the on chain information is only part of this story and the peer-to-peer network is another part and it's really cool to capture the historical state of this information my ideal would be that you know maybe the foundation runs something like this and and captures this historical information so that you know 10 years from now if people wanted to know the health of our peer-to-peer network now there that information is available track once we start tracking this historical information you realize that tracking node health over time is is almost like tracking node reputation and could current client implementations use this data to improve their ability to prune bad notes from from the peering table so oftentimes in the past I've run into issues where my node would just stop syncing data and when I would look at it I would you know SSH and take a look at the console and find that a lot of the peers that I were paired with were not in sync and so I was stuck in this area of you know having to manually kick off peers so that it would hopefully find healthy peers for me the sink those issues have slowly gotten resolved but I'm sure that there's room for optimization and I'm hoping that this information can help client developers take a closer look at that so now let me take you through the code because it's pretty pretty quick all right so like I was saying the the code is really simple being built on top of this theory MJS module the the total lines of code is you know 144 and and majority of this is boilerplate JavaScript one of the key things that I did here was as so we initially initialize our private key we make sure to identify ourselves as a aetherium JSDF p2p client I put research in here to be a good actor in here if people want to filter us out they can and just reporting simple capabilities each 63 and each 62 and then start starting to connect to the network so as we connect to the network we send out our our LP message or start our opx protocol message of the hello message and then as soon as that comes back like I mentioned we do the Geo IP lookup so there's an embedded geoip database to quickly get the location information on that IP and then we enrich this data packet do you some additional splitting of information so that we can index on things like client name client version operating system name etc and then store that into the database so if anybody wants to try hacking on this it should not be daunting I'm not a JavaScript developer regularly I mostly work in Python it but you know working in this was extremely easy so if somebody wanted to take this and extend it I hope it's pretty easy to understand where they get started additionally in here to make it easy to start running I docker compose file so that you can just do a doctor compose up and you get three things you're going to get this application along with the Postgres database and an application called meta base so I wanted to put this data in a way into something simple like Postgres and then find a tool that would allow people to build better visualizations then I could build myself but without having to kill them from scratch and tools like meta base are things that allow people to do that because ultimately the data is more important than the the initial visual visualizations and so I'll show you some examples that I was able to build and I'm sure people can build better ones so so this is this is the package I'm going to I'll go ahead and open source this right that are unlocking and making the public repo right after this is call this is an example of the data so me being me I messed up my actual live production data set right before the demo so this is a snapshot of information from a couple of weeks ago but it looks you know a little bit better than the data that's on on ether ether notes ether scanner and actually started showing some of this data as well but it's not something that you can run yourself and and in this ecosystem I think we we all like having something that we can run ourselves yeah times that all right so these are the visualizations just really quick I just want to show you the peering table so that this is what's great about this meta based tool is you can really drill into this and search for a specific IP addresses block numbers hashes and you get a better sense of the information so so that's it I want to say thanks again for having me and everybody else enjoy Gourley con just hit me up on Twitter or anything if anybody has any questions or wants to collaborate in the future exactly [Applause]