Devp2p | Felix Lange | Lecture 18

so now we are at the uh week five of our study group um we are in the second phase of the study group where we are diving deeper into various parts of the protocol with uh live uh lectures talks from uh uh Clan developers and researchers and uh uh this week we are looking into networking we are looking into the uh peer-to-peer layer of execution and consensus starting with execution now and I believe we have the best person to talk about the um the networking of the execution layer here the de peer top protocol uh Felix who is U working on go ethereum team uh but he specifically focused on the uh networking and uh he's maintaining the the peer-to-peer spec so yeah honored to have you here Felix thank you so much for joining and um yeah if you want to if you want to start talking if you want to tell us about the um uh the uh e networking Dev peer-to-peer uh yeah yeah I can I can I can do that so um hey guys um this is actually my first time on this type of event I'm just not really sure how these generally go and also I'm not really sure how long I'm even on for so I think it's going to be about an hour yeah we have 90 minute slot um with questions and everything so it's uh it's up to how much time you use I'm I'm pretty sure we're going to be finishing under 90 so the main thing is that I don't have a big presentation because I'm not this type of slides guy and I think also like what uh what I'm talking about doesn't really need so many slides so mostly I think what I've planned for today is just that we're going to be looking at the specs for a couple minutes and then basically we can just straight go to some kind of interactive thing where people can ask questions or we can discuss interesting uh Challen es and so on um I don't have a lot of like prepared content because it's just not um it's going to be quite boring if I if I I think you'll just see like once we get once we get started talking we will we will find something anyway um yeah so the way I was actually going to do this is basically just share my screen and um go to the specs so you guys able to see my screen I guess so uh yes yeah all right yeah looks like I'm getting some positive reactions about this so all right so the um for the um execution layer P2P networking the specs are actually located uh in this repository it's the def pwp repository and it's one of the older ones in the orc um this one um goes back all the way to the very beginning where we had a single file kind of uh EXP explaining the P2P protocol I'm not really sure like there isn't really a good way to um a good way I think to go to I guess we could kind of go back to the Early times to discuss it um but if we I'm not really sure how this works anyway so but we can actually because it's only like a couple hundred comments we can actually just also go all the way back pretty easily uh so like the way it kind of all started is with this type of thing so this was kind of the first um specification of uh ethereum's P2P uh networking and uh at that time there wasn't actually like consensus layer and execution layer we just basically only had the execution layer uh or what is the execution layer now and in this document we kind of tried to a bit explain like I was there when we created this and it was kind of this idea that um basically in in in the in etherum PP we had like two main Concepts and the two main Concepts is the node Discovery and the other big concept is this like idea of the um subprotocol connections so these are the the two big things and then so um and we have retained this difference all the way up to now like 10 years later we still operate in this basic Paradigm where we have no Discovery as a separate thing and we have um the other protocol which is traditionally called The Wire protocol and that one is like the thing that actually where the nodes actually exchange the blockchain information and um I mean the basic thing that we wrote here is kind of still valid today so I mean our objective is that in ethereum um the nodes should be able to have a uniform Network topology so basically the network topology shouldn't be drastically different from note to node um like their view of the network I mean and um everyone should be able to join the network and um it should be authenticated and encrypted well we'll get to this part but anyway that was a goal and um for um for Discovery specifically there was this idea that like you actually got to be able to find like new nodes to connect to pretty much reliably and some of the other stuff in here is a bit weird but basically it was kind of this this was the goal and um yeah I mean the whole spec is U not very long it's kind of weird but um it's uh it's there and um nowadays I guess we like ever since this document was created if you go to the now you can kind of see that we still kind of got this document so it's a bit different now but basically it's the same one nowadays this document only really specifies the wire protocol or not actually not even like it doesn't specify the actual wire protocol it only specifies the mechanism like the underlying transport but this protocol has not really been changed for the last 10 years um and then on top of this protocol on top of the RPX uh TCP protocol we actually have the real sort of ethereum wire protocol and this is um it's called eth and this is specified in this document and uh nowadays uh we actually call these like it's possible to run multiple protocols on top of RX and they are called capabilities now and so eth is just one of the possible capabilities but it's basically the only one like nowadays the only capability that's really used is eth and also to some degree the snap protocol for the snap sync so we're going to get into this a bit later and um so the eth wire protocol uh we're going to go over in a second but basically it's just uh this protocol that the nodes use to actually talk about um the blockchain and um nowadays since the merge this protocol Ed this protocol used to also be used to exchange the blocks but nowadays that's no longer the case so now nowadays this protocol is only used to synchronize the existing blocks um but it's not used to announce any blocks but it is used to announce transactions so the main two functions of this protocol are syncing blocks and exchanging current transactions and so yeah I mean this is basically what we uh explaining here is well we still have this section somehow about like how the block propagation used to work by the way this spec isn't I mean the absolute most up to date I mean it is kind of up to date but it's also just like probably this section should be deleted now but we've kind of kept it for like reference and so on and um but everything that's in here is kind of um still pretty much valid and it is how it's done today and um let me just quickly go over a bit like how this how this actually works so um for connections between peers we um exchange messages and the most basic message there is is the status message which exchange exchange right in the beginning so both peers send this message in the beginning and tell each other about the version that they are running um the uh network ID that they're trying to connect to and their Genesis block and then we have also various other information like the fork ID we'll we can talk about that later if you want but basically the the idea behind this message is just to tell the other guy you know what is the uh which network am I operating on is it going to be the main net or is it one of the test Nets or is it like another custom Network nowadays there are a lot of networks actually they are can be found in this this website and um yeah there's lots and lots and lots of different networks and all of these networks did at some point probably use this protocol so it's quite uh it's quite good to uh to be able to check uh the network at the beginning and then um yeah this message is kind of unused this used to be part of this this the block relay but the transactions message is still used today so it's basically just like a message that gets sent between the peers whenever there's new transactions um we also have these like get Block headers get Block bodies types of messages these are used to sync the blocks and then we have this mechanism here new pool transaction hashes get pool transactions and pool transactions these are used for the transaction Exchange and then we also have receipt syncing for this is important during the snap sync and here at the bottom of the spec you can find all of the different versions going back all the way um to the very beginning even before the mainnet launch where uh and all of the changes that were done and it's actually not that many like the protocol has changed a little bit but it hasn't really drastically changed so it's kind of still the basic concept is very simple and is still kind of there today and you can also see that actually um most of the bigger changes to the protocol were made using eips so basically whenever there's a drastic change to the eth protocol we have proposed in an EIP and in these eips which are also linked here you can uh learn a lot more about uh for example the changes and why they were made and any kind of considerations we had at the time and so on and also like who proposed the change originally so it's kind of a useful resource I feel being able to click through to these definitions and seeing in more detail why people wanted to make certain changes to this protocol and we are currently debating another update to this eth version 69 uh which will um further change some of the data structures Exchange in the protocol and probably also be um an announcement it will also change the status message in a long time to remove some of the stuff that's actually for example the total difficulty which is no longer needed after the merge but also in the next version we are planning for example to announce um the history expiry support of the pier because this is kind of a New Concept now nowadays the nodes may be able to remove some of the blockchain history from the node and uh this is something that we may also want to announce in this handshake anyway um are there any questions so far anyone feel free to raise your hand um people are complaining that the uh video is blurry but you can just check the specs uh by yourself in this repo Sobe you can follow the website yeah um but if there are no questions I think we can we can move on but feel free to raise your hand if there is anything no I'm just kind of curious because I have no idea about like the familiarity of everyone with these topics I may be going over some material that is completely known to everyone already or it may also be that I'm going to fast I have literally no idea no worries no worries you can you can assume that people have some idea about the execution layer uh and what kind of communication is happening uh but uh folks if you are unsure or if you want to provide feedback like don't be shy you can you can always speak up okay well seems like nobody really wants so it's fine um the M the main thing is that um I can also quickly go over like some of the other parts so we do have this lower level protocol this protocol um this is kind of this uh this is like the middle layer sort of like so we have the the peer-to-peer like the connections between nodes I made using TCP and at the very high level the peers just want to exchange these messages that I showed uh moments ago in The eth Wire protocol but in order to do that they need some kind of mechanism for transporting these messages and how they should be encoded and so on and this is what this protocol provides so uh RX is like I said kind of an old protocol it was invented about 10 years ago uh not by me but by someone else and um it's kind of a weird one also like nowadays it's kind of the the question you'd be asking yourself is why would you create something like this but at the time um this is just where you may want to understand that like in in the early days of ethereum people were just doing stuff they thought would be useful and they didn't really think too much about uh the long-term consequences of it or if there would be a better solution they just kind of went ahead and did whatever they thought would be the best solution and it's kind of the case with this as well so it's kind of this is a protocol that attempts to establish an encrypted and authenticated uh connection between two peers um it does kind of fail at that because as we later learned the encryption part of it is kind of broken it hasn't really been fixed so far because there's no fundamental need to encrypt ethereum Communications anyways and so it's more like the the protocol now provides basically some kind of like masking so it's it's like the information is not exchanged exactly in plain text but if you do know how to how to break the encryption which is pretty easy and it's explained here in bottom of the document then um from the point of view of um of of you know like real privacy this protocol is is is not is not working um however the authentication still is working so it has not been broken so it means that if you connect to a certain Pier by its peer ID then you can be sure that this connection will also go to this peer ID in fact it's not possible to impersonate another PID using this protocol that said the the mechanism for authentication is pretty wild and um the message for the mechanism for message authentication is also pretty well they do work but they are kind of strange anyway um yeah so um I'm not really going to go over too much of the details if anyone is interested I'm happy to answer questions about this but I also don't know what you guys want to learn so I will just tell you quickly that this protocol proceeds by having a kind of handshake after the connection when the nodes connect they will PR first perform this handshake in order to establish a cryptographic um like some cryptographic key material and after they have done that they will go uh and exchange the messages and the messages they are uh kind of uh described a bit further down in document and here where we is where we enter these capabilities as well so after the handshake all of the messages exchange on the protocol are associated with the capability and there's like a whole number of them like I mentioned before we have the mainly the eth and snap capabilities but but there's also a very basic P2P capability which is about protocal internal messages so for example when the peers very like start talking to each other for the very first time they will initi initiate the uh exchange using this Hello message that also exchanges a bunch of very fundamental information about the node like for example the client ID in in for example gu's case it's just going to say gu and the version and also the list of capabilities because uh like I said they are Dynamic and um there's also the disconnect message that is sent when the peer disconnects and here we have a bunch of reasons uh most commonly is going to be this one uh so usually when you try to connect to another P just going to tell you that like yeah it's already kind of all of the connections are already used so it will just reject the connection using this status code but there's a bunch of other status codes here and and they they have uses I guess uh we do track them so we have a we have a we have some metrics around these disconnect codes and they are all used it's just that use uh too many peers is the most common one um yeah and then I mean apart from that um there is uh in this basic RPX protocol there is it also defines this concept of the message uh IDs and the message IDs or commands are basically also the messages that are defined in the higher level protocol so if we say for example within the um eth protocol that the status message has the Code Zero then um this is a concept that is that then translates into rx's idea of a message ID and uh it has a system for basically like at startup the um like when the connection starts up and uh the hello messages are exchanged after that both peers will compute sort of like this like message ID table and um then they will be able to associate each message which is received with a capability so for example for eth um the messages will be renumbered so that they can be told apart from the basic PTP capability not sure anyone got that but yeah that's something that's in there anyway and then finally uh just to conclude my presentation there's also the discovery part and for that I'm just going to zoom way out and go back to the uh read me here so uh in the readme you can see that uh basically everything is laid out here we have the RPX protocol that we just looked at and we have the ethereum wire protocol that we just looked at uh before and then we have the two Discovery protocols so I'm going to start with the discovery version 4 because it's the older one so this protocol was basically the original discovery protocol that was created back in the day before the ethereum launch and um it has been updated a little bit since then but not really and um this protocol is a UDP based protocol so the nodes will open a second port with UDP and uh upon on this port they will form a kind of uh uh Dynamic uh distributed hash table so uh what this means is that each node will maintain a data structure inside containing some references to some other nodes and they will um occasionally um exchange some packets with these other nodes the primary function of this protocol is basically providing a way to query the network for any peers U that exist so using this uh protocol you can walk this data structure ask different noes about which other noes they know and then when these notes are returned um you can attempt to make connection to these nodes using the RX and this is the primary purpose of this and um it this is achieved using the kademlia system so kademlia is a very old um distributed hashtable uh protocol that we have adapted here and um cademia was originally invented as a proper distributed hash table so it would also it would would contain keys but also values so basically you can just in like any hash table in a programming language within cademia you could store keys and Associate values with them but in this protocol in the discovery protocol we do have the keys uh but we do not have the values so basically this protocol is only used to store the nodes and um there's a lot of there's a lot of content here the discovery protocol is uh supposed to be quite resilient since it is exposed by every ethereum node so uh there's a lot of definition here and a lot of stuff I'm not really going to go too much into this unless there are questions and um as with the um RPX uh and the ewire protocol there's there are also some revisions here and you can see that there have been very few revisions so we made a tiny revision to the Discovery protocol in 2017 in EIP number eight uh this was written by me and this was just when we started introducing like a version number into the protocol so uh um and actually this was created uh not just in 17 but it's created in 15 so the basic idea behind this IP was just to um add a version into into into every packet because we wanted to be able to upgrade the protocol and uh in the in the initial version of the discovery protocol there was no version number so we weren't really even able to to upgrade it properly and this changed it was kind of a kind of a tough change but we made it work and then much later uh in October 19 we made another change to the Discovery protocol where we started adding support for the ethereum node records and as an extension so this uh extension is nowadays also supported by every um every ethereum implementation as far as I know and um this one um basically changed some of the packets to um allow exchanging uh more information about each node so prior to this change uh the only information that could be exchanged is the IP address ports and the Noe key but after this extension was introduced we can now also exchange information such as the uh uh blockchain Genesis hash or the fork ID uh to further check if the node that we have found in the discovery system is actually suitable as a connection Target for RPX again I hope everyone got this but yeah if you don't then by all means please ask questions curious if there is a reason why you're using why you started using cema if you're not using the full functionality of of that table um yeah so I can I can go over this um so the basic idea behind cademia is kind of a good one so it's it is a way to EST to establish a network uh with many like it supports an arbitrary number of participants and and um it is able they are basically able to form a network but each node Only Stores a fixed amount of information and and yet the the network is very scalable so this is originally what Drew us into cademia it's because I mean there are other designs but basically we we chose this one because we felt it's the easiest to understand and most importantly I mean and this is still true today there's the note only has to store like about 150 or so other nodes even though the network can be much larger like nowadays the the the the Global Discovery network of ethereum which is actually shared among all of the other chains that are using the same protocol as well um we have uh many many members in this DHT uh but still um it is able to function no matter the number of nodes because um it is built in such a way that each note only has to store about 150 and and it still kind of holds work together anyways so that's why we went with it all right cool so um yeah nowadays we also have uh the uh Discovery version five so this is something new and um it's something that I uh created I'm actually not really sure about the year but I guess yeah about five years ago or so yeah kind of also 19 so the idea behind uh this thing is more like six years ago so um the idea behind the version five was to finally remove some of the very annoying limitations of the uh Discovery version 4 Protocol so one problem that the version 4 Protocol had is that it has a dependency on the clock so um and this goes kind of all the way back so even in the very initial version of the Discover for this was a known issue but uh we never really bothered to fix it because at the time it was kind of tough to to get these changes in and um so in each Discovery version four packet um in the basic packet frame we um have a few actually it's not in the basic packet frame sorry in each message submitted via the protocol we have this field the expiration and this is basically just an absolute Unix Tim stamp that says how long is this message valid for and the idea behind this is that usually the note would send messages that have an expiration that will be about five seconds or so into the future the problem with this and then the when you receive this packet if you see that the expiration doesn't match your current time then like if it's if it is earlier than the current time then you will just reject this packet the real problem with this is that it requires nodes the nodes clock to be synchronized and um this is this is something that uh I guess the initial designer of this protocal just didn't really want to acknowledge but it has been a real pain in the in in in practice because often people's clocks are not entirely synchronized and it can lead to this weird issue that basically they they run their node and just nothing happens because they just cannot find any other nodes and for the longest time this causes a ton of pain now we even tried to work around this by by adding code into the node implementations to detect whether they had configured ntp or even later and still to this day in gu we have logic that uh once we hit a sufficient number of network timeouts we will actually perform an ntp query ourself to detect whether the nodes clock is uh off and then they will we will notify the user about it it's kind of a it's it's just a very messy situation and this is one of the reasons why Discovery version 5 was created is to uh work around this stupid issue and then the other issue is that um it has a bunch of other things so in order to ensure the protocol is safe the packets have to be sent in a certain order but this order is weirdly undefined and it kind of relies on some like there's no clear State machine for this protocol to work even though it's four it's only four messages or nowadays it's actually six messages but anyways like even though technically there are only very few messages it is really complicated to send them correctly and the notes have to do some very strange stuff to make it safe so it's just this protocol was never like a good design in the first place and the discovery version five was to fix all of this so um the main thing about this is that uh compared to the version four um there's no dependency on the clock the protocol is encrypted unlike Discovery version 4 and it's authenticated this time the encryption is also actually working um also um for it is a bit future compatible in the way that like we actually don't really use we're not relying 100% on the sep 256 K1 elliptic curve but actually we could use another crypto system later if we wanted to I didn't go much into this before but uh ethereum's P top nodes they um all use SE p256 K1 it's the same curve that we also use for the account cryptography in ethereum to uh establish their node keys and the node key is this fundamental private key that's created by every um node uh and that is used to establish the cryptographic key material for the P2P connection and in the discovery version five it's actually possible to use different types of keys if we ever wanted to the protocol is extend extensible in that way whereas the older Protocols are not and then finally um it introduces this concept of relaying arbitrary metadata uh between the nodes uh using the ethereum node records and um this is a topic that we may actually go into later so um there's another spec here that explains a bit how this looks like and um all basically all you have to know is that we can we can relay using this protocol much more information than just the IP and the port of the node we can also relay like I said before um uh the uh network ID and uh Genesis information and so on which is very useful but also some other protocol specific information like for example we can uh uh uh Mark whether the node supports snap and things like that so it's kind of a useful um a useful thing and then finally there is the topic advert system this is something that was unfortunately not realized so far um uh I am still working on this we have uh published a paper about this a bit earlier like last year and uh we are still working on finalizing the implementation and so on but it's something that nonetheless is mentioned a couple times in this spec and this mechanism was the hardest to get right we still not quite there I mean we have a basic mechanism now we know how it's supposed to work this was not the case when when this document was created but um either way this is not fully accurate it just it says this on here because it was a goal at the time but it's actually not not not there um yeah so um if you want I can go into more detail related to the DB5 as well but it's definitely something that will require a bit of time and nowadays also the discovery version five is the same protocol used by the ethereum's consensus layer so it's not just used by the execution layer but also the consensus layer nodes use this protocol to find each other um it is supported in some execution layer implementations notably G and Aragon and I think some others but it's not universally supported so for example nethermind does not have an implementation of it right now and neither does Buu so um we're still waiting on these guys to fully uh uh be up to date and also start using this protocol and then we can finally drop the uh Discovery version four but for now we are operating Discovery version five and Discovery version four in parallel um yeah so this concludes my presentation about uh the networking specs um there's a bunch more stuff but I would say that now is kind of the time where we could have a couple of options I can either start answering some questions about the specs or um I can start answering some questions about uh further topics in networking or we can um go over the GU implementation together and uh look in detail how these Protocols are implemented in G um either way I would like you guys to let me know what you want and um yeah yes thank you so much Felix uh yeah let's take a minute to take some questions and then we can uh look into the go eum but first folks please raise your hand if you have any thoughts any questions on the specs or generally the networking dimitric has a hands up yeah yeah hello thank you for the presentation uh I'm sorry if I missed it but so uh the version four had this problem with clocks which were very complicated but um I didn't I I think that I didn't get the solution so in the version five you remove the the whole issue with clocks but what does replaces it yeah that's actually kind of a good question so um thank you for this question so um the basic idea is that um in in the discovery version four we had to use um the clock because the packet requ the the protocol requires some kind of Replay protection so the only reason why the clock check was added in the first place is because there is this kind of aack in the network protocol where for example someone can just keep sending you the same packet over and over and um this has some implications so because the packets are signed and thus authenticated by the sending node it's kind of annoying if someone else can take this exact payload and just keep repeating it to you so um it basically means that yeah it's it can be used for example for very cheap uh uh denial of service attacks but it can also be used to for example break the protocol and things like that so we didn't want this so uh this is why we introduce this clock check and then in the discovery version five because it is properly encrypted and authenticated each node that you are communicating with you have a dedicated um session with them and within this session messages cannot be repeated because they have to use nons and they have to be encrypted using as GCM so you cannot just repeat um an encrypted message from another node because first of all you won't really know what's the meaning of it but also uh it won't match the session then actually that's not true so you can exchange the same packet but uh since the sessions are kept by IP and Port it's like it's not really going to find the session let me put it like that thanks awesome thank you so much uh we have another question from samart like I want can you tell something about like why use Li P2P and versus de P2P like in execution they mostly use Li yeah that's also a very good question uh it is one that I have attempted to answer uh in the read me um so um one of the things is just that this protocol is older than the P so I guess this is the main the main answer um basically uh when confusingly actually def PDP used to be called lip p as well so that is something that doesn't really make it a lot lot easier to understand but we said we since moved to to Def PP um yeah it's a very good question I mean nowadays we could probably change the execution layer to use Li PP as well it's just that it hasn't really been any need um it works pretty well as is and it's a custom it's a custom prot so we would have to Port uh our wire protocols like the eth wire protocol for example would have to be uh ported over to lip and the snaping as well and um yeah we just haven't really seen the need for it that said it is kind of um it is kind of a nice project and maybe eventually we will get around to it um awesome uh we have two more questions shash wat yeah so far in Discovery so far in Discovery in Discovery five like we used to like use TCP or orp so like on what parameters do we decide that which which protocol to Essen use and yeah I'm sorry I'm I'm I think I got your question but I'm not sure are you asking why is it is one using UDP and the other one TCP or yes because in because in like portal Network like they were they were using to so that they could send a larger payload but uh like onward parameters like decide it for the execution yeah I I hope I got your question I'm I'm sorry if I'm answered incorrectly because I I don't know but uh I don't understand 100% what you were saying but um so just uh what I can uh refer you to also uh for example here or actually no this doesn't really have it I guess but it's it is explained a bit in the U in the um in the disc in the dis5 uh spec so um one of the things about the discovery protocol specifically why it has to use the UDP is because um it is a protocol that uh requires you to talk to a very large number of of nodes with low latency so um that in order to uh use the discovery you basically have to walk the network and this means you have to talk to a lot of different hosts in a short time to get some answers and then um this is why we chose the UDP because it means um there is less overhead per connection if this was using TCP we would have to establish a basic connection to every node that we talk to which can take quite a bit of time and then there would also be something like for example the noes would have like an accept Quee and so on so in the in the operating systems uh TCP is much more has a lot more overhead than UDP for a single connection and that's why we chose to use UDP just because we wanted to have this very low latency connection establishment uh which each with each of the nodes that we are talking to for the main PDP protocol for the ethereum wire protocol and RPX we are using TCP because it is uh better for the long lift connections and it will for example also properly uh uh regulate the traffic when uh a lot of data is sent if we were using UDP for this purpose then we would have to implement these kinds of mechanisms ourself uh and this is not something that we have chosen to do however nowadays the discovery version 5 protocol is also used for systems such as the portal Network and they chose to go the other way and they have actually implemented the data transfer over UDP as well so it is a choice that can be made I hope I answer your question correctly yes thank you you answer yeah thank you much it was very interesting uh we have another question from pinaj yeah so in the discovery mechanism is there any validation entitled about validation of node before the establishing any connections like uh node current status or node reputations Health Factor all before the establishing the connection with any node um yes so there is a little bit of validation but you fundamentally it's important to understand that this protocol is supposed to be about the open participation so we cannot really make any assumption about the nodes at this level at the same time we do require them to have an active endpoint and the node has to be responding on the Discovery protocol in order to be considered uh for announcement to other nodes so this is a fundamental thing so then we will we will not pass on Noe record that we cannot validate ourself so this is a very basic rule that we have and it's like knowing more about the node before the connection yes so it is not it is about announcing more information about the node before you connect that is 100% correct but um still we cannot really verify a lot of the information that we get for example if the node announces that it is a participant in the ethereum mainnet we cannot really verify this it's up to the the peers that connect to it to verify it when they do but um at the same time there also there isn't really a lot of incentive for noes to lie about this type of information it's just they they are just a participate in the protocol like all the other participants so it's kind of in their interest to announce the somewhat correct information in order to get the correct connections like you're not getting lots out of this by out of it by announcing something incorrect got and there's no um like basically sorry uh just uh before we do the next question I just want to quickly mention also that there is no built-in reputation system and this is actually something that uh we find it's kind of an important point because we found over the years that the use of reputation systems can actually have a somewhat harmful effect in the cases where the network is broken and this is something that we recently also saw again uh in the um heski t testet uh incident where for example the reputation systems within the consensus layer PDP implementation cause the nodes to have a hard time connecting to each other uh during a network fault and this is why um within death PTP we do not recommend and do not specify a reputation system within the notes that makes sense yeah thank you thank you so much thank you so much Felix yeah that's really interesting with the peer scoring on CL being one of the problems with their peering during hki incident yeah thank you so much and we have another question from vant yeah hi uh I'm not very sure like is this the right time to ask this question but uh like in general I wanted to ask like what do you think are the attack vors in the current implementation of like there are like multiple things like uh uh Network legs there are delays there are disconnections and like there could be malicious noes like how how do we handle this and like what are the uh attack vectors like which are there now so uh I mean in general we are hopefully there are no attack vectors I mean it's supposed to be secure you know um that's the thing it's this this protocol is designed or actually all of the protocols in ethereum are designed uh under the assumption that the uh remote party is always going to be malicious so this is something we assume even in the basic wire protocol so when we talk to a PE we will always validate the information that they send to the extent that it is possible and the design of the protocols takes care to ensure that this validation can take place and that you do not have to accept for example the information as is so um as far as we know there's there are no like big attack vectors I can go over some of the things that we have seen ATT tax in and I would say the number one thing we have seen ATT tax in is the transaction exchange mechanism so uh within the ewire protocol there exists this facility for exchanging transactions uh it's also known as the transaction pool or the me pool and um the transaction exchange is a very simple process but um what it is maybe deceptively simple here in the spec because in the spec it literally just says you know if you have some transactions you're supposed to send this message and then the other side is supposed to give you the transactions is or basically it's it's a it's it it is kind of primitive but what it doesn't tell you about is that the node actually kind of has a complicated data structure like each node implementation has to maintain a pretty complicated data structure to make this work and um There Are Rules uh in this data structure for which transactions are acceptable at which time and how the transactions are supposed to move through this data structure these rules are not spelled out on the spec but it's kind of implementation you find and we have seen a lot of attacks especially also in G because it used to be or maybe still is the most popular implementation so there have also been for example academic papers about the security of this mechanism where people found specific issues in our data structure implementation that enabled them to for example uh publish a lot of transactions at once or remove other people people's transactions in favor of theirs so this is one of these areas where there's always going to be something um and to this day we are still receiving for example at etherum Foundation um a lot of reports of like um um a lot of reports of attacks within the system on the basic layer though there are not really a lot of attacks that we have seen or known about it seems to mostly be working awesome uh yeah thank you so much Felix it's it's uh very interesting to learn um bunch of these edge cases I am waiting for some more questions if there is anything else you folks want to ask we have still like half an hour to go so we can uh look into the get code meanwhile if there are no more questions for now yeah yeah I think we can jump into it yeah so this is something where I thought this is probably how where it's going to go because a lot of people might be interested in it and a lot of people might might also want to know like just where is the implementation in the first place like this type of stuff is not always super easy to discover um and I can just quickly tell you a bit about like where is it and uh how can you read more about the about it in the code and then I can also show a bit you know like even though for example like I said the specification is quite simple then there are certain parts that actually have a lot of code and other parts that have very little code um let me just show a bunch of quick example so this is the go ethereum Source tree and um we have uh the PTP directory that contains most of the basic peer to-peer implementation and actually within this package the P2P package um this is kind of where the uh um where the um entry point into the whole system is so um I'm just going to show the go documentation now because I find it a bit easier um to show the apis and so on with this um yeah I hope you guys are at least a little bit familiar with go because this is obviously going to be a lot of Go stuff but I mean I guess the basic concepts should be fine so the fundamental entry point in the system is the server type so uh the PDP server in go ethereum manages um both the T the TCP uh connections and it also launches and configures the um Discovery system and um you can see it's actually kind of configurable even from the outside in all of these options that I'm just quickly scrolling over here they are kind of setable also from the outside using G Flags But ultimately using G Flags will just translate into these settings internally so for example um you need to specify when you start it up you need to specify um the node key like I said and at the moment this is a sec p256 K1 private key uh this key is used to establish all of the key material and so on and it establishes the identity of the node on the network then there is also the fundamental setting in in in in go theem where you can configure how many peers you want to have in total and this um this uh usually the default for this is about 50 um but it can it can basically go up to any number it doesn't really give you a lot though like even if you specify tens of thousands of peers here it's pretty rare that you will actually get that many peers and also why would you want to have that many peers it's just going to cost you a lot of resources to serve all of them so usually we recommend that people just stick to 50 as a default or maybe 100 if they somehow feel like it but it's not really it doesn't give you anything and then um we have a bunch of other settings here but one thing that I want to highlight is that as with any P2P system we have to make sure the system fundamentally allows other nodes to connect to you so it's not just about connecting to other nodes and in order to do that we have this setting here it's called the dial ratio and at the moment the dial ratio is oneir so what this means is onethird of the connections created by gu will be initiated by this get node whereas the other 2third of connections are reserved for other nodes to connect to you and uh we have this ratio because we know that within the network there will be quite a bit of nodes that are maybe not able to accept connections and in general it is better to allow a bit more incoming connections than to make outgoing connections yourself um finally there are some options here for enabling the different Discovery systems and we have for example Discovery version four which can be enabled Discovery version five can be enabled these are both on by default and then we have things such as the bootstrap notes uh which are for configuring the initial nodes that the discovery system uses to connect to the network and finally we also have static nodes and trusted nodes what this is is um you can also just configure um the GU to connect to certain peers that you list within the configuration and it's called Static peering uh finally we have a system for restricting connectivity to certain IP ranges this is useful for testing and U I don't know specialized use cases and then finally the basic P2P server contains the list of protocols and the protocols these are basically the capabilities that I showed earlier so there will be an instance of this of this object for each protocol that get supports notably the eth protocol and the snap protocol and um there actually will be an instance of this object for each version that we support because we do support multiple versions at the same time sometimes like when we for example publish a new eth version we will commonly accept also an older version and I think right now we do actually only except one eth version 68 but when once we start introducing eth version 69 we will be supporting the previous version e68 for a while longer and this is uh when we will have multiple instances of eth uh in this protocols list and um uh notably within the protocol implementation uh each protocol has a run function it's called that and the Run function is invoked for each pier and it receives the uh messaging stream as an argument so based on this stream it can start exchanging messages with the other peers so it can read messages can also write messages and all communication with the peers is handled by these two things we have uh these two functions and they are ultimately invoked for uh every message sent through the P2P system and then there's a bunch of other callbacks also provided in the in the protocol uh most notably also each protocol can nowadays specify its own source of notes to be dialed so this dial candidates field configures um like a source of nodes that the PDP server will then try to establish connections too in order to uh uh get on the network and I will show what this means in one second um yeah so the server object is the most basic object I will now show it a little bit in the code so it's defined here in this server P2P server. go file and um yeah uh internally I'm not really going to go much go too much over it but yeah basically the server is just an object that has a start method and the start method will kind of you know like do everything so here we will set up the node uh set up the discovery system set up the dialing and then finally we launch the server run Loop is which is the thing that manages all of the peers and uh here we will also configure the discovery systems and so on but then um when it comes to the um to the protocols is actually quite interesting because the the the wire protocol and so on is somewhat decoupled from this P2P system so I'm basically going to show this first and the wire protocol is kind of set up in go theum in a bit of a more high level Place uh it's set up here within this file e backend or go it's kind of the main file really of go etherum like this file is where the action is and it contains the ethereum object and this object is the fundamental sort of like holder of almost everything related to G like pretty much everything that gu does is somehow done ultimately reachable from this object and uh one of the things that this object is also responsible for when it is created so here we have the Constructor function of the ethereum object is um it will somewhere also um register the protocols let me just quickly show yeah so here is where we will link the protocols supported or the capabilities supported by the ethereum object into the P2P so here you can see that basically how this works is that within this make protocols function we go over the supported protocol versions at the moment we only have one protocol version I'm not sure where this is defined actually but I think at the moment recording is on all right we are back so um within this function we uh iterate over the supported protocol versions and um for each version we create an instance of the P2P protocol just like I show before and here you can see kind of like where this run function is actually created and notably this sits on a back end and the back end is actually defined back in the other eth package they show before this is kind of where the confusion uh gets even more confusing because even though this this is defined here as an abstract sort of interface actually it is implemented by um by the other package with the same name and um yeah basically within the Run function we just tell the backend to run this pier and then even more confusingly we actually pass a callback that will say that uh uh um uh it should then invoke so this this triple uh in Direction here is just done because of the split of the two packages was actually very hard to to take apart the Thing Once Upon a Time we just used to have a single eth package containing all of the logic uh for the PDP protocols and a lot of other things and when we split up that system we kind of had to make sure that like there are a lot of interdependencies between the basic protocol implementation and the like high level objects within the system so that's why it's so confusing but ultimately I mean it will invoke run pier and um run Pier I can show you is back in the um other um package like I said actually we may have it in the a yeah because we have for eth and snap we actually have it in um one second I need to click again yeah so this is where we have handle not really sure where we have run ah here's run Pier okay all right so we have run Pier which will defer to this run e Pier method and um let me see where that is I don't have that damn it yeah sorry I'm not sent in in this in this window so I won't really be able to show you but um yeah so here we go it's back in this in this file so this is kind of where like the this is where every connection will ultimately end up as well in this function so this is when the basic cryptographic hand check between the two peers has passed and they have started communicating basically the the peer was either discovered in Discovery order it has connected to the local node uh uh as an inbound connection and it is running the eth protocol and at the compatible version then we will end up here so this is like the common case so here the first thing we kind of have to do is that like if the if we detect that the pier is also running the snap protocol we kind of have to synchronize with that because snap kind of requires this like it's it's a what we call a satellite protocol it doesn't actually it does it cannot stand on its own it requires the E protocol to to function as well and then here we will we will perform this very basic handshake so we will basically check that the node is running a compatible blockchain um by uh um checking that the information in their status message uh uh matches with with um with ours and um I will try to go to one second so if we so here this is this handshake function so we just we are just now here in this flow where basically once the connection is created we end up in this function and here is the handshake function it calls this one so uh and then here basically what you can see it uses the basic PDP apis so uh we will simultaneously uh send our own status U message containing the information like uh the protocol version network ID total difficulty is no longer used so we just send zero there our current head block the Genesis block and the fork ID and then we will also wait for them to actually provide the status back to us and here we just wait until both of these things have happened or there's a basic timeout so the handshake timeout I'm actually not sure is about 5 seconds so if they do not manage to send their own handshake within five seconds and we know that this is an invalid connection and we will just disconnect uh it so and we will disconnect it with this disconnection reason so here you can kind of see these reasons also used so there is actually a reason for this case so it can be detected in theory by the other side or we can uh I'm not really sure when you would want to detect this but either way it is detectable and then also I mean here you can see that these are all of the checks so we will read a message from them and we will check first of all that it's a status message we will also check that the status message is of like it doesn't go longer than a certain size because there is a there is a limit on that as well um it has to be decodable uh from its basic encoding and the network ID has to match the version has to be the correct one the Genesis has to match and the fork ID has to also match so all of these things have to match in order for it to be considered valid and if any of these things is violated uh we will end up in this Clause we Mark appear is invalid and we will disconnect it so this is how uh this kind of logic Works anyway we go back to this flow now and you can see that now uh we will basically go back to the synchronization with the snap and we will just do some further check to ensure the snap protocol is also launched because the protocols are kind of initialized concurrently so we kind of have to make sure that it's all it's all working and then yeah we have some further checks um ultimately we have to basically register this pier within a bunch of subsystems of G so for example we have a basic map for the peers but we also have the downloader which is responsible for sync and it keeps track of the pier set in its own way so we have to register it with this as well if it's a snaps Pier we also have to register it with that and then we will actually go into this flow where we start synchronizing the transactions so um swing transactions kind of the entry point into this um I'm not 100% sure where this is defined but I guess we will Discover it quite soon um yeah it's defined right here um so what you can see is that it's actually quite simple so what we will do uh upon the initial connection is we will um request from our local meol all of the current pending transaction hatress we collect them into this list and then we just sort of start uh sending it to the pier and um I'm not sure where that's defined but I can show um I guess I could show yeah I guess um this is kind of where you do it and this is where you start seeing that there's a lot of just like there's a lot of stuff here where like the implementation is kind of sort of like it's kind of spread out over a lot of different places so even though it's like the code is supposed to be easy to read and has a lot of functions and in practice you will have to follow quite yeah I was just thinking about when we go move towards veral uh I believe then like go the we'll be passing around like veral witnesses as opposed to Lea nodes and extension nodes and Branch nodes to try to verify state for like a new node that's trying to sync uh just curious what are some interesting changes we expect to go ethereum to support that um I don't think there's going to be a lot of changes to be honest I mean the main thing is that the this would be then handled by the snap protocol so I guess the snap protocol would have to be adapted and um yeah I mean it's just it's going to have a bunch of new message contents maybe or we maybe be able to even fit it within the old because um at the moment uh we don't actually really Define what the trotes look like within the P2P protocol we just say that hey sorry can you can you give me this these these nodes and it gives you a bunch of bite arrays and then it's up to the implementation to figure out the validity of that so I guess this could fit in but you you may also be right in a way since the snap protocol does make a decision uh like it has a distinction between storage trees and account trees but within the veral um world there are no there is no distinction between between these things anymore so I guess we will have to adapt the protocol as well to uh remove the notion of stor trees eventually cool cool that's interesting I somehow assume that the wi the witness protocol will be used for like uh sharing the witnesses from uh from notes to stateless PES um in like stat right um then realize that it's false as you mean this width capability that is in the ri oh yeah so that one was kind of a we can quickly go over that so this was um kind of an early attempt like the witness protocol was supposed to be for this thing called beam sync um it didn't actually work so this was something that was introduced by not 100% sure I think it was by done by Trinity uh which was the python implementation or of of the ethereum execution layer but it I'm I think this never went anywhere like they added this protocol and and I was like sure you can have your spec in there but then it it it never actually went so in snap I can just quickly show this um in snap we have a bunch of messages including account uh what's it called get account range and get storage ranges so I guess this thing would be moved or we would have to figure out and then I mean here it's kind of like if you if you request accounts and you're going to get the basic accounts and you're going to get the proof and I think the proof is just a list of Bays so I guess it's same with veral maybe or I don't know like either way I mean this is something that can be changed thank you than pH yeah any any more questions um still a couple more minutes to go uh please use the time to ask something can learn we some time there but uh let's utilize the rest there is something comments what could the potential addition what what could be potential addition to additions to thisk V5 so maybe like or more in general like what are some advancements some improvements that U that you can think of and or generally here what is being worked on yeah so um at the moment uh we are kind of working on the next protocol version for this discovery version five version 5.2 um there's a couple things there so we want to we want to uh improve the protocol with uh a new mechanism for Nat hole punching so this is like something where if if you are within a restrictive Network environment um like that has a a net then we want to be able to we want other notes still be able to connect to you and this can be achieved using that hole punching and we have strong strong feeling that this is something that uh should be within the discovery protocol because the discovery uh protocol is kind of it's in a good position to do this and um another thing that uh we have been thinking about in dp5 is uh improving a little bit like some basic crypto but then there is the the the much bigger question of for the portal Network that I mentioned earlier they are currently using a bit of a weird mechanism to actually transfer their data so we have started to uh explore some alternative mechanism by which we can have some kind of let's call it like a like also kind of like subprotocols or capabilities like we have on RPX like to have this type of stuff also for the discovery but more for the purpose of transferring like large amounts of data like uh historical blogs or things that they want to store in a portal Network so this is something that is ongoing right now they are looking into this we just had an event with portal Network last week where we discussed these things in detail and it's something that is ongoing you think there's a world in which portal replaces the P top no I mean the basic protocol could I mean it's not so much I'm not sure like I mean portal itself is kind of a it's a different system and I don't think there will really be a replacement is more like they are trying something with the I mean they are building a real DHT containing the blocks and so on and that's something that's useful for sync and so on but and for long-term storage uh of of the ethereum data but I don't think it will be the only place where that data is stored it's just one of the places and I mean it's good you know yeah awesome yeah we at a full time here so let's wrap it up even though we lost a couple of minutes here I'm sorry about it again FKS but U I think we can can uh cleanly wrap up here uh we learned a lot thank you so much Felix it was really great very educational uh great dive into into specs into get code uh yeah was was amazing um really appreciate your time and everybody who joined uh if you have any further questions you have any things to discuss please use the thread in Discord we Shar the materials from the talk all the repost all the links are shared in the in the Discord so you can you can look in there to study more yeah and uh that's it thank you so much again everyone hope you enjoyed thank you