libp2p | DappLion | Lecture 19

Once again, thank you so much for joining everyone, especially the plan. Thank you so much for being here. Really appreciate having you here. Um, so we have now a networking week where we had on Monday Felix covering the execution layer and now we have uh dep uh on the consensus layer networking or the lip peer-to-peer protocol and uh uh its enhancements. And uh I'm very grateful for Lion to be here because he's one of the best people to be able to give this talk.

He's an expert on consensus layer before was leading the uh uh development of loadar. Now he is part of lighthouse team and other co's client and uh he's mainly focused on peer which is uh um big part of it which is the networking. So uh yeah he's really expert. So thank you so much again for being here and yeah the stage is yours if you want to present. Yeah.

Yeah thank you so much for having me. Yeah, as uh Mari said, I'm currently developer at Lighthouse Consensus client of Ethereum and since I joined of January last year, I've been focusing mostly on sync and Pas. So lot of networking from my time at logar. I had to become very familiar with all the specs and I implemented uh I think we we did a uh reimplementation of this V5 partial uh also of course so yeah I'm going to share let's see how do I do it here start screen sharing the window this Okay, let's tunnel vision. Uh let me let me know if you see a blackboard.

Okay, perfect. Okay. So yeah, I think for this talk I wanted to basically start from the basics uh just give a basic overview of what's going on uh on P2P on Ethereum concentric specifically what are the basic concepts of LIP2P and then from there I want to jump straight to the specs because uh in my opinion the lipop specs are very well written and I they are very approachable. So I would highly encourage anyone who used to work with them just to give them a read and get familiar with them. Same with consensus specs.

There has been a lot of uh talent and effort to make them understandable and approachable by a very wide audience. So, anyone that wants to work with Ethereum or understand the thing, if you can go to the specs first, that's like the best possible source of data. It's always up to date. It's the most precise. Any any other resource that derives from the spec could be outdated wrong.

So, it's always good to just go to the specs first. And in case you need some help to start digesting them then it's then it's good to look for alternative sources. So let's start uh for the very basics. LIP2P is uh kind of a competitor to dep Mario mentioned that there has been uh uh Peter here talking about the execution stack. uh dep structured relatively similar.

I think both of them started in the same hackathon of the same years but they diverge slightly on their approach and lip to be end up I think having a more uh mature comprehensive codebase and approach and that's why it was chosen for the Ethereum consensus because at the time when the beacon chain got speced it it was an independent chain so we had the opportunity to choose a Neo networking stack. But at the end of the day, um most of the things are relatively similar like we have the same need of moving data from uh peers to anothers in a reliable way that scales and it's resilient to attacks. So the most basic thing we need to do in P2P uh consider someone proposes a block and then we have a very large amount of peers in the network. We want this block to propagate through the network as fast as possible to everyone assuming that not everyone is connected to everyone. So we have some sort of peer graph peer mesh.

So let's consider we have one pier here. It's going to be square. I'm not sure if Scolro has funny figures. I think so, but I never I never know where they are. So, the first thing these two peers need is some some way to move data from one side from the other.

And in terms of terminology in lipb, that's that's what's called a transport. So we need some underlying way or baseline protocol to move bytes from one side to the other. In Ethereum case, we do that over a TCP channel. So I would assume at this point you know what TCP is. Uh otherwise it's it's a protocol built on top of IP with order bytes.

Uh so the the the other alternative we could use is uh UDP which we will talk on uh on a second but at least now for uh block propagation we use TCP. So now that we have TCP we have some way of moving byes from this pier to this pier but we need more things like we we are not only propagating blocks but we are also doing other things like identifying the pier syncing other stuff. So then the next step uh that we need to do is to make this connection secure. So now this this TCP channel is uh unencrypted. It's running plain text uh which is not ideal and if we have some men in the middle attack we can get compromised whatever.

So the first thing we want to do in terms of um lip terminology is upgrade this uh transport. So what we would do is construct on top of it uh an uh an encrypted transport and we have a bunch of uh protocols that we can that we can choose. In the case of Ethereum, we use noise. And what noise does is a basic handshake from one party to the other. So when you actually connect from this pier to the spear, the first thing you need to know is the IP address to start up this TCP connection.

And then as a second point data point, you have a public key. And with this public key is the one that you will initiate the the noise handshake. So uh we will look at the details now but uh it's I think it's a classic Dy Helman um handshake where you know the public key then you come up with a secret you agree on it and then you upgrade the connection to uh symmetric encryption uh channel. Cool. So now at this point we have uh a connection where we can send encrypted bytes but nothing else.

The next thing that will become very important as you see now is we want to have different protocols going on at the same time. We want to have some popsup protocol to propagate messages. We also want to have some protocol to to sync or to do pings whatever. So what we will build on top of it now is a a multistream protocol such that we can have in the same channel uh different streams of data such that the application can say pipe this bite of data into stream number 43 and then this this channel uh just pumps this data to the other side using basically reusing the same connection. So in uh the web to world in classic land uh TCP connections are not that expensive.

Well, it depends on how browsers are implemented but sometimes in some websites if you need to get some data then the the website will just do another HTTP connection to go to whatever API and get an image or something. uh but if you want to do that then you have to establish a new TCP connection and you need to find a new port you need to negotiate a new HTTPS handshake so that that can be a significant amount of overhead if that has to be done over and over and so in the case of li2p we we choose to get rid of that inefficiency and have one single transport like one single TCP connection with every single P just one and then all the data that have to change regardless of protocol it goes to this one center. So what the protocols we have for multiplexing are mplex and yamax and then slashqu which quick is not technically multiplex channel quick does everything uh but I'll I'll touch on that later like for simplicity let's assume that it's not 2025 anymore we are in 2020 so quick quick is not here yet so we we could choose between endless and yamos but for practical terms they do the same. Cool. And now now that we have a ro transport which is encrypted and it's multipplexed now we can start to do uh user application streams.

So in the case that we were talking about where we want to propagate the block, we would start a new protocol here which would be gossip app and uh gossip allows us to propagate uh arbitrary messages in uh in popsup. So from the application level we would be like uh your client dot gossip dotpublish the message and then we would reuse this existing TCP connection open that is encrypted multiplexed find whatever uh stream corresponds to coip sub push that message and then it gets to the other side and then on top of it it could we could add now a new substream different to gossip for for example blocks by range B2 and then ask send a PR request and be like hey can you give me blocks from slot one to slot 16 please and then we'll give back blocks and look at them cool uh let me go back here and ask you Mario do we want to do questions at the end or in the middle I would be glad to stop and I was just curious um why not TLS 1.3 like why specifically are we using noise for Ethereum like uh isn't TLS more widely adapted? So, yeah, I was just curious about that. Uh sorry, you mean TLS?

Yeah, that's a good question. I don't know now on top of my head. Let me check. There's a in the specs. So, now I'm in the consensus specs uh P2P interface and then we have rational here.

Why are we using noise? Like I don't know now, but I would say probably because TLS may be doing too much for our DS and noise does exactly what we need and nothing else. Like that has been the rationale for some protocols that we use like uh SSD. But as I as I said like a a way a way that Ethereum tries to adopt some existing technology for this purpose is not TLS itself but Quick and I'm not sure if I don't know if Quick uses TLS under the hood or if it's a its own encryption protocol to be honest. Um, as far as I know, I think u DLS is generally used in client server models and u I don't really have much information about noise but I think quick also uses TLS under the hood as far as I remember but maybe it's peer-to-peer systems and maybe that's why noise is maybe preferred but I'll have to give it a read.

I was thinking that you must be knowing something about why it was picked specifically. So yeah, thanks. Yeah. Yeah. Sorry I can't give context.

Yeah, no worries. Yeah. Hello. Um, hi Line. My question is, um, somewhere where the noise is, I feel like it should be discover.

Um, so how do they interact? Uh, okay. We jumped. So you're talking about this V5, right? Yeah.

Yeah. This V5 uh it kind of breaks this a little bit because it's it's separate thing. So this vi from the point of view of lip2p is actually an mostly independent protocol because from the point of view of lipop uh peer discovery is happen somewhere else and then whenever you get the peer then the whole lip machine of legos happen. So in with this schematic in the case of this B5 it would just be a different connection on a different port over UDP and then on top of this we just have the whole this V5 protocol like no no noise nothing just this 5 on its own and then I think that this 5 protocol by itself is not encrypted but the messages are authenticated by uh signing Okay. So, they just run in parallel.

Uh, yeah, correct. I got it. Thanks. Okay, awesome. Yeah, thanks for touching on this five.

like we're going to get there eventually, but for now we're going to pretend that discovery is magic and we just get peers and then the whole stack here kicks in. So again, I want to read the specs and now I'm in github.com/lippy2bspecs. Everything is really datedly speced out. uh most of the basic the basic concepts are here and what what I want to start with is this.

So the connection life cycle is exactly this this structure I was talking about where we start with a raw connection then we upgrade into an encrypted channel and then we upgrade into a multiplex channel and then from there is where we can open streams. Wait, I have way too many tabs here. Yeah, look here as an example, we're using TLS. So, in this example, uh yeah, another another important point. Everything goes through whenever we want to do an upgrade or even do uh a user substream, we use uh handshake protocol.

Not exactly handshake protocol but this multistream protocol which is something very basic. Whatever the the initiator asks the responder hey I want to interact with you with this list of protocols which one do you select which one do you support? So to start with it would open the TLS connection and then it would uh ask the PR to do an upgrade and it would say hey I support TLS 1.0 and noise 1.0 zero then the other mode will reply yeah let's the noise and then with the noise and then on this upgrade has happened then we will do the mlex upgrade now going through the upgraded channel and do another round hey do you want to do I I support mlex I support yam which one do you support blah and then as a substream the same I want it just asks to open a gossip substream and then the responder will agree and then we will open the stream.

Cool. Now let's go let's go to Emplex. I I'm not sure if I want to go through noise but Emplex is actually I think quite simple and interesting to go through. So, Mlex and basically any uh multiplexor protocol in itself it's quite simple. The only thing we are doing is we are wrapping the messages of whatever underlying thing we want to send with some header that we attach to a specific substream.

So here you can see the the format pretty simple. We have a header with with an ID. And here we would basically do a new stream. You want to open a stream. Then we send the messages, close the close the stream and then reset the stream.

Like this terminology of close and reset uh works because you can have uh a channel where both ends are open. So it's it's constructed in a way where uh one like the initiator can send data and the receiver can send data. If you close the stream, it's only one directional and then when you when when it gets reset, you just uh close both. So yeah, as as you see the spec is super simple like the uh the annoying bit of implementing this and uh maybe we can touch on it later. Uh these these things are the mind for memory bugs and you want to implement them in a way that they do as little copying as possible because we we receive a lot of data through here.

uh also any of these protocols are uh facing adversarial uh players. So it's quite easy to screw up and then yeah they they can wreck your node and this has happened in the past. Okay, the the protocol I want to spend the most time with and I think we should go with now and then we can take a look at the rest is gossip. I think osis is by far the most interesting one and I think the most important. Uh also a lot of research regarding uh pure das and scaling is on gossip sub itself like now we have different proposals to improve this also gossip 2.

0 protocol. Uh so starting from basics where we were talking here about this example gossip is a popsub protocol. So it's it's a protocol where you push data into into users. Let's do a new one. So let's consider this example.

We have one pier, two peers, three peers, four peers, five peers and then they are meshed in this way. So now what we want to do is we want to make so the beer on the left would publish a message. It's a publisher original publisher and then we want this message to propagate through the network as fast as possible. If we just if the only thing we want to do is to propagate the message as fast as possible then the best approach is to spam. So this this spear will send the message to this one, this one, this one and then this one will send it to uh well this one because it already has but it will send to this one and this one and then this one will send it to this one, this one and this one from this one.

So what we end up with is a a massive amplification factor. So that the message if we have a very high peer count say 100 you could receive the the same message from 99 peers. And then in terms of network as a wall the bandwidth consumption for papsub is astronomical like if if we have an amplification factor of 100 and I don't know like I think the the end goal for ethereum is to have these massive blocks of 32 megabytes. So it means that the whole network is like spending this bass massive amount of bandwidth which makes it infeasible to to have uh nodes at home. you will need some some data center.

So this amplification factor is one of the the key things that gossip uh tries to limit which is approach and the way it works is in gossip we have uh established connections with all our peers but we choose a smaller subset of them and we build something that's called a mesh. So in this case, this spear would randomly select uh according to scores and other stuff this pier and this spear as its mesh peers. And at the end what a mesh pier means is that we would push the the message to the pier only to the mesh pers. And then these two and these two maybe only have let's say only have this one mesh pier. So then it they will push to this one.

This one only has this one mesh beer. And let's say as an example that this beer is not meshed with anyone. So it doesn't receive the message. Now what happens is the the name gossip up comes from uh the word gossip. So gossip is like when you say ah this person blah blah blah.

So the the you push the message directly to your matchps but to everyone else you would randomly send a list of messages that you have. So maybe this beer or this beer after some milliseconds they would send a list to this beer saying hey I have this all these messages and then this spear would be like oh I I don't have this message which it he doesn't know it's a block but at least it knows it's some message it doesn't have so then this pier would reply to this one and say hey I want this message please send it to me and then the peer would reply with the message and then this peer gets the message. So thanks to this mixed approach, we have very fast distribution of messages through the mesh and then the gossip protocol. So the I have I once basically cover everyone else in case there's partitions or bad tiers or whatever. Um so going back to the amplification factor in gossip sub it's scoring as eight.

So on average if you have enough good peers you will just publish to eight and then the next peer would forward to eight and the next forward to eight. So uh this this behavior is exponential. So in just like a few hops even a network of tens of thousands of nodes get the message and then everyone else would just gossip the I have I once list and and get the rest. And now we'll talk on a bit more on security features and how we prevent against attacks. But the lip team did a bunch of uh research and studies and they found that with with this protocol even with very very adversarial networks like I don't know what was like 80 or 90% adversarial nodes you still have uh decent propagations within seconds which is really cool.

Oh, and I think I think I haven't stopped for questions when we were talking about the the P2P specs. So, I'm going to stop for a second if anyone has questions about uh Nlex and so far goes out. Um like you explained that we have a different uh we use discovery 5 protocol, right? So in this u how are we uh are we are we connecting to discovery 5 as a as a layer to get to know about about our peers in here or like do we have our own ambient environment or how how are we discovering nodes here? Yeah.

So from the point of view of the lip spec uh you can do whatever you want basically. Then in practice most consensus clients they discover almost every every single peer they have through DSP5 but I think some of them have a discovery protocol withh I don't know the name but there's there's a way you you can you can find uh all your IPs on local host and then try to do a discovery with them. So that would technically be a a separate uh discovery mechanism that runs alongside lipop and this v5 and this one. But okay. So the gossip gossip sub needs to like discover the nodes right.

So it is using disk 5 for that. Yeah. Correct. Yeah. Yeah.

Okay. And like um if uh like are things random or we are trying to use some distributed hash table or something of that sort of an algorithm for this discovery. Yeah, exactly. So this V5 works with a cadmilla table. Uh and you okay yeah what you do is you you do queries to to peers randomly and you try to walk this table and discover as many peers as you need.

Uh so it it's possible you could you could walk the whole table and discover all the peers but that's a bit wasteful. Usually what you do is you start from a boot node and then from there you would query some random set of peers and then if you if you need more then you just keep going. But after you have basically nodes usually have like a 100 peers. Once you have that the set of peers like usually you stop doing this discovery until you need more peers for whatever reason. Yeah, got it.

So like because we have a DHT we are we are sort of u not really that problematic when it comes to the churn of things. Yeah, got it. Got it. Thank you. Yeah, exactly.

So I wanted to ask like you mentioned there are two multiplexing protocols but like why have two? We can just have one so that we can also reduce one hand shape asking which protocol to use right uh I think you asked why do we have to multistream protocols right multiplexing yes yeah uh let me remember why like everything has a reason uh but there are so many things and so many reasons it's hard to keep up like sometimes we are not uh in secret but I think for M. I think MPLEX is a custom protocol developed by LIP2P themselves but it has a fundamental issue where it cannot handle uh back pressure. I think I think yeah I think that was the the main the main issue. So let's let's touch on this very quickly.

So consider that uh TCP is an ordered protocol. So if if the if the PR doesn't receive the the packet number 100, it cannot receive the packet 101. So depending on how all of these protocols are implemented here, consider that uh the concept protocol sends uh a lot of data and then this PR doesn't consume it. So in in terms of a stream you have pressure from one side like you send send send and then the pier has to receive the data. If it doesn't receive the data the data uh backlogs into a queue and you you don't want this queue to be infinite otherwise the client would just run out of memory and explode.

So usually they have some queue let's say one megabyte once this queue is full then what do you do? So in case of mlex there is no way to communicate back pressure back to the sender. So it will just backlog the the the underlying transport itself. So if if gossip subap sends a lot of data then the receiver doesn't uh yield this data. All other protocols will get blocked uh which is kind of uh pretty bad.

Then Yamox doesn't have this problem. It has a way I think the way it works is you send window updates. So whenever you receive data you tell your PR back like hey now you're allowed to give me um one megabyte of extra data and unless you give unless you get this allowance you're not you're not allowed to send more data and that way you can communicate back pressure on a substream uh level so if gossip app is very slow then gossip sample stop working but everything else can still send data then what happens is yambox is very complex so when we started on I think mlex was the preferred option because it's very simple implement. So I think the issue was like teo and loadar and I don't remember who else only had mlex and not yambox. So we had to support both at the same time.

Uh and then basically everyone took so much time to have a stable yamok implementation because yeah again like these protocols are the the spec is simple but the imple the implementation is very complex. So the Yamok implementations were not stable for a long time like years. Uh so we just kept rolling with two and then quick came along and everyone was like hey yeah whatever why why do we have to bother with Yamox when we can just go to quick. So yeah that's the quick summary of why we have to so finally we are going to move this to quick. Yeah.

Yeah, I think I think that's the plan. But again like uh I think quick quick is has not well support on all the languages. So it's we will have to basically wait for those implementation. Uh I think JavaScript is an issue here like their quick implementation is not very mature and I'm not sure if another language. So if we basically deprecate all of this and don't do quick then the load the the load start client uh is is dead and we already have a working implementation.

So it's kind of kind of a dick move to kick out a client. Uh so unless there's a a critical vulnerability on the protocols, it's not too bad to let them run. Um yeah, my question is uh could you please tell a little bit about I don't want and uh if it's adopted already or not and uh I wonder uh how uh clients update libraries. So do is it a must to have the simultaneous version all across the clients or or or not how it happens? Yeah, really good point.

So I would say if you can can you ask this question again after we have gone through the gossip subspec because I wanted to go if you have time through gossip subspec 1.0 then 1.1 which is the scoring one and then 1.2 two is the introduction of I don't want so we could talk about it uh then okay awesome so the big boy gossip sub 101 Uh yeah, just to recap, Gossip is the popsup protocol that we use in in Ethereum. It works with this dual architecture like where we have everyone connected to the to all the peers, but we built a mesh with a a small selection of peers such that we have fast propagation with few peers.

But because of the exponential effects of this propagation in uh optimal network conditions, the message reaches everyone in a few hops. And then to to supplement that we have the I have I want uh control messages to exchange these message lists and make sure that all the messages get distributed eventually through the peers. Cool. So all of this is explained here. Here we have the messages we were talking about mesh dependencies.

Yeah. So as we're talking about this relies on ambient discovery which basically means somehow you find peers. In our case it's this V5. Uh but yeah then these parameters the one that's more important for us is the D. So the D is the it it refers to how many peers on average like what's what's the target number of peers that you forward a message to and it it also uh determines the the total bandwidth of the network because in so in case of PRA for example we rely on PR on gossip sorry heavily for sampling and the main issue is that whatever sampling we need to like how much data we have to download to check that the block is safe.

Now we have to multiply that by eight because of the cool. Okay, let me go through the control messages first. So all of these mesh architecture we're talking about it it's not global in all of gossip app but instead is per topic. So in this like hierarchy of things, gossip sub is a sub pro like a substream or a protocol within lippy itself and then within gossip sub you have another degree of uh differentiation which is the gossip subtopics. So within gossip we can have the I don't remember the name but blocks the blocks topic we can have the beonization topic and each topic has one message type for it.

So whenever we need to publish some new type of data say uh blob cipher or slashing then we would use a separate topic. Each topic has its own message type its own validation rules and each one of these has a distinct mesh. So maybe for the blocks topic this pier could be meshing with this one and this one. But for the beacon at the station we could be meshing for with a different set of peers. So when when we start to to form this mesh, this mesh is birectional.

So let's let's go back to the start. So this spear is just connected and it has zero mesh pairs. But because zero is less than eight, it wants to have more. So then it will go through the list of peers and be like, "Oh, you and you have really good scores. You look good to me.

So I want to mesh with you and then it will send a graft message and the graft basically tells the pier. So this beer would tell this pier, hey now you I'm I'm messing with you just let you know and then the peer can either do nothing and then the mesh is established or it can send a prune message rejecting this uh this meshing. Also you can send a prune at any time. So if the P is misbehaving or doing things that you don't like, then this PR could tell this one, a prune, you're not no longer in my mesh. Cool.

Easy. And then then this is the these are the the modes here. So I haves are every gossip. So I think it's every harbit. We we will talk on that later.

The pier looks at all the messages it has in its cache and then it chooses a random list and it just sends this list random to the pier. So it would tell hey I have these 500 messages like just just the hash. Then the peer checks against its own cache like do I don't have any of these messages. If it doesn't, it will send back um I want with the list of messages that it wants. So um all of this works because we have this uh scene cache.

So we have a data structure where we keep a list of message hash and the message itself for some period of time which uh I think it's determined by the spec not here but here so now now I went back to the Ethereum consensus specs uh phase zero P2P so phase 0 is the initial phase of the network uh most almost everything spec here like in the the way the specs work is this is the fa the first fork so the genesis one and then alter also has a P2P interface but this one only specifies the new things and almost all these like basic directional and rational questions are answered here on the original P2P so here we should be able to see do we speculate the D or we on maximum clock [Music] figuration and second go submain the ah yeah yeah yeah okay so for example the the d is eight in ethereum but in the default lippy spec is six so all also always the spec prevails then the heartbeat interval is 0.7. So this is the frequency at which we compute this I1 lists and we send them out. Again there's probably a rational for this. I I think I remember this used to be 0.

5 and then we changed it to 0.7. There must be a reason somewhere. Uh probably someone did simulations and then they figured out that this number is better uh to have less something and then the yeah the mcach this one this one this one this this is the important one so expiry time that we of of seen message which determines how number of let's go Yeah, these messages are the ones that determine how much messages and data you keep on your cache. So this directly relates on the memory consumption of the protocol itself.

Cool. Okay, I'm going to stop again for a second. Any questions so far? uh what happens when we lose a mesh pier does it like connect to a new one automatically or yeah really good question so for example let's consider that this pier I don't know their their computer sets on fire and then we lose the whole thing. So when when you lose the TCP connection it will unravel everything up including goips up.

So for from the point of view goips up it would get like a disconnection and then this spear is gone. So now this spear is under the the D level in this case we are setting D to2. So on the next harvest, so the harvit is not only a moment where you do the I I have lists but you do a lot of uh maintenance tasks. One of one of them would be uh let let me see for the blocks topic how many mesh pers I have. Oh, I only have one.

That's not okay. I want two. So then it will uh we can check the code later but basically that there is some privation list of where do you go to to look for mesh peers first. Uh let's say for simplicity you you check your peer set you you see which peers in my peer set have good score and are subscribed to the blocks topic. Then you would randomly choose one and send them a graft.

And then at that moment when the graft is acknowledged uh so because basically the peer doesn't send a run back then you go back to having two peers two mesh peers. Uh yeah, I wanted to ask that on what basis this score is being determined and uh like where is it kind of stored and that what essentially determines your score? Sorry, I the audio quality is not excellent. I I couldn't fully grasp the question. Are you asking about when peer scoring is necessary and when it's not?

I wanted to ask on what parameters do we decide these on what parameters do we score the peers as to whether they are a good pair or not. Okay, really good question and we would answer it extensively with gossip 1.1. So hold on for a second and we'll touch on that because gossip 1.1 is literally all about scoring.

Okay, good. Uh so hi actually I wanted to ask like you mentioned that the topics and uh as nodes we are subscribed to topics. So is it optional to subscribe to some topics and some and some not or is it compulsory to subscribe to every topic like how is it decided right really good question so on Ethereum we have a list of topics uh let's say for phys uh here so even from the start we we already had what we call global topics And global topics are topics where everyone should be subscribed to. And then the attestation subnets are topics where only interested parties would be subscribed to. So let's say for the global topics the beacon block every single participant in the network should subscribe to the to the beon block.

then we specifically nowhere in our code as far as I'm I'm concerned we don't enforce that to be the case. So you are not required to subscribe to to the beacon block. Uh however if you don't subscribe to these topics we will see now like in in gossip sub 1.1 you have something that's called the mesh score. So when when when you do bad things you get a bad score.

when you do useful things like giving us new messages or giving us valid messages your your mesh score goes up. So if you don't subscribe to anything at all then your mesh score is zero. So let's say that you have 100 peers and 99 of them they are subscribed to everything and they give you valid messages that eventually their score will go up to some value that's more than zero and the peer that's not subscribed to anything will will remain at zero. Whenever you you do pure rotation this pier will just be kicked out because it's the pier with the lowest score even though it's uh not negative. So you you we can see that in clients whenever we have a network scraper or some tooling like this.

Uh you you can do that you can just connect to a to note and so you have to have a gossip protocol that that's true like if you don't have gossip app uh then you will get kicked out immediately but you could have gossip enabled and don't subscribe to anything that's fine that's legal as per the spec but you will probably not survive for too long. Okay, that answered my question. Thanks. Sure. Hey.

Yeah. So, I'm my understanding is that it can be quite large blocks and I'm just curious if like you know for the last like 8 seconds of a slot um all the nodes need to be propagating their at the stations. Are we seeing issues in like latency and uh you know what are some ways that we may need to mitigate this in the future because you know they're they're gossiping these uh attestations like at many many uh pattern right so sorry so your question was if we have uh propagation issues with the decisions themselves under uh a limited box or like some limited hardware. Yeah, it's like for like large block size like that I would imagine be difficult to propagate between the nodes, right? Like across a distributed network around the world.

Yeah. So uh at the stations themselves are actually uh already scaled in a I'm not sure what's the term but like a quadratic network. So this the this pattern as I I explained where all one message like the block goes to everyone that applies to I think blocks and basically everything except for attestations sync committee messages and now u the blobs or no okay let's focus on the stations so the as you say at the sessions there are a lot of them so we don't have a one single decision topic because that would be crazy. Uh you would be getting like 30,000 messages per like per slot but they all come in at once like so it's too much. Instead we have 64 channels individual channels.

Uh so let's pick let's say channel 04 will only get uh 164th of that traffic. So a much much more reasonable number let's say 500 attestations. Then these 500 attestations go through a specific mesh into a type of node that's called an aggregator. And then this aggregator gets these 500 messages constructs an aggregator at the station and then this aggregate station oops is the one that we actually uh propagate in a global topic which is this one beon aggregate improve. So everyone only has become great and then this this channel has way less traffic like 64 times less traffic than the sum of all the beaconization topics.

So it's much more manageable and that's why we have these two steps. So in we we have the block then 4 seconds at the stations but 500 on every subnet and then the aggregators aggregate and then 500 messages on the global topic and this this scales really well like we have never had issues with the stations so far like blocks are the annoying thing always because it's like it's this one thing that everyone has to get as fast as possible that which is really and everyone is playing me games and timing games. So we have a limited time and then the decisions are played but the attentions itself has been has been doing great to be honest. Thanks. Hi.

Um I hope you can hear me. Uh so first of first of all the gossip sub version 1.1 does have evaluation report and it shows the results why 0.7 was chosen. Okay.

But my question, what I really want to ask is uh it might be quite new, but for a heartbeat of 0.7 seconds, uh when the heartbeat fails, uh uh do I tell everybody else in my peers that this guy has dropped off the net or I just simply do not send the message or to this uh peer anymore? Huh, that's a good question. So I would I would say the harbit function shouldn't be fable like it should be impossible for it to fail like it should only fail under some weird internal error. Uh and again depends on how you architect it.

But what you could do is you could have like a loop on a thread where you just wait until the next uh tick of 0.7 seconds and then run the hard function. If it fails you lock an error and then you just try again. So if if it's some weird internal bug that only happens once then that harid would not send any lists of course because it error and then on the next one you will send them but I cannot imagine why would that fail like the maybe like a delay in the network or something network congestion and the response right I see so when when on the harvit you decide to send an I have message that message basically goes into a queue So that doesn't interfere with the hard function itself. You just queen for for later because we do something that's called piggybacking.

So we we don't want to waste uh we don't want to waste one message just to send a control message. So what we do is we um schedule the control message to be sent eventually and then whenever we want to send an actual message with data to the spear then we will piggy back like attach the control messages on it. Oh, and all these oper all these operations should be infallible. Like the only thing that could fail is the delivery of the message itself because the queue is backed up like what we're talking about with MLEX. Like if if the other peer is not consuming gossip sub then the message will just queue up.

But that's not a that's not an internal error. That's something that can happen and the application handles that gracefully. Okay. Thank you. Sure.

Okay. Okay. So I think there's there's a lot more details on 1.0 uh but I don't think they are as relevant as what we thought so far. Like just to recap 1.

0 the basics are this dual pier grid where we have the mesh where you you prune start the mesh sorry graft start start the mesh with the pier prune to remove them then do the I have advanced list to gossip everyone. So the issue with this approach just with this is that you you want to know who to to peer with. So peer scoring is absolutely essential. Uh you want to know who uh who's who's acting faithfully and who's not. That's why 1.

0 comes in. So 1.0 Zero defines a very uh detailed and specific way to calculate scores and then use those scores to decide when to do things. So when when do you prune, when do who do you graph with, who do you send the Ihavs, etc. So this function looks a bit scary, but it's not.

So the yeah the score function is this crazy thing but at the end it's just uh all these p1 p1 p2 p3 p4 p5 p7 uh multiply together by some weight and then added together. Uh we can go we can go over them because I don't think it's that complicated but some of them are penalties some of them are positive things. So, first one, very simple. You mesh with a to with with a beer. You mesh with them for some time and then you reward.

Basically, you're rewarding how long the beer hasn't attempted to prune you or you guys haven't kicked each other out. Then P2 is how many messages have I seen from this pier which are new. Like if you recall here when you have a degree of two like on average you would receive the same message two times. So here we reward the peer that send the message first. Uh I'm not 100% sure if that's uh exactly true.

Like I think some some implementations I think would reward messages that are within specific window. So if you get the two messages almost identical, you will reward both of them. But I don't remember off the top of my head. But this is basically to incentivize sorry to penalize. Let's let's assume that this pier is actually malicious and it's not doing anything.

If if this one sends the message to this one and this one and this one just forwards the message to to this one then we don't want to reward the forwarder. We want to reward the original publisher. So that's why uh the first receiver has value for us. So we want to basic we want to reward people that's that create messages not the ones that just forward them. Then the P3 is on.

So again this is not specifically for for libid2B but on on Ethereum some of the messages like for example at the stations we can predict how much traffic we expect because every epoch everyone has to do an atestation and we have specific number of validators we can compute how many atestations we should get from a topic on a specific rate and then we we use that variable to expect this number of messages from the peer. If you give us less than that, we will penalize you because what what a malicious peer could do. So this guy, bad guy, very bad um is mesh with us and then censor everything like do not forward any messages. Imagine that we have eight peers and all of them censor everything. then we would get like stuck in this black void and not don't get anything from mesh and have to rely only in the ives because of this P3 penalty everyone is giving us under the level so the scores will go down and then we will boot them out and get rid of the bad peers then P3B is is basically if you send a message that's invalid we would penalize you and you have half less Uh me wait no message delivery rate this a threshold for expected messages three parameter h true sorry so this is this is the one that you penalize if you don't get as messages the p3b let's ignore it for now it's a bit more complicated but it's it's a way to keep like make this score sticky if the pier disconnects then the P4 P4 is the one I was talking about like if you send us an invalid message, we penalize simple.

P5 application specific, we don't use it. P6 is to uh penalize or correlate uh peers that are within the same uh IP prefix because the idea is uh I think like most of the data centers and colloccated servers have the same IP collocation factor. So I'm not sure if this one is just for the whole IP is the same or if is of some IP prefix is the same. Like I think in Lighthouse we do both. But in a way it's it's to like let's say that you you have one computer and then you can have 1,000 peers in that computer and then just spam the network.

So what this tries to do is uh catch that type of attack and penalize it. And then the P7 is a special one. Like there are some things that you can do in gossip that are malicious. So for example, if if you give me an I have list and then I send you an I want, I'm expecting you to give me the message and if you don't send me the message within some time, I don't remember how that is. Is it spec here?

Well, there is a there is a parameter called the delivery deadline. Uh let's say it's 5 seconds. So if I give you an I1 and then you don't give me the message within 5 seconds, I would penalize you with a B7 penalty. Uh okay, I'll stop for a second. Any questions about scoring so far?

Where is the score stored? Sorry, the what? Where is the score stored? Like ah okay. Yeah, usually clients they maintain a a hashmap with all of these uh numbers.

So you would keep like a a statistics per pier uh to be able to compute all of this uh within the gossip protocol itself. Yeah, I wanted to ask like what if uh what is what if a client is offline and because of like uh these these I have control messages and uh if the client fails because of this like are there any provisions for those clients like who are offline or who are time or part of the network okay if if I understand the question you're saying if the client is offline time then it will fail to do the I1 deliveries and then it will be down scores right. Yes. Yeah. So yeah exactly that's what's going to happen and that's fine like you you you don't care if the beer is faulty if the peer is an attacker or if the beer is disconnected you want to have uh all the peers being honest and healthy and productive to you.

So it doesn't matter like if the beer is not doing the duties that you expect from it, you you have to kick it out. Recording has stopped. Oh. So is this score only stored by us or do we propagate this score to other nodes also? Yeah, the the score is always local.

It's it's it's your subjective view of everyone else. No, basically everyone is keeping a view of everyone else but it's only local. Okay. Okay. Thanks.

uh if the score is stored locally, how we can kick off the mesh just from from the local uh Yeah, that's a good question. I I don't think anyone persists the scores. I think so. When you kickstart the mesh, you just basically select a random beer and then it will eventually converge into the most stable reliable peers eventually. Okay.

So yeah, 1.1 introduces this set of uh penalties. Again the document is very extensive on the rational and I think I sorry I don't remember who but there is a a report from from lip2p that it it it went in very in depth on analyzing all these all these cases and it shows that one.1 with this set of uh scoring parameters it's it's quite reliable against very advers networks. Cool.

Let's quickly touch on 1.2. two. So, or actually one moment I want to specify the consensus specs are a lot of information that's super relevant to uh to networking. Uh as we were talking here in the spec it's very abstract about we have these topics where we get messages but we don't specify what are the message and what are the uh validation rules.

So that's that's what's happening here. So the way to read this is this is the name of the topic and then uh the following maybe yeah this is the the message type that we will send and then this is the important bit these are the rules like the validation rules we have to check per message before forwarding it. So if if we have an ignore, what that means is that we just throw the message away, but we don't penalize the beer. And a reject is we throw the message away and we consider the beer invalid. So we apply a P4 penalty, right?

Yeah. Uh we apply a P4 penalty, an invalid one. So just to make it more relevant, let's go to Electra. No fu follow where is fu here. So in fu we have a new topic the data column sidecar and here we have a lot of conditions.

So for example if the commitments are invalid that's considered to be uh malicious behavior that should never happen. So it's a reject because we don't want to propagate an invalid message and we know that uh the sender is malicious because it should never send such a message. But for example if the citeer is from the feature that could happen if the sender beer has a the wrong clock or something then we'll just ignore. It's not strictly malicious and we can just ignore the message uh for now. Cool.

Let's go to 1.2. too. So until now all the messages in gossip sub have been uh relatively small. So big stations which is the highest traffic the messages are tiny I think on the order of less than a kilobyte or even bytes and then blocks before the merge they used to be very small like a beacon block I think it's just like I don't know 30 kilobytes now with transactions they could be like 100 could be way more if they are bigger but it's not too frequent what has kind of break a little bit.

Everything is blobs. So blobs are this new data type that we have since uh the net. And the issue with blobs is they are quite big. I think it's a I never remember. 496 time 32.

So one yeah 131 kilobytes which is it's re it's decently big like let's uh if you if you don't have a very strong connection to download such a piece of data it takes um a non it takes some time like let's say half a second which could be sufficient. Okay let's let's draw this. Let's imagine this spear wants to this pier wants to send a blob to these two peers and this pier now is good and this pier is our target. Cool. So this will be progress bars.

So now let's say it takes half a second uh but this pier is on a data center. So let's say this starts to download on a home connection and then this is qui instant and then at this at this moment this other starts to download. So then we have some progress some progress. So we we're spending bandwidth spending bandwidth to download and then this one gets first but almost almost as this one gets first then this one arrives also. So now we have basically spent twice the bandwidth to download the same message that we already had which is kind of annoying.

I think I think a more annoying case would be if when this one is about to complete then this one starts to download. Uh we also have this uh other risk condition. So we could already have download this and this start sending the message to us. So if the message on the target has already been received then we have the scene cache. So we will just discard it.

So it's wasted bandwidth. What the I don't I don't want. Yeah. What the I don't want message wants to achieve is whenever we get the first message like this big one then we will tell everyone else with a very short message. So it's quite instant.

Oh, this is uh will be hey I don't I don't want this message like I already got it so please don't send it. So then then all these wasted bandwidth that we just had will not occur. It again it depends on the timing but for for these slower peers that have a lot of scatter sense uh different times it it has shown that it can save some bandwidth. Uh I'm not 100% sure but I think that the results are a bit disappointing in terms of safe savings. Uh but basically the idea is this one like the the moment that you get the first message you tell everyone hey I already got the message please don't send it.

Uh so you waste these double downloads. Uh so now Katya could you ask the question again um about the libraries? Uh yeah, is it one 1.2 is it already in use or or not yet? And how how clients upgrade uh the libraries are they compatible different versions or not?

Yeah, good question. So the way it works back back to lip uh when we have established uh transport encrypted multiplexed then we will try to open uh substreams all gossip sub is essential for everyone. So basically on startup as soon as you have all of this completed you will immediately try to open a gossip sub connection. So what you would do is you would do a a multi- multi- select sh I forgot multi multiream select yeah you will do a multi stream select protocol and you will send gossip I'm not sure if it's gossip or meshap uh b1.0 uh well I I think I think no one supports b1 but you will send b1.

1 and b1.2 2 and probably this order. So you you would you would be hey I support these two. I would prefer to use B2 if you have but I also support U1 and then the receiving client would be uh oh I don't support this so I will support this one. So then the client replies with B1.

So in practice what it means is clients support both at the same time. They have two H because this one is required by spec but they can support B2 on top of it and then whenever they connect with a client that has the same like this then both of them will select B2. If you find a client that only supports B1 then they will support B1. So some of the peer connections in the network will be B2 BB2 B12 B12 the other ones will be B1 B1 even within the same pier. Yeah.

Thank you. Uh yeah, I think we we got short on time, but uh maybe I wanted to touch briefly on 2.0, but I think that's probably too much. I think I'm decently happy like we covered all the basics of lip connection establishment, uh encryption, multiplexing, and gossip, which are the the most the most important protocols. Maybe very quickly I can touch on the other protocol that's pretty relevant which is a very small protocol.

So in in the specs we have two basic ways of u moving information. So the first one is with with gossip which is a popsup but then we have what's called the recresp domain. So rep rest is very simple. You you send a a packet like a request asking for something and then you get data back. So as I was saying here, gossip is its own protocol uh at the uh lip substream level.

But then we at Ethereum specify a bunch of other custom protocols which are these ones. So for example the blocks by range the full name is by range. Yeah this this one for example or this this is to get the status message. So just to check the pier. So this would be another protocol that we have uh at this level and these ones are are speced quite simply like this is the request and response or for blocks.

Yeah. So for for blocks for example when we want to sync we want to hey PR give me blocks at this specific range blocks by blocks by blocks by range. Yeah. So hey give me blogs in this range and then the peer goes like here here are the blocks and again we have a bunch of conditions that we have to satisfy. It's not as well written as gossip subap where we have the ignore and reject but at the same it's kind of the same idea.

We have specific rules that we have to abide by and uh yeah that's it with with these two things. It's most of the uh P2P architecture may maybe ignoring this V5 that we didn't have time to cover. I I cannot hear. Yeah, while we were at while while we were discussing like saving bandwidth like true I don't want control signal. So mean I don't want control signal like other control signals is it also piggybacked on top of on top of a scheduled message because like if that is the case then it would be like uh it would be ineffable because like the time delay would be for the schedule for it to be py right uh yeah I'll probably check the implementation but you're correct like we don't want to hold the control messages forever.

like probably what again I I don't remember the implementation off the top of my head but you could just wait for one harbit to hopefully have a message and after that har then you just send the control messages but I guess that that would be that's up to the implementation or maybe it's back I don't know test is it audible now yeah yeah we can hear you oh okay uh so I my second question I start I have two questions I like to start with the second one so the Second one would be for block transactions. Have anyone considered actually sending a preemptive message to decide whether or not to send rather than uh do not want message the other way around I guess. Yes. I think I think that's gossip v2. Like I only read the spec once uh but I but I think gossip sub 2 works in that way.

like yeah we're not going to go through it but um I remember that that is the case. Yeah. I see. Okay. Then uh my second question would be what else that uh uh what are the topics that we should have we should read about that is not covered today.

Uh what you should read about that is not covered today. Yes. Uh I would recommend the DB5 protocol if you're interested in that. We didn't have time to cover then app B1.1 report that we're talking about.

Yeah, this one the evolution report. Yeah, I'll keep a list of these links here and I'll send later. Thank you very But this is very I think this is fascinating to basically understand what type of uh adversarial environments we are defending against because the the happy case looks very good but this is what we want to protect against. I see. Thank you very much.

Uh when using recresp uh does the peer sends requests to all the mesh peers or maybe the the high score here or how it works. Uh sorry could could you repeat the first part? I didn't call it. Yeah. when using recrisp uh ah ah okay yeah that that's not specified and I it's up to the to the implementation in the case of lighthouse when we have to request blocks so we first try to figure out who has the blocks that we want depending on the the type of sync like lookup sync or rain sync and then we prefer peers that has the least connections possible to load balance and then it's random but you could you could have an implementation where you prioritize my score.

But we believe that then that could be gamed and you don't want to always ask the same peers. You want to have more random distribution of who gives you information. Got it. Thank you. Uh in the consensus specs there is something attestation subnet com.

So I wanted to ask like uh the attestations are like uh the subnet is created uh for particular block or for each block as well there are multiple subnets for attestations. Yes. Sorry I think I didn't got the question. Could you say the the last part like uh for the attestation for each block like is the subnet created for each block or there are multiple subets for each block? No, the the questions are sorry the attestation subnets are uh longlasting.

So even though you attest to different blocks, the at the station for block A on channel 30 33 gets sent on the same channel that at the station for block B but still at the station 33 because you want so to build this this this stable mesh it takes some time like building building a mesh is not an instant thing. It it's on the it's on the order of minutes not on the order of seconds. So we want to have a longl lasting mesh of of peers. That's why we we recycle the subnets for all blocks. So what if like uh like in the pre-built mesh uh if like that validator is not in the next step for that same mesh, right?

So uh the mesh is the message is mostly relevant for forwarding like when you publish you actually can send the the message to whoever you want because it's only one it's only the initial publisher. So there are two strategies we have now. The first one is flot publishing where you send the message to literally every single peer. Uh and that's what everyone did until recently. But now that we have blobs the the cost of doing flat publishing is too much.

So now now we just send to I think a selection of peers which is the mesh pers. But you could you could actually find a strategy that could be different or just send to randoms. If if you are the publisher and for some reason don't have any uh any mesh peers that's fine. You can still publish to to your topic peers. So topic peers is people you don't have you have not meshed with but you know they are interested in the topic.

Uh for the blog that's basically everyone for the blob subnets that could be more of an issue. Uh but hopefully if you have 100 peers you should have at least one of them um that that that is interested in that topic. Otherwise you're going to have issues. Yeah. Yeah, sorry if I missed it, but uh why are publishing to blobs more expensive than blogs?

Uh in terms of topics, topic messages. It's just it's just the amount of data. So blobs blobs are 101 kilobytes and we have at the worst case now I think six. So that's the much data and then blocks I think on average are like 100 kilobytes and also uh as the the previous question pointed out uh everyone subscribes to the blog topic but not everyone subscribes to the blog subnets. So that makes it more difficult.

That's why we have the get blobs B1 optimization. Uh not sure if it's going to show up like this. Get blocks. Yeah. So, we recently added a new engine API endpoint where the the publisher of a block can choose not to publish blocks and expect someone else in the network to do that for them.

And because of the way transactions work, that's possible. And it it allows it allows to save against this case. So if let's say that the original publisher is a homestaker and doesn't have enough bandwidth or no peers on whatever blob sidecar subet we're talking about. H wait, wait, wait, wait. I'm tripping now.

I'm tripping. Both everyone subscribe to the blob sidecars. Like it's it's also a global topic. Everyone subscribes to them. So my previous comment does not apply.

What what is the issue is you you want to propagate the block and blobs uh within 4 seconds before the the the attestation deadline. But on on a homestaker connection the up link is usually not good enough to do it reliably like it it takes maybe one or two seconds which is too too slow to hit the deadline. So what the strategy we do now is we publish the block first and then the blobs. But if the blobs are too slow to publish someone else in the network just with the blobs with so just with the block and access to amend pool they can recreate the blobs on the on their own and republish and that's that's what this optimization does. Yeah.

Thank you. It was great. Uh I think we learned a lot today. Really appreciate appreciate having you here. Yeah.

Awesome. Um sure. Thank you. Thank you for having me. Thank you so much.

Yeah. Yeah. It was great. Um yeah. And a lot more to study as well.

So, um, thank you so much and, uh, these emoji sounds are so enhanced, right? up