Protocol Berg v2: Mikel Cortes - The eternal research of broadcasting messages, limits of GossipSub

Hi everyone. Um so yeah so I'm working at probab uh we are basically focusing on metrics on the lip side of protocols with the idea of trying to make them better probably optimal up to the standard that we want to have right now. So the talk of today is pretty much a summary of the research that we have done over the last years with the main idea of seeing like where are we now at gossip which are the challenges that we are experiencing and which are the proposed roads that we can take in order to move them forward. Um the disclaimer this is my personal take so there might be many other takes so happy to have a discussion afterwards but yeah um for those ones that maybe are not that familiar with gossip it's pretty much the main way that liptop offers uh to broadcast messages over the network um is it a popsup uh like protocol which means that there might be many different topics but you don't need to be subscribed to all of them this allows you to disseminate and receive messages only on the things that you are interested at. Um, gossip is a a small um a slightly more efficient way of doing it like instead of subscribing and getting all the events, it allows you to have like a small representation of the network which is what we call mesh.

Um, and you keep a small mesh for each of the topics that you want to have and is the way that you are kind of saving a little bit of bandwidth there. Um it includes several features with the idea of uh making it more resilience in order to facilitate like civil attacks and many other stuff. Uh those are what we call control messages. Um and the main idea is that uh we achieve resilience only by not only uh eagerly pushing messages to peers that we are connected to but we do have also like a separate layer of lazy push where we notify others that we saw a message in case they didn't see it um so that they can request it from us. So it's been a a long journey.

Uh we are now at the 1.2 version. Um this is the list like the the grand list of what I think it's relevant to mention. We have dynamic meshes like um I've mentioned that we have like a small representation of who are we connected to in the network but this is not rigid like this goes and changes over time. Um we do share uh these I have said once um on heartbeats like every given period of time we are sharing with other people that we are connected to like which are the message that we saw and which ones maybe the other one didn't get so that we can have like this conversation uh filling gaps that the other peer couldn't see.

Um we also keep a list of fan out peers which is like okay I have my I know that there there are these peers in the network that they are subscribed to these topics uh and I only directly connecting or sharing information with few of them which are the messages. So the founder is that we can still use or make available like all those extra peers that we know which topic are they subscribed to in order to receive from them or even push into a topic that we by the moment we need to send a message we are not using or we cannot do it through our me. can do it throughout these founders. Um it also allows like message validation like ideally you don't want to propagate messages that are not valid validated somehow. Um for these resilience like we we also included like a peer score.

Basically, if peers are misbehaving with you, you keep track of that to the point that if they go below a threshold, you can pretty much understand that that peer is not a good citizen in your mess and you can prune it and and go for another peer on the connection. And the latest addition was the I don't want messages. Um we realized that gossip sub tends to be chatty uh in the sense like you are broadcasting to your mesh peers directly. Um which tends to create many duplicates. So an add it on one message was included there in order to prevent the like trying to be slightly more efficient notifying others before they start sending us messages that we already received it.

So it's been a long journey like gossip sub uh was pretty much designed at the early 20s like maybe also the the first versions were slightly before but we have to keep in mind that um we are currently having something really close to what was designed at the beginning. At the beginning it was designed as a way of propagating messages but with an idea of having them small enough uh so that the main focus was broadcasting was not throat boot wasn't any other thing but disseminating messages in a resilience way on a way that no one can eclipse us. So we can see the meme on the left side where is like gossip trap was just a way of doing it while the main focus was probably on consensus uh and many other things. So what we have now is that this this protocol has come along to the point where we have many applications uh with high different like many different um ideas of how do want to how do they want to use it and we see is that the throughput requirements that we are imposing into gossip subab are way bigger than what it can actually handle. Um as I mentioned generally like with there were few talks yesterday talking about the main restriction that we have now is broadcasting.

Uh it generally comes to the point of bandwidth. You can buy higher CPUs but you cannot always buy like a more bandwidth or you cannot always have access to faster bandwidth or more bandwidth. So the idea is that we do actually see limitations in gossip shop because it's pretty much not fitting the requirements that we have at the moment. So yes, there are definitely limitations at the gossip hub level like and and the main issue is that it's it's not a mess but it's a complex system that we need to first understand in order to see like uh how can we overcome this these issues at the broadput level. So the main idea is that at the moment um it's quite rigid which means that when a a node a client wants to set up a gossip instance in instance you can subscribe to many topics but the parameters of gossip subab for the service itself um and there are definitely multiple applications that is using it's using gossip subab for many different things across th those different topics.

Uh an example of this would be um I'm mostly working with Ethereum. So the main idea is that you have some topics to disseminate at the stations which are really small messages but that are happening uh on a large blast like many me small messages and then they tend to go out of control because um for those control messages that we are sharing in order to be resilient. What we end up is that we have a huge overhead exch exchanging control messages because yeah the parameters are kind of if you think about the distribution the parameters are set to be at the mean when because we pretty much cannot distinguish them between different use cases and perhaps like the most relevant thing here is also like the large messages. Gossip shop wasn't designed to disseminate large messages and now we are paying that that that price mostly because uh by the moment we are sending a large messages we are already receiving them from others in the end we have many duplicates and we are saturating our bandwidth for just receiving a single copy a single valid one. So yeah the idea is that we should be thinking about overcoming all these issues.

So this is totally to my interpretation. There are many things that I haven't put here but there are different caves that I think are relevant to what we have now and it should be the things that we should focus on in order to move forward. Uh but they are not independent from each other because many of those things interact with each other and they are kind of leaving us where we are now. So the first one is the users. Like I'm not saying that we should blame them, but there is definitely kind of feels like when blockchain was new and everybody was trying to put blockchain where there was not even a need like we have to think about whether gossip is the right like okay we might need to want like we might need to broadcast messages but maybe we can do it more efficiently in another way like so we need to understand what can it offer and we need to use it smartly.

Um it has happened that maybe gossip shop was the right fit at some point but then we have scaled it so much that it's not it's becoming a bottleneck. So we need to play smartly around gossip and definitely it's a complex protocol at the uh configuration side. Um so users need to understand that they need to pay attention to these configuration parameters in order to get the maximum out of it. Um an example of this is something that even has happened to us is that um many people has been using gossip to propagate large or many small messages to the point that the validation queue like the user experience is that you are missing some messages around and the main reason is that your validation queue is filled up. So you cannot add anything else and therefore you'll miss those messages.

It's a simple thing that can have like a big impact in the application layer so that you stop missing messages but it's still there and there's still people facing the same issue. So from what we do at problem where we take a look at how the protocol works and we try to monitor it. Um we've identified like that there are still like not cutting issues but there are still edgy cases timings and mis interpretations of these uh specifications. Uh what I want to say with this is that we should probably take a look at what we have now. Uh pay attention of what are these identify edge cases that are happening.

Um try to see whether we can reduce them as much as possible and have like a solid background like base a solid base on top of which we can build like new features. um edgy cases that we found for example are that um if a peer um sends us a message through an IH that we didn't see before we are supposed to send with an IHV because someone has a message that we didn't see. The issue is that there is no limit on the number of high ones that I have to send. This is up to interpretation on the implementation like like side. Um so there are many things like that that you can find in the code at least the co code that I was reviewing.

Um and there are definitely also many timings that incentivize these edge cases. Like something that we figure that we were able to see for example is that 60% of the I1s that I was that our node was sending were happening like 10 milliseconds before we were actually receiving the message through our mess. So we were because of these educ cases and the timings we are pretty much saturating our bandwidth in a way that we should be handling smartly. And the third one is misinterpretations. Um okay this might be like punctual stuff but we found a bug in the JVM implementation where um so the specification says that if someone sends a message ID on an IHF like someone notifies you that it has a message on a topic that you are not subscribed to you're pretty much going to discard that message because it's not of your interest.

So the JVM implementation uh was pretty much adding all the message ids in an empty string uh topic which means that for four years they weren't getting used of I have I want messages and no one realized it until we were actually paying attention at this level of tracing and at the coding level. So there is definitely big improvement feel in this before we can actually move forward. And the third one would be like yeah pretty much on the lines of what I explained like there is a it's not an coordination issue but perhaps we we should coordinate more efficiently like there is a single spec which is meant to be brought as possible so that implementations can choose whatever they want based on their needs. We have to assume that gossip is not used only for Ethereum and big projects but by many other stuff. Um and those teams have been facing like their own problems and they have adapting their implementations to overcome them but that knowledge hasn't been shared with many others or it has been shared but we haven't been paying attention to actually learn from those.

Uh for example like with the I don't want edition it was something that the NE implementation has already there for over two years. uh while it took us like eight months like it it was implemented for the rest of the implementations like eight months before. So okay, we know where we at. Um what can we do now? Like we know that we have a system that we definitely need have or we need to upgrade because it's not fulfilling the needs that we have now.

But how should we do it? Um out of this coordination like most of the like extensions that have been proposed go over the same line which is like uh because right now is too rigid and you pretty much rely on the um eager push which is creating duplicates. The idea is that we should be able to handle our mesh in a more efficient way or in a more smart way. Um I've listed here a bunch of proposals and even implementations uh that are on the lines of yeah trying to combinate the eager push to a small percentage of their mess with the idea of having eager push like people notifying you when they receive a message um on a on a larger scale that the people that is doing like the the eager push um yeah we can go in detail maybe in the questions of afterwards like I don't want to pay much attention like most of them are are on those same lines like gossip zap is not working well with large messages and the only way of uh making it better or scalable is pretty much like having these combinations of how to broadcast the messages. Um okay I want to take this like as the summary of the talk is like we do know what is gossip chow how it works which are the limitations but there is definitely a set of common targets that I think are relevant in order to make it better.

Uh the first one is like we should shape slightly the specs so that we describe the main functionality of the protocol at the same time that we include like which are the best practices like all this knowledge from all these teams that have been using gossips have been implementing it for many years it's it should be shared across the spec or at least link at the spec so that others can understand okay I'm facing an issue where my node is receiving many duplicates why is it so maybe there is a m mitigation technique already happening and I don't know I note that the Rust implementation is playing smartly of how you cue messages over the the wire. So maybe this is something that we should extrapolate to other implementations. Um another thing like there is already work going into make it more parametricable uh with the idea of let's make parameters per topic instead of per service. Like it's not the same handling at the stations in 4 seconds that having to deal with many large messages like blocks or blobs in Ethereum. It's not the same.

We don't need the same requirements. And the main idea is once again um either with direction we take definitely there needs to be an upgrade in gossip subab in order to make it um able to disseminate those messages um reducing the dlicates with a smart combination of eager push and lazy push um it's probably the way to go. Uh there are many other lowhanging fruits like cell level cell level in messaging which means like you can instead of transmitting and broadcasting the message as a big chunk of data maybe we have to chunk it and divide it into cells that are easier to broadcast and you can play around to who do you notify them and how can even you can apply eraser coding to actually recompose it without having the whole set. So yeah this is pretty much my view on gossip. Um hope I didn't disappointed anyone that wanted more details but we can talk about them afterwards.

So yeah happy to take any question if if others um small disclaimer like we at problem have um many dashboards like there are even some dashboard that are monitoring gossip shop at the athereum minet level which are updated daily um in case anyone wants to check them and come up with any questions like yeah feel free to reach out. [Applause] Uh thank you MueL. Uh we actually have a bunch of questions here. Um I'll go through them via the most popular one and if the question is a bit like ambiguous we can just ask the question um ask her to write it again. So the first one is when you say small or large messages what kind of size do you mean?

uh transactions, blocks, hashes, is it order of few bytes to kilobytes or is it a few megabytes? I think this question was a bit early in the presentation so it might be referring somewhere there. Yeah. So I can talk I mean this is pretty experimental like you don't know how big is something until you see how it behaves in the network. But for example for I don't want messages um it was something clear like clearly useful only for what we consider large messages.

So the more it takes a message to propagate, the more time we are giving to receive duplicates and also for someone to let us know that they have a message that we are receiving but we didn't we didn't receive it yet. So we kind of agreed on a consensus for having like a threshold of 1 kilobyte. Everything above the 1 kilobyte was worth spending that message like that extra RPC in order to consider it large. Um we also should maybe consider like super large something beyond the megabyte like what we used to yeah if the message goes beyond another threshold probably we should be looking about chunking and sending like this sub level in messaging but I mean it's hard to say like what is large what's small that's pretty much the consensus that we read um at least at the gossip sub level we consider something large if it goes beyond that threshold Okay. And the next question is um to how many topics can cosap actually scale?

What is the overhead? That's a good question. Uh so we know that there are networks out there with like Ethereum has over 70 topics. I guess this also goes on the throat boot that you want to put at them. um as it is right now the overhead of gossip subab is is shared across all topics like it will be different if we want to parameterize it um yeah like I don't know how to answer that question to be honest well maybe we can go more directly to yeah sorry like maybe maybe the the right people to answer this is the wacko team uh which are using many gossip subtopics for different application levels so maybe yeah those guys can give us more information there.

Yeah, maybe we can go to the next question which goes more at the Ethereum context. From your perspective, which parts of the Ethereum protocol suffer most from gossip sub limitations, especially the ones that you mentioned? Yeah, I mean I guess the clear answer here is that now that the the um is pushing many blobs. We can think about a blob as a large message with arbitrary bites uh which is call data pretty much that is not onchain. Um the idea of Ethereum is that they want to scale the number of blobs.

uh and because they are large enough and they want to be sent in a small number of time um yeah like gossip is becoming a bottleneck because it's pretty much draining the bandwidth that we have available at the network. It's it's not per se a problem of gossip because if we had enough like unlimited bandwidth and unlimited resources like gossip wouldn't be the problem but it's probably a combination of how the setup of Ethereum is in terms of topology and the resources that they have available and the current gossip like that's why I was mentioning like the we should play a little bit the smarter with how we compose a mess and who do we forward messages because that could allow us to free bandwidth that could we could be using to actually send more new messages. And the final question, do you think the extensions are applicable more widely to all gossip sub users or are these quite Ethereum specific? So just for context, like not all the extensions that I listed there were proposed by people working with Ethereum. Some of them um like the preamble of I'm I'm receiving I I want like that was proposed by the name fox.

So, it looks like Ethereum is not the only one facing the issue of having to broadcast like many large messages. Um, but it's true though that Ethereum is pretty much now getting more conscious about how we could actually resave Ethereum to make it better. But once again, like just because Ethereum will benefit from it doesn't mean that anyone else can benefit from it. I guess that every other chain that now can broadcast sooner like large chunks of data can actually benefit and maybe we actually see more of use cases of gossip because of that. Okay, awesome.

Uh thank you everybody for Oh, there's one final question. Uh does 2D data sampling of blob play better with gossip subub since it would reduce message size? It's a very specific question if you can answer it. Um yeah I mean okay 2D if if you erase your coding in two dimensions yeah you have like lower messages which should have but at the same time you have more messages you have those extra uh cells that you created from from erration coding. I guess we have to keep monitoring it.

Like big shout out for ProLab to keep doing it maybe like. Okay, awesome. Uh if you still have some extent to those questions, uh please catch Miguel in the hallway afterwards. Uh let's give him a final round of applause. Thank you, Miguel.

Thank you.