Neighborhood Doublings & Setting Target Neighborhoods

if you've heard the term doubling but are unclear of what it means or whether it affects you then this presentation is for you in this presentation we explain doubling and discuss how it affects node operators and data uploaders in part one we need to First cover some basic information about neighborhoods in swarm your overlay address is your node's permanent ID and neighborhood address and it takes the format of a 64 digigit 256bit heximal number to keep things simple and easy to read for this presentation we are shortening all overlay addresses to their leading four digits only just keep in mind that the concepts apply in the same way to the actual 64 digigit overlay addresses a neighborhood is defined as a group of nodes which share the same n number of leading binary bits this number is defined as the depth of the neighborhood and also referred to as radius in some contexts to see this more clearly we've converted each number to Binary we've also highlighted the first 10 matching bits which are shared among all the nodes in this table we can then see that these nodes form a neighborhood of six nodes at depth 10 nodes in each neighborhood are responsible for storing the same set of chunks which match their leading binary bits up to the neighborhood depth when a chunk is uploaded it's assigned an address with the same format as a node whichever neighborhood shares the same end leading bits with the chunk is the neighborhood that's responsible for storing that chunk let's take a look at some examples here these two chunks in the first table share the same 10 leading bits which place them into our example neighborhood here for these other two chunks in the bottom table we can see that they have differing bits from the example neighborhood at the 10th and fifth digit respectively meaning they do not fall into this example neighborhood above and that about covers the basics of neighborhoods in swarm and we're now ready to start talking about neighborhood doubling next section we Define neighborhood doubling and discuss its implications for both node operators and also for data uploaders on this slide we'll start with a highle description of doubling before moving on to a more detailed example in the next slide a doubling occurs when the chunk reserves of nodes in a neighborhood are filled Beyond a certain threshold when that happens the neighborhood will split into two with each new neighborhood being responsible for storing half of the chunks of the previous neighborhood a few details which are important to note are one the actual mechanism of the doubling is that the storage radius also known as the storage depth for each node in the neighborhood is increased by one and two this results in both a doubling of the number of neighborhoods as well as a doubling of the network storage capacity and three the doubling will not occur across the network at exactly the same time as different neighborhoods will fill with chunks at slightly different times and finally four a having can also occur if the number of chunks in a nod's reserves fall below a certain threshold with all these details in mind let's take a look at the next slide for an in-depth example here's the neighborhood from our previous example which was at depth 10 with the 10 leading bits highlighted in yellow once the reserves of nodes in that neighborhood are filled with chunks past a certain threshold they will increase their storage depth or radius by 1 from 10 to 11 and we've highlighted the 11th bit here in blue and then the neighborhood will split into two neighborhoods based on that 11th bit as you can see nodes with a zero at the 11th bit split off to form neighborhood a and those with a one at the 11th bit split off to form neighborhood B on the next slide we'll take a look at how chunks get get redistributed between these two neighborhoods here you can see that the chunks which were in the same neighborhood at depth 10 now fall into two different neighborhoods at depth 11 this means that the nodes in each neighborhood now have freed up about half of the reserve space and can start storing more chunks at depth 11 until they've hit the threshold required to move on to the next depth depth 12 now let's take a look at how the doubling has affected reward chances in the redistribution game these are the same addresses from the previous slide with the binary digits removed for easier visibility as you likely already know a node's chances of winning when its neighborhood is chosen to play the redistribution game are weighted according to their percentage of stake within the neighborhood here in neighborhood a at depth 10 each node has 10 stake and each node has the same 16.6 repeating perc chance of winning according to its percentage of stake in the neighborhood now here at depth 11 neighborhood a has split into two new neighbor neighborhoods the new neighborhood a with two nodes and the new neighborhood B here with four nodes the stake Remains the Same as before for each node this means that in neighborhood a here each node has a 50% chance of winning when its neighborhood is chosen to play the game while in neighborhood B each node has only a 25% chance of winning if it's if its neighborhood has chosen to play the redistribution game so we can clearly see the advantage of making sure that our node falls into the less populated neighborhood a in order to maximize its chances of winning after a doubling event now the good news is that there is an option in your B node configuration which allows you to do just that we'll go through how to do that after discussing implications for data uploaders on the next slide now let's address how doublings can affect data uploaders doubling can can create situations which could potentially put data at risk for data uploaders in the previous example the doubling resulted in one neighborhood with only two nodes Which is less than is considered a safe number of nodes and that could potentially put the chunks in that neighborhood at risk for being lost to prevent the loss of data data stores can make use of the new eraser encoding feature which adds additional protection that allows for the original uploaded data to be preserved even in the case of the loss of some chunks now in part three we go over exactly how node operators can find and set Target neighborhoods for their nodes to maximize their reward chances the easiest methods to find Target neighborhoods for your node can be found at swarmm scan swarmm scan has both a neighborhoods page where you can manually review the least populated neighborhoods at the current and next depth and also offers an API for recommending Target neighborhoods the API even has a depth query parameter which allows you to look ahead to Future depths for the least populated neighborhoods let's take a look at both of these tools now here we can see the neighborhoods page and at the the top we can see the uh neighborhoods with the least amount of bees for radius also known as depth 10 and we can see that there's a single neighborhood which is the least populated with seven bees or seven nodes now let's go down to radius SL depth 11 and we can see that there's quite a few neighborhoods with only two nodes in them so we can pick any of these and save it for the next step where we're going to actually set the target neighborhood but before we do that let's take a look at how we do the same thing with the API so on the API page under the network tab you can see this one suggest a neighborhood that needs additional capacity and you can see the end point here and then also make sure to take note that there is a query parameter depth uh if we do not use this depth query parameter it'll just give us a recommendation for the least populated neighborhood at the current depth however we want to look one depth ahead uh so we don't want depth 10 we want depth 11 11 and that's exactly what we do right here so uh here you can see when we call that API endpoint we get a suggestion for uh a least populated neighborhood at depth 11 now that we've got our Target neighborhoods binary digits we're ready to set it in our nodes configuration so first let's open our configuration and then we should see here that we have the target neighborhood option uh and if if you do not already have this option you may need to add it to your configuration and then we simply paste in the binary digits for our Target neighborhood and then save and exit and let's check that to make sure it's been set and there we have it and that's all you need to do now the key thing to make sure is that you do this before initially starting up your node so that uh it will uh generate this this uh Target neighborhood address when you first set up when you first start your node to wrap things up let's first review everything we've learned so far number one uh a doubling event occurs when the amount of data in a neighborhood exceeds the limit for the nodes reserves in that neighborhood the neighborhood will then split in two allowing each neighborhood to free up half the reserve space this eventually results in the doubling of the neighborhoods on the network number two node operators should make sure to set their target neighborhood using the target neighborhood option in their nodes configuration settings and should also make sure to account for future doublings by using a Target neighborhood at a deeper depth than the current depth and third in case of a doubling event which might result in underpopulated neighborhoods data uploaders should take precautions by using the new eras recording feature which will protect their data even in the case that a few chunks get lost and that wraps up our presentation thanks so much for your attention and if you want to learn more please join the official swarm Discord and also check out our official documentation page