Ashraf: Nethermind Client & Optimization Sync Time | 14th Oct 2023 | Sunway University | ETHKL 2023

our next topic is from nethermind it's about nethermind cin and optimizing nethermind sync time all right our speaker is Ashraf we're going to welcome him to the stage Ashraf where are you can you wave your hand at me oh there he is let's give Ashraf a big round of applause all right he's going to be speaking about nethermind you give slides right okay I'll get the slides ready just get your slides ready um this is about the nethermind CL this is our second last uh session for this evening after NE mind we've got a consensus layer on ethereum okay let's see this one right yeah there we go okay uh your microphone is over there in the clicker you can just test it out okay our second last session for this evening um all the way from NE mine Ashraf the stage is yours take it away hello all right it's working it is working okay uham Alum hello uh my name is Ashraf I work uh for nethermind as a CF team so I'm an ethereum codf meaning that I work on the ethereum client for NE mine so what is neine so n mine is the second most popular ethereum client so everyone knows about gas guess is uh this guy over here which is the the most popular one and right behind it is SN mine and according to Ether notes it's around 30% market share um I think it's on Monday or something this is ether note ether note is not very very accurate there's some you know something about it that people talk about but yeah next is nether mind roughly so why is it important it's important because of client diversity uh yeah it's important for client diversity you know ethereum have a lot of client different type and there's Aron there's gas that's rest and everything else and then there's also nether mine so why do people use Nether mine from what I heard is because of its uh syn time which is which is faster than gas and also because uh apparently if you look at Reddit it's because of the atation performance which is usually higher than than other client that's why people switch to it so I'm here to talk about the same time because uh a lot of the time I work on it right so I work on it uh I used to work on the the merge and then after that usually bug fix and also the the syn time right so there's a lot of optimization at the syn times so syn time is important because it makes it easier to switch client right if you try to swing switch to gas at least for me it will take around six 6 hour to syn but for night mind it will take around 3 hours which is uh which is half of the time and that's a a lot of reason for that uh but it's largely because of engineering decision in my opinion gas focus a lot more on efficiency and nether mind focus a lot on on throughput meaning that if you still have some SSD bandwidth you will use all of it as much as it can uh and syum is important because of many reason uh aside from the easy barrier of Entry slow sing is also essentially a hanging sing meaning that uh because the chain is keep moving if the sing is too slow you will need to do more work in order to catch up with the with the with the with the network so so you need to have it to be reasonably fast uh and also it's because it makes it inspiring confidence because NE mind have a 95% sing rate the definition of sing here is a bit Yeah it depends on how is notes interpret it and it also make development easier for me because you know I don't have to wait 6 hours so I just have to wait 3 hours or something like that every time I need to change the database configuration or something so so if you look at the syn rate over here NE mind is I mean that the statistic for the one that is sync nether mind is actually really close to Gas like 38% and guess is 39% so you know at last week that's like minute of time and neine have a higher percentage than gas according to this metrics so it's pretty close there uh this is more statistic so we have neine at the highest sink rate uh definition of sink is a bit blurry and that's gas at around 60% and and the rest so this is a graph of the the SN percentage and also the ne mines market share which is around the blue and the yellow line as you can see around the merge the same percentage goes up to around 95% and the the market sh also goes up that's last week there's like a big jump there so we are at around 35% it's actually go down a bit so actually when it actually goes up what happen is like gas goes down a lot like that's a 100 or gas note that goes down and when it goes down over here there's like a 100 also gas note that actually goes up so the percentage is not it's a bit flaky because of gas uh this is in terms of the absolute number so as you can see net mine is is still going up uh I don't know until when I thought there would be like like like a little stop over here around 30% uh but then again last week it goes back back up so I don't know you guys are still switching to NE mind so pardon the impeccable handwriting so this is how uh sorry let me just there we go much easier I just stand over over here right so this is roughly how neine syn and this will explain what does it mean by syn rate over there so actually NE mind does not SN from Genesis it SN from a a block that is close to the Head we call this a sync pivot and from there it will once it sync close enough it can start snap sync so snap sync and after that there's a snap healing because snap syn which is downloading the try that is required for the block processing actually doesn't download the try completely yeah so there need to be a a small pH called Snap healing and this whole phase after this uh neine can process block so at this point neine canman is essentially following the head it can return the new payload uh message correctly to the consensus layer however it has not download all of the blocks and the receipt yet and that will take another two two hours and another different from guess is that NE my doesn't actually download all of the blocks and receip it only download it to something called the engent barrier which is around block 12 million or something roughly that so so which is the block that is starting required for the consensus layer to work correctly so the whole process take around 3 hours if you if you have a setup similar to mine uh yeah and this first phase before the start of the process block will take around 80 gab of uh bandwidth and the D will take around 500 GB of bandwidth so yeah uh yeah so there's a lot of improvement over the years on each of these phase uh yeah the first phase which is the the the the the the fing which is the traditional fing but we don't actually start from Genesis the S pivot used to be configured inside the the sync config and recently it was changed so that the syn pivot will be get from the the the the conser the conser will will call nman periodically for what it thinks is the best what he think is the canonical block and the finalized block and such so it used to be configured manually within the S config file so what happen is usually is that people will notice that older netan version take longer time to sync that's because the configured trusted pivot is actually older so it needs to take more time before it can start uh snap sying nowadays it's it's no longer an issue because we just take it from the the consensus layer and it will and it will only need to download around 60 block or something so it will take like one minutes to to complete that assuming that the con already sing which is depending on how it syn something usually it's a checkpoint sy so it's like I don't know exactly one minute or so and after that we have the snap sying phase and there's some there's a lot of optimization done on this part part because this part is the most processing heavy part of the sing uh sing process so the two most I guess important part before is on this part uh which is a 53 42 and what happened before a few version before is that neine still have B still have processing power but it doesn't know what to do exactly because it's blocked by a a certain request which is not paralyzed so this change do uh paralyze that and just split the top level address range to eight and this cut down the the what we call that the S time from roughly around how much is this 45 minute or something down to 35 minute or or something like that right another big Improvement is that uh a sync DB tuner which is nowadays enable by default on 1.21 so what happen is that during sinking nther mine will change the the database configuration a bit so that it will be right optimized it sacrifice a bit on the rate performance but because snaping is very largely just just rights and very little reads this reduce the the overall Sy times quite dramatically so the overall bites return is around 30 to 40% reduced and this cut down the the rate of the rights from roughly you know how much this is 60,000 K 600 K notes something to roughly 1.5 million uh notes or something so it really depends on the SSD and the processing power right in this case is about 30 minutes slightly less than 30 minutes for the snap syns yeah yeah so snap sying is so it's very close so this is what happened this is probably what e not counts as sync right so that's why the sync percentage of n mine is quite High because it only take around around you know 40 minutes or so for that for this particular phase to complete and at that point NE mind will return a status quote that saying that yeah our latest blog is definitely close to head because we are processing the block and such we just don't have the rest of the blocks to to respond to RPC uh another thing is that concurent read and write during State healing after the snaping because snaping doesn't doesn't produce a correct try so it cannot really process the block right so so previously it turns out that the read and R for this particular phase is behind a single lock so it cannot be paralyzed and if you try to Benchmark SSD before with a single thread the maximum IOP is around 12.5k iops so so it's not like 1 million iops that the manufacturer usually advertise right so this change changes that to use a reite lock and it can paralyze the the read by quite a lot and after this this you this phase goes down from around 50 minutes to 2 minutes typically on a fast SSD and this particular phase is quite important because if your SSD is slow enough essentially you cannot sync because it will have to do more work if it's slow right if it goes too too long then it will need to find another try and restart the whole process so that's why if you go to the support page of nethermind some people finish this in around 15 minutes another people take overnight to to resolve this because of uh it needs to catch up to the to the to the network right so this changes that so that it's no longer become a bottleneck and then we go to the next part which is a fast block and this is also another recent change enabled by default on 1.21 also and previously the process of downloading the blocks take a huge amount of right band wiist of their SSD this is largely because of uh database competion because of how roxb works there needs to be some kind of competion happening and this changed that by basically just enabling a feature of roxb called blob CB and this cut down the total rights by about five times so that's why you see inside the release note of the of the latest version there's a saying that some configuration it it reduces the Sy Time by about five times it depends if you are limited on the SSD or not uh my case I'm not very limited on SSD so it doesn't change that much in my case and another different is the fast receip which is done before the previous changes that's why you see uh it doesn't consume much bandwidth over here even though that the database of receip is actually quite large it's probably around 100 Gig compared to blocks which is 300 gig right so the changes is quite actually quite simple which is just to prefix the the key the Roxy key with the block block number so this trick which if you look at the database design of Aron which Aron does for every of database of it actually reduces the the compression by quite a lot so this reduces the band we the total RS for the receipts to be face by about twice or something but this particular pH usually is limited by the CPU so it doesn't make much of a difference for most people and yeah that's I guess that's it that's it that's all in my presentation so what's next for neine that's canun and all of the protocol level stuff there's also a flat DP layout still working on it I'm not sure it's it's a very problematic system and there's some other uh things I'm thinking of doing yeah so that's it all right thank you ashra speaking about the nethermind client and the upgrades and all that anybody got a question about nether mind we got one here we'll pass the mic to you is just coming in a minute please introduce your name and your question hello um wean so I wanted to ask uh regarding the snap healing process so my understanding is that uh because it's trying to keep up to the network that's why you have to do some healing on the outdated try notes in the try so what happens if a let's say a small or a big REO happens in this case would it affect um if if my understanding is correct it's it it the main point isn't really to catch up with the with the network the main point is that the after the snap sing the try is not correct because different part of the try comes from a different block so so it needs to need to finish it and state seeing the head must be from a canonical block so it picks a a try from the canonical block that's how it prevent that issue okay I think I also have my second question which is on um let's say we will to compare GFF and Ne mind so for my understanding from the the very nice handwriting that you had uh impeccable handwriting that basically net mind it feels like the architecture's philosophy is that um let's say for gaff um is if I remember correctly for SNAP saying is it will download all the blocks and download all the data that is not needed to process blocks but for net is that okay what do I need to process a block which is just the try the up updated state right so once you have the complete State then only you download all the So-Cal unnecessary things um that needs to soal keep up with the network or to continue with the network so is that uh is my understanding correct in that sense yeah yeah more or less okay yeah that's there really Point answer yes that's that's quite a lot more different than than that uh for example nethermind doesn't download all block right so so it it download from the engine barrier which is blocked 12 million or something so if you want something further you have to change your configuration so that it downloads that that also another different is that gas has an what we call that ancient database right and the restriction of the anent database is that you must insert the blocks in a strictly increasing uh order right and the problem with that is that you have to reorder the block somewhere in the memory or something right so that's a sacrifice they have to make and if the memory buffer is slow they have to throttle it down right so nether mind doesn't do that nether mind just dump the whole thing to the roxby and let Roxy do whatever he needs to do that's it different okay um if anyone doesn't have any question are there any questions no it's all yours go okay uh G also has U they recently upgraded to Pebble DB so is there any plan I mean do you think it makes sense for neite to upgrade to P DV in the future uh if there's a binding it might make sense so it makes a lot of sense for gas to upgrade to P DB because mainly because level DB previously doesn't have a concurrent competion and that really limits the right rate so that's another that's another reason why gas might be slower uh but another different reason is that pable DB is kind of like rock DB Plus+ you kind of can make rock DB as fast as P DB but you have to tune it uh quite a lot and until a certain point where you don't really have a choice but to switch to Pebble DB but that point is quite high so is does it make much sense for us to make a a c binding for p DB to to you know have a right limits higher than 1.

5 million Keys per second you know it doesn't make much sense if you ask me okay understand I think my last question here is um not technical related just want to understand more about yourself is that um I I because I previously talked to you before but um but what's the difference between um you know um building a distributed system in blockchain Space versus like let's say in the traditional web tool space yeah the I think the main differ is that with distribut system like this you have to validate every message you have to validate every single message uh in one way or another uh in a distribute system in a big tag you don't have to do that and this actually add quite a lot of overhead for example in my development setup I have four guest note learning locally takes up 70 gig of RAM so that I can saturate NE mind and why is it for no instead of one because a single connection have a limit to how much bandwidth it can because of the RPX decoding because we need to encrypt the connection so now that's a limit over there so that that's that's probably the biggest different in terms of networking okay any other questions uh feel free to raise your hand if you want to ask about the nethermind client any of the upgrades going on going twice all right let's give our speaker a round of applause thank you very much Mr asra