Devcon 5 - Ethereum Swarm Breakout Sessions: Running and testing large scale Swarm deployments

so morning everyone yeah so I'm Rafael this is Anton we work for a swarm team we are kind of mainly responsible life for DevOps related tasks and also like testing swarm and testing swarm is not that easy like even like a big part of it is like observing how swarm is behaving and this presentation will explain you like some techniques that we apply during our testing process so I'm gonna start a little bit with observability and basically cover the tools and the techniques that we use to debug swarm so most of the observability tools like blogs matrix tracing that part of go at your own code base for a long time and we've had them over the past few years so what we've done over the last year is basically put them in our automation tools so that they're about everyone who develop transform and they can yeah get meaningful signal part of them so yeah the first part the observe ability to set is the logs yes form pushes go to the standard output you can set different purposively and yeah also include the falsification of all the lock lines it's quite useful to bump the verbosity up from the default value of info so that you get also like block debug and trace locks and understand what your mouth is doing when you when you run this world metrics metrics are basically posting to time series database where we aggregate them from the distributed environment that we can actually visualize what us from network is doing not just one node so basically we aggregate locks within our deployments with elasticsearch and visualize the new Capanna we'll be showing this yet presentation so that you see how we do is do this on a day to day work and metrics we I can get them within in Flook CDP and we have that protein graph and where we visualize basically different counters and timers so that we have an idea of how performance permits and what part of the protocols we need to optimize and whether things are true we are working correct it basically gives us an overview of what our nodes are doing when we run different scenarios on them we also have tracing and basically the flags that we'll need for that term yeah the tracing clock system we're using the open tracing framework most of our internal IP eyes are already instrumented so you have the possibility of having tracers and bloodlines within any part of the process and then have them visualized in this central system for which we use Jaeger from Google and that also gives us another system to basically have visibility over our networks private deployments here testing in general this is what it looks like yeah here we've just an HTTP GET request which goes through the internal IP is a swarm and also goes to multiple nodes so from not sixteen we communicate launch tree and all 29 and basically the timings of those requests so next time tell us a bit more on the what we basically improved on last one year so the points warm like in the past so what we did is many times we deployed one on our machine we use for example containers to deploy like multiple nodes on a single machine and then of course we have the bottleneck of your machine you don't have enough memory you don't have no CPUs one point in time it gets slower you got just able to like provide a cluster oh I don't know ten nodes and that's it also like on our public gateways in the past we've used like some virtual machines on like public clouds they don't yes and we used an smooth to configure these machines this didn't like really scale very well that has been like there were like many manual steps that we had to do like adding PM's and then running the and zero scripts manually and so on so we have to improve this yeah I mean yeah the problems mentioned that we had and basically what we did was we started using our human ages to vanitas for those that don't know it's just like a container orchestration tool so you just have like a bunch of servers and you just orchestrate like applications you say like run most of these and that's many it so it makes like this full automation process easy for us and in communities you have the thing they call it helm helm you can see it as a package manager in Cuba teacher so imagine if there's like a package swarm which we actually did develop this you can like install swarm in cuban eaters and just configure it as you want and so this swarm helm short that's what it's called it provides a lot of configuration and it has like everything integrated that we've mentioned before this means like the metrics tank like influx to be in pure fauna the tracing stack would be acre and also like the logging stack like in it's all in one chart so it's just really one single application that you deploy and configure and you will get like everything out of the box on your cluster additionally we also have some smoke tests smoke tests you can see this as some like essential basic test that in the case of swamp they just vertically upload some files and retrieve them and we even gather like metrics from these so that we can see over time or the cluster is behaving if we keep consistently like uploading files to yeah as i've mentioned that this is what our swamp young chart looks like so we do actually have swamp itself as an application there and then we have like all these dependencies that and honestly i have everything working like from scratch the metric stack which is in flux DP and for fun I'd like to visualize and Jaeger life for the tracing stuff and actually also now since we have swap and like incentives we it's not here but there's another component that we've added like for testing purposes which is there should be also a carton of ganache so we use ganache like to simulate the blockchain and connect our smanos to it yeah now I'm handing back to Anton he's gonna show you actually a live plaster that we have to quite antigo go through like other things we've mentioned about the transdimensional this is open source so anyone is free to check out the ripples and yeah basically one portion now and everyone is basically free to run it on your own or even on your local machine I guess I need to be so basically I'm just showing the parts of the department that we started yesterday and we see that it's like a 50 not strong deployment and we've been running some periodic smoke tests on this deployment and based in order to get this what we need to do is best just provide one yamo file which basically specifies that you want to make a strong deployment based on the storm chart which is open source the number of radicals that you want and then basically deploys a song for you if it also comes bundled with the debugging tools showing those the first one is profile so we just pop over to instance [Music] and then we can open it up we basically see automatically we get both dashboards that we are using in the scrum team you don't need to import anything or set of data sources they all just come here and this is the defaults from that port which gives us an overview in the warning stairs but mines pick up times of the nodes the HTTP requests etc we can check also some other dashboards for example the smoke tests so the smoke tests over the past let's say 12 hours we can see that we are very obliquely uploading and syncing with also the church that have been synced across the entrance and we are other used to information for example in the stream network we have information on the syncing protocol domain bouzouki protocols for the number of years per node received change the offered Jones The Wanted tracks and basically a very nice overview process all deployment of the various protocols so that's pretty much what Ravana gives you advice to visualizes or your metrics and the nice thing is that it's available immediately when you run your home chart and it comes bundled with all the boats that we're developing on a daily basis so the next one I'm gonna show you is Jaeger so basically this is the distributed tracing so we're going to port forward to the Jaeger tech service and for that I also want to see the last smoke test that front so that we can visualize something that just run and we take care that we uploaded successfully to not 0 and then we fetch it successfully from not for so I'm going to just check this request from node 4 and these bro because I just killed the party we go and we look into basically the requests that we are kind of interested at let's pray for the Democrats oh yes thank you and we can see that the request is like from 9 59 minutes ago and we can basically see on the bufferedreader on the section we get requests the reason it's sequential it's because we support range queries and you can actually just ask for a chunk of that content but we're uploading enough in the coding so we see all of them and we need those individual read buffers we can see the remote requests and also the local stock request so the local ones are those strands that we have on the local nodes the remote benchmarks I'm not sure if you see that our basically does that go to remote notes so in this case we see that the amount for ask note 33 for a child and we can see for which chunk it asked when that happened belonging to etc so as an overview we check what happens within a deployment and we measure 95 percent times 99 percent of protocol deployment Ravana with the matrix and weeding the distributed tracing we have a more in-depth look over specific requests so so much about yeah distributed get the login Saturday which basically we use cabana and elasticsearch tracking at the logs and again the port forward [Music] service and there we can filter for anything that basically is mentioned within our logs on the roads so this is either getting logs from across our whole cluster which means that all deployments are sending looks so I'm just gonna add all the logs of the last 15 minutes within the swarm private zero note and we see all the logs here why this is useful if we search for example for HTTP request we can see for instance this request and search for one individual request based on its request identifier and we can have again all the information about a specific request to a specific note that have identified as interesting so yeah we have the caustic location and the request so that's quite useful when you're developing protocols if you want to debug what's pretty much going on within your deployment and it comes out of the box when you deploy yum chart so with one command you get those three systems it helps to build on top of soil so that's pretty much it you wanna try that so also with another thing we have so sworn as the HTTP interface and we automatically have like reverse proxy that that generates like these host names and they are publicly accessible so we can target any of these nodes lives through our browser easily and yeah it's just targeting one of our nodes or we can just load balance through them as we want everything out of the box we mentioned like large-scale deployments we just show up like a 50 node cluster but this also works for more nodes we the biggest one we demoed I think it was 1,000 to 126 so that's it there's also a list of useful links we put here so this there are our young trans if you know about Cuban eaters and want to try this out so just have a look at the young charts and yeah thank you much where's it this is deployed on burn it is on Amazon so we're using Amazon for our development but it can very well be on HR on any pattern operator kubernetes you can deploy it on your local machine as well on your own hardware it's quite transferable across thank you yeah thank you for that this is actually a really big deal because when when we were running a single small note and things don't go right something's really hard to know what's going on in there and we used to have like log dumps and you can try to figure out what's going on but these two have done is not just made these logs really nice and you know graphs you can see things going on what they were just showing with that distributed tracing is you get to see what's happening in the entire cluster when somebody makes a retrieval request and that gets passed from one node to the next to the next to the next and you can see the logs go all the way through this so that's a really important thing because when we're developing something like swarm which is a peer-to-peer dynamic network you're not only debugging what a single node does you have to get the emergent behavior of the entire network correct and that's hard because you get reading sometimes very subtle things changing getting having big influences so that's why this was really great so if you're into DevOps look at what they've done look at the repo you get a lot of really cool tools so yeah we schedule a bit that wasn't enough well Victor's talking about 10:40 the next talk coming up is from Wojtek and drink here right here