Anton Evangelatov & Rafael Matias: Running large scale Swarm deployments

so I guess we can start my name is Rafael just a quick intro um I started working for the swamp team last September but it was really just part time one day per week and then I started working actually then mostly full time from beginning of the year for the swamp team mainly focusing on the DevOps aspects of the project so infrastructure related with me I have Anton his was the swamp team now for like one and a half years three years yep hello everyone yeah I've been with this foreign team for Oh for the past year and a half and yeah also doing foundation wide DevOps work and yeah let's start so first a short recap about observability is formed so basically the tools that we have available to help us develop test and monitor swarm so that we can achieve our vision and this is the LOX matrix and the tracing so basically swarm is sending blocks to standard output and standard error as you all know you can set the verbosity the debug and basically see where the bug knives come from and we're using Cabana and elastic search to aggregate our locks which helps us to debug issues and monitor our nodes that we maintain as part of the public test net about metrics as you know also you can expose metrics from your swarm node and we also collect those and aggregate them and they scale much better compared to logs so the more nodes that you add like the metrics don't really explode and we use in flux DB and graph ana to visualize those so on this screenshot you see the peer-to-peer dashboard that we have where we measure all the peers different nodes have the total number of a peer-to-peer messages sent and also the individual messages sent as part of different protocols like PSS stream hive etc so you can have an understanding of what individual deaf b2b protocols do within swarm and we also have tracing we have instrumented the codebase with the open tracing library which we use together with Jaeger to visualize traces and we this is how it looks basically so with Jaeger you can actually see a request going from one node to another and also monitor and see how notes communicate with each other in the screenshot here you can see it basically an HTTP GET request for a piece of content and every lazy chunk reader read is basically either getting data from the local store on your local node or if you don't have the particular chunk issues a remote request and you can see in this screenshot that a node basically does two hops to fetch the content that it needs so now I'm going to talk about how we actually run swarm as part of the team to help us develop it where we are coming from and how we came up to the current system and basically why we build it so previously when we want when we were developing swarm we had pretty much the only possibility of running for means for small deployments so as the day to day def you would basically run a bunch of notes on your local machine and you can interact with them through the simulation framework but yeah and within the public gateway that we maintain for the community we had a bunch of static virtual machines which were running a static number of containers on them so that you can try swarm without having to run your own node that's the from gateways dotnet and these were all managed with their form and as well playbooks the issues that we had with O'Day's was that for small test deployments on your local node basically you're limited by the resource limitation of the machine and you can't really learn run large-scale deployments let's say hundred nodes because you just don't have that much power on your machine and some problems manifest only when you're on larger deployments I mean if you want to test code emulate apology and thinking you need at least like 15 to 20 nodes so that you build a somewhat healthy academy a topology if you just run two nodes you're not running yeah half of your codebase basically also simulations are kind of hard to debug you cannot really interactively inspect them after they have completed I mean this is not part of the code base right now and something that we want to improve in the future and in terms of the public gateways it it was very time-consuming and hard to scale them up and down basically they were mostly static with virtual machines provisioned with as you go play books and yeah that was a problem so I'm heading over now to Rafael who was a very instrumental in bringing in the new syrup and he's going to talk more about where we currently stand and what we want to improve in the future thank you and John so as I mentioned to previous setup was pretty static scale about scaling of things was very difficult and we had also the goal that we wanted our developers to easily spawn up I don't know 100 swamp nodes and observe what's happening in this network so on this slide we have a list of tools that we are using I mean in flux to be in Corona we used this already in the past for the metrics Jagr anton already mentioned it is for the tracing purposes so this is these are things that we had already what we didn't had was a proper Orchestrator to deploy our workloads so for that we started using kubernetes we previously you already were using like docker containers statically deployed on some VMs and kubernetes gave us like the benefits of having a pool of worker virtual machines where we can just deploy our 100 knots one cluster and don't care about saying oh yeah this should be running on this machine or that machine the orchestrator will deal with this problem for us another thing related to kubernetes that we are using is helm helm you can see this like as a package manager for communities so it includes all the resource definitions so everything you want to deploy within kubernetes inside like one package I will show you like briefly for example what swarm helm chart looks like and then I'll turn already mentioned this we started collecting logs so once you have like a system like this running containers everywhere it's very useful if you have a centralized place where you can visual visualize your logs so for that we use elastic search cabana and fluent pit and we also so once you deploy like a hundred node cluster you want to like run some basics most smoke tests on it to like make sure that the most important functions of swarm are working so now back to this home chart so as I mentioned the helm shot you can see it as a package like a group of applications like things that are going to be deployed within your native I'm not going to talk that much about kubernetes details I think that's not important for this talk but within swarm so this home shot we have of course swarm itself it might be important to reference that like the minimum deployment unit within kubernetes is called a pod so these are these boxes here so we have a swarm port which consists of warm itself which is a container and there's another container on the side which is the Jager agent so this is collecting traces from our swarm container and also then forward stresses to the Jager collector on the right side which can then store this on some datastore and we also have a UI to query the traces so these are traces and then of course for metrics swarm has generates a lot of metrics and you can forward these metrics to in flux DB so we have a separate port for that and on in flux DB we want to visualize these metrics using some dashboards that's why we have grow fauna and on the left side we have capacitor this is kind of the alerting tool that based on some thresholds you can generate alerts or warnings so that you can identify problems quickly and this is all all this is within a single package so let's go to demo and showing you guys some code that's not what I want to show I just want to like explain you clicky quickly what a helm shirt is like in simple terms so we have this repository where we store all our helm shots let's this one for our like swamp private deployments of a private cluster it has like this requirements file which contains like dependencies for example which we see a we need in flux DB we need this capacitor we need the Jager this is just like a reference to like other helm charts and then at the end helm what it is it just like a bunch of templates like resource definitions within kubernetes and you see it's it yeah it's basically templates and all these values within these templates that are defined here you can override them and the defaults are defined within this value siamo and whenever you like use helm to install something these will be the default values but you can override this with your custom values file and this is mainly how we can I use warm like deploys form currently another thing that we use on top of helm this helmsman so what if you want to manage multiple helm deployments you can do it via the comment line but at some point it will get annoying so for that we have films men we have this repository that it's open source it has the definitions of our staging cluster let's have a look at one of the definitions so you can see we do we want to deploy some apps one of them is our private swarm and cluster we just say what chart we want to use basically here the address first one private chart and then we kind of provide like a reference to our custom files where we are overwriting all the configuration configuration can be for example saying which docker image we want to run for example recently we released 0.4 so there is a docker image for 0.4 together with the swarm deployment we also have like smoke tests which are the entire gating this other smoke from deployment and the good thing about this once you have like this file created and you configured everything properly running then a swarm cluster is one command which will bring like up all those components that you see a so on the graph previously good then once you deploy your cluster you want to access it somehow to make it easier we also have like web server in front like a reverse proxy that can target your individual swamp containers or just load balance across them and this is an example I mean in here I'm just accessing like one of the containers number 23 within a deployment that Anton yes I can easily change this to any other one or I don't even care which one I'm targeting I just removed this number and I'm just load balancing across one of them yeah and this again it's like one command and all this magic comes up and you have this working so this is very good for developers because you just provide them like an example on how to run like a hundred node cluster and then they can reuse this and just change the middle configuration to test stuff so this is like ingress how to access and then an important part that Anton mentioned are like then the metrics like how we visualize this maybe you want to talk a bit about this yeah so basically once you run this deployment with Gravano you we also automatically provision all the dashboards that we use internally to monitor our deployments and monitor our tests which basically is a lot of dashboards which kind of change often because we are developing storm but I'll just show a few of them for example the peer to peer dashboard so this is a deployment that we started yesterday I mean currently just shows you're the last 30 minutes but you can see like the total number of peer to peer messages the number of registered peers for individual container then you can see how fast some of those requests are and then you can see all the individual protocols for example this one is not running psso you don't see anything here but you can see all the message exchanges and the number of messages sent between the actual wire protocol messages which is very helpful when you debug stuff for example one thing that comes is interesting is that we know here that we're running smoke that's every 10 minutes and when I say smoke tests currently these are the very simple ones that we run you just upload a file every 10 minutes and you try to download it from a different random node in the network so it's kind of interesting why we're seeing you know counters for those messages all the time and that's actually unknown bug that we have in swarm right now that we're gonna fix for the next release but it's immediately visible that you're not expect I mean if you're building on this protocol you know that you shouldn't really be seeing those message exchanges at that point in time and it's you're very helpful other dashboards that automatically get provisioned for you for example you can see the smoke tests like we can see the last or is it twelve hours we can see all the runs how long it took to upload the given file how long it took download it how are chunks synced across the deployment so that we see what kind of overhead we have and yeah other interesting information that you can see by Spee what your form note and in this case your whole deployment is actually doing the local store one for example shows you the data usage on the given nodes the number of gates would cause [Music] timers and yeah or other interesting information so that you can further Bute and so on yeah and now Rafal is going to talk a bit about the locks one thing that I want to mention is that you automatically get this every time that you run those camp charts so even if you don't have a clue how to run let's say graph Ana because you've never used it before this gets deployed for you and it's quite handy as a tool to understand what's going on that's right that was a very good point then after metrics logs so all these different containers running within kubernetes they're generating logs we are sending all the logs that containers generate to elasticsearch and then we can visualize everything within here on kibana we can filter like by namespaces I see for example that a lot is running a deployment here that is generating logs so I could just filter his namespace to just get his containers and then you can start filtering by specific containers or different applications whatever so I mean that's mainly it about logs then also important is the tracing part let's see if this works let's just try to upload a file to Anton's deployments I'm just gonna upload some image okay we got here a hash and I'm just gonna download it from another node okay so we uploaded something we downloaded it there should be some traces we downloaded it from note 1 so I expect let me just refresh the page to be sure that this works and downloaded it from node 1 so I expect this HTTP cat is easy and let's try last hour find traces ok that's actually the redirect because you did not the slash in the end so the first one is the redirect and the second one is the actual trace right and then if you collapse them ok that's not the key I expected anyway I mean yeah it's very useful to learn the shortcuts of jäger because you can then collapse and expand all the traces otherwise it's a bit overwhelming but right now you can see that this node 1 is actually fetching some of the content from node 63 and then it's fetching it from not 46 and you can also see how long different messages actually took you can also attach locks to those individual spans and you can see basically you can trace from wearing the code that happened I mean you can for example validate that your requests are going closer to the chunk and not just fetching it from a random node basically it's very useful to to visualize how different protocols work within swarm yeah that's pretty much it traces yeah that's racist now let's try something else so I already prepared like a deployment of I don't even know swarm demo get stateful set mm-hmm let me just verify okay we have like a cluster deployed with hundred nodes running I already prepared this so that we don't lose like one minute that's the time it takes to have this running here and let's let's just try to here print out CAD Emily our table here beehive on one of the nodes so here we can see like if you check the population we know about 100 nodes and we are connected to 15 and now let's try to scale this to 1000 to see what happens and you guys saw this was one command I didn't have to care about statically defining where to run stuff and so on this was very simple now it's going to take well it's not instant you won't see like 1,000 nodes running it takes some time even like to start a container so they the container images have to be downloaded stuff the starting is quite quick but downloading images and so on take some time then also swarm starting we are not persisting data so we are generating keys for every strong node so this also takes some time but you already can see that it's increasing so we have 184 nodes right now we are checking I think every 5 seconds on changes here so yeah we just leave this rack 350 so we just leave this running and continue on the talk and at the end we will check out if this worked out let's go back to the slides so yeah one thing to mention is that everyone on the team has access to this cluster environment everyone basically can start their own deployments in their own namespaces and metrics logs they're kind of isolated so if I run a deployment it doesn't interfere with the deployment he's running or somebody else on the team and yeah do two scalars make sure that we are also not wasting resources on Amazon so basically those machines that are gonna start up right now are spotting instances which are like five times cheaper than the on-demand instances and they will shut down after like five to ten minutes after we should down this 1,000 note deployment so that's also quite cost-effective right and before we get to the 1,000 I want to mention like some next steps some improvements that we want to do so currently like applying these developers do this manually like even though it's one comment they kind of do this manually what we want to have is to integrate this into our development lifecycle that for example a cluster gets automatically spawned up during a pull request and report like back the result of the smoke tests to that pull request and for example if something goes wrong there the developer should also have the ability to then further on debug on the current deployed cluster to figure out where the problem is also another one we have so we have this simulation framework we want to create like an adapter there so that we have the possibility of just spawning up directly terminating spots running a swamp container and way more in the future I mean all this metrics traces stuff this works very well on private deployments but you might ask yourself okay what about the public network how can we debug that well that's a very complex problem and in my opinion for that in the future we will need like some friends so people that run notes we will probably try to collect an anima metrics this is of course opt-in and yeah those are some goals about the anonymized metrics yeah Peter from the get team actually has nice too that basically allows you to send your metrics of a tour to a centralized server so that would be one way of doing it for example we wouldn't have a clue where those metrics are coming from but we would still collect data from the community and have an idea of how the network works and yeah it works so we have 1,000 swarm nodes running I'm just gonna scale this up even more for fun while there are like questions I don't know what the limit is right now and so what one other important thing to mention from a developer perspective is that even though this is well automated compared to what we had before it's still not a very tight feedback loop so unless you're familiar with the system I mean obviously there is a learning curve but it's not as fast as you know just running go test on your local environment so what we wanted to do ref I mentioned it has next steps we have the simulation framework it's very useful and we want to move some of those tools back to your local machine so that you can have a similar feedback loop with let's say four or five nodes because usually that's enough to run a bunch of unit tests and to have a very tight feedback loop and debug and develop protocols and form once you're there then you can run your tests on much larger deployments and basically have a better safety net with the end-to-end smoke tests and the rest right now we are at a point where the simulation framework can do some of this stuff on your local machine but it's not that easy for example grabbing for locks is difficult interactively checking instances after the simulation has finished is also difficult whereas with this environment it's actually very easy okay it's just last test let's upload a file to this 1000 North cluster now it's actually 1184 okay I just uploaded an image got the hash let's open the web so I uploaded it to not zero let's just access access it on 55 I'm gonna put the hash here and one last test what was their address this is from yesterday so you can see that ENS resolution on Robster also works which is pointing to that hash that we previously upload and I think that's it do you have any questions we have like I guess some minutes is there any question