Decentralize All The Things: Deploying Your Own Node Infrastructure by Carl Youngblood (Devcon5)

morning guys thanks for coming bright and early so I'm gonna actually tempt the demo gods today it'll be interesting because the things that I'm going to show you today and demonstrate take long enough to deploy that I kind of want to get started right now so I also want to thank the conference organizers for providing us with these free LTE connections because that's what I'm using right now to be able to do this the Wi-Fi hasn't been great but fortunately we have these wireless hotspots we can use so I'll just deploy this and then break into it in just a minute so changing my config params to suit my needs then going ahead and setting up my dependencies and then I go ahead and kick off this cloud formation template that will deploy the solution that I'll be talking about right now so you jump back into that I also want to say hi to one of my local counterparts from Tokyo Yuki naka taki which is right here joining me thanks for being here so I'm Carl Youngblood I am a senior blockchain Solutions Architect with AWS and I've been doing work with blockchains for quite a while I first became interested in blockchain in 2010 when I read Satoshi's white paper and it was one of the only white papers that actually came with working software it was really exciting to actually set it up and run it and I was lucky enough to be in living near London in 2014 around the time that like the first theory and meetups got started as well so before I joined a AWS my brother and I co-founded a consulting company called black scale and we deployed some node infrastructure solutions for companies like my crypt oh and the Tesla Foundation and I'm gonna be sharing some of the things we learned in that process today so first I want to share some thoughts about decentralization and how it can sometimes differ in theory and practice so and after that I'm gonna explain a strategy that we used for deploying highly available node infrastructure and then some ways that we DIF mitigated against some different challenges and problems with you VMs and also how we made our deployment more cost effective and then finally I'll show you a little bit more of this solution that I just deployed so one of the great promises of blockchain technology was that you know it enabled greater decentralization so in the same way that anybody the Internet is a permissionless network that anybody with a computer can and the right equipment can just hook up to and as long as they're speaking the right protocols they can they can join in this network blockchain also eliminates the need to trust in these large central authorities to track information even private permission block chains have the goal of leveraging decentralization across all the participants to increase the security and the reliability of the network but some of the initial ideas about decentralization have been kind of naive so Vitalik his earlier claim that aetherium should be able to achieve on chain scaling while running on nothing other than consumer laptops has been forced to confront some perennial challenges in computer science about data locality and constraints of networking so while that scaling work is ongoing and I certainly applaud it and grateful for their work their companies still need practical blockchain solutions for deploying their infrastructure that are robust and reliable enough for their applications right now so some companies rely on third-party RPC services and these are very helpful and they definitely have a place but they also represent a significant reduction in the overall level of decentralisation in the ecosystem so there are times when you're building an application that's important enough that you want to reduce your reliance on external dependencies as much as possible by deploying your own infrastructure something that's fully under your control and then you also have the added benefit when you do this that you're actually contributing to the overall decentralization of the blockchain so in practice we've kind of observed that the best way to birth tomorrow's decentralized infrastructure is to build on the reliable solutions of today so if you need your DAP to be highly available the underlying node infrastructure that it relies on needs to also be highly available by leveraging the services of an experienced cloud provider for your blockchain deployments you're gonna be able to take advantage of just a lot of different things so their unprecedented scale and efficiency and global reach and also the cost savings that come from like the pay-as-you-go model instead of the sort of over-provisioning your resources for periods of peak load right so before if you're doing an on-prem solution you have to kind of over you know stock up on resources just in case you need them and clearly this is less efficient it results in a lot of waste and one other huge benefit is that you know these cloud providers like AWS have been doing this for a long time they really know what they're doing and they're already running on over fifty percent renewable energy and our AWS is committed to getting that to a hundred percent so that's kind of a nice bonus so I'm going to talk a little bit about how the infrastructure works the AWS cloud is divided into several regions and each of them has something we call availability zones or AZ's so they have several of these and these ACS consist of one or more discreet data centers and they each have redundant power and networking and they're housed in separate facilities that are unstable floodplains though they're geographically isolated from each other for disaster recovery purposes they're also close enough that the pink times between them are very low so the RPC service deployment that I'm going to be showing you today distributes if there are M nodes across different AZ's to ensure that the service continues to function even if one of the AZ suffers an outage so I'm going to go through some of the architectural approach to this deployment first we have requests that are coming into the Amazon API gateway and in you know this Gateway checks to make sure that the requests are recognized and it blocks certain forbidden methods that are not really needed for everyday use that might incur additional overhead so it kind of keeps your waist down and then if the call is valid we pass it on to a network load balancer and that serves the request to several container instances and each of those container instances is running in a separate availability zone so in our experience the primary bottleneck that you incur when you run nodes is around IO so these are storage optimized I three instances they're optimized for very low latency high random i/o performance and they each have like an SSD based storage that you have access to on the on the container so to optimize the resources we run several processes or tasks per container depending on the container size and that technique is sometimes called bin packing but it's the idea that you know the container size you have available to you may not be the perfect use of the resources that each node requires so we add we choose how many nodes we want to run per container and these ratios that I've listed up here are you know in our latest testing these are the ones we recommend at the moment but they could change as block chain sizes increase or change so or you know if the efficiency of blockchain storage changes so one particular challenge that we ran into is how to get around the incredibly long sink times for new nodes so if one of our nodes becomes unhealthy we want to be able to start up another node quickly and with death taking several hours like almost a day to sink a new blockchain and parity taking several days we needed a better strategy than just starting our nodes from scratch and having them sync with the blockchain so we created a special task on one container that we call the updater task it's down in the corner there and this task runs a node that's periodically taken offline every four hours or so and copied the s3 and when you copy within the same region to s3 it doesn't cost any you don't incur any bandwidth charges so when new nodes start up they copy the contents of the blockchain state from that same s3 bucket to their local disk before they begin and then this you know enables them to start up within a few minutes if we're lucky it'll be like actually running by the end of my talk here so let's take a look then at the code for just a sec so I've got the repo right here for those of you who I'll leave it there for just a second longer for those of you who want the link of course this would be published later and in here you know I'm a strong believer in a project having all the information that it needs and it's read me to be able to reproduce it and run it and execute it sometime I think Chris landstrass one of the github founders called this readme driven development but anyway I think this is important so hopefully everything you need to be able to run this is there and I'm gonna just bounce over to my AWS management console to just see what the state of this thing is going so we launched this additional stack and it looks like the connection is so slow that it hasn't even fully kicked off here yet so it's probably not going to be finished by the time that I'm done so for that purpose I prepared a video just in case this hat was going to happen right I kind of had this feeling last night that I might run into this i tempted the demo gods right so I'm going to bounce over to kind of what you would see as this progresses a little further along so that kicks off this CloudFormation template and you get this result saying okay we've just started this and I've got it serviced that's I've already pre deployed that we can actually test against so we're seeing some of the RPC calls now going out to this node cluster and they're being responded successfully and because I have another node cluster running already I can actually do that as well so if I want to for example I first need to go to my API gateway and find out what the URL of my service is so we'll do that so while this is loading very slowly I can take you through some of the things that we're still working on this so this right now supports the parody client that was one of the you know at the time we first developed this parody was a little more stable so that was what we were focusing on and I am very close to releasing to merging in a PR that will support death as well but it's actually pretty easy because of our approach of just kind of copying the contents of the blockchain state off disk and syncing it with s3 it's very easy for us to kind of support whatever node service we want to so as you know eath 2.0 comes online and a lot of new EVM implementations are coming out this shouldn't be too hard so let's see alright so got that URL and we get a successful result very slowly because of the network it's it's not the service I promise anyway so this is this has been a very stable successful deployment that's been running for some of those companies that I mentioned earlier for quite some time we continue to maintain the tezo's foundations note infrastructure in the similar way and happy to answer questions about how we did anything and talk with you all offline after this but thanks a lot for being with me today you