Powering Progress via Partners: Middleware's Scaling Role | Brian Robbins - Coinbase Cloud

[Music] hello good afternoon uh my name is Brian Robbins I am with coinbase and I'm here to talk to you today about uh powering progress via Partners uh basically what kind of Technology we use in our staking platform and uh how the different ways that we look at the underlying technology the types of blockchain technology that we use and overall the way that we think about things and look at the different technology options that are out there and how that leads to the types of decisions that we decide to make with them uh so first I'll start off with a little bit of background uh I lead the engineering team for our coinbase Cloud staking this is all of the validators that we run for our internal customers our uh Prime clients retail as well as a number of our external customers my teams operate validators and all of the tools in Tech that support this operation so we run validators across over 15 protocols that represents over $15 billion dollar in assets that are staked and we've been doing this since 2019 we started running ethereum in 2020 uh We've one of the longer staking operations that's out there it's really about doing this at scale and doing this really well uh for further background coinbase we're known as the most trusted name in crypto this comes from one of the most important things that we do it's always about trusting and making sure that our customers can trust us it's having the security and reliability making sure that that always comes first and the number one thing that we can do is make sure that we protect the assets and protect the networks that we operate that's something we take very seriously and that's always our first job job zero is making sure that we take care of customer assets and that's how we protect ourselves as being the number number one trusted name and crypto after we get past that we start talking about scale I mentioned the 15 different networks that we operate the large number of assets sted and it means we need to do everything and think about the scale of operations that we bring to the table it's really about bringing efficiency the highest levels of per of performance across all the networks that we operate and trying to do that with as minimal human effort as we possibly can uh we need to not just throw people at the problem we need to be able to do this reliably and with scale so looking at how do we eliminate the manual operations that are out there the oneoff oneoff processes uh it's all about repeatability you if we have to do something once that's fine but if we have to do it more than once we're going to write scripts to do that we're going to create tools to automate those processes and overall we think about this as you know humans humans exist to introduce errors so we try to minimize the opportunities that humans have to introduce those errors into the the systems and the things that we do uh we consolidate our tooling as much as we can uh try to consolidate our configurations it's really about trying to use the same tools across all of the different networks that we operate because ultimately if we can get to a place where many of the operational issues that our teams face don't require int don't require deep Network experience that lets us scale it lets us scale the people it lets us scale the tools it lets us scale our operations a whole lot better and ultimately that's where we are at is that many of the operations issues we have today don't require that deep Network expertise because we've done a good job building out some of these tools and trying to eliminate snowflakes or places where we have a unique setup for one protocol or one validator Trying to minim minimize and prevent that from happening as much as we possibly can and all of these are the ways that we bring to the table so that we can operate at the scale that we need to so bringing that to the underlying technology and some of the the choices that we've made here uh first off we're multicloud multi- region again large number of assets large number of protocols uh we primarily host through AWS gcp I and we also further decentralize through multiple regions we've have uh regions across Asia Pacific uh North America and Europe so again making sure that we're decentralized across the full footprint that we can within both AWS and gcp and it's all about ensuring that there's no single place where a failure can take down everything uh we have to accept the fact that there are places where you know an AWS region may go down a gcp may have networking issues we need to ensure that regardless of what what happens there our networks and our protocols and the validators we run are resilient to those kinds of errors and this helps us build and scale quicker uh building on top of AWS and gcp especially really early on uh you know back in 2019 2020 as we were getting this started let us not focus on how do we set up and and maintain these machines but how do we just sit on top of AWS and gcp not have to worry about that underlying Hardware this was a really big part of our early growth and now we feel like it it really helps us scale and make sure that we have a nice mature scalable platform to build everything on but it's challenging if you've ever tried to build things on AWS and gcp uh there's a lot of differences there they're not exactly the same and so this is where we bring in kubernetes and terraform and uh so you know using eks gke this lets us build everything on top of the cloud providers and and lets us normalize away a lot of those differences and the the changes that are between it we also very much bring to the table infrastructure as code you know build in our infra um our metrics alerts dashboards everything gets configured everything gets set up through code so that we can easily recreate things we can manage the change we don't again have snowflakes we don't have one-off processes we don't have a lot of manual errors that can put get put into the system again all of this bringing back to how do we scale better and how do we uh bring this stuff together and it's about abstracting away these differences and making all of the different protocols all of the different places we host look as similar as possible to each other then sitting on top of that and you're coming up the stack Again One Step Above It uh is our cicd platform so we use uh Concourse CI this manages all of the builds and tests and all of that kind of process we also built something that we call protocol pipelines and this is a a set of internal framework that sits on top of concourse it's very specialized for protocol deployments it helps us set up the test Suites it helps us set up the staging environments the test Nets uh the main Nets build out all of our deploy groups all of this stuff kind of sitting on top of concourse CI again bringing it back to how do we do this at scale across all of the different protocols that we work and operate and then we have a bunch of other internal build tools that we've built out over the years most most L again looking at what are things we have to do more than once and anything we have to do more than once we look at how do we build a script how do we build a tool around this how do we make it repeatable so it can either do it automatically or do it at the push of a button without needing a human to kind of go in and set manually set the right parameters and configurations every time they need to go do something that we're going to do regularly day over day week over week so coming up the stack one one more step uh I want to talk about one of the pieces of Technology we introduced in the last couple years that has really been one of the largest unlocks for our team in the scalability we call it protocol cdk and this follows the AWS Cloud developer kit model um or there the AWS cdk model and what it did was it brought a single language typescript in our case uh and through that interface we now configure every protocol that we work with prior to this if you were setting up a protocol you might be dealing with four five six different languages depending on the protocol depending on where you needed to go you had uh various you know configuration files a bunch of different technology places and once we introduced protocol cdk we got that down to one single place generally one single script so anytime you're changing something with any of the protocols we operate you can do that in one place it makes us build faster it helps us actually onboard protocols when we're launching new chains and new networks we do this three to four times faster than we did before but more importantly we do that with even higher confidence we're able to bring in the same set of testing sues we're able to bring in the same set of alerts and metrics and alarms all of that stuff comes in for free because we have it built within cdk and just by virtue of using this tool set we've simplified that work we've abstracted away a lot of that work so we move faster and more reliably at the same time and this has been again a massive unlock for our teams over the last couple years and really has helped us scale and get to where we are today so that's some of the kind of generic technology that sits underneath our stack next I want to talk a little bit about some of the blockchain technology and some of the things that we're excited about some of the things we're looking at some of the things that we've introduced over the the past few years so first up is me and we started with flash Bots we introduced this in 2022 uh We've since then introduced a lot of multiple other relays uh we added jeto for salana um and the goal here is really we wanted to increase the yield for our customers but we also being very crypto native we want to make sure that we maintain proper proposer Builder separation and this is again one of the things we're trying to pay attention to is when we're looking at technology what does it do for our customers uh in this case me helps increase the yield that we have for customers but we want to do this in a way that sticks to the ethos of crypto and make sure that we stay credibly neutral we want to keep everything very aligned with the mission and and the ideas behind the folks that are trusting us to help operate these networks and ultimately we've seen a lot of success with me it has increased yield it has brought better returns to our customers and we're able to implement this and do this at scale uh something that we're paying a lot of attention to now and looking at as I'm sure many of you are as well is the reaking so looking at you know igen layer obviously there's a number of other players in this space too uh and the idea of reaking if you're not familiar with it is you can take the e that you've already staked on a validator like coinbase and you can then go restake it and basically commit and and use that uh that asset to help secure an additional use case and those additional use cases could be things like a data availability layer oracles other networks which is kind of similar to the way Cosmos Hub works and the consumer chains they have there but this is a place where it also introduces a huge amount of complexity there's a ton of unknown risk you're allowing uh the IG layers or the the resters to have the ability to slash your tokens again as a result um as as part of the the tradeoff of using it to help secure their their Network or secure their additional uh tokens or uh tools that they want to introduce and so this is a place where we have to pay be very careful and pay attention to the balance of innovation security and reliability you know security is always up up front and security is always first and foremost for us but we have to pay attention Innovation and ultimately we would like to increase yield for customers and we see reaking as a great way where there will be lots of opportunities to go ahead and increase that yield but we've got to make sure that we're doing it in a way that makes sense and and doesn't introduce too much risk into the the products and the tools that we're offering and as we're looking ahead a little bit further uh DVT and this is something we were talking a lot about last year here at eth Denver something we're still very excited about you know there's tools from uh multi multiple providers Obel SSV are two of the ones we've paid a lot of attention to Super interesting potential here the idea behind DVT is that we can instead of having a single validator run an eth node we can distribute that across multiple validators maybe a set of where we need three out of five to run or four out of seven to run and be online and that gives an operation like ours the ability to scale even further it lets us do the kind of Maintenance and upgrades and security patches that we need to be doing and we can do that without ever having to take a validator offline you know one of the most important things to us again security means that we have to stay on top of security patches we have to stay on top of everything that we need to do to to trust and and keep everything reliable and tools like o obal SSV and the DVT that that comes in may let us do that and may let us increase that uh that security posture and do it in a way that lets us scale reliably but it also introduces a very complicated extra layer into our platform this is why we're still paying attention to it it's still something we're looking at because that complexity piece hasn't been worth it for us yet but it's uh it's something that I'm still very excited about looking ahead and that kind of brings us to the way to that we make decisions and what are the key things that we're going to pay attention to I've talked a little bit about some of these as we've uh been going today but it really comes down to three major components and the first is trust security and reliability is always the number one first and foremost thing that we can do that's the number one thing we have to pay attention to when we're looking at something like Obel and SSV for DVT uh some of this comes back to the teams and the underlying teams that are building it as we're looking at uh reaking and Igan layer not just the igen layer team but which AVS operators are we going to uh potentially introduce which uh which projects are we going to allow our customers to to support and we need to to understand we we do have a role here with our limited bandwidth that we can't just adopt everything and so a big uh one of the first big places that we're going to pay attention to is who's the team behind it um what does their code look like how are their development practices and is this something we can trust where it comes to Smart contracts you know have those smart contracts been audited are we able to look at the code are we able to to give our own stamp of approval onto the things that we operate and bring into the platform because again security has to come first and there's nothing that we're going to put before trusting and and taking care of those customer assets the second piece we pay a lot of attention to is the operational impact of what we're introducing again large scale lots of protocols we're very careful to introduce snowflakes and and prevent that from coming into our systems and that means we need to pay attention to the operational impact there's some of the protocols that we've operated and have actually decided to stop operating over the last couple years primarily because they were so difficult to run they were so Hands-On they did have so many challenges and so much impact uh on our teams and on our ability to run them effectively that it just the tradeoffs didn't make sense anymore so we're going to pay a lot of attention to that operational impact um part of my job a big part of my job is to protect our team protect our operations and this again comes back to protecting our customer assets if a team is up all night every night dealing with issues because something that we've deployed is unstable that makes them less likely less able to to respond to to things during the day and we've got to make sure that we keep our team fresh we've got to make sure that we keep our team operating at the highest level possible so we're very sensitive to anything that'll reduce the operational impact uh or operational effectiveness of what we do and the third thing is is customer impact you know after we've taken care of security after we've taken care of the operational impact what is it doing for the customers that want to use our services is it increasing rewards uh is it bringing more optionality for customers uh we are here to serve our customers and our goal is to make sure that we're taking care of the networks the the primary networks that we're doing and that's really on behalf of the stake that our customers are putting their trust in us uh to to represent and so we need to make sure that we're delivering the things that our customers are looking for that we're offering the opportunities that they want to see and that we're giving them the chance to to be able to allocate their assets to help protect the N networks and tools that they're most interested in and excited about um so that's really you know what it comes back to at the end of the day when we're looking at what are we going to support it's what do our customers want what's the operational impact and can we trust it can we trust the team can we trust the technology and everything else around it and that really kind of wraps up uh a very quick highlight of how we look at things at coinbase and the way that we approach our staking business and our validator infrastructure I really appreciate your time today and uh I'll be over here on the side for for a few more minutes if anybody has any questions I'd be happy to talk to you thank you