🛠 Working with Blockchain Node Provider Infrastructure - Jeff Wentworth

foreign [Applause] [Music] my name is Jeff and welcome to working with blockchain node provider infrastructure I'm one of the eth global mentors I'm also a co-founder to a company called Curve grid we're a blockchain infrastructure company based in in Tokyo Japan um so this this session's a little bit different in the sense that a lot of the things that I'm going to talk about today are less applicable to a hackathon and a little bit more applicable to after the hackathon building you're taking your dap that you've hacked together over the weekend and turning it into a production decentralized application so that's just something to keep in mind this weekend this hackathon do whatever it takes to get things done to get things working but I think this is sort of after the hackathon after the after the weekend what are some of the things you can do to build a production dap so uh we're gonna go through sort of three different sections today the first is a quick overview and reminder refresher on how dapps interact with the blockchain we're going to talk about the blockchain as a distributed system and these two parts are very important um uh to think about uh the third part which is how we actually work with blockchain node provider infrastructure everything I'm going to talk about today is also 100 applicable in the case where you run your own nodes but there's some extra things to think about if you're going to use a a node provider what is a node provider node provider is a company like quick node or organization like pocket Network infra alchemy that runs blockchain nodes on your behalf and that you use as a service so blockchain 101 I think everybody knows exactly what is going on in this diagram you have a Dap with a front-end a back end there may also be other components like a database it may talk to a third-party API and of course it's talking to the blockchain a blockchain node this is the part that we want to focus on today right the blockchain node interaction I'm going to go fairly quickly through this part because I think a lot of this is refresher but again I want to set the stage uh for for the kind of the meat of what we're going to talk about today so how does a typical dap interact with the blockchain well it can call Smart contract functions to read data it can query and listen to events and those are two distinct things right querying events is reading past events from the blockchain and listening to events is listening to a present and future events from the blockchain we can do other things that don't involve smart contracts like reading each balances and then we can write to the blockchain by calling a smart contract function to compose a transaction signing it and submitting it to the blockchain so these are different typical dap interactions with the blockchain you would use these interactions or you would compose these together to build daps that would do any number of things here's a few examples of things that dapps typically do display a dashboard of token balances of a particular user uh display those token balances sorted by balance so taking all of the token balances whether it's 10 or 100 or a million and just and sorting them by balance displaying all the nfts that are owned by an address and other kinds of essentially blockchain state aggregations right so the first three of these really fall into the category of I'm reading some data from the blockchain I'm processing it and I'm doing something with it I'm displaying it to the user or using it to make decisions um Etc still setting the stage here some common smart contract interfaces and I just want you to keep these in mind when we talk about uh some of the the things later in the talk so I think we're probably all familiar with the erc20 balance of function provide an address get back a balance another one that many of you are probably familiar with is the ERC 721 innumerable function which allows us to basically provide an address and an index of the nfts that that address owns retrieve a token ID and this is in a case you would use something like this ERC 721 maneuverable or any kind of innumerable smart contract capability when you want to basically ask a smart contract what are all the things that you know about I want to go through them basically in a loop from off chain um and then ERC 721 transfer event right and events of course are emitted by smart contract functions and you listen to them off chain and process them and they're intended to be used by smart contract developers to indicate a state change on the blockchain to make it very efficient to build scalable dapps so again just keeping these in mind for the the rest of the the rest of the talk so some strategies for reading data from the blockchain right again keeping in mind what we've just talked about you can call a smart contract function you can call a balance of or you can call one of the enumerable functions but then the question becomes how often do you call it do you call it every certain number of minutes and cache the results do you call it every block are you doing this per address if you're building in a hackathon it it's fine because maybe you've got one or two or three addresses but what happens when you need to start enumerating through the balances of nfts in a loop for 100 000 addresses is that the kind of thing that you can do every single block or not Well turns out a much more efficient way to do that of course is to process is to listen to events process them and cash them off chain um and this is of course something you can either build into in-house or use open source or use what we would call a third-party chain indexer service this would be something like the graph or you know our own multibass event event monitor capabilities or you can also use a third-party higher level API so something like an nft Marketplace API that will just give you all of the nfts at a particular address owned or owned at a particular point in time or all of the defy prices and balances the key difference between these three is the first two are essentially you know much more decentralized and you have complete control over uh the data that you're getting how you're processing it um the the certainty or or um quality of that data right if you're reading it directly from a blockchain node or from a node provider um uh you're calling a smart contract function or processing events if you're using a third-party higher level API the trade-off is much easier to implement right because they've done much more of the hard work for you probably cheaper but you're giving up some of that control and your uh and and knowledge over let's say the quality of the data and I'm I'm not advocating for you should do this or you should do that it's these are all things to be aware of right these are the trade-offs that we make in software engineering when we're building applications Okay so we've talked about the apps we talked about reading data and some you know if there's anything to take away from the first part of the talk it's that there's different ways to read data and the way that you read and process the data depends a lot on the size of your application right another thing that uh how when you're architecting a Dap and thinking about working with blockchain nodes is the blockchain itself right so we're going back to blockchain 101 this is a blockchain it's a decentralized ledger it consists of nodes communicating in a peer-to-peer fashion over the internet uh and every node of course keeps a Blog a copy of The Ledger I think this is is pretty um familiar to everyone here the key thing to think about with this is that a blockchain is essentially a type of distributed system and classic distributed System Theory or or uh you know practice we have the cap theorem consistency availability partition tolerance you can only have two of three of them uh ethereum and most evm chains optimize for availability right you'll always going to be able to get data from a node that you're talking to and partition tolerance they're able to survive the blockchain uh partitioning in different ways right the there's a network break I mean there's an earthquake and an undersea cable gets cut and Asia is separated from North America um or you know there's some kind of network congestion or an attack or a bug in in node software um the compromise that that they make is on the consistency and the key thing here is eventual consistency right and when we're building a Dap when we're thinking about how we interact with blockchain nodes or node providers we've always got to be thinking it's eventually consistent it's eventually consistent right and that drives a lot of the complexities around the systems that that you're going to that you build in a production dap so um I mean there's so much to talk about here but we're talking about uh ultimately node provider working with blockchain node provider infrastructure and not uh um you know different ways that nodes re-synchronize but I so I'm just going to keep this very simple and say um the nodes can get out of sync and I've written here not just nodes but also the network right and so what you have to start thinking about is I might be talking to a blockchain node but actually I have to think about all of the components in the system as being a distributed system right and again in hackathon it's very very simple you have your python script or typescript or JavaScript or go or see that's or Java that's talking to a blockchain node uh doing a very a single blockchain node potentially or a test Network it's doing you know very low volume of transactions but as you start to add more components into the system the entire system itself really becomes a distributed system and you have to start thinking about wait a minute there's these eventually consistent portions uh throughout the entire system right not going to go into full distributed systems 101 but it's just something to keep in mind that again this isn't just blockchain notes this is the entire system that we're we're thinking about and so the key thing here is how does a client that's connected to one or maybe multiple of these nodes handle conditions like this and when I say multiple nodes again I don't just mean multiple blockchain nodes it might be different parts of the infrastructure right my client might be connected to a blockchain node here but then sourcing some data from a pricing API or other data from an nft API do all three of those have the exact same view of the world at the same point in time and the obvious answer is no they don't so how do we start to think about that so the big question should you run your own blockchain nodes and um this is not controversial at all this is just Jeff's opinion um but it's it's um uh sometimes kind of becomes the elephant in the room my overriding long-term view is yes we should all run our own blockchain nodes no question but we're not quite there yet right um and so what I want to do is try to enumerate some of the reasons to think about running your own nodes versus Outsourcing this to a node provider and again I don't work for a node provider but we do make heavy use of them so we have a lot of experience with both running our own nodes and um uh and using blockchain node providers great you you know good reasons that you may want to run your own blockchain nodes over education hackathons you're learning for development and testing right you're doing a lot of iterative potentially destructive uh uh r d activities and testing activities you may need access to the internals of a blockchain node um for example you may want to enumerate what we what some people term internal transactions which is State changes that aren't exposed through the the standard RPC API like each balance changes or contracts being created within other contracts or you may have other very specific performance requirements right I think in a production dap or sorry not a production app in general these would be great reasons to run a blockchain node a reason to use a node provider the first one I think is cost right anybody here who has run blockchain nodes in the past understands or currently understands that if you're doing it in production and we're talking highly available you're running adapt that's being used by uh you know in in Anger let's say or in uh um you know at a good scale by a large number of people it costs a lot of money right because you can't just run uh you know start blockchain node and you're done I mean we're talking uh you have to think about what happens if the node fails what happens if the server goes down what happens if there's a problem on the network right now you're you're running real you know production infrastructure and there's a cost behind that not only the actual let's say server and software cost but engineering time and effort and as with many things in this world it's often easier to let somebody else do this I think a good analogy for this that we can all keep in mind is it's no different than making the decision between running your own server whether it's a Raspberry Pi on your desk or a server in a data center or a VPS virtual private server through a service like digitalocean or vulture or use a cloud provider right Microsoft Azure Google compute platform or AWS and there's no right answer right it's going to depend on your use case and what makes sense to you but I would I would argue that um for most production dapps the answer today is node provider right so we've talked a lot until this point about some of the ways that we think about reading from blockchain nodes interacting with blockchain nodes now we're going to talk about um not just no you know how how we think about interacting with and architecting adapt to work not just with blockchain nodes but the particulars of node infrastructure uh as well node provider infrastructure as well so speculative node provider this is um you know just a block diagram um and and how it might look if if uh you know Jeff who is not a node provider was was thinking about building it this is what you think that you may be talking to because this is what when I'm developing on my laptop it's spin up a local um you know ethereum uh Dev client and I I'm I'm connecting to it um most dapps many dapps I should say will be using one or two kinds of clients right https for um doing smart contract function calls potentially and then web sockets for listening to events and that's that's critical right you many dapps need to have both of these capabilities but what you're actually talking to when you're working with a blockchain node provider looks something like this and you can Google uh I mean there's there's lots of talks on the Internet by various node providers and some of them throw up diagrams that look very similar to this if we start at the at the bottom of course it's not just one node right they're running multiple nodes again for redundancy on top of that there's some kind of load balancer that's uh taking block updates taking event updates from the nodes and um you know essentially arranging them right am I getting this this node is having a problem and it's 100 blocks behind let's drop that and we'll take the um uh you know the data from the other nodes that's what let's say the node the load balancer the node load balancer would be doing on top of that I probably at some point want some kind of distributed cache right the reason when you um when you do certain kind of uh um Json RPC calls to a node provider that the responses come back faster even than your local node is the that they're not going all the way down to the node they're saying oh you want blocks 0 to 10 000 from three to go uh well great I'll just serve that for my cash and there's some kind of local cache and then they've probably got some kind of large scale uh distributed cash off to the side as well then on top of that they're going to have multiple https and websocket servers again with a load balancer in front of that and all sorts of anti-ddos protection to actually talk to you the DAP at the top so you know we talk about the cost of running your own node in production in a generalized sense like these are some of the things some of the infrastructure that you invariably end up having to build and operate that you end up being able to offload to a node provider that can be very very helpful it's great everything is sunny and wonderful rainbows rainbows and unicorns um you know I've got my dap it's got a high performance node provider I don't have to worry about running nodes but what do we have here we have a bigger more complex distributed system right and the key Point here is that any one of these components can fail and um you may be the one to notice the failure right you may be the one to notice the issue or you may be one of the one to notice the bottleneck maybe I want to query not 10 000 logs but Ten Thousand and One logs and I get an error back which I didn't get from a different node provider which allows me to query up to 20 000 logs or um you know I have multiple websocket connections from adapt let's say four and one of them is misbehaving well maybe there's an issue on a particular websocket server right and so um these are some of the things when you're starting to architect or build adapt at scale that you you have to start thinking about um so um yeah I've talked about some of these but just to kind of enumerate them right um using a node provider doesn't get us away from node issues right if a blockchain or a you know a blockchain has issues at the at the core level or is taken offline for a period of hours like you know there's going to be note issues we can't we we can mitigate some of that because we're offloading the upgrade and um management of those notes the node provider but at some point we could certainly be exposed to that there can be load balancer issues I mentioned that again you think you're connected uh you know some of your websocket connections or some of your HTTP requests are fine but others fail uh cash coherency I'm getting back um data that does not match what maybe I'm seeing on an uh block uh blockchain a block Explorer like etherscan right or polygon scan the data here is not the same as the data there and um you know what do I do about that um there could be an issue with the overall node provider there's many node providers just like there's many Cloud providers um there could be an issue with a particular one that's let's say a concentration risk if I only work with one node provider and then I have to deal with all the fun things around the blockchain right uh reorgs blockchain reorganizations uh blockchain forks and I have to be thinking about that and these are these issues are no different uh than if I was running my own my own blockchain nodes and then of course dap issues like I've there's something broken in my dap and is it my dap or is it somewhere in the infrastructure as well so for all of the benefits that we get from uh performance increased performance not having to worry about upgrades etc etc there are other cons there are other things that at some point we're going to have to deal with right um I would love to give you all the answers and you know you should use this pattern and do this and do that but I think probably what you're figuring out is this talk is more about to getting you thinking and posing questions uh rather than um handing you the answers because you know like many things in software engineering um it's it's the journey not the destination and that's that's how it's always going to be and that's a good thing that's that's interesting and exciting um so just adding some questions to this so thinking about all of this together right and what does this really mean the first question is how much downtime can you survive and anybody here who's worked in um big it for big corporate companies the answer is always zero I can afford zero Town time but that's not realistic right everything goes down sooner or later the question is how much you can survive in classic sort of let's say business continuity um you know Theory or or thinking you've got two two considerations right one is recovery time objective and the other one is recovery Point objective recovery time objective really means uh how long does it take to get yourself back online either through failing over to Alternative infrastructure in this case if you're running your own node it would be spinning up another node you know Docker run Docker uh and and getting that working and switching everything over if it's a node provider it's you know switching over to the other one if you've got an account or madly creating an account and putting your credit card in or whatever and and switching everything over that's recovery time objective how fast can you be on uh be back online and the second one is recovery Point objective how much data can you afford to lose right the blockchain is eventually consistent and and very strongly eventually consistent and it's proven that over the last uh seven years with the ethereum blockchain and longer than that for for some other blockchains as well um but what happens when you're doing things like storing data that's coming from the blockchain in your own database right and that gets out of sync um somebody you know uh put some data into uh mint and nft and that's some of that's gone into your database and some of it's gone on to the blockchain because you're doing some kind of semi-decentralized um activity or maybe it made it onto the blockchain but not into ipfs right there's data loss there uh how much can you survive and that would be recovery Point objective and again there's no answers here it's questions it's going to be different for each production dap um how much data loss can you afford uh and um um yeah this relates again to recovery Point objective actually uh how much how much data can you can you afford to lose how much redundancy do you need again I need uh I need one one server on every continent I need uh two servers in in every city I mean that's that might be realistic someday but there's a cost to all of that right and and it's it's not the kind of thing that is going to be realistic at the beginning so you have to think about that how much do you automate right if something breaks do you even have the ability to tell is this a real problem or is my monitoring just broken and I think uh my one my one suggestion here would be automate very little at the beginning um and then uh and then enough so that you can sleep at night and not get not get woken up and then what's your budget right um one thing I want to talk about here around uh budget is that um you know no providers can be inexpensive uh they can you know I listed cost is one of the reasons to use a node provider not potentially and not run your own nodes for production applications but a lot of that depends on how you architect your dap right and it comes back to everything we talked about in the first part around am I am I enumerating every nft that all of the the the users or that I know about uh every single block am I um am I making requests from six different servers in parallel and uh you know pretty much every blockchain node provider out there has some kind of concept of number of requests per month and I think one Trend that we've noticed in the past six to 12 months is often have been differentiating different kinds of requests based on the complexity so for example if you just want to know the network ID they may charge you at a multiple of one if you want to make a smart contract function call maybe that's a multiple of five if you want to query past logs which uses their you know very Advanced caching infrastructure and gives you response very quick maybe that's a multiple of 25 and so on and so on and so on and I think one of the challenges there is trying to compare apples to apples uh in terms of cost and performance and what they offer and that's just just something to to keep in mind right um and of course you can trade off some of these costs by doing your own caching as well but that that of course brings in additional complexity again not for eth Bogota this is all production uh dap for for when you want to scale after the hackathon um so thank you I do want to highlight one of the things so again um we have this uh we get a lot of questions around uh oh first of all let me also say we actually work with basically all of the node providers so we have a uh the company that I co-founded curve grid we we have a blockchain middleware and we have relationships and business Commercial relationships basically all the node providers and they're all great in their in their own special ways but one question we often are asking ourselves is um uh uh which one should I be using at this point in time or which one should I be using for this use case so we've actually put together a public dashboard um called web3 Arena we haven't uh officially launched it yet but I'm happy to share it with you here and basically it gives you a few different metrics and we have more that we're going to expose over time around um the responses that we're getting from different node providers so right now I think I have the last 30 days polygon mainnet selected I'm sorry that we don't have all the node providers out there we're certainly working on that we'd love to talk to you if you're a node provider and add your your data into here each of these different boxes here these four boxes sort of describe the kinds of data that we're looking at so we're looking at the latest block number by calling F block number um on a five minute interval and as you'd expect they're all pretty much in lockstep with one another we're looking HTTP errors so it's really quite brain dead simple we um we just call F block number again every uh five minutes and see how often we get back an HTTP error if there's no if there's no lines that's because we haven't received any errors in the last 30 days I think one of the ones that we're very interested in because we do a lot of work with events is websocket disconnections so many node providers that we've found uh architect their hdps stack very differently from their websocket stack but websockets are the only way that you're going to be able to listen to events and so in this way what we're doing is just doing s subscribe which is subscribing to new events to the zero address so an address that's never actually going to receive events and we just want to um we just want to see how often we're getting uh disconnected the reason a disconnection is uh important is it's effectively like we've lost connection with the one part of the distributed Network and so when we reconnect we have to do go through like a re-synchronization process and anyone who's run a chain indexer used a chain indexer before knows that that starts to get very very complicated so web3arena.io would love your feedback ahead of ahead of the launch you know looking to add more and more metrics to that and and just you know operating it as a service to the community who can um you know in case you ever want to know which node provider should I be using for my particular blockchain my particular use case at you know every given point in time thank you my name is Jeff at the global Mentor I'll be at the mentor table and walking around it in the Discord you can also find me on Discord as well um and that's it we've got a couple minutes left any questions in the room I presume we can bring the mic around or you can just shout it out and I'll repeat the question going once going twice all right I'll hang around afterwards happy to have you come up and ask me any questions thank you very much