Streamlining Blockchain Data Access | Carl DiClementi - Quicknode

to access it as well as how to make a blockchain ETL that doesn't S now this is all going to get a little bit conceptual right um blockchain data slides don't really go well together so we're going to have our friend help us out meet Chad the djen um Chad invests in dog coins Chad sends GM messages in hundreds of Discord servers every month just to try to get nfds right um Chad has his crypto spread against dozens if not hundreds of wallets connecting conducting hundreds of transactions so that he can Farm all the tokens that he can get um for most of us that are building products right uh Chad is our Target demographic he generates volume he pays transaction fees and he's very active on social media talking about our products so um you're right Chad you don't struggle with blockchain data everyone building products for him does so um you know what we do is uh blockchains offer this really awesome decentralized global data sore right no one needs permission to be able to work with it um and with that Chad's able to send crypto to anyone in the world for really low transaction fees 24 hours a day right he can even interact with smart contracts he can mint nfts do nft swaps ET Etc cool uh so what about all of us that build products for Chad right uh how do we retrieve the data that he needs right all of his D gen activities Chad expects a web2 quality experience with personalized experiences and fast load times this should be easy enough right all the data on the blockchain is public um how hard could it be well just because something's public doesn't necessarily mean that it's easily accessible right what is accessible is access to blockchain nodes and as what many of you may know right access to blockchain nodes means that you're using an API typically referred to as an RPC API well this is you just out of blockchain development 101 Building your first application right you're attempting to make your RPC calls and you're finding hey this is really easy I can make an RPC call to get a transaction and I can see a nice and easy right um it's free in a lot of situations you can go sign up for a free account on quick node various other providers that you probably shouldn't sign up on um or you could run your own node um and that's great right you can immediately get started you can access the data right away and the apis are standardized right no vendor lock in what other place in the world can you access that that's awesome but when you first start using RPC API guys then you get humbled pretty quickly you start to find that the data is encoded right uh decoding interactions with smart contracts requires access to these ABI files that are hard to find um and you know it they're also not on chain which makes it extremely difficult um data when you want to extract it you have to extract it block by block you can't just say something like hey for Chad give me all his transactions that he's had you have to slowly go through and say okay in Block one let me filter down to the transactions for him block two block three block four which is extremely inefficient right and also the transactions that are happening that are associated with Chad aren't easy to just see hey let me see his two and from address of a transaction and that's the ones that are associated with him no it's a lot more complex than that based on the smart contracts he's interacting with so really to uh get all of the information that Chad needs you could be doing billions of API requests across chains so in summary right node apis are good for requesting a single transaction right or are they let take a look right here at the AP response for a simple transaction for a usdt transfer yeah I can't read it either um so to decode this right we need access to the ABI and while once you can find it it's actually pretty easy to show something like what ether scan is showing right here right uh for those of you who over there can't see it just says hey it's a transfer of 40 tether from Chad to random address so Chad's happy right he can see his transaction but what if we want to show him a more advanced experience something like hey what are are all my erc20 token transfers well that would require millions of API calls to the node something that Chad is not going to wait for or be willing to pay for so enter enhanced apis um some really good examples that you guys are probably familiar with are things like token apis nft apis transfer apis Etc um this is really nice right they handle some major use cases very well um they provide the data decoded easy to display to Chad no longer worrying about finding Avis Etc they're typically really inexpensive which is very nice um and you can immediately start using them you're starting to hear me say that um because that's not going to be true for some of the ways to access the data L uh that I'll mention later unfortunately not all enhanced apis will meet your most basic needs um there are exceptions to everything if you want to have an nft API right that might not support something like Crypt punks which doesn't meet nft standards right um and if Chad owns a crypt punk you better assume he wants to see that in his wallet um so you know uh the premium providers of course they're going to support those things but they start to become more and more expensive so where you'll start to run into more challenges is something like hey I want to do something unique that's different from every other platform out there right I want to present the data in a unique way I want to be able to pull some unique data off the chain well with enhanced apis you're kind of stuck with the standard interface that everyone else is getting a fixed set of API methods with a fixed set of fields that'll be in the response um and if you want to start enhancing that data then you have to figure out all right well I'm going to call this enhanced API to be able to get an nft purchase and then I'm going to call this other one to be able to get the price of ether at that time so that I can say hey this is how much it was in nft uh in US Dollars and that just starts to get pretty cumbersome at scale um another example is if you want to start to do some you know custom logic like hey show me the total balances across 50 of Chad's addresses right Chad doesn't necessarily care about like oh I want to go to one wallet and then the next one and then the next one and then I want to do all this math in my head right he just wants to see his total balance right his total net worth to do that you're going to be stuck making at least 50 API calls if not also calls to a uh price API so nft apis where they're good for really straightforward standard use cases if you're trying to do math or any form of aggregation with the data you start to run into issues raise your hand if you've heard of subgraphs great okay uh raise your hand if you've worked with something like the graph or Z good okay so people are familiar with some of the challenges here but for those of you who aren't um subg griffs are an extremely powerful tool um it really actually seems like magic when you first start using it uh all you have to do is you start to select something like hey here are the contracts I'm interested in um make some additional minor conf configurations and then come back in a few days and it'll have indexed all of the transactions events Etc um once indexed you can really query it very powerfully usually through a query language like graphql um some people find this to be advantageous um others find it to be a challenge because it's a completely different networking layer uh at their Edge that they have to interact with um but it does allow you to do flexible queries something like hey give me all of the addresses that Chad has ever traded with right uh which for a wallet could be a really useful thing to provide so that you're not accidentally putting in the wrong address you just reuse a previous address you've worked with um also there's a lot of value in some of the features that the graph provides where there's like pre-made off-the-shelf graphs uh subgraphs right a really good example is misari they make a lot of them available and easy to use where people start to run into challenges with subgraphs is kind of the initial indexing time this is where I was getting into with the fact that enhanced apis and RPC allow you to immediately start using them that's usually not the case if you're making your own subg Griff there's a time in which you need to wait for the index to get created um and that's not really that big of a deal in isolation if you're just launching a new product it's okay all right I'm going to take a few days I'm going to build up my index and then I'm going to launch it but the problem becomes if you run into an error during that indexing you have to redo that index again and if you have a bug in your system Chad's not going to be happy to just have to come back three days later for it to be resolved right um so that's not actually the only delay you'll experience on some graphs a lot of them have a little bit of a data lag for the data to be processed on chain then go through the data pipeline um so if you want to give Chad the ability to be up toate on the latest activity on chain so he's most informed this might not be the best solution for you um one of the most significant challenges that we found that people run into actually is that subgraphs and the processing are actually a little bit of a black box um so if you want to be able to process something like offchain data and Pull It in while some platforms do allow you to do this and have this composability it becomes a really big challenge to be able to debug that um and we found that our customers prefer to be able to pull things into their own data warehouse and process it through their own data pipe pip lines so overall subgraphs extremely powerful Tool uh but the feedback that we hear is that they prefer to have their own data warehouse so uh this data warehouse right by pulling the data into a data warehouse and building your own custom index you've unlocked Jedi Chad uh the possibilities are pretty endless when you have the data in your own data warehouse right you can do any kind of visualization or personalization for your users you have access over SQL or you can write apps above it uh to interact it with it in any language of your choice um and it has tight integration into your existing data pipelines let's say you have a web 2 platform and you're doing some form of processing uh maybe bot detection as well things along those lines you can do that powerfully within your own Warehouse um also you have composability with additional data sets that is a huge challenge with subgraphs something like even cross chain data gets very difficult with subgraphs so this is a way that allows people to do that at scale uh and you can enrich that data right uh like I mentioned something like being able to have bot detection or uh run analysis to determine wash trades right start to get to that real data that might get away from this banter that we have on Twitter these days about oh hey this is how many dius I have well like are those actually users or are they just addresses that are being farmed so um you know let's say we want to be able to help Chad report on his taxes with cost basis info that requires offchain data of dollars right let's say we want to show Chad the Twitter handle for the owner of an nft he wants to buy well again we need to marry in offchain data uh chat Chads Roi over time right or chart Chads Roi over time these are all really important use cases that any you know user wants to be able to have solved so building all of this of course comes with its own challenges um again there's an indexing toay you have to go and you have to scrape the whole blockchain again that could be Millions hundreds of millions to billions of API calls um and an advantage here though as opposed to subg griffs if subg griffs you usually have to redo your whole index if you have an issue uh with this you might be able to identify hey I only need to reindex a subset of the data um reorg handling who here knows what a reorg is okay real quick reorg means that historic data on the blockchain chain had changes means what a surprise blockchain data actually isn't immutable to start um this actually is a bigger problem Than People realize they only find it when they start to build something like this uh for example ethereum has over 30 reorgs per day uh so it's not something that's a edge case you have to solve for this uh polygon will have reorgs that go back up to 10 20 30 blocks that you might have to go and reindex that becomes a huge challenge for people building these systems and obviously building all of this comes at a cost you have your engineers that need to try to build out an indexing system scraping the data from the chain identifying okay well uh I want to be able to do this as quickly as possible I only have a rate limit up to xrps how am I going to maximize that build in my retries for erors uh handle my reorgs it it's a lot and then obviously there's the maintenance and the infrastructure to run that so maybe the key qualifiers if a custom index might be right for you is you want to do any kind of advanced differentiated experiences for your customers you want to be multi-chain you want to marry an offchain data and you want to be able to have data enrichment pipelines things like wash trade detection or bot detection but what if there was an easier way so today uh we launched our streams product where instead of you having to pull each transaction each piece of data off the blockchain Via an API we actually feed it directly to you uh we take the raw data and we feed it to whatever data store or uh messaging service you might be using you can see some examples right here for the people on the other side we have examples like web hooks S3 Kafka uh postgress snowflake Etc what is it that you're using we want to make it as easy as possible and available to you um and with that you know if really solves some of the core challenges associated with extracting the data you know the reorg handling that is a huge problem for people we provide tools to do that for you we'll restream the data if you want it to overwrite within your system you can do that as well you can set up a trail delay so that it doesn't end up uh running into the reorg at all Etc early customers of this product have seen a significantly faster backfill versus RPC uh especially when you can hook directly into your existing system and you don't have to worry about some of the latencies associated with apis that's where they're seeing the most impact so while streams might significantly reduce your data extraction infra and Engineering overhead there's still obviously some limitations to it um one is that we only give the raw data we're not processing the data and this is actually a purposeful design decision on our part and that's because we want to get it to you as quickly as possible we have customers that say hey I want to be the one processing it I have my own pipeline set up I'm providing differentiated experiences through this so we want to just get them the uh standard onchain data also still to provide these experiences you need to build and maintain infrastructure for the in-house transformation or do you also being announced today is functions functions is akin to AWS Lambda razor hand if you know what AWS Lambda is okay great so that allows you to just take arbitrary code run it in the cloud um in the context of streams this really allows you to do things like filter the data let's say you only want to give uh send data Downstream for I don't know erc20 token transfers Um this can significantly reduce the load on your infrastructure as well you can also transform the data right at the edge so the decode the blockchain data or structure the data in a way that you're Des destination might want let's say instead of sending it into your data warehouse you want to be feeding the data directly to twilio to send TW text messages to your users you can do that right within our function configure the request directly how twio expects it and then no longer have to run any infrastructure on your side to make that happen uh and obviously you can also enhance the data with API calls to additional Services right so let's say you want to make a call to a coin gecko API to be able to get the price data or you want to make a call to IP FS to be able to get the nft right we'll you can do that right Within These functions this is all available in your choice of language uh out the box we support JavaScript python rupe go PHP java.net and even Docker images so pretty much anything that you want to be running you'll be able to run at the edge on our infrastructure um and now this means that you've freed up your engineers and infrastructure budget to be able to provide more strategic experience to your end users so what are the challenges that we still have left right obviously you still have to build those functions right you have to figure out how to transform and decode that data um as a head of devil that's actually one of the things I'm most excited about is start to provide a library of functions for people to use so that it's oneclick deploy and it does it the way that you want enter in your code whatever it might be um and you know I'm really interested to hear from you guys if you're looking to build something using these tools how do you want to use it because something like Lambda is infinitely configurable whatever you want to build right I'd love to hear it um also you know we don't necessarily know what other challenges people are going to run into yet because we don't have customers uh using it it's in private beta we're ready right now for people to start up sign up and start giving it a try so to summarize this all in one chart this is my attempt at it uh on the left you'll see something that I call Total costs some of you might be familiar with the concept called total cost of ownership TCO that is your infrastructure GE cost your engineering time as well as the services that you're using again note apis very low cost enhanced apis very low cost and then on the bottom we have your ability to provide any unique experiences associated with blockchain data as you come to the right something like subgraphs you start to be able to do more advanced features and when you bring in a custom Index right that becomes very costly but we're starting to provide you the tools to make that as simple and inexpensive as possible so thank you all for coming to my Chad talk and if you want to follow up and look into any further information here's a QR code for streams as well as functions thank you all for