Blockchain Indexer for Highthroughput Blockchains | Christoph Niemann - AWS

Hello everyone. I'm Kristof. I'm from AWS in the web3 team. And what I'm going to talk about today is indexing blockchain data. You all know of course that blockchain has a lot of data on it.

It's stored immutably there, can never be deleted. But it's actually quite hard to access. So the question is how can we use that data in our applications and then build on top of that? And that's what I'm going to talk about. Specifically building your own indexing solutions.

So let me just start out with this very simple architecture that you probably know of. Um you have a front end on the very left side. You have the blockchains on the very right. Then on the lower path you have this transaction management actually writing data to blockchain. Your users can use the front end to do stuff that creates a blockchain transaction that will be sent off to blockchain.

Of course you have some wallet interaction in there. I just put MetaMask there or you can use AWS services for it to sign your transactions and then they'll make it into blockchain. There is this other path on the top, querying data. Which is the one that we are concerned with. Um of course your front end or whatever application you're building always needs the data.

It needs to access uh the current blockchain data, the current state of things. So just imagine you're building a DeFi application. Of course you want to know the current prices on the very DEXes that the tokens have. And there are various ways to get that information. Of course the very simplest thing that you can do is just query the smart contract directly.

And that's what you usually start out with. It's simple enough. You can just ask a smart contract what is the current state of the state of things, and then of course work with that data. However, you'll soon find out blockchains are of course not really meant for querying data. They're really good at storing data, and they're really good at securing that stored data.

So, we have all these proofs that data that sits on blockchain has never been tampered with, because we have of course the entire all the guarantees that blockchains gives, which is really good, but it's fairly hard to get the data. So, if you're building an application that just needs the current state of things, that's easy enough. You can query a smart contract for its current state. However, once you want to do something like historical data, give me the prices half a year ago, it's of course the data is on blockchain, but it's almost impossible to query directly. Just because the way the blockchain store data, it becomes fairly hard to do.

So, we need something something different, and that's what we're going to talk about. Just this query path, we want to optimize for that. We're not concerned about the transaction management down there, just about the querying things. Now, there are really three solutions. The first one I've already hinted at.

You can of course just query the smart contract directly. That's the easiest thing, and that's just what you can do immediately off off the bat, if you will. And then we'll come to the more advanced solutions, either using existing indexes or indeed building your own indexer for the blockchain data. Just talking about the very very first one, query smart contract directly, very easy to implement. Here on the right side, you can see a token that wants to do an airdrop for based on the on the ownership of another token.

So, if you do have the BAYC, I think it was, Yeah, BAYC token. Then you can claim this other token that we've developed here. And of course, uh it's a fairly simple line in this in solidity where you can query another smart contract and say, "Does this uh particular um address have a token of a BAYC token here?" It's simple to do, but again, it only really is useful for the current state of things. If you do want to do something like historical data or range queries or anything more sophisticated than just querying this single bit of data, then it's really out of out of the window.

You can't really use that. So, we need something else for that. That's what we're going to talk about uh now. Using existing indexes. There are really two flavors to this.

One is of course, there are very many companies that actually have indexes that you can just use. And we at AWS are one of them. We actually do have a managed service that allows you to query uh data for the Ethereum and Bitcoin blockchain. And if that is what you need, um to say token ownership on either Bitcoin or Ethereum, then of course, the probably your very best bet is to use an existing indexer because they have gone through all the heavy lifting. They have done the hard work for you.

And normally, they just provide you some sort of API that you can just query and have all the historical data and and whatever they integrate. And there are a couple of them. So, recommendation first, look if there are existing indexes. If there aren't, and there are very many cases where there aren't existing indexes, and in particular one of them is if you have custom events. So, if you're only interested in something like token ownership, that's what most of the existing indexes support um right out of the right there.

So, you can almost always query them. But, of course, there are smart contracts that have the custom events that are specific for these smart contracts. And most of the existing general-purpose indexes, of course, don't cover them. They, of course, will see those events, but they won't decode them, so you don't have the data available. So, that is one of the major use cases where you probably need to look somewhere else.

The other uh second major use case is, of course, if your chain isn't supported. Um not all indexes support all chains, and in particular, if you're looking for high-throughput chains that have very uh very small block times, something like uh say base or so, 2-second block times, that's really a lot of data. Most of the existing indexes won't cover that. And if you're looking at something like Solana, it's it's even worse. Um even fewer indexes cover that in a real-time fashion.

There are many that cover the historical side of things, but not really real-time data. So, using existing indexes is really good if you're in this standard event space, if you will. If you're looking for custom events, do something different, build your own indexer based on an existing framework. This is what the uh how the architecture will change then, of course. Instead of the front end on the left side calling our blockchain directly, instead we put this blockchain indexer in between.

So, the blockchain indexer does the heavy lifting for us. It queries blockchain the blockchain for whatever we want the indexer to query, and then provides this data in some sort of aggregated form, in some sort of um custom schema for our front end, so the front end can do really simple queries only. It doesn't need to do all these complicated queries or aggregate different API calls or anything like it because a blockchain indexer does that for us. Excuse me. One of the major, I'd say, frameworks for that is the graph, of course.

It's an open-source framework. They have their own um own dedicated graph network that you can use to deploy your your sub graphs to. But you if you want to, you can also spin up your own graph node. This is what it would look like on AWS, and we do actually have that QR code on the lower left there. We do have a CDK script that does it for you.

So, if you ever need your own particular graph node, you can just use that, and it's more or less a three-line deployment or so, which would spin up this this um entire architecture for you. An ECS, uh elastic container service, so a container management solution with various Docker containers, one the graph node, one an IPFS node. And then, of course, a storage, RDS or data database, and then EFS for for the metadata and for backing up the IPFS node. The whole thing is then fronted by API gateway, so you have a simple enough um interface to do the GraphQL queries that the graph node supports. If you're using the graph, the really nice thing about it is you really have control over everything you can possibly think of.

For one, you run the node on your own, so you define which blockchain you want to index, and then you also define everything that the this particular node needs to index. You define the schema, you define every single event that it tracks. You can even do smart contract calls if you want to and then all put it into your own custom data schema. I've just drawn it on the left the these little boxes there. Just a simple example where you would have a contract and then that contract has a token and then the token has owners which are really of course blockchain accounts.

So, this particular sub graph would then map or well, really subscribe to the blockchain for this particular smart contract and map all the transfer events that that come in into our custom data schema. And of course, if you have custom events that you want to track because you're building, I don't know, a DEX or so, something like that, then you would just integrate that. Um the you define the data schema, you map write the mapping itself. So, the mapping from the events into your data schema is entirely under your control and then you just let it do its thing, really spin it up. It index indexes the blockchain from whatever block you want to start off with, even historical blocks of course, and then provides that data in a nice easy to query format, namely GraphQL.

So, it's fairly nice, but if you've ever Who has ever worked with a graph? Just show of hands. Some have. You probably hated it. It's really, really slow.

It's a very steep learning curve, very hard to get right. So, it is there, it's it's really useful if you get the hang of of it, but it does have some um disadvantages. One of them, it's it is fairly slow. And that is because what it does is the graph node, if you just go back to the architecture here, what the graph node does, it queries the blockchain for new data and of course it uses the normal RPC calls that we have. And then of course, it needs to do all the mapping.

You need to get the mapping right, which is complicated in and of itself. So, many things could can go wrong with it. So, we want to do something um some Yeah, something better, if you will. And that then would be, all right, we're not using an existing framework. We're just building the thing on our own.

If I had had this talk a year ago, this would have been my very first solution, and I would have highly discouraged to do that. And one of the things why this is really complicated is of course because blockchains have this thing called reorgs. So, it can happen that a currently valid block block in the blockchain, on the canonical blockchain, does become invalid because we have forks down the road and something like that. We have reorganizations of the blocks. And of course, your indexer needs to be aware of that.

So, there are many pitfalls that you can fall into, but very luckily, we have some really nice uh solutions or at least approaches nowadays that make this thing much easier. So, what are the current problems with our existing uh solutions? One of course, I've hinted at that, the indexer currently polls our blockchain by our RPC. So, it continuously polls data, calls the RPC for new blocks, for new transactions, new events, and so on, which makes it really, really slow, to be honest. If you go about, there is a sort of the canonical, if you will, Uniswap subgraph.

If you try to deploy that on the graph, it takes about 3 months or so to actually index the the entire Uniswap uh smart contract because it's just a lot of data and it needs to do millions and millions of RPC calls to get all the data into the database. So, that is one of the problems that we want to uh tackle. The other one is we have a fairly late transformation. The node processes all the data. It then sends it or gets pulled by the indexer for the data and then only the node indexer itself transforms the data into whatever you wanted to have.

That is fairly late in in the process. There could be an interesting idea of moving that closer to the node. And then the final one, transformation definition at compile time. With the graph you define the structure that you want to have, you define the mapping, then you deploy the subgraph, it starts indexing. If you see that there was some some kind of mistake in your mapping, you'll fix that mistake, you'll start from scratch.

You need to re-index from scratch again and again if it's a large smart contract, that's going to take very very long. So, we can be thinking about putting that transformation layer somewhere closer to our uh to our actual indexer so it doesn't happen at compile time but rather at run time. Now, going through each one of these, indexer pulls via RPC. What you see here is the uh existing uh existing architecture. Front end pulls the indexer, blockchain indexer uh queries via RPC the blockchain nodes.

What we'd rather have is instead the blockchain nodes, when they are processing new blocks, transactions and all that, instead push the data into our indexer. So, that would of course make it much simpler for the indexer to actually be aware of new data because it gets them continuously via the node. And then we can the front end can still pull the data from the blockchain indexer or it could be some sort of push mechanism there, too. But the main part that we are concerned about now is really the blockchain nodes pushing data into our indexer. Late transformation.

Currently, we the nodes process the processes the blocks that's the upper uh blue blue box there. Then pushes data or gets the data over to the indexer, and then only the indexer starts transforming the data. This is highly recommended blog post. It's not by me, it's by um Andrew Hong. But really highly recommended.

That's the QR code down there. That's where this whole um whole diagram comes from. And he makes the point saying, "There is this tendency of pushing transformation closer to the actual execution to the node itself." Which of course makes it probably much quicker and probably simpler to do because the node is well, it's processing the transactions anyways. And then finally, transformation definition currently happening at compile time, so we need to re-index every single time.

We'd rather want to put that into the our runtime. So, while the indexer's running, we can push a new definition into it, and it would immediately pick it up and start indexing on that. So, what does that look like? And this is if you want to take home something, this is the slide to take home because this is really all the um all the approaches combined. For one, the first thing is blockchain nodes have done considerable advances in during the last year or two or so.

And in particular the Reth node came out. And the Reth node does one thing that is really, really useful. It gives you the ability to define so-called execution extensions that are directly compiled into the node itself. So, what an execution extension does, it it gets a block whenever it's processed by the node, and then it's up to you what to do with that data. And what we can do here, of course, is use the data the new block data and push it out to our our sync to our indexer in this case to work on that.

So, that's what the Reth why Reth is in here the Go Ethereum Geth client has something similar called live tracing, if I remember correctly. I haven't looked at that much. I've only looked at the execution extensions of Reth, which are really useful. Only caveat that they're written in Rust, so you need to know Rust to use them. And so, now the Reth node pushes data into our indexer, and we restructure the indexer itself.

And what we do here now is we're pushing the data into a Kafka queue, and then use that Kafka queue as a source for data for everything else. And of course, Kafka has this really nice ability that we're pushing data into it, and we have we can have various consumers independently use that data. So, I've just only drawn two different possibilities here. One is um our Kafka Kafka service has the ability of directly pushing data into S3. Uh and S3 currently supports Iceberg um uh Apache Iceberg.

So, what we can do is have Reth push the data into Kafka, Kafka push it into S3 Apache Iceberg, and we have a continuously updated Apache Iceberg table in S3 which we then can use. Or of course, we can use the streaming Kafka to actually push the data somewhere else. A database here, I just put Dynamo here. But we could have whatever we want to and we could also use some kind of processing engine, something like Apache Flink, that takes the data and then transforms it and then push it in somewhere else. So now what we have done is really we've killed three birds with one stone.

One is no more polling. Really our node pushes data into our indexer. Exactly what we want. We don't have to do this whole RPC business. It's just well not needed.

Second well, second one no RPC calls. I put it again because it's so important. It makes it so much quicker. And then of course we have an optional early transformation. The rest node itself can start transforming data and then of course we can also transform data after it's been stored in our Kafka queue.

So we have really these two choices where to put it and the idea is then of course have the rest node do the basic transformation, push it into Kafka and then the application dependent individual transformation on say Apache Flink or so that then pushes into your data stores. Yeah. It's the same. So that was my last slide. I know I've taken the time.

I've only put this QR code. That's again for the self-hosted graph node. I couldn't get the internal approval for the GitHub repository for the for the Kafka based system in time so it'll come out soon enough. If you want to be be of it, just contact me on either X, WebCaster, or LinkedIn, and I'll let you know. Thank you very much.