Lightning Fast Multichain Indexing - JonJon Clark

[Music] Now, and for this one, I'd like to bring on John John to tell you all about indexing onchain data lightning fast. So, please give a big round of applause.

Thanks, Cartik. Um, hey everyone. Really nice to meet you all here. My name is John John. I am one of the co-founders of Envo and I'm here today to chat to you about lightningast multi-chain indexing.

Right? Who knows what indexing is? Maybe many of you do, but I'm sure there's some of you who don't. So, I'm going to give you a little primer so you also understand what actually is indexing in the context of blockchains. You know, why do we need it?

Is this thing even important? And chances are it's actually been powering many of the things you've used recently, right? So there it is. What on earth actually is indexing? So you often want to find things fast.

Has anyone here used an encyclopedia back in the day? Yeah, some people have used them. Before information was just quick and you can Google it and now chat GPT it. Back in the day, you were doing a project on dinosaurs or whales and you wanted to go get more information. You would open this encyclopedia and as a young naive kid, you would go page by page and it would take you forever to find the section you cared about.

But when you leveled up, you realized, you know, actually there's this page at the back of the book that tells you exactly where everything is. And that brought you directly to the information you wanted to find. Right? And that's essentially what indexing is. It helps you find things much much faster.

And in the context of blockchains, this is really important because we're not just dealing with encyclopedias, right? We dealing with thousands, millions of blocks, tons of transactions, tons of logs. And as we've been talking about, it's actually a shared ledger, right? It's not just your information on the blockchain. Your information is interspersed with millions of other memecoin mints and stable coin transfers.

And very likely, you actually only care about a subset of that information. That's what you want to show. That's what you want to understand. And that's what you want to broadcast to your users. Right?

So, let me crystallize this with a a little example for you. You land on on a website. You've been doing some borrowing and lending of various assets and you you click on your username and you expect that instantly information will show all your past transaction history, everything that's going on, right? It's something we all take for granted, this snappy developer experience, but that moment you click, it's too late then to all of a sudden be like, well, now I'm going to search the entire blockchain and I'm going to find the relevant information, right? You need something that's constantly getting this information and allowing you to really then show it to your users, right?

And that's exactly what the index is. that kind of sits in between this beautiful blockchain and these applications and it's this thing that listens to these blocks, these logs, these transactions, things you deem relevant for your protocol. It saves them and then it makes them available really quick so that you can provide this really nice rich experience. Okay, so that's a little primer. We now kind of all know what exactly indexing is at its most basic form.

Again, here's another example. Sablier, you want to see your stream, how far it is. Again, it's all this onchain information. You want to be able to show it. Now, we get to a problem where we in this increasingly more multi-chain world, right?

More chains, more problems. Maybe, I don't know. It does make things really tough, right? You're not just looking for a needle in one hay stack. Now you now have tons of hay stacks.

You don't just have to find them there, but you also got to find them there and there and there and there. And it's it's a hard problem for those of you who are engineers and have had to do this stuff before with blockchains. It's difficult now getting all the information from this chain and that chain and that chain and making sure everything doesn't break. It's tough, right? And again, you can see this beautiful picture of all these chains driving home that concept.

Anyways, in response to these issues where multi-chain indexing is really difficult as well as the speed of looking through those hundreds of millions of blocks for the information you care about, we've spent a significant amount of time, love, and energy building this developer framework that you can visit right now on GitHub called Hyperindex. and definitely give us a star. That would be massively appreciated. You can see in the bottom right corner there's a couple familiar faces used by uh Nosis there. I think that's a a Paul Berg emoji and a few others.

You'd be in good company. But this framework is allowing developers to do this process that was generally very hard and cumbersome and turn it into something that's a lot more friendly and and developer orientated. So, here's a little GIF of it in action. And what you can see is when you configure things and you get it working, it's kind of like you have this hot knife just slicing through butter and you absolutely motor through these chains, scanning millions of blocks at a time, getting all the information that you really care about. And that's the experience that we want to bring to you, right?

Um, so I think one of the best ways to always see this is obviously looking at some examples. What kind of things can the indexer help you do? What does it power? So, I just wanted to share a few quick cool examples of that. And this first one is for example showing real time activity of unis swap v4 across multiple chains and what the swaps look like.

So, it's it's a very busy couple of smart contracts as you probably know. And you can see on that that little left column over there um tons of different swaps are happening between various assets and you can bet your bottom dollar if you try to just you know hit the RPC and get this information on the fly it would be an extremely difficult task. So it's it's really integral in building these these rich kinds of applications. Another one, I mean, how many of us here have wondered, you know, what the price of ETH on one chain compared to another chain might be, right? You have all these different swap pools and, you know, it's something I was curious about, too.

So, built this little realtime analytics that actually shows across various different chains um how it trades as well as the volume. So, all kinds of things like this you actually relying on an indexer to actually make happen. And finally, one more one is oracles. I think we all know we need to understand them better. Oracles are often a bit of a blackbox to developers how they actually work.

And in this case, we just show some real time updates of how the oracles are actually posting data to chain. And this is actually on Mega E. You can see it's happening pretty quickly. So, a few examples aside, back to a few reasons that we think hyperindex is really great. And of course, we'd love you to try it.

It is fast. It's multi-chain native. It's got a really nice local developer experience, which all of us devs will know is super important. And it's got some pretty robust hosting with some good GitOps, which I'll tell you a little bit more about, but just to talk a few more bits on those points. Uh the fast, I won't go into it too much, but you can check this GitHub repo.

It's actually not a a benchmark we performed ourselves, but we really do uh do things really really fast. And in the context of development, you know, when you need to scan a chain and make some changes and do a backful, it's it's a really really difficult task and this really helps to speed that up. So, it's really loved by developers. Um again, you can see the multi-chain aspect over here. It's not just one chain at a time, but you can index all your data into a single unified API.

And this is really coveted by a lot of developers right now like for example the Sabia team we work with because they deployed you know across many many different chains and having to maintain 30 different indexes that's just a big big ask. So it's really loved by them. And you can also actually see at the bottom local host. That means you can go click it out and obviously index all the data locally which as us devs know it's always a great experience. Finally, you can see we have this nice robust hosted service powering hundreds of different applications.

So definitely there's a nice development tier that's free. Go and make sure to to check it out. You'll be in good company. And if you love that slick Velike experience where you kind of just link it to GitHub and when you merge your PRs things just deploy and work and you can promote between versions. It's really a great experience and it's something that we we emulate here.

Um and yeah, you should definitely go check that out. Um, I won't go over this because I don't think there'll be time, but there's also some nice alerting and and other cool features that you can really go check out or um, honestly, if you find any of us in these caps, chat to us afterwards about it, too. So, I'm going to talk a little bit about why Hyper Index is fast. And just so you remember, you know, this thing needs to be fast cuz when you have like 260 million blocks on arbitum and you just care about a subset of that information to go and find it again, it's like a needle in a hay stack and it's like having one person looking through the hay stack versus like some machine that sucks hay and finds the needle quick. You know, you're going to choose that machine.

It's going to help you find your stuff a lot quicker. So that's why it's important. it it's a better experience for everyone when you're trying to find that needle. So, we have this uh multi-layer data approach and again there's no silver bullet, right? It's a lot of blood, sweat, tears, and hard work everywhere to make something good and fast.

But one of the really nice points is this multi-layered data approach where we're not just using the RPC which is uh great and and it works well but we're also using hypers sync which I'll get to in a second which really helps also make things nice and fast and smooth. Okay, so getting a little bit on to hypersync. We've talked about hyper index which is the indexer batteries included everything. I've kind of shown you what it does. And then one layer below it, we have something called hypersync, which you can kind of think of as say a data node is maybe sometimes how I like to explain it.

Or imagine G, right? But instead of just G, it was built for only one purpose, just serving data. And you know if you build something to do just one job you can make different tradeoffs and you can make it really good at doing that one thing and that was what we wanted to do really really good at serving data and really flexible in how it serves data so that for cases like indexing etc. it can do it really fast which people like okay you can get this data in a variety of different ways. I I won't go into the the clients, but I'll rather again show you a few examples of how you could use hypersync and what kind of applications that might power.

Right? So the first one is called chain density and and I guess many of you have probably wondered like regarding a specific address, how how active has this been over time? You know, are there more transactions, more events? It's like how popular is a smart contract? And it's actually not an easy question to answer.

Normally, if you take an address like a a unis swap router and you want to see how many events has it emitted over time, you would be stuck with millions and millions of RPC calls trying to actually figure this information out. But if you go to chain density.xyz, XYZ. In just a couple of seconds, you can paste an address and on more than 70 different chains, you kind of get this nice graph showing you how active addresses are over time. And this helps you understand your indexing efforts.

It maybe helps you understand uh yeah patterns, anything else interesting like that. Another cool one is called Hypertui. It runs in your terminal. you you paste your address and you see all cool kinds of things like all your transactions, all the information on it. Again, just running in your terminal.

Um, another one I'm going to go and show you is actually called uh log 2. And you can run this command right now in your terminal if anyone has their computer out npx log 2. And and what you're basically doing here is scanning through an entire chain. This is this is real time. Uh, believe it or not, this is uni chain.

and we're looking at all the ERC20 transfers. Show me every single one. Let's like just figure out how many transfers there's been on this entire chain. And again, very quickly, what it's doing is scanning through. And you can see at the bottom there, it's at nearly 1 million events per second, which is really, really fast.

And you can figure out, okay, we've scanned the entire chain, 20 million blocks. There we go. That's finished. I think that was maybe about 40 seconds. There's yeah, whatever it was, 20 million ERC20 transfers.

Um, again, here's another one, Scope. It's a really cool block explorer focused on account abstraction. And again, what you can see with it is, uh, they actually using hypersync. They don't even really have a backend. and they're making hypers sync calls on the fly megapost through information to get basically unlimited pages of logs, transactions, any kind of data.

So again, it's this it's this powerful engine that can help a lot of kind of data processes and yeah, these are just a few use cases. Um, okie dokie. On to the next one. Cool. This this one's uh called Snub and a big shout out to Revoke who actually helped inspire this little fun idea.

But this is another tool that runs in your terminal and you paste your address and many of you know if you approve some contract to spend your ERC20 tokens, it's a pretty dangerous thing if that contract ever got hacked. So what this does is it scans multiple chains for that address and it finds any outstanding approvals and redirects you to a website like revoke. So you can remove all of those outstanding approvals again running in your terminal and something that would genuinely take millions and millions of RPC calls and days to do for tons of different chains. But using something like hypersync, you can actually do it really quickly and package it into a nice little uh terminal and yeah really cool that we working with revoke on this and helping them with some of their data too. So that's another nice example.

So yeah, with all those examples in mind, I just want to say thank you very much for having a look. Thank you for spending the time understanding a bit more about indexing and data and really understanding the you know the difficulties and the intricacies that can be involved with data. We've built these frameworks to really help developers overcome a lot of these issues. We want to see more and more developers building applications across all chains and we want that experience to be much much easier. So yeah, we really appreciate you you trying it out, supercharging your app.

And if you have any questions, I've got a cap. There's a few other NVO team members with caps. Come have a chat to us. We also have a very lovely website with all the good things, documents, um, good examples, pictures, check it out. You can also send me a message on Twitter or Telegram.

And yeah, thanks again to the ETH Global team for putting this on. It's been fantastic as always. And I'll see you all at lunch. Thanks everyone. [Music]