Index, Query, Ship - HyperIndex + HyperSync Live I Jonjon Clark

Hey everyone. Yeah, excited to be here today. This is going to be an action-packed workshop. So stay tuned for lots of live code examples, hackathon ideas, all the all the kind of things you want to see. So without further ado, let's get on to it.

If I didn't say already, my name is John John. Um, yeah, I'm one of the co-founders at NVO and I'm excited to be talking to you and showing you what NVO can do. And just a massive reminder, Discord, there's plenty of help. We'll be able to help you all the time. And today we're going to talk about Hyper Index and Hypersync live and show you how you can do that.

And why would you want to do that? Of course, because there are prizes. We are having various prizes. the best use of hyper index of hypersync AI and NVO tooling and live web 3 dashboards. So go check those out.

I won't go too much into detail now. But just super quickly if you've stumbled upon this talk and you may be even wondering uh why indexing, what is indexing? Why is it necessary? I'm just going to give you the very quick TLDDR of why on earth a product like this exists. Now, indexing really just helps you find information faster.

And if you've used an encyclopedia before a long time ago, there's something at the back of the book where it kind of tells you what page number everything is on. And then it helps you basically find the information really quickly in the book. And that saves you potentially hours going through page by page to find the information that you care about. And this is a very relevant thing in terms of blockchains because in blockchains you have millions of blocks and you have information that's scattered throughout these millions of blocks and often you only care about certain bits and pieces of that information. So, you know, indexing is this process that helps you surface those pieces of of information you actually care about so that when you need them, it's quick to retrieve them and show them to a user.

You know, just one example is you have a user comes onto your web page and they expect to see straight away, okay, I've deposited D, I've deposited MATC, I'm borrowing this or that. They can't wait for you at that point to scan through 20 million blocks and find that information. So this is why indexing is really really important. It's that thing that sits in between the the blockchain and often a front-end application so you can surface the information you care about. Well, that's indexing uh at a high level.

And I just want to quickly tell you about multi-chain. More chains, more problems, right? Here we go. Uh so now you have this problem of not just scouring you know one chain for information but but many and it's a problem you often face as a developer. So in response to that one of the first tools we've built which I'll show you is called hyperindex and you can go check it out on GitHub.

You can see a lot of uh people are using it and you can see the the open-source indexes they've built. You can go check out on the latest developments and everything there. So definitely go check that out. And what I want to actually show you is just to start off with some examples of things being built with Hyper Index to kind of frame this talk before we go in and actually build something. So, I got this class A GIF over here where you can just see hyperindexes absolutely steaming through four chains at once and scanning millions of blocks at a time and syncing events which smart contracts emit when certain actions are taken.

And we'll show you what that looks like again now. So, yeah, here we go. And let's look at some of the examples. So, one over here, here's a real time UI. you can find that is essentially uh you know doing swaps for unis swap v4 it's showing them in real time across all the different chains.

So very cool to see that you can also see the latest pools that have been deployed. And by the way if if you have any questions throughout this please be dropping them in the Q&A because I will endeavor to try and answer every single one. And yeah, there'll be some people on the team who who can hopefully even answer while I'm talking. So, please do do that. So, yeah, there's the first one.

Real time swaps. Now, you can see real time liquidity for specific pool. What are the ticks? What does the order book look like? How are people adding and removing liquidity?

What does it look like on chain? Volume fees. Another one. This one is showing the pricing of one particular pool. ETHUSDC, but on multiple different chains and how in real time that price is changing as people trade.

So, there's tons of applications that you can get creative on to really understand what's going on in the world of web 3. And here's another one. Stable coins are interesting, right? This is tracking all stable coin transfers uh USDC, USDT across multiple different chains and showing you in real time the value of what transfers are actually being sent. Another really, really interesting one to check out.

This one is showing oracle prices being posted on chain in real time. Uh here we're looking at CryptoKitties where if you know about CryptoKitties, it was deployed maybe eight or nine years ago and was one of the first I'd say big NFT type projects where you had these onchain kitties with various genomes forming to to breed other kitties. And all of this information is is scattered amongst all these millions of blocks and indexing is allowing you to find that information. So you can have that responsive API to build a really nice front end or a leaderboard or a dashboard, any kind of thing like that, which is often extremely important. This one here shows liquidations.

So it's showing liquidations across tons of different chains and what it looks like. And finally, this one's showing blacklisted stable coins. You're interested in seeing which uh which addresses have been blacklisted from interacting with various stable coins. So, in a nutshell, you know, Hyperindex, it's fast, it's multi-chain native, it's got a local developer first experience and it's got super simple robust g uh robust githops deployments if you know of something like you know, Versel. So, think of that kind of experience.

Anyways, that is a lot of information, but I'm just showing you all the cool things that can be built. And right now, we are going to go ahead and build something quick to show you how easy it can be to simply get started and build something yourself. Maybe it's even an idea for something that you want to build. Just a reminder, we have all of those different prizes for Hyper Index and Hypers Sync. Right now we're talking about hyper index which is all batteries included full indexer really easy to get going and we'll come in a little bit later to talk about hypersync and we'll also do some examples there.

Okay so hyper index actually let me I'm just going to actually quickly see if there's any nice chat things from here. Yep I'll have a look and I'll I'll come on to them in a second. Uh yes these dashboards are listed places. um I'll be able to share them with you and you can check them all out. Uh so that's very cool.

Thanks for that question. Hopefully it will give you some inspiration and hopefully we can build something now that gives you some inspiration. So if I go to this next lovely slide over here, this just shows the performance time and sync results. I won't go into it, but NVO is bestin-class and really fast. Again, you can see some lovely syncing happening there.

Uh here you can see we powering hundreds of different applications. You can go check a lot of the indexes online at nvo.dev and uh check what people are building. And this kind of shows you if you deploy what the setup kind of looks like online. Uh you get all of these different deployments.

Uh you can see the chains that have been synced and there's a playground for you to you know make queries. There's also some logs over there, which is nice. But let's let's do something. Let's look at indexing some flash loans. If you don't know what a flash loan is, it's this crazy concept where you basically can borrow a whole lot of money um and give it back in the same kind of atomic transaction.

So essentially you can borrow it for free because you're borrowing it for basically zero seconds. Um, and a is a protocol that allows you to make flash loans. And I've just listed two addresses here, uh, one on mainet and one on the arbitum network. And if we wanted to, for example, find all of the flash loans and build a cool dashboard, you know, how would we go about doing it? Well, I'm going to show you right now.

I'm going to uh actually have to quickly exit my slides to copy and paste that uh that lovely figure over there. And what I'm going to do is, you can see I've got a completely blank uh terminal or blank folder opened up over here. Very nice. Uh all you need to do to get started is run this nvo init command. Bang.

Just like that. And it asks you for a few simple things. I'm just going to click enter. Most of them are quite straightforward. We're indexing EVM.

Uh we're going to do this contract import. And basically what we're going to do is we're going to select the Athereum network. And then kind of all you need to do is paste the address of this contract that we want to index, which uh went and I got ahead of time. And now if we hit enter, what's going to happen is it's going to show us all of the events available in this this contract. And you can see one of them here is flash loans.

So that's actually the only one that um I want to index. I'm going to actually remove all these others. Although there is a lot of other interesting data, you can go and select them. So, we're going to be just indexing this this flash loan event here. I'm going to hit enter.

And I'm actually going to choose this option. I want to add a new network for the same contract. I also want to actually index it on arbitrum, right? Why not on on two chains? Is two not better than one?

That would be lovely. So, I'm going to select that. I'm going to go back to my slideshow so you can't peek the rest of my slides early. And now I'm going to paste this address. And I'm going to say I am finished.

And I'm just going to simply say skip for now. I'll add later. And what's happening is our indexer is now being yeah autogenerated, which is really really nice. So now if you if you have a look in this folder, you can see there's actually source, a config.yamel, a schema, bunch of different things.

And um I'll show you what those look like in a second. I'm going to actually open a say a cursor terminal for example. Um so I can show you that in a second. But what I'm going to do is I'm going to run this pnpm nvo dev command. Now the index is already set up.

All I need to do is run pnpm nvdev and what's going to happen is we're going to start indexing and I'll come back to that in a second. But check this out. So it's starting at block zero for both. You can see it's already got through 36 million blocks. I'll just quickly talk you through what's going around while it happens, but essentially it's it's doing this uh indexing.

That's what we call it. Basically searching through all of these chains for all of those flash loan events that we care about. And now we actually have the bedrock for building something cool because we have all of this flash loan data which is very very nice. So, this won't take long to finish at all, thankfully. Um, you know, a big part of developer experience is making sure it's super quick to scan through tons of information and find what you care about.

So, it's a lot of what we optimize for. And to show you how to actually make sense of this information, um, I'm going to just open up this Hura console over here. I'm going to just zoom in a little bit. But what we can do now is we can make queries to this kind of data to get things like flash loans. Right?

Firstly, you can see all of the data uh is over here. You can see the asset, the amount, the ID, uh the premium, the target, you know, all this kind of flash loan information and your cool front end showing the best biggest flash loans might be making some kind of query. uh where you are for example going to order by um amount right sorry not where but order by amount you want maybe the biggest flash loans um so in descending order an amount ID and asset and initiate or whatever it might be you kind of create this GraphQL query and then you hit enter and you get your data back so your kind of cool front end would be able to request information, make these queries, and very quickly build a very nice kind of front end or application. It could be multi-chain. And that's how a lot of those examples that I showed you earlier were very simply built, right?

Um, thankfully, it's really quick to get all of this information. We've already got more than 100,000 flash loans and it will finish really, really quickly. And just to show you how this code actually looks and how to actually now go about changing this code or or making it uh work better, I'm going to bring another piece up over here. And just to show you the structure of what a hyperindex project looks like, you have this config. And to start this config, what it really does is it defines the the networks you want to index.

that's mainet and arbitum. Uh the addresses of those contracts and the data that you want to index which is these flash loan events, right? So it's very simple to add more events, add more contract addresses, um all of those kinds of things. Very very easy. Then what happens is this is kind of your defining like where's my data source?

What am I actually listening for? Then when I actually get that data, what do I do? Then you kind of come to your event handlers and you run this code. So every time I get a flash loan, this is what I'm going to do. And right now, every time I get a flash loan, I'm basically just getting all the data from the flash loan, uh, all these parameters, you know, the asset, the amount, whatever it is, and I'm simply saving it.

Nice and easy, right? Very simple. Finally, you have this uh schema over here. And the schema is basically just telling you what is the the shape of my data look like. So, here's the flash loan.

And this is kind of defining this this flash loan table, what these flash loans look like, uh all the information inside of them. So, you could do um various amendments or changes to this indexer. You'll see we ship with a cursor rules, which makes it nice and easy. Otherwise, you can write some handfreshed code uh yourself, but you could say something like, "Hey, please add a time stamp to the flash loan entity and update handlers. Thanks."

You could do something like that. And thankfully, this should help you to uh do various pieces in your indexer a lot quicker. Can see time stamp over there. will change the handlers too and it will even run things like codegen and make sure that things are compiling, things are looking good for you. Uh all those kinds of lovely things.

You can see what it's done over here. It's added the time stamp which is a thing that we um essentially expose. And then you can run your indexer again and you can go ahead and now have the timestamp information which is essential for you. So, it's all about enriching and building on this and and really making sure that you have the data that you care about. All right.

So, a couple questions. Those flash loans are for this particular user only. No, these flash loans are every single flash loan for every single user. If I'm understanding you correctly, that's what I'm indexing here at least. Um, yeah.

How would I index all the token approvals on any given chain? That's true. Uh so you can go to firstly our docs.nvo.dev and I will show you a feature called wildcard indexing where you don't even have to specify the address of the contract you want to index from.

So for example, you want to index all ERC20 transfers or all approvals. This shows an example of how you can actually do it. uh you specify this wild card equals true parameter. So if this event is emitted regardless of what address it's from, you can go ahead and index it. So yeah, there we there we go.

That's answered for you. Uh Chris, is it necessary to index for certain contracts or can I track all tokens? Yeah, there we go. This is answered again. You can track all tokens.

Uh that's the same. So yeah, you can do that and check out the docs. There's lots of cool examples like for example a unis swap v4 indexer which you can check out in GitHub. There's things on multi-chain indexing. There's block handler indexing where you want to every certain amount of block run some indexing process which is really nice.

It also tells you more about what I just talked about the config the entities uh the the handlers as well as many other lovely lovely features. things like indexing, dynamic contracts. So yeah, you'll find it um all in here, which is really nice. And yeah, again, I showed you how we can go ahead and we can make changes very easily to the code. You can see it's run.

We can rerun the indexing and you can see it's going through and and steaming indexing all the various events. It's indexed, you know, more than 300 million blocks. Uh so yeah, it's a it's a nice quick process to get the indexing going. Okay, so that is essentially hyperindex. Cool.

I see I'm going to answer a couple more questions, but uh yeah, please keep them coming. Any more questions on hyperindex before I get on to hypersync? Uh can you fetch data real time? Absolutely you can. For example, look at this.

This is uh real time fetching data from unis swap v4. All the swaps literally happening in real time. Isn't that great? Absolutely. Um cooking.

You can see the order book of people adding things like that. You can see the TVL's changing the liquidity. Yeah, absolutely. You can fetch stuff real time. What makes NVO so fast?

Uh it's a good question. Won't dive into all the specifics right now but essentially um there's lots of engineering where we've focused on making it fast in terms of a data source to go and fetch the data. We don't rely purely on the RPC which has various bottlenecks. We also have our own fast data service or data node hypersync which I'll get to which is part of the reason it's very fast as well as many optimizations inside hyperindex. Uh why isn't it possible to get get a call data from the RPC?

I think you're referring to Hyper RPC. It's because Hyper RPC is an abstraction over Hypersync. It's just there to get um logs, blocks, and transactions really fast. Yeah. So, thanks for those questions.

And now let's get back to the slideshow over here. And let's start looking at hypersync. So like I was saying, uh if you want to get data, RPC is a one good source. Uh hypersync is kind of our take on what happens if this node was only built to serve data and not do everything else. Then you can make a lot of different tradeoffs to make it really fast and really flexible at at doing that one particular thing, right?

So, Hypersync has a node client, a rust client, and a Python client as well as you can make some curl requests and yeah, you can try out lots of examples with it, which is really, really cool. And I'm going to show you some examples of why you might want to use Hypersync. Hypersync is a little bit more low-level. It's a little bit more flexible. If you want to do massive amounts of large data analysis, it's perfect for it.

If you want to build your own custom experiences, it's perfect for it. So, uh, very much so for that realm of even larger amounts of data, do it yourself, store it yourself, uh, kind of getting into the weeds. For example, I built this tool called chain density with hypersync which you can paste any contract address and you can basically scan an entire network and plot how many events or transactions have happened on this contract address over time which helps kind of give you an idea how busy is this address can you know unlock some some insights which is yeah pretty cool I would Okay. Uh, next there's this thing called Hypertui that was built at a hackathon that again runs in your terminal and you can paste an address and see its entire history because Hypersync's very fast and flexible to to fetch all of that data on the fly. Um, next we have uh what is this?

Let's see. NPX log 2. Yeah, we have log 2 which helps you scan entire chains for logs. So you check rescanning for ERC20 logs. Um you can check we scanning at close to a million events per second and it's checking how many transfer events are there.

This isn't sped up by the way. Um I'm not sure which network this is. I didn't see when I typed it in the terminal here what it was. You can see it's found more than you know 25 million ERC20 transfers. So if you wanted to build something that's checking the number of ERC20 tokens that you can see we're getting all the approvals that's really really nice.

Um okay and yeah that's just a screenshot of what it looks like. This is scope which is a block explorer which again is something um really really nice. And yeah let's see what this example is. H this is of scope. Um yeah, a block explorer that's built largely on top of hypersync making requests on the fly which is really really cool.

So there's lots of custom type applications and things you can build with hypersync. Lots of interesting tools. Um this one is to allow you to see and visualize token approvals and revoke them. So you can put in an address and we scan seven different chains here and find all the approvals for a specific address and basically see all the tokens. Uh so yeah, you can see that's all in the terminal and you can go ahead and play around with this example too which is yeah really really nice.

Um yeah, so this is now the the last example is hypersync query builder and this is how I recommend you actually go ahead and and play around to start off with. It's just in the in the the web UI as a way to play around with hypersync and get to grips. Again, you should be using uh the client if you want to do the client if you want to do anything uh super serious. Let me get to it. Sorry.

Here's the hypersync docs and try the quick start. The quick starts really good. Try this repo. But yeah, you'll use the node Python or Rust clients. There's various examples that you can try run out and you can read lots more about everything.

But if you just want to play a bit around with it, yeah, you can go to builder.hypersync.xyz and there's basically two parts to it. You choose your config, you know which network you want to make a request. There's some advanced options which I won't go over.

And you add filters. What are you filtering for? You select fields. What data do I want? And then bang, you can make a query and get your results.

So extremely flexible and extremely fast. So if we wanted to get every single EOCC20 transfer for an address, I've got this little preset example over here. Um, you can see this is the filters and what it's looking for in terms of topic zero, topic one. This is the data that we've asked for it to return. If I hit execute query, you see that data is going to come back lightning quick.

You can look at it over here or interactive JSON. You can also have a look at the table and see, you know, all the logs, transactions, blocks that have matched our filter. You can download it as a CSV. You can play around this builder.hypers.

hypersync.xyz in itself is a cool hackathon project and it's an it's an extraction abstraction kind of over hypersync. So another cool example of something built right so I don't have massive amounts of time yet but yeah I would like to leave the last couple minutes for some Q&A and again a super quick summary and where to get help. Uh, you can get help on our Discord. You can nvo.

dev is the main link. You can chat to me on Twitter, Johnh Johnh Johnh Johnh Johnh Johnh Johnh Johnh Johnh Johnh John Clark. You can uh answer questions all over. That would be great. Um, and yeah, there there's lots of places for you to get help.

Cool. I see there's some questions. I see uh I see there's a hand up. I don't know if that's like a a live question. We can take one of those or maybe I'll just take the the chat ones.

Let's go ahead and take that live question if that's that's good.

Hey, hey, John. My fault. It came a little bit late. I literally just got the notification for the event. But um the question I have it might have been answered.

If it did, just let me know.

Sure.

But basically, uh looking at the price pool, we did we were looking to use uh Nv tooling project utilizing it within our AI agent. Uh very quickly to summarize, we're building basically a protocol where AI agents could communicate with one another and send payments. Uh so be a reservation agent and a customer agent like you tell Siri to reserve your restaurant appointment and then your agent will contact other restaurant agents accordingly and book that reservation uh with some kind of payment process. uh uh uh can you please just maybe uh guide me to how we can use the NVO tooling toolkit to kind of um help uh help that happen in a smooth process and possibly even like crosschain if there's like different agents on different chains.

Sure. Sure. Yeah. I mean I I won't go into too much detail now. dropped it there.

But basically, if there's an onchain component and someone makes a payment, something like that, you need to know when that payment is made, have that information so that you can then react and your app or your front end can show payment received thing booked. Thank you. So, you need some process or way to be able to show that and index is often the thing that sits between yeah, that payment uh process and and the actual app. And yeah, everyone here you can you can send a couple um ideas. You can DM them through.

That's uh great. Uh yeah. Um you can consume data from your indexer in real time, Chris. You just make queries to that essentially that GraphQL endpoint of your indexer. It's live.

Um cool. Yeah, I think that is mostly it for questions and we are bang on time. Okay, let's We got one more over here. James, go for it.

Hey, can you hear me?

Yeah. Yeah, got you.

So, I was just trying to understand in the example you're using what the data source is. Is it like a node running hypersync that NVO manages and that's where the speed advantage comes in.

Yeah, thanks James. Yeah, that was a that was hypersync um enabled syncing. So, that's where it's really fast. There's support for more than 70 different networks and that's the the default option on on indexing but there's also RPC as a data source too so you can mix and match you can use both uh for resilience you can yeah switch between the two but that was yeah hypersync based syncing

so is it like a data provider involved at all um I'm thinking of like infuro or other kind of services like that

it's the essentially um us it's it's NVO so it's our hypersync is like a a service like I showed you with that builder where you can make requests to it and get results returned. It's yeah it's as simple as that. It's just a a faster more flexible pattern to access that data you care about than perhaps only using RPC requests.

Okay, I I think I understand. So you could like run your own node with hypersync software and consume that as a data provider or is that

So we we run that. So we run that service and all you do is just consume it. So yeah. Gotcha. Thank you.