The Graph 🛠️ Faster subgraphs with substreams - Sahra Karakoç

Today I'll talk about subgraphs powered by substreams—a powered by substreams—a new technology in The Graph created by one of the core teams of the project, StreamingFast. First, a little about me: my name is Sandra. I am a medical university student in Turkey. I am from Turkey. I also work as a developer in GraphOps, which is another core team at The Graph.

Today we will talk about what The Graph is, quickly about bounties and subgraphs, then compare subgraphs to substreams, and also discuss substreams and how subgraphs work on substreams, which is the main topic of this workshop. Regarding The Graph: it is the data layer for Web3. It allows you to index and query blockchain data, and we will soon see in detail how exactly this works. About the prizes. So, if you create a subgraph or use an existing one, you can claim the bounty.

Also, if you use substreams or deploy your subgraph in Subgraph Studio, you get bonus points. To use substreams, pay attention, because I will show you how. So, subgraphs are, in short, the ETL process: extracting, transforming, and loading blockchain data. They consist of three main components main components . The first is subgraph.

yaml, a manifest file, which is a configuration file. Here we see that we are listening to Ethereum—it is visible here, and the subgraphs are tracking contract events. Therefore, we must specify which contracts we are tracking. In my example, there is one contract—it is the GRT contract. GRT is The Graph token.

Next, for each event, we have a handler that transforms the event data into the entities we need in the schema in the subgraph store. Here we see a simple handler. It takes the event data and then populates the entity fields. In the schema, we define the form of the data we want to see in the repository, as I already said. So, substreams are a new technology, and if you're familiar with subgraphs, it might be a bit confusing.

Therefore, we will compare them Therefore, we will compare them . As I said, subgraphs are an ETL process. The entire process: extraction, transformation, and loading. But when it comes to substreams, I write "ET" there. In fact, this is a transformation layer.

But extraction is also built-in if you develop substreams. Extraction is done via Firehose, another technology Firehose, another technology developed by StreamingFast, and transformation is done by substreams. But there is no mandatory boot layer here. So you need to define where you're going to store your data at the end, and we call them " we call them " sinks," er, at Substreams, that's the term we use. And in my case, my " my case, my " sink" will be the subgraph; hence the name "subgraph powered by Substreams", uh, actually, that's the name.

And another difference is that subgraph handlers, functions that transform data, are written in AssemblyScript, but in Substreams we write them in Rust. Also, Substreams are composable, so Substreams are made up of modules, uh, lots of modules that transform and, you know, filter data, and if you have a module or even an entire Substream that's designed and developed, you can reuse it, and other developers can use it too . And Substream...Substreams can be parallelized, which means, let's say you're processing, I don't know, two million blocks; you can process the first million in parallel with the second million, which speeds up the work significantly. That's why they're faster, I mean faster, I mean , that's one of the reasons why they're faster than subgraphs.

Also, so subgraphs. Also, so ...uh, in the case of a subgraph you can only listen to events and maybe sometimes do maybe sometimes do , uh, you know, contract calls, but in Substreams you get the full block data, which is what I like most about Substreams: you have calls, transfers, logs, and even changes to the repository. So everything that happens with the block you can see in your Substreams, also because you get the whole block and the block comes with sequence numbers sequence numbers . So everything that happens has its own order, has an order within the block.

So, you know block. So, you know , what happened before, you know, you know, rank in the block for everything, that's another advantage. Now, when we talk about Substreams instead of subgraphs, we get rid of the mappings where the transformation takes place. We replace them with Substreams and leave subgraph.yaml and the schema, but subgraph.

yaml will look a little different. I'll show you this in a moment, because our subgraph is no longer listening to a block, it's listening to a module with Substreams. So, Substreams, as I said, are made up of many small modules. Here I will show an example, and this is the Substreams module graph. We have a block.

The block is the input for map, er, map transfers. This is a module transfers. This is a module , and this map just, you know, filters the data and feeds it to the next modules, and then the other modules do their own transformation and filtering and pass it further down. Finally, in our example, since this is a subgraph managed by substreams, we have a special module called graph out that will push data to the subgraph store. And I have two more maps, map accounts and map block total supply chain, these are just for demonstration purposes to show you how to use repositories, which are functions of repositories, but we won't be using them for our subgraph.

So, in substreams there are two types of modules, as you can see: maps and storages. Maps, they accept bytes as input and then output bytes, and in the process they pass and filter the data, and these bytes are encoded as protobuff messages. Here you can see the protobuff definitions that you should make if you are a substream developer. You must determine what type of output your handler will have. I defined transfer just to make things easier.

And here you can see that the input data of my module is a block, and the output is the block, and the output is the transfers protobuff I defined . When it comes to repositories, well, there are repositories with state, but it's not the final repository. This should not be confused because in subgraphs when we say "storage" we mean persistent storage, a database, but here we have temporary stateful storage that your modules will use. So anything you put in the repository won't be available for queries or anything like that. You will simply use this in your other modules while your substream is running.

Here we see that this object has a value type and an update policy. The reason is that for a value type, every store has a value type because stores are made up of key-value pairs. So you set it at the beginning and you can't change it. Each storage will have a specific value type. The update policy stems from the fact stems from the fact that, as I said, modules can be run, meaning they can be parallelized during execution.

So, if you don't have an update policy, you won't be able to merge them. Let's say you process the first million blocks and then the second million blocks. Now they need to be combined. This is where the renewal policy comes into play . If I have an addition, here this repository will store GRT balances for users.

So, the first million blocks will have a balance for the user. The second million will have a balance, and we'll add them up because that's how you calculate the balance. So, we have different types of values. You can check this. This is from the streaming fast documentation.

And we also have different update policies, you know, " update policies, you know, " install if not exists." In the case of "set", the last key wins, and in the case of "set if not exist", the first key wins. We have admin, max, and append. OK. Now let's see what a substream looks like .

Let's start with the substreams.yaml file. This is the configuration for sub-streams. Here we have defined, I don't know, a file for importing photos and so on. But the most important part the most important part is the modules.

This is what I showed you before. I have a transfer module, store modules, other store and transfer modules, sorry, map and stores. Here, as I said, you define the value type, update policy, input and output data, everything else. data, everything else. I also showed this in the proto file .

You define the input and output form of your data in deliver. This is where you actually write your modules, your handlers. The first example of a module is map transfers, which we saw here. Here is this first guy. So what it does is it has a GRT token contract.

Again, GRT is the graph token. We simply loop through all the logs coming from the block. As you can see, the input of this map is a block, the entire block. But I'll focus on events because it's easier for beginners. But if you want to see more complex things about block information, there are, for example, store changes.

If you want to know more about them , I can show it to you if you come by our booth. So we loop through the logs and just skip everything that doesn't apply to the GRT token contract, and every time we see a transfer event, we create the protobuf that we defined and output it. Where is this going? We saw that here we define input and output data. This is where the subflow graph is defined.

Here, the output of map transfers is the input for store balances. Store balances is where we track each user's balances with each transfer. We simply add, for example, increase the user's balance if it's to them, and decrease if it's from them. It's very simple. But the reason I have this here is because I want to show how to use repositories in delta mode.

So, the logic of delta So, the logic of delta mode is that mode is that , let's say you're calculating a user's balance, but the user might make 10 transfers in one block, maybe one block, maybe he made a multi- he made a multi- call, maybe he has a bot or something like that, and then the value of his key (the the value of his key (the account address would be the key, and the value would be the balance) would change 10 times in one block. So these changes within a single block are called deltas are called deltas . The entire graph of substreams is executed on each block. This is what happens on every block. So we have block deltas if one repository changes multiple times.

Even if there is only one change, we will get one delta. In the repository, when we receive it as input from another map, this " another map, this " map account" receives the repository balances as input. You can iterate over deltas, and here to show you how we do it: for each delta we create an account protobuff . This is just an example. You would n't actually do this, but we can run this "map account" run this "map account" to see what it does.

I have my own stories. Let's "make an account". We can also run this or use the graphical interface, which looks a little nicer. So, this is the output of the "Map accounts" module. As you can see, there is only one version so far.

By the way, this is currently being broadcast from the block, so there is only one GRT so there is only one GRT transmission. We see this "probab" here, so if there was another one I could switch, but there aren't any . I can also switch between modules. We can also see "map transfers" here. This is because "map transfers" were the input for storage balances, and storage balances were the input for "map account."

So, all the cards are working now. As we can see, there's another one, we can just go here: another activity, another transfer, another transfer, another transfer, and so on. So using the graphical interface is really fun. Let's stop this and move on. We have another mode in which we can use repositories—this is "get mode".

So, with deltas you can see all the changes, and with "get" mode you can access— access— remember ordinals? —you can access a specific ordinal , the value of a key in a specific ordinal specific ordinal . So in this example we get, um, we calculate the total supply in the first repository; It's very, you know, straightforward. In another, um, map, in another module that takes this repository as input, we use get functions. So here I just created, let's say, let's calculate on each block how much the total supply of GRT has changed.

We do this with a special function get first that gives you the value at the beginning of the block, the key ' block, the key ' value at the beginning of the block', and that's the key that we get at the beginning of the block here. And we have get last; we also have get at. If you know the sequence number, let's say you want to know, um, the user's balance on a particular transfer, you'll get the sequence number from the transfer and then query the value here using that; this is another option. Let's not start this. Okay, so far, it's just a substream; we want to actually build a subgraph powered by a substream.

So, we've done all these transformations, and I think you can use this project, it's on GitHub, and I'll share the link with you . You can use this as a template because there are a lot of files that need to be created and, you know, a lot of things that need to be done. I would just clone it and change the necessary things. So from here we actually want this data to be stored in subgraphs. How do we do this in a subgraph?

We have a diagram again, a very simple diagram for today's example. We have an entity grt that contains a general transfer proposal transfer proposal . Maybe I want to list all GRT transfers in my interface and the balances of each account. Now the subgraph yaml will look simpler and will accept graph out as input. As I said, he is no longer interested in the bloc.

No handlers, nothing, and we do this nothing, and we do this , as I said, using a special graph module, we use entity changes. So entity changes changes. So entity changes are again protobuff definitions that simply go and change the storage for each entity changes protobuff each entity changes protobuff that comes from graph out. So, let's say your transfer entity transfer entity is the name of the entity from the subgraph repository, the transfer entity comes in, you specify the ID and sequence number. Then there is the operation; it can be an update, delete, or create, but if the object does not exist, it will be created when using the update operation.

So I always use the update operation, and this change function simply updates the field of your object with the new value; Here's how it works. And, uh, another object change uses deltas. This is simply an update of the account balances on each delta. And this guy updates the total supply on each delta. Um, and the output data will go into the subgraph, like I said.

And in the end we will get a subgraph that is powered by substreams. Now let's run graph out just to see what it outputs. Again , I use GU. We can also run, for example, the regular run command. Let's take a look.

Okay, profile output. We see changes in objects. What is happening in this particular block? This is the current number, the current block number. There was a transfer, so we see a change in the object of the transfer.

Of course, if there is a change in the transfer object, there will be a change in the account balance object and a change in the GRT total offer object. Since this is running a bit slow over the internet, I have no other activity for this block. So we can stop it. And finally, we will deploy our subgraph in the studio to claim not only rewards, but also receive bonus points. As we do this, let me open Subgraph Studio.

Okay. From here, we connect with our wallet like this. I've already expanded on this here, but when you create an account, connect through a wallet, just let me show you how to create a subgraph. You will enter a name and select a network. It's not really important because you're not going to publish it.

You're just going to deploy it. These are two different things. You come here if you've already created a subgraph and it's working, you already have graph CLI, and you're not doing this because it's for initializing a regular subgraph. You simply skip this initiation. You authorize the studio, then codegen, graph build, and deploy.

Then your subgraph, this is a subgraph that is fed by subflows. This is the example I showed you. I have expanded it, and it can be queried as a regular subgraph. So, to wrap up, we have...where is it?

Okay, it was here. We have a QR code where you can see the prices. Um, there are jobs, resources for hackers, and links to the repository and to these slides. Thank you.