🛠 Indexing Smart Contract Data with The Graph - ETHGlobal San Francisco 2024.

[Music] what's up everybody thank you so much for coming uh my name is Kevin Jones I am a devil engineer working for the graph I work for Edge and node which is one of the core developers um I'm going to be talking to you guys today about why you should use the graph why it's important uh to have some data indexing um talk a little bit about how the graph works and how you can configure a subgraph really quickly we'll try to build one as well using the graph CLI my name is Kevin Jones again based here actually in the San Francisco area uh but you'll see me a lot of events um happy to answer any questions afterwards or if we have time I'll I'll answer questions uh so again basics of like what is the graph and why you need it we're going to show the configuration and the architecture and then do a demo um so first of all why do we need uh data indexing um I think we know that like for uh something like Google or some of these uh larger search engines that are very web too uh it's to categorize and make information easy accessible by searching um same goes for uh crypto um blockchain data is pretty complicated um and the fact is that you really need a way to uh easily access that data rather than using the built-in tools that are on the um the blockchain itself like the RPC um is what I'm referring to and so if we if we talk about blockchains blockchains are incentivized for transactions right you write transactions to the blockchain you pay some gas if you're a good smart contract developer your smart contract adits events right that that lets your frontend know what's going on uh and then these are kind of like cheap logs right we were able to kind of send some data and keep things um aware from a Global decentralized Perspective uh but these can get commonly or or kind of lost on the blockchain if they're not categorized in a way that's easy to access um so we look at like an example simple nft right this is the Crypt buunk contract uh all the crypto Punk contract contract is is a list of the IDS the punks who their current owner is and then where the metadata which is all the offchain data right sits um so this is a very basic use case of kind of using um you know the blockchain as kind of a little bit of a database um and as these o uh owners of the contracts change hands um we need a way to be able to know what's going on and know who owns a punk at a particular time um but there's some problems when it comes to getting data using the standard like interface the RPC interface um the first of all is there's just a lot in one given block there's lots of protocols that are like you know inside of a current a full block um and so it can be very difficult to just look up like you have to know exactly what contracts for each protocol in other words like Unis swap how many contracts are there what are all the addresses keep those in your application and query them and if you're trying to get um lots of data from that particular protocol or multiple protocols it's just it's really hard to do um and we also have this idea of like an initial dap where you have a user interface that you deploy with something like ipfs or R weave uh and you have the uh application sitting on the blockchain and for writing like I said to the blockchain using gas paying for the gas using the um you know whatever consensus algorithm to you know secure your your transaction is great but when it comes to reading from the blockchain technically the RPC is not designed for reading uh it's originally was used as a tool for people to query their node and do some kind of interesting things to see what was going on with the the blocks it's never really been truly um uh you know created to do to solve this kind of data problem we also have this problem with Providence so how do we get like historical data from the blockchain how do you know like what exactle block number that someone sold their Punk to another owner or transferred it to an exchange or whatever it is uh there's no way to get that information right uh so what we really want is something like the time machine where we can go back in time and see historically does anyone remember time machine show of hands did anyone ever use it I used it a little bit uh but it's kind of cool idea right you go back in time and see what's going on we want to do the same thing with uh the graph we also end up with like something like this where someone wanted to see all of the nfts in a wallet which is like you know this is definitely not an ideal way to do that but if if a user only has like two nfts it's great but if they have hundreds of nfts it's going to be making hundreds of RPC calls to the uh node or to the RPC endpoint and that's going to be very inefficient as this is just trying to Loop through and and get all that data um so this is where the graph comes in right so we know the problems the graph uh is over 200 indexers across the world that are indexing data what are called subgraphs Uh there's delegators who provide stake there's curators who provide uh GRT to to incentivize people to pick up that data and store it and there's over this is a wrong number it's actually closer to like I think 10,000 subgraphs now so there's lots of daps using the protocol in a decentralized way um and there's a large amount of queries that go through the network as well um and so the idea is to take this you know mess of a blockchain and create subgraphs for uh protocols and for for daps and so create these subgraph configurations which are basically databases uh with an API associated with them so show of hands who knows what an API is everybody who doesn't know what apis and want me to explain that basic understanding okay cool um so what we want to do here is do something like this instead instead of doing that for Loop we can make one graph ql query and one HTTP request and get all the same data using the graph right and this is uh kind of the the real beauty of the graph is we everything is built around graphql and if you're a front end developer you know that graphql is perfect for for front end development and especially for hackathons so this is kind of what it looks like user interface you still have you have an indexing protocol uh instead of the graph for reading that data and indexing it and storing it and you still are writing to the blockchain for all of your secure rights uh paying gas and all that good stuff okay let's look at the configuration so uh the the subgraph configuration is essentially three files we will look at these files uh in a little detail but you can think of it like this the Manifest is like the highle configuration file just like any software you need this highle configuration file that keeps track of like what you care about so what network you care about what entities what what how you want to store the data it keeps track of the address it keeps track of like how you want to handle processing data and it kind of references these two other configurations so it references the entities and it references the WM uh Handler okay the entities are how the data is stored like in other words if you create an event the event is emitting a data type and a and a and and some data uh The Entity needs to be like relational to how the data is coming off the blockchain's event so you can think of it like how that data needs to be uh available to your front end with those proper data types like integer unsigned integer string oleon address whatever it is uh so that entity is actually a graphql thing but it's basically we'll look at it but it's basically like a table you can think of it almost like a database table and how you want to store that data and then the third part is like this assembly script so um assembly script is a subset of typescript so it's not quite typescript but it looks like typescript it has Type safety but you can't do like uh Imports of like typescript stuff you can you can only import uh the stuff that we support which is like the graph TS and a couple other things uh but it allows you to create a web assembly um module that processes the data okay and so that will that's what actually stores the data in the database and that is what um is used to listen for the events as they come off of the the blockchain so you can think of it like this the the blockchain uh application that you have the smart contract emits events you El might have like uh an ipfs hash or something that's emitted uh for R weave that tells you how to get the data from R weave for it so let's say it's like an nft so we can do offchain data and then the wasm will process that data as it comes off or it will go out and get the data from ipfs or R weave uh based on the event and then we store that stuff with postgress in the database okay so the hard the the data is actually stored on the indexer inside of a database and then for uh the indexers they also run a copy of what's called graph node so graph node is a rust implementation uh of basically it's what runs the the wasm modules and makes sure that they're running to process the data and it's also what exposes uh the data in a graph key well endpoint so imagine your user or your consumer is going to your application they're making a graph ql request getting that um giving that to the graph node over the API the databas is then responding with appropriate SQL query data and then serving that as a graphql response so it's kind of like middleware that's the best way to think about it so again just review this is like the probably the best diagram I I know we have of explaining it but you have a Dap right you build your dap nextjs whatever react whatever it is Javascript CSS HTML doesn't matter you're still proposing transactions to the blockchain you're still uh using that you know gas that you pay for to change the state the smart contract is emitting events on the right and as they get picked up by graph node uh using the web assembly module the rules that you have defined inside of that module they're stored inside the database and then they're exposed with a graphql API right so then your dap is just able to query that graphql endpoint directly another good use case to for you guys if this is your who's your first hackathon anyone here first one first one oh perfect okay this is a cool package uh it's called scaffold there's a special extension for it all you have to do is add- e subgraph and you will get a copy of nextjs hard hat or Foundry and a subgraph repo that has a subgraph already built for you for the contract that comes with scaffold e so if you're just starting out this comes with everything you need to qualify for the Bounty you just need to make changes to your subgraph and make changes to your smart contract that relate and you can um get started quickly building a d I'm going to use it in my demo but um yeah yeah we we'll go from there there's also a telegram chat everyone who wants help uh we have a booth uh we'll be there for some reason there's no one there should join the chat um there's not a chat Channel yet but we will create that right now uh there should be a general one though so if you have questions you can ask in there anyone need to scan that cool there's also cheat sheets um I don't know if we have them here we probably don't have them here but they're at the booth those are going to be super useful you should come grab one um they're going to help you get started pretty quickly as well okay let's do it so I got n minutes for a demo cool um okay so I am basically going to use I've already deployed an application with scaffold eth I put it on Base main net uh it's a very basic contract we can look at it right here well actually we can look at it in VSS code because it's a little little nicer it's a it's like a hello world contract the only thing you need to know about it is that it displays a greeting this is the contract that comes with scaffold e it displays a greeting called uh public uh building Unstoppable apps and then it emits an event anytime someone changes the greeting so if they come in here and set a new greeting they kind of shout something then it adits the event right it's very basic smart contract nothing crazy so I've already deployed that and uh if we come over here to scaffold e we can see that it's actually deployed to base uh it doesn't have any balance but it has a a string of hello eth Global San Francisco right so we want to index the data for this contract I also verified the contract already so if you click here on uh contract you should verify your contract before you uh deploy it uh or after you deploy it not before you can't uh but after you deploy it you should verify it because um the CLI will reference the ABI um to make sure that it understands uh all of the events that are in there and it will automatically build you a subgraph if you don't verify it you have to pass the ABI in the CLI and I'll show you what that looks like right now so if you're going to get started on the graph and You' let's say your smart contract developer has built the smart contract or you've built it and you want to build a subgraph for it you come here you go to the gra.com you go to products you go to the subgraph studio okay this is where you go if you want to build a subgraph okay um if you want to um query a subgraph that's already out there you can go to the Explorer okay so it kind of depends on what Bounty you want to get there's three bounties there's the best use of a subgraph there's the best new subgraph gra and then there's I think best substreams so there's three different ways that you can um uh qualify I'm not going to talk about substreams today but you more than welcome to come by the booth and I'll explain some some things about it or afterwards uh so really if you're going to get the new subgraph you should be going to the studio so let's do that let's go to the studio you'll sign in so if it's the first time you've ever been here you should sign in um you connect your wallet it's going to ask you to sign a message create an account and you'll get on this thing you'll get a free plan with 100,000 free queries oh some I have 200,000 sweet um and I will say create subgraph I will name it so we'll do it uh eth GSF or something I don't know there we go it will create it here you click it okay you install the CLI so it gives you all the commands here this is kind of like a skeleton like a barebone skeleton for you to start building so you come over here you can install this CLI with npm or yarn so let's just say we do yarn I've actually already done that so let's uh skip that then you do a graph and knit so we come over here we do graph a knit that's going to spin up a new subgraph on my local machine so I'm in my terminal right now I'm going to go to ethereum I'm going to call it e GSF e GSF I'm going to go down and choose my network so in this case it's Bas if I can find base where's Basse here it is base and then it wants the contract address so if you've deployed put your contract it's available right you know what it is you're verified go here to the contract grab the contract paste it in it's going to try to fetch the ABI um it might have issues on certain chains so you might have to provide the ABI so if you want to do it manually you say no um it's also going to try to fetch the start block if it can't find the start block you might need to add it manually I'll show you how to get that so we'll say no so where is the ABI well uh I have it in my project here so it's inside of the artifacts I believe um no sorry it's in uh deployments here it is so here's the ABI so I have that you'll need that if you are on a chain that isn't pulling it automatically ethereum main net it should or like sapoia it should pick it up so you should be fine so I'm going to copy the path paste it in the start block is available on the Block Explorer so you can see here this was the transaction of when this contract was deployed we can click it and then we can get the start the time that it was deployed so this was the block so I can grab that paste it in name contract index of entities is is true yes you always want to say yes because that will build out the subgraph for you four minutes let's do this this last part's going to be a little bit of a speedrun uh because I'm going to have to fly through the explanation but okay add another contract no I'm going to go into that folders so eth GSF and I can take a look at the code let's look at it real quick so I mentioned earlier there's a subgraph yaml which is like the highle configuration file it automatically generates this as long as you provide the ABI and as long as you um you know if you verify your contract it might do it automatically otherwise give it the ABI this tells you what network we're on so we're on base we have the address we have the start Block in here we have the entities are automatically created based on your events we have the Handler which processes the data right those three things I told you about so if we go to schema we only have one one entity and it keeps track of the ID which is like the the actual ID of the unique identifier which is generated it has the who sent the greeting what the greeting was was there a premium paid was there some value in the transaction what was the block number the time stamp the transaction hash and all the data types that we know we need to use use based on the ABI and then inside the source folder is the contract TS and this is where the Handler is processing the data so we we import the schema we import the contract and the event and then when an event comes you know comes through creates The Entity using the new function here creates what the ID should be the hash uh and converts it to i32 and then we set all the values as they come off of the block so the event par parameter for the greeting for the new greeting sorry the greeting Setter the new greeting what the premium was or did they pay a premium what was the value in the transaction what was the block number the time stamp in the transaction hash okay so let's deploy it so I'm going to do it real quick subgraph studio uh o did I already do that no I do that let me o I have to authenticate then I can uh build so this is going to generate the the types for me so that I can interact with the contract and it's going to build the WM module and then all I need to do is deploy so it gives you all the commands you literally just have to copy paste copy paste so version this is the last part it's ask you what version you want let's do version one now it's going to take the subgraph deploy it to the the network uh it's going to deploy it to the studio I should say uh and then it's going to give you the API endpoint but we can also grab it just from the studio up here so now it's synced 100% we can go to playground and we can start getting the data and there it is so I only have one event and it's hello eth Global San Francisco I didn't pay any value and I was the one who set the transaction cool so you get you get the idea um it's pretty easy to get started with the graph you just need to come onto the graph Studio create a subgraph follow those commands and it will build the subgraph for you we obviously want you to do more than that like you should probably do some interesting things in the subgraph there's tons of features and functionality um but yeah hopefully that was suful I have time for one question who wants it not not one question yeah so when I deployed this one the question was is where does the uh the subgraph live in this case I'm deploying it to the studio which is a a a special setup that that we run to run like a development environment uh when you eventually publish it to the network it goes onto the decentralized network but here it's on a special kind of like development instance but it's not on your local machine um if you use the subgraph extension of scaffold eth that will give you a Docker container that you can run locally and that's that's an option cool I'm going to step outside and answer any more questions you guys have but thank you so much appreciate it [Music]