Indexing the New Economy with The Graph

all right good morning everyone welcome to the workshop on using the graph to index data off of ethereum I guess to start I was going to just do a quick explanation of what the graph is but can I get a show of hands on who knows what the graph does and in general what the goals of the graph are great ok cool so quickly just an explanation of the graph we are a data access layer and tools for you to index the data off of aetherium chains that you are interested in for your applications which is typically for a specific smart contract that you are either created or you want to look at data for and what we're gonna do today is we're gonna go through how to start up a sub graph which is what we call our data sets on the graph and we're gonna initialize that from a contract and deploy it to our hosted service and start that sinking so you guys can see that process and go through it with us today and then build your own sub graphs this weekend or later on if you want to before I get in to graphing it is there anything just as far as the general idea of the graph and our goals that you guys want to ask questions about or talk about before we dive into the specifics like one of the very useful things is just being able to integrate it into like a front-end app so a lot of the one of the first ideas that we came up with was basically people were querying directly to aetherium nodes and it was loading super slow so what the graph does it'll sync from block 0 to 9 million or whatever it's at now index all the data you want and then you have this back-end where you're hitting this like endpoint and you design the schema yourself it's a graph QL schema so if you have the dashboard you have in mind that you want to build you can actually just bill so the schema will come out exactly how your front-end developer wants to grab it and it makes it really simple basically to grab data off with your aim yeah yeah yeah yeah so if you design the schema correctly like it could just be one API endpoint instead of grabbing a few putting them together and then displaying it yeah go up so I'm just gonna open up a terminal here and so you can there's a bunch of different ways to start up a subgraph okay and I'm gonna go through the process of starting it from a contract and what we do there is you enter in the contract and we pull that down from ether scan and we scaffold out a subgraph for you that will run in a very simple way to start and then what you can do from there is build out your schema based on the application usage that you want and so to do that you first install our graph no daman jewel oh yeah is it to smaller okay cool so I've ever you have our graph CLI installed it's pretty easy to install it's a node module and I'm not going to go through that but we'll just start out by running graph in it and it kind of takes you through the process so you just put in the subgraph name which has to be your user name and then the subgraph name that you want it to have what we're going to be doing yeah yeah yeah so this is just a blank directory I haven't said anything up but my intention here is that I'm gonna make a directory inside this with my sub graph in it if I wanted to I could make that elsewhere on the machine but just to make it easy I'm gonna keep it in this directory before I keep going through this I will go over here and get the contract that I'm interested in so what we're going to do today is the gravity contract which is pretty simple so it's easy to understand what it's doing and it is just a tool to have to store Gravatar azan chain so you can build your own avatars post them there and then with a sub graph you'll be able to query the different avatars that people have made list of users stuff like that so I'm gonna go in here and get the contract address go back to my terminal and it's gonna ask me what I want to name my directory I'll just name it gravity it's gonna be on main net and I'm going to paste in my contract address here and it's fetching that ABI from ether scan creating my directory scaffolding out my sub graph and installing all the dependencies with yarn so cool and then we gives you some next steps for how to then deploy your sub graph but before we do that I'm gonna go ahead and open that up and see what we've generated there so I'm gonna go over to my IDE and we have this gravity folder that was just generated and in it we have the made main pieces of a sub graph I'm gonna try to make this bigger for y'all it's probably pretty hard to see I'm not sure how to make that one bigger but I can talk through that a bit so the things that are generated are graphical schema a subgraph manifest a package.json that has your dependencies in it and we also auto generate mappings and what mappings do is they transform your data to the entity structure that you want it to be in in the database in the end and in this schema here this one's really nice because it's pretty simple we just have one entity here with counts and owners and I use this example because it shows one of the potential pitfalls of using graph in it which is in this case the contract has an ID in the contract so our auto-generate created two IDs here which is quite hard for you guys to see on the screen but this is something that might you guys might come across if you guys are initializing your projects so all you have to do here is just remove one of those IDs so you don't have one of them and then I'm gonna go back to the terminal and I'm gonna kind of go through some of the steps that will be next so we will run graph build I'm not in that directory yet so I'm gonna go into our directory gravity and we generate a package that JSON file for you and in that we have some NPM scripts for building your sub graph deploying your sub graph and running code Jen what code Jen does is it takes your schema and your abis and we generate classes for you and then you can use those classes in writing your mappings so we Auto generate the entity types with the fields on them and getters and setters for those and you can then write pretty quickly write nice transformations for your data so I'll show you guys what I mean by that if the ABI isn't on ether scan and you have access to it you can just download the ABI yourself and you will in the subgraph manifest which was auto-generated in this case it has a reference to where the ABI is located under the ABI is portion of the manifest so you'll just put your ABI in the directory somewhere and then reference that and then you'll run code again and it'll build your typescript classes that you can then use in your mappings and the nice thing about that is then you get some nice autocomplete and type checking for you so I'm going to go back to my terminal and I'm going to build our sub graph and you can see I have another error here related to our ID and the error is requires a specific code join again did you fix it no so so this is a type big int our our IDs need to be strings and the data coming from the contract event is a big int so I'm just gonna go in here I find that error and I'm gonna cast that I can convert that to a string before saving it to my entity and that should fix that so I'm gonna run yard and build again so now I need to run yard cogent because I had changed my schema and now that's gonna update all my classes yeah so it's not easy to develop a subgraph if you don't already have some understanding of how the contract works yeah like so in it is pretty basic it it goes over the contract and looks at the events and creates identities based on the event signatures so depending how like the events it's very like basic right it goes over and just creates it most likely if you're developing a complex smart contract you're gonna look at the events yourself and start building them like you're gonna make this schema file yourself basically you're gonna say I want the CDP's and that might combine four different events in the maker contracts and you kind of just map it out yourself put it down there so graphing it is more based for anybody who's getting started with a graph and like their first project and looking at maybe a contract that's simple online and it'll generate it for you but yeah if you're if you're developing a subgraph to be used in production you're you're probably gonna like you probably might not even use graphing it you might just create the schema from scratch yourself okay thanks for the good questions guys yeah so now that I've built the subgraph I can either run a node locally and deploy my sub graph to that node to index all of my data and then build my graphic you'll server or I can deploy it to our hosted service so today I'm going to use our hosted service which I suggest everyone doing because then it's pretty heavy on your own machine so I'm gonna go sign in at the graph and I'm already signed in here and you guys can just sign in with your github accounts and then I'm going to go to my dashboard and I'm gonna create a new sub graph that I'm going to call gravity I'm just gonna really quickly put an X a description in here Gravatar for aetherium and then create my sub graph and once I've created the sub graph through our UI on the hosted service then I can deploy our my data definitions to that so to our house in service so I'm going to go back to my terminal and we have all these scripts already built for you in our package JSON so I can just run yarn deploy actually let's let's go make sure I don't think that we have the name my github username in there before you forward in gravity okay yeah it looks good so I'm gonna deploy that and what it's doing is it's uploading all those files to ipfs and then using the unique IP FS hash sending that to our hosted service which will then pull it down take the data definitions and start start indexing that's so that we have storage to keep all those sub graphs around so we use IP FS is distributed storage to be sure that we're not losing that base data and then because that's basically our data definition and from that you could always rebuild the subgraph data yeah so like long term when it's a decentralized network what it allows for is like the subgraph files that allow anybody running a graph node around the world if they can just grab them from a decentralized data source like ipfs or file coin all you do is grab that file and then it'll start indexing yeah so I got this error here that I have an invalid account name or access token which is because I did not use my access token for deploying so it didn't authenticate so I'm gonna go back here and I have my access token shown on my my dashboard I'm going to copy that over and I'm gonna add that in to the deploy script which is just access token and I'm gonna try that again uploading the ipfs builds and it's deployed so then let's go back to the hosted service here and there we go so we have the gravity subgraph has deployed and it's actually already finished syncing and I believe that's because yeah so that that subgraph has already been deployed to our service and we D dupe using the IP FS hash so if there's an already identical sub graph that's been indexed we won't redo all that work we're just going to reference the data that's over there so that's why you see I already have 73 entities that have been stored and I have this bar this showed I've synced all the way to the latest block 8.9 million and then you can see the the simple schema that I had generated here and we can actually try to query it so I queried for the first five entities and their ID count and owner and so you can use this page to try queries out and explore datasets that we have already on the graph or your own and then the other really important thing about this details page is the logs tab which you can use to see progress of your sub graph syncing and debug by looking at errors there in the mappings you can log out messages or debug statements so that can be really helpful as you guys try to figure out and debug your your sub graphs so that kind of is a really quick walk through of using a really simple sub graph to point into the hosted service initializing it and what we want to do now is go into like a slightly more complex sub graph the eunice sub graph which I think everyone should be familiar with Union swap here so we'll kind of switch over to that and use that to explain how our mappings work and how a more complex schema might look yeah let's do it so how many you guys know you know swap okay yeah other people are here nice nice awesome so we built can you keep stopped yeah so we built a sub graph for unit swap actually so this kind of relates to like we didn't use graphing it here as you can see with this schema it's very calm I got I'll zoom in here too nice for this one we just basically like because we're familiar that we set it up you could use graph in it as a scaffold and then start deleting stuff as well we could also totally help with you guys like anything of that this weekend yeah go ahead for sure yeah no problem one more okay there we go perfect so as you can see like this schema is massive there's a lot of data here what we're doing with you and swap actually is that you can query data that exists on the smart contract like live like right now you could vary the specific like amount of liquidity that is available in unis WAP but what we also did is we tracked historical data as well so every time there's a transaction we're making an entity which is a piece of data so as you can see this this is a pretty long schema 200 200 lines took us a few weeks to do but what it ends up allowing you to do is get to the get a sub graph like this so as you can see this I'll try to zoom in here as well so you know swap is is a is a decentralized token exchange basically where you just swap tokens and one of the queries that you're able to do with on the graph on the graph on the sub graph for it is right here we're organizing exchanges by trade volume and as you can see if you look over here we got dye which makes sense if you if you ever look at you no swap die is the most traded token so you start to get all these values like we're recording the price of die each time the price in USD the price in ether the balance of the contract and the liquidity of the contract and we're organizing it by a token trade volume which right here is is basically around six hundred thousand dollars a month getting traded so right now actually Yunus swaps main website for info is actually being powered by the graph right now so this website right here which is ran by Yunus off because we've worked with them this is all being produced by the graph so whenever it goes from block zero to two nine million it records all those historical data points and allows you to build historical data and it also has the values down here so the values down here for synthetic F and and maker and stuff these are more like live live values as well as 24-hour values so what we're doing is we we pick a day like the zero hour of of each day and we pick 24 hours and combine those values and then make day entities so you can also instead of querying all time you can query one week and it shows you like more condensed to see that doesn't really show you that much but when you go to three months you get a little more finer detail and you also get over like over encompassing values for the whole entire unit swap protocol which is interesting because each unit swap contract is a different or each exchange the different contract so we actually allow you to index those dynamically so it's we follow the unit swap factory contract and then index the 800 exchanges every time a new contract gets created from the factory we index that start recording that data we index it start recording that data yeah go ahead yeah so it's essentially it's written in the mappings and that's where it gets very complex like and that's why I took us a few weeks but right in here we're basically every time an event happens we checked what time it was and it's if it's within a day and we're keeping track of that day somewhere as well in the mapping and then every time it like crosses that day mark we we cut off adding to that entity because we're adding like how much trade volume happened in a day and then you start a new one and you keep adding them and keep adding them yeah go ahead yeah so you can bind to the contract in your mapping and call any of your viewable call functions within your mapping return data from that and then save that into an entity which is a nice paradigm to use yeah yeah yeah so yeah you can trigger on events as well as function signatures because some functions don't have events but we added that ability and yeah you can basically query any aetherium contract at that block that it happened you just bind to it you have to have the ABI so it knows and then it just calls it directly and we have a few different types of triggers so there's event triggers there's call triggers which will trigger anytime an external source calls into that function so then you'll index and get an event for each each time that call happened or you can use block triggers so your mapping will just run on every single block yeah yeah so we're working on a RC 720 subgraph right now that pretty much has tracks all transfer events without filtering on the contract and that is taking forever to sync it's yeah a lot of data so we're kind of actively trying to optimize and and make that something that can can run in like a week and that's like something we're working on right now but yeah but you don't have to filter on the contract so you can filter on just a signature and then get all contracts that that are applying that that function signature so yeah we're we're offering five prizes of a thousand died for the hackathon essentially if you're building your own smart contracts you can use the graph and and use graphing it and start getting the data from there and we can help you guys with that you can also just go to existing contracts that exist like today like you could do you know swap you could do something else and we have some ideas we could share with you guys as well and we're just looking for people to either build a subgraph of existing smart contracts or your own smart contracts we can help you guys with that likely you'd start off with graphing it but come on down and we can show you you know changing the schema or anything like that go ahead like within one of the width of winner within one of the event handlers so so we generate a URL every time you make a sub graph and on that sub graph details page you have a WebSocket URL and an HTTP URL you can then use to query directly in your application so we recommend using something like Apollo which is nice for working with graph QL and then pretty quickly using our URL you can write some graph QL queries and have a front-end right away yep so you can subscribe over WebSockets and you will you'll get the full query response every time new data comes in it's stored in a post quest database and we actually just recently updated our structure of how its how it's stored so we're using fully relational tables now and our queries have gone a lot faster just in the last couple weeks so we're pretty excited about that yeah any more questions no that's it all right thank you guys very much [Applause]