Building dApps that Load Fast on The Graph

just as a reminder we've got some bounties for this weekend so we're giving out $5,000 three so it's $1,000 for each winner and three of them are for best new subgraph and two of them are for best new app it built on the graph and I'll kind of explain what exactly that means so so the graph is a protocol for indexing and then querying data from for web three so that would be you know from blockchains or like storage networks and the problem that we solve is that as soon as you start building an app on a blockchain like aetherium you quickly realize that you kind of can't just rely on getting the data directly from that blockchain from like an etherium node for example and the issue is usually if you want to put something up on a screen you need to like filter to just find specific data usually you need to kind of transform that data ahead of time and so you end up doing you know a lot of like filtering stuff on the client and that really slows your app down and so there's this kind of like missing indexing layer that allows you to just run you know queries like you're used to when you're building an application and and so that's where we come in so you know today there's kind of two ways that you can try to solve this problem yourself and most teams have done one of these too right so either you build a custom proprietary indexing server and this is what actually most teams probably do where it's just a bunch of custom code that's like talking to an aetherium node stuffs the data in a database and then serves it up over an API and you know that's really fragile because you know everyone's doing this kind of processing work differently there's no way to really have consensus and you know if people are building applications on top there's no way for me to know if you did that computation correctly so we think that that so the the other way you can try to do this is to just load everything on the clients and then filter locally on the client and if you do that you know you can kind of keep it decentralized but you end up waiting a very long time for that data to sync so at the graph we're building this protocol for it in you know having a network of nodes that can index that data for you but it's pushed off to the network so you don't have to do all of the indexing client-side but we're kind of doing this in stages and so we open source our graph node implementation July of last year and then in January we launched a hosted service where we run a bunch of these indexing nodes for you and then we're going to be launching our decentralized network as a next step so let me kind of walk you through first the graph Explorer so who here is seeing this suite okay cool so then I'll walk you guys through it so this is where you can see like all the data that's kind of being indexed on our hosted service [Music] it mirror cool thank you for that okay so this is where you can see the data that's being indexed on our hosted service and so these are what we call sub graphs and they define how to index the data and make it available over graph QL who here knows about graph QL okay most of you so so graph QL is a query language from Facebook that's kind of a replacement for rest and it's just a really nice easy convenient way to fetch data from a server so here we're looking at the Moloch sub graph Moloch is a Dao for funding aetherium infrastructure and you can browse kind of the domain model here right so this is the schema so you can see that there are votes and proposals so a proposal has this information it's got applicants for example token tributes write these all have types and then you can traverse across to see the applicants etc so you define the schema upfront here's all my objects here's how they relate to each other and then you can query using graph QL we're here for example we're getting the first five votes for each votes we want to display the members and the proposals and for the members we're asking for the you know ID shares to the debt and for the proposals there and you can see the response perfectly kind of matches what we asked for now one good trick we should have a little thing that calls this out to you is you can hit control space in here and you'll get a drop-down with the different kind of fields that are available and then you can query and you can seamlessly traverse these relationships so for example ppthe I think is a really cool one so they have data that's they have like a contract on aetherium and then they've got a bunch of the data is on ipfs but this is basically like a Twitter kind of thing where you know these tweets or peeps are stored on IP FS and we index all of that data so you can see like a peep has you know this content reference is an account and the account has like all of this info so you can query for the peeps and then get the account like name and about avatar URL for example and then you could like easily build a Twitter client that you know gets all of the data like this through the graph so let's take a look at what it takes to build one of these sub graphs the first okay so step one for everybody if you're interested in building on the graph this weekend is go to our website and join the discord so it's the graph comm slash discord there's a link on the right discord drop in there and jump into the hackathons channel and say hello and if you have any questions we'll be there all weekend and we can help you as you're going along your second biggest friend is the docs so from the website you can click on Doc's and we've got a really great you know tutorial and it'll you know define everything and so this is pretty comprehensive and she gets you most of the way there but the first thing I want to do from here when you want to get started is you'll sign in and I should probably do this with sign-in so you sign in with with github and we we piggyback on github for permissions basically for like teams because there's a lot of like teams that use the graph so basically if you're an admin of your org on github then you can manage basically you can you can deploy these sub graphs under your organization name so here you can see like the accounts for each of these sub graphs are basically it's like your github in phone.we and and so we pick it back on the permissions for that then you can check out your dashboard and this has like sub grass that you've created and this is where you get your access token you'll need this when you deploy your sub graph from the CLI and you can start by creating a sub graph just through the UI so you come here and give it a name subtitle which shows up in the card so you know my awesome sub graph and then subtitle here and then your card will show up in the community sub graphs with the others and then the next thing you do is actually what you can create that it'll show up and then you can go build your sub graph so what is a sub graph if you follow the examples or the the kind of quick star in the docs we have these really good CLI tools so you just install graph CLI and then you can use that to like graph in it which will create like a scaffold for you and if you already have a contract that's deployed either to like a test net or on main net you can do from I don't think we have it in here but we'll actually yeah I don't know there's like a blog post that's where we have this but it's - - from - contract and then you can give it a contract address and it'll it'll just scaffold one for you that basically brings in all of the events and then index of them so that's just like an easy way to kind of get started but then you end up with a directory that kind of looks like this and the first thing to look at is this sub graph yamo file this is kind of the the entry point and this is where you define your data sources so probably you have an aetherium contract and you can point to the specific address so this could be on you know maenette or it could be on one of the test nets rinkeby Kovan you can also run this locally with ganache so we've got some getting started guys that'll walk e through having like a local development environment using docker compose if you've got if you're going to be working on our contracts locally you can do that as well then you basically specify these like event event handlers and these handlers are going to get called anytime one of these events are triggered and then you can process that data at ingestion time and and that data will get indexed I'm gonna walk you through that part and then I'll open it up for questions because I'm going through a lot of stuff but this is the graph QL schema so this defines the shape of your data and and this is actually like a really great place to start as well you know even before the manifest if you're just thinking through basically how what would the perfect API be for consuming whatever I'm building I am this is where you're defining you know the domain model for your application and so you want this is really just ready to represent your app as best as possible so this is the Moloch example where there's proposals votes applicants so you set this up and then this is how people will be able to you know query for this data use a graph QL and then the last piece is the mappings so this is how you specify how to like transform the data at ingestion time so you've got an event and then you want to process it somehow and basically store these entities into or store abstraction and then that's where they get indexed so these mappings are written in a language called assembly script which looks and feels just like typescript but it's strongly typed so that it can compile to wasm and that way we can transform the data in a deterministic way which we think is really important so from these mapping handlers you have access to the event data the transaction data the block data in the smart contract state as of that block and then you can you know do arbitrary kind of you know logic on that and then you know save that data to the store so one thing that's really cool is I highly recommend that you get something like a Visual Studio code which has strong typescript support and when you build your subgraph and then there's like a build step we auto generate all of these types for you so you can write these handlers with autocomplete and and type safety so for example here I want to handle this proposed event and on the event I can see that I have access to like the block and the transaction data so I can see like what's on the transaction so I can get that or you could get like the params for the event and then we'll like decode the name so if this is like your and if the event has named parameters you can see what those are called so this is how you get data from the event transaction block and these things that you you know you're creating these entities they're like you know objects that you're saving to the store these are also auto-generated for you from the graph QL schema so here you'll have this directory that's generated for you that's has all of the these entities that are just the types that you define in your graph QL schema and so you can just create these objects you put them in the store and they're indexed okay that was a lot so let me open it up for for questions how much of that made sense raise your hand if you followed awesome sweet yeah so so this is all real time we react really quickly to new blocks getting mind so yeah basically as soon as you know the transaction gets included in a block it the graph total will pick it up and process the handler right if the if your event was emitted and then that becomes available immediately yeah yeah exactly so yeah really I mean like 90% of you know daps probably have like a server that's running that's actually doing this kind of thing and because you can't really do it directly on a theme you can try to use like web 3j a switch like in a hackathon you can kind of get away with it cuz there's not a lot of data but like you know if there's like ten transactions that have only ever happened on this contract you'll be fine but if you have like hundreds or thousands of transactions suddenly you have to like load them up load all the you know if you know logs so like then filter and then like transform things on the client so it really doesn't scale so so kind of anyone who's building on aetherium should probably be using the graph to like index that data so that you could just query what you need from your applications I did I didn't go into a lot of detail but but yeah that init tool which is described in the docs so if you just do the QuickStart in the docs it'll walk you through using like graphing it and it'll scaffold that for you so you don't have to start it from scratch so that's why yeah here I'm keeping it a little more conceptual so make sure everyone knows how to use it the anit tool just creates the scaffold then there's these build tools so you do like a yarn build which is calling like you know graph now by scaffolding I mean it's just creating these linked example files for you so it'll create like an example manifest file an example graph QL file and an example mapping type script file was just like like an empty function just as a starting point and then when you run I yarn build so if you look at the package jason so these are like you know if you're familiar with kind of javascript tooling you know you can just run like yarn build and that actually runs graph build and then you can also use it to like deploy so you can say yarn deploy and this one i pointed it to this URL but it'll it'll actually come setup to deploy to your their to the hosted service so you can just deploy it and then while it's deploying you'll see it like sinking in the explorer so let's see this was just updated but this is just created oh cool people are probably already starting to create stuff some of these most of them are fast sometimes they can take a little bit longer this my people are doing a lot of like active development we're running two hackathons concurrently so if for any reason like your sub graph fails after you deployed probably because it was like a bug in the handler you can click on the logs here and you can see basically all the activities so this would show you every time an event was triggered what handler had tried to call and then you can you can basically yeah just just see that and then you can filter on errors if there's an error and this is how you can see why that would have happened yep sorry no yeah so this is just public endpoints oh yes good question so does everyone know here know how to like build you eyes with react or what's the there's maybe different like UI libraries of choice if you're using react we highly recommend Apollo and we have if you go to our github there's a eath denver dap example which is a really good kind of example that for building a front-end that uses graph QL and so this uses apollo as the kind of client library and you can you basically just pass in a graph QL and point into apollo and it integrates nicely with react and it'll in a hit your graph QL endpoint for fetching that data so yeah these are just public endpoints so there's an HTTP endpoint that you can use for queries and then a WebSocket endpoints that you can use for subscriptions so we actually handle like live queries where if you just use the word subscription at the beginning of the query and graph QL that makes it a subscription and then you'll get push updates over WebSockets whenever the results that changes you know for the time being we don't have any rate limiting I mean we have like DDoS kind of protection but like other than that it's it's all just there now at some point in the future we're gonna add a page here where it's just kind of to cover the costs of the servers and we're gonna be experimenting but with basically like gasps costing is essentially you know this is gonna be running on a network of nodes and those nodes need to be compensated for like the the compute that they're doing and so there's going to be like a gasps kind of costing model but for now it's all just we don't have too many limitations yeah yeah so yeah so if y'all are interested you can go to our github the main repo is graph node it's written in rust and yeah we use Postgres under the hood is the database it's kind of an implementation detail we kind of like reserve the right to change the database in the future but you know you just access it through the store abstraction and then you know we use it when you run the queries what specifically are you curious about so what we use ipfs for is well there's okay so what we use it as the protocol for is we store these sub graphs on ipfs so what happens is when you build and then to deploy what you're actually doing is oh okay right what you're actually doing is we take we take them the manifest and we like upload all of the Constituent files to ipfs and then we take the manifest itself we replace the links with the ipfs hashes and then we upload that ipfs and then we get back the hash of the manifest and that's like a unique content hash that you know uniquely identifies your your subgraph so that's the subgraph ID so anyone could basically get this file and it would give give them everything that they need to index the subgraph so that's how we use IP FS and then you can also use IP FS like you know if you have data that's on ipfs in order to access it you just have to make sure that you like log ater somehow the ipfs hash is anchored on chain because right now we only support like aetherium as the data so like route data source but in response to an event you can for example get an IP FS hash and then you can like cat the data for my PFS and then access that from your mapping handlers yeah so anytime you make a change you get a new ID back and then you can deploy updated versions and and so we support kind of versioning yeah it's the ipfs hash exactly one more thing I should mention is jorge our engineer back there just recently added call and transaction trigger so this is a cool new feature before you could only trigger on events and now you can also trigger anytime a like smart contract function is called that can also be a trigger if you need that cool all right I think that's it for time all right thanks for dropping by