The Graph by Yaniv Tal

alright hey I mean even one of the co-founders at the graph and the graph is an indexing and query protocol for blockchains so you know we really believe that you know we want software to be built on a solid foundation and I often get this feeling that the way that software is deployed globally today is a bit like building on quicksand and so I think you know all of us in this room are probably here to try to work on this problem and we're at the very kind of you know beginning stages of this and you know it starts by building on protocols that have certain guarantees right around stability openness security verifiability and you know right now we're focused a lot on like financial use cases and you know defy dows there's you know a lot of these early cases coming up but you know we we really believe that like this is just how all software should run eventually that you know you have protocols that provide these guarantees and you kind of build up from there and that that's gonna allow you know society and civilization to you know scale and coordinate in ways that you know we can't do today so the problem that we solve is once you try to build an application on top of a block chain like aetherium you quickly realize that it's actually pretty hard to get the data that you need directly to your front end application alright so you're building some kind of web or mobile app and you want to you know run some kind of query to get some data you want it maybe filter search paginate and maybe try to do this using like web through j/s or something like that and basically it either takes too long or you know you kind of have to just load all of the data on the client and filter it locally and and you kind of get stuck and basically the problem is that block chains are missing this indexing layer so when you have a database a database maintains indexes so you know on at the bottom of the stack it's using the file system to like store essentially like logs and data and then on top of that it builds these indexes so that it can run these queries or it can efficiently retrieve the data that you want to get from the database and blockchains they're really good at transactionality so if you want to set up rules for who's allowed to update some state the blockchain can enforce that and they can build you know these blocks up but making data efficient for retrieval is you know kind of outside of the scope so that's where the graph comes in where we're building a decentralized network of nodes that I can index that data in a reproduce and in a reproducible way and make it available for querying so the clients can access that data efficiently so basically the way it works is you define what's called a sub graph and a sub graph basically defines how to index that data in a reproducible way so all data on the graph comes from a web 3 data source so that's kind of the Box on the right there and that's really important for us that in order for this to be reproducible it's important that the data doesn't come from any sort of web 2 data source so if we allow you to just hit some kind of HTTP API that API could return different data every time and you wouldn't have that reproducibility so right now we support aetherium and ipfs and we're going to be adding more data sources more block chains more storage networks more protocols over time then you define these mappings which are compiled to wow them so you can do any arbitrary data transformations on the data from those data sources so if you need to do aggregates if you want to kind of denormalize the data transform it so that it matches you know an arbitrary schema then you can do that with your mappings that gets put into a and then served up over a graph QL API so who here has used graph QL before okay so not a ton of people so so the quick pitch on graph QL is it's the easiest most convenient way to fetch data from a server and instead of rest where you have one endpoint per resource we're likely the endpoint doesn't have exactly the data that you want which then forces you to make like multiple round trips and there's kind of just this mismatch between what the server provides and what the client wants you can define the entire schema upfront you say here all of the different entity types this is how they relate to each other and then the client can ask for exactly the data that it wants and you get just that data back in a single response so it's really an API that's designed for consumers of an API where you can get exactly the data that you want back so I'm gonna show a brief demo of the graph so so this is graph Explorer you can see if you go to the graph calm slash Explorer and you can see some of the featured sub graphs we have at the top here and then there's a bunch of sub graphs that people are like building and deploying every day so we can take a look at some of these so there's this like melon port one and I'm going to try to zoom in a little see if that works so here you can see that like this the sub graph is fully synced so this is the latest block on on main net and you can see this kind of graph QL schema on the right here maybe I can make it even bigger so in melon port they're like a fund protocol on a hiriam and you can see it's a really rich schema there's all of these different like entity types is to just kind of graph QL so you can see these like investors and trading and shares and policies and like all of these things so if I check out like a fund so a fund has a name it has a manager it has like all of these fields on it and you can see that it's all like relational so you know a fund for example refers to like the investment history and I can click on the investment history and see what that's got like the shares and the share prices and the amounts and then you can run queries kind of in this playground here and we stand up this graph QL API so these are public endpoints HTTP and WebSocket endpoints that you can hit directly from your web or mobile app so the graph acts as like live API back-end that you hit from your apps to get all all your data so now everything's like super exploded and big but let's take a look at another one so this is P pyth which is a fun app it's got like a Twitter clone so we've indexed like 53,000 entities here and so these peeps are like like the tweets and we recently added this like saved queries feature so you can you know type a query and then give it a name so it's easy to find so this for example is like PP users and their followers so if I want to grab the accounts and get their names and then their followers I can do that I hit play I get back those responses and and what's cool is even if I don't know what data is available I can hit like control space and I get this like autocomplete so I can see that if I want for these followers I could get like any of these additional fields so maybe I want like see if they have an avatar URL for the follow ease and and if they have them then it shows up in the response there so you can see it's like relational this way another cool one might be going to drop this for a second so here I'm looking for the peeps and I can get like the content and let's do the I mean do you like replies or let's just do the timestamp so that's that's just like a super simple one so here's a bunch of like peeps so you could see you could like really easily build like a Twitter type clone on top of this API you can ask for just the data that you want you get it back and so that's how you kind of query data and use the graph to get the data that you want so now I'll show you what it's like to build one of these sub graphs so the first thing is you can come here and just sign up or sign in with github so we do like an OAuth thing and I already have an account so just pop them right back here and then you get a dashboard where I can see like my sub graphs and here's a sub graph by this project called affogato that's doing like a coffee supply chain on it area which i think is pretty cool because you know coffee is an example of like some you know market where you know prices are really opaque and you know these farmers aren't getting a lot of money for their labor and so you could create this kind of direct-to-consumer or any kind of you know marketplace to connect people directly to these farms so you can see that like you know farms have names and they're in like you know regions and you can check out like the batches and all this stuff for this coffee so I've got a subgraph that's created here and it starts with this manifest and can you guys see that cool so in your manifest you specify the data sources so this is going to be an aetherium contract here's the contract address and then these are the events that I want to listen to and so there's different types of triggers so we support events and recently we added call handlers as another kind of trigger so you can trigger anytime a smart contract function is called you could use that as a trigger and then you specify the handlers they get called then you specify your graph QL schema so this looks very much like what we were looking at before this is standard graph QL it defines basically the the data model and then you have your mappings so so the mappings are built in a subset of type script called assembly script which compiles to wasm so so this is pretty cool we kind of looked at all the different ways that you could build for wasm and we thought that this was like the easiest to learn since everybody knows JavaScript or typescript it just has like a few quirks but we've added a bunch of tooling to make it really easy to build on top of this so here we're handling this add cut profile event and you get the event and then you have access to the event data the transaction data the block data you can get smart contract state as of this block and we basically like event source so we'll start at the first block and we'll run through everything and we'll just like run all the handlers index all the data and it's available over graph QL and what's cool is you know we generate all of these types for you based on like the schema and on the api's so you get autocomplete on everything so here we can see that we've got like these cut profiles and they have like some data but they don't have like all the data that we might want and I happen to know that this contract it has like aroma and some like other information about like these cut profiles for this coffee that I think would be cool to show to users so I could easily add that data and index it in my sub graph so basically I come down here and I say I want to add some data to this cup profile say I want to add like the aroma and I can get that data the event so I get like autocomplete here and I know that that data is available on the event parameter so I can look in there and I can see like here's everything that's on the event so I want to grab like the aroma and then call to string so this is the kind of assembly script II stuff for wasm to make it happy and then let's get just a few more fields so we'll do the flavor and the acidity let's say that's just do those three and I had an extra dot so I just extended the subgraph to put some more data underscore a thank you that's I deleted the wrong one see and it gave me the red squiggly so I knew at compile time cool so now I can jump into my terminal and I can basically deploy and that's gonna I could start first by just doing a build so basically what it's doing here is it's compiling the assembly script it's uploading the waz M files to ipfs it's hashing the manifests uploading that to ipfs so everything's on ipfs and then i get back this sub graph ID which is that ipfs hash which uniquely identifies this version of this sub graph so it's like content addressed and then I can deploy that to the hosted service so right now we're running a bunch of nodes for you this is this like intermediate step on the way to the fully decentralized network so right now I'm just going to deploy this to the hosted service I should be able to come back and see it indexing here so we support making like zero downtime upgrades so that current version is still running and if you know my app is hitting that end point it would keep running but in the background its syncing this new version right now so you can see the whole progress bar is going through its chugging along through all the blocks its indexing data you know whenever it sees the events and in the meantime I can come here and I can see that in the cut profile I should have some more data and I should be able to query it so so that's the graph let me go so how am i doing on time I don't when did I start perfect so the kind of data that you know I'm excited about in kind of the short term I mean basically we want to just be indexing all of the data from all different networks make it really easy to to access and so that people can build applications on top of fully decentralized protocols and you know in the short term you know the stuff that really excites me about this is the opportunity to create like global open markets for everything where you know people don't have to you know give away power to these like large centralized entities that exert their market power you know to extract and you know in the in the short term examples of that would be just you know dows so there's a bunch of like Douw protocols dow stack uses us and we're getting started in engagement with like Aragon and colony and all of these dowels I think are really exciting milwaukee's with us you can check out like all of the proposals and and everything on on mohawk things like you know get coin in places where just any kind of marketplace where you know people you can basically vouch for each other skills and you can like perform tasks and you can like find people to perform tasks I think you know all of this kind of stuff should be on web 3 and when it's all available like those people be able to build more and more applications they'll be able to customize them and you'll know that all of this software can continue to run forever and you can build on it you can you know trust in it it's you know kind of a stable foundation for large-scale coordination so if you're building in the space if you're building on aetherium or other block chains please reach out we'd love to help thank you very much [Applause] thanks for the talk and a quick question about the interaction between the query layer and chain is it like decouple in a way that can be used on any other chain as well or is it tightly coupled with etherium yes so in in the first version it's more tightly coupled than we would like but we're specifically going to be adding support for multiple block chains did someone change something there because I would bring closer okay so as part of that effort we're going to make it like much more modular specifically so that it uses like a plug-in architecture where we'll build like the first few plugins to add support for multiple chains we're really interested in like cosmos and polka dot for example adding support for those and then eventually anyone should be able to build their own plugin for their own data source you said in well thank you for the talk and also great project you said that you currently use support theorem and ipfs roms because you want like cryptographically audible input right to have it be reproducible well what do you mean with or what is your vision or roadmap for supporting ipfs can i push index data into ipfs do I try to index data from ipfs right now I think it's used to like initiate the node with instructions oh so right now we do use ipfs just ourselves internally as a protocol for basically storing all the metadata for the subgraphs but so like when you deploy a subgraph what you're doing is you're uploading you know the wiles and file and the manifest and everything onto ipfs getting back a hash and so a node anywhere in the world could just look up that hash on ipfs to get everything they need to index that data so that's just how we're using it internally the way that you know projects that are building on the graph can use IP FS today is if you have some data that's stored on ipfs and you have a hash that you then anchor on chain so you have like in your smart contract you law for example the ipfs hash then in your mapping script you can cat that file you can get the data and you can index it in your mappings so for example I didn't point that out but with the PIPA sub graph most of that data is actually on ipfs so that was a pretty complicated so I'll grab that till I get a lot of data from ipfs so we support that today and then there's a question of like how would we support ipfs outside of the context of aetherium altogether and that just becomes a little bit harder because so one way to do that would be for example to support IP NS as a data source and then whenever the IP NS reference changes then you'd kind of you know that could be a trigger but the problem is you don't have any strong consensus with like IP NS it's like eventually consistent and so we need to figure out how we're going to deal with that because we want our protocol to maintain consensus one of the important requirements we had from day one is that if you issue issue a query for example as of some block then there should be one and only one correct response to that query so we need to have strong consensus and so that's an open area of research for us because I'd like to be able to you know support like an orbit DB kind of model or like IP NS as a data source and we still need to figure out how to do that so so you on the three points like the first point just because I am selfish and I want to dig deeper um so is this when you say like we want to make sure that the input instructions to the node is also verifiable so that different nodes have the exact same instructions is this sort of in a vision towards having consensus on the indexing of different notes as well and where what do you see happening there yeah it's it's just I mean we think ipfs is just like a great abstraction for like file storage and so we have this problem of like we need to store the files and so we weren't going to reinvent the wheel and so ipfs was a good solution to that problem we do want the protocol itself to be open and permissionless and so the easier it is for any node to kind of have these guarantees where they know that like the data will continue to be available to them that's important and that's why like we only support these strong web 3 data sources as inputs so that you you know really have these guarantees anyone at anytime could like rebuild the state and you know index a subgraph and be able to provide that service great thank you so much again Ian if thanks [Applause]