ETHParis: Optimizing dApps with EthQL A GraphQL Ethereum API

hey everyone my name is Michael this is Chris maybe else we're from in Fira today we're gonna be talking about optimizing your DAPs using something we're working on called eath ql so just about in fear a quick background maybe probably have heard about us but we were founded a couple three years ago almost originally as a Mista project inside of consensus meant to deliver infrastructure for other projects inside consensus and quickly saw the demand from the entire ecosystem to you leverage our infrastructure to really simplify that that component of the death of their adapt development so our mission all along was really just to simplify and enrich the developer experience so for three years we've been just providing easy scalable and reliable access to the etherium data and we also have like done our very best to support an ecosystem projects that make running full infrastructure well known infrastructure on aetherium easier things like VIP node we've been a financial sponsor for that project big fans of that project Alexia Kuhn offs work on turbo death is another example that we've sponsored and helped as far as eliminating some of the requirements it takes to run a full node I know there's a lot of misinformation people claiming that running the etherium notice so difficult that only if you're knows how to do it not really true at all but at the same time it is actually pretty hard for a normal you know a non tech person to really know how to run in a node so we work to address that as well so if you look at the evolution of aetherium so on the client side I went from one client to many clients as far as developing it you know go from the EVM assembly and to like more rich languages for Phil Silla tating that serpent solidity Viper started with absolutely no frameworks you were just kind of like on your own back in the day now we have truffle embark Paris whole other things out there that make kind of that whole development lifecycle much easier but then as far as accessing data from the from the blockchain you we started with JSON RPC and that's kind of still what we have so if you've developed add app or you've worked on trying to pull data from a blockchain you know how painful it can be sometimes to get data from the blockchain using JSON RPC so that leads us to Eve QL which is what Chris is going to take from here and explain about thank you so let's talk about JSON RPC JSON RPC was developed to give you access into the node right to send transactions to pull data off of the chain it's not really designed for as I ficient querying mechanism or efficient querying API the JSON the JSON RPC interface like a lot of resource based interface suffers from over fetching and under fetching problems right it lacks support for things like explicit data relationships hierarchy if you're just getting into the ecosystem and trying to create a Deb you have to read the yellow paper to figure out what all these data types are how they relate then you have to refer back to the you know the JSON RPC wiki or you know some documentation to figure out how to access it it's not a great developer experience the simplicity of the JSON RPC is great but it also is not does not really support a developer ecosystem doesn't give you a lot of great tools doesn't give you a way to describe the scam at all so there's a lot of things lacking within the developer experience so essentially a theory and data is a graph right I would posit that any non-trivial data that you're gonna have in an application is a graph right you have different data types or different documents you have relationships between those documents you know what I'm showing in this slide is a simplified view of aetherium data blocks transactions logs accounts they all exist this sort of different calls different data types that have relationships but they're only sort of implied via hash references when you pull a block or you pull a transaction and this kind of relationship is not easily captured in a resource based API it comes down to that you know getting useful data off the blockchain is hard right if you want to pull a block you want to scan headers that's pretty easy to do but when you try to start getting into application level doc application level querying you know things that are even as simple as this you know get only transactions related to my contracts list addresses that were that had sent transactions in the last block so while token transfers etc etc this stuff is not easy to pull off in JSON RPC right usually requires multiple calls you have to aggregate that data have to filter it down and then you can display it to your user so that's when we started thinking about graph QL as an interface into into aetherium so aetherium plus graph QL is what we're calling each QL it's a little bit of background into graph QL graph QL natively views its data as a graph so it's all about relationships it's all about being able to traverse the data in a very easy way it's been in use in facebook since 2012 the open sourced in 2015 and essentially what they were trying to do is figure out how to hyper optimize mobile clients right if you're talking about a rest interface or something let's resource based for something like Facebook you have to grab do a call for user data then a call for say newsfeed data than a call for you know this data and that didn't compile it all in so they created graph QL as a way to optimize this experience it's a single self-describing endpoint so it doesn't have that hierarchy of of sort of rest paths that you have to memorize in a rest world it's self describing so you don't have to have an overlay on it like JSON schema or swagger and here's the where it gets really interesting is that really flips the script on who's presenting the data or who's in control of the data controls not a great word but if I'm developing a server if I'm developing a rust interface you know I have the data in my database and I'm saying okay here's a resource here's all the fields in that resource I'm gonna give it to you because I don't know exactly what you're gonna need but you'll probably need something in this data so I'm gonna give you everything and then I define other resources that relate to it right but the server is control of how the data is displayed with graph QL they flip the script and say the client is in control of the data right so rather than the server giving you just blatant a document of fields that you don't need the client specifically asked for okay I'm interested in a block and I just want the number in the hash of that block just cuz I want to see where we're at it only the server will only send that information it solves a lot of problems with rest not only the over fetching under fetching problem that I just was talking about but also you know data relationships versioning deprecation paging extensibility which is a big one extensibility is a big one that ether QL gives to an API interface and my favorite is that it's strongly typed and it provides query validation out-of-the-box so you have a you can easily build tools on it you can easily query the schema and know exactly what you can ask for and exactly what you can receive so why would we want to use SQL so finally presents data as an applicable graph it integrates with a lot of powerful front-end libraries that you might want to use Apollo client you know maybe relay things like this it fetches exactly the data you need and no more no less in a single call which is important always taken to that later we reduce round trips to the server so we were improving the user experience by reducing bandwidth by reducing wait times by reducing rendering times and that allows the developers to build on a modern toolset using an extensible API it really eliminates the need for def developers to use JSON RPC so sql's been developing pretty rapidly there's a lot going on in this space it started as a sidecar server to originally start bringing graph QL to two aetherium a few people wrote Kripalani who started the original server plus nic Johnson's that were working on parallel tracks to bring graph QL we eventually found each other and we've now started to formalize it into an EIP so that's now the IP 1767 is up it's still open for comment if you're interested in it please check it out but comments use cases open dialogue there's a POC implementation of it currently in get master so if you wanted to play around with it and guess you can clone master and compile the server - - graph QL enables the graph QL interface and you can play with a node and it the standard version is focused on providing a simple yet powerful JSON RPC equivalent on the other side of things the server that row will Kripalani started is still being developed and it's more focused as an overlay on top of the the standard spec we're really focused on application level features things up pluggable back-end where you can create a plugin for your own ABI custom contracts and it handles a lot of the decoding and encoding for you flexible deployment model as a server sidecar kind of features that the SQL extended project has is you know flexible block selection single block list of blocks range of blocks navigating through blocks to transactions to accounts we're currently built on web 3 but we've optimized it through smart caching data loader batching we have transaction filters so you can say okay give me all the transactions in this block that only have blogs or that only have input data that kind of a thing we provide x-ray x-rays into transaction logs so rather than getting back the raw hex data you're getting back something that actually is decoded and is actually meaningful to display to a user we currently support er C 20 we support ENS address resolution and you can traverse storage you know Maps arrays of maps maps of maps etc so getting started is easy if you wanted to try hacking on it this weekend there's three ways this is really going around you Kewell extended server so you can go to this go to VAR repo clone it if you're running your own node you basically start it with with your own flag - J and then whatever your nodes json-rpc interfaces and then you can go to the local host the local host link easier than that if you sign up from Fira you can use your fira account download or clone the repo user and fury ID start the server go to the local host 4000 the easiest is that we have set up a URL for you already so you could just go to that bottom link it'll pull up the graph iql interface so you could start playing around experimenting with queries yourself if you're building something you can use that as the URL your graph cue ball client URL as well so let's talk efficiency gains we did some testing early on in the project to see how that how it would compare so essentially we pulled five blocks from the chain we're fetching the block cache and the sending address of each of the transactions the value and for each of the sender's we're getting their current balance so on the left side that's what a graph QL query for you know for this data looks like we're selecting a block a range of blocks hash transactions the account information and value right so you can already see if for those of you who have are new to graph QL it's a bit strange as a query language but the data types are nested in here right you have blocks transactions nesting blocks accounts nests in transactions so you can already see a query tree getting formed of that original diagram that that I showed before this is the query tree that's getting selected so if you were to do this over JSON RPC it essentially amounts to a little over 1,800 requests so there's three or four transaction at three or four hundred transactions in each of these blocks we're pulling five blocks and all the transactions plus a get account for each of those transactions right so that's 1800 the sum of all the data coming back is a little bit over a Meg with SQL it's a single call and 218 K so it's an 81% reduction in the amount of data of data sent over the wire all right I'm gonna try to demo this well all right so this is the graph iql interface for those of you haven't played around with graph QL before this is the the the link that I was showing on the other slide so if we wanted to get a block say latest block and we just wanted to see what the number was all right I'm just gonna try to peg it to this number really quick see how many transactions 74 okay so you can kind of see already the kind of tool set that you can have with with graph QL auto-filling based on the schema it's it's a introspective schema so you can basically pull anything back in a second so you can pull back the information regarding the structure of the API and you can get these things like autofill or let's say that you wanted to you don't know what you can query on there's documentation built on the sides right so we're doing a block we're looking for transactions and then you can kind of drill into the documentation all right so here we see we're pulling back all the transactions all the indexing maybe I just want to see the ones that have logs with logs true all right so now we're starting to filter out the transactions that are coming back to only the ones that we care about you know let's say we want to see what the decoded transaction looks like right raishin so with this I'm decoding ERC 20 transactions I'm like and see what operation never see 20 transfer so that the decoding of the ERC 20 is happening in the extended layer the way that I like to think about it is that the standard layer is optimizing on your querying from the node perspective right it's remapping a more efficient interface on to the node itself the SQL extended is a sidecar that you can attach to the node either through an RPC interface or the IPC interface and all of the logic for the decoding is happening there and what that gives you since it's built on a plugin architecture that gives you the ability to add standards like ERC 20 your own contracts if you wanted to add not it so within get the the idea of indexing is more than I think a node should handle the indexing layer is going to happen probably off chain or off to the client itself and into something like on the East QL extended server indexing is something that we actually have on our roadmap to build into into the extended project yeah oh and so like so they'll be like great if you wanted to write like an EQ extension for that and package it in with the rest of the node you could write you know plugin that would use any arbitrary indexing it could use your own data it could use you know something from another service you know that's it's definitely possible so with just those few lines of code I'm able to pull back you know the late not the latest block anymore but this blog number and then information about all of the ERC 20 tokens that have been traded or transferred so kinda to extend on the demo a little bit I have an example app that that I built for if you go to a repo which all I'll show you at the end there's a really simple react based app that I did plus a node an example and note of how to query graph QL so this is a simple table and basically just scans scans the latest blog for TRC 20 transfers it a grits it based on the symbol and then sorts it descending right largest value transfers top to the smallest transfers on the bottom and this is basically the code that does it there's not a whole lot of code to it again on the left side sort of what I just demoed this this is the query and on the right side for those of you that build and react that's really all you need to connect the data source to your application so I'm using Apollo client you have an Apollo components query component we are inserting the query you can tell it to pole so like all of the polling is handled for you and basically it once it returns it shoves all the data back into your into your components so really quickly you can start building you know data centric components connect them into these higher-level queries and the data just sort of resolves and propagates down the componentry so what's coming next for the roadmap for the East QL for East q1 general we're trying to finalize the 1776 spec 1767 spec so that we can start building on it propagating it into other clients right now like I said the proof-of-concept is in Gath but we're try to get into parody and Pantheon and all of the rest of the clients as well we wanted on the extended side we want to add support for more standards popular standards like 721 the RC 165 we have ENS address resolution support but we also want to add decoding ability for ENS events we want to make it so that's easy and flexible to deploy and embed into your applications graph QL subscriptions is something that's big it's not going to be part of the core specification the court VIP but we want to be able to add that as a layer on top so opening an EIP for subscriptions which will make sort of all the filtered and subscriptions event subscriptions things like that I'm really simple to deal with so and eliminate the need for polling we want to add schema variants for paging and also relay clients integration with indexing backends to level cache and new filtering strategies you know things that support the top developers because at the end of the day the East QL project is made for DAP developers for you guys to use it break it contribute to it you know let us know what you think about it we're on getter this is the repo again you know come say hi with a star open a pull request comment and then here's some links for some more information so we set up some documentation slash Doc's if you wanted to learn more we have a SQL playground the thing that I was just demoing with at the hockey's QL and sure that IO and that examples repo which has the example application I showed plus a node example at the bottom and that's it thank you very much questions so the graph is interesting that you bring that up so the graph is focused on event data solely and there that as far as I understand it they're abstracting the chain away right like they want to be not only for aetherium but for other chains as well but they're solely focused on indexing event data what each QL does it provides that and a complete replacement for JSON RPC so you can do things like get block block scans for transaction hashes or you know whatever your use case is additionally what it provides is the call and send raw transaction functions right which which the graph does not so really each QL is a complete meant is a theorem specific graph QL interface yeah so it's not necessarily a query planner we use not a query planner per se but we do use batching and caching so the nice thing about graph QL is because of the structure and because of the way that on the spec is defined every graph QL query basically turns into a query tree right yeah that's correct currently yes yeah that's so the nice part about sort of the indexing thing so right now it's a very simple interface I would say right it's basically a skin to be able to reduce the data and round trips but since it has like a plug-in architecture basically you can start putting in more sophisticated indexing you know things that are more specific to your use cases what we're trying to do is provide enough to make it useful but not so much that it's opinionated on for people's particular use case and leave that opened to the developers to be able to put specific things in yes I think there's a little bit to unpack here I might need to dive into that more specifically [Laughter] stay deaf I'm not sure I mean we do support it in a pretty simple way like we have the ability to decode storage so that you can at least see what's what's in there doing it an actual state of like built into the query language isn't yet supported yeah that's a good question I think it's I'd have to check on that I think it's MIT but I wouldn't I would have to check the license on it so I miss like yeah [Applause]