Best use of Goldsky - Superhack 2024

hey hackers and welcome to super hack 2024 we're so excited to be partnering with eth global today and to be a part of the super chain ecosystem it's incredible watching it grow and expand this is just such a cool time to be in web 3 my name is Andrew I'm the customer success manager at Gold Sky and I just wanted to give you a quick overview of who we are in case you're unfamiliar with gold sky at a very high level our goal is to essentially streamline how projects use onchain data when it comes to understanding what's happening on chain making sense of it all and effectively maintaining that data in the back end to use it for your project it's a lot of work can be really challenging and it's a quite difficult process to have to you know always think about and just always be working and allocating resources towards when maintaining web3 data tooling you know there's many variables you have to consider and the obstacles can quickly stack up so just to name a few pre-made endpoints that can you know limit customizability of what you can build SQL apis can often be complex and not as efficient database sharing could be really slow spoke indexing could cost a lot to maintain I mean to be honest it's just it can be quite a pain especially when you don't have a whole team dedicated to maintaining that infrastructure 247 so that's where we come in we do it for you so that you can focus on building there's many solutions out there today already but sometimes you can say that the batteries aren't included and at Gold Sky we aim to provide a comprehensive solution that includes everything you need to get started started and also keep running smoothly so some of the key principles that we really value are speed efficiency uptime customizability we offer robust tooling to allow you to build the exact use case you want and have dedicated support to walk you through that process so that we can ensure you get there so the two core products that we offer are subgraphs and mirror and today we're going to be diving into how both of these work so that you can start building with gold sky but at the most basic level subgraphs allow you to pull onchain data and query what you need whereas mirror allows you to push onchain data directly into your database to store where you want it so why don't we dive in and learn more about how these work hey hi there my name is Javier I'm a Solutions engineer here at Gold Sky and in this video I'm going to demonstrate the multiple ways by which you can deploy s grabs using the platform and so what you see here on the screen is essentially a um essentially Gold Sky dashboard uh and I just created a a test account for the purpose of this video so when you create a an account uh and you navigate to sap graphs uh you will be presented with this quick start model which is basically a very easy and intuitive way for you to uh deploy what we call Community sap grabs and Community sap grabs where they are are uh essentially curated um sap grass by the Gold Sky team that you can use for free and so here we have a bunch of them but by the purpose of this video I'm just going to go ahead and um deploy ethereum blocks for example so when I go ahead and click deploy I'm going to have these Community sub grabs uh immediately being deployed onto my project so this what we see here now we are into the uh sofra detail page and where we can see a bunch of very useful information like for example healthiness of the sof graphs where we is some some very good metrics on it also logs and we also got a a few things like you can create web hooks uh so for example um when certain entities of your sub grabs they get saved for instance you can communicate this information into your external HTTP endpoint uh and then you can do all the things of you know on your infrastructure um so this will be the first way to go about it uh as you see here we also have a graph Q endpoint that you can use um so this will be public by default and essentially you canuse s it um there's also the possibility for you to create a private graph link so in this case um for you to be able to query this endpoint uh you will need an authorization header so this is sometimes this is better in case you want to restrict the number of API requests onto your s graph um so this will be the first approach right uh the second approach that we can go about this is essentially creating or deploying the sap graphs from the from the source code and so uh just a second second method I wanted to go over uh in this case we have this poab s graph here and so what I'm going to do is I'm going to go ahead and deploy this s graphs onto my project so um I already have this let's actually go ahead and maybe delete this we can go um right so and then clear this bit there we go and so now uh let's just clone this okay I just clown It uh C into it um so what I'm going to do first is I'm just going to install dependencies while this installed we can take a look into it um essentially is a very very simple s graph from po app um we need to install dependencies and it uses a a template so I'm going to run the template so we can get a proper sofware of Jamma file uh and then we can just deploy it using the CLI command from Gold Sky onto my project um all right so this is now done uh so what I'm going to do now is uh jump rep peret to get that sub graph G file okay now I create a codm and build the subgraph okay good all right so now um what I'm going to do I'm going to deploy this uh s graph into my project and so to do that um I'm going to go ahead and use the Gold Sky um C command um one of the things you will need to do first uh is run the Gold Sky login command so this will allow you to input an API key which you can create here onto your project we go to settings and you can create a new key over here um this will allow you to basically connect CLI into your project I already did that so I'm already connected to my dashboard no need to do any of this so what I'm going to do next is around the the Gold Sky SRA command so just going to do it with Gold Sky s graph um and then deploy put on a name for it so I would just say something like um o sub graph a version for it and then Define that this is going to run or create be created from the source so I'll deploy this from source using the Gold Sky command and I'm I'm given here a essentially another another Endo right we can use to query so actually that's I'm going to run here so if we do actually run another one maybe something like this there we go all right so this will be one um now uh if we go into my project we can see that this is now deployed here um and now we also have you know the ability to get into the software detail page um one of the things that is uh actually very useful when it comes to you know this graph C end points is that in in many occasions you may want to update your s grab logic um could be because of the mappings have changed or maybe you just want to um create it or redeploy it from a new start block uh in those cases what's going to happen is by default the graph ql endpoint will change uh but you may have a ton of client facing applications stock into that endpoint so it's going to be very costly for you to update all the clients right so instead what we can you do is we can put on a tag onto the S graph so we have a fixed uh endpoint and then we can easily switch tags as we go and up data sa graphs so I'm going to very easily uh very quickly go through it um so we go into let's actually look into this if we go ahead and look into the code what I'm going to do now is I'm just going to create a new version of this F graphs for example um you know we new start block for instance like this and hit uh pretty much the Coden command again and code let's actually do uh code your buil yeah all right so now we're ready to deploy a newer version of this St graph so before I do that what I'm going to do is I'm going to tag my existent s graph with this prod uh tag or whatever we're going to call it right so I'm G to go ahead and say Gold Sky SRA tag create and this is going to poab S graph uh called it gra one z z and just going to call it pro what I'm doing now here is basically I'm going to have this static prod endpoint which is basically the exact same one as this one here so you see the deployment ID is this one while this is this one right okay so now we have this fix endpoint the next thing I'm going to do now is I'm going to deploy a newer version of the sa graft with you know with that little change that we did before so for that I'm just going to go ahead and deploy version two of the existing SRA um let's go ahead right so now we have another sub graph here with another URL uh we go back here into my project we can see this newer version um which is going to be version two now what we can do is we can move the tag so that we don't have to query this new endpoint instead we just connect that fix static endpoint onto the newer version of the SRA so uh to do that we just basically run the exact same command that we did before so now two and now we're adding that tag into it so if I go ahead and uh this is prod end point refresh it and then we run it now we see the deployment IDE has changed so this is one of advantages of using tags which are actually very powerful um so that's one thing uh we deployed subgraphs using Community um subgraphs from Gold Sky from Source um one last thing which I wanted to go over is what we called um instant soft graphs which is basically as the name imply is a very is a very easy way for you to deploy sofra without actually having to code them um and so without really getting into the need GRE of it I'm just going to show you a quick example of how that works um so over here I've created this uh no code FR Tech Sacra U if we look into it let's actually take a look um you can see that these instant s grabs um they basically rely on two main things which is basically one config file uh this config file defines the name of the sa graph as well as uh where the AI of that sa graph is located and you also need to input some information such as the chain uh the address of where the sap graphs so the contract is located and the start block this will be the simplest way of it you can also complemented and use what we call enrichments uh and then we can basically add event handlers Etc so basically create a no code s grabs without you having to actually get into create the full Blom um project or or scaffold of that Sacra ID um so we have this and we have we're pointing to the AI JM and the API here um with this we already have enough to be able to deploy this s graph uh using Gold Sky so how we go ahead and it's going to be with this command let me just switch here um this is going to be Gold Sky s graph uh then deploy since uh this is a FR Tech I believe yeah this is fr Tech in in in base yeah in base I'm just going to call it C like friend take um 101 uh and then we passing the the option from maybei which is basically maybei is going to be my friend Tech config Jon file right so this way it's actually the SRA is been gener ated by gy based on that config file and now it's available right here so now we have our deployed sof gra is here let's just see what we have available to query in terms of uh data just here so for instance what we can do we can now for example see and we can trade we can see the trades for this for instance we can query this information all right so this will be instant s graph there's a lot to talk about it there's a lot of great enrichments that you can use um so that's actually pretty good and so the last point which I wanted to go over is you can also very easily migrate uh existing sub graphs um onto Gold Sky and for that you will go ahead and create the deploy command again and just Define a from URL this would be for the host service by default from the graph host service which is an out set it so it's better if you use from ipf hash which is the graphs decentralized network but you can also go go ahead and it doesn't have to be from the graph if you're running your own graph node endpoint we can also migrate over it from it so you will just need to define a another option which would be something like um um something like from URL so this will be something like from um let's actually gohe from API see if I can find an example yeah so here for example we are defining that we want to migrate uh this sa graph from this ipf hash on the Ia ipfs Gateway is located over here so this will be another example where you can very easily migrate softw off your existing graph node Endo onto onto Vol and today in this video I'm going to show you how to create miror mirror pipelines um for linear decoded blogs and so before I go ahead and show you how to do this within the the dashboard and the CLI I wanted to just give you um high level overview of the different components that make up the mirror Pipeline and so just basically a mirror pipeline allows you to stream web three data into your data warehouse um now the actual data that you can sync into your data warehouse it's either going to be sub crabs that you have deployed on that specific Network or maybe uh data sets that we have available for you to consume directly and add into your into your database and so the three main components that make up a Mir pipeline are going to be the sources and transforms and syncs and and so sources as I said before this could be sap grabs or this could be our own data sets that we have created and already readly available H in the example here what we're going to do is we're going to choose linear decoded logs and next you will want to Define uh transforms uh so transforms allow you to basically as the name implies right uh process and transforms uh the data that is on the pipel uh in real time and so is quite useful in cases where for example you may want to filter the data um from a specific address uh as an example it would be if you know if unit swap is something you're interested in and you want to uh track uh swaps from Unis swap then you will as well uh add the the Unis swap contract address in here so that the actual um resulting data sets gets filtered on that contract address so transforms are quite powerful and I will talk about that later in a second and then the third part of it is the uh syns and so uh syns basically allow you to uh or would say syns is the destination where the data gets written and so this could be H this could be an S3 bucket it could be a postest instance like I will do in these examples uh it could also be a click house instance um so we support a number of things and this is something you'll be able to see on the documentation um one important thing here is that for us to be able to write data into your uh data warehouse into your sync we will need to have uh the credentials uh on the information needed to to be able to do so and so that information uh about us being able to write to your sync is stored under secrets so it's very important that before you even go ahead and create a pipeline that you create a secret uh for us to be able to create uh you know to add that data into your uh into your sync and also important is that uh for example let's say the we're writing data into your poist DD uh and you create a secret for us to kind of write into your database in the name of a existing user you you should be able to you know um create that user beforehand those credentials beforehand and make sure that that uh that those credentials can be used indeed to write data into your database that they have the correct permissions all right so this is pretty much it sources you can Define one of many transforms are really optional you might as well use transforms or not uh since you can use water many and and so one question I saw in the chat before was um about being able to query this data um the thing is once the data is in your database uh it's really up to you how you actually interact with that data and so typically what we see the most here when with real world implementations is that you know customers will have an API fronting the DB so that the dabs or whatever type of client application will consume that information via VIA an API but so this part over here is really up to up to you on how you would want to architect this all right and so that's actually pretty I think this uh pretty clear there um now how do we go ahead and create pipelines um there are three ways that you can go ahead and create pipelines uh you can do it through the web pipeline Builder uh which I'm going to show you first and then you can also do it through the through the CLI so the web pil Builder uh I would say is the easiest to start with um it's quite nice and so you'll see in a second uh is a little bit more restriced in terms of what you can deploy because it uses defaults for most of the options um and if you want to have more granal control over the actual pipelines so say for example you want to Define transforms of your pipelines or you want to have do multiple syns or multiple databas or multiple sources then you're better off doing this via the CLI with a definition file okay so let's go ahead let's start with the AC uh the part will be with the web pipeline Builder and so how do we go ahead and create uh pipelines using web pipeline Builder uh you will go here go into your dashboard go to pip lamps click on the new pipeline button um select the data source and as I said before here we are going to go ahead and use linear coded Lo so I'm going to choose direct indexing uh right now here um I'm Pres Ed with all the uh multiple data sets that we have available uh there are quite quite a few and it really depends on your use case and you can you may sometimes find it more interesting to consume Ro blocks uh traces logs in our case again we are going to go for the coded locks which is right here so I go ahead and choose the Cod logs and then I will choose the chain we are interested in so I would choose linear um you see here down there that we have some example data from this data set and you can also here uh this configuration file which for now we will ignore but this is the kind of the backbone for the CLI if we want to deploy uh this in the CLI later but for now let's ignore this and let's continue with the UI experience so we already chose the cogs for linear I'll go ahead and create the pipelines here we have some config for the pipeline so uh choose a name for your pipel just make sure that uh it's Unique and you don't have any other pip lamps with that existing name choose a description if you want to basically differentiate this pipeline for others um resource size basically this is the amount of compute and uh memory that you want to allocate for this pipeline uh as you can imagine the more the merrier but it's also in terms of processing but it's also more expensive so uh for this example I start with small U but if you're consuming a ton of information and you want to back field pretty quick uh then you you might want to go for larger options uh again sources we said the code logs for Linea here you can choose U if you want to back field from the very very start block of the chain of this will be the earliest block or if you want to start consuming pipeline from the latest block Edge again here you can uh have some sort of uh little transformation uh you can if you want to add a filter from a specific address you could easily do so here uh by adding it in here but in this case uh I won't be any of that so we just leave this out um I would just go ahead and click next as I said before pretty important uh when you configure a sync which is the next step here um you need to define a secret right and so the secret will basically security represent those credentials that information for us to be able to write into that database um well as you can see here I have already two syns or two secrets for two syns um I'm going to go ahead and choose post I'm going to choose this one uh but if you wanted to create a a secret as you go here you can also do so uh so let's say if you choose pogress you could add here the connection stream the secret name and then you will be making use of this secret for now on to to talk to your database uh as I said before very important make sure that um you know this username and password that we use here uh it's already pre-created on your side and has the correct permissions to be able to write into your into that database or that table um so I'm going to go ahead and choose pogress uh and then here we need to choose the schema where uh the table will be created I'm want to choose by default it will be public I leave it as it is um uh this pipeline will actually create a table on the corresponding schema for for that for you know for that sync for that for this data uh the actual table name is not something you can edit here when you use this um approach via the UI but is something that you will be able to configure uh if you chose the the CLI option with a configuration file uh and this is just to give you a small H so let me just go next here you see here that I'm I'm ready to confirm and deploy and I could just go ahead and deploy this pipel as it is the other options that you can take uh the existing pre or let's say the resulting uh definition file that's been created by this stepbystep uh approach that we've taken with dii and deployed with CLI since you already have here this uh configuration file created for you but we're not going to be doing this for now so I'll just go ahead and and deploy the pipeline now I deploy the pipeline and we're pretty much finished with the process here um right so now here you will see that the status will will change as we as we go um now it's inactive I think this is because I pre-recorded oh you know I pre deployed this before but we should see this changing to to active uh life pretty soon so now what's happening under the hood is that the Pym is being created uh and what the first thing that I will do is that within my um database which is here in neon DB which is the one that's going to use py my secret is going to create a table and it's going to start writing data into that table um so over time we'll see how this that table is been um so that would be the the first step into how to create the pipeline uh using the web UI which pretty simple uh just a little bit restrictive when it comes to certain things within the definition file now the second option is uh via as we said here via the CLI and there are two ways to do it via CLI interactively or from a definition file so let's go ahead and let show you interactively what it would look like um hopefully see I can make this so um first of course make sure you have the Gold Sky CLI um install in this case um just see here in this case I will just go ahead and click Gold Sky I already have it installed of course um as you can see here gives you a nice overview things that you have already deployed Etc and it asks you here interactively if you want to um create a midor pipeline so I can go ahead and initiate this process which is veryy similar to the one we just did on the UI so I will just say yes I will just choose a name of the pipeline um logs for example I will choose the source types to be direct indexing I would choose my chain again which is linear and I will say that I want color logs and I want to process all historical data so as you can see here and I would say no source is the exact same um exact same type of experience that in the UI but only the CLI right so say do you want to filter you want to add a transform for filter data I would say no filtering and then the sync type I would choose pob and because I already have that secret which was created before uh I will also just go ahead and choose this one I can choose the database schema here as well and and this is more configuration options just say yes no more things ET and I will go ahead and create the pipel I just cancel because I don't really have to deploy that pipel anyways so that would be the second way to deploy P which is basically interactively and so the third one would be how to De how to deploy it bya definition file which is a lot more flexible a lot more Gran um one thing that I just noticed here is that as I came back to my to my pipeline the one that I created to created through the web uh through the web Builder I realized that the records have been written into my database which is great great and I can also confirmed when I access my database I can see here that um you know within the public schema a table was created and that it contains this data so so that's actually pretty good news we have a function in pipeline already now as I said before the the last um method to create pipelines which is the most powerful one but it also involves you know some more knowledge of the of the PM itself um it would be through a configuration file uh so pamps are defined by yaml files and again they have their sources transforms and syncs I'll make sure to send over the reference file for you to see what needs to be added into this confli file but basically it's all about that right it's all about choosing your sources that can be not only one source you can choose many uh so for example uh you're not really rest limited to the coded logs here you can also do uh something like roblocks uh within the same pipeline uh then transforms you can as a said before you can write your own logic here to filter on on data um or maybe do things like decoding on the fly as well and so this is uh any SQL uh Flink compatible um queries that you can you can imagine you can also write in here so that would be the transforms and you can add many transforms as well and then since as we saw before this is where data gets written um you can have um you have more control here because you can not only Define the SK scha but also the tables so in many cases you may be writing different tables if you have different entities or different data sets right um so as as you can see here there's a lot more power in doing it this way but it's a little more tricky if anything and so one thing I could do right is every since every Pam is referenced or is um represented by a configuration file I can make use of this existing one from the one that I just deployed via the UI um copy it and deploy a modif version of this pel let see some sort of version two right so I'm going to go ahead and de U download this file and which is now going to be here so if I do something like then um linear decoded logs yeah I have thata access to this here so say for example that in this new version of this new pipeline which is a cop is based on the other one I don't want to have access to this I don't want to have a transaction has I'm not interested on that um not even in this I just want the block number on the other paramet right so this is also valid what the resulting of this is that the newer table that will be or let's say this pipeline will have less information than the previous one essentially so this is just an example so if I go ahead and save what I can do now is that I can deploy this new pipeline um this way so we'll just go Gold Sky pipeline create uh let's just call this let's see what's the name of this one linear coded logs version two um and now let's actually say definition path and then we choose L [Music] logs so with this command now here you would validate the the actual file if everything goes well you would say hey this is good and now you can monitor the status of the P line this way and so this will just give us a nice overview every 10 seconds of how the pipel is going um just like a monitor you can also monitor the any pip lamps even those created with the UI Builder this way as well and you also have access to where the p is located in your dashboard via this link welcome back hope that was helpful and you learned a lot you know we work with tons of exciting projects in the space poly Market R weave Dora hash flow poab just the name a few and we can't wait to see what you build out maybe one day your name is going to be on that list so the next step for you is to create an account and get started take a look at that guide that's attached to our prize page if you haven't already there's a discount code in there to get you access and for easy access here it is to it's eth Global 2024 that'll give you access throughout the whole hackathon so that you can deploy subgraphs and use mirror pipelines but yeah connect with us on Discord come say hi I'm going to be in there in that channel so drop in and you know introduce yourself ask some questions feel free to reach out and of course don't forget to follow us on X to stay up to date on all things Gold Sky but thanks for tuning in excited to see what you build good luck and have fun