Ceramic 🛠 Building Relational Data Composites - Matthew Davis

foreign [Applause] [Music] Global hackathon I'm Matthew Davis I'm here with ceramic the developer Advocate on the team you're probably interested in composed DB or decentralized data as a whole here in web 3. so today I'm going to show you how to like get up and running with compose DB if you've attended e-global events before some of this might be a refresher but we just recently shipped a developer preview that added even more features to compose DB now we can build relations between documents very similar to what we would see in a left join from a regular SQL table so I'm going to show you how to get that up and running I'm going to do a little bit of schema design with you here in graphql and then we'll go ahead and we'll fire up some of the new developer tools like the graphic or the graphql server that we ship along with the command line interface just to help you get up and running with some of the new stuff that we ship so join me over in the editor and we'll get started alright over here in the editor it's just going to be the editor I'm going to kind of keep my face out of it but I'm going to walk you through all of the ex the steps necessary to get up and running with compose DB quickly if you're new as well as to get the latest feature set if you're not new so the the first thing we need to do is install some dependencies so the first dependency we're going to install and we're going to want to make this a global install is the compose DB CLI but very specifically we're going to have this tag at next we want that tag to be in there that's going to give you the latest greatest release candidate now this is a release candidate so it's not suitable for production use and that's very very important to keep in mind the other thing we're going to install is the release candidate for ceramic itself so the ceramic Network um slash CLI at next as well so this will give us the latest and greatest of the ceramic protocol and the latest and greatest of the libraries and necessary tooling needed in order to act with that interact with that protocol so we're going to go ahead and install those by hitting enter and pmpm is going to run through and do its job now I highly recommend that you use pmpm as your package manager it tends to work a lot nicer with the packages that we have out so now that we have that we can continue on with this setup the first thing we're going to need need is to generate a private key this private key is going to help us create a decentralized identifier or a did and to do that we're going to use the compose DB which is one of the things that we just installed so compose DB did colon generate private key now this private key is going to get blown up at the end of the video but I am going to blur it either way but I'm going to show you a couple shortcuts I use in order to make this easy so I'm going to go ahead and hit enter and you'll see now I have a private key although mine's a little blurry for you on purpose you can then use this key in the next step what I find valuable is I'm going to create a file in my workspace right now but you need to be careful when you do this I'm going to call it priv key and I'm going to copy and paste my key up there now do not Commit This to a repository ever not today not tomorrow make sure you save that file this is a private key for a reason so right now this is a bad practice but it's a little bit helpful for development so in your git ignore file or whatever make sure you're ignoring this private key all right then the next command we're going to need now that we have a private key is we actually need to take that private key and create a did from it as you can see here's a private key I used in the past this is not the one that just got generated so this command takes an argument you could just paste your private key in right there like we just saw my my terminal auto field for me um but you could also use a little command line Kung Fu here which is why I stored it in a file and it's a whole lot less to keep track of I don't have to constantly go back and copy and paste this private key over and over again but you'll see now I have a did this is a whole lot less uh secret but still you know don't want to let this get into the wrong hands this is what's going to be your controller for data so make sure you protect this as well I'm actually going to go ahead and copy mine and store it in a file as well which you don't need to do for this demo I'm going to name my did key I'm just going to paste it in there you don't need to do that for this demo it might make it easier for you to work with things though as you're doing some development work but again do not commit this to a repository super Danger Don't Commit This to a repository all right the next thing we're going to do is we need to set an environment variable you'll see it's ceramic enabled experimental compose DB this environment variable basically allows us to use the new features that ship with compose DB now here's the thing is this environment variable needs to be set in the terminal that you launch the ceramic Daemon from right so if you set this in some other terminal session and then launch the Daemon it's not going to be able to see that so it's very important to know so we can go ahead and we can start the ceramic Daemon now by typing ceramic Daemon this will make sure that we get configuration files along the way you'll see it's it's now running which is great and I have some previous models and things in here so it's actually indexing these we'll end up removing them as we go forward so ignore this if you see this I'm going to go ahead and open a new terminal now I'll rename this one just to keep it simple so that we can see uh what's happening ceramic Damon cool and over here we're going to go back to being able to run command so this is going to be a super useful beneficial spot for us so ceramic's still running okay that's important to know uh and now I have a new terminal in a new workspace so for the Sega organization I'm going to go ahead and I'm going to create a new folder called schemas and in that folder I'm going to create a new in schema for an address book I thought it'd be really cool if we could start building an address book so I'm going to show you how to do the world's most basic address book but I'm going to go ahead and create a file in here called addressbook.graphql and now I'm able to work on the schema using graphql's schema language for my address book so instead of typing this all out all crazy like I'm just going to go ahead and copy and paste it in here from a previous run of this but I'm going to go ahead and say that this is a demo address book so I'm going to change this just a little bit so that we can see it so a simple demo address book and right here basically what we're doing is we're naming the schema we're running a directive on it that's going to go ahead and create a model for us and in there it kind of has some metadata about where it's going to live basically we're saying any did can have more than one of these so you could create more than one address book if you wanted for a given identifier and then we've given it a simple description which is helpful for us when we go to search for it if we want to reuse it somewhere and here it's uh just the name of the address book so you might want to have a friend's address book or a co-worker's address book or whatever it's just got a property of of address book name with a string and I decided that 100 characters is probably enough for you to to you know name it um so once we have that in place and that file is saved we can now go ahead and use the compose dbcli in order to create um a composite from this right and which is how compose DB works so we'll do compose DB and that's going to be composite create and as you can see done this just a few times here we're going to say that's in schemas in its address book right and then I need an output file so I'm going to say output and that output file is going to be eth address book really just for the hackathon and that's a Json file right so we're going to want to make sure that we're we're including that file extension and then the last thing we're going to pass in is the did private key and this is where our command line Kung Fu can come in again for us and we can just cat that private key file okay and this is going to create a composite for us you'll notice that we we now have a new file up here in the left hand side and it is uh basically a big Json representation of our graphql schema we can hang on to this for later we're not going to need it right now at the given moment now that we have our address book set up what good is an address book unless we add entries to said address book so I'm going to come over here and we're going to create yet another graphql schema and I'm going to call it address book entries Dot graphql and in here is where we're going to start outlining the shape of an entry for an address book but the first thing I'm going to do is I'm going to grab the previous graphql schema that we have so I want to get this demo address book into this file so I'm going to go ahead and do that we need to add demo here but you'll notice now the directive being used is a load model whereas when we developed this we said create model so what we're telling this portion of the schema to do is hit like say hey load model and then we need to pass it an ID this is an incorrect ID for what we're using so the question becomes well how do I get that ID because without the ID I can't actually use this at all the way we get that ID is again using the compose DB CLI we're going to go grab our Composites and we want to see the models in the address book.json which is incorrect for us because we actually named ours something different so this is the output file so it's going to be eth address book dot Json and you'll see here we get a field that says ID it's uh right on here it's kind of tough to highlight but it it begins with kjz and it's just this random string so we're going to copy that and we're going to paste that up here into our schema itself so now we're off to a good start we are loading that previous model so we're reusing that shape so for any reason the address book itself changes more Fields get added to this they will become available into these models once you rebuild the Composites it's not going to happen unless you rebuild the Composites but it's less to actually type in so the next thing we need to do is let's use that let's create our entry because right now all we did was load that demo address book model so we're going to name this a demo address book entry now this demo entry has a create model directive and just like before it's a list because many entries can belong to an address book on this account right and it's a simple uh demo address book entry and in this case we're going to do a little magic here in order to make this work so the first thing is we're going to say the address book ID I just made that up it could be anything we could call it the the address book we can drop ID all together and that's fine and matter of fact let's do that I think it makes it a little more readable um has a type on it a data type called a stream ID and if you use the document reference and point to the model we just created so the demo address book which we we have right here on line you know one two three you're going to say hey like go get the reference for this document so basically we want all the key value pairs that exist inside of the address book go get those and bring them into this shape right here and storm is something called address book and this is great it would allow us to get just the stream ID which is basically this crazy string up here the kjz L6 whatever they would store that here which is fine if all you need is a mapping to an ID but if you want direct access to all the key value pairs within that shape you need to do this line here which is create a new property in this case I'm going to call it a book rather than than book ID we're just going to call it book um and the address book is going to contain both the ID as well as the key value pairs that exist so we're going to say hey that's a demo address book that's the data type right it's a demo address book and what we really care about is extract all the properties from this model so this address book string is is mapped to this this property here and then we can add the custom fields for an entry which in this case the entry name so like I don't know Matt Davis right like and then my wallet address and wallet addresses you'll notice is also its own shape so let's go ahead and bring in a wallet address now this is called an embedded type notice how it does not have a directive up here we have like create model up here we have load model but for wallet addresses there is no directive so this type is embedded meaning these two things are in the same file this just allows it to be defined and a little bit more readable way but a wallet address it contains an address which is a string and contains a network which is a string so we can type ethereum we can type Solana we can type whatever but basically a list of addresses can exist inside of this entry because if I have a friend maybe I want you know to add all of their wallet addresses I could do such a thing this is our schema for an entry so let's go ahead and create a composite from it to create a composite from it we're going to use compose DB composite create and then that's going to be schemas and that for us is an address book entry this time address book entries dot graphql we need to set an output file and that's going to equal if address book entries you could name this whatever you want dot Json and that's going to then need a private key to work so did private key and back to our command line Kung Fu you can now see why I I did this I don't have to keep track of and copy and paste that did but you'll see now over here we just got a new file generated which is address book entries it's just like the other Json file we had only now we we have build time representations of each of these Composites that's nice but now we kind of need to put them together and in order to put them together we are going to use the compose dbcli to merge them so let's take a look at that next to merge Composites we are going to use the Json files so we're going to do compose DB composite merge and then we are going to specify ethaddressbook.json and each address book entries.

json and lastly we're going to supply an output file which is going to be our demo address book composite or let's actually call it merged .json you'll notice there was no need for a private key this time and now we have another file here which is the merged version of our composite so it contains everything we need to know everything ceramic needs to know all in one file for these two models to work together now that we have them together let's deploy them over to our ceramic node we haven't done anything to our ceramic node yet so let's go ahead and deploy them over and just like before compose DB is going to be the tool for that it's going to be composite deploy and then we're going to say demo addresses merge.json and again no private key necessary for this but what is necessary is that we capture the output so don't capture the brackets but each of these is the stream ID for the model or the Composites that we have deployed for indexing to take place we need to copy these because we need to now edit our ceramic configuration in order to tell our ceramic node to index these models and pay attention to them so in order to change the configuration I needed to move back into my home directory by default the ceramic Daemon will create the config files in your home directory there is a folder called ceramic and it's hidden so it's dot ceramic and if we move into that you'll find a file called Daemon config.json so let's actually go ahead and open that config file here in vs code gaming config.json and if you scroll down into this file you'll see that I already have some some models in here so I'm just going to add mine here to the end and we'll save that so I'm going to add mine here to the end and now my node is indexing those stream IDs which is really nice but it's not really doing much at the moment so let's go ahead and change that let me go ahead and delete this terminal we won't be needing it all right now that our our configuration file has indexing ready to go we need to come back over to where we were running the ceramic Daemon I'm going to hit Ctrl C to stop its operation and I'm just going to go ahead and restart it here and let's just make sure it starts up effectively you'll see now we have a whole lot more indexing taking place so we're going to leave this running and it's fully configured to pay attention to our new Composites and now we can move on with actually interacting with these newly created Composites that we have it's important to note that everything we've done up to this point has been what we'll call a build time step so this was just preparing the data in our schemas for our applications to use and for us to interact with we're almost done with that we actually need to take a step now to go from build time to runtime and we do that using compose DB and we're going to say composite compile and what we want to compile the composite file this merge file that we have and we're going to turn that into a runtime file so for ours it's demo addresses and we're going to say the output is demo runtime.

json and you'll see we now get another file here this demo runtime and it looks very similar but it just has a little bit of different metadata into it and and helps take care of the graphql scalers and things that we need right so we now we have a runtime representation that we need to use this is what we would pull into our application we would say like hey read this file this contains all the data you need to access the other files that we've created along the way and that's exactly what we're going to do here we're actually going to start a server called graphical which ships with compose DB but we need to pass this file to it so that it knows what to look for so in order to do that compose DB I'm going to say graphql server and that's going to be graphical a lot of this is going to be very very similar we're going to put it on a port I'm going to put it on a different port for me I had some some trouble with Port 5000 if for some reason you're running something on Port 5000 you can change this it's not going to ruin anything by you doing so and then we point it to the runtime file which for us is demo runtime and lastly we are going to do a did private key like we have been and we can continue using our command line Kung Fu in order to make this easy for ourselves now we need to pass in the private key so that graphical uses our did without doing that we uh we won't be able to write data down to ceramic the ID is the controller for our data stream so meet me over in the browser let's go ahead and take a look at this thing you'll see in order to get there we're going to copy and paste this URL that just came through in the output and uh it'll take us to a graphical Explorer okay you should see something very similar to this if you have followed along to this this far and uh you've opened that graphical Explorer in your browser so now we can actually start making queries for things I think the easiest thing to do is is actually uh query to see if we have the demo address book all right so we can write a very simple graphql query you'll see that demo address book is is what we're calling on and I'll go ahead and I'll write this query right here for you just so you can see how awesome graphic Killers with this if for any reason you don't know what to type you can just type the word demo because we know that was in our address book and you'll see that we get the demo address book entry index so right now we're concerned with the address book so let's go ahead and grab that and let's go ahead and say we want the first five of these things okay so show me the first five demo address books that are attached to this account because remember we said you could have more than one and now it's just a matter of standard graphql to make this query so I'm going to say edges nodes and then we can pull the ID which is automatically generated for us when we create in an address book um no so that's why you didn't see that field in the schema but what you did see in the schema was the address book name field so if we come back into this and we look at our address book you'll see we have this address book name field and that's where I'm getting this from so we can go ahead and run this query another cool thing with graphical is if you hover over this you can get some documentation about your types you can come back here you can look at the schema itself you can look at different queries you'll see we have the the demo address book index all this stuff is nice which is kind of built into graphical well right now we don't have anything it did return a value but it's empty right because we haven't made one of these so let's go ahead and actually make one of these and to do that we'll call it a mutation and the mutation is going to be called create new demo address book I know that's a mouthful and we're just going to go ahead and pass in a variable for input and that's going to be a type of create demo address book input the data type is the exact name of your mutation plus the word input right so that like that's what you need to pass and that needs a fancy exclamation point um once you have that we can now call one of the methods that were created for us this we typically get a create method that's automatically created when we do the runtime part of the composite so we can say we start typing create and you'll see we get create demo address book autofills for us we're going to pass it some input and that is going to be the value of I which is up here right we wrote that on line one and from there we can then auto fill all of this and say hey the document go ahead and return a CID the address book name after this runs but now we want to actually like fill in this data and to do so down here in the bottom left hand corner you can open this query variables Tab and you can paste in an object so since I is the variable that we need we're going to go ahead and paste I and then we're going to say the content of I is the address book name is um we're going to name this address book eth Bogota right that's going to be the name of our address book notice how we're not passing ID that's auto-generated for us and if we go ahead and hit the play button you'll see we get a document created for us so now we have an address book it has an ID of this a stream idea this and it has a name of that so if we were to go ahead and run that query as before like we had so we can go ahead and we can delete this and we can come up here we'll say query that query is going to be demo address book index show me the first five of these things and and then with the edges nodes and really I just want the address book name I'm not really hard-pressed about the ID right now and you'll see we do in fact get an address book returned let's go ahead and create an entry so just like before we named our mutation we could have named this anything we wanted to we grabbed the automatically generated scalar for us which is demo address book entry input right and that's because we have demo address book entry as the name of the shape so if we come over to this uh and we look at our schema it's demo address book industry right and that's how we're getting that so now that we have that all we need to do is fill in our variables that match the shape so I'm going to go ahead and paste these in just to be a little bit quick here because this video is getting long um we did need the ID from the pre the address book that we want to put this in so let's go ahead and run a quick query to get that now we went back to the query so we're just going to go ahead and query the one we're going to copy this and we're going to replace this string down here and now we can put our other query back in okay so you'll notice that the content is now address book entry name and an object containing wallet addresses right and that's because that was the shape that we had outlined in our schema file over here for a demo address book entry you'll see there's address book there's entry name there's wallet addresses right each address has an address and a blockchain network associated with it so that's all we have going on down here right here's we wrote These in they're strings a demo entry we'll call This And if we go ahead and do that now we're interacting with this composite but where it gets really cool is when we when we query this so let me show you that and we'll wrap this video up so this query gets a little bit more complex we're going to go ahead and grab the entry which is the key here the address book entry show me the first five of them and in here I'm specifying book and I get to do this because over in our schema we extracted that address book object itself into a field called book and this is the real power of relations at the current moment so we've extracted that into books so I can get the name of the address book that this entry belongs to as well as the ID to that address book and I'm actually going to come down here and I'm going to add ID so this bottom ID is going to be the ID of the entry which we we see here and the top idea is going to be the ID of the address book that it belongs to and all the rest of this should just show us very similar to what we already have so you can see this is the address book object and it's been embedded into our schema really simply by just loading it in and and then specifying these two lines here to reference that document and then create a relation to that document so now we can expand that entire document as a part of this object and we get enhanced composability the rest of this is just data from the entry itself so that wraps up getting started with relations if you guys are here live in Bogota look me up grab me on Discord names right here you can go ahead and ping me on Discord let's link up I'd love to talk to you about your projects and what you're working on I look forward to meeting all of you keep your eye out for the prizes there is 5 000 the equivalent of five thousand dollars in cryptocurrency available for uh for grabs for prizes for the projects that you build so happy building hit us up in the Discord if you need help check out our Forum all the typical spots and but most importantly try to link up with us we got a bunch of team members here in Bogota right now so hit us up on Discord try to link up with us we would love to meet you and uh Happy hacking