Ceramic 🛠 Composable Data Using Ceramic

[Applause] [Music] hey everyone good morning good afternoon good evening depending on where you are we have matthew here from ceramic doing a workshop on composable data using ceramic questions will go into the zoom chat if you have any uh and matthew will answer them all at the end and with that being said let's get started hi good morning good evening good afternoon wherever you might be i'm matthew and we're going to talk about composable data using ceramic hopefully you can see my wonderful screen according to the test that it worked so uh if it's still working then we're in business um some of you may have heard of ceramic before some of you may not and that's okay we're going through a lot of changes right now so this isn't going to be a code based demo uh but the good news is we're going through a lot of changes right now and there's plenty of code-based demos to come and it's going to make using our stack so much easier um in the future so i'm excited to kind of talk about that a little bit at the end and a little bit throughout so who are we if you're new to our project we have a team that's built from extensive enterprise level web 2 folks as well as pioneering web 3 people that have written core web3 standards specifically around like identity solutions and things like that all of us have come together to work on sovereign data and getting data back into the hands of the users rather than uh with these big companies and stuff so that's what our passion is um and we come with a lot of experience to the table here so pretty solid team working on this the big question if you're new is what is ceramic and it's easy to kind of break this down into one sentence which is decentralized network for composable data but that leaves a lot of ambiguity so i like to tear this apart into two sections first we have the decentralized network and then we have the composable data but it's easier to talk about them if we switch them i think everybody understands the data a little little easier so we're going to start by talking about composable data excuse me and then we'll go into talking about like what we mean by the decentralized network and how our network might differ from like an ethereum blockchain or what you're used to so the easiest way to dive into composable data in my head is to take you through this user's journey we've probably all been here way too many times uh where you're going to sign up for a web 2 service so i have twitter's example instagram and facebook pulled up right now uh you might notice some similarities in these forms like for example twitter asks for your name and phone number and then i'm pretty sure facebook is asking for your name and either a phone number or an email but a lot of these fields become the same thing and we can see that here where we're repeating this data entry every time we go to sign up for one of these new web 2 services that we want to join there's a lot of problems with this one is your time is being taken up because you're filling out the same information over and over and over again and sure you might have that stored in a browser but what's the browser doing with your autofill form i don't i don't think i know i don't know how many people know exactly what that browser is doing on the back end and that leads us to the second problem is once we fill this in we're kind of giving that to uh to the service itself so we're saying hey i gave you my name i gave you my phone number i gave you my email i gave you all this information about me um even the people that are in my social circle or whatever it might be and you're going to go store that away on your own application specific database now once that happens i now have to trust you i have to trust that you're going to do the right things with my data that your database isn't going to be breached and my data isn't going to be compromised i have to also trust that when i ask to see my data like for example i want to see my friends list that you're actually showing me the data as i entered it not as you altered it and that brings us to the third problem is as a developer you're often building a one-off api integration through one of these services and that's fine they're usually well documented but what happens when one of these services decides to close off their api now you can no longer access your data in the way that your application might need to because you no longer have access to what's living on their database all of these three problems lead to the same uh big theme which is you have no agency over the data that you're putting into web 2 apps and i don't think this is news to anybody but this is what we're aiming to solve with composable data so let's look at how ceramic approaches this instead of asking you to fill in your name your birthday your phone number 10 different times we give you the ability to create a data object um with this at this stage we're calling them data models in just a couple weeks we're going to be calling them composites and i'm going to talk about both a little bit because data models are not going away you're still going to be able to use data models composites are a little easier to use so data models rely on json schemas and a rest api composites are going to be more of the graphql query system so you could plug in any graphql client and create and query data composites so it's uh one of the improvements that we're making over this they both kind of point back to the same thing and that's you define the data one time in one location and then all these other applications can just globally read from that data so for example a developer would sit down and say hey my application needs data that has this kind of schema to it let me go ahead and create a model for that or a composite for that and now if the the second project's web 2 or web3 app decides that that data is valuable to them they can just read from that and the user gets the ability to update that wherever they want so our user sits down and says oh here's a schema for me that that holds my profile let me just fill this in and now every single application that leverages that profile can read from it so this makes it to where we have to fill stuff out one time it's also tied into your identity a little bit so you have the ability to change it to delete it to update it to whatever you want to do with it and it's 100 up to you the user and these applications can just access it because it's also important to point out this is globally readable uh so you don't want to store something like a payment information without encryption ceramic doesn't do encryption on its own at the moment um and we're actually relying on other protocols like lit protocol for you to do the encryption on your own and you could store it that way but we don't recommend storing super super sensitive things at the moment we can dive into this composability piece a little deeper so if we think about that basic profile data model or composite it actually on the network it's a ceramic stream and streams are just what we call the the data itself um it's not like streaming data that you might be used to if you're using some sort of like big data solution or something like that we're just calling streams the the objects if you're familiar with mongodb you're probably used to what a ceramic stream looks like our most common stream type happens to to be very similar to a mongodb document it's called a tile document for now uh our new common stream type is going to be called a model instance document most likely we're still kind of working on naming so it's important to know like we filled out that basic profile cool it gets stored as a stream on the ceramic network and a stream is just a list of commits so we're not actually storing the the data itself we're storing the changes to the data that gets made so uh and then we can like kind of reduce them down and show you the the aggregate view of that data so when you go to change something it's um it's let's say i change my name i'm only changing that name field and what's happening is just like when you use git we're just tracking the commit for that each commit is an ipld object right and it has its content identifier and that's how we access that commit and then stream types i've said stream type is like a tile document and i even said like model instance document stream types are just these these bigger kind of uh object or bigger definitions for like how and what we can do to a particular stream so maybe some streams aren't that mutable and that's where the logic lives is in the type of stream whereas other streams might be highly mutable so they have different logic that comes with them and so we identify them as different stream types you as a developer you're going to identify as a different stream type when you when you code it out too and then these streams get blocked or get anchored to a blockchain but they don't at the same time the streams themselves never make it down to the chain it's all the the commit ids and the ipld information that gets uh written down to blockchain there so it is something that we can look at and say hey like here's proof that the data is what it is we we can't argue the fact that this has commits that do get anchored down to chain um but we're not storing the data down there just to make it kind of quick to do our reading rights so that's kind of the gist behind a stream type and that might seem a little uh worrisome if it's globally readable how do we make sure that uh it's gonna be accurate yes we have the blockchain but how do i stop somebody from coming and changing my data and we do that through authentication which you might have guessed by this slide currently we have three major authentication methods if you're been using ceramic you've probably used the three id uh package that we have this is like an in browser iframe thing that pops up and asks you to connect your your three id to your metamask wallet or whatever wallet you choose that is the most common one that that we use and it it leverages the tile document stream type on the back end which is again a stream that we can change it's mutable the second most common way to authenticate is to use key did and a lot of times this is build time so you can kind of separate these in your head with three id being run time and then um sorry the chat just took me out for three id i'll get to with three idv and runtime and key id being build time so as a developer you might sit down and say hey like i need to define like the schema or the data object itself i'm going to do that with a developer id that i have but it's not limited to that let's pretend that you create a service that's kind of like an autonomous service and it chooses to store its data on ceramic it's not going to have a wallet to interact with so you could then give a key did to a service and have it interact with the ceramic network the newer and nicer way that we've we've built things out are to eventually replace three id with cacao which is uh chain agnostic capabilities objects it's the more preferred way of doing it right now we're leveraging the sign in with ethereum aspect to to allow this to happen um but basically what what we get is sessions out of this so a user can log into the app they say hey connect my wallet and then we're we're kind of giving the application permission to act on behalf of us for a set session time and from there we can um until the session times out right the application can do a bunch of stuff on behalf of the user what this does is it eliminates you having to sign every single message every single time you try to do the smallest thing on the network all right so the big picture of this is if we look at our developer or our person in purple here they're going to get a did it's probably that key the id and that's where they build the definition and the schema from there our users our yellow and our green person there they're probably using like three id connect or sign in with ethereum or sign in with x we're currently working on specs to expand that and when they log into your app they're creating data that matches that schema and writing it as an individual record that record stays with their identifier you'll see here that user two can't change anything from user one it's not writable because in order to write or update data you have to use the same did which is decentralized identifier in order to do that so that identifier becomes the controller for the data the only time that that could maybe not be true is if you're using something like cacao and you're letting the application act on your behalf and even in that scenario it still is using your it's using a hash of your did in order to to perform actions so each data stream like we talked about earlier stream of data not not streaming data but data stream ceramic data stream has to have a controller and that controller ends up being a did cool what we get out of this is user agency you have full control over your data it is tied to your decentralized identifier um nobody else can write to or control that data however it is globally readable right so um unless you encrypt it and then understand how to to decrypt it once it gets to your application it's globally readable the big deal with that is if you start making a bunch or start storing a bunch of data that's very um encrypted then you're limiting other web3 applications from leveraging the fact that they don't have to build out an entire social profile schema there's already a social profile schema you could just call that into your application and get that user's information so be careful with what you encrypt is i guess is really where i'm at like if you're building some sort of financial application and you want to encrypt transactions or whatever that makes sense but if you're adding layers of encryption to data that makes sense to be public you're making it really hard for other applications to use the data schemas that you come up with we won't be able to use your composites we won't be able to use your data models and that completely defeats the purpose of the composable data composable data is all these hundreds of apps can just read from one type of composite or one type of data model or many of them to compose what that view is for the user so be careful if you're going to go down that encryption avenue that covers like the data part um hopefully i'm not frozen okay cool not frozen let's jump into the network there's always a lot of questions and fun to come from the network uh so we're probably all pretty familiar with the traditional blockchain right like we get like the single global execution environment and what that does is it communicates this state out to all these different nodes and this is also why sometimes these things are slow right is we're trying we have like one entry point to get the current state of things and we aim to solve that by providing ceramic nodes so the nodes back here are not ceramic nodes ceramic nodes are something different that was just like hey we're running different nodes that plug into the blockchain a ceramic node is a small piece of software that you run you can run it on a laptop locally you could deploy it in a cloud if you want to you want to put it over there wherever it doesn't take a whole lot to use we actually provide nodes for you to get up and started with we have a developer test net we encourage you to use it you guys can discover each other's stuff if you're using testnet but ultimately a node provides a single individual execution environment so unlike a traditional blockchain there is no giant global ledger every single node is responsible for validating transactions that it knows about for the streams it knows about uh this allows us to say oh things are getting a little slow plug in a new node we can scale horizontal horizontally for this and that allows us to increase the number of streams and stream transactions that each node can look at at any given time these nodes take an offline first approach so the way that works is a node cares about what it cares about so if i'm building an app and i say hey i'm going to run a node or i'm going to put this on testnet or whatever those nodes are going to say there's a new app there's a new stream type out there or a tile document or whatever there's a new data stream let me make sure i care about that thing and now that it cares about it it begins running that individual execution environment and if node b comes along and it doesn't know about that that's okay it doesn't need to but it could if the developer needed it to communicate over to that other node and then actually care about it and start worrying about checking the the the commits and and doing the offline first approach stuff validating those transactions so what this means is let's say you're on an airplane some of us fly on occasion and we're doing some development work and you're running a node locally that's fine and when you land though you want to reconnect that node to the bigger network and then boom uh it can replicate over so you can work offline and when your node comes back online it will then populate the rest of the nodes that care about it your your streams and with the the current data right with the current state this works because of this global namespace so all nodes kind of live within a global namespace and this allows uh them to reference other stream types even if they're not specifically pinning is what we call it pinning that stream to their their node themselves right um i see a question about best fit application i'll answer that at the end that's a super good question we're almost at the at the end of this year so more things about nodes is is this is what they're responsible for and these are the characteristics right every node cares about its own storage it cares about tracking its own commit log back up that stream state storage is that that pinning thing that i talked about that's saying i'm a node and i'm going to care about this stream or these mini streams one node can care about thousands of streams or one node can care about one stream um so they're responsible for their own commit log storage this gives us that offline approach they're responsible for persistent connectors this is something that we're working on it's on our roadmap which i'll show you at the end um it's on a roadmap this will allow you to say hey i need to backup the data on my node like how do i do that well you can plug in many solutions whether that's some sort of file coin solution or r weave solution or whatever it might be this is something we're working on to give developers their choice instead of saying you have to use this one specific uh solution so um nodes are also responsible for responding to queries like hey i don't have this stream can you find this stream type for me and those are the big pieces uh in order to answer those queries we broadcast right the node is responsible for broadcasting changes and commits and other transaction stuff to the other nodes the last thing that's not really like a responsibility but it's more of a characteristic is that nodes can exist on multiple networks and you're going to need to make sure that if a note is on testnet um and you're trying to read something from mainnet you can't do that those nodes can't communicate they're on different networks right so if you're having a hard time getting data to and from your node or from another node chances are you're on the wrong network so there's a couple major networks the first one is a local node easiest thing to get up and running really quick well second easiest you just npm install the ceramic package run it and now you can do your local development the actual easiest thing to use is testnet you just enter a url and you're good to go you don't have to install anything then we have our development network where we test our new features highly unstable don't recommend using that one and then that which we do have a process of getting on mainnet we'd like to see a project a lot more mature before we do move it over to mainnet so if you were gonna start today what should you do well you should create a node uh or you should just use the testnet node sorry don't create a node uh use testnet right that's where all the other applications are building that's where you're going to be able to get the access to their data so the read-only data remember so to get started we have three things to help you i'm going to speed through this because these good questions and i want to make sure we get to them um first one is self id this is a high level abstraction if you're a react developer you're probably really going to love self id there's a ton of react hooks that will make managing data on ceramics super easy glaze is a little lower on the stack but it gives you more development control if you want to create a schema or a data model or composite you're going to end up using glaze to do that and then ceramic is the lowest level of this this is like hey i'm running a node or i'm interacting with the protocol at a much much lower level most of the time you're probably not going to be digging into the ceramic stuff unless you're doing something super unique but it is out there we can kind of see there's a raw http client in that ceramic package if you want to manage tile documents or other stream types directly you would use that ceramic package but this uh the did data store from glaze makes managing those packages or those stream types a hundred times easier um and it helps us use the the composability uh a little easier as well and then self id pulls it up that's where that three id connect stuff is and really a lot of our our react stuff so if you don't really need a lot of control and you're doing a lot of read only stuff you probably won't just want to hang out in that self id area if you're doing that read write stuff you're going to want to be in the in the glaze area and if you're doing something super low level you're going to end up over on the ceramic side all this is kind of documented as you might have guessed we do have our documentation a major major thing is the roadmap this is a new thing for us you can find this on our community discord so if you join our community discord you'll be able to click this link and you'll see the roadmap so you can see exactly what we're working on it's public facing nothing's hidden and your input helps steer that road map to some degree so don't hesitate to end up there and then the last call out here is if you do want to get started with this uh basic profile and and maybe building yours and then getting ready to go and have that populate in different applications this profile app self.id you can go there connect your wallet and create your profile and then you can like kind of play with that as you do some development along the way um last little piece i'll leave this here for a second just in case some of you guys want to scan this um there's also a link here to our website we are hiring positions change all the time feel free to apply we'd love to have you join the team um you're obviously here so you're you're enthused and that's uh the number one thing we look for is enthusiasm specifically around sovereign data so um i'll leave that for just two more seconds while i answer these questions i'm gonna scroll up here uh where do we store data if not on the blockchain yeah data gets stored directly on the node i think um that was asked before we got to the node responsibilities thing um every node stores its its uh stream state so that's where the data is um and with some of the changes that are coming into play like one of them is indexing which is going to be out here pretty soon uh there's the node is now also going to run a small database to keep track of the indexing stuff but the data itself does get stored on the node and it is stored uh all through ipfs that's that's the correct answer there right so we sit on top of ipfs ipfs is here and then everything that ipfs sits on top of can be any user-defined storage can you read data from another did yes um yes and yes you could make your own user-defined storage uh if whatever your object needs to look like for your application to work you can stuff pretty much anything in there there are some limitations if you're storing video it's not going to work well right so like large uh pieces of like binary data don't don't fit really well but lots of if you could if you could see it over like a rest payload or a graphql kind of payload then that's the type of data that we're really uh supporting more than any other type so um let's see can we read data from another did absolutely if you know the did right and so one of the ways that this has been handled in the past which we're hoping to solve with composites and this indexing service that's coming out uh applications would use three id you sign in you do all your stuff and then they ran their own indexer that kept track of dids so that they could you know re-reference data as they needed to orbis is a good example of this uh orbis is one of the most fully fleshed out applications we have that uses ceramic it's a it's a twitter clone so all the posts and things like that and and what you've written are all tied to a did and then they do have a small indexer so you can read data from another did if you have it the bigger problem is if you have the did then what stops you from from writing data so there's there's a better way to do it you wouldn't really keep track of the did um directly right uh so there's that each stream gets a stream id and you can read data from any stream as long as you have a stream id so that's the safer way to do it would be to say hey i have an app um the idea is doing all the controlling great i'm running an indexer or i have some sort of index service that's available to me which ours will be soon and i can keep track of stream ids so i can globally read everything because i have the stream id and that's a read-only thing you can read it if it's not encrypted yep it's globally readable you just can't write or update exactly you need a did to write or update um is there any way to encrypt and decrypt data documents stored on ceramic we recommend lit protocol for right now that's been the most commonly used encryption layer for this uh best fit applications for ceramic we're really focused right now on things like social um we think that data makes a lot of sense because of how it's typically shaped so social applications um credential based applications like like some sort and that well maybe create i don't want to tell you what to build but um like reputation and verified credential type applications are are helpful um we just did a major integration with with git coin their passport feature they're leveraging ceramic quite a bit um so things along those lines um not necessarily financial transactions if you're building a financial app it could go either way like we've seen an integration with superfluid using ceramic but what they are storing is more of an address book so that's more of like the social side of a financial application is there gas fees for did transactions if you're using test net right now the answer to that is no and there will not be gas fees at the moment on test net i don't know what the the long term plans are for that right now even if you're on main net i'm pretty sure we're the ones covering the gas as it comes to that um any timeline for oh caps look at the the road map would be the number one place i could point you as engineering changes their priorities that probably changes so if that's not on there we'd love to have you raise a concern on the community discord so we can get it on there if data is stored on ipfs who's paying for the pinning you know file coin again we're we're taking care of all the the transaction fees at the moment especially on test net um if you're doing it local uh so if you have a local node then your local node also comes with ipfs so it's all just local and it's not wholly discoverable like it would be on testnet data storage can those be pointed towards storage like s3 r weave or not ipos not at the moment again that's a coming soon feature if you remember we looked at uh so that i don't know if i read the question loud enough it was data storage of nodes and can they be pointed to our weave or s3 um that's a upcoming feature that was in our our thing so i'm guessing this was ask prior who do we reach out after the hackathon if we want to talk to someone and how you can use ceramic me hi i'm matt um you can reach out to me on discord in the ceramic discord server honestly the whole team is there but you can reach out to me i'm more than happy to help you we also have a forum um and if you start asking a lot of super technical questions i'll probably point you over there it's just a longer lived easier thing for us to track so feel free to reach out to me i'm more than happy to circle back and help you um i've answered pinning um as an app developer how do i discover the stream ids used that that is a struggle that we are currently dealing with um and that's some of the indexing that we have planned for the very near future check that road map you'll see how near it is it's right around the corner uh this has been a long term thing that we've tried to solve is how do i discover those stream id's um the answer is we're good that's what that small database is gonna come along and help with hopefully um other applications have built in their small indexing service to keep track of stream ids that's one way you could do it for now check the roadmap though that's a big point that we care about deeply that is uh that is coming to light self idea is a little bit not easy for me is there other did services based on ceramic if you're struggling with self id i feel your pain um when i build stuff using ceramic i don't typically point to self id directly i like using the glaze set of things check out the did data store i think you're going to find it a whole lot easier to use from the did perspective plus very recently let's say within the last three to four weeks we have uh fully released the the sign in with the ethereum part part of the glaze stack which makes not using self id and not using three id that pain point kind of they go away it's a whole lot easier to use the sign in with ethereum flow and then the the the glaze sweet sorry i keep saying the id in my head the glaze sweet so i feel your pain i try that and reach out to me on discord i'm more than happy to help you get that up and running would like to discuss on building an instagram s yeah we'd love to discuss that too again join our community forum i'll put that up one more time here for you join the forum please bring those discussions over there we would love to have these these talks with you we do have an entire like partnership person manager business partnership manager that will take care of those type of relationships um i can get you in touch with her she's awesome does a wonderful job um you just gotta reach out to me on discord and i'll be more than happy to take a look at your project and make that connection for you so would you recommend start with glaze i recommend look at everything uh look at everything and see what you find easiest i find clay's easiest there are plenty apps that use self id um i come from a different type of programming background so the self id stuff is a little bit uh too high level for me i like to have a little bit more control and not have so many opinions in there and self-id is the most opinionated piece so check out the glaze stuff again reach out to us in the forum these are great conversations that we can go deeper in over there i think we're a little over time so um yeah so we are at time um i know there's one more question in the chat but feel free to ask that in the sponsor ceramic discord channel uh matthew and the rest of the ceramic team will be available and like we'll be more than willing to answer that um but with that being said thank you so much matthew for taking the time to do this workshop for hack fest i was super useful i think a lot of people are really interested in ceramic which is really great um yeah so once again uh everyone is available on the sponsor ceramic discord channel so feel free to reach out there um also matthew did put up all of the other ways to contact the team if you want to uh yeah and thanks to the rest of you for tuning into this workshop we do have another one after this one so hope to see all of you there as well have a great rest of your day or evening or morning depending on where you are thanks everyone you