Standards of Encrypted and Mutable Data on IPFS - Joel Thorstensson

Hey. Yeah, I am. Fancy, fancy, fancy seeing you here again. So, we're going to go into standards of encryption and mutable data on IPFS. Cool.

So, you ready? Yep. Awesome. All right, take it away. Cool.

Well, let's see. All right, you guys see my screen. Cool. So, yeah, I'm Joel and I'm going to be talking about DAG-JOSE. Um, and actually it's it's kind of interesting or funny.

Um, it was about 1 year ago that we actually kind of kickstarted the work on DAG-JOSE. It was actually at at Denver. So, that's that's pretty funny. Uh, so DAG-JOSE and basically I'm going to talk about what it is and like why we need standards for encrypted and signed data on on IPFS. So, the things I'm going to be go going over uh today is why why do we need encrypted and mutable data?

What why is that important? Uh, we're going to look at some existing solutions to this. Um, and then we're going to dive into the DAG-JOSE um, standard which which is kind of like something of like an amalgamation of three different kind of existing standards. Um, so this is going to be fun. And then finally we're going to talk about some some applications of this stuff.

All right, so uh why why do we need encrypted mutable data? What why is that important? Um, IPFS kind of already is an authenticated uh data format or rather IPLD. So, IPLD is this like linked data structure and you can with IPLD you can kind of represent of course like IPFS data, but you can also represent like Ethereum, uh Bitcoin, get and like other types of linked data structures. Uh and if you have the hash, you can like kind of authenticate that when you sync the data, you can uh know that you you got the right data.

And since you have like links uh in in kind of the first object, you know that like all the data that's uh subsequent or like linked from that object is the data that you expected to be. So so uh Wha- what what would we use encrypted encryption and mutability for? Well, what if you want to know like who created the data? The the kind of uh hash doesn't really say anything about like who authored the data. Uh or who changed the data uh if we get like a new new hash.

Um And and who can read the data? Like by default, like put your put putting stuff in IPFS IPFS means that it's like pretty public. Um and the first thing like who created and changed it, like we can use signatures to kind of uh deal with uh signatures and like you know, you can use a public key to verify that this is signed by a specific uh party. And who can read the data? We can actually, you know, use encryption to uh manage access control.

Uh So the the kind of existing solutions that exist and have been used and is still being used um for these different things. Uh but like the first thing kind of I think most commonly known about way to do this is to just take the IPFS hash or the CID, uh put that on chain, and now you kind of have or you have the signature for the user that updated the transaction, and you have um you have like uh the latest update. And then you can like make another transaction to update the the uh the hash in the data. But, this is obviously like super expensive, especially given like current gas prices. Um and then you have stuff like IPNS, which is basically like taking a public key and signing some data and then publishing that in the IPFS DHT.

Um but, it's kind of slow and you can't really do interesting stuff like change changing like who is the controller of that data. Uh then, you have more sophisticated systems like OrbitDB and Textile. And these are kind of append-only logs that each each and entry in the log is kind of signed. Uh so, you can verify both the data integrity, but also the the kind of cryptographic integrity of the signatures. Um the kind of not so nice thing about those like OrbitDB has their own specific way of making signatures.

Uh Textile OrbitDB has like another way of doing signatures, and they're not really interoperable. Uh for encrypted data, uh we kind of have the same uh same same situation there. So, Textile uses AES encryption, if I'm not mistaken. And Peergos uses XChaCha-Poly encryption. Um and I mean, using different types of encryption is fine, but we don't really have any way to interoperate between these because you just like use a specific encryption algorithm and just put the data in some way into IPFS.

All right. So, uh we can actually do better. Um and that's where DAG-JOSE comes in. Um so, DAG-JOSE is like a composition of of different standards. And I would say like three different ones.

So, the first one is is JOSE, which is JSON Object Signing and Encryption. Second one is well, it's IPLD, uh the the the IPFS native data structure. And the third one is not mentioned on this this page, but I'm going to talk about it later, is the IDs. The IDs um decentralized identifiers is a way to go to represent uh digital uh identity kind of and allows you to do stuff like key rotation, stuff like that. Um But anyway, so JOSE So, JOSE itself is an IETF standard.

There's an RFC for it. Actually, there's several RFCs um and they're a subset of JOSE for encryption, JWE, a subset for signatures, JWS. You might know a lot of the better known uh there's another um kind of subset that's called JWT and JWT can be either like encrypted or signed. But the cool thing about JOSE is that we can like uh use different keys. Like the the data structure of the JOSE object tells us, "Oh, this was signed with uh ED25519 key or a uh maybe it was signed with like a secp256k1 uh public key."

Uh so, you can like represent many different types of uh signatures and different types of ciphers as well. So, we could use AES or XChaCha or other types of ciphers without kind of breaking interoperability. And it's widely implemented. So, uh I I just looked at GitHub before and like I couldn't find the last page so when I searched for um JOSE implementation. Uh and and it's adopted being adopted in in or is already adopted in the decentralized identity community.

Uh so, DAG JOSE uh Well, why do we need that? Well, actually it turns out that JOSE is kind of uh great and uh but we can't really use it use use its kind of full capability in IPFS uh if we just put it into IPFS like a pure data object, so we need like a special IPFS codec. So, an IPFS codec is basically a way to represent new data formats within IPFS. So, there's an IPFS codec, for example, for Ethereum. There's an IPFS codec for Bitcoin.

So, you can like actually traverse the entire Ethereum blockchain inside of IPFS, which is which is pretty cool, uh but it's not very uh well known or like commonly used yet. Um Anyway, um so let's let's look at a signed object. So, a JWS normal JWS has a payload and has the signatures. And that signature consists of protected header and a signature. So, now um why couldn't we just put this into like the IPFS DAG and be done with it?

Well, usually when we deal with like IPFS data, is that we want it to like kind of link to something else. Um and if the payload actually is used like this gibberish text, this is a base 64 URL encoded string. Uh we we can't really follow any links. Like IPFS doesn't know how to like follow something that's encoded as as this this base 64 URL. Um so, therefore, we kind of need a specific codec to do something about that.

Um so, we created a codec and that's the DAG JWS codec. And it introduces one constraint. The payload needs to be a CID. And what that means is when you create a DAG JWS, you take the CID you want to link to and code that as bytes or or you take the bytes of the CID and code that as base 64 URL, and then you basically put that into the payload, and then you make the signature over it. Now, when you retrieve the object the the database object from IPFS, the the dag-jose codec will automatically see that, okay, it's a dag-jose object, take the payload, and encode that as a CID, and put that on the link property, which you can see here below.

And this is allows us to like, once we retrieve a dag database object from IPFS, we can automatically like follow the the link property, and then follow any kind of other um path down the the the the Merkle cap the the Merkle bag that is associated with like top level um dag-jose. And that could be like just pure IP LD objects, or other um dag-jose objects, or or yeah, really really anything like a Bitcoin block or something. Um so, this is great. Like, this this works really well. Uh but, now is the problem, right?

So, we made the constraint that the payload, or the clear text, needs to be a CID. And that that's fine for for signed objects, but what about encrypted objects? Like, if I encrypt something, and you only encrypt the CID, the hash, like, the the data will live in clear text on IPFS, which is like super bad. So, we don't want that. So, what do we do about that?

Well, uh interestingly, there's there's a way around this in in IPFS. And I'm kind of going to walk through you how walk you through how it works. So, essentially, we take the clear text IP LD data that we want to encrypt, we encode that as an inline CID. And an inline CID is is a special type of CID that where the hash function is is the identity hash. So, what the identity hash is is basically takes the data, and puts the data itself.

So, you can think of like the identity hash of ABC is ABC. The identity hash of 1 2 3 is 1 2 3. So, the identity hash of your object is basically just your object. So, you encode the IPD object into some IPD coded, put it into inline CID, and then you encrypt inline CID. So, now we don't actually point to any other data in the IPFS network.

We just encrypted the entire data object as a CID. And then we can put that into IPFS. And then when you like retrieve it from IPFS, you can use like the encrypted and um you will be able to like get the inline CID and interpret the data. Okay. So, um what's this thing about DIDs that I talked about?

So, uh DID or decentralized identifiers is a standard for uh well, this is essentially does one thing really well. And it's going from a kind of tiny identifier that's like did colon three colon uh a bunch of random numbers, and which is you a unique identifier, and goes converts that into a DID document which contains uh public keys like your public uh signature verification key, your public encryption key. So, I can get this identifier and then find your your public encryption key and encrypt a message to you. Um and this kind of may makes kind of very well suited to do something like that with say so you can like easily verify uh who made the signature and like easily encrypt data to another uh user. And uh we implemented or we kind of created something called EIP 2844, which is like a JSON RPC extension to well, uh really any wallet.

So, it's essentially like three methods. and the authenticate then create JW or create JW and decrypt JW. Um and any wallet can implement this. It's not really tied to Ethereum. You can implement that as like a polka dot wallet or like a near wallet.

Uh and you kind of get the same benefit like because the DID spec is blockchain agnostic so you can use DID that's like native to your blockchain or you can use the 3ID DID method which I'm going to talk about a little bit later. Uh we we also build like a consumer library of this JSON RPC interface called JS DID which allows you to like encrypt and decrypt objects and like sign objects and verify signatures of of objects. And then on the wallet side um there is like the key DID provider for ED25519. It's like a very simple public private key pair based DID. There's 3ID connect which allows you as to do like more complicated things like have multiple authentication methods and stuff like that.

I'm going to talk about that a little bit more at the end. Uh and yeah, essentially like you can have the wallet separated from the app and the app can do stuff with JS DID like create DAG-JOSE objects and put them into IPFS. All right. So, can we actually use DAG-JOSE? Like that sounds super cool.

And yeah, you can. In JS IPFS, it's fully supported. And you can get it by importing the JS IPFS library and importing DAG-JOSE like putting them together. Uh and then it kind of just works. Um or you can uh install our bundle that we built.

So, @ceramicnetwork/ IPFS daemon there's an IPFS bundle that bundles these two things together. In Go IPFS, uh support is coming soon. So, the DAC Jose Kodak has already been implemented for IP ID Prime. So, IP ID Prime um is the kind of next version of IP ID, which is using Filecoin, but it's not yet using IPFS. But, the IPFS team is telling us it's coming very soon.

So, uh we'll be compatible there very soon. Uh so, yeah, looking forward to see like more applications built in kind of once more support in IPFS is there. All right, so what what what what are the applications for DAC Jose? Like Yeah, we can store data. What do we do with it?

Um Well, the first thing is of course like we can upgrade existing protocols. So, um Textile could start using DAC Jose. And actually, the DAC Jose was kind of inspired from a discussion with Carson from Textile last year at at Denver. Um and they're kind of already start experimenting with it. OrbitDB also has on their road map to kind of look into DAC Jose.

And there are some other kind of protocols that looking into it as well. Um and we could potentially create like other types of protocols um for like event logs, different ways to receive your ET in different ways. Um you can do like one off signatures or one off encryption. It's not really a protocol. You just want something that to represent this uh signature or encrypted object.

That's also fine. You don't need like a protocol that uses this. And and finally, um Ceramic Network, which is a project that we've been working on at uh Three Box Labs. I'm going to dive in a little bit to um Ceramic and kind of explain like what it is and and why it's using DAC Jose and what we can do with it. Like once we can once we have like built the the the kind of core layer thing and build the protocol on top, like what can we do with that?

Uh so, Ceramic is a decentralized kind of framework for mutable information. Uh so, in Ceramic, uh we have something called documents, which is basically uh an object which has which has like a persistent identifier that never changes, but it's still mutable, so you can change documents however much you like. Um without changing the the identifier. And we kind of get strong data integrity from from party the signatures, but also from uh updates are anchored into a blockchain, uh which gives us like timestamp and and provides us with the ability to do secure key revocation and stuff like that. And since we can represent like a bunch of different documents in Ceramic, and we can also link the documents to each other, we can create this kind of graph of interlinked documents that is like ever evolving and like changing.

Um And yeah, so uh in Ceramic, we have the concept of document types, and we have implemented two types, uh which is tile, which kind of like a generic JSON uh object uh document. You can just store JSON object and update that. And then 10 link is a doc type that allows you to link a blockchain account to a DID. So, you can link like your Ethereum account to a DID, you can link your Polkadot account to a DID, you can link your NEAR account to a DID, and you can link all of them together to one DID. Uh yeah, that's pretty cool.

You can like link a bunch of blockchain on-chain accounts to this kind of decentralized off-chain data system. Uh and you can potentially like create your own doc type. And there there's full support for like Ceramic is is of like has this framework which allows you to build different types of uh documents. And this is useful if you want like different some different type of logic that goes into uh your mutable data. Um and right now you can do like a few different things.

We're working on like abstracting this a little bit more. So potentially in the future we could represent like an OrbitDB database in Ceramic or maybe like a Textile database in in Ceramic. Uh but but that's pretty like it's still quite far off, but there's still like a bunch of powerful things you can do with like custom doc types. All right. So let's let's actually jump into kind of a data structure.

Like how do we actually use that Jose in in Ceramic? Uh so the what in this picture what you see is is basically a Ceramic document and the data structure of this Ceramic document. So first we have a genesis commit and this is an like an IP LD object which contains the first kind of initial content of a document. Then this document has an anchor commit. The anchor commit basically um is another um update to the document that contains like a proof that this document has been anchored into a blockchain.

And here this document has two signature commits. And these signature commits uh are of course using that Jose and um yeah, this is signature from the uh controller of the document which was specified in the genesis commit. Uh and yeah, they're stored and the the updates to the content are stored as like JSON patches. So that Jose commits the signatures and then like another anchor commit and in the signature commit yeah, and then kind of goes that way, uh into the kind of future. Um So, that's kind of the basic data structure of a document.

And of course, like this would be super expensive if you needed to pay for every like anchor commit yourself to anchor it into the blockchain. But, what you can do with Ceramic is you can actually take a bunch of documents and uh create a Merkle um tree that we put into IP LD, and then put the just the um uh sorry, Merkle root into the blockchain. So, we have like one transaction for a bunch of different documents that are being updated at once. Um and the cool thing about having them in IP LD is that we can sync like a specific Merkle witness uh over the network. Um so, if I only want to sync my document, I don't need to sync the uh data of like a bunch of other documents.

Uh and yeah, there's a bunch of other details here, but I'm not going to go into uh any more detail uh there right now. Um There's some stuff about like how do we do uh conflict resolution and stuff. And basically, the uh thing is that the the earliest anchor wins. Um Uh If you have any questions about that, you should jump into our Discord. All right.

So, what what what can we build on Ceramic? But, like now, once we build Ceramic on top of I will say what what can we build off that? So, uh I think there's a bunch of interesting things. Uh one thing is governance. Uh and we're working with some projects that are building some interesting things using Ceramic there.

Uh so, like discussion forums, uh potentially like voting uh is a really good use case because we have this kind of uh integrity of the data where it's like timestamped and anchored and uh signatures of every participant. We could potentially do different forms of like this future solution because like the surrounding document is a digital audit trail. Uh maybe we can do some cool recording contracts where the contract can be updated if it's signed by uh you know, your lawyer and your counterparty. Um Publication is a really interesting use case. You could do like research publication, journalistic publication, uh publication of open source intelligence.

Uh yeah, that that really interesting aspect. Um one thing that we're building in Three ID Labs, it's something called identity index. An identity index is essentially like uh kind of like a decentralized user table. So, like putting um give it giving uh put putting data about the users in control where the user is. So, you don't really need to store a bunch of user data on your servers, rather they just live on a decentralized network where the user has full control over the data.

It could be like any kind of blockchain metadata or or metadata about the blockchain accounts. It can do be like uh profiles for the users, like skill trees, like things you've done or things you can do. Uh different types of reputation could be built there. Uh private data storage, you can put encrypted data into IDX. You can uh take IDX and point to data that lives in like Textile or BTB or some uh centralized data store.

Um Uh and you can build a bunch of other cool stuff with with IDX. All right. Finally, I mentioned that I was going to talk briefly about 3ID Connect. So, 3ID Connect is kind of like uh a thing that allows you to use the EAP 2844 uh DID specification uh with any wallet. Um So, 3ID Connect is like a wallet agnostic authentication system to your 3ID did.

Uh and um uh it's also blockchain and so I could authenticate with, you know, MetaMask or through wallet connect or some other Ethereum wallet or like Polkadot.js or like uh or near wallet or some other kind of blockchain wallet system. Um into authenticate to like this this DID. And the cool thing is you can actually have the same the same DID be authenticated to from both your Ethereum and your Polkadot wallet, which means that all the data that you kind of created for a profile and stuff like that, you can actually take that with you to the Polkadot ecosystem and use an application there. Um which makes kind of uh data really kind of interoperable, which is like the power of using these standards for like signed encrypted data in an IPFS and and by extension like uh you kind of get this interoperability of data that can move between um different blockchain systems.

And yeah, the the 3ID did did method is really cool because it's just a ceramic document and by virtue of being a ceramic document uh you we can do like secure key revocation without having a blockchain or we kind of have a blockchain we kind of anchor stuff in a blockchain, but the the document itself is not really a blockchain. And since we can do secure key revocation, we can have multiple authentication methods just like the example I just mentioned. All right, so I hope you found that interesting and and really want to dive into creating stuff with with signed encrypted data on IPFS. And if you're interested in like learning more about Ceramic, you can go to our uh documentation page at developers.ceramic.

network or check out IDX and you can find us in this board and probably in all of the other cool venues here at at Denver. All right, thank you all.