J2 RF 08 Daniel Nagy Swarm

so I'm going to talk about privacy and swarm swarm is the subsystem for aetherium where distributed applications can store large amounts of data which would be prohibitively expensive to store on the actual blockchain and there's an obvious requirement to for some of that data to be private that is only accessible for parties that are authorized to see it now let me just give a brief outline of my talk so first I'm going to give you the minimum necessary information about how data is represented in swarm then I'm going to discuss how content is actually symmetrically encrypted encryption is crucial here because in a distributed system unlike in a centralized counterpart it is impossible to control the access of people to data or other entities to data by providing or not providing that data because all the data is by definition public so the only way to restrict access to data is to encrypt it and in the third part of my talk I'm going to discuss how key management is done and how access control is actually implemented so first a brief introduction swarm consists of these four layers so at the bottom is a layer where we have chunks of data each of them at most four kilobytes in size so they're fairly small fixed size chunks of data and the network only knows about this layer so the when when swarm nodes communicate with each other all the communication happens in reference to these little chunks above that layer we have a layer which represents big binary blobs or potentially big binary blobs which are orbit which have an arbitrary length well whatever is expressible in 64 bits and there's no metadata attached to it so on this layer every every blob is identified by the hash of its root chunk because these blobs are actually organized into a tree of chunks which I'm going to show you a little bit later and above this layer we have a virtual content address web server which has all the typical metadata of web content like content type HTTP response status and things like that and above this layer on top we have this warm hosted distributed applications which can if they want to access the two layers below them but a typical application would only access the divertor web server so what's inside the chunk a chunk consists of two parts first is what we call the span which is kind of the file length of the file represented by this particular chunk and the subtree of other chunks that are below it and at most four kilobytes of payload data so if the span is less than four kilobytes or equal to four kilobytes then it is basically the payload is the data so the payload is the binary blob if if the span is more than four kilobytes then this chunk is an intermediary chunk which means that it's a non leaf node in a tree and it has references to sub trees not the actual content so here's the here's the representation so here's how you would store like 10 kilobytes of data and swarm so you have the root chunk which declares a span of 10 kilobytes and since 10 kilobytes require at least 3 chunks it has three references to three chunks and those three chunks have 4 4 and 2 kilobytes of the data so this is how 10 kilobytes of data would be represented inside swarm and it takes 4 chunks to represent it the layer above it contains objects which are called manifests which are merkel eyes devalue databases somewhat similar to etherium state trees and the key in these databases is the paths that you would use in the URL and the value is a reference to the content and all kinds of HTTP metadata that you can think of typically and as a minimum you need to have content type so if you address a collection of files in swarm than what you actually enter is either the hash of the route chunk of the manifest or an ENS name which contains this root hash so that is how a swarm site is structured if it's unencrypted and are there any questions actually at this point ok so then I will go on to explain how can we so all this was completely unencrypted and is completely unsuitable for uploading your home directory for example so in order to overcome this problem we're going to introduce symmetric encryption to swarm symmetric encryption means that you use the same key to end trip down to the Crypt the actual and cryptic schema is counter mode encryption this is a this is a diagram from the Wikipedia page for modes of operations of block ciphers so this is currently the most most widely used and most secure mode of operation of a block cipher because it is very easy to reason about and what it actually does is that you can see that the the actual one wave function the block cipher is always used in encryption mode so you don't use the reverse reverse direction of the block cipher and what it achieves is that it takes a key and using a counter creates a pseudo-random bit stream out of it the so called key stream and then this key stream is XOR with the plaintext in order to get the cipher text or it is ax sword with the cipher text in order to get back the plain text and because of the counter mode it is actually possible to do partial decryption where you don't have to dick generate the Hokie stream only its section which covers the data that you're interested in so this is the basis of what we use but also please note that here we don't use many of the properties of the block cipher the only property that we use is the one-way function nature the random Oracle nature of the block cipher namely that given the output it is hopelessly difficult to wreak find a matching input consisting of both the key and the actual input of the block cipher which in this case is the nonce and the counter so for our purposes we have created a special case of this structure oh sorry wrong button so instead of the block cipher we use the shot three hash function and we use it twice so what's the justification for this first of all shot three is already used in etherium for identifying all kinds of things so the block hash is in shot three the links inside the inside the state try is Shari and also in swarm the references that we use are also Shari so basically as long as Shari is secure those parts are secure and if Shari is broken then it's broken anyway so in order to keep the attack surface small in order not to introduce other cryptographic primitives which would constitute yet another point of attack we stick to sha three because if shatter is broken we're screwed anyway and if and we can use Shari for the purposes of counter mode encryption but why do we use it twice that's a more interesting question so using a hash function twice is still a hash function it still has the one-way property however it gives us one extra opportunity that in linking encrypted data to the blockchain and in particular to smart contracts because it becomes possible to reveal that intermediate step so the one that is between between the two Shari function calls and if you reveal the preimage of the second encryption it will not reveal the key it will not reveal the counter but it lets you decrypt the data which means that you can selectively decrypt a small part of the chunk provide a merkel proof that it is that part that your covering submitted to a smart contract and since the Italian virtual machine also has Shari as a pre-compiled you can actually verify that piece of content now those of you familiar with cryptography would immediately notice that this schema this proposed construction is vulnerable to what cryptographers call an existential forgery because if I revealed the wrong preimage not the actual one I hash it and then I acts or the result with the ciphertext I will get some plaintext and I can still claim that this was the original plaintext and with this particular preimage I can produce this ciphertext which is the encrypted content that's stored in swarm but existential forgery is turns out to be not that mature problem because with existential forgery you can find some matching plaintext but you cannot place any constraints on them so if there are any integrity if there's any integrity protection built into the data itself then you will not be able to forge it at all because existential forgery will not allow you to actually fulfill the constraint of the integrity protection and as it turns out there's plenty of integrity protection there so in the intermediary chunks of the of the swarm representation of binary blobs the the integrity protection is the very fact that hey the plaintext happens to be the hash of another chunk and with existential forgery if you can do that it implies that you can actually find a collision in the hash function which by definition you can't or by assumption to be more precise by assumption you can by the assumption of security of sha-3 you can't so the intermediary chunks are already secure against existential forgery and for the actual content you do have to provide some integrity protection so whenever you need to reveal part of the actual content there also needs to be some hash or some Macker on the digital signature that integrity protects the content that you reveal and in that case you won't have a problem with existential forgery and therefore you can make statements in smart contracts about the plain texts of files that are only available in encrypted form in swarm and obviously reveal parts of it only in case of a dispute so that's that's the rationale behind using sha-3 twice so just to recap the reference is in encrypted swarm consists of a cipher of the hash of the cipher text and the corresponding decryption key so instead of the 32 bytes of the hash in the unencrypted swarm here we have twice as long references but otherwise everything else stays the same and in particular the API itself is completely unchanged so if you have some distributed application that you have put together on swarm for example if you try to use the applications that are provided with swarm as example application like the photo album or the swarm Explorer or the the text editor so if you use any of those distributed applications if you turn on encryption they will just work without any change to the code because the API is unchanged and these applications do not assume the length of the reference the only cryptographic assumption that we make is the security of char tree which means that our attack surface is very narrow and so much hinges on char tree in aetherium anyway that if char tree is broken then we have bigger fish to fry than the problems of swarm encryption but as long as char tree stands up which we hope is going to be a very long time then the proposed scheme is provably secure and the most important property is that this is all very smart contract friendly meaning that smart contracts can depend on content inside encrypted swarm content so that's the symmetric encryption part any questions at this point ok so the last part of my presentation is about the key management like how do we make sure that these symmetric key is that encrypt the swarm content get to become available to exactly the entities to which we want to reveal the information and not anybody else so basically this is access management so the cryptographic primitives that we use here is the elliptic curve version of al-jamal encryption which means that the identity is the sac P 256 k1 public keys it's the same public keys that are used for idiom addresses you can keep those the corresponding private keys in hardware wallets adding swarm decryption functionality to any of them is a trivial matter because it uses the exact same cryptographic primitives as Bitcoin or aetherium and the vast majority of other block chains the plaintext is the symmetric key so the only thing that you encrypt using the alga MO is the symmetric key that we used in the previous section and the way we encrypt it it's there so we generate some random K we multiply a generator point by that K and the ciphertext is going to be a pair which has this capital K point which is the generator point multiplied by the random index and then the public key of the permutation of the party that can decrypt it multiplied by this K the small K and combined in some reversible operation for example XOR with the actual ciphertext and the decryption you can see there how it works so here we're using again the same cryptographic primitives on which the security of the whole aetherium hinges we don't introduce anything in new or controversial and we crucially do not increase the attack surface by introducing new cryptography so this is the exact same elliptic curve cryptography that in theory amuses for digital signatures and how do we manage those keys how do we enumerate the the people who have or the entities that have read access to a particular file we have a ciose access control lists the reference to the ACL concerning a particular file or subdirectory in sites worm is simply added to the manifest as metadata and what's inside this manifest is oh and another piece of metadata added is the public key of the owner of the ACL so the entity that has the power to modify the ACL who has permit who has the permission to give and take permissions and the ACL itself is just like the manifest a key value database it's actually in the manifest format it's exactly in the same format as the manifest since worm and the key is a difficult secret of the owner of the manifest together with the public key of the permitted party why do we do this we do this so that a party that can decrypt that has access to a manifest can very quickly retrieve the encrypted secret key moreover a party that doesn't know whether or not they have access can very quickly check so they calculate they they look at the manifest they take the public key of the ACL owner they calculate the diffie-hellman key crate and then they use that as a lookup key so they can very quickly check whether or not they have been granted permission to read yet they cannot see neither them nor anybody else can see who else has read permissions the only thing that is apparent from a ACL is a lower bound sorry an upper bound of how many members the ACL list has because that's the length of the ACL but the owner is perfectly justified to stuff it with a lot of garbage if he wants to keep it a secret how many actual entities have access to a encrypted entity an encrypted file and the value is the algum all encrypted content keep that I have in that I have shown in the previous slide so you have this ACL what you do is you look up whether or not you have permission to read this if you do then you decrypt the corresponding el-gamal encryption with your private key and that gives you the symmetric key with which you can decrypt this warm content and so again to summarize what it means to the end-user we're using hybrid encryption so that means the asymmetric encryption of a symmetric key and for everything else we use a symmetric encryption which means that large files and collection files only need to be encrypted once you don't have to re-encrypt them if you want to give read permission to yet another party modifying a CL requires a very small number of new chunks to be generated because it's basically modifying a miracle structure and if you want to store the route in ENS to have it a name then a single transaction on the etherion blockchain so each time you make modifications to the ACL no matter how complicated those modifications are you can completely change the scope of the audience to which your content is available and yet the only thing you need to do is update a single root hash inside ENS a single transaction to the ethereum blockchain so it's very cheap so read access in this distributed model equals ownership of the key and the ability therefore to decrypt encrypted content and write access is simply your permission to modify the corresponding record in the ENS resolver so it is governed by the smart contract that is registered as the resolver for that particular ENS name so that's that's how it works so thank you very much and I'm open for the Q&A session [Applause] no I don't think Oh when you want to like let's say I have random right access into the file do you have to re-encrypt the entire like you have blocks all over the network and you need to decrypt write your data but it might be more than four kilobytes so you need to cut it and paste it paste it in the following blocks and everything gets shifted what is the performance of that I'm just wondering so let me go back to how counter mode encryption works here so if you just change data and don't move data like don't insert and don't remove then you only need to modify the corresponding the corresponding chunks and also walk all the way up the Merkle tree to the root which is a logarithmic number of chunks that you need to modify which is a small number and you also need to calculate a logarithmic number of hashes and update 1 root chunk root hash in the end in ens so any modifications that doesn't involve actually changing the distance between information within the file so which is just over writing is very cheap it's actually so cheap that we already have a fuse integration within swarm so you can actually mount a swarm collection and have POSIX level file access to an encrypted word non-encrypted swarm volume and change stuff within it so that thing is cheap if you need to insert stuff or remove stuff that's expensive but that's expensive not because of the encryption that's expensive because how how this whole thing is organized and that is assumed to be expensive in POSIX as well so whenever you have any data manipulation model that can live on top of a POSIX file system you can also have it on top of swarm because what is assumed to be cheap in POSIX file model it's also cheap and swarm if that's answers your question do we have a question there so as mentioned achieve to modify the access so is there a part in this scheme where you overcame the malleability of Hamelin corruption for it's a greatly not advice to use a camel in public crypto systems because of this malleability if I capture you a message encrypting something I can create another ciphertext of my message under your key and this could lead to decrypt your storage on a public network so al gamma by define is homomorphic and malleable thus it's not used in systems where I can see your encrypted messages okay so this is a somewhat involved topic so that's thank you it's a good question though I like it so al-jamal type encryptions indeed do suffer from this that we even if you don't know the you don't know the ciphertext you can it don't know the plaintext or II don't know the plaintext and don't have the private key to decrypt it you can still change it in such a way that the content will be encrypted to some other key right so that's that that's the concern that you have now that you can change it means that you can insert somewhere what you have changed but as you can see the ACL list itself is integrity protected so there's this is not a communication protocol this is a storage protocol which means that like you can eavesdrop on it but you cannot insert messages because the whole thing is integrity protected through a chain of hashes that is rooted in the blockchain so the fact that you can apply a transformation to the encrypted content does not mean that you can actually fool other parties by inserting this encrypted message into any data stream because we don't have a yep is it to check the owner of the access control list as the same as the owner of DNS because the the owner of DNS can also be like a private key it's not addressed is there any reason why you want to have an extra layer in between to have a manifest or can that be the same in other words can you would you recommend that if you encrypt the side and you use a different public key than what you have for ENS well there's one so I would like to leave this this degree of freedom open so we I do not want to hard link the public key of the access control list to the public key of the owner of the iana sentry so I would like to allow for these two things to be different because the the key the shared secret it that you compute between the a non-public between two non public keys even though you cannot reproduce it you can recognize it which means that if you always use the same public key for access control lists and you give permission to the same entities in different access control lists you can so this information that is the same entity that has access to these different different files is revealed it is leaked so actually I would treat the public the public private key pair of an access control list as somewhat of a ephemeral encryption so if you really care about not revealing any information then you don't reuse that key for different access control lists because if you give permission to do the same and it is you would have the same key in both in both ACLs and you might want to to avoid that so that's why I don't I don't insist on those two keys being the same and I actually do recommend them to be different no because the car well depends on how you add that of course so like you can do a key derivation so basically to this scheme you can add the key derivation but you can also do it like outside of the structure so of course you can derive this key somehow and it's a good idea probably to derive these keys because that way you don't have to actually store that many public keys but even if you're deriving those keys it is it is good practice to actually have different keys there even if they're derived from the same seed there's some question up there hi I assume you used access control list carefully or you choose it carefully I just wanted to know are there any reasons in choosing this one instead of other acts based security mechanisms like leaks a purpose base or yeah just wanted to know so actor based access control lists are kind of difficult to implement in a distributed setting because the only thing that you can do is to either allow encryption allow decryption or not allow decryption by sharing the key you cannot withhold the information because the information is there it's accessible so you cannot say that hey you're not authorized because of this complicated set of criteria therefore I'm not giving you that information you can do that on a centralized entity but you cannot do that on a decentralized network so the only thing that you can do is to encrypt and if the only thing you can do is to encrypt then you're basically your only reasonable option is to turn your access control lists into lists of publicly encrypted secret keys no further questions well thank you so much