J3 SAY 03 Louis Holbrook Mutable resources

my name is Liz Holbrook I am part of the Jack & swarm core back-end teams and in my 14 months of excursion down the crypto wormhole so far among other things I've implemented the PSS messaging messaging system over swarm and also implemented something called mutable resource updates I gave a presentation about PSS in Cancun in November and now I will try to explain you what mutable resources are so mutable resources in short are a predictable way to look up data and swarm but before I dive deeper into the topic I'd like to give you a little bit of background first now as I'm sure you already know swarms go aetherium implementation of unsensible distributed data cloud storage using etherium nodes swarm isn't sensible because storage and retrieval of data and swarm is routed by common properties between the content and a network thus there is no torrent like tracker or any kind of central entity in charge the discovery of resources in the system like torrent data since wormus cut up in chunks and distributed in the network but the difference is that a node is not expected to seed a whole file on the network instead nota responsible just for chunks whose own properties relate students to themselves in a particular manner in fact swarm is designed so that you can upload and disappear whereupon the network will make sure that what you uploaded will be available now the relation I talk about is a notion of proximity and swarm proximity has nothing to do with geographical distance but instead we speak of and when we speak of distance and swarm we mean how much the addresses between entities in the network differs from each other specifically this means how many leading bits one address shares with the other now both contents and nodes in swarm share the same address space namely a 256 bit hash for nodes it's the sha-3 hash of a public key of the account you used to start the swarm note four chunks it's the sha-3 hash of the data content of the chunk itself now when it chunk is sent into swarm every node will look through its list of connected peers and compare number of bits shared between the peers address and the address of the chunk if a connected peer has a better match then the current node it will pass the chunk on to it like this this is quite the superficial explanation of whoops this is a quite superficial explanation of the dynamics involved but what's important to note is that the chunks in swarm will always tend to end up with the notes that's closest to so we established that an address of a of a chunk is a char 3 hash of the content that means that any content in swarm even if it just changed us a little bit and bit being quite literally here we get a completely different address in the network and that's trying to guess in advance where a chunk will end up in swarm it's kind of like playing a game of whack-a-mole except the volume of the holes would be substantially larger than the universe and even divine powers will probably struggle to find where to smash them the net mallet next so there's no way to predict what the address of the chunk will be without actually having the data and hash again solving this problem is exactly what Ian s does so like DNS in the internet for the common people it's a persistent identifier that points to an address that can change over time however for those that are inclined to bench and data objects update DNS has two important shortcomings it's relatively expensive and you're at the mercy of the speed of the miners to get your update through so in the spirit of true modern consumerism wouldn't it be nice if there was a simple way to do this fast and cheap instead and fortunately there is and this is why I'll try to explain now first such the scheme to work we first of all have to decide on some predictable rate of change to use as a reference of course in our space the most obvious choice here is blocks in the blockchain because they have a relatively constant pulse and they're just deliciously incremental now what we do is we decide and advance in a certain frequency the updates will be made and we also need to choose where to start so let's say a mutable resource will start the block 30 to 42 and we announced that update will be made every four blocks so this means 46 50 54 and so on now we can bind an identifier that's a block number to specific data now let's say we want to retrieve this data now let's pretend that we're currently on block 44 since we know that we start on block 42 and then with a frequency of 4 we know that 46 would be the next block in the future therefore we make a query to the identifier that's equal to this and as you can see here they're so would slight disappointment we Trevor's back or skip back to the previous number in the block here we find an update so we can retrieve it of course if there was an update at number 42 46 then we would just retrieve that so whenever we can't find an update at a certain block number we iterate backwards in time too we either reach the first block that has data or reach some sort of limit that we set on how many hops we want to make sup t up to the implementer here to keep the promise of the update frequency and let's see where were sigh yeah it's up to the implementer to keep the update frequency and it's up to the client to decide if you want to bother if they get too flaky now just a while ago I accused ans of being slow but you know even though propagating the updates through the storm itself instead of mining blocks having a maximum update frequency of every four blocks doesn't really make things better it actually it makes things much worse so the answer for this is allow for multiple updates for each period and in the above example we have three updates bound to the block number 50 these of course can be numbered sequentially and will be queried after another so to summarize first we go to the next block next future block after the current block that we're at optionally stepping backwards until we find an update we then ask for sequential versions of this update and we can't find any more versions we conclude this must be the latest one in fact we can think of these block numbers had major version numbers and the different updates belonging to the same block as minor version numbers in the implementation I've written the four mutable resources major version numbers are referred to as periods and the minors merely as versions also when we refer to periods we don't use the block numbers in the code but just incremental numbers starting from one so let's have a look at what these predictable identifiers actually are and how we store them so to create a new resource really means that we just create the root chunk which contains the period and the frequency the key of this chunk is to show three hash of an arbitrary identifier that we call the resource name typically in DNS name hash this chunk is just different from normal data chunks in swarm which are all content to rest as we saw before it actually you might be so rude as to call them fake chunks but they're recognized by the nodes as being special resource update types so while normal data chunks when they travel through swarm are being checked that their key matches their data these guys are left off the hook to create an individual resource update the chunk is created where the key of the chunk is the sha-3 hash the concatenation of the major version number of minor version number and this identifier the name the chunk contents themselves are not just data it includes a header in which the elements of the key data are stored and citizen resource number can be a varietal and the header includes a header length as a 16-bit number it also includes the data length as a 16-bit number because an optional signature can be included at last as the last element to the chunk and if the actual data length is longer than a given data length the excess data is interpreted to be a signature of of the chunks data and both the period and the version is our 32-bit numbers if the data length is set to be 0 the data portion of the chunk is treated as a multi hash so this just means that an extra check will be performed to verify that the multi hash is in fact a valid multi hash and as you probably know multi hashes are self describing data structures so you can tell from the data how long the hash is and what type it is etc I won't go further into that but just note that there is a proposition of making ens itself multi hash compatible so it can point to a much wider range of data or resources note that the size limitation of a chunk and swarm is 4k so this does not mean of course that it has to be this size just means that the data together with a header header and the signature can't exceed that limit and resource updates and swarm are always just one chunk so apart from cheeky chunk forgery resource update works like any other data and swirl so when you upload data to note it will pass onto the swarm seeking the peers that are closest to it as we saw but if all did that if all the if that was all there was to it you know anybody could come along and just make an update on behalf of somebody else now if the mutable resource that you have is equivalent of your personal twitter feed let's say this might not be what you want so we need to make sure that the resource actually belongs to the person attempting the update this is of course done with signatures chunked data is signed signatures inspected by a node compared to an authoritative instance that can tell you if the corresponding signature matches the owner or not and typically this is done using en-us meaning that the address of the public key of the signer much the owner entry of NS of the name that is the name of the resource and given that the name sorry that the data is validated or the signature validated it's passed on to the peers now till now we only seen how to retrieve the updates as this traversal of periods and versions and this can be potentially a bit inefficient or at least demanding on resources so but there is a way around this there is an easier way you sing PSS messaging we can actually send notifications when we make updates to parties that are interested so this is done using a SUBSCRIBE method and what happens then of course is that when an update is made they instantly know what is the latest version so they exactly know exactly where to go to look for the data now if Ana validation should fail of course no updated sensor swarm and no notification to the listeners either the update validation will also be performed when the data is sent among the peers so if that would not be the case of course then you could just create a very very naughty node that sends updates that would claim to be valid and everybody in between would be none the wiser so speaking of ENS it actually will be possible to update the ENS record itself upon sending a resource update and since the update will be signed by the node which then has to be owner of the ENS name if in effect makes it theoretically possible for the node to vet third parties to update the ENS pointer with a multi hash to update the with the reaffirm the the multi hash from the resource update I mentioned the NS as an example because this is what the system is primarily designed to cater for however the implementation the methods for signing for hashing and for validation are all pluggable so you could theoretically create you know your own exotic signature and validation scheme to plug in or actually choose not to have any if you want now a mission of signatures and valuations will incur some restraints in the system properly to mitigate from flooding and malicious activity so we're still kind of discussing how if we should disallow it outright or finds the middle garage I hope for the latter so let me add a couple of side notes in order to get the correct block height you'll have to be connected connected and also fully sync to the block chain and this is not always a simple task to achieve as we've seen lately so of course you could just query a different node that you trust to be synced but what you could or which you would do anyway if you're a light client of course but depending on the granularity of the update it could theoretically also just suffice to look at what time it is and kind of infer what so they're supposed to be this week green ray here showing that you're scanning this block I don't know if you can see it but anyway and for kind of approximately where we are of course now they're supposed to be a green way pointing to the future of course if you're off your guess is off then you here heuristics will point to a block that is beyond the next block in the period but this is necessarily a problem it should be possible to to change arbitrary updates in the future so at the time the only thing that the system really will stop you from doing is trying to change the past also mention PSS a couple of times here so it's short for postal services or swarm and uses the same kind of routing mechanism that you use for passing data along and swarm to send messages instead and in these case you see attach manually so to speak and address on the envelope and this is addresses used to find your way to the to to the peer that is the recipient but you can also specify partial addresses then it will be targeted to a neighborhood of peers all of which the address prefix match system or you could send without an address at all which then sends to all peers a PSS also used a symmetric and asymmetric encryption and a recipient in PSS is basically whoever gets the message through them and can also successfully decrypt I think on the DEF CON three youtube channel there is there is a recording the talk I made on this Cancun so if you're interested in this topic then have a look we provide a HTTP API for the operations for mutable resources with the prefix bzz - resource so to create a new resource you issue this with the ID which is the name of the which is the name of the resource that you want to create and the frequency it will take the block the current block number and use that as a starting block then to update a resource you would post data to the same but just using the ID it will automatically find the next upcoming period and it'll also iterate and find the current version that we're at and appended to the next version number to retrieve you can retrieve the latest one by name you can retrieve a certain period by naming the period you can also retrieve a certain version from a certain period by giving those pieces of data we also provide a transparent retrieval of resource updates using the normal normal retrieval calls for swarm so this is what you use if you want to retrieve a sworn normals were manifest but if the code recognized that this resource or the is actually a mutable resource type it will transparently retrieve and return that instead so resource update engine signature invalidation the HTTP IP I swim manifest pointers and verification of multi has datatype were all implemented and still to come in the code are the PSS update notifications and the ability to update DNS directly when you make an update and yeah hopefully it will be put in the code shortly this is all part of that proof of concept three halves warm that's now pretty much feature ready and we're just weeding out the last bugs I won't be so bold as to try to announce any any date additions but it should not be far away so lastly this is our repository where you can follow the cutting-edge latest code swarm where these mutable resources and PSS also is included code examples are listed here cheekily is just my advertisement for a p2p programming tutorial that I wrote earlier it's just code examples that show you how to do p2p programming in swarm no sorry in aetherium and also gives you a crash course of how to employ PSS as well and of course if you want to get in touch there's at the bottom is the usual stuff I think that's my last slide and if there is time for something then maybe some questions you talked about the storage components in swarm I know that swarm oh so does this is like the new feature whether it's streaming the streaming also utilize the immutable resources in any way it's like when you turn this off just so you know yes it uses the same framework so basically what I was talking about earlier when you're checking whether the data or the chunks are valid or not the sinkers will always have also have this feature where they would detect it's a resource update and let it through so it uses pretty much the same network I think the results an idea that we will be using this system also as a notification channel if we can kind of hook it together so in this in the case of streaming does it mean that every new chunk of data in the stream is a new like resource update or is it like or is it like only when I start a new stream well well the resource update consider it as a like a new resource you know I mean no I didn't quite understand sorry so like so in you know in a data stream right like you can look at it as this data is constantly being updated because it's a stream of data that's coming in or or you can look at it as this is this stream is one piece of data that will present itself in the future but you will only uh like it's only an update when you have a completely new stream you know that with a different different reference or something I guess I guess maybe my question is how does the sinker work with with the immutable updates ok so I'll have to disclose that we actually haven't done that part yeah so yeah that's actually also the upcoming thing is to just wire this into the new sinker sinker framework so but basically you know it'll use the sinker for propagating the chunks and swarm as normally of course but we'll also look into the possibility of using it as if no notification channel as well yeah I'm not sure how to answer your question more precisely in that because