SWARM Team Update by Aron Fischer & Swarm Team (Devcon4)

so it's my pleasure to start off this team update we're gonna hear from everyone in the team or at least a big part of the team on everything that's been happening with swarm so normally if you've seen my talks before this is the part where I tell you what's forum is and how it works it's all about decentralized storage server less content storage distribution and all the great stuff that web 3 needs appear to be a communication in all that great stuff but we don't have time for that today so you know come find us talk to us if you want to learn more about swarm today it's all about the updates what's been happening so one of the things has changed in the last year we've updated our homepage because it needed an update and it constantly does but anyway we've changed the address so if you don't remember anything just swarm a theoria morgue as it says in the bottom that will redirect you to the swarm homepage it'll redirect you to one hosted on swarm so most of the time it actually works hopefully so you see up there that I want to point out it says installation downloads and documentation those are the most important links because I've got updated documentation the downloads are downloaded bubble binaries I'll get to that in a moment anyway what I should have said is that the swarm homepage is that bzz dot the swarm dot eath but for that you will lead us form compatible browser not that many around but this is advertising for the future so we have a new release process and swarm now lives in the same github repository as gift so we used to live in a different fork different repositories as gift if you get gift code you get swarm code that means whenever that gift release is triggered as form release is triggered there's no more separatrices they all come at the same time that makes it nice and easy whenever there is a release of gifts that there's a release of swarm and simultaneously all of the nodes on our cluster the swarm gateways cluster are updated as well so they're always everything is running at the newest release version the lowest get release tag and we are under heavy development so we can and do introduce breaking changes quite regularly so always make sure to update to this latest release and there is actually right now a breaking change in the master branch set the next guest release will be next four release that breaks changes so always stay up to date let me say that again as important always go for the latest before you ask us any questions okay so having said that let's talk about how installation works because that's also gotten a lot easier in the past few months how to install swarm well there's always the option of installing from source code I said it's the same repository as guest so that's no mystery anymore it's the go a theorem repository the second one is to download binaries if you go to our page and click on download so the compiled binaries for Windows Mac and Linux in both the release branch and the master branch unstable development bleeding edge if you're on Ubuntu it's even easier because you can install from the theorem PPA and then you'll it'll automatically handle the updates for you so you install it from there once the theorem - swarm package you will always automatically be on the latest release and finally we have docker images also now available so makes it easy to deploy it in a dock raised environment it's the F devups swarm repo so there will always be newer binaries they are both in the the latest tag will be the release version and the edge tag is the bleeding edge okay so now we're gonna move on to team updates we're gonna hear from various sub project we're going to start off with feeds which is a nice way of doing dynamic change constantly changing data in swarm with only a single on chain transaction to the NS then Donnie will talk about encryption and access control how to keep data secure make sure that only the right people see it Anton is going to talk about observability which is about how we can actually observe what's happening in this form and in our cluster with some real nice logging and tracing Louis we'll talk about PSS which is the communications protocol that's piggybacking on top of the swarm network then Victor will talk a little bit about light nodes how we plan to into mobile phones into the swarm network and then what's on our roadmap what's coming up next so yeah let's have Xavier pon say trees [Applause] all right thank you very much for the chance to present my name is Javier played here I'm the city of epic laughs I've been contributing to the swarm since since May since this May and today I'm gonna introduce you to one of the features that I worked with with the rest of the team called from Fitz so what are some fits in a in a nutshell so as a as a summary is a way to update content in swarm okay so it's a publisher subscriber system okay where you can post updates about a topic and then also read others others updates about that topic okay and you can also retrieve all their values for those posts and you can do all of that without any single blockchain transaction okay working okay so you can think about feeds as a key value store okay where each user can only write to their own key space okay you cannot overwrite somebody else's value okay and you can read your own values okay and some and everybody else's values provided that you know their address and the topic they are posting at and you can also retrieve all their versions of the values that you post okay and you can do all this without any serum transaction okay so this opens up the door to many dubs okay that do not have to do any turn instructions at all okay so applications that we can we can alter alter continue on without ens okay you can use ens as well but you don't have to you can enable taps to persist content easily so from the browser your browser application called safe to swarm some value and retrieve it later you could use it to improve communication protocol among your applications and also a iot devices could push information to swarm just by holding a private key and that's it okay not it not even ether not private key so how does from feet work okay to push to a certain feed you need just two things you need a private key and you need a topic to post that information under okay I didn't say that you need ethers to post that update just a private key that you can generate but and to read a certain feed the only thing that you need is the users et Rome address that you would be reading the feet from and the topic under which that user is posting information and optionally if you want you can provide also timestamp which would allow you to retrieve data in the past okay all their values for that key you could retrieve as well so just to put up an example let's imagine that we have these sets of of topics on the left for example other it could be avatar another topic could be local weather another topic could be website and then we have publishers or users on top you know if you're a user be and user see so in this case for example user a has posted an update under the avatar key and the update is well her profile information here profile picture so if we will use the query feed primitive instruments we can retrieve that value that she has posted which is her picture okay if the next day for example she updates her profile picture the same primitive code which is a query field would return the new picture but if we provide a timestamp in the past we can retrieve the older picture as well okay so this enables again ended up to interact with farm in this way to save information and the polish information okay so how would this work if we want to use this to publish our website it's not the only application but imagine that we want to use it to update that website as of today when we want to publish a website with ens what we have to do is upload the content to form and get a hash out of that and then we do an on chain transaction to part with their cash and that way everybody can reach the content by the name but then the problem is that we want to update the site we need to publish the new hash and that means going again to ENS and update the condom has which is another transaction cost okay so it means that every time we update we have to pay a transaction cost withdrawn feeds this is simplified in a way that you only have to do a one en s transaction just to have that en s trans NS record my site dot if for example point to the feed instead of to the direct content and that way every time that you want to update the site the only thing that you need to do is to to update the feed instead of to update the contract so that way you have a way to have data site without having to pay so meters you know to get it ok so the key takeaway here is you can update content without doing transactions and you can start doing that now because this is already available in the in the storm cluster so there's the front guide that explains the step-by-step how to start messing with it so I recommend that you guys take a look at all the examples in there together start step-by-step okay great yes that was about how to get comped don't dynamic changing data in swarm next up there two sections are by Daniel he will talk about first encryption which is the low level encryption of all the content and swarm and then about access control which is making sure that only the right people with the right content so hello everyone so I'm going to talk about how you store confidential information in swarm and how you limit the users which you can access it so confidential information is in swarm is encrypted using counter mode encryption which you can find the basic security properties in Wikipedia however we use our own version of it which is a slight modification so instead of the Seifer which really is just a one-way function with a reverse gear which CTR encryption doesn't use any way we use Shari twice which is the same hash function that is used throughout aetherium and we use it twice because with accessing that little piece of data that is denoted by that pink arrow what you can do is you can partially reveal plain text inside a encrypted volume or a encrypted file in such a way that a smart contract can actually verify it which means that you can make various commitments on the blockchain in smart contracts regarding the plaintext of certain encrypted content and if there's a dispute you can reveal a piece of data that does not reveal your access keys or anything else just reveals a little part of your encrypted content whatever it is this is vulnerable to what cryptographers call existential forgery but if there's any integrity protection in place then this is not actually a security problem so this is how we encrypt and in practice for application developers this means the following that compared to unencrypted Swarm the references are no longer plainly route ashes but hashes of the ciphertext plus the decryption key so they have grown from 32 to 64 bytes but otherwise nothing else has changed the API is exactly the same and so is the data model meaning that if you have already written distributed applications for swarm or looked at the example ones that we have in our public repository they would work without modification on the encrypted swarm so this is a native encryption API on top of swarm and every swarm node supports it and the only cryptographic assumption that we make is the secure the collision resistance of the SHA tree hash function now so we have not actually increased the attack surface beyond what etherium already assumes and you know if char tree is broken then we will have much bigger problems and finally we have designed our encryption so that it's a smart contract friendly as possible so now I'm gonna move on to access control in swarm which basically boils down to how do you change the sets of users that have access to certain content so it's important to emphasize that in a permissionless trustless and distributive system you can not really act as a gatekeeper or you cannot really delegate nodes to act as gatekeepers instead read access corresponds to the ability to decrypt and write access corresponds to the ability to register so registration can happen on swarm feeds or directly in the blockchain for example in an ENS contract and that's about what you need to know about it but read access which is the ability to the crypt is a bit more interesting so I'm going to talk more about that so there are three strategies in which you can identify authorized users to access content the simplest and most intuitive perhaps is a passphrase so you can make content accessible to users that know a particular passphrase so in this case both the publisher and the consumer needs to know that passphrase which of course has the disadvantage if that if the same user needs to access content by two different publishers then those two publishers can also access each other's content so in order to mitigate that we have a more sophisticated access control mechanism which is public keys public private key pairs and at this point I would like to draw your attention as developers that these public keys are not aetherium addresses these are actually points on the elliptic curve of which aetherium addresses are the hashes so you need you need those in order to as a publisher to identify your consumers and as a consumer you need to have the corresponding private key and the most sophisticated and most finely grained access control mechanism that we have implemented in swarm is access control try or act for short this is basically a efficient organization of access control lists and in these lists you can have users of both kind so both pass phrases and public keys in such a way that there's no information disclosed beyond a upper bound on the size of the ACL so anybody who can access the access control tri can have a upper bound on how many items are there but beyond that they have no clue how many users can actually access it and who those are even if you are one of them you can only check whether or not you're one of them but you cannot peek into other people's permissions very importantly since access means ability to decrypt you cannot really withdraw access what happens if you exclude somebody from the access control list is that they're not gonna be able to read future updates they're still gonna be able to read what they have read before especially if if they have cashed all the keys that they have access to but if the resource is updated then they won't have access to the update anymore only to the old versions but it's also a very neat property that if you change the act you don't need to re-encrypt the content because we're using multi stage encryption and you basically only need to re-encrypt the reference that I talked about earlier the 64 right reference also granting access extending the access control tribe is a logarithmic operation so if you want to add another party to your access then the operate the number of encryptions that you need to do is a logarithm of the number of grantees revoking is a little bit more expensive it's linear in the number of grantees precisely because you want to reintroduce cryptid reference so that the those whom you have excluded will no longer have access to the updates and finally I would like to draw your attention that the granularity of this is very similar to that in a UNIX file system so you can have per directory permissions per file permissions and so on so you don't need to have an entire swarm volume under the same access control so before I finish I would like to give you one warning and this is the last thing I'm gonna say and hopefully this last one you remember that if you're dealing with access controlled content please please don't use public gateways run your own swarm node because if you're using the public gateway that Gateway will have access to the same access controlled content and you might not want to have that so thank you so much [Applause] alright so next up is Antone about observability hello everyone so I'm going to talk about observability in the last few months we've implemented a lot of observability to things form and also instrumented the code so that you can use those tools what is observability serve abilities basically answering the question what is nice from not doing at the moment and if you are an operator of let's say a larger deployment like us for the swarm gateways what is my food deployment doing so basically an aggregate view of is your network coyote and basically how our abilities so observability inform what do we mean by that so there are three pillars to it so logging metrics and distributed tracing so we do aggregate logging on our soar nodes so that we can basically trace requests from one to another we do matrix aggregation and statistics and we also do distributed tracing so cross node propagation of traces logging in matrix even though they're very useful they're not very interesting they've been in the going to VM code base for a while now what we introduced is the open tracing framework and we instrumented the holes from code edit which is basically vendor-neutral api's for instrumentation so what do we actually measure with those tools as you can imagine the infrastructure metric so CPU memory disk utilization so in case we have a backings form we can detect like memory leaks or CPU starvation if we have let's say goroutine leaks and also very useful for us is the application matrix so basically we track how many errors happen how many warnings and other types of counters when I say errors I mean things that developers must look at so let's say that we have unit I mean we have a elaborate unit tests and integration tests food that catches a lot of things but errors is what might happen saying production on our public test net and that helps us debug problems for which the test suit is not enough other types of application matrix number of peers per node so basically the different def be to be notes that your node is connected to different protocol messages between the peers and pretty much anything that you're not is running and that you want to gain visibility into so I'm going to give you examples of how those tools look like so that's one of them we're using the open source here girl it's a tracer that hooks up with the Opera tracing framework and here you can basically see a request that propagates from one's form not to another and helps you get understanding of the underlying protocol I have to mention that we do this only for debug purposes obviously we don't run this in production and you shouldn't really run it in production but it helps for development purposes and if you're building on top of swarm these tools are available for you to use and basically gain visibility the other one is the aggregate logging so as I said we have request identifiers so correlation IDs and we can trace requests from one internal API to another and also track them across different nodes all those tools are available within the repo there is swarm readme so if you're building to some type of storm this will be highly useful for you to answer the question what yours forum is doing can basically gain visibility over it so next yeah thanks that's actually I mean it sounds you know it sounds just like looking at logs but that's actually really cool stuff when like you see an action triggered in one node on to the next on to the next time so next you can track like the action throughout the entire network if you miss that that's what this is doing so it's really cool view of what's going on talking of moving things from node to node our next speaker is Lewis about PSS hello everyone how does this work the green button to go forward and the red to go backward this is pretty clear hello I'm Lois Holbrook I'm responsible or maybe a responsible party of the implementation of PSS and PSS as Erin said earlier messaging platform that piggybacks the doesn't work or that green button I see yeah that piggybacks to swarm routing mechanism to send ephemeral data across the across the network enter node messaging and what does that mean well it means that we that we increase the efficiency of delivering messages and therefore it comes at the expense of secrecy so this answers a question that a lot of people answer to a lot of people asks and really about why do we need this PSS thing when we have the whispers and whisper the messaging platform of etherium well that's your answer that is whisper primarily focused on the property of privacy in darkness PSS gives you the possibility to actually efficiently efficiently root over the over the network and since we all know that there are no such things as ninja mail man you kinda have to choose between one or the other all nodes and swarm take part in this routing so and it's also unveiled by default so you can't deactivate PSS from swarm as such so what are the general features of swarm well first of all this is not really a feature but I already did a couple of talks on PSS they have gone last year that's why I won't go into technical details about how it works but at that time it was in a very obscure branch in development since then it's actually merged into main code base so that means that when you download a swarm binary this actually has PSS and stuff in there it also means that it passed some tests which is good news right so what does it provide custom luminosity so I said the DSS gives you the possibility of efficiently routing so it means that what happened now yes I thought well anyway so it gives you the possibility to define exactly which address in the network you want to send the message to in this case it will it will it will reach the message in maximum of a logarithmic ops of the address space but it also gives you the possibility of partially selecting an address or not selecting matters at all now not selecting a dissolve would be gave it the same consequences whisper it would be spread all over in the network and whoever can decrypt the message for example list and tend to do so the recipient still no slide I don't know what's going on and oh I see so I have to remember what the slides are that's pretty cool and parcel addressing could be you know for example you send in that a message to Prague or you send a message to the Convention Center in Prague or you send a message to Luis at the Cancer Center in Prague anomalies Luis because I'm selfish that's my name and I know there happens to be another Luis here at the conference or you could say Luis with the fried egg tie at the convention center in Prague which probably pretty much would narrow it down to me and that would be the full address oh they're back and I let's see we can go back right swarm has the PSS has built an encryption by default so it means that it handles generation of keys and also sending autumn at all the encryption happens behind the API there are pluggable called handlers which means that for every M so every message has a certain topic and associated to that topic as a recipient you can register code to be executed when you get a message for that topic and it can be any number of handlers so it's extendable very flexible there is a handshake module which enables you to do diffie-hellman handshakes and exchange session keys behind-the-scenes happens automatically the keys are valid for a certain number of messages and there is also a buffer of keys so that if you run out if you the exchange the refreshment of keys happen before you run out of keys so that you won't be stuck if if if so it's less likely get stuck protocols yes so it has a framework so PSS has a framework for inter machine communication it also so happens that it's assigned away so that if you have already deal dev p2p protocol that exists between two directly connected peers directly in the sense of TCP you can with minimal minimal code actually port this to PSS so by newest features I mean so all the stuff that I said now I kind of said last year but again the difference is it's in the code base now right or it's in the incent the main binary this is new since last time but also part of the merged deduplication last when we last heard from this yeah you had the risk of getting the same message twice and BSS this is now much less likely when messages are exchanged on the PSS network they have an expiry and the nodes will also have a certain period of time where they don't allow the same the message an identical message to pass or an identical message to go in so that will make it less likely not a hundred-percent guarantee but actually less likely that you have to handle the duplication messages yourself raw messages which means it doesn't you actually have a possibility of handling the encryption outside of PSS yourself or just send plain thanks messages if you will if if the secrecy is not important and last of all a notifications packet so it's basically means the protocol which makes it really easy to subscribe and publish notifications and it just provides a kind of a channel which you your code can pump stuff into and then a very simple protocol of where notes send subscribe message to know that's publishing and then automatically gets notifications yep ESS for example in combination with swarm feeds as we heard from Xavier this can be a really powerful feature yeah pool as well as push now there is an API of course most applications in swarm use are available through IPC WebSockets HTTP and CLI PSS unfortunately or it's only IPC in WebSockets because we need subscriptions to get incoming messages there is API is of course all documented unfortunately though it's kind of low-level so if you want to interact with it to kind of looks like this at least that's what we provide fortunately there is a community out there that are also collaborating to sperm development among the mainframe that have created a JavaScript library called Erebus which as you can see makes it a little more human-friendly to interact with the SS as such so for the documentation swarm guide of course Erebos also has some documentation page and at the bottom is my repository where there was in tutorial except more code examples for tutorial for PSS i would also like to mention that victor is having a talk later today on the higher level vision of PSS and how it can interact for the feets as i mentioned also access control as Donghae was talking about I can't remember exactly the where it was or when it was but if you follow this link I mean everybody takes pictures of the screen right then these things come then you probably can find it and I think that's it [Applause] so yeah the talk is at the fluence meet up tonight at six o'clock there's going to be four hours talking from line life pier new cipher fluent sentence warm and I'm going to talk about my vision of how to how to basically reformulate communication protocols yes you heard it well all of them in terms of in terms of tools that are available for web 3 so basically they have communication that's decentralized incentivized and secured thanks to our friends from beefy for the motto now talk about like notes so we realized early on that we need to distinguish between two types of notes once that that are constantly online and therefore a reliable force for storing chunks and the notes that have high churn basically that you just open your laptop and close it and we're all or you have your mobile device and in general that's we call this this is distinction light load as opposed to full load or light mode of operation maybe you can call it it's basically about mobile support let's let's let's simplify the things so it's concerned with not only the high churn but also the difference in the in the resources of the environmental resources that are available for swamp solo band it's usually probably ole ole ole memory as well and short on disk space and and there's various issues that we came across by when when trying to support the mobilize the some of them are kind of boring platform specific issues I'm not gonna talk about the permission issues and so so in general what was the was the chance of a restricted resource client to to to connect to swamp where does this post potential gay twist like it which that might be accessible through adders like for example the one that the this one team were the Syrian foundation is running the Swan cluster which we run a public gateway this is not intended to be long-term this is kind of a just fun for for easy access now through the development that obviously when the incentive is Asian is coming on the main that is gonna cost money so we might only do that for marketing purposes for short term but and as you heard as well you remote gateways are not appropriate for for hosting to to retrieve the encrypted content from there's also privately free solution so you run your own remote node for example then that notice is supporting that and what we are going to talk about here is just light mode of operation which is like how it natively support this this type of operation where you have low resource environments so basically we distinguish these two doing 200 types and in the in the zero phase what we already accomplished kind of almost almost deaf ready is that light nodes when like odds are are treated differently they are not saved in the address book of your peers they are not dialed so actively looked for they just accepted as a connection they don't state since they don't store Chiang state they also not doing syncing process so they're not sinking but taking part in the in the transport of chunks to their to their local neighborhood where they belong in the distributed storage of so they're not serving retrieve requests since they cannot store and and also be - in order to respect the redundancy properties of swamp they also not counted in the local neighborhood so once we have that we we are we gonna be a lot more tolerable towards high churn and especially if we set the the light node as as the default mode of operation and people will run like permanent nodes we explicitly have to set it to full nodes so this this will contribute to the resilience of of our network now I talked about the roadmap so what's coming up next we started rewriting the the swap protocol the swarm accounting protocol which is basically the peer-to-peer for tenth accounting that's serving as bandits incentivization and serving to regulate and and basically optimize bandits resource allocation and we reported the odd code and and actually we really were quite quite a bit of it and simplified the code so that's that's what's gonna come up next in in a few releases time we have been planning on introducing spam protection so basically protection against people flooding justice for network with chunks by having to attach like making it making sure that people have to attach a proof of burn so some sort of basically proof of work to the to the to the to the chunks and and also the first phase of the ratio coding is going to be implemented very soon which takes care of basically it's not the racial coding so that this two layers vary according one would be to take care of catastrophic loss and this one of the first twist is just guarantees basically gives an upper bound to the to the 3d Retriever latency in the network so let's let's that scene in the immediate roadmap also under on the immediate roadmap that we were kind of moving to the new cluster setting so we're gonna be able to spawn really big scale scale scale in the network test in a real setting so that's Anton Anton and Rafael are working on a kubernetes cluster that's that's gonna be able to start spotting sense this end and therefore neural network testing framework is going to be able to test like really complicated emergent Network scenarios okay so what what si is on the roadmap well as usual we still very heavily researching how how exactly we do storage insurance then we mean let me now have like a two layered system that's that's we're planning the the substance window frame work which is generic framework for four basically are at the Coney station or networks and and also a contract support form for service level agreements that are challengeable on the blockchain so it's kind of a our take on state channels combined with the blockchain as a judge paradigm so the sorcerer and since we know the contracts which has been developed and and tested with by Bhairav you'll hear from from Riyadh so our it so narrowed not to finish this this part and and introduce the the option to have like service networks and so on the Savi research on on databases or swarm and we we realize that there's a lot of usability issues in general with swarm so so we're gonna put a lot more effort into into developing and supporting deaf tooling so basically bindings in various language then most prominently provide better JavaScript support together with some of my friends and and allies in the ecosystem so that's about it for the update yeah Thank You Victor so I said that's about it it would be a mess at this point if we didn't mention that there's more to swarm than just the swarm team the community has grown this year we've gotten a lot of contributions from third parties the always awesome mainframe status has been helping us with the mobile devices we've got volcon databases data fund is awesome they make orange sweat shirts and other orange gear for us that's wonderful guys so thank you everyone on this list and everyone else who's joining the swarm community this is the team at the orange summit last year we have once warm orange summit every year we're hopefully have one next year so you're welcome to attend keep a lookout for that here's our contact we have I've already mentioned swarmed of the theorem org we've started our own Twitter and reddit called eath swarm I promise there's not a lot in there this is a low environment but that's where you get updates will make release announcements and when we have a talk somewhere or when our or in some it is coming up that's weird look that's our Gator swarm at a theorem org if you want to talk if you want to write to us directly otherwise come find someone in orange at the conference and come talk to us thank you very much [Applause]