Viktor Tron: Web3 Communications | Swarm Orange Summit Madrid 2019

so thanks everyone for coming and thanks for giving me the opportunity to kick off this summit with the talk that I hope will serve several purposes once first of all I would like to give you a bit of an overview of the vision of Swan so we walk through the components on the other hand it's much better now I guess so but at the same time I would like to present a little bit novel and and little bit and rehearsed conceptualization of house form is basically be at the top from from from the ground from from most simple simple components and and give you a give you a clear idea how how the different modes and and modules and components build on each other and what what constitutes as the course warm layer and and what but products are layered on top of it for the purposes of this talk I I mean the word communications in the broadest possible sense so it includes in its scope required quite a number of things including publishing and and consumption so it's it's a bit understood the the motivation that that we got for when when we're building building web 3 communications services is basically centered around censorship resistance and privacy so so the the idea situations that when I when I present you with with the will be the particular services and how how they are implemented or can be used in swamp the the idea approach would be to do to evaluate them in terms of their privacy and censorship resistance fishes and and wee-wee so we can focus on that and some some of these will be will be picked up and and we go into details during workshops so so just just is to give you an overview in this talk and of course we want to when we want to in the in the in the light of our motto it with scalable sustainable and sovereign infrastructure we may be obviously want to want to provide tools and and and and and communication services that that give you several ownership of data that you that you produce and sovereign right to the grant access to this information who is hosted on an honest scalable and cheap infrastructure which allows for messaging services that that we need to be to be autonomous and also have it have it as a system which is economically sustainable through built-in incentives and in the incentives obviously ultimately connect this peer-to-peer technology with the with the with the Syrian blockchain generals the consensus so just just to give you give you a crazy idea what the scope of this talk is so and I'm really I'm really and well that this is this is very ambitious but this is basically what I'm claiming that all these all these different things that you do on the web the the basic back-end infrastructure that you need for that is is coming in a fully decentralized and incentivized and secured away from swarm and hopefully hopefully I'm gonna show you show you how how to build such things and so on so as I mentioned if you if you approach the communications modalities from the point of view of privacy and censorship then then these are the these are the dimensions that you should evaluate them in but weather and and and and here here in as I as I say communications use this in quite a broad sense so when I say sender and recipient which is the normal term used in in the context of messaging you can you can just substitute the publisher consumer in terms of the consuming static content so it's kind of the same thing so in terms of in terms of the communication modalities you it's important to recognize how like what the relationship between the sender of publisher and and the recipient and and consumer whether whether the sender needs to know the recipients identity or what what aspect of this identity is revealed to them there's an important dimension I call imposition which means that is whether sound whether some communication modalities basically for its forced on the recipient in a way it's a bit like advertising it's put in your face even if you don't want it if it's if it's impossible then basically you if you have a public email address for example that's impossible in this sense because you can you can just spam it with with a message you don't need to know that the the recipient the sender to get an email from them so and and and also interactivity is very important so to evaluate these different communication modalities in terms of their their required throughput and and how much latency they tolerate typically and in general what order efficiency metrics are and and and most importantly in general in continued setting of using a communication modality how much information is and what what information can be guaranteed to her to be an to remain confidential so let's let's move on to some components these designer basic components are products that we that we currently and advertise for mass and and that's what the components that do you think swamp comprise itself so let's let's go from let's go and via the map from from ground up so what is Swami so basically Swami is just a bunch of nodes in a network which the assumption is that if they have their samantha lee connectivity protocol in in this case it should be an authenticated encrypted multiplexed framed stream which in in our case historically statisti are a pikestaff peer-to-peer protocol that the etherium is using which is runs under tcp/ip transport this there's a bit of a link to to dimitri in the audience at the leap peer-to-peer could just replace this at some point not so so this this is this is basically i consider this part to be outside some of it so we can rely on this this that there's such such an underlay network but we don't consider spare part of this one so so we have this node switch which can establish this semi permanent connections channels with with the underlay connections and and they can connect so the question is what should they talk about if not well they should first of all they should talk about how they can organize an overlay network so this is the this is the hive protocol so first first day they organize them themselves in the in in the in overlay network what is an overlay network is when when nodes generate some some key which it defines their address in the in the overlay namespace this is basically a 32 byte 256 bit integer address space which we can define an extra distance metric on it and and with this distance metric you can form academia topology so academia topology is is basically a graph a particular graph connectivity pattern which guarantees some sort some properties of graph traversal namely that you can have route between any two nodes and and the number of hopes that you take is is logarithmic to the size of the network this is a based a very strong strong property because this is basically allows internal messaging with with relays so what what what what we can do here is categorize the types of types of message villages so this does obviously want the three-way one when we when you just broadcast the messages to your peers so now again what you see in the in the in the graph here is the overlay Network so where I have where I have the edges these are overlay these are these are underlay connections so to the semi permanent connections and and the the connectivity pattern poorest academia topology which means that apart from apart from gossip which which you can value in a particular area you just gossip and broadcast a message to all the other nodes you can have efficient routing with with relays which we call forwarding Khatami I mean and this is this is central to the workings of so on and it's important to notice that it's foreign in Columbia allows basically makes message passing and sender or at least at least it makes makes the standard possibly deniable and and and and difficult to get but but it's it's it's its recipient if recipient is revered because that's that's what makes it efficiently route so so this forward in Columbia which I indicate with the with the yellow arrows the message passing was to several hops is central to due to the workings of Swan so the the other type of message message passing modality is when we combine the the Academy are acting with with gossip which is who a neighborhood addressing is it's basically when when you you do rousing up to a certain after certain neighborhood and within the neighborhood you do gossip and and and there's yet another modality which requires when you when you when you do message passing with forward Academy Ahab's then then each hop denotes open a little ticket so that they remember that some something went through them and they pass back the response so basically we can define round trips with this formatting Adam yeah it's it's also very very important so so so far so far so good we have messaging Ruthie Ruthie and a neighborhood ad estimates forwarding Adam Lea so we already have a component we just stick just tickle API to it and and combine it with some encryption clever clever encryption system and what we have is basically PSS so non messaging system so you can see a little label here so in in case of if the case the overlay address of the of the recipients is fully fully disclosed then then you can have direct routed message to one node to another if if the recipient he is known to the sender then then you can do encrypted messaging it's not known then then then plain message if if the underlay addressed is only partially revered or not revered at all not revealed at all and then basically PSS gracefully falls back to the whisper the original whisper which gossips done the message and it's picked up by the owner just by virtue of the owner being able to decrypt it with the public there's a special case on the right bottom corner even when the no public keys is known or given and and there's a there's a unencrypted BSS message sent to the neighborhood and shipped around in the neighborhood so this this this is used for meaningful for service service messaging so mystery notes so what does that mean I always talked you through it in the in the in the context of the next component but first first let's see what PSS can only be already be used for it can be used for real-time chat for notifications and service no messages so this is this use cases that have been heavily used in in in so on so so what well here comes the storage layer so it's not so so far we being concerned with with a bunch of notes that want to message each other and they do that addressing because they want to look for another node but in fact in swamp there's other other creatures that that live in this in this network it's not only the nodes but it's actually data and swarm can be interpreted as a distributed chunk store because this data are stored by the by the node staff that are closest to them what does this mean it means that chunks should have an address and the this address is is it's meant to be interpreted in the same address space as the as the Oval address of nodes which means that it makes sense to say that a certain chunk is stored with with the north let's close us to them that's that's exactly how housewarm distributes them in the data so far so far it's quite similar to the dhts in a way but this is a very very big difference is that in DHT is normally its information about the particular chunk that that that you find at the address that the closest node here here in swamp it's the actual data and and since since the since the chunk size is fixed and and and the hashing that that determines the address of the chunks is uniform or as long as these two properties hold then we can guarantee that that this one will be a lowest balanced storage over over the nodes and now now you can see why this is this is important so this one also it's very important that when when the chunks have to be retrieved or or sent to the place where they belong where they where they taking custody then then it's important that that we validate that the chunks are varied their integrity just because that that that means that you cannot spam the network with we just be junk junk chunks so it's very important that you have a scheme which is which is locally makes makes a chunk locally verifiable and and this was very important this assumption that a chunk wants a chunk is always a chunk so once you see the address of the chunk then then you know that you've seen it and that's that that's the chunk you've seen sits there's no there's no mutability for from the chunks perspective so there are two ways to to guarantee this one is the one is taking the content and rest so basically when when the hash when the other rest of the chunk really depends deterministically from the content of the of the data blob this is this is done with with with her.she so on the level of Chinese tea sister Chun cash but this is another tricky way which is proof proof origin in I will talk about this in a second so but first let's let's let's talk about this is in so here as well so I understand we talked about storage and and because no it's no it's store so the actual charge that that they that are close to them but in fact it's in fact using these chunks for anything is basically communication because what you need to do is send send the chunks to every where they are should be stored in the end and maybe maybe get back some receipt some some statement of custody and and in for retrieval you are so it's all sort of communication round-trip because you send every three requests and yet delivered the chunk back so this is these are exactly the the types of message passing events which which I told you that there's one modality well and you see the round trip like forwarding and and then back passing so the examples for this are this storage and retrieval so it's basically we can we can we can characterize it like this so basically so think of it the timeline timeline going from from up to the buttons so basically the top node is the publisher so it publishes a content which is the which is the middle node and that content is consumed by the by the nodes in the bottom so this is just illustrates that this type of publishing content is simply a one-to-many communication modality and the two errors just indicate to you that these are typically round tip so the the publisher publishes the content chunk to the to the neighborhood where it belongs and gets back statement of mastery and and the pattern nodes the the sender retrieve request to the to the actual address of the child which is picked up by the local that stores it which delivers the the chunk to them which is the green arrows pointing down so the important takeaway message is that is that with this with this case this week we can cover all kinds of static content storage and delivery so we can we can have blogs websites and apps I am she I'm cheating here because we're talking about chunks chunks at the moment and of course you need a layer where you can talk about files and and and insights and I will touch on this in a second but basically when you when you retrieve for filer or a collection ultimately on on the on the lower layer this is what happens so ultimately ultimately really requests chunks from this one and and you upload chunks to this one when you when you consume and publish respectively so but but one important message takeaway is that since since these publishing requests and and and consumption requests you use the same forward in Kalama Lea thing is basically inherits some of these properties so it basically means that you have anonymous anonymous publishing and anonymous consumption so you can do it anonymous browsing you know in a way so just just just at how much time they have thank you so so to top it up with will act this is another component like a basically adds a layer of of access control over over over swamp chunks and it's it's basically treats the encrypted content it has a method of distributing the decryption keys that that the chunks to two particular grantees and I don't want to go into the details it's a very very very elegant solution you can how you can stick basically access control over over content that's that's published this one I'm just mentioning this because the the use cases will become clear so when we talked before I said that with the storage layer you can just have you know public naps well make you pay the hour or any kind of blocks implemented with the with the expert on you can have you can implement a retreat received access forum like you can have the authenticated login area for a website you can have a private group check east history and and okay private sharing so basically you can create create an act that stands for a sense for access control and try so you can create create an access control structure that that basically controls who we want to grant access to with your content which is basically private sharing so now we come to feed which is hobbies hobbies I find I find this is quite quite an interesting construct because it's it's so simple and yet so powerful so so feeds is used another modality tool to to make sure that a chunk is immutable and and locally verifiable how does it do it instead of instead of the the chunk hash for as a Content address it's basically says that the address space is partitioned in a way that each user that has Blackie has the right to occupy certain parts of the address space so for example you can create you can create an address from from a topic and and and your and your public he has together and you have the right to occupy that that part of the of the chunk space give given that you you sign off on the content and the assumption is that here is very strong assumption is that it's by the Swami's it's it's it's assumed to be to be kind of immutable or unique because because it's it's it's at the Publix in other words it's a part it's the public sense description discretion to do the double sign on on on a particular address and therefore basically screw their screw their own feed but but but if they don't do that because they are the other ones let's control the feed then then they then this can be it can be guaranteed that that for one particular feed address you you have longer particular content and this is this is this is important because because because you you can you can rely on the principle that once you see the address you know what the chunk was you know what the data was that that belong to that address and so what what what feeds do is basically being being a kind of temporal continuity to to Chauncey so what what this means is that I'm I just put it put it in a simple simple diagram like that in in that the the top publisher note basically successively publishes content and this is this is successively read by by the consumers and and and the feed feed addresses basically map map some sort our succession schema or versioning time stamps to the parts of the parts of the address space and therefore it can implement implement mutable content updates so basically with feed you can you can track changing content you can you can you can link together the the the unrelated content addresses off of your changing I don't know website to one particular feed and and basically used use the feeds to get the latest update or the history of a particular website thing you can use it for for to update database state and and you can you can use clever clever topic topic naming conventions to do to link feeds to the certain certain relational information so for example for example with feeds you can you can implement Facebook Likes website or you can you can publish upwards upwards for a particular post or even comments on a particular post and most importantly that's the first one this is in this list they can they can interact at zero the communication channel because if if if two parties of band agree opened on some or some other communication variety they agree on a particular feed succession scheme the fit succession scheme means that they agree that at any point in time they know what topic and name the useful for looking up the the looking up a chunk and and they know what publicly the use for the session which which mobility has two can be can be completely talk not related to to the public identity of your of the party you communicate with this basically it's a zero communication general so what what this means is that is that it's it's a bit like finding finding around rendezvous points from from from from the two nodes that I put out the message here in this corner and you later later come and pick it up we just have to know have to know which match which marked in the corner I put the message and if we have a scheme that we agree this morning I leave you the message here do they and the eye in the door then we can have some some communication channel so we're gonna go into into more detail about that in the messaging workshop and and I think Louis we talked about this scheme in much more detail and hopefully we're gonna have time to analyze a bit of the security and privacy properties in a bit more serious way just as a piece and so most importantly the the incentivization so in in this talk now now I don't want to go into too much detail by the incentivization but relating to how I how I introduce this working installation should definitely focus on how incentivized relaying to message reading because that's central to how how the network operates and we we have a solution to that which is which is the peer-to-peer accounting system where where you can count service for service provision surface or service or exchanges so basically when when you pass me a chunk that I need I can't do I credit you one and when when I pass the chunk to you then then then it's the other way round so equalized and and if if the consumption and and and and provision of a service is roughly equal smartest and they get away without without too much exchange to them the value stays with the user if if the consumption tests in some way then there's a there's a scheme to to to to to settle that with the with a kind of with micro payment solution really basically be sending a check that you can cash in a child book contract we're going to go into this it's more bit I think in in the context of rafts talk about the swaps and swindle framework later today and I'm sure we're going to touch on this within the incentives workshop which is also today so we got me gonna talk a lot more about this in detail but just just to the takeaway message is that very important that we have a very scalable chain cheap micro payment solution to incentivize relaying and you if the if that is so then basically we have the the sustainable scalable and and and so for an infrastructure for for for further for the communication at least under under under level of chunks basically so this is this is about but I was quote really the the the the bare minimum that that the core of Swan if you have this then this is basically the the the necessary network layer that we can implement is this this network protocol and then then this can already go a long way and let me continue it with little bit more so so over chunks there's also an extra layer so you can you can you can interpret chunks is not completely independent of each other but somehow assumed that this is the structural relationships over them this is a whole new layer which you can you can even think about this whole layer as as application-level in actual fact we most most of these solutions are rose to the core of Swan and therefore therefore considered the core part of it but was in fact if if you if you want to think of swimmers just just the chunk store and the chunk handling that I introduced so far then you can think of it as SSM SMX tradition so what what was the general approach here and I only I only just realized this myself recently how has this have this things fit together so very new so basically we had we had a talk with with Ella's verbena in the end approximate where we represented this general generic result how how the distributed chunk store can basically map and wrap and used to represent any functional data structure so how the set backs functional data structures are basically notes and pointers and and you can turn them into into distributed normalize data structures which means that nodes are interpret the nodes are serialized as chunks and the pointers to to to know from from node to node is basically the chunk reference or use the chunk a sure if it's encrypted and unencrypted reference as the point and therefore you can you can immediately get to Americanize distributed data structure now what is what this implies is that any query query that you do by my traverse traversing this data structure so for example think about the tree that you want want to get to the leaf off and you have to traverse the data structure along the branches but so this this this process of traversal basically translates into successively of chunks and in and between the success successive retrievers of chunks you just interpret the canonical serialization of node and and see which which of the pointers is it that you need in the in the next hope in the next round of the torso so I give I give use cases for this some of this already exists and some of this I can should exist so so the swamp ash tree the represent eyes is already an existing example so so how have you represent files in in Swan is basically bye-bye we're building a localized three with-with-with intermediate nodes just packing packing the the hashes off of their children so it's basically a super simple super simple representation of over three using using this this division or Christ structure and when you retrieve a file or or do do so many ways when you retrieve something bigger than then less than one chunk this is exactly what happens in swamp that you you basically take take the route ash of the of the file that specifies address and unpack it and see what what are what are its children you successfully will retrieve them not too much you get to the date chunks and and you have to fire the same happens on an even higher level when you organize files into into directories or collections this is how we this how we do virtual hosting as well so basically this is how you how you represent websites and so on so manifests are also also this kind of this structure that that you either to traverse according to certain rules how you interpret chunks and how you how you want to traverse them so manually you retrieve a fire from from a directory then you take the route manifest interpret that chunk as a note as a note in the in the in the in the functional data structure you take the pointer etc so obviously so so the theorem state try could be another example which could be directly mapped the Patricia try that I actually mapped to them to the designated this section as one and another example I will not go into and and and parallel to this of course the the the data structure operations is ready to random access to files typical example of or the traversal operation pilot you were to manifest all right when for example when you want to get an account a particular account through the theorem state that's also a traversal in the over these data structures the interesting thing about this this this this interpretation is that is that if you if you have successive retrievers from through two chunks interpreted as as no dinner in a functional data type then actually traversing the quiz by traversing can be interpreted as network protocol and and proved provable queries into data structures and be interpreted as as as forwarded requests and on the way back they accumulate Mercker proofs so it's basically outsourcing outsourcing and creating massive data structure so one more point is aggregators so I can get a seasonal is another concept where basically it's it's it's it's it's a it's a kind of node where several sources publish publish to it and it it basically process process systems plus those input and and and perform some computation on them and publishes the result and that result is consumed by several parties on the bottom layer so so this this construct is basically using using the same tools that that we had before feet and and and PSS messaging they they can do in the generic formulation of that is that obviously then they need convex resistant data types so that so that there's no consensus problem basically on updating a data structure and therefore they can do simple things like indexing aggregating merging merging like timestamp indexes or counting and and they can also be used so in successive iterations to do some MapReduce operations etc so this is a quite powerful concepts and the they could they could in principle be used now because if you if you if you program the application logic it's using basically the same lower-level infrastructure as well now of course of course this these these aggregators are typically ones that perform something that's that's not immediately seen you don't immediately see the result so it's kind of a promise sorry type of activity which means that ideally it needs staking to be to be trustless so if you have service provision by by by Minoans without reputation then you have to have to prove ability or verifiability of their computation this is this this can be done with with this with the sponsor and spindle games and and aggregators can be used for various various real-life cases so aggregator scam can be used to for example process create create an index for you for for a public forum for for for forum where we're we're polling polling the individual participants feeds for the contribution not scare so for example think of a think of a ready to and Swan you typically we have the forum comments be indexed by an aggregator it can also be used for offline mailbox in a special use case with that is is that we I think we've it's upon that we've done it the other day that when you have pinned content and when we implement print content then when when somebody doesn't find a particular content and and that's there's there's a indication that there's a content maintainer who who is responsible for having that content then then you can you can use this aggregator to to notify them without them revealing their their their actual under layer overlay address so the user can notify them through their public key this is this is also powerful use case and of course it can be used for for counting likes and outputs so this is this is a important ingredient in social network context where you where you need a gracious of some or some result so these are typical use cases where which I would describe as multi-party updates to a database so when we have to solve the problem that several part is one to update something and and we don't want to go to the blockchain to do that of course of course when the computation is is more complex and there's no conflict resistance data types possible so there's this there's this genuine transaction ordering problem for census problem then you have to use the blockchain hello obviously but you can also use it for very powerful things I almost put Google on swarming as the last point but I changed it to food with textures sounds bit less pompous but basically that's that's how I use accumulate full text index over spawn to to basically and chain chain chain is aggregators together okay so just to wrap thank you so the the claim here is that swamp can provide base their infrastructure for communication modalities in you do it to be used in in diversity context in the Webley context it means that these modalities should be decentralized incentivized and secure basically they should come from to the to the principles that we set out in our most oh that that this infrastructure should be scalable sustainable and sovereign and and it's basically infrastructure that requires no no middleman and all the services allow no barrier of entry and and participation for Peru and the the claim is that this is really you can build it out of this very simple building blocks of stone and hopefully hopefully we see the richness of such applications which use this this these ideas in in the future more and more often so thank you very much for your attention