Scalable Responsive Đapps with Swarm and ENS

[Music] hello again so this is going to be a longer talk the intended audience are developers who want to build scalable and responsive distributed applications on top of swarm using DNS so first I would like to talk a little bit about the distributed application paradigm itself because in many ways it's a very radical departure from the client-server model that is prevalent on the Internet today and with which most web developers are intimately familiar then I'm going to talk about the various challenges of scaling and what the bottlenecks are then we are going to discuss how reliability issues are being addressed in a dynamic decentralized and essentially low trust environment of a distributed application we are going to discuss availability of content which is a very important issue and finally I'm going to talk about how a distributed application can be operated and developed and improved over its lifecycle so first the distributed application paradigm distributed applications are much more heavy on the leaf nodes so endpoint nodes perform a lot more work than clients in a client-server applications very large part of the business logic are actually done on the client side so on the slide you see several possible nodes which have their own requirements but which can be the nodes in a which can be the nodes of a distributed application so it's not necessarily web-based there can be in native mobile applications Internet of Things devices and also we need to keep in mind that key and identity management is sometimes outsourced to this hardware keys and we need to take that into account the backend is typically generic and I will talk a little bit later about the deeper reasons behind that but the kind of backends on which we rely are the ETH and less protocols for consensus so it's either a full node or a light node on the etherium network so each node needs to be an ethereal node we use swarm for permanently storing large amounts of data and accessing data and we use either PSS or Whisperer for node 2 node communication so all of these backends are entirely general-purpose they're not application specific and even though I told you that there are no clients and servers in a distributed application you should still keep in mind that the contribution of resources is by no means equal so peer-to-peer applications are not necessarily not necessarily balanced in terms of the use and contribution of resources so even if you think about BitTorrent there are Cedars and leechers and in the general terminology we distinguish between consumers and suppliers of resources and of course each participant can choose and even change over time to what extent they supply and consume resources but typically consumers are are characterized by high churn so they leave the network and join the network frequently you cannot rely on them staying on the network for a prolonged period of time typically they spend accounting units whether they are ethers or special-purpose tokens and they have resource limits which in some cases can be quite severe suppliers are typically there in order to earn accounting units so they are providing a service and expect to be paid for it they have a launcher so they can still leave the network and join the network but they don't do that frequently they have high availability and they have adequate resources for providing the the services that need to be provided for this distributed application so in the next part I'm going to talk about the challenges of scaling distributed applications and identify bottlenecks and proposed ways of dealing with them so the most important bottleneck is the blockchain of course we have heard wonderful talks about how scaling problems on the blockchain are going to be solved over time and in a ideal world in which we had a essentially infinitely scalable blockchain everything I'm talking about would be unnecessary because we could host the entire distributed application on the blockchain keep all the data on the blockchain and submit every transaction that the application ever does directly to the blockchain but there are two reasons why I feel that the techniques that I'm going to introduce are still relevant one of them is that the scaling of the blockchain is a slow process and it's a much harder much harder challenge than scaling the particular dab bottlenecks that I'm going to talk about it so people want to create scalable applications even before all the problems of blockchain scaling are going to be solved and secondly because the techniques that I'm going to talk about are actually useful for scaling the blockchain so for example the initial motivation behind swarm was to store his historical record of the etherium blockchain so archive nodes which store the entire blockchain history as far as I know there are very few of them maybe none and but we still want to keep the entire history of etherium available and swarm is the perfect vehicle for that so all the techniques that I'm going to talk about can be used for scaling various aspects of the blockchain so on the slide you see in what ways the blockchain constitutes a bottleneck so because all information needs to be replicated over every node that means that storing information on the blockchain is very expensive so you don't want to burden the blockchain with too much information also the blockchain can only be updated at the block time speed which is on average a dozen seconds or so which is not exactly responsive and you have to pay for submitting transactions to the blockchain so if you do updates very frequently or your updates are very large then it again becomes overly expensive the other bottleneck that we are going to encounter are individual nodes so basically you don't want individual nodes to perform all the work of a particular area of the distributed application so you want to distribute all the tasks of ADAP or as many tasks of the app as you can over a large number of nodes that are working in parallel and also bottlenecks can be in network links so for example you do not necessarily want to broadcast every interaction to the entire network because then you're going to have a network congestion and in case of systems like whisperer you may need to pay in terms of proof of work for doing so so these are the bottlenecks that we're going to be dealing with so the storage bottleneck is best overcome by storing everything that you can inform and put the root hash of your data in iana's so the root hash is just 32 bytes and it can integrity protect as well as identify data of arbitrary size and complexity and structure meaning that no matter how much data you're updating you still on the blockchain you only need to change 32 bytes which is obviously a lot cheaper than changing megabytes the transaction bottleneck can also be overcome so for those of you who don't know EMS has resolver contracts so each time you ask Ian us what the particular address corresponding to a name is you're going to have a contract answer you and those contracts can be more complicated than a simple storage of 32 bytes in particular you can implement Radin style updates on els meaning that you can send those updates to all the interested nodes but you don't have to commit all of them immediately to the to the blockchain and yet all the nodes that are interested are going to know that the content has changed and any of them can commit it to the blockchain in case it becomes it becomes critical that a consensus is reached so this way we take the burden of updates of the blockchain and we can have in frequent updates to the blockchain and yet have immediate and very very frequent updates to the root hash of the data that we are using broadcast bottlenecks they can also be overcome using PSS so just as Louis had explained before me PSS can be used to broadcast messages to parts of the network instead of the entire network so if you have this pops-up model then PSS will make sure that even though there is an overhead you won't burden the swarm network with every message broadcast it to every node of course this only becomes a problem if your application scales really big so for many applications broadcasting is perfectly adequate and if you are more interested in darkness you can still use whisper but then you will have to keep in mind that you have a network bottleneck so in the next part I'm going to discuss challenges of a trade-off between responsiveness and consistency it's a trade-off because the faster things happen the more difficult it becomes to agree in a timely fashion so the consistency is only guaranteed on the blockchain but this consistency is a eventual consistency which is rather slow so you can see a back-of-the-envelope calculation that tells you that you will reach consensus in a time that can hardly be called responsive and you don't want to wait that much time for every update to be visible to all the users of your application so essentially you need to broadcast the updates to all the interested parties and then you only need to commit to the blockchain once it becomes important to to guarantee the consistency over a longer period of time so here I would like to give you a few examples so one one is a online discussion where there's some kind of content be it a video or a blog post or a picture that several users are viewing and commenting and the other is a massively multiplayer game in which different players are interacting with each other obviously submitting everything to blockchain is hopeless in both cases so in both cases you can notice that it is relatively easy to scope the circle of interested parties of a particular update so in case of a online discussion it is perfectly enough to make updates fast to those who are actually viewing that discussion who are participating in it and it is acceptable that others who are not browsing that particular discussion right now will only have the update much later and they are only interested in the final state of the discussion they don't need to receive every update at all times and similarly in online games there are typically locations and there are some players in that location and you only need to send quick updates to those players whose characters are in that particular location you don't need to send updates to all the players and the update structure is also more complicated than simply a hash because if all you broadcast is the new root hash of the swarm content then you're running into a responsiveness problem that it takes time for swarm content to sync even if we will improve and we will improve the performance of the syncing protocol it will never be as fast as a direct message passing simply because a much larger amount of data needs to be transported and it also needs to be stored along the way so if you only send the root hash then the response time is going to be much longer than the message passing time because you also need wait for the corresponding sworn content to become available for the interested parties and instead what you can do is you can also include in the updates the actual information that was changed so for example if a comment was added then in addition to the new - you also send around in messages and PSS messages the actual the actual comment that was added so that the nodes that are monitoring the discussion will be able to immediately update their local model and yet they can verify whenever the swarm of date happens and especially when the blockchain update happens they can verify that they haven't been lied to so this does not present a security problem it just speeds things up and also I would encourage you instead of creating very complicated update rules and then you need various ways to actually verify that the updates were legal or so they were the kind that is allowed instead what I would suggest is to do aggregation on the on the front end side so for example if we're looking at a online discussion then every participant would update the route hash of all the comments that they have sent to different parts of the of the service and whenever a participant looks at a particular discussion they should gather the participant the addresses of the participants of that discussion on ENS and then monitor the updates of those participants and independently of each other and do the aggregation in to a discussion themselves on the client-side so that's a much simpler and much more reliable way of dealing with concurrent access in the next part I'm going to briefly address issues of reliability and availability so here I would like to go back to the things that I said about about distributed applications whenever possible you should not try to build your own infrastructure try to rely on generic infrastructure because that way it becomes somebody else's problem so in order for a distributed system to work well it would it needs to have a sufficient number of nodes those nodes need to have sufficient incentives they have they need to have adequate resources each and all these things they are difficult so for example if you only use swarm as a generic information storing framework then making sure that swarm content will be available is our problem as infrastructure developers it's not your problem as an application developer and in general so there's a reliability kind of ranking between these services so ETH is the most reliable light client protocol is less reliable bzz is even less reliable and PSS is the least reliable of these generic services simply because of the popularity of the of that particular service on the etherium network and if you do something yourself it will be way way way behind all these on the reliability scale so also you should keep in mind that the only thing that is certain is the blockchain everything else is ephemeral so if there's some important state update eventually you have to make sure that it reaches the blockchain and finally you should be paying attention to incentives so in case if you want to enforce compliant behavior then the two tricks that can make things more scalable is instead of using pre-emptive measures so writing complicated contract code that can only drive the application mechanics as the smart contract executes that block time instead you can use reactive security that everybody updates the way they the way they can locally however if there's some dispute then you can turn to the blockchain and figure out who's right and who's wrong and this way if everything happens correctly you rarely touch the blockchain so that helps you scale so instead of using proactive security measures you use reactive security and the you only bother the blockchain in case of disputes and the final part of my talk I would like to introduce some ways of maintaining and improving and developing and adding feature to a distributed application it is very different from from the client-server model because in a client-server model you can do everything on the server and you don't have to consider the fact that your application is not entirely under your control so when you're rolling out a new version obviously you need to update ENS to point to the root hash of the new version but also you need somehow to notify all the active users of your application that things have changed because otherwise they just keep continuing using the old the old version and it might even interfere with the workings of the new version so in this way it is very different from client-server application also with apps we have the same thing as what we have with block chains we can have Forks it's entirely feasible that somebody likes a particular version over that and they register it under their own iana's domain and people keep using that so whenever you introduce a change that the community disagrees will or at least a large part of the community disagrees with you will have Forks I think I'm running out of time so the last thing that I wanted to mention is analytics which is much more difficult than in a centralized environment and it's ADAP in itself that will be eventually developed but unfortunately I ran out of time so if you want to talk about analytics please see me in Saturday at the breakout session thank you [Music]