Protocol Berg v2: Thomas Hsueh - Coordination Avoidance: Rethinking Decentr. Netw. Beyond Consensus

Yeah, that's about it. Thomas, take it away. Hello everyone. I'm Guilty Gyza. Thanks for the warm introduction.

Um, today I want to talk about coordination avoidance. um rethinking decentralized network beyond consensus. Now I think from first glance coordination avoidance may sound very strange because this industry is founded on top of algorithms that encourage coordination not avoiding it. Um but there there's a there's a technical meaning to that that we'll get to. Before that let's talk about global consensus.

So, Ethereum gave us um global finality um in a trust minimized flavor. It is a truly unique system, one of a kind. But um we all know that Ethereum operates on global consensus which is um sort of the finality algorithm is the the modified uh two-phase commit protocol um where the supermajority um among all the validators have to agree on a certain block for it to be finalized and the consequence of running a consensus mechanism globally leads to number one latency issues. So latency is heavily dominated by consensus. Uh which means that the system is not highly responsive.

Um also the cost of transaction is dominated by having to run global consensus. Uh these fees of transactions make layer one really unsuitable for very frequent interactions which actually is outside of the scope of its design. So that's all well and good. Um but as a result we have a world computer that's not really suitable for interactive contentrich kinds of applications. So we're asking the question is it possible to make decentralized applications that are actually responsive and free of cost to use.

Enter local first software. Local first software is kind of software that functions on the local device seamlessly and it uses the global network only when it needs to synchronize with other replicas or users or it needs to perform a remote backup for example. Um how does it do that? Local for software places a copy of the application state on the client side so that the client side are essentially only interacting um with the local copy or replica of the state. Behind the scene these copies um on the network synchronized among each other to maintain consistency.

The key advantage of this architecture is number one applications become super responsive because locally you can read your write and observe its effect immediately. Um it also gives you offline capability just like the the git protocol. You can um edit your local clone of a repository at any time. You can synchronize with the remote uh repository when you want to or where the internet is available. But how does um how is concurrency handled?

If everyone is editing their local copy of the same application at the same time, how do you handle consistency and collisions and so on? Well, we talk about CRDTS. Uh CRDTS stands for conflict free replicated data types. It's a kind of data type that is um with instances is replicated across many machines and when these copies are synchronized uh they would synchronize just fine without any conflict whatsoever. If you have no conflict to deal with you don't have to resort to ordering systems or voting systems that means that you can run a um you can run a distributed application without using consensus mechanism.

So it's a spatial data structure that I I like to call it multiplayer data structure. It allows you to build multiplayer software. Um sort of the the the ma the mathematical guarantees behind cdts guarantee that whichever way you merge these different copies of the same piece of data, you will always get the same result without having to um resort to a central coordinator or resort to voting mechanisms. uh famous case study would be HackMD. It's the collaborative note-taking applications that we all love to use.

It started off with the operational transform approach um OT which is a predecessor to CRDT that's um 20 more than 20 years old. Um but the hackmd later adopted YJS which is a very popular open- source CRD library written in JavaScript which also um was uh rewritten in different languages including including Rust and Python. The motivation of HackMD adopting YJS is that it wants to support offline editing capability. It also wants to uh allow users to manipulate the folder trees concurrently. Um OT is um was designed strictly for uh text editing.

Uh it's difficult to generalize to more complex structure for OT and CRDT was designed for uh designed specifically for that. And there as with all engineering approaches there are limitations. CRD is not a silver bullet for all possible multiplayer software. We can look at two case studies. The TL Draw is um the pretty popular multiplayer canvas application when you can users can draw on the same whiteboard uh prototype designs together.

They also integrate AI so you can generate drawings. Um the draw initially used CRDTS but they dropped CRDTS because of performance issues. Imagine you are on on a collaborative canvas and you want to draw a line from point A to point B. The typical typical uh user experience is that you would click on the canvas with fix which fixates a point. Then you drag the other point uh deciding where you want to place a second point and you'll have a line that is connecting the two points at any given moment.

Um when you generalize this to multiplayer scenario where everyone is um estimating and manipulating their geometries in real time you get um pretty serious performance trouble with CRDT because operations or states are frequently transmitted across the network. you get uh you require very high bandwidth for uh correct operations and um and so for TR draw an application that requires very responsive kind of experience they had to um handcraft their own synchronization mechanism for themselves. Second case study would be the Penguin World prototype built by topology. Penguin World is a um is a game prototype that looks like Super Mario but only multiplayer. There's a sidescrolling um it's a sidescrolling sandbox where each player is a penguin that can jumps around and collide.

Um the game was prototyped using YJS. So the entire game state is structured as a YJS document. So the sort of the the thesis behind the prototype was that well it's uh it seems to be um it seems to be a dimension jump from a collaborative document to a game because because um what makes it different between multiple people editing the same document and multiple people participating in the same game state. We thought it's a dimensional jump. So we um um as in topology prop uh proceeded with prototyping it.

Synchronization works out of the box. So you can make a simple uh baseline multiplayer experience right out of the box with the YJS library which wasn't built for game uh scenarios at all. But the tricky thing is with uh sides scrolling games one of the basic physics you would expect would be collision you want interesting thing happen when things collide whether that's bullets or NPCs. um collision turned out to be very tricky on a on a peer-to-p peer situation because without an authoritative server who's who is to say which collide with um whom first um tricky thing with bullet for example um did I headsh shot you first or did you dodge the bullet first and if if it's uh if it works on top of P2P network you can there are sort of various different ways for you to for example pretend you didn't receive a packet or pretend you receive a packet later um then you dodged the the sort of the bullet carrying packet. Um how we prototype the penguin world was that we um we we implemented a collaborative kind of collision mechanism where when whenever anyone collides with anyone else, a message passing happens uh that says uh A collided with B uh both A and B should back off by this much and change their velocity by this much uh which obviously is uh which obviously welcomes cheating behavior.

Um so if collision is this tricky let alone more interesting game physics this turns out to be um a very challenging proposal for video games and so we can move up the stack because uh if you follow the local first software movement uh nowadays CRDT as a algorithm is not so much heavily discussed anymore as before as let's say five 10 years ago uh nowadays sort of the trendy topic would be sync engine so sync engine sort of recognizes with the fact that when you build a applications with multiple people interacting with it, all you really want is um for the synchronization to happen magically. Um and so sync engine is a middleware that handles synchronization for your application. It takes takes care of network transport and it takes care of any possible conflict resolution. So your software on top of a sync engine becomes a multiplayer version of it. Um a good sync engine should um satisfy should have a few properties.

One, it should be fast obviously even with very large data volumes. Um and um specifically there's there's something about initial load when you first load up an application. So how how long does it take to to load up the initial state sort of the uh the time between visiting website to um sort of the first render of the website. Um if you have to synchronize the entire application replica it might make the initial load very slow which would uh drive away um internet users. Nowadays all of us are pretty impatient online and so um sync engine solution to this would be partial sync and there there are different algorithms for partial sync.

you would sort of synchronize um uh pieces of your application state based on their query patterns or based on uh client side sort of uh request or client side uh patterns. So that's partial sync. Uh number two, it should be very easy to use. So CRD has this issue sometimes that you expect programmers to understand concurrent programming to use a CRD properly. Um concurrent concurrent programming is not very easy to understand.

Sync engine abstracts all of that so that your software on top of it becomes um multiplayer instantly ideally. Um third property should be um it should flexibly support many transport options uh from websockets uh web RTC and so on. And furthermore it should it should offer access control out of the box because multiplayer software um invariably needs some kind of access control to prevent chaos. But there's one thing about off-the-shelf sync engines. On the bottom right, you can see Replicash and Electric.

These are two bigname sync engine projects um on the market. They rely on centralized servers for synchroniz synchronization. Uh this is not to blame anyone. They these communities come from very different um different very different directions. They're often sort of quite orthogonal to the blockchain community.

Although all of us are trying to build distributed applications with different guarantees. um we want to make decentralized applications. We want to avoid or minimize the presence of centralized servers. So uh what can we do? So uh we can imagine the existence of a decentralized sync engine which um uh to my knowledge does not exist at the moment.

But we can imagine what should a decentralized engine look like? Well, first of all um it should work on top of a permissionless P2P network. Um and with with all P2P networks there are sort of bas basic functions that typically are fulfilled by centralized servers such as bootstrapping but um we should minimize the presence of centralized servers um as a whole. And second we would crucially need very good P2P transport layer. This is this is to say that when multiple users are working on an application on the sync engine, their um their deltas or their updates should be propagated on a network as as efficient as possible and also not just efficiency but also accounting for um topology changes and u membership churns because people going online and offline all the time.

uh sort of a few um approaches to achieving this will be gossip mechanism um lip P2P has a gossip protocol um broadcast trees which um has a sort of a more efficient way of multiccast information on a on a sort of structured network um and RLNC is more um catering to unstructured network where you have a network topology that's very unstable but you still want to make sure delivery a guaranteed deliver delivery as much as possible. Um so network coding breaks up your payload into pieces and encode it in such a way that um any recipient receiving enough amounts of these uh fragments can reconstruct the original data that will give you uh sort of uh robustness against topology changes of the network. Um potential applications of this will be censorship resistance social medias forecaster being the the best example of this. um collaborative applications of different kinds. Fileverse with their DOC is is the great example for this.

Um in this talk we want to um propose a pro probably radical idea um but maybe it's time for this um decentralized governance platform for Ethereum itself. So um this is taking a detour from the technical aspect. Um we can sort of look at Ethereum governance from a constitutional perspective. uh the three power system legislation execution and judicial branch. If we look at the Ethereum spec as the law then uh all the full nodes collectively form the judicial function of Ethereum.

Um execution function includes different entities and um organizations that are um that are working towards turning spec um into uh production code and um managing the sort of the project management aspect. But what about leg legislation? Well, obviously legisl legislation is driven by the EIP process. So this diagram uh the state transition diagram is modified from EIP1 directly. Um you have the state transition diagram where every EIP goes transition through these different statuses um on different conditions.

Uh the entire process uh stewarded by the core devs. Now the issue is that there um are certain aspects of this process that somehow lacks codification. So number one the how sort of the the arrows in the diagram uh can lack codification the conditions the triggers and the entities that determine a transition um somewhat lacks codification. Um the whole problem so precisely the authority um and the accountability of the special role EIP editor may lack a little bit codification. Um aside from EIP editor, client teams have significant voice in the process but um the way in which these teams are qualified and their voices weighted um or not uh is not exactly uh codified.

And finally the wear problem. Um discussions about EIPs are scattered on the internet across uh discourse server ETH R&D across the Ethereum magician forum um in the PR comments of the EIP repository and of course u in the ACD the synchronous ACD calls um it's not very formally codified how the voices of the communities is incorporated into these discussions um well is it time to change in fact the lackings quote unquote were actually very crucial because um for example it provides flexibility for the foundation to steer in the environment where the regulatory uh situation changes. Um it also prevents regulator uh it also prevents um sort of for formalized modes of capturing the system because when you have rules that are um excessively formalized you invite kinds of attacks um for the for the rules. So um when we look at Ethereum's history as it is it's now approaching the 10th anniversary um the ecosystem has grown explosively um every change to the law or the spec can have profound implication to many stakeholders. So we believe that Ethereum can foster stronger trust by increasing structure and rigor in the legislation process.

um an entire process may be very difficult to change um but we can start from the technical aspect just to demonstrate the visibility um sort of proposing sort of fantasizing the concept of uh concept of Ethereum Congress um doesn't have to stick to the schumorphic language just just as food for food for thought Ethereum Congress is a a platform that centralizes all the essential discussions with respect to uh spec upgrades on a decentralized platform uh it consists consists of two locations. The floor where the top level deliberation happens. It also consists of hearing rooms um allocated per each EIP so that they can be uh transparently and um centrally discussed and edited and reviewed. On the layer one, we can have uh small contracts that manage the permissions of these locations offchain locations. We can also have the um a special contract called the EIP manager that basically canonically manages the statuses and the transition of every single EIP.

Um so we've built a prototype using orbit DB which we see as an approximation of the uh decentraliz sync engine. Um EIP manager is the special smart contract on chain that canonically records the status of every EIP uh whose events are tracked by um orbit DB nodes. And here we have two hierarchy. We have the admin DB nodes that are responsible for tracking the Ethereum events and creating new databases for for hearing rooms. And we have all the users that are locally running the datab orbit DB nodes with the local first uh software principle.

There's screenshots of the um we can't do live demos here. Screenshots of the PLC. Not everything works but the the repository is open sourced. Um you can the the floor is the place for top level uh meetings and there's hearing room for every EIP where you have a um where you have a forum based interface just for uh just for last um food for thought a side product that is unlocked by the system is you can then build a prediction market on chain that um sources information about EIP from a special small contract and have the British market bet on for example will EIPX reach final status by DDATE and that prov sort of provides a way to pull community sentiment which can be factored into the legislative process in some way. Um so um we conclude the talk uh by saying that global consensus is a very precious resource global finality provider Ethereum's uh uh core strength but um there is value for building coordination avoiding systems on top of the system and one value proposition is using this coordination avoiding system to govern the coordination heavy system both are decentralized in nature and we proposed a proof of concept called the ether uh congress just to spur discussions.

Thank you. Thank you very much. Thank you very much. Um, we have a few minutes for Q&A. You can, you still have some time.

If you want to get a question in, just scan the QR code up there and access Mircat, log in, um, vote or upvote some questions. We do have one question here. You partially answered this one already, um, but, uh, I'll give it to you anyway. What happened to Orbit DB and Radical? Do you use them?

Are they part of your stack? Right. Uh we didn't use Radical, but I think it's a great point. Um a version control system for um code should be in the picture. Ethereum spec is not written in natural language.

It's written in code. And so there should be a there should be a place for such a system. But we do use orbit DB um and um orbit DB um to um those who may not be very familiar with it, it builds on lip P2P and a CRDT layer and a database abstraction. Um so um it follows local local for software principle nicely and that's why we use it as an approximation and then I think we have one last question here just a quick one. Uh no, we have two more.

Okay. Have let me give you this one first. Have you tried gunjs per amazing things about it? U personally haven't tried it but I if um if I have spare curiosity that will be the next project I try. Great.

Okay. And this should be the last one. Are you aware of the work being done with ethereum forum? Actually, I am not