Havoc Net How to Spot Consensus Bugs in Large-Scale Decentralised Infrastructures Stefano De Angelis

I'm taller and today I'm going to present you a new tool it's an approach to spot consensus bugs in large scale decentralized infrastructures like blockchains so when I did when I do this kind of talks I would like first to fix the stage um setting the ground on decentralized systems there is this very nice uh citation about how a decentralized system is and it is a distributed system in which multiple parties control uh components of the of the network but nobody uh is fully in control uh of the whole network uh I would like to do a parallelism between web 2 and web 3 infrastructures uh in web two we are familiar with uh we have this kind of Paradigm of client server uh and the server is usually under the control of a a a very big uh provider like for example Amazon uh these guys can give us uh very reliable and secure infrastructures ensuring certain guarantees for example of high availability high performance systems uh at the cost of a of a centralized service on the other hand if we talk about decentralized networks like blockchains we we have a autonomous servers that interconnect with each other and they are governed by a protocol um they provide certain property like uh no single point of failure L they there is no need for trust anybody to use their service and nobody's in control of the infrastructure and this is actually where the problem is because if you think about a blockchain right we have a number of nodes that interconnect with each other and these nodes can be myself in my house with my laptop running a no on ethereum for example and another big provider like in fura which is running a lot of nodes uh with different guarantees so in this such a large deployment uh the the challenge is how do we ensure the maintainability of the protocol we do have governance for doing that but we don't not control the infrastructure and this is what we want right the drawback of that is that when we want to test uh production ready applications if you think about I don't know Bank use cases fin finial use cases we need to do a lot of test also on the infrastructure uh if you so we do have the possibility of running the nodes for the test on the nodes for example inura can perform a number of tests on their infrastructure but they're not testing the underline protocol we can test the applications as well so we build applications on top of the blockchain we have some techniques like fuzzing static Dynamic analysis we can do formal verification but still we are relying on an infrastructure that we cannot control so what if we want to test consensus bugs or what we if we want to test decentralized applications that behaves under uh difficult condition of the network well you could say hey Stephano we do have test nets for that right so what is a test net a test net is a parallel Network that can be ATT tested for for ethereum for example that runs in realistic conditions so we have node Runners again that decide to engage with a new network that is that might run the the protocol rules of the main net but might not so the drawback here is that we have discrepancies between realistic main Nets like ethereum main net and the test Nets and this is why the reason why is that sometimes it's cost it's very expensive to run test Nets so note providers prefer to run a test that is uh running with a different consensus protocol for example or has different conditions so how do we uh leverage test nets for running uh our test use cases so just for giving you a snapshot of what happened on ethereum in the last period so we had a number of test Nets uh someone was running a proof of work consensus protocol others were running proof of authority and others proof of stake so I hope you can read it this is like uh the the line up of all the test Nets that we had on ethereum uh robsten was the first one with proof of work and then we moved toward other networks that were leveraging a proof of authority which is let's say a consensus protocol that is run by a c a fixed number of actors that with a lightweight protocol propose blocks all those test Nets got somehow deprecated in favor of a proof of take based solution but still we don't have control of the infrastructure so what we propose is a um a different approach uh leveraging simulation and the idea is to have a realistic decentralized system that provide also infrastructure control so the tool is called Havoc tool and the idea is to deploy simulated networks leveraging tools like kubernetes and Docker but in a decentralized manner with this tool we can distribute the the workload according to our needs and the workload is say let's say for uh firing transactions on the network to observe the behavior of the network and then the cool part is that you can test it according to network issues so there is a techniques called CA testing that allows to simulate issues at the network level so you can crush the nodes you can inject Network partitions and and You observe what happened to your workload so this is a very basic uh diagram of the flow you deploy a network with a state state a Ste State you define a workload you simulate the C testing and then you observe the metrics so some of the the stuff you can do here summarized so you can deploy network of different uh topologies uh you can even deploy nodes that are running different software versions uh you can Define workloads according to your needs and inject folds the metrics you can extract are a number of metrics that uh move from performance metrics and security metrics uh according to what is of Interest we did a CID study in particular we tested the three two test Nets famous test Nets of ethereum kovan and rink by and uh we compared them with another kind of private Network called Quorum that was running a consensus protocol called ibft so the the first two were running a proof of authority version of the protocol while Quorum is a bft like protocol uh very close to classical distributed System Theory uh what we did we run some tests injecting partitions and clock skes in a network of in a small network of four nodes and deployed some workloads and we observed what happened at the network level so I don't want to bother you with diagrams but uh the idea is to observe the number of forks that we measured during the test and also the number of finalized transactions that were uh emitted uh you can see from the vertical do red line the start and the end of the adversarial uh situation in which we had a partition and a network que and if You observe on the on the black line this is the one of the proof of authority networks we tested we observed that after the partition got resolved it never returned to a single Fork um situation so the network basically forked forever and no got out of sync breaking consistency of the network and we observed that that also from the capacity of the network to finalize transactions of course during the the network issue we had here it wasn't possible to finalize the majority of transactions but then always looking at the black line we observe that after the partition resolved the network wasn't able to finalize transaction anymore at the same level of the input rate that is the Green Dot line here so what happened basically is that the network went out of sync and the result was that most some of the transactions got failed and wasn't able to finalize them so in conclusion there are some consensus bugs that we cannot spot even if we have test Nets uh um on real world settings we need control of the infrastructure the cast testing technique is a good approach to do that but we need to do it in a in a in a certain uh deployment scenario in which the network is still decentralized and reproduce realistic environments and this is more true now that ethereum is moving ethereum and other decentralized system they are moving toward a modular architecture in which we have layer twos we have layer ones we have data availability layers we have Bridges all this stuff together that gets integrated needs to be tested in a proper way so thank you for your attention and happy to answer questions