Ethereum Protocol Testing

[Music] again hi my name is Demetria I'm working with a foundation for about three years and for a long time a serum testing was performed by one person Christophe judge he neutrally invited me to the Assyrian foundation and then I continued his work so today I will describe you for you how the testing process goes in the serum and what we achieved for the last year science Shanghai so what we have is a yellow paper which is a protocol yellow paper describes the serum protocol what it doesn't make it easier to understand actually this picture represents yellow paper really well nobody able to understand this unless we tell it my and we have lots of implementation of this protocol written on different languages different clients and the testing problem is to make sure that every of the client is in consensus with each other and they interpret they implemented this yellow paper rules in a way that there are no consensus issues between themselves how to do that we have our testing sealed each test file is a JSON instructions encoded in JSON file and we could select three main times of the test so the first one is the VM test is basically number of instruction for IBM white machine it's the most simple test in the whole test year so if if there is any developers here you could if you implement an EVM you could start with VM tests a most simple ones the next either state tests is a bit more complex and VM tests it has accounts transactions and state changes state hashes and so it's a bit more complex and the most sophisticated s is a blockchain it has actual mining difficulty and those algorithms implemented well and for a long time we require to all the clients to implement custom tests though suppose they had to read this JSON files and implement this and compare the results it was not so so good so we use a C++ client to run a test source which is my job is to keep those sources for test sources updated with the most recent issues discovered and what we are trying to achieve is to create lots of lots of different test source files describing all of the possible situation which we could get in a VM code execution and then C++ clang execute those instructions and it creates a final test final test is being processed by other clients and say compares their results to the C++ clients and that's how we see if everyone agrees with C++ client which is like a golden star user is an issue we open up these put and discuss whether we have to make a change in the C++ client or other clients so we had a couple of problems the first problem is to create an ultimate test which covers all of the code or all the possible situations which could run and possibly find all of the bugs and concerns to see here's the second problem is that clients they should execute our tests and because before we are required to my exact clients this custom tool I had to manually ask every client developer have you run this test have you updated the test repository is it updated and one time I discovered that core client wasn't running the recent tests for about half of one a year and the ghosa there was a heart for cover radiantly it still not cut medication abuse the test not to mention that I don't know the status of other clients [Music] so how we solve those issues to get the best code coverage just we have tests cases list which could be found on Google Documents this is the many o test cases if you came up with an idea for test case you could manually add this to this also I want to mention if you think you know how to make a consensus Asian's is a reward for bug bounty from the foundation if you discover a consensus this year you could get money and so to cover even more cases you can you can possibly come up with every situation but I think it's impossible so we did some tools that basically run a random code random even bytecode on not really random like fast testing is a smart random code and this code hope we hope that it will find some consensus ratio and it did another approach will be to use a fancy tool that you analyze a source of a client and then run different white-coat executions and maybe some fancy other reasons and you'll produce a very very good test ok two solves the next issue is the clients being updated last year i mentioned that we have a hive tool in development and now it's life and working now we no longer require time to implement a custom test oh they just need to provide a client and an RPC interface and then any client could be integrated in a hive tool and this have to perform a test execution automatically and we have results equals C on a high stats page there is also of all the rest execution on every client also clients have continuous integration bills in a github each commit executes this continuous integration test in a github which synced with our test wrap or a serum tests and I still personally do ask people on a Skype our channels versus a pay phone most recent test executions another field of work which was performed by Casey he did standardization of IBM logs and by doing this we could now compare that not only we have the same hash after the test execution but the way we get to that hash is exactly the same on every client and for the past year we added some debugging for the test if to which I'm a developer and it's basically a whole new field of work with those traces for again and better debugging okay and another update for the way for this year will be my personal for coverage by fork I mean serum has certain hard work points like constant and other proposals which we actually don't have a consensus about naming those Forks I called like EAP 158 and others codes and some kinds of beers dragons and those points means that from this particular block the protocol has changed and we have test that all the changes and all of the clients implemented correctly but we have a test cases and there I had many only to copy all of the test cases for each heart Fork and then I came up with an idea of general test format so now I described one single test case and then C++ convert this test and executed on every fork and produces a couple of output final tests it's actually a result of this particular test cases on every fork that that is what I call general tests and for the past year I realized state test which is now general every status being executed on every fork and in future all of the test cases that we have will be executed I just have to run one comment and all the test cases will be generated on the recent Forks same with blockchain tests and next will be the transaction tests also thanks to your G he did awesome documentation about how he learned because he has joined us this year he created the documentation of his steps when he was learning the process of test generator generating the test and then he graded this documentation if you interested in running the test execution yourself or if you are developer and you need to know how this works there is no updated documentation you could take the pictures of the links and you could see that now our testing team is not a single person we have like lots of people joining and most of them joining us this year so special thanks to all of them working together on tests okay we still have time for questions oh yeah not all of the tests are running on a - portion they're so general state tests are being converted in a blockchain test this is another update so I just run a common and although general state has been converted blockchain tests which then will be run on hive okay thanks Dimitri [Music]