Fuzz Invariant Testing: What Is It and How Can It Help You? | Chris Smith - Independent Contractor

[Music] for all right how is everyone doing um my name is Chris Smith I'm gonna be talking about invariant fuzz testing uh how you can use the power of chaos to better secure your smart contracts and your protocols um this is going to be a bit of a technical talk we'll be going through some code and it's primarily targeted for people who are um responsible for ensuring the security and correctness of smart contract code and protocols a little bit about who I am uh currently I'm an independent security researcher and consultant doing traditional audits in variant security engineering and reviews and protocol engineering previously I was a protocol engineer with uh pre- in-game maker Dow core unit and uh worked as a smart contract engineer for the launch of MCD with the maker foundation and was with uh consensus prior to that um if you want to follow along my uh latest tweet has a link to the code and this presentation or you can see it on GitHub uh here at I am Chris Smith uh and that link at the bottom all right so we're going to be talking about invariant fuzz testing um invariant testing is a tool that you can use to augment your unit testing your integration testing um uh and what you're already doing to secure and test your protocol to make sure things are working correctly it can also be called invariant uh testing as I said Forge invariant testing or stateful fuzz testing uh I'll be talking specifically about how to do this within The Foundry Forge tool set um and basically an invariant is a statement about your code or system that should always be true invariant testing checks that those statements are true after running a random series of inputs against a random uh sequence of calls so it's the power of kind of unit testing with fuzz testing but in a stateful way that maintains State between individual test calls um one of the nice things about invariants is that they can be and probably should be defined initially in natural language as the rules of the system um you can probably Define them from your white paper or your spec before writing any code code and then they can be part of your development and Engineering Process to build out your code um the randomness that they have both in the inputs that go into the functions and also in the sequence of calls that are maintained throughout the the test run is one of the really powerful things about invariant testing it can help simulate the adversarial environment that your code is going to experience in the public blockchain and in a way that your unit and integration tests might not because you may not be thinking of a hundred different calls that could happen in a random order that don't make sense to you as a developer because you know what um what should be happening in your system but invariant testing will allow you to to test all of those edge cases so I thought to show kind of the power of invariant testing we would run through uh code repo as I said I used to work on maker Dow code so I selected the uh D erc20 smart contract um I updated a little bit so it works with 0824 uh and removed some of the safe math stuff but B this is basically the die contract that's live on Main net now um as far as the tests go I removed some of the extra ones to be a little bit more on point with this presentation um we're going to be talking about transfers minting burning approvals uh for this invariant test Suite uh so for example this first test here talks about valid transfers to you or a user can send a balance from one user from thems to another user uh it's has a specific amount calls transfer on the die token and then it asserts that the correct amount was sent and the balance is changed but it also tests that the total Supply didn't change so pretty simple straightforward unit test here for you um as I said this is testing a very specific sequence of calls our fuzz tests our invariant tests are going to to test against a random input of uh and random sequence of calls so let's look at how we can get that Suite setup you'll see we now have this directory here under the test folder um an invariant Suite in Forge is going to have three parts you'll have your regression file your Handler file and your main invariant file that's going to be what's in charge of running the invariants for you regressions are going to be empty initially basically if you run into a problem where one of your invariants is violated then Forge is going to give you a sequence of calls that will tell you how you got to that uh violation and you can use that to create essentially a unit test here with the random inputs that will um backtrack and show you how you ended up in that in that uh buggy State and you can run those to recreate the exact sequence of calls and exact inputs uh so we'll put that away for now your Handler is going to wrap around the contract that you're testing so in this case we're testing the die contract we have a die Handler right now it's just an empty contract but this is basically where we're going to write the functions that will get called in the random sequence for by our invariant test Suite so we'll get more into the meat on that one later for Now setting up the die invariant test file this is the main test file this is the entry point that Forge is going to use um we need to generate a new die contract for our test and a new Handler contract for our test um a little bit of MCD work here we want the Die Handler to be able to Mint die when it makes calls so we're giving it permission to do that by calling R uh these are for the Forge test Suite to know which contract it's interacting with what it should be fuzzing and what selectors it should call right now we don't have any selectors in our Handler so that's going to be empty um and then I put in a very basic invariant here this is not a very helpful and probably not even a really valid invariant as far as true invariants go it's just saying that the die contract is deployed with 18 decimals that's hardcoded into the contract um but I wanted something in uh in place here for us to uh build off of and so we can look now at um how to set up the the Handler a bit more um get to the right commit so we want to go to our Handler now we're going to start building out our Handler a little bit here um as I said invariant testing is a good way through its Randomness both in the fuzzing uh inputs and in the sequences to replicate what's going to happen or what could happen in in the live uh blockchain from that perspective we need actors and in this case destinations uh to be our actors in our invariant testing so we're going to build an array of actors um that we can keep track of and we'll init this contract initially with 10 10 actors so there are going to be 10 users that can either make transactions or receive transactions they're all going to start with a thousand die just for testing purposes um we have two separate arrays here that we're going to be pulling from one is the actors that's basically going to be become your message senders and the other are the dsts or destinations um we create a separate array for that because I want to add the address zero into the destinations in case users transfer tokens to address zero or function calls go to address Z we want that to be part of our recipient class uh but we don't want to be making function calls from address zero because if a message. sender of address zero shows up in your protocol we probably have bigger problems on the Chain uh than uh something that's happening in your protocol as I said your die Handler is going to be a wrapper around the die contract that's what this function is doing so we want to test all of the external State mutating functions in D in this case we're going to start with the transfer function and wrap the die transfer call in this Handler function where we're going to use a random actor as I said this is going to become your message sender uh we'll talk about resetting errors in a second and then it's a transfer so we're sending from that sender the to someone else and so we're just pulling another uh random destination from that array here this is where you start to see some of the forge fuzzing come into play so the inputs for this function are an actor index a destination index and a wad if you know D wads are what we use for amounts um that's just the amount of D that's getting uh transfer the actor index is a random input all of these are random inputs of type un 256 Forge is going to actually plug that or plug a a semi- random value in uh those values tend towards the edges so they tend towards zero or towards Max end and get fewer values in between we'll talk a little bit more about that later uh and why that's important to keep in uh keep in mind but basically these are our random inputs we're going to use those to pick random actors with this select destination and use random actor functions we wrap the actual function call the die transfer in a TR catch because we don't want to fail every time the uh the protocol actually reverts and if you look here you can see that in some cases we actually want to catch an error and a allow it and allow the suite to keep running so when you run an invariant Suite you're going to define a number of runs and a depth and the number of runs is the number of times that the whole Suite runs the depth is I think the more interesting number because it defines how many function calls occur within that stateful test so it's going to maintain state if you say 100 for your depth it will maintain State and make 100 random calls from your Handler against your protocol and keep track of everything thing so we want to make it all the way to 100 unless we hit an unexpected error in this case we know that if we're trying to transfer an amount of die that is greater than the user's balance that's doing the transfer we're going to get this die insufficient balance error we don't want to stop our test Suite if we hit that error that's expected that's our code working the right way so that's all this uh if die balance of the actor is less than the amount that we're sending add the expected eror that's what this is doing expected error here will check if we hit an expected error and if we didn't then we're going to throw a revert and we will get um and that's when we'll get the sequence from the Forge test Suite uh and that's kind of what we're looking for with these invariant tests because then we know that we found a problem in our code as I mentioned uh we have these select actors and select destinations their internal functions their helper for how the Handler Works um one interesting thing here is this bound function this is part of the cheat codes of the Forge test Suite all that's doing is saying give me the actor index which was between Zer and maxu and 256 and narrow that range down to between zero and the length of our actor's array so that we get a valid index here that we can pull from our actors array we only had 10 actors in 11 destinations so we don't need maxu 256 here uh let me go back and type in proper replace with that initial transfer setup that makes building out the rest of our external functions sorry they're not showing up here here we go the rest of our external functions for the die contract fairly straightforward kind of repeating the same pattern we're selecting random actors random sources in the case of a transfer from or an approval and a random amount in a random destination you can see here we start adding additional errors so in a transfer from you need an approval if you don't have the sufficient approval we want to catch that error and allow it because we know that you're uh not supposed to be doing what you're doing it's not an error in the in the function um and then the other piece is I mentioned in our setup that the Handler needed authorization in order to successfully mint you have to be a permissioned contract on the die contract if the actor calling the mint function is not permissioned we expect it to throw a not authorized error here so like I said uh kind of more of the same as far as building out our external functions um now our Handler has all of the all eight or sorry seven of the external selectors that we want to be testing here we add those to our selectors array in the invariance test main file which is getting passed into the target selectors as a fuzz part of the fuzz selector object for Forge these are now the seven functions that it's going to call randomly in a random order in our 100de stack so we might get a you know a transfer transfer from transfer mint um in in those calls and uh start simulating kind of the churn and and the chaos of your of the public blockchain um let talk about now that we have our uh Handler built out we have our test Suite functional but we didn't have a very interesting invariant we were just testing that Dy had 18 decimal which can't change it's a constant on the contract a more interesting invariant and one that should hold true for all erc20 contracts is that the sum of all user balances equals the total supply of your token that that is what this invariant now tests so we grab all of our destination users we sum their balance of and we check that against the die total Supply the real power of this testing comes into play because this rule gets tested after setup when you run your functions or when you run your tests and after every single one of those Handler calls so every time the invariant test site makes a new Handler call call to a depth 100 or a th000 or 10,000 it's going to come through and run this invariant test and make sure that this rule holds true so it becomes something that can never be wrong or never be changed about your protocol and if you run your invariant test Suite many times over a long period of time you have enough Randomness that gets introduced that gets that your system gets tested against you can start to say that this is an absolute rule of your code that the user balances can never exceed the total Supply or never be different than the total supply of your token um and that can be a really really powerful thing to have alongside your other testing uh let's talk a little bit about the benefits and limitations and how you could take this um to the next level with your project so every approach to security whether it's unit testing traditional auditing um integration testing Fork chain testing they're all going to have limitations and invariant testing is no different some things just don't work well as invariant rules you need those invariants to be things that are always true no matter what you can program some conditionals into it but you can't really write good invariance if it's only true after this function calls this other function sort of thing or if you have things that are happening outside of your smart contract code like governance actions or if you're dealing with a up gradable smart contracts then anyone can change the list of actions that your contract can do the the proxy actions that are getting called and uh the invariance wouldn't necessarily hold anymore but as an addition to your current testing suite and to your current security practices with your code this can be a really powerful thing as I said you can look through your white paper look through your spec and start writing these rules before you even start uh writing the code and then run them as part of your Ci or run run them as nightly jobs to see if you're violating these rules that should never be violated as you're developing your code um one of the other limitations or not limitations but things to keep in mind is the fuzzed inputs that Forge is giving you as I said tend towards the edges towards zero and towards the maximum values you're not getting true Randomness in those values um most of the time that's not going to matter but it's something to keep in mind how can we take this further we can build out more uh invariance more rules there are other things that have to always be true about the die contract by itself there are other things that have to be true about the whole multic colateral die system and you could start building integration tests where you have invariants that are that apply to the die contract and invariant that apply to the multic collateral die system as a whole and run those um against the protocol if you need Randomness true Randomness or if you need less Randomness you can start to kind of put your finger on the scale to to prefer certain actors do certain things or prefer certain values come through uh at in your functions and that can add some really powerful uh ways of checking your um checking your invariance and making more successful calls more frequently which can mutate the state faster in your protocol and maybe get you to more interesting end States um as I said you can run these on overnight jobs for like really long depths or multiple runs and see what you get when you wake up in the morning so that can be a powerful kind of let your code work for you while you're sleeping type thing you can add them to your CI and run them in the cloud so that they're part of your uh development workflow as you check in your um pull requests um and yeah I think that they are uh really really interesting ways of improving the security of your smart contracts um adding a new tool for your testing environment and I hope that this helps get you set up and let you run tests build out your first invariant Suite um add it alongside your unit tests the code all the code that you saw here um on the different branches is available on uh the GitHub here on the screen or at my Twitter if that's easier for you to type and you don't have to type so much um so thank you very much