2 years and 200 audits later: DAO learnings from building Code4rena with Sock

foreign I'm going to go ahead and get started because I'm standing up here the laptop is hooked up and you're sitting there so I'm gonna start all right anyway so uh I'm talking about uh two years and 200 audience later um my name is sock from C4 uh code Arena and some of you are probably wondering what the hell code arena is um most of you may have heard uh because I see a lot of folks that I recognize um but uh basically cut Arena pioneered the competitive audit and the basic concept is rather than a fixed time audit with a few uh quality Auditors taking a look at the code the approach that we use is we bring in 100 plus Auditors competing to find bugs and get a share of the prize pool and they we do this generally in a few days or a week or a couple weeks in some instances uh sometimes longer with larger Scopes but basically based on the volume that we have based on the surveying that we've done of uh the the Auditors in our community and the time that they put in on audits uh we estimate that we roughly have about six months worth of in-depth code review delivered in one calendar week which is um pretty stunning actually um and and the the nice thing also is that when folks have a desire to incentivize more top talents out of their code they just can add more funds to the prize pool um really the core concept I think is really is outstandingly uh explained uh by Savannah talk that Sebastian bettiscu gave uh from Quant stamp uh at I think was ethcc 2021 and based on what they what they identified from a large volume of data in audits they looked at the more Auditors that you have with eyes on the code the more bugs are found and so the model that we're trying to do is how can we get as many Auditors to be reviewing the code simultaneously and the model that we use isn't uh like a bug Bounty where essentially your first in or you're nothing right uh the model is essentially everyone gets compensated based on the value of the bug the value of the bug is it is determined based on how severe the issue is as well as uh how rare it is so if you find a solo High severity issue you're going to make a pretty decent Penny or in the matter of really well Arden code that's already gone through several rounds of audits uh as was the case in a recent event uh we had one person who had a single medium that was the most severe case ended with a seventy thousand dollar um result so really the the reality is that uh this this community ends up finding more smart contract bugs faster than anyone and we openly share all of the findings and learnings from the community so you can go to the GitHub for code Arena and see all of the past audits you can see everything that anyone ever submitted and you can see them matched up by their duplicates you can see the ones that made it into the report the ones that didn't uh and in fact there's actually uh I just saw this last uh I think yesterday or maybe the day before there's academic research that's being performed based on the public data that we've made available from uh these these audits that are out there um and a big thing and this is something I'm super passionate about in in how C4 works is that we really Empower new web 3 Auditors uh to become leading Security Experts there's so many people that have come through our community who have started as really pretty early on in their in their pet in their path as an auditor and have been able to progress through and become some extremely uh dominant folks in the space with some uh really well-known uh well-known names and it's been a joy just to see that and the cool thing is it's entirely permissionless and so it doesn't matter you don't have to sign up for an interview you don't have to show up you just show up and compete uh sign in with your wallet and Away you go um and a big part and this is really fundamental is we're always working to ensure that people are paid fairly for their contributions and this is a big part of why we don't take the straight up Bounty model it's a big part of why we ensure that people get paid for and have a way to get paid even if indeed the code is extremely well hard and so we have uh QA reports and gas optimization reports that people can submit and uh and also earn a share of the prize pool one thing that's really unique I think about C4 is that it was started not by Auditors but by audit customers so Scott Lewis and Zach Cole originally uh created the the design for code Arena and I was just some Rando on the internet um that happened to see them tweeting about this concept that they were going to try out for slingshot so they're co-founders of slingshot and uh Dex aggregator and they needed an audit for their platform and they said you know let's let's uh let's get as good a one as we can but they couldn't get one uh booked with a top audit firm for months and so they said let's let's reach out to all the white hats we know throw up a pool and see what we get and the result was uh pretty pretty awesome from the very get-go um and a key thing is that these are Builders who wanted very good very fast and very fairly priced audits which is pretty hard to get because the market is just so so much demand there um so ultimately what C4 is is a multi-stakeholder dow that ends up being made up of Auditors and Builders investors and users who are all interested in the same goal of seeing the space be more secure so now I want to talk about Dows I want to talk about what we've learned in the process of building uh our Dow uh and in in uh and in operating it and ultimately at the end of the day if you're building a dow it's Choose Your Own Adventure the the you know I heard it set in the last panel you've got to make the decision yourself uh about what your design looks like you're making choices ultimately along the way and this is critical whether you acknowledge them or not um and you're making decisions about priorities so um there's a really good there's a really good talk uh and uh essay called choose boring technology from like 2015 by a guy named Dan McKinley and he talks about the idea of innovation tokens and basically it's like if you're thinking about a startup as a startup you're trying to do something probably hard and probably you know trying to disrupt some area and if you're doing that you also are making technology choices along the way well if you spend all of your Innovation tokens making uh you know interesting technology choices you have less Innovation tokens to focus on the hard problems you're trying to solve and so the concept that he talks about is you basically should be trying to climb up Maslow's hierarchy so rather than focusing on the fundamentals of what the technology is doing you should be focusing on how are you going to solve the big problem that you're focused on and that is a big thing we made a lot of trade-offs especially early on as we form the Dow in terms of technology I mean the the early tools that we created for people making submissions it's not on chain at all it's basically I threw together a form and it dumps into GitHub and then people use GitHub to triage the rewards it it could have been more decentralized but that wasn't the point of what we were trying to do ultimately we were trying to figure out how do we how do we bring in as many Auditors as possible um and and that's the the thing we're most trying to to decentralize and scale I want to talk about this concept that I saw Corey doctorow talk about um on Twitter a little bit ago called insidification because I think it has a lot of relevance in the context of thinking of particularly a multi-stakeholder dow and one that has multi-sided Market involved so in shidification is what he described as a seemingly inevitable consequence arising from the combination of the ease of changing how a platform allocates value combined with the nature of a two-sided market now he gets into just he gets into describing this a little bit better um so he says here's how platforms die first they're good to their users and he gives an example of Amazon in the way that Amazon was just outstanding or Google was outstanding in what they were trying to do for users at first so they first they're good to the users then they abuse their users to make things better for their business customers and finally they abuse those business customers to claw back out all the value for themselves um this is insidification surpluses are first directed to users then once they're locked in surpluses go to suppliers then once they're locked in the Surplus is handed to shareholders and the platform becomes a useless pile of [ __ ] so the biggest thing when we look at how do we manage the growth that we're trying to achieve the question we're trying to answer is how do we avoid acidification um again you're making choices about or your organization whether you acknowledge them or not in a dow uh Joe Freeman uh badass uh may have been uh maybe somebody anybody in the Dow space has heard her name before and this essay the tyranny of structurelessness which basically says that you may you may think that you are aiming to create an unhierless uh organization or a flat structure whatever but the reality is that every organization has hidden hierarchy even ones that have hierarchy uh and so if you're not working to figure that [ __ ] out uh you're just kind of having this game that you're playing where the real hierarchy nobody can see and nobody can participate in and that is a big that's a big challenge uh I love this quote from uh Anarchist uh Enrico uh maleteza uh organization far from creating Authority is the only cure for it directly in line with the thoughts of Joe Freeman uh another badass that I think Builds on this uh concept and takes it a little farther is Eleanor Ostrom uh who uh her piece of contribution to um this discourses is big enough and impactful enough that as a political scientist she won the Nobel Prize for economics which has never been done mutually Nobel Prize for economics it's run by economists as one by economists but um she basically was addressing there's an essay from uh that we we've all heard that the term um of the tragedy of the commons and basically she was trying to say that there are ways to avoid the tragedy of the commons and she did an enormous amount of research in identifying what are some areas where the comments has been preserved and strengthened and what are the systems that make that happen uh and I think this is a completely interesting way to look at Dows and whether we're being successful in the actual Pursuit that we're that we're going after um as an organization and one of the big things that she comes away with the institutional design matters a lot um she's got these eight principles so I'm gonna I'm gonna go through them and I'm gonna and then the thing I'm gonna do is I'm basically gonna walk through the ways that I think uh and I'm gonna be a little more generous than my first grading but the ways that I think uh code Arena has graded uh against the scorecard and and why um but I'm gonna be a little more generous based on some initial feedback that I got on on my scorecard but anyway so her eight principles are these clear boundaries for everyone using Commons rules in line with local conditions those impacted by rules have a voice in changing the rules monitors keep tabs on behavior and are accountable graduated sanctions for out of bound Behavior conflict resolution mechanisms and then nested groups with different roles and responsibilities you'll note that there is no seventh principle on that which is because the right to design your own institutions is not challenged by governmental authorities which just the side eye to the entire uh us uh regulatory apparatus right now um so we'll focus on the eight because I don't think that's something that we can fit into the scorecard uh for how coloring is doing so what are we doing well um I would say from the Austin scorecard again clearly defined boundaries for everyone using the commons um I actually give us an A plus because I think one of the main things that we got right at the beginning and I could say that because it wasn't me is in the design there were specific roles that people could play you could be a judge you could be a warden wardens or Auditors you could be a sponsor sponsors are the folks with projects bringing in their projects and everyone has a specific role to play in the dance and then ultimately we identified people who in the in the Dow were organizers and that was the I was the first organizer uh in in the Dow where I was basically just putting together these different uh audit contests and bringing people together on that and I feel like the fact that we had these really clearly defined roles that people could come in and play and it wasn't everything in the world is wide open come pick a role decide whatever the hell you want to do but these are the things that really matter that we need to fill and we need to fill well and I think we did a really good job of that but I would say I have to give that an A plus sort of because everyone is big and this is the problem ultimately is that when you talk about everyone using the commons they're also a lot of people that um have uh taken things that we've been building in our community and then have built other things on top of them ultimately that's uh challenging when you're trying to grow something um and being open will make you an easy target um and ultimately just do it anyway uh because really doing being open makes you more resilient uh it it helps you to create more value than you capture you know that you're doing that it keeps you accountable to always improving um and I think those are things that really I think those things really matter um then again the host from scorecard nested groups with different roles of responsibilities is a big part of what we did we actually formed an organization that was hired by the Dow in order to facilitate um the way that uh the way that these audits worked um and again we're trying to create an uncertifiable platform I really believe that if we ran C4 purely for profit we'd ruin it if we ran it as a pure on-chain democracy we would ruin it if we ran it to where Auditors were making every decision then we would ruin it if customers were made of every decision then we'd likely ruin it and so ever we've we've got to set boundaries for everybody and we've got to identify what are the roles what are the processes involved so we have independent judges for each audit and then we have an independent commissioner basically is the best way to that I think of uh the org that that I end up working for that's hired by the Dow and basically just like a commissioner in a sports League we are not putting our hand on the scale in terms of individual outcomes but we are there to ensure that the rules are constantly being made better for all the participants on all sides and trying to balance the needs of the folks the folks who are wanting Audits and the folks that are providing Audits and making sure that neither side is getting taken taken advantage of by the system so uh next item on those from scorecard rules in line with local conditions I I think that we're a B plus on this one and I would say because it's in progress because this is something the local conditions are constantly changing a big one that we're experiencing is um you know I think for the first half of uh of C4 we were in the bull market for the second half we were in the bear market and in the first half uh we needed as many Auditors as we could get to fill the need on uh on the side of um audits that were in demand and then we had so many Auditors that uh you know right now we have about 3 500 registered Auditors competing on our platform and keeping keeping that in balance and keeping that reasonable uh is a big challenge in terms of the overall systems staying on top of and then we've got the challenge of AI right um and as you grow any group uh you have early participants who are going to be by the book and and you have the relationship strength and the trust strength of the choices they're making being in line um as you grow you start to have more Bad actors and so trying to balance that is a constant battle and trying to figure out how do we have the Right Touch in different policies and so this is something that's always in progress and it feels like we're always a little bit behind the curve but always working on it monitors and keep tabs on behavior and are accountable um I would say the greater B broadly accountable in that the organization that serves as the commissioner is uh you know is essentially hired by the Dow and therefore accountable um but ultimately I think there's more accountability that could become there um uh then yeah then what what really should improve individuals impacted by rules have a voice in changing rules they absolutely have a voice the process is hard the process is really hard because of the fact that when it comes to design we have an outstanding mechanism designer in Scott Lewis and ultimately there's an enormous amount of pressure and desire from Auditors to participate in shaping exactly what the designs of the mechanism should do however they're only representing one side of the perspective in the whole picture and so it becomes really difficult to entirely say well Auditors are the vast majority in the community they'll make the decision um we have to take that in that input and then we have to balance it also against the fact that we have customers we're serving and they have specific needs and demands and we also have to look at the whole so I think that this is a really really hard thing we we agonize over it a lot and we but I think that uh it's something that we can always do increasingly better and a big part of how we approach it really is keep things as simple as possible as long as possible um I'm a big fan of uh the idea of non-intervention that like a lot of times uh jumping in and reacting and making a decision on something can be far more harmful than uh letting things sit and see how they work themselves out and we've seen that play out a number of times um we're really big on Paving cow paths rather than cowing everyone onto paved paths and so we see new approaches and ideas emerge from the community gain some steam and we say yeah let's standardize that we also really big on authority of expertise not position not numbers and ultimately that's a big part of why we're like when it comes to design we're we're gonna go let's bring a mechanism designer in and take all the input and make the best decision possible I love this quote from Miguel bakunan uh well-known Anarchist reject all authority no in the matter of boots I refer to the authority of the boot maker and that's a big part of how C4 Works in that we are absolutely trying to find the way to highlight the expertise that exists in our community um graduated sanctions that's a big one that we're trying to work on especially with the growth of AI conflict resolution mechanisms in place we have a lot of ideas for this but we haven't implemented a ton of them um generally these things work themselves out in a very ad hoc way that's been Pro positive but I think systematizing that stuff is really important that's the core of it but ultimately the last thing I want to leave is a high scale multi-stakeholder Dow is an intentionally guided Market um and ultimately that means that it's a vision for how you believe the market should function so when it comes to what we're trying to do let's make the world we want to be part of thanks everybody foreign [Music]