Vulnerability Coordination and Incident Response in a Decentralized World by Jessy Irwin (Devcon5)

[Applause] so [Music] difficult so here's a little bit today I'm just - morning this is not going to be super technical talk I couldn't hold up to github the shooting or super support so the first thing I always have to talk about generally is what is security how does it work why is it there how long has it been there we'll talk a little bit about our first main that security and did I will refer to this is story time and my team who are here are probably tired of hearing the stories but if go time so I or how we started thinking about these things I'm talking a little bit about some of the things that come up when you're trying to manage to centralize yet coordinating in response especially when you have lots and lots of stakeholders and in our case you know Adam poor people might people in Syria and then and also we'll talk a little bit about how governance is very uniquely positioned to be able to help us when we're trying to be more proactive and prepared when responding to sue areas so just to front just to talk a little bit about what security is I will skip over about ten thousand years of examples of human seems horrible and managing keys there's some great stories from ancient Rome about how you if you had a key would have invented a piece of jewelry let's get that today but security is a scary face it's the seven principles that are extremely important to player and implement correctly and the right places to make sure that you are protecting and defending what's important to you your communities and the conditions right for one there especially public activity there's opportunity rigor minimization and the compartment ation is extremely important making sure that you're not putting all of your most valuable things in one place but you're distributing them personality and what we do a little extra things but these principles especially if you help improve your thinking about practice in the security space there's some other aspects of security that are incredibly difficult technical piece of us get very excited about it but there's also a human thing because humans are very very difficult to Courtney and organize and manage when we have very strong intense emotions like fear which is kind of a big deal when you're in the middle of a security emergency and you think this thing you might have done is zero day there's also the question of being proactive about security meaning that you try to stop bad things from happening and you identify risk early and you step in and you take time to mitigate problems before they really become emergencies and then there's reactive security incident response is an area where you can be proactive and run books and planning for emergencies that will naturally come up in the course of running the watching but reactive because you basically have to detect that this badness has happened where this incident is occurring and then spring into action there's also a very very deep line between offensive security and defensive security in the security space I am hardcore blue team you name I wanna build defenses I'm going to protect people I want to protect a news I want to care for and maintain things a lot but offensive security can be very important are in space freighter that much work and attacking things and going after them and deeply understanding there are they break can inform how do you run your defense some of the ways that you see security happen in planning organizations tend to become security programs depending on what your organization does you can have product security could have application security or infrastructure security they can all be in one room they would be separate programs it really depends on how critical each of those things and the operation of your network but then there are also things that we don't really cover as much that are just as important as the technical ones operational security is one of the things that watching space and very significantly behind where it needs to be at the airport penguin assessments are difficult I run security on it's in the last two years okay well it would be important to make sure that you have someone independently evaluating your codes all of the business negotiation that goes online and scoping out and on it and defining the testing you need to do these are things that are hard they are immensely difficult to do if you've never heard before like how to think about them in a comprehensive and programmatic way to make sure that that bought it you're releasing to the public actually does attest to the things that you're claiming it does and we've chosen the correct remediation steps to close bugs or issues that were surfaced within me on it and then there's education that's also extremely difficult if you're in a smaller company an incident response which is the one for me that has been the most interesting and watching space because other people in conventional technology but I also get challenges that they maybe wouldn't know how to solve even if we told them which is kind of exciting and I personally am very competitive with security like I want us as a space to win I want to I want to be like looking while everyone is making stupid blockchain jokes so this incident response areas thanks for thanking me a huge opportunity to shine Wow and then there's just a couple of other things about security that I've really learned and taken to hearts and practitioner security is very much a shared responsibility there's only so much I can do is core maintainer or a core developer to make my community secure there's lots of stuff that I need them to do and to be set up to do to help so that we're able to quickly resolve issues and build immunity and security is definitely much much more than funding all of the bugs in code the next time I'm in it tells me it's my job to find all of the bugs I'm going to get a level of staff see that it's relatively unmatched for me and give them a lecture about stopping the bugs from definitely more than anything else though Security's really don't enable in and one thing that's really important we all know Security's never going to be a late 100 percent guaranteed but it's more successful when we make it more expensive for an attacker to launch a successful attack all we have to do will be one is where the attackers are and make them jump through hoops and lots of money and have to do things that are not male individual scalable I'm too exhausting and maybe if not make them go away make them leave us alone for a while there are things that we can do in order to plan and prepare for incident that other organizations and other technologies can but more than anything else it's really important to recognize the decentralization might make parts of security more complex it doesn't require us to reinvent the wheel if we are more busy reinventing the wheel than anything else we're gonna miss a lot of opportunities to get very small very nuanced very important things right so this is the story of time once upon a time I made a fatal error of accepting a keynote in Australia and so I was like 17 and 18 time zones away from my core development team now I made it a point to make sure I still showed up for my meetings and did all the stuff I was supposed to do but in the middle of this week that I'm in Australia and like I'm getting late chewed up by the ocean and spit right back out at the beach I have a high severity security incident planned in my security app in once so I have been planning for this for a year and a half at this point I'm like our parts go time all the stuff that I do it works somebody give me a bug oh my god I'm so excited and a few of my colleagues but this is not exciting what is wrong with you but how did you come in and it looked at it and I realized uh-oh this is a problem basically the issue Croker security model in that it would allow for someone to be able to find bond their stink and not have to wait three weeks which is something that is built-in to our perfect to economic layer and it under my practical II anything that you would want to do on the network on not later so what we realized was to fix this we were going to have to do an emergency one er ability coordination exercise meaning you're going to have to patch the code we were going to have to develop some willing to investigate exploitation because if exploitation happened I better be ready to show people where it happened why it happened is how it happened and additionally we had to think through notification so my office wall at home has all of these sticky notes on it it looks like kind of looks like it would be received earring up fine basically what my office walls does for me is best it's got the process mapped out just document to the three or four different places and I because they already planned on having to do this knew what was next [Music] how may I have a moment but what we knew we wanted in the case of any security emergency was if we wanted to be able to be the best dependency that we could mean to people who rely on our code we wanted to make sure that we reduce harm we wanted to make sure that we've provided visibility transparency and the way that we specifically do that case is very comprehensive incident report post that I put up within a week or so of something bad happening that requires emergency exercises a couple of things we knew we were going to come up against we knew that we time was really really important here we are monitoring for exploitation we were able to see where the person who reported the vulnerability had been exploiting the buyer just to check and see if it works they did this in a way that didn't cause harm but we also knew if one person knew about the spot it was likely other people could have Co discovered it and just didn't say anything because we knew time is of the essence here it was especially important for us to make sure that our communication and coordination was fast and it was good and it was clear and when we were asking people are recommending that our validators do specific things we're not telling them what to do but we specifically aren't making sure everything we say is actionable and clear one thing that we committed to very early on was making sure that we pre notified people who are who rely on our code meaning instead of just dropping a security bug into Tuesday Monday I send our message through our forum through a mailing list or through every nail any other channel that we need to send it through to say hey heads up you're probably going to need to patch we've got a bug here and if you're high or critical disparity but the patch will land around this time part of the reason we do that is because we have an extremely global community and if we have everything decided on Pacific Standard Time always waking up at 2:00 in the morning and somebody's always you know having to jump away from the dinner table and go deal with some security patch that's incredibly important for them to apply so in addition to this we knew we were going to need a little bit of discretion we were gonna have to handle this privately we can't release an exploit to the entire world look what we found because badness happens when you do that so we realize we have to quickly stand up a safe space where we can talk to our network operators in this missive encased we already know the topology of our network pretty well so we created telegram room which was a moment that we want to hide a new table but we have new opportunities to do things much better than getting 100 people in a telegram room and talking about explanation of scenarios and security flaws one thing that was particularly rough in this specific spot though when we were trying to clean notify people but we have a critical emergency in the hopper we were trying to go and make sure that we reach out to all of these different projects and all of these different companies for environment are toast more than anything else the most frustrating thing but there were no operational practically no operational Security Act email inboxes when a human behind them there was no way for companies to get a straight security message from us and a lot of cases when we were trying to proactively reach out to exchanges they told us to file customer support tickets which is not available and it put me in a position where I had to hope that when we have emergency part forking happening in 24 hours some exchanges going to pay less attention in public process or in advisory from support all the way to engineering their security in enough time for them to act but leave and not be harmed because realistically the moment attach plans in your code Facebook and your github repo not really trivial if you're good at rehearsing hearing to figure out what changed and to find and exploit and then turn them around and automate attacks against anyone who hasn't passed yet it's basically relatively simple reconnaissance and so if you happen to know anyone in a position at an exchange to set up a security act if it company who relies on blockchain code if you're a developer who you know works with a bunch of different projects Holy's security a Kitfox but some form of obnoxious alerting so that instead of just having that be a reactant place we can at least quickly good to you and tell me what's up so we have this giant perspective and basically what happened as part of the retrospective what's this I had to figure out a bunch of ways to make the process that we had to go through this first time much much much better and I will be I'll make sure I post these slides online and it links to all of the things that I've screen right here but basically what we realize this is we built a public watching and it's probably time to figure out how and where the firefighters and our emergency services so that was interesting about me being an Australian Spence I happen to be speaking at a conference that was their national conference or their computer emergency response teams which I wanted to go there because we wanted to learn a lot about this public-private academic partnership of a bunch of organizations who get an emergency can pop up and they immediately help organizations of any size of any kind all of these problems they come when big security problems happen this particular sir even though hard to eat alone I think all had some fantastic stories about how they helped their members and how they were able to help civic governments and university is basically patched and quickly identified what we have public stuff it's important so what we realized is we should probably stand this up we can leave it someone decentralized you probably need some folks from our management probably need for developers we definitely have to give our network operators as a table you don't just have to have everyone around for an emergency we can also use this group as a way to make everyone's murder about insecurity instead of being in a position where we're the ones doling out advisories and maturity models and all kinds of tips for how to run your AWS staff or your GP put out proactive educational information to help mature the entire network so here's what we came up with as potential core refinement it's probably best when you are responding to an incident and not have to deal with like 107 people at once it gets busy it's usually better than five your end as less than 10 I was thinking about fun outfits to plan for when we have to respond to emergencies spacesuits make it really hurt too tight so I killed that idea but that being said we basically decided you know what we are going to take this group together we need to make sure that we've got people who are from different parts of the world that aren't just North America and we probably need to make sure that it's distributed across multiple kinds of stakeholders so we have valid leaders from tons of stake and we have small validator community aren't as mature on the security front and who need a little bit more support from us to be doing the best big canvas particular mix those who realize we probably should outline expectations for service so initially we were thinking maybe running here that we would have people as a part of our certain which we are working on coding now but we wanted to make sure that we're keeping them accountable especially in the areas of potentiality and respond if direct or thirteen and you cut access to the internal 'ti information I don't have to be worried that you realized you couldn't use some learner ability that you have access to to exploit you know a validator in a single environment post too much and telegram channels so there's a couple things we still like to do to make this successful we have to encourage everybody to start the security act email address so we kind send out our signal flare when bad things happen we are working on adding a security perimeter to the cosmos SDK basically a place to state your security I email address where I can query the network daily for our time note and have a Security Response list instead of going through like 110 websites and any given time and hoping and by the way to get to human and also realized we should probably go ahead and start working on a model that allows us to elect individuals to be represented in this instead of just picking on a bunch of people I know that's probably not the best way to make this particular group work especially because of the need for trucks what's really interesting about this is we internally have been playing this on the movie called inquanok receipt and we've been thinking about how floor to garments can be a tool to help us accomplish things in a wider context but I'm pretty proud that I got to work with a copy of our protocol apparently this is really the first application of that liquid democracy principle that we have been talking about over the past when a delivery to a new module is we're making sure that we are writing the code that will govern the ability to turn out transfers if there's an emergency we have ten people and we need to go interesting but there's one particular thing where basically they have a complex ceremony the other goal if I flip the room key among seven that's the example that I'm looking to engineer so finally really quick plot chains and forward mediation tools more than anything else nine-tenths of security is making sure that you're using the right tool to get the job done and happen to think on the incident response part we could build the security incident response that we want to be in the world that being said I wanna make sure that I have a shout out to my core developers who have been helping make this idea come to life especially rifle into the cloud he is one of the first people to get excited about liquid democracy and whose natural state is just one so thank you very much if I have any questions and rant over a little bit but happy to take them in the hallway [Applause] you