Security Summit 🦺 Why DevSecOps is Essential to Blockchain Security

[Music] today in in this talk I really wanted to go over what it means to secure the blockchain in a continuous basis of course we've all heard about getting the audit of course insurance is a huge part of that but as developers the question comes down to us you know what can we do to kind of go ahead and protect our protocols protect our contracts and that's what I just wanted to go over at a highest level in in this in this talk today to give you a bit of background on myself and the CEO and co-founder at guardrail it's a real-time security monitoring platform for the blockchain and of course when we're talking about security monitoring that's a huge part of deaths offs so this is something that spent a lot of time thinking about and what I wanted to do today is share some of the learnings that we' have seen working with teams across the space just to bring to you guys what is the best that the best teams are doing so given the audience today you know it's going to be a mix of developers security you know maybe amateurs and Security Experts didn't want to go to Deep deep into a specific area so today we'll be um going through um a bunch of different you know Realms of security and then I'm happy to pick up this conversation offline and talk to any of you guys on any of the areas that are most relevant with that being said let's Dive Right In um this is a rough agenda we'll just be going over what devops is how it relates to crypto and then a framework for how I like to think about Dev seops into space and then we'll wrap up with some concluding Thoughts with that let's Dive Right In so first of all some of you guys might be wondering what is depths offs um it may be a confusing term it may be something that you haven't heard of before um so to simplify it really it's just devops and security put together pretty simple now so a few guys might be thinking okay that's uh taking an easy way out so what you know what's the really the reason behind def gols entirely um the reason that we've been talking about this at all is that you know over the last decade the way we ship code and the way that oig are building are are changing pretty rapidly especially in blockchain but even in the broader you know web 2 ecosystem um the cus of deployments is growing very rapidly this means that you know there isn't a a feature freeze or a security testing time you know once a quarter um now code is being written every day it's shipped you know every week and that means that security is something that can't just be tacked on at the end but needs to be integrated into the code development life cycle so that's where you know devops is talking about you know bringing in agile performance agile performances to code development so that you know it's not something that takes a long time but it's something that can be built in um in a more continuous uh deployment process and then def cops talks about how we can build guard rails around that to be able to not have it be kind of a gatekeeper that has to approve everything and add additional delay to what developers are trying to do so really the challenge here is how do we redesign the security function to fit into agile and of of course this is something that that not just is relevant for blockchain but is relevant for the broader space too um if you're interested in learning more about this actually just yesterday Google announced a report it's called the state of devops it's released every year it's really interesting to hear about how the broader Tech ecosystem is dealing with devops and today of course we'll be focusing on blockchain but if this is something that you know you're going to be doing deploying contracts securing them continuously at launch definitely encourage you to check that out so you can bring in the best practices so with that with a rough understand what def Cyclops is let's talk about how it relates to blockchain so first and foremost when def Cyclops meets crypto we have to recognize that you know rapid growth and experimentation is pretty core to everything happening in blockchain space um the entire space is less than 10 years old in terms of the applications we're talking about today and in that case you know even the the applications that are considered mature today like l1s um you know they constantly have their own development plan and a change on the L1 ripples up to l2s up to Da rles up to you know all the applications that we're building um so really I think speed is the Crux of something that you know crypto is embracing and that also means that security doesn't want to be the one thing that slows it down I that's what even the previous two two speakers have been talking about so while that's the premise of of the blockchain space and we actually look at um how applications are being deployed today generally it's still following what is the previous model which is you know folks spent a long time building and then it ends up into this code free of a month two months waiting for the right auditor or like multiple Audits and then once the auditor signs off on it it's like you know main net ASAP launch it and then start start growing pretty aggressively so there's this tension between like wanting to go fast and then you know not necessarily having the security systems and processes built to do it in a continuous manner but then in the meantime we're still doing kind of the the initial version of this so that's kind of one tension that we've been noticing across the teams that we've been working with um it's even harder because once a contract is posted most times it's you know it's immutable on the blockchain of course upgrades are are caveat to that but upgrades themselves have their own challenges but you know with smart contracts being infrastructure that's being pushed being completely immutable it can be very challenging to launch and come back and make changes to the system uh to add to this um of course it's a very challenging thing to do correctly and if you look at some of the accidents that have happened in the depth sovs World um it's not uncommon but even the biggest projects in crypto space um there's like three images right here first one is from compound having 162 million um that you know was stuck because of a bug upgrade um second one is from Wormhole which you know submitted had a contract on chain that wasn't initialized all the attacker had to do is just initialize it set themselves as an owner and then go ahead and take over um that contract um and then after that was also a also that was doing an upgrade and you know had over 100 million so that's you know some of the biggest project in the space all three are like nine trigger hacks so this shows you that you know it's it's a very core part of what it means to build a successful protocol but also when done wrong can quite a few different accidents the third one which I want to touch on here like why Dev offs is relevant for crypto is actually in the exploits world so of course there's the speed angle and there's the um the accidents but even for exploits having good devops can really be a huge game changer um first of all uh you know the attacks we're using today are existing in what we're calling attack or kill chains so it's very rare that it's purely a single transaction we usually see some level of reconnaissance configuration funding the actual exploit and then you know multiple different steps before and after the the transaction of course this may change as things move on but today this gives us the chance to be able to have the right systems in place to be able to stop this at any different point in the life cycle to be able to stop the kill chain so that's really exciting to think about how demos has that potential um additionally with one of the teams that we've been working with um we actually set up some monitoring for them to be able to identify you know you know what was happening with their tokens you in the broader block screen and we identified that if there was some tokens being trated on FTX then unexpected EA um that they were able to you know using devops and and monitoring identify pull those tokens out and this was before kind of those tokens would have been frozen under the under the FDX um kind of situation that ensued afterwards so that was a huge win for them to be able to you know have the right monitoring in place then additionally one of the teams that we worked with also they were able to set up a PG Duty call so that uh you know when a major pool imbalance happened during teral Luna they got a a call on a Saturday and were able to emergency move some funds out um if they didn't get that call they would have been 100% exposed and with that call they were only 20% exposed so not an absolute get out of jail free card but really the ability to limit the damage i''d be able to kind of help teams in in critical situations so that's setting up some context for you know why devops is important uh and of course as a blockchain industry we don't want to give up speed uh we don't want to have accidents and we definitely want to have better Tools in transex flits so let's talk through how we can integrate defs Ops into the workflows that uh we're all building today and with defs Ops just want to call out that it's not necessarily like a simple tool or Plugin it's not a linter or code analys code analysis plugin uh it's more of a framework and that's where it becomes really interesting to think about how it's applicable across every stage of the protocol um we've been seeing this in practice with quite a few different teams that we work with um and I rly have been able to see it in four key areas um the design code transactions and and protocol is how I'm thinking about it so right now let's dig into each of these and really understand how def go is implemented each of these different phases of the protocol so when it comes to design this is actually where a lot of us start when we come up with a project either hackathon or you know one of the some of the biggest projects today started off with just an early design of what it could look like um of course that design involves you know mechanisms toomics and key application flows but the the one of the cool parts that touches def cops is a thir model so when it comes to thir model this is essentially you know what what what we would call expecting out scenarios a thir model talks about the scenarios being both like good and bad scenarios and specking out who the stakeholders and interactions are um here's a quick kind of mental model I wrote out that you can use and you know to spec out you know who the users are going to be for your contract whether it's you know which DBS would they integrate with um if it's hosted on web to infrastructure what would that look like in terms of the different apis and providers which wallets you'd want to support integrate with blockchain smart contracts in terms of how many and you know what what configuration you do there and then of course you know what admin functionality there would be across the whole team so those are it's a very high level version that I just put out here maybe even the starting point for some of some of you all but I just want to point out that um as good as application design is I think threat model or something that's first step of defit go for any contract next um definitely definitely you know early on integrated logs U you know predesign some testing scenarios and come up with these invariance that you want to test for defi teams this may be you know over collateralization for nftd teams it may be some level of liquidity or some invariance around their order book uh but definitely definitely think through logs testing and invariance for your application um and what some of the best teams have done here is actually limit the area of the code base that they expect to work perfectly a good example here would be if you look at unops core and for free contracts and you're welcome to go check that out uh what they've done is with core it's very simple almost like too simple but that's kind of by Design to make sure that there's very little scope for anything that could go wrong very liit very limied chances for um you know exposure to happen there um and that way periphery is where they can do all the interesting experimentation with user accesses interfaces and apis while core continues to be like the the part that's battle tested into QR so that's a really good approach that we seen some of the the best teams in the space too of course even with this it's really important to stay on top of the protocol changes that keep coming out so definitely you know review VIPs and standards as they're happening I think when you know the the gas optimization EIP came out 437 came out a lot of these eips have been really um kind of conversation inducing in the whole space and I think that's why it's really relevant to make sure that you're not ending up you know ending up with that incompatible with the existing VIP protocol um and the last thing I want to say here is it can be very frustrating if you're doing nine out of 10 things right and and one of them goes wrong so really it's important to be very consistent and continuous here there's a couple of examples I can point out here but the idea here is you know if you get three or four Audits and you have a really strong developer team but the processes is is not great you have a locally deployed three of three multisig all in one place or you know you have a developer pushing code directly um you know from the laptop and the laptop has malware all the hard work you're doing on on the smart contract side or even on testing side can can go to shame so definitely you know be consistent across the entire stack because I can actually beware vulnerabilities so that's on the design um talking through code which is where I think a lot of us are going to spend our time um first and foremost it's you know before you start building anything really start with battle tested contracts where you can this is where 's contracts like you know the ownable initialization utilities um these are amazing because they not only save your time but they also save you vulnerabilities these are you know battle tested and then definitely where you want to begin if if you're going to be building some of the more kind of templated versions of of contracts definitely where I like to start if I'm going to be experimenting with any of the smart contract flows um when it comes to devel deploying a contract um you know Fork testing used to be very hard I remember trying to do this three years ago but today it's much easier to do using things like Foundry or hard hat um and the idea here is you know any contract that you want to deploy you can check contract State and account State before and after using I mean that fork using Foundry hard hat not that high lift in terms of work but extremely high impact to know that there's no unexpected changes happening in terms of Integrations looking at time and I think I might need to speed up a little bit here um but there's code security testing which I think is really relevant and can save you a lot of pain a couple of really awesome libraries put out by really smart Auditors like with slither kid now which I recommend you to check out um dependency monitoring which has come up in a lot of hacks recently uh not knowing that you've forked one version of a contract can be you know quite an area of demise for some projects so here there's tools like cod SL that have you know verified contracts or Pata that give you a bunch of different verifide contracts too and using those you can make sure that any contracts you're including are battl tested and don't have dependency risk and of course if you're using an audit make sure that your deployed code doesn't stray too far from that audited code if it is stringing far then just maintain that diff very carefully and go ahead and revise that very continuously I'll speed through transactions I think transactions is really interesting since it's both as a user of a protocol but also as your protocol interacts with other protocols there's going to be a ton of best practices that you should be considering when it comes to transactions first one's being test ads I think it's fantastic to see all the info that you know the entire Community is building around test Nets ourselves we' actually at guardrail implemented a bunch of test net support and our our teams that are launching really large main that projects will you spend months on on test net just creating bots and simulating test cases so cannot say enough good things about using a test net to verify you know that the protocol is working correctly investigation tooling of course you're welcome to use ether scan but you might want to think through some possible scenarios that come up and be familiar with tools like breadcrumb token flow and a bunch of other investigation tooling you know we I have my own stack and I'm happy to share if you're going to go down deeper down that route a specific monitoring for transactions I think this makes a ton of sense when um you know you might want to think about what are cases that you you want to be alerted about this could be you know bad sources of funds it could be unexpected contract that just got posted funded by a tornado that you want to know about so there's a couple of these specific scenarios that you know maybe don't happen as often but if they happen you do want to know and be able to kind of get a chance to react so this is where you can spend some time designing those and using realtime monitoring platform um and then early on set some limits and expectations for your own transactions um this is helpful to since you know as you as you launch you you might want to you know create your own definitions of what a whale transfer is what me looks like or maybe even roughly some expectations of what hourly or daily contract interactions look like and this can make it clear when maybe front running or exploits are about to happen I'm just going to double check that my audio is still working all right awesome awesome um yeah and then I think touching on protocols which is I think where we spend most of our time um definitely with design code and transactions it's it's where it all starts with protocols that's where you know we're able to really work closely with our teams to build in Def cops and then obviously build in monitoring for them um definitely feel like this is where Stakes get higher um there's more operations more chances for failures and a lot more moving Parts um so first and foremost is really to be able to track and defend your own protocol's perimeter and also the internal moving pieces so this means really understanding what are the web two API keys and what privilege each of them has on the web 3 side you know which external dependencies are there whether it's oracles or Price feeds or just making clear you know how often you'll be doing upgrades and things like that of course on the internal side we've seen this also put a couple of teams in trouble whether it's like Tech permissions or malware so these are all things that I think are are really relevant for a protocol to you keep an eye on when it comes to Deaths Ops um next it would be actually standardizing key workflows around um some of the more operational pieces that as we saw earlier could be places where things go wrong deployment upgrades and initialization are are three come to mind um Additionally you know building in multisig for admin or high privilege actions is a really good idea and of course these multisig should be people that are you know able to piece out what is um you know authorized versus unauthorized but this could usually be like the tech lead and one or two other folks but I think multi are a fantastic idea to be Implement at the protocol level um of course we're all hoping this doesn't happen but in case something wrong does happen definitely be prepared have an incident response plan with a clear point of contact um a clear list of you know who based on criteria who gets invoked and pulled into a waru it's really helpful to have controls in place to begin with so that in case you do need to take action on on pools you can go ahead and do that um and then of course some documentation and data to be able to jump through into any investigations um and of course all of this might be a lot to do and that's where I think these tools can be really helpful to monitor what's happening um the question I like to ask a lot of teams are are are really on the way to growth is you know if there is like a 100 e transfer or proof of concept that's hopes it on chain would you get to know that um you know this this hack just took place because I think it's really important that you're at least able to detect them if not go ahead and take some actions based off that um case in point if you look at some of the largest attacks that have happened recently most of them were detected by other folks security researches on Twitter which is never a good look for a project and then when it does happen it's also you know takes quite a long time to triage coms so getting ahead of this is something that you know the whole industry is learning but I think extremely vital to be able to do looking at the time here I think I'm coming up so really quickly just some concluding thoughts here um I think as an industry we're getting a lot more aware about security I remember in 2017 it used to be the case that audits were optional and maybe insurance was part of the cation today it's mandatory that you know everybody's looking at best-in-class security tooling with their lochain think customers are demanding it and Protocols are bringing it and that's just awesome to see around the whole Space and indust including that too I think there's the whole industry is getting a ton of practice with internet response which is really getting the whole Space a lot more mature which is really exciting to see I think I'm at time so I hope you found this talk interesting if it was reach out to me on Telegram and happy to continue this conversation um yeah thanks for attending