Coinbase Security Series Best Practices for Key Management and Wallet Security

Hello everyone. Thank you for joining us today. My name is Joseph and I and Eli are blockchain security engineers at Coinbase. Our team focuses on securing all blockchain components at Coinbase. This will be our third Coinbase security series session and today we'll be talking about best practices for key management and wallet security.

And Eli, you have the stage.

Cool. Thanks, Joseph. Everybody, my name is Eli. I'm a senior blockchain security engineer at Coinbase. I work closely with Joseph and the other presenters, Opel, Matt, Emra, and Alexis, but on a slightly different team, CryptoSc.

So, CryptoSc mandate is to help design secure custody solutions to manage the more than $300 billion worth of assets that Coinbase holds on behalf of our customers. Specifically, I focus on the systems that integrate and interact with blockchains. So these include wallets, key management systems, nodes, and protocol integrations. Over the next half hour, we'll break down three critical stages of custody: key generation, key storage, and key usage, and explore the concrete strategies Coinbase employs to secure billions in in digital assets. Think of this as a guided tour through the life cycle of a private key, how it's born, how it's protected, and how it's safely put to work.

Whether you're an engineer looking to harden your own wallet infrastructure or a security leader evaluating custodial solutions, my goal is to leave you with practical battle tested takeaways you can apply right away. As is the case for every one of these talks, please note the disclaimer at the bottom of the screen. First, here's a quick road map for our session. Start with custody basics where we'll break down a cryptocurrency wallet into its three core functions so we have a common mental model and then we'll dive into these three functions starting with key generation. In key generation, we'll look at how to create truly random high entropy private keys and the ceremonies that protect that process.

Next, for key storage, we'll discuss cold storage principles and compare multisig, shmeir secret sharing, and MPC techniques, highlighting Coinbase's real world approaches. On key usage, we'll walk through how keys move from storage to signing, covering payload preparation, policy enforcement, and safe liquidity management. Finally, Joseph and I will wrap up with a Q&A segment to tackle specific questions. With that map in mind, let's dive into custody basics. Before we get too far into the technical details, it's crucial to share a common mental model for what a crypto wallet really does.

At Coinbase, we boil it down to three elemental stages: key generation, key storage, and key usage. Why does this breakdown matter? Because it turns an overwhelming threat landscape into three focused problem spaces, each with its own controls and monitoring. When something goes wrong, we can immediately pinpoint which pillar is affected and respond accordingly. Now, let's walk through a single transaction life cycle to see how those pillars connect.

Keys are first generated and locked away in secure storage. When a signature is needed, a subset of those keys is brought to a signing ready state. The unsigned transaction payload travels to the signing component and the keys do their one job, which is to produce a valid signature. Finally, the sign payload is shipped off to a blockchain node for broadcasting and the keys retrieved back into cold storage. I know it's kind of awkward language to say signing ready state.

Sometimes like with Shamir, the signing component is truly a geographically set the centralized entity and the key is whole in memory during the signing operation. So in this case, the keys are indeed brought to a signing ready state and that they reconstructed live. Alternatively, with NPC, the keys are kind of already in a signing ready state. And so for NPC, we'll just say that signing ready means the shares have been decrypted or whatever. We'll talk more about the differences between these two techniques in the key store section.

But I find it helpful to break down a wallet into these subsystems and subcomponents because it helps to illustrate where the threats exist. All right, with that framework in place, we can now zoom in on the very first pillar, how to securely generate private keys. Let's begin with the private key itself. On the screen, you see a 256-bit hexadeimal string. That single number grants control over one blockchain address.

If it's lost, the funds at that address may stay lock. In cryptographic terms, a private key is simply a large randomly selected integer. The key space is so wide that guessing a valid key is infeasible with current computing power. Liptic curve math turns a secret into a corresponding public key and address that anyone can verify, but is practically impossible to reverse engineer. Let's very quickly quantify that randomness.

So the total number of atoms in the observable universe is around 10 to the 80 key space is in the same ballpark with 256 bits. It's around 10 77. So what does that mean for an attacker? Even the fastest classical supercomput would take around 10 the 53 years to brute force every key. Even like a hypothetical but classical so like non-quantum machine operating at the upper physical limit of computation would still need roughly one quintilion years to brute force every key.

So to frame that concretely, the sun will only shine for 5 billion or so more years. So these numbers give practical assurance that if the key is chosen randomly, brute force attacks are not a realistic threat. So that leads us into a discussion around secure entropy and randomness. Entropy is the unpredictable input that feeds your random number generator. If an attacker can model that input, they can regenerate the private key no longer how strong the map looks.

So some good sources of entropy are hardware true random number generators that sample physical noise for example the diode inside a UBHSM but also the kernel's entropy pool on a properly configured system. So for Linux/dev/random and which gathers like jitter disc timings other kernel events on the other hand weak sources of entropy include functions like javascript's math.random and in the image it's sort of cloud cloud player's lava lamp wall which falls into the first category. So BFO uses this basically they take a photo of the fluid motion hash the pixels and then push the result into an entropy pool. In this case the randomness is coming from physics not software.

So the key point here is confirm your entropy source is rooted in physics and verify its health before every keygen run. If you need to generate a small amount of keys sampling/dev/random from a fresh MacBook will do the trick. But if your goal is scalable keygen to the tune of millions of keys in a single ceremony, purchase a dedicated hardware random number generator and follow the tips on the next slide. A key genen ceremony has one job. You want to prove that the random numbers are strong and that the secret never gets excfiltrated.

So if you have the budget, you can create a keygen route of trust through a dedicated ceremony setup. Here are some some tips we've learned over the past decade. Begin by isolating your workstation. So use a single purpose laptop with secure boot. Strip out ne networking hardware.

Be wary about plugging in USBs or other removable media. Once you've done that, create a key batch entirely in RAM, maybe encrypted RAM using your Harbor random number generator. Once the keys exist, you'll want to encrypt them because eventually you'll have to ship them outside the tent. So encrypt them with a symmetric key that's shared and stored securely. For extra credit, compute a Merkel route of the public keys and publish that route so that anyone can verify a given key was created in the tent by creating a Merkel proof.

Lastly, move the encrypted bundle onto right once media. When the exports are safely out of the laptop, zero the RAM, wipe any temporary storage, and if practical, shred or melt disposable media. All right, we figured out how to create strong keys. Now, we have to store them securely. So when I say key storage, think of everything that happens after the key is born, but before it's asked to sign anything.

So let's start with cold storage principles. Cold storage is the first safeguard after a key is born. The rule of thumb is keep the secret offline, keep it split, encrypt it, and back it up. In practice, that means an air gap laptop or HSM generates the key. The encrypted export is written to onetime media and that media never touches the network port.

Copies of that key live in at least two different distant facilities. So a regional disaster or break-in at the single vault can't halt operations or cause fonts loss. Each shard stays encrypted. So stealing one drive is meaningless without the quorum required to decrypt. Finally, maintain multiple verified backups under the same quorum policy and run periodic recovery drills to prove that you can restore share on demand.

When these four habits work together, an attacker would need physical access to several sites plus the means to defeat both encryption and quorum controls. An unlikely combination, which is exactly the goal. All right, let's talk about a classic multisig, which is a classic way to perform cold storage and to meet the principles we just discussed. So, at a high level, a multisig wallet is just a smart contract that refuses to move funds until it sees t of n valid signatures. Most chains have mature tooling for this.

So, it's the first stop for many teams and people like it because you can use off-the-shelf hardware like leed or treor as key holders. The underlying contracts are open source and battle tested and you can bolt on like time locks and guardian accounts, social recovery if you want them, but there's some trade-offs with this. Blind signing is a risk. If the wallet UI lies about what you're approving, the T-signers can be tricked. And so this kind of reminds us of the private incident.

Another trade-off is you need to have a contract deployed per network. So if you support a bunch of L1s, that's a lot of deployments, grids, and audits to keep in sync. If you plan to use a multisig, keep the human quorum, but add an independent policy signer that can parse the raw transaction and veto anything outside your rule set. It's a simple way to stop insider collusion or UI spoofing without giving up the transparency and ecosystem support that make multisc appealing in the first place. Next up is Shamir secret sharing.

Think of it as the original break the key into pieces strategy. Shamir allows you to take the private key, split it into n mathematically linked shards, and set a threshold t. say any three of the five shards can rebuild the secret. The math is from the 1970s. It's well understood and you can find libraries for almost every language.

Shmeir is great because as I mentioned the algorithm is simple, well understood. It's also chain agnostic. So just because you're you're splitting bytes so it works for Bitcoin, Ethereum, Salana, anything that uses a private key. But Shamir can be problematic because at signing time you have to pull a threshold of shards onto one machine and recreate the whole key in a single geographic location. So the signing machine must be just as secure and air gapped as the original key gen.

And if it's compromised a mal or an insider can grab the full key and all your storage and key sharing efforts evaporate if that machine isn't locked down. So, if you can't guarantee an airgapped singlepurpose signing machine, treat the key as burned. Sign once, then immediately move all funds to a fresh address controlled by a brand new split. Keys are cheap to generate and you won't run out of them. So, for multi-party computation, usually shortened to MPC, it takes the many pieces one key idea a step further.

Here, the private key is never reconstructed, not even in RAM. Instead, each custodian holds a share and the group collaborates over an authenticated channel to produce a signature and fragments. The math assembles the final signature, but no party ever sees the entire secret. Teams adopt NPC because it eliminates the single signing machine. The quorum policy also lives off chain, so threshold and participant list stay private and it's blockchain agnostic, so it works at the cryptographic layer.

One set of shares can sign on any chain that uses the same curve. However, there are fewer mature audit implementations compared with Shamir or multisigs. And also protocol implementations are subtle and if network faults aren't handled correctly, uh it can expose information. So if NPC fits your threat model and your needs, use the NPC library Coinbase recently open sourced CBNPC. CBNPC is Coinbase's production MPC engine released under an MIT license and maintained public.

The codebase implements threshold ECDSA from key birth to signature so teams can adopt distributed custody without writing the crypto layer themselves. Out of the box, you get a world-class implementation audited by Cure 53 and shipped with constant time tests. You get a system that supports distributed key generation, meaning the key never has to be in plain text in a single location, even during a ceremony. You get a practical tool chain, C++ 17 core with Go bindings, Docker builds, and make files for reproducible builds. Lastly, the project includes a sample HD wall implementation, verifiable backups, and benchmark suites that exercise the full protocol.

CBMPC is a great way to mitigate geographical single points of failure without having to be a cryptography expert. With generation and storage covered, we now focus on key usage. We'll trace the path from payload assembly through policy checks to final broadcast. Key usage covers the final signature, but it really starts earlier. At the moment you build the payload that will be signed, if you're operating in a straightforward multisig with no tight latency requirements, a few discipline checks go a long way.

First, decode the call data and make sure you understand what it does. Blind signing raw hex defeats the whole purpose of a human form. Tools like Tenderly's transaction simulator or founders cast command will show you in plain language which function is being called with what arguments and what state changes to expect. Second, treat contract upgrades with the same skepticism. Recompile the new version locally using the identical compiler settings.

Then diff the bike code against the proposed deployment. If the hatches don't match, stop and investigate. Mismatched bite code is often the first sign of supply chain error or a malicious insert. These steps slow the process, but in an offline multisig workflow, that trade-off is usually acceptable. Lastly, we'll look at architectures that can automate parts of this for higher throughput systems, but the core idea remains validate before you sign, not after.

When a platform moves from a handful of manual payments to thousands per hour, the process we just discussed needs extra infrastructure. This diagram shows a common pattern for custodians that let customers withdraw to an external wallet. It starts with a user sending a withdrawal request to the send service, the public facing API layer that handles authentication rate limits and basic sanity checks. Once the request passes them guards, send service forwards details to the wallet component. The wallet's job is to handle blockchain specific logic.

Send service outsources these abstractions to the wallet layer to remain agnostic as a general payment layer. Wallet crafts the unsigned payload using ethers.js. JS for example and then the unsigned payload is dispatched to the key management system for signing after the signature is returned wallet sends the fully formed transaction to a blockchain not broadcasting. Next we'll kind of talk about the threat model of the system and we'll map out where the payment architecture could fail with the goal of adding some policy controls and some mitigations afterwards.

So starting with threat one, let's assume there's a malicious insider with access to send service, maybe SSH. But because send service translates user requests into wallet instructions, an attacker with shell access to that host can fabricate withdrawal orders. The architecture assumes send service is honest. So wallet will build and forward any payload it receives from set service. Once a forge request reaches signing service, keys sign it just like any other transaction and funds flow to the attacker chosen address.

The next threat we consider is a supply chain compromise in the ethers.j JS library. So while it depends on external code here, ethers.js to assemble transactions. If that dependency is back to back door, the library can modify the destination address and amount while still returning a syntactically valid payload.

Because no independent check compares the payload to the user's original intent, the malicious change sails through signing and broadcast. Both issues stem from implicit trust. The first in the API layer, the second in a third party package. And there's definitely a lot more issues with the diagram, but we'll just focus on these two for now. And in our next slide, we'll show how we can basically shrink that trusted surface.

So here's kind of a step-by-step way to close the two gaps we saw. We're adding a few components here to our diagram. And I want to go through it one by one. First, we start with an intent signature at the user level. So we equip the userfacing client with its own private key and require every withdrawal request to be signed.

This creates a tamper evident record of what the user actually asked for amount, destination, and currency. Then you pass that intent through each downstream service. So send service now forwards that intent rather than re-encoding the request. And the intent is signed. So it's tamper proof.

After wallet constructs the raw transaction, we're adding another component here. We're routing that payload and the intent to a new component called policy engine. And this engine, what it does is it runs a transaction simulator. kind of like in our previous slides when you're verifying call data and it extracts the concrete effects of that payload and what the policy engine does it it takes the output from the transaction simulator and compares it against the original user intent. If the effects line up then the policy engine can sign an add astation and we can pass the add a station to the key management system and it can automatically verify the add astation against the payload and sign.

And so this workflow pushes trust outward towards the user and adds an independent policy gate before the keys get involved. So now let's see how these controls would neutralize the threats we outlined. For the first threat, assuming we've got, you know, a malicious insider with access to send service. Because the attacker at the send service layer lacks the user's private key, they can't forge or modify the signed intent. Any requests they inject will fail the signature check at the policy engine.

So it never reaches the signing stage. In other words, the policy engine when wallet sends it payload and it doesn't have an intent or it has a false intent, policy engine will reject it. It won't issue an addestation. For threat two, assuming there's supply chain compromise on ethers.js, even if the library modifies like the destination address or the amount, the simulator will surface those changes.

When the policy engine compares the simulated result to the intent, the mismatch causes the policy engine at a station to be withheld and the KMS rejects the signing request. In both cases, the decision point moves outside the component that could be compromised, forcing the attacker to break multiple independent controls to succeed. So yeah, that's about all we had for for you today. Thanks for your time and I'll pass it over to Joseph for questions.

Yeah, thank you so much, Eli, for the presentation. If there are any questions, kindly drop us in the comment section. But hi, I have a few questions myself. So I know we went a lot over um how to handle like signing and generation of the keys, but how would you advise projects that have that have like need for multiple keys? So you have multiple private and public keys or let's say they have a contract that has multiple roles and need privileged addresses for each of those roles.

How do you advise handling those multiple keys? Yeah, that's a good question. So ultimately it depends on your latency requirements for those keys. So if you don't need low latency signing then follow the cold storage principles we outlined earlier. So keep them offline distributed encrypted with backups and perform transaction parsing and simulation.

Otherwise if you do need your keys to meet like low latency they need to sign very quickly follow the guidance around automated signing validation. So you want to capture the original intent or the source of truth and then evaluate that signing payload against a list of policies to make sure it matches what you expect.

Okay. Thank you. And um I know you mentioned um having like multiple backups and testing recoveries. So but I I also wanted to ask like how do you ensure is it possible to fully guard against like permanent loss of a private key that will lose that can cause controls and how can you do this securely in such a way that it doesn't involve you know just backing up your key in a Google drive which can like be exploited.

Yeah. So it really depends on your budget. So you know you obviously yeah you you need these redundant encrypted backups. So you could use private S3 bucket, you could use multiple S3 buckets, you can put them in different AWS regions, you could go multicloud. The more complexity you add um in distribution you add Yeah.

is a benefit to you. But you know, you can also use like a safe deposit box at a bank or multiple banks. You can use physical data centers with badge readers. Yeah. If you have the budget.

So, um, it really depends on how many funds you're custodying, um, what your threat model is, what your expected attackers are, and then you can kind of scale up from there and create a a system that requires multiple layers of manual approval or is easier to get to for less critical resources. Oh,

thank you. Um, so I know we've seen cases of and multiple hacks running billions of dollars have originated from compromised keys. So I would like to say what's what would be like the best action or what would you like advise as like a process to go through when you find out that the private key of your project is compromised or the signing key is compromised like let's say ownership takeover of your contract or some ownership withdrawal happened on your contract or you still have access to the private key which is now compromised. Yeah, it's a great question. So I mean like ultimately the goal is to pause the contract if possible if it can't be unpaused and by the attacker has access to those keys and and rotate the the keys.

But it's also important to think about key compromise before it happens. So when you're designing your contract and your your rules, apply the principle of lease privilege. So if it's a minting key for a token, give that key a rate limit. If it's an upgrade key, consider using a time lock if it meets your your business needs. If the key holds funds, then distribute your funds across multiple keys.

So look at the permissions you're granting to that key and see if you can limit it and maybe have a a for a minting key example, you can have like a rate limit override key that sits in cold storage and if you breach that rate limit, you can increase it. But yeah, that's all proactive. In terms of reactive steps, like I said, you ultimately want to rotate those keys uh to limit the damage, contain the damage. So you should build like a robust key rotation process runbook. Yeah.

Uh do like fire drills with the runbook and ideally with the goal of being able to activate it quickly and securely at the event of a real threat. And then on top of that, another key component is monitoring. So you want to observe basically suspicious activity from any of your privileged keys that you're worried if they were compromised, it would be damaging to you. So yeah, just you can create alerts on anytime a transaction comes from those keys and triage those alerts and if they don't come from a legitimate source, then that's when you execute your runbook.

Yeah, thank you. I guess one more question for me. You mentioned rotating keys and I guess that would be a straightforward thing to do if there's a compromise, but what's your advice regarding rotating keys without a compromise? So is it fine to just use a set of keys for a long period of time if you have like a secure and extremely secure ceremony and setup and storage. Is it fine to just use those keys or is it there like a better theme like okay let's say you rotate in over a certain period of time?

Yeah. Uh so you know when you rotate cryptocurrency keys you you have uh you know a gas cost and if you're like automating it based on some time stamp then you got to make sure your automation is locked down and very secure because if somebody can compromise that then they can rotate your key automatically to something an attacker controls and if you're not automating it then you're taking on some maintenance burden. If these capabilities are within your arsenal, like you're okay with the gas, you're using base and it's really cheap and either you're automating it really securely or you're okay with the maintenance burden, then I think it's a a best practice. I think it's good hygiene. I think SSL certificates, they, you know, have a maximum life of around 13 months and they try to expire them even shorter than that today.

So I think that's a good paved road for a pretty sensitive secret for a lot of websites and companies is like the rest of sale certificates. So you could use that as a time frame. But if you don't want to go through this sort of heavy lifting process, one thing you can do is define what constitutes key exposure. So, for example, if you take your key with you through airport security and your bag gets confiscated temporarily, ask yourself, does that warrant key rotation or are you okay with that? If you know you've got uh encrypted key material um at a data center and somebody gets access to the encrypted key material, yeah, they got access to it.

It's encrypted. Ask yourself, does that cause different key exposure? and write down sort of a list of all of the things or or the burden that an attacker would need to meet in order for you to consider something compromised and warrant rotation. Key rotation is not super expensive if you're doing a onetime thing. So, you can set a fairly low tolerance for that as long as you're okay with the maintenance burden.

Okay. Thank you so much, Eli, for the presentation and for answering the questions. I guess that brings us to the end of the presentation today. In the next two weeks, we'll be having another presentation about fishing and social engineering awareness in web 3. And thank you all for tuning in.

Have a great day. Bye.