Breaking down the Bybit hack: Lessons learned from crypto’s largest heist

In potentially the largest crypto heist in history, Dubai-based crypto exchange Bybit announced they’d lost more than 400,000 ETH and stETH, worth almost $1.5 billion at the time, to a sophisticated attack, which the FBI has attributed to a North Korean state-sponsored hacking group called “TraderTraitor.”

The hack compromised Safe’s multisig interface used by Bybit to take over one of Bybit’s storage wallets. When Bybit attempted to reload their hot wallets, the transaction they put into a Safe interface didn’t match what was eventually signed. Instead, the signed transaction upgraded their multisig contract to one with methods for the attackers to drain the wallet and further upgrade the contract. The hackers used this upgrade to drain the wallet immediately after executing the transaction.

Attacks of this caliber can leave people feeling helpless, particularly those trying to shore up security and ensure that their applications can’t be manipulated into stealing money. By analyzing exactly what happened, we can identify a few lessons for crypto teams to ensure their products are as secure and as fail-proof as possible. 

So, in this post we’ll explore what happened to Bybit and Safe, and what you can do to make sure you’re not susceptible to a similar campaign.

What happened to Bybit

The transactions used by Safe use the EIP-712 signing format to make it easier for signers to understand what they’re signing. When looking at the transactions involved with that lens, we would have seen something off:

Likely intended transaction

Modified transaction

If those details were available, then it would have been easy to spot the issue and refuse to sign the transaction. However, using specific hardware wallets may have limited Bybit’s visibility to something like this:

Likely intended transaction

0xcb9526896b19c507e261ae34f1217c6e20751bd27a727a5f378be2e9735c1a0d

Modified transaction

0xc5d1e6fffacbf0402919e52108d1571c4df45beb0e375c9bdccd039143005941

These hashes are impossible for people to discern whether the transaction is malicious without using some auxiliary program, so if they trust what is telling them to sign, they’re likely to sign.

In this case, Bybit would have trusted the details presented to them by Safe, and not been able to verify the details independently when signing them. Thus, all of the $1.5 billion inside the wallet depends on Safe and their network connections to Safe.

In short, if something were to happen to Safe, Bybit would lose a lot of money.

Recognizing this dependency, Bybit could have had some method of verifying transactions being signed by the wallet software — prior to allowing them to be signed. The whole reason for using EIP-712 signatures is to allow users visibility into the transactions they’re signing at the wallet-level, allowing them to make sure data is correct.

Given the amount of money Bybit had inside their multisig, losses could have been mitigated or prevented entirely if they had been able to:

1. Custody funds in separate wallets to make losses less catastrophic
2. Use different interfaces for each signer, possibly locally compiled and deployed versions
3. Employ specialized tools for analyzing transaction information prior to signing

Even if Bybit had taken these steps, their security is only half the equation — let’s take a look at how Safe may have been compromised…

What happened to Safe

Safe provides a set of audited onchain multisig contracts, and a set of web and mobile interfaces for users to operate them.

Safe’s interfaces have no login and no way to directly sign transactions on the user’s behalf. Users provide their own wallet software, with which the Safe application communicates to get transactions signed.

Preliminary reports show that someone was able to inject the following code into the signing and execution process of the Safe web interface:

This code does a couple things:

1. It denies a Bybit signer (0x828424517f9f04015db02169f4026d57b2b07229) and one of their test addresses (0x7c1091cf6f36b0140d5e2faf18c3be29fee42d97) from using the signing functionality
2. It checks to see if the Bybit multisig (0x1db92e2eebc8e0c075a02bea49a2935bcd2dfcf4) or their own test multisig (0x19c6876e978d9f128147439ac4cd9ea2582cd141) is doing the signing
3. If it hits the targeted wallet with a transaction using a normal call (operation==0) then it substitutes the transaction data, asks for the signature, and then sets the original transaction data back in the response.

If you happen to read the code, available here, then you’ll notice the snippet above does not follow the same formatting as most of the code. This suggests that attackers placed it directly into S3 after the code was minified.

The current hypothesis on how the code got inserted into Safe’s interface is that a single developer was compromised. It’s just as likely that the attacker used developer credentials in fairly routine offensive operations for a while (typically a few weeks or months), to eventually get enough access to place code into Safe’s interface.

There are a few different steps Safe could take to help protect themselves from this sort of attack, but preventing it completely is a difficult, if not impossible, task. Best practices include having:

1. Advanced capabilities to detect and respond when production assets are being modified
2. Advanced capabilities to detect when a developer has been compromised
3. Tight controls around code repositories, build environments, deployments, and more to keep them from being easily modified; and, if they do need to be modified, to ensure that the right people are using the right process

What you can do: Process fundamentals

None of us want to get hacked. There are some basics everyone should know about, but what we saw happen in the Bybit attack was sophisticated and difficult to avoid entirely. That said, companies are not powerless — there are steps teams can take that become increasingly important when dealing with large sums.

Double check

Refuse to sign any transactions if your wallet can’t tell you what the parts of the transaction are, or if you don’t fully understand what you’re being asked to sign—especially if the wallet contains a large amount of funds. This may involve finding tools, or writing your own, to help you understand what’s happening. It may also involve performing simpler transactions to other smaller wallets in your possession.

Compartmentalize

Some wallets simply don’t allow the ability to view transactions. In this case, users kind of have to trust what’s being signed unless they want to spend a lot of time verifying it. If this isn’t something you or your team have time to do, then compartmentalizing helps reduce the impact of a bad event. This means that if you need to sign transactions you don’t fully understand, it’s best to use wallets with an amount you’d be willing to lose. Worst case, you lose the contents of that wallet which is a lot better than losing effectively everything.

Perform verification of and use static interfaces

Get a local copy of an interface, and verify that it cannot pull assets from the internet, or have code loaded into it from outside of your device after it has been deployed. Then use that for signing. This can include local web deployments, browser extensions, and mobile applications. It’s even better if the interface is on a machine or phone that is completely separate from the other parts of the business. 

Secure your software development lifecycle (SDLC)

To prevent attackers from using your company to attack others, you’ll need to ensure that code deployment processes are locked down for anything that creates transactions or helps users perform transactions. This means that no single person can easily modify code repositories, build environments, artifact repositories, deployments, or any dependencies. And that if something strange happens, you receive an alert.

All of this is a lot easier said than done, but it’s necessary to mitigate risk — tightening up processes, including which processes are happening and who is initiating them, goes a long way. Still, many companies do not have best practices down. All of these come with varying degrees of complexity, but it’s important to take them seriously as a first line of defense. 

What else you can do: More strategies to consider 

The elephant in the room, however, is that even the most diligently administrated systems are still vulnerable. 

The truth is that attackers — nation-state attackers in particular — invest untold amounts into perfecting increasingly sophisticated attacks. As a result, teams must take a more holistic, more locked-down approach for their most sensitive systems. These systems include any wallet with a lot of money in it, any high-privilege admin account, any high-privilege cloud account, any account that contains important keys, or any other device that needs additional guarding.

A few strategies to consider: 

Decentralize

 It isn’t difficult to hack one thing. It’s much harder to hack 100 distributed things all at once. Spreading responsibilities to as many entities as possible reduces the risk of a full compromise. This involves creating multiple independent systems that depend on organizations relatively unassociated from one another. 

Think of it like a multisig, but to the nth degree: A number of unassociated systems that all serve to verify something independently.

Decouple

Get devices that are as decoupled from your other tech as possible. Importantly, never power on the device in proximity to your phone, home, or office. Additionally, be careful to ensure that, when you are trying to operate the device, no one is tailing, monitoring, or otherwise watching you. 

The reason for this level of vigilance is that nation-state attackers will bridge between any two environments that have any common connections. If you plug in a USB device in both, they’ll use that device to compromise both environments. If you have another laptop on your network, they’ll find out where it is and find a way to compromise it.

This isn’t hypothetical: It’s actually happened. So it’s important to disconnect sensitive devices from all other aspects of your life. This way, anyone monitoring your use of any given device will have no idea that others exist.

Harden

Some devices come with good hardening options — like lockdown mode for iPhone or Android, and virtualization-level separation for specific Linux distros — that make the device a lot less friendly to use. 

The tradeoff here is making sure your device isn’t as susceptible to being hacked by removing features prone to issues — including most direct messaging clients, all graphical email clients, most browsers, all social media, and more — for any devices that help operate these sensitive systems or wallets.

What would this look like in practice? Some admins are going to need laptops with access to the network and the ability to perform necessary admin operations. If a machine has a wallet containing $1 billion, then there is no reason for anything to be on the device — except software that’s absolutely necessary for signing a transaction (that an administrator inputs into the device manually or using a QR code) and the embedded camera. 

Minimize

Similarly, make sure sensitive devices and keys are doing as little as possible. The less you need to use them, the less likely you are to make a mistake while using them. The name of the game is to make as few mistakes as possible and minimize the surface area for future mistakes.

For example, if you have a multisig signer and need that wallet to be secure, it’s best to have everything necessary for signing to be on the same system responsible for signing, but only those necessary parts like a simplified interface, a local signing library, and a way to send the signature to another system. By reducing the number of components on that system, we reduce the number of ways that system can be compromised. This is what security devices, like hardware wallets, are meant to do, but some lack features necessary for users to securely use multisigs.

Obscure

A veil of secrecy makes it more difficult to find weaknesses in a system. If those involved understand the reasoning and importance of this veil, then information should flow out slowly enough that putting everything together would be difficult for an attacker.

That said, security-through-obscurity should never replace state-of-the-art security measures, but it can complement the security best practices you already have in place.

***

The good news is that, short of being on the bad side of a major government, you shouldn’t have to be too worried about a Bybit-level hack. And if there’s anything we’ve learned from our analysis, it’s that there is a lot of room for teams to improve, secure their processes, and double down on device security.

Stay safe out there!

***

The views expressed here are those of the individual AH Capital Management, L.L.C. (“a16z”) personnel quoted and are not the views of a16z or its affiliates. Certain information contained in here has been obtained from third-party sources, including from portfolio companies of funds managed by a16z. While taken from sources believed to be reliable, a16z has not independently verified such information and makes no representations about the current or enduring accuracy of the information or its appropriateness for a given situation. In addition, this content may include third-party advertisements; a16z has not reviewed such advertisements and does not endorse any advertising content contained therein.

This content is provided for informational purposes only, and should not be relied upon as legal, business, investment, or tax advice. You should consult your own advisers as to those matters. References to any securities or digital assets are for illustrative purposes only, and do not constitute an investment recommendation or offer to provide investment advisory services. Furthermore, this content is not directed at nor intended for use by any investors or prospective investors, and may not under any circumstances be relied upon when making a decision to invest in any fund managed by a16z. (An offering to invest in an a16z fund will be made only by the private placement memorandum, subscription agreement, and other relevant documentation of any such fund and should be read in their entirety.) Any investments or portfolio companies mentioned, referred to, or described are not representative of all investments in vehicles managed by a16z, and there can be no assurance that the investments will be profitable or that other investments made in the future will have similar characteristics or results. A list of investments made by funds managed by Andreessen Horowitz (excluding investments for which the issuer has not provided permission for a16z to disclose publicly as well as unannounced investments in publicly traded digital assets) is available at https://a16z.com/investments/.

The content speaks only as of the date indicated. Any projections, estimates, forecasts, targets, prospects, and/or opinions expressed in these materials are subject to change without notice and may differ or be contrary to opinions expressed by others. Please see https://a16z.com/disclosures for additional important information.