Open Sourcing Subzero

Late last year, Square began offering the ability to buy and sell Bitcoin with Cash App. As part of that launch we invested heavily in building out our cryptocurrency infrastructure to help protect our customers’ funds, and Square itself, from both internal and external threats.

Cryptocurrencies like Bitcoin require private keys to access and move funds. Securing these private keys is important because transfers are irrevocable. An unauthorized person who gains access to the private keys can move funds and due to the nature of cryptocurrencies, it would be practically impossible to identify the person or recover the funds.

Since launching Bitcoin support, Square developed a robust approach to Bitcoin cold storage, and we recognize the importance of sharing our work with the community. As a result, today we’re open-sourcing the documentation, code, and tools for “Subzero” our HSM-backed solution for protecting Square’s Bitcoin holdings.

Background

Cash App customers have the ability to buy, sell and withdraw Bitcoins. They can view the balance for their account, but the actual Bitcoins are held by Square in a combination of hot and cold wallets. Cold storage is a term used to refer to a Bitcoin wallet that is offline, meaning that the private keys for the wallet are stored in such a way that they are inaccessible from any network. Storing Bitcoin in a cold wallet reduces the risk of remote attack. Without remote access, it is more difficult for an attacker to obtain the private keys that give access to the Bitcoin stored in the cold storage wallet. Transferring funds out of cold storage requires physical access to the private keys or the device on which they are stored. On the other hand, an online or hot wallet is able to send funds at any time. Keeping a small amount of funds in online wallets and the majority of funds in cold wallets is a good way to get the best of both worlds.

There are many ways to implement cold storage. You can print the key material on a piece of paper (a so-called “paper wallet”), carve it into stone, save it on a computer that is kept disconnected from networks, or use a specialized hardware wallet device, such as Trezor or Ledger.

Hardware Security Modules (HSMs) are specialized hardware devices that are commonly used in the payments industry to store sensitive cryptographic key material and perform operations with those keys. HSMs provide guarantees around the security of sensitive key material, such as strong access control, and active protection against physical tamper. HSMs also typically allow replicating keys (for backup or disaster recovery purpose) without exposing the plaintext key material. These features make HSMs a natural fit for a cold storage solution, as our primary concern is securely storing the wallet’s private keys, while still being able to use them to transfer Bitcoin when needed.

What is Subzero?

Subzero is an enterprise offline Bitcoin wallet. It is useful for those who don’t want to use an off-the-shelf hardware wallet because they need to create unique customizations or implement business-related rules. We decided to use the same HSM vendor that we use for all our other payments-related needs, as we are already familiar with the hardware and software. The HSM is programmable, allowing us to support Bitcoin wallets today while giving us the flexibility to implement other protocols in the future.

One specific customization we implemented is the ability to enforce that cold wallets can only send funds to a Square-owned hot wallet. Such layering provides defense in depth; forcing an attacker to compromise multiple systems in order to extract funds. It is also possible to build additional layers, where each layer can tradeoff convenience with the amount of funds being stored (onion model).

Our multi-party signing ceremony leverages the multi-signature feature available in Bitcoin. Moving funds out of cold storage requires m out of n people to participate in a signing ceremony. The participants use a combination of smart cards and passwords to authenticate to the cold storage system. QR codes are used to exchange the minimal amount of data needed between the offline and online world. The cold wallet, which is physically secured, remains offline throughout its lifetime. This provides assurance about its integrity; while the supply chain needs to be secured, unauthorized modification of the active system is unlikely.

A geographically distributed setup makes it both harder to compromise (m different sites need to be compromised) and provides redundancy (we can lose any n-m-1 sites). A nice property of our signing ceremony is that we mitigate the risk of handling an incorrect transaction due to corrupted data (either in transit or in memory). The exact same bits would need to get corrupted in an unnoticeable way on each of m different servers.

Subzero Details

A signing ceremony starts by having an online server generate a QR code. The QR code contains the minimal amount of information necessary to sign a transaction. QR codes are efficiently encoded using Protobufs. These QR codes can be printed and archived, which is useful for forensics purpose. QR codes also constrain the amount of data that can possibly be exchanged.

The people performing the signing ceremony interact with servers. The servers are located in undisclosed secure locations. The servers boot from a Linux Live DVD. DVDs provide two desirable properties:

1. Unless an attacker is able to modify the drive’s firmware, the media is immutable. Immutable discs make it harder for potential malware to exfiltrate information or for malware to erase itself.

2. When a software upgrade is required, DVDs can be swapped without having to bring the servers online.

Our long term goal is to have a reproducible build process, similar to Tails.

The server runs a custom application (Subzero UI) written in Java. The application prompts for a QR code to be scanned. The participants are able to review the transaction being signed before they unlock the private keys.

The application communicates with the HSM. The HSM runs Subzero Core, a custom Bitcoin wallet implementation. The code is written in C and implements Hierarchical Deterministic Wallets as well as Segwit. We leveraged Trezor’s open source projects (such as trezor-crypto) as much as possible.

The business logic rules are enforced inside Subzero Core and the private keys are never exposed outside the HSM + Subzero Core boundary.

Beancounter

Finally, we wrote a tool, Beancounter, to audit our wallet balances. The tool is written in Go and addresses needs that aren’t fulfilled by existing wallet software, such as the ability to compute the balance at any given date in the past, and the ability to handle wallets with very large number of transactions. Beancounter also has some other useful features, such as mapping dates to block numbers.

Our repo contains documentation, code to build our DVDs, the GUI, the wallet that runs on the HSM, and a few other utilities. Some of our code is tightly coupled to our specific hardware, but we are open to contributions enabling support for alternative vendors.

We hope that by sharing our work, we can make it easier for others to fulfill their security needs, enabling even more innovation — and better protection for all players — in the cryptocurrency space. In the long run, since we had to solve problems that other companies may face, we are interested in standardizing some of our work.

We look forward to hearing your feedback and suggestions as we continue to improve Subzero, the code is available on https://github.com/square/subzero.