When bitcoin exchange Coinbase launched 2 as well as a one-half years ago, Heroku, a uncomplicated hosting platform, provided the correct solution, said Brian Armstrong, Coinbase co-founder, writing in Medium. Heroku provided a to a greater extent than battle-tested solution than anything the founders could exercise on their own. But the founders realized that they had to laid upwards the side past times side version of their infrastructure from the soil up, giving exceptional attending to security.
After ane year, the squad completed a transition that has operated within AWS. The sense inward developing this solution tin serve as a starting indicate for edifice productive infrastructure inward the cloud.
Coinbase currently stores 10% of all bitcoin inward circulation. Armstrong noted that the company’s safety measures are constantly evolving.
Two Key Security Principles
Two of the fundamental principles they followed are eliminating unmarried points of failure as well as layered security. The concepts are based on non putting all your eggs inward ane basket. The take in is to essay consensus as well as redundancy amid dissimilar parties. Such concepts are deployed inward nuclear launches, corporate governance, banking concern security, certificate regime as well as human resources.
One illustration is the means ane secures an administrator line of piece of occupation organisation human relationship on AWS alongside a two-factor token controlled past times some other person.
If ane someone controls the password to the account, the second-factor token goes to some other party. The minute ingredient is stored inward a condom deposit box or vault for some physical (in add-on to crypto) security. This volition preclude a unmarried someone from accidentally or intentionally destroying the company.
The developers should non demand production SSH access for their regular tasks of deploying code, debugging, spinning upwards intelligence services, etc. But it is difficult to take away the demand for SSH access. Some employees volition ever require a means to debug problems.
The Lock Down Process
When someone inward the fellowship needs SSH access, they tin follow the next lock downward process:
1) Add two-factor to the SSH. Each SSH should require a minute factor. Duo two-factor authentication for SSH pushes a asking for approving to a phone. Another selection is a FIDO U2F fundamental that is similar to a small-scale hardware safety module on a USB stick. The fellowship tin require all SSH to survive “pair programmed” to separate the keys.
2) Use exceptional laptops for SSH access. It is of import to preclude the mightiness to SSH into production using a regular laptop. Most high profile breaches are due to malware arriving on a laptop caused past times pike phishing. Some people assume that 0-day vulnerabilities or other sophisticated techniques displace hacks. The fact is that uncomplicated pike phishing – clicking on spoofed e-mail links – is a greater cause. Some attackers dedicate half dozen months or to a greater extent than to developing relationships for the piece of occupation of pike phishing. Companies should allocate sure enough machines inward a locked room that only sure enough individuals can access. There should also survive a dropcam inward the room to tape who enters as well as exits. These machines should non survive used to browse the Internet or opened upwards email. These machines demand wiping regularly.
3) Audit SSH access heavily. Establish bastion hosts for all SSH requests. Restrict access to these least-privileged hosts as well as advise the squad when they are accessed. PagerDuty is available to warning people based on sure enough commands. To preclude untraceable activity later on access, log every activity as well as keystroke going through the bastion host. Coinbase wrote a custom software for this part as well as may opened upwards rootage it. SSH log storage is every bit of import since the logs convey sensitive information. Coinbase runs a separate disaster recovery surroundings to guarantee storage of each activity inward its surroundings for 10 years minimum. Immutable logging provides an audit trail to determine the displace of a breach.
4) Limit SSH access to those who are less probable to steal. Special rules should regulation production access. Every employee granted access needs to convey a background check. The fellowship should survive prepared to number an arrest warrant if something goes wrong. This tin exercise controversy, but consideration should survive given to granting access to citizens of the province of operation.
Coinbase decided to shop 98% of client bitcoin offline inward a condom deposit box.
The kickoff version included USB drives as well as newspaper backups stored inward a condom deposit box at a bank.
The New Cold Storage
Version iii of Coinbase’s mutual frigidity storage looks different. The fellowship generates keys inward a secure environment. The keys are split upwards using Shamir’s undercover sharing. Every private fundamental consists of dissimilar parts. Some subset of the pieces is needed to restore the secret. In this manner, the undercover tin survive recovered if some pieces teach lost. The organisation also requires a quorum of fundamental holders to restore a key.
Coinbase distributes fundamental holders geographically as well as follows a protocol during fundamental signing ceremonies to verify holders’ identities.
An illustration of generating a 2 of iii key, where at to the lowest degree 2 of the iii pieces are needed to recombine the secret, using Hashicorp’s opened upwards rootage vault project, is shown below.
A fellowship tin require 5 of 10 pieces or whatsoever threshold they choose.
It is of import to log everything occurring across all containers inward the infrastructure.
Having a adept audit trail is critical. Worse than getting hacked is getting hacked as well as non knowing how it occurred. The only selection inward that scenario is to hope yous convey patched the correct affair earlier relaunching.
Proper logging also prevents theft. People are less probable to pocket when they believe they may teach caught.
High Variety Logs
An surroundings focused on depression latency as well as high diversity logs required a novel blueprint for Coinbase’s novel infrastructure. To minimize the complexity of logging, the fellowship sought to force all logs through a unmarried identify that could survive consumed inward many ways. Running bitcoin nodes globally required logging endpoints that are accessible across numerous networks.
To trim down the complexity of adding consumers as well as log producers, Coinbase instantly pipes each resultant through a streaming, distributed log providing flexible, at-least-once guaranteed, processing as good as a multi-day information buffer that tin replay as required.
The fellowship runs a fleet of Docker containers to procedure the entirety of this pipage to perform dissimilar evaluations, transformations as well as transfer information to to a greater extent than permanent homes for search, archival as well as more.
Coinbase built some other slice of software that seeks irregularities inward the logs that flow through Kinesis. When it detects something, in that location are iii warning levels.
Warnings appear inward the infrastructure slack channel that the squad tin honor for context. Someone attempting to beast forcefulness passwords as well as running into the charge per unit of measurement limiting would survive an example.
Errors triggering PagerDuty to warning someone represents a to a greater extent than serious number calling for immediate attention. Unusual displace of funds would survive an example.
Critical issues tin trigger a kill switch that shuts downward critical services, such as outgoing payment processing. Kill switches demand their ain fundamental signing ceremonies. Unauthorized access to sure enough services as well as machines are an example.
Addressing Common Tasks
Deploying novel code is ane of the almost mutual tasks. Coinbase has developed tools approximately the take in of consensus. It has a three-phase procedure inward which anyone tin suggest a change. Consensus is required to apply a change.
A tool called Sauron comments on every draw request, requiring approvals earlier deploying code into production.
This branch requires a +1 from 2 developers also the author. More sensitive services require to a greater extent than approvers. In periods of higher risk, similar a compromised laptop, an employee tin dial the number of +1’s as much as needed organisation broad without blocking all deploys. This protects against cases where ane or to a greater extent than developers have malware on their laptop. Consensus is also used when updating the environment, such as the docker-compose files for launching services.
Coinbase has run alone on Docker inward production for to a greater extent than than ane year. Before the novel deployment tools, the fellowship began edifice its ain tool called CodeFlow. This tool provides each developer the mightiness to deploy their code past times combining Docker-Compose file, Dockerfile as well as Envars for 12-factor applications.
Other topics to consider are: scarlet squad drills, põrnikas bounty programs, pen tests alongside exterior firms, working alongside vendors storing PII, incident response, as well as educating novel developers.
Images from Medium/Brian Armstrong.