Luckily we were able to restore the data, but now I (we) really want to learn what a proper setup would look like.
If you have any clear overview reading on the topic I'd be very interested to to know about it.
In particular I'm wondering: how do you back up your encryption keys, or even put them in escrow somewhere? Assuming we don't rotate the keys constantly I would love to just save them in somewhing like a passsword manager that's secured with 2FA/FIDO.
Would love to hear your thoughts!
The company solved this by giving me a root username and password that worked on every single important database in the company, at least every customer database.
I had to beg them to create a somewhat restricted account.
The same company was however deeply sceptical to all kinds of remote work. The security equivalent of penny wise pound foolish I guess :-]
To keep the charade up I sent one of every 20 requests to them to do for me.
If they slack off, at least they do it in the office and not freely at home (imagine the possibilities!)
But we aren't allowed internet access on our workstations because "security"
I wish you the energy to keep trying to change this, don't let it get to you too much, I know this stuff can be frustrating as hell.
There are multiple automated systems that scan public repos for credentials. 5 minutes later you are mining bitcoins for them.
Not particularly germane to the discussion, but really disappointed in how SendGrid handled things. I notified them immediately, rotated all API tokens, and tey could not turn it off, so the spammer sent messages for days and eventually my SG account got suspended.
The CEO was not pleased with the 30K (or maybe it was 60K) bill... and I just pointed at the CTO and was like "I fought this battle and was overruled"
It seems convenient until the credentials change (which they ought to now and then). Then when you check out an old revision of the project, it is broken. You end up having to copy and paste the new creds back in time and it's finicky as hell.
IP Rules are super helpful in this case, still need to rotate when exposed but can limit the exposure.
> When you push to a public repository, GitHub scans the content of the commits for secrets. If you switch a private repository to public, GitHub scans the entire repository for secrets.
> When secret scanning detects a set of credentials, we notify the service provider who issued the secret. The service provider validates the credential and then decides whether they should revoke the secret, issue a new secret, or reach out to you directly, which will depend on the associated risks to you or the service provider.
https://help.github.com/en/github/administering-a-repository...
in the last startup I worked, all jwt tokens were created from a 10 letter long shared "secret" stored in json config files all over the place :p
even dev environments had same key lol
If they're dev-keys, I think this is pretty common.
Since it's of interest to HN, I am working on educating our very small team on how keys should be protected and used. I am the youngest developer by about 15 years. It's a very rural company and it often feels like all learning and passion for development stalled around 2005. It's a company that gave me a chance to grow into a development role with no previous experience so I feel indebted to try my best to keep the lights on.
I don't judge too harshly- anyone who has black and white principles on these matters has never worked in any other industry most likely... all you can do is your best to steer the ship and convey the downsides.
I think it's important too because it helps us understand how much friction people will tolerate.
In many cases, even a small amount of friction will cause people to stop functioning completely; I recently tried setting up vault and it was a nightmare, I understand why people avoid picking it up.
That doesn't mean we should not try; we have to become the advocates, arbiters and helpers for those systems.
Good luck, you're not alone.
It should be no surprise that people do insecure stuff under deadline pressure.
This is definitely true, but not actually surprising. It's much easier to notice that, say, a violin performance or plumbing repair is done very badly, than it is to actually do it correctly yourself.
Which also leads to a great deal of exasperation when people (either apparently or actually) don't even notice that what they're doing is insecure. There's a big difference between "yeah, it's broken, but it'd be a huge pain to fix and we'd probably get it wrong anyway, so we'd rather take our chances" versus "there is no problem".
We use Hashicorp's Vault product to manage SSH credentials, TLS certificates, as well as application secrets across thousands of users, tens of thousands of virtual machines, and hundreds of applications.
We pay for the enterprise version, but the free version is more than capable for most needs. Avoid a password manager if you can, it leads to poor security practices and availability issues depending on how and where the data is stored (IMHO, YMMV). Use single sign on where ever possible.
Disclaimer: No affiliation other than a satisfied customer and frequent end user of the product.
1) Secrets checked into code means when the code gets stolen, this is an unimaginably major breach. Code tends to get stolen eventually and most tech shops will never know / only know years later because they don't have access to the channels who will sell your code.
2) You can track secrets you have in storage, who has access and who is using them
3) When a secret does get stolen, they can be rotated with ease. You do not want to be spending on the order of man months manually auditing a huge code base and manually indexing and rotating every secret. With good setup you can do this without taking any systems down.
4) Most support HSM / KMS systems which encrypt the data using a key in a hardware security module, which, in theory the key cannot physically leave – the HSM will re-encrypt all your other keys. The HSM, if used properly can be a continuous bottleneck to decrypting any information, meaning, if your encrypted objects are stolen, the attacker needs active access to your (e.g. AWS) KMS system to decrypt them.
Password managers are good for individual users, but provide bad security characteristics for development and deployment.
AWS Secrets manager, AWS's competitor to Vault used a similar system I worked on that was developed within Twitch for this purpose.
https://github.com/sethvargo/vault-on-gke/blob/master/terraf...
The KMS ring itself is only accessible by Vault. People with high enough privileges for our Vault GCP project could technically grant themselves access to it, but on day-to-day business, nobody can view the project.
At some other place, where we were using AWS, I wrote a script that would store encrypt unseal keys (need multiple due to shamir) via pgp using their keybase public key. IIRC you can store encrypted keys in Vault that can be accessed only for unsealing purposes (please correct me on this). When needing an unseal, the script on the user side would then decrypt the key and submit it to Vault for unsealing. It worked well enough, and it felt like being in the movie/game GoldenEye, but nowhere as slick as auto unseal.
Regardless of the setup, yes, at the end of the day any solution is really just pushing the problem further down the stack.
It does admit the possibility that an operator's share could be copied. To work around that you can get a proper HSM that needs a quorum of smart cards presented to unlock. (The offline, low QPS, root of trust-oriented ones are not exactly cheap, but much cheaper than the network-attached ones targeting high QPS transactions). Vault Enterprise has PKCS#11 integration.
With the Thales nShield stuff, you can replicate key material from one to another for redundancy while allegedly still preserving the "can't ever get keys out in plaintext" property. Not sure about others.
* You can split the key between different persons, and you can even implement "n of k" schemes, like you specify (at key creation time) that you need any 4 out of 9 shards to unseal the vault. You can then keep those shards on separate operator's laptops, in separate backup systems etc.
* You can use a hardware security module to unseal the vault (support for that is not included in the free version, IIRC).
But even if the vault wasn't stored encrypted, it'd still be a huge improvement over "keys on NFS", because only machine administrators get access to the whole DB, and you can limit and audit the access of everybody else in a sane manner.
That means that each application got a ServiceAccount (SA) and each user got a username/password. Based on your identity, you get access to specific secrets from Vault.
Smart cards are held by two separate people, each card set is different. Both have to be present to restart the HSM.
Our HSM can require up to five different cards be required for certain operations.
For us, we only require two for normal operations.
For HSM management (key generation, card authentication, etc..) we require a third card set member to be present. This protects against accidental key erasure, or fraud.
Secrets may still need to go to config files for third party stuff, but you can write your in-house applications to fetch secrets at runtime and hold them only in memory. That's less opportunity for compromise vs. both memory and disk. Also have heard that Linux's process address space isolation is less prone to vulnerabilities than user account separation or filesystem permissions. Not really sure how true that is.
At scale, you can very granularly define policies for each secret. When a secret is accessed, it is done so through a user or application identity. Each access is also logged.
you should only let the instance access the secret it requires.
As OP wrote, you did not solve it, just moved it to a different level.
A typical deployment might involve placing the manager on a secure host that has access to generate and rotate keys, for example.
The manager can then configured to re-generate keys and vend them on-demand to instances that need require them. You can configure these keys with very limited access, and also make them expiring.
The manager then becomes effectively a keystore that can never export master keys, but only vends out some limited-scope keys to other instances.
Other instances would have to authenticate using some pre-configured host keys or even be authenticated directly though the cloud provider.
If your instances are compromised, the worst someone can do is to get access to a limited-scope key that will expire.
Hopefully you have other measures in place that would prevent and detect someone from just sitting on an instance and sucking up all your data and exporting them.
Not only it is opt-out, but the mentioned "off" toggle doesn't even exist.
The privacy policy is here: https://www.hashicorp.com/privacy
The only thing protecting the unseal key is access to your KMS. So one rogue SRE can unseal the vault rather than requiring collusion of 1+ SRE members.
Again, this comes down to your risk tolerances and what you are protecting. I think for most workloads, the value KMS autounseal brings is worth the risk, but if you want to have tightest control, then the Shamir Split (M of N) is the best option.
I find it hard to build initial trust in the system, without involving the trust of an administrator + subsequent automation.
Automatic password rotation for AD credentials[1] is especially useful to us.
Meanwhile we use the project level variables Gitlab API[2]
Secrets are kept in Gitlab and requested by applications through a token that can easily be revoked
[1] https://www.vaultproject.io/docs/secrets/ad#password-rotatio...
[2] https://docs.gitlab.com/ee/api/project_level_variables.html
1. Buy a bunch of Yubikeys, minimum of 2.
2. Create GPG keys and store them on YubiKeys. Follow this guide: https://github.com/drduh/YubiKey-Guide (if you want to, keep the secret keys, but in case of multiple YubiKeys I would not keep them anywhere). Remember to set the keys to always require touch.
3. Use GPG to encrypt your backups to multiple recipients (all of the YubiKeys).
4. Take care of the physical keys with proper storage and procedures. Do not store the keys together, have at least one in a really secure location, check if you have all the keys regularly, etc.
5. Test restores at least once per quarter, with a randomly selected key.
The advantages of this solution is that it is simple, works pretty well, and gets you a lot of mileage with relatively little inconvenience. You don't have the risk of keys being copied, and guarding physical keys is easier than digital ones.
You still have the problem of guarding the passphrases to the Yubikeys (if you use them), but that is much less of a problem than guarding the encryption keys. A passphrase without the physical key is useless.
This setup works for organization from size 1 up to fairly large ones.
Note that some recently fashionable security consultants crap on GPG from great height, but do not provide an alternative. It's a tool that while having multiple flaws, does many jobs better than anything else out there.
> 4. Take care of the physical keys with proper storage and procedures. Do not store the keys together, have at least one in a really secure location, check if you have all the keys regularly, etc.
Would be great to see what folks think this concretely looks like for joe random startup in Capital City, Somewhere.
e.g. Does "really secure" mean "find a bank that still offers safety deposit boxes"? Does it mean paying for something like Iron Mountain (http://ironmountain.com/) or one of its competitors?
Realistically? Yes. This is what several of the companies I've done contract work for have done. You can still find at least one bank or self-storage place (look for the ones that don't have a nationally-advertised brand and don't look like they're made entirely out of corrugated metal) that do regular safety deposit boxes in pretty much any city. They may only be offered at a couple of locations and I've noticed credit unions bailing the hell out of this market as fast as they can decommission the vaults but boxes still exist.
Let's assume all variables work in the other way, though. If you can't find a safety deposit box and don't have somewhere that's not your office you can drill into a floor or wall and you're storing a small device like Yubikeys or USB sticks, buy the heftiest portable gun safe you can find, one with a steel cable that loops back into the device, and stick it under your bathroom sink with the cable wrapped firmly around the water supply or drain pipe.
What's the idea behind this? So that when someone tries to break in, they flood your bathroom? Or do you just mean secure it reasonably?
But if for instance, your whole security team got in a car accident and all the keys burned up, you'd have a way to recover the creds and save the business.
There's also the ability to just leave it at a lawyer/notary (they already handle deposits, they might even have a secure box at a bank, so you can piggy-back on them for this).
Directors/Founders of Random Co. should just make a few copies a few pieces of papers that contain the passphrase and store them at their own home and ask a few relatives to do the same for them. Depending on their recoverability/safety/accountability trade-off they can increase the number of copies, they can increase the separation between the parts (eg. keys and passprhases), and so on.
The big-big-big advantage of the yubikey approach is that it's a HSM, and you can't accidentally copy the key and leave it somewhere.
My main point was that the use of hardware keys makes many things much easier, and you do not have to worry about your keys being copied and used without your knowledge. That's a big thing. Also, the often-ridiculed GnuPG is amazingly useful with Yubikeys (using the setup I linked to), because you can use the same keys for SSH, thus ensuring access to all resources as needed.
https://docs.sequoia-pgp.org/sq/#subcommand-encrypt
https://sequoia-pgp.org/status/ is coming along nicely ( https://sequoia-pgp.org/blog/2020/04/26/202004-towards-sequo... )
GnuPG is a command-line tool that is omnipresent, keys can be stored on Yubikeys, can be used with ssh-agent, and I can use it to encrypt files in an automated fashion, using both symmetric and asymmetric crypto.
Vault stores all secrets needed by running services (ACL tokens, access keys, credentials for databases, PKI for certificates, what have you).
For the rest (Vault unseal keys/cert keys/operator token, other operator secrets), secure those with the GPG keys mentioned above and store them in some way that suits you (GNU Pass/git-secret/there are several alternatives).
Where you draw the line between what's stored in Vault and not will depend on your org and its needs.
Have an offline backup printed along with the disaster recovery checklist and documentation and put them in a safe in your company - the checklist should be dumb enough that your drunk self can use it at four in the morning, because you were the nearest employee when everything went down.
Ensure that you have stupid manual processes in place on rotation of the safe's PIN and encryption keys in general, including a sanity check if the newly generated keys actually work (e.g. if they are used for your backup storage, actually back something up and restore it). Ensure that the safe's PIN is available to at least another person and used regularly (e.g. because you store your backup tapes there).
If you feel that you need to change from this very simple system to a more complex one, ask yourself why. What does your change actually add in terms of security and what risks does it add.
In the end, you want your system available to customers and the security you add is to not only secure the data, but actually to know who can access it (the auditing part).
As a side note: for example we had some backups that are probably useless, because they are way too small. Catching this would mean more manual regular checks, or some automated rules, at which point it becomes quickly more complex again.
Storytime: I did not check the restoration of my backups some time ago and had a faulty harddrive, so I needed to restore the backup. Backup was also as dumb as possible - essentially a tar and encrypting it with openssl. So I reinstalled the server, tried to decrypt it - got the error, that the key was wrong. Took me a good weekend to find out, that openssl changed the default hash algorithm between openssl 1.0 and 1.1. This would not have been catched with the proposed system, but now I really pin all default options in my scripts as well.
have you ever tried typing a private key from a piece of paper? once i was in a similar situation and gave up and just drove to the colo.
OpenSSH private keys are armoured by default, gpg-keys can be exported and imported in an armored format - and everything else can be just printed as hex representation with whatever tool (e.g. `od -Ax <file>` or any other).
fd3223ec 20f55ae7 6fddc979 d41e2276 25255516 b08f5cd4 3d66d676 a054d2bb
- All "hot" keys were stored in an offline credential manager in specific vaults depending on who needed access to them. Only staff with actual clearance could request temporary access to a vault (fully background checked, 1 year employment, etc).
- Copies of each vaulth and our master CA cert were written to 4 encrypted USB sticks. Two stored on-site in the fire-safe and two off-site at our safety deposit box that only c-level staff could access. (We had the same process with our tokens and master logins for AWS).
- Any work using those keys was on a pair-up basis, so at least two people, one doing the work and the other observing.
- We had a detailed policy around this that covered each step in the process and who needs to approve them; everyone who could feasibly need to access the keys was briefed annually as part of our security awareness training.
We handled a LOT of sensitive financial data, so this was the most appropriate way that we could find that maintained both sensible availability and key control.
So in order to get to the keys you needed:
- Access to the fire safe (Senior Ops, Senior Security and C-Level only).
- The LUKS passphrase for the USB sticks (Senior Security and some C-Level only).
- The passphrase for the specific vault (Senior Security and some C-Level only).
I don't know how the passphrases were managed by our sec team, but I know that the C-Level staff had physical envelopes in their home safes.
Seems like we hear more frequently about the actual secrets being stored encrypted (potentially with hardware protection) in a central place, and only the keys to unlock them being distributed like this.
Accessible over USB or HTTP, it supports every major crypto algorithm [1], and keys can be backed up onto another HSM via a wrap key (if they are marked as exportable -- you can also control what can and cannot be exported -- in fact, every operation may be allowed or disallowed per key).
Every operation is logged for audit, of course, and the device may be setup to require logs to be read before they are overwritten. In combination with configuring a special authentication key to access the logs, you can ensure that every operation on the HSM is logged to a remote store before additional operations may be completed.
It does depend on your existing physical security, so that has to be taken into account when designing architectures including it. The micro form factor at least makes it trivial to put into an internal USB port.
And of course, if you require a more enterprise grade tool, you may want to use an HSM in combination with a tool like Hashicorp Vault to manage your keys throughout your orgnaization.
[1] https://developers.yubico.com/YubiHSM2/Product_Overview/
FYI: Hashicorp vault just uses Shamir's Secret Sharing scheme under the hood: https://github.com/hashicorp/vault/blob/45b9f7c1a53506dc9722...
* The pricing for just storing keys is incredibly cheap.
* At least with Google KMS you can't delete the keys without a 24 hour waiting period (and you can alert on the deletion attempt), so that's a huge safeguard.
* You get key access auditing out of the box.
There’s also a feature you can enable that automatically rotates your key once a year. KMS is great!
Why do you like their auto-rotation? The keys that are rotated out are not never disabled, so I don't really understand the benefit. In what scenario would their auto-rotation improve security?
The costs of that obviously scale in a completely different way from the per-key storage costs (which are actually zero, I think).
At $0.06/key/month, that's practically free for most reasonable use cases. For example, if there's 10k secrets that's $7,200/year.
If you encrypt/decrypt your secrets 1 million times per day (~11.6 times/s), the access charges would be $1,095/year (1 million operations/day * 365 days/year * $0.03 / 10,000 operations).
[0]: https://cloud.google.com/kms
In addition to it, we use envwarden[1], which is a simple open-source wrapper around the Bitwarden CLI to manage our server secrets. It's super simple, but does the job for us well. We can then manage both passwords and keys in one place.
Disclaimer: I created envwarden. I'm not affiliated with Bitwarden in any way however. Just a happy customer.
[0] https://bitwarden.com/
[1] https://github.com/envwarden/envwarden
We migrated from bitwarden to offline client machine for the reason that we didn't wanted a copy of our encryption key be available anywhere else online.
I first wrote about it back in 2017 (1) and we released an open framework for multiple languages/frameworks (2).
1: https://neosmart.net/blog/2017/securestore-a-net-secrets-man...
2: https://neosmart.net/blog/2020/securestore-open-secrets-form...
It's designed to integrate really well with your existing CLI tools like vim, xargs, and diff. It offers user-based permissions, and secrets are encrypted into a single file that's safe to commit into your git repo. We can stream secrets out of it directly to our remote servers during deploys.
Unlike Vault you don't need to manage infra to run it -- it's just a file. Unlike cloud secret managers, there's no lock-in.
1. Does every client have a copy of shh to interact with the secrets? Or are the secrets in the file served from a single centralized node?
2. What is your process of exchanging user keys with shh?
3. If someone leaves the company, what is the process you go through to change the secrets and rotate keys?
2. Exchanging public user keys is done via email when an employee starts, then added to the .shh file and committed in the repo.
3. You'd need to write a script for this, which probably involves `shh rm-user {email}`. There is no silver bullet here for changing secrets; since developers have access to secrets, any secret they had would need to be regenerated. `shh` makes no assumptions about how you deploy, what secrets you keep, or their formats.
Application repos have the encrypted secrets (meta) stored in their repos using Ansible-vault and the .vault_keys are stored in Bitwarden.
Usage includes SSH authentication, file encryption (backups and exchanges), git commit signatures and password/secret storage using `pass`.
Copies of the offline master keys keys are stored on flash in safes onsite and offsite in bank vaults, and sub-keys are valid for one year.
We use Hashicorp's Vault for secrets that require automated access.
https://dev.ionic.com
Utilized globally by individual developers, large enterprises such as JP Morgan & Chase[1], and integrated into the KMS services such as Google Cloud[2].
1. https://venturebeat.com/2019/02/27/ionic-security-raises-40-...
2. https://cloud.google.com/blog/products/identity-security/clo...
This was early in the “cloud” epoch and at any rate they preferred in-house iron for entirely understandable reasons. Also, they needed to do some pretty interesting things, so they had a NAS populated with enterprise-grade SSDs and a native AES-based encryption scheme. This kept the key on what was basically a glorified USB key.
(For those of you who wish to know these details, the network between the blades and between the blades and the NAS was a nicely spec’d fibre channel network, and there was some iSCSI involved. The blades featured Itanium processors, which kind of gives the manufacturer away, and the firm had invested quite heavily in producing very high-performance code for those ill-fated microprocessors, but I digress.)
So... it happened that somebody lost the USB key. Well, not quite. Somebody took it home during a weekend (whilst the system was shut down) and their kid used it for school work.
This proved to be a “significant problem”. There was a backup, and it was encrypted and stored on an adjacent SAN. It wasn’t exactly stale, but it wasn’t entirely pristine either.
There was much woe and gnashing of teeth.
Nobody was fired because the dolt who maintained custody of the only USB key was the founder/CEO, so he couldn’t exactly blame himself.
But, yeah. That happened, sadly.
One threat model might be: Burglars sneaking in during the night and stealing the hard drivers. Then you would store the keys on a different location then the disks.
Then you make it a routine to reboot parts of the fleet, like scheduled simulation/training so that everyone knows what to do when you actually need them.
In general I would suggest using a key vault. AWS, GCP, and Azure all have cloud versions that are backed by virtual HSM's built on top of actual physical HSM's. For the vast majority of usages they're good enough. Use admin account management to enforce 2FA/FIDO for all AWS/Azure/GCP logins. (You should be enforcing 2FA with phone/FIDO auth anyway.)
If you need truly paranoid backups, you can back the key up onto a portable hard drive that you lock in a safe in the closet, with a few key people who know the code.
I recommend against using a (cloud-synced) password manager. Cloud key vaults do the same thing but offer specific features relevant to server stuff. And if you want more paranoia, a physical safe is probably safer than extending your attack surface to a cloud-synced password manager.
Also: make sure that you set up a ~quarterly ritual of opening and verifying the backup. For crucial backup fallback systems, you want to make sure you actually use the system so that you know if it fails.
Poorly seems to be the answer industry wide. Both encryption and disaster recovery are hard tasks. Combine them and you have recipe for a total mess.
Think of a lighter vault, with ACLs for people and/or machines to access keys and versioning to rotate keys.
In our case, Knox depends upon AWS KMS to "lock/unlock" its storage.
For context: We run a centralised salt-master, salt master unencrypts content using gpg filters as part of variable generation (salt "pillars"). So it's encrypted at rest and encrypted in our git repositories.
What we do/did, is:
* grab a pair of differently branded USB sticks.
* LUKS encrypt the USB sticks; we used a keyfile which is encrypted on our machines with our GPG key.
* encrypt salt's GPG private key with all of our keys.
* encrypt some of the "irreplacable" private keys (IE: CA roots) with all of our GPG keys.
* store it all on the pair of USB drives
* put the USB drives in a real-life vault, give the keys to the office team.
We haven't needed to recover, but it's clearly documented how to recover if anything went wrong.
It'd be better to take a page out of the cryptocurrency handbook and print the keys out as QR codes (maybe with a simple password so it can only be restored if you know the pw).
1) If you don't need it - don't store it. If you need it - but it needs to be encrypted - probably! don't store it.
2) Think one way hashes with salts, think deletion policies, rotations, small disk drives.
3) You will get owned!!
LDAP is ubiquitous enough as an auth method (how do you auth to Vault? You auth to LDAP with it...) that any service you run or use is likely to speak it.
Why this isn't done more often is a mystery to me and probably the number 1 source of credentials being baked into things accidentally: oh we need a service account into the <enterprise system> which uses Active Directory.
Might have answered my own question there though.
It's been quite a few years since I interacted with them, but for some keys there is a server somewhere with an HSM installed, and two people have credentials for it. If you need something signed you send it to them, with a justification for why it needs to be signed with the real keys, and they will send you a path to get the signed file, and remind you to delete the signed file when you no longer need it.
This is overkill for some things, and probably would be considered sloppy for others.
For other parts HashiCorp Vault/AWS kms
For standard password style secrets used by Ops, we use Team Password Manager. Which we chose about 5 years ago because it was self-hosted, the database was encrypted, and it had fantastic audit capabilities.
There are different CAs for different purposes. There's an intermediate for device management, and another for user or service auth purposes.
For anyone who needs a super simple place to store their encryption keys that works with Heroku and has versioning, I think Doppler could help. It doesn’t have all the fancy (and really cool) features of KMS as it’s designed to be a kv store for secrets, but it could be helpful. We have a free tier for anyone who wants to try it out. https://doppler.com
Signing and authentication keys are expendable but encryption keys are worth keeping even after they've been rotated since decryption of existing data may be necessary.
The key can be printed on paper and stored in a physical safe. Paper isn't a high density storage medium but it is remarkably durable and perfect for small amounts of data such as encryption keys. It also counts as an offline backup.
Keys can also be printed as QR codes. They support error correction and enable automatic data restoration. Even 4096 bit RSA keys fit in a binary mode QR code and the smaller ECC keys allow use of high error correction modes, making the data even more durable.
I wrote a binary decoding feature for ZBar in order to support this exact use case:
It's available on version 0.23.1 and above.http://docs.oasis-open.org/kmip/spec/v1.4/kmip-spec-v1.4.htm...
Catch: Client machine is encypted with VeraCrypt. Veracript hidden drives, one password is kept by product owner and another password is kept by head of security.
Offline client machine is key for us.
We rotate encryption keys after quarterly security audit.
Not ideal, but it works... At least until one of us resign (but turnover is quite low here, so crossed fingers).
Quite old school but it works quite well, alas we're small though (120).
Check it out and if you have any questions, feel free to ask here or open an issue in github. We also have a Rust version in the works for those interested in something native.
https://cloud.google.com/secret-manager
The main observation I'd make that if you put your keys somewhere only accessible in production, you've made it impossible to test anywhere except production. If you do that, you need to create a process where people can ship some small bit of code to test if the production key setup genuinely works (hint: it won't).
AWS ACM for SSL certs.
AWS SSM for ssh, eliminates the needs for ssh keys.
Not everyone loves AWS (me included), but this stack works nicely in removing the need to ever touch raw encryption key files locally.
I don't know your company size but we use stackoverflow's blackbox. It's very nice because you can check in the repo into version control. You can add and remove users on the fly as well.
All of the key management is "built-in" and managed, so there really isn't much overhead. All software-based, with FIPS certified key management. It's very easy to encrypt data this way. It is expensive though.
Disclaimer: while I used to work for Vormetric / Thales, I no longer do.
Locked in a vault.
Keys are an actual secret key (within the HSM), unknown to any human.
Two people with two different access cards have to be present to enable key operations.
So, you don't have to worry about changing keys as employees come and go, because no one knows the actual keys.
There is a whole structure of spare cards stored in offsite secure storage in case a card is damaged, lost, or stolen.
Card set one is stored separately from card set two.
None of our secret material is particularly hard to revoke or replace; we don't run an internal CA or anything like that.
self promotion * You did ask how people do it :), this is my way, Ive written my own service which has been in production for more than 3 years, http://pkhub.io (if you would like to try it send me an email to admin@pkhub.io). This was before aws secrets manager, the tooling is usefull cause I wrote: running your app with its needed secrests dev/stage/prod, accessing dbs, downloading and installing ssh keys to ssh agent, utilities.. end
of course you could write all these yourself with aws secrets manager.
there is hashicorp's vault but tbh it always seemed like way to complicated to setup.
my advice in general would be: to get something secure but simple enough that your engineers can do their work and access the resources they need, without the oh only bob has the keys on his laptop situation.
[0] https://kubernetes.io/docs/concepts/configuration/secret/
How the 4 shareholders store their shares is up to them. Mine is in a secure note in 1Password.
[1] https://en.wikipedia.org/wiki/Shamir%27s_Secret_Sharing
I will be very interested in hearing how others do it.
You can rotate keys and facilitate key pinning scenarios.
Cheers!
Sadly now forced to figure out what to move to. We're considering 1Password with it's CLI as a short term option, but will be wanting to move to Hashicorp Vault or similar on the mid-term
The extension makes use of the per-user keys of Keybase. This allows you to manage access permissions using the Keybase GUI client. Once a user is added to the appropriate Keybase team, they will immediately have access to that teams secrets that are stored in the encrypted Git repos.
To use it you just need to download the 2 shell scripts to a directory in your path (e.g. /usr/local/bin/) or you can clone the git repo.
My team uses lastpass
Worth noting accessible using api via REST https://docs.microsoft.com/en-us/rest/api/keyvault/
and powershell https://docs.microsoft.com/en-us/azure/key-vault/secrets/qui...
https://imgur.com/a/FE8sslv
and that team now keeps growing and the feature never improves :)
Most people have their employer or work email on their HN profile.
Really people?
But also, at that point you're fully committed to security through obscurity, so folks should get at least the obscurity part right :)
Here is an air-gapped solution:
https://www.arcanus55.com/?trusted55=A55PV2
We don't have SSH keys because it's not the 90's and we don't have servers.
This seems unnecessarily snarky.
There are lots of businesses in 2020 that still maintain their own servers, use ssh keys, and are staffed by admins and developers who very much know what they are doing (and are not at all "behind the times", as this comment seems to imply such businesses are).
If that's not what you meant, well, OK, but I find it difficult to interpret it any other way.
probably the hard way.