Secrets Management
Vault, AWS Secrets Manager, SOPS, git-crypt - keep secrets out of your code and your logs.
“There are two types of companies: those that have committed secrets to git, and those that will.”
What you'll learn
Assume the secret has already been committed. Then stop making it worse
There are two kinds of repos: those that have leaked a key, and those that have not run git log -p yet. Rotate anything that ever lived in a commit. git rm does not unpublish history.
Install a pre-commit scanner (git-secrets, gitleaks, or the equivalent in your CI). Block the merge. Feeling lucky is not a control.
Pick a store that matches how you operate
HashiCorp Vault if you want leases, dynamic credentials, and a thing you now have to run. AWS Secrets Manager / GCP Secret Manager / Azure Key Vault if you want the cloud to be the store and IAM to be the lock. SOPS if you want encrypted files in git and a KMS key that is not in the file.
git-crypt is the older cousin of SOPS. It works. It also trains people to forget what is encrypted. Prefer SOPS plus a documented key ceremony if you are starting now.
How secrets get to the process
The app should read a secret at runtime from the store or from a file the orchestrator mounted. Baking secrets into an image is how the image becomes the incident. Env vars are fine if the platform injects them and they do not end up in debug dumps.
Kubernetes Secrets are namespaced objects. They are not Vault. Encrypt etcd, restrict RBAC, and prefer an external secrets operator when the org is ready. Until then, do not also put the same values in a Slack thread.
Rotation is a design, not a New Year's resolution
If rotating a database password requires a maintenance window and a priest, you do not have rotation. You have a wish. Prefer short-lived credentials (IAM roles, Vault database secrets) so expiry is normal.
When you must use a static secret, write the two-step: introduce the new value, make clients accept both, drop the old. Do this before the leak, not during.
Logs, CI, and the other places secrets go to die
CI variables should be masked and restricted to protected branches when the platform allows it. Do not echo env in a debug step 'just this once'. Ticket systems, screenshots, and kubectl logs are where the second copy lives.
If a secret appears in a log, rotate it. Then fix the log line. Apologies do not revoke access keys.