Secrets Management

Secrets Management is the systematic solution to one of the most common security vulnerabilities in modern software development: hard-coded credentials, API keys in .env files that are committed to Git, or passwords in environment variables without rotation. Every day, thousands of API keys are accidentally published on GitHub.

The Problem: Secrets Everywhere

Common Anti-Patterns

1. Hardcoded in the source code (most common mistake):

database_password = "SuperSecret123!"
AWS_ACCESS_KEY_ID = "AKIAIOSFODNN7EXAMPLE"

2. Committed in .env files:

.env contains: DB_PASSWORD=secret

.gitignore is missing or .env is committed anyway. The Git history contains the credential—which can be reconstructed even after deletion!

3. Unencrypted config files:

config.yml: password: admin123

4. CI/CD variables in plain text:

# GitLab CI:
variables:
  PROD_KEY: "abc123"  # not masked!

5. Docker images with secrets:

ENV DB_PASSWORD=secret  # ← in Docker layers, visible even after `rm`!

6. Log Files with Sensitive Data:

logger.debug(f"Connecting to DB with {password}")

Consequences

• Credential scanning (TruffleHog, GitLeaks) finds keys in seconds
• GitLab/GitHub: continuous secret scanning (free since 2023)
• One AWS key = potential 6-figure cloud damage in minutes
• Rotation required, but dependencies unclear? → uncontrolled secret chaos

HashiCorp Vault - Enterprise Secrets Management

Vault is the most widely used open-source secrets management system.

Concepts

Term	Meaning
Secret Engines	Plugins for different secret types
Auth Methods	How clients authenticate
Policies	What a client is allowed to read/write
Lease	Secrets have a TTL (Time-to-Live)
Audit Log	Every Vault operation is logged

# Docker:
docker run -d --name vault \
  -p 8200:8200 \
  -e 'VAULT_DEV_ROOT_TOKEN_ID=dev-token' \
  hashicorp/vault:latest

export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='dev-token'

# Enable KV (Key-Value) Engine:
vault secrets enable -path=secret kv-v2

# Store secret:
vault kv put secret/database \
  username=dbuser \
  password="Sup3rS3cure!" \
  host="db.intern.firma.de"

# Read secret:
vault kv get secret/database

# JSON output for scripting:
vault kv get -format=json secret/database | jq '.data.data.password'

# Dynamic Secrets (Auto-rotating Database Credentials):
# Vault generates temporary DB credentials on-demand!

vault secrets enable database

vault write database/config/my-postgresql-database \
  plugin_name=postgresql-database-plugin \
  allowed_roles="readonly" \
  connection_url="postgresql://{{username}}:{{password}}@localhost:5432/mydb" \
  username="vaultadmin" \
  password="admin-password"

vault write database/roles/readonly \
  db_name=my-postgresql-database \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN VALIDITY '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

# App call: new credentials every time!
vault read database/creds/readonly
# Key      Value
# username  v-readonly-2024AbCd
# password  A1b2C3d4-random-40chars
# lease_duration  1h (account is automatically deleted after that!)

Cloud-Native Secrets Management

# AWS Secrets Manager - Create secret:
aws secretsmanager create-secret \
  --name "prod/app/database" \
  --description "Production database credentials" \
  --secret-string '{"username":"dbuser","password":"secret123"}'

# AWS Secrets Manager - Retrieve in application (Python/Boto3):
import boto3
import json

client = boto3.client('secretsmanager', region_name='eu-central-1')
response = client.get_secret_value(SecretId='prod/app/database')
secret = json.loads(response['SecretString'])
db_password = secret['password']

# AWS - Enable automatic rotation:
aws secretsmanager rotate-secret \
  --secret-id "prod/app/database" \
  --rotation-lambda-arn "arn:aws:lambda:eu-central-1:123456:function:rotation-fn" \
  --rotation-rules AutomaticallyAfterDays=30

# Azure Key Vault - Set secret:
az keyvault secret set \
  --vault-name "mycompany-keyvault" \
  --name "DatabasePassword" \
  --value "SuperSecretPassword!"

# Azure Key Vault - In-App (Python):
from azure.keyvault.secrets import SecretClient
from azure.identity import DefaultAzureCredential

credential = DefaultAzureCredential()  # Managed Identity!
client = SecretClient(
  vault_url="https://mycompany-keyvault.vault.azure.net/",
  credential=credential
)
password = client.get_secret("DatabasePassword").value

# Kubernetes External Secrets Operator (better than plain K8s Secrets):
# Note: Plain kubectl secrets are Base64-encoded, not encrypted!
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: database-secret
  data:
  - secretKey: password
    remoteRef:
      key: secret/database
      property: password

Secrets Detection and Prevention

# Pre-Commit Hooks - Stop secrets before pushing:

# Install git-secrets:
brew install git-secrets
git secrets --install      # Configure in the repo
git secrets --register-aws # Add AWS key pattern

# gitleaks (modern tool):
brew install gitleaks
gitleaks protect --staged  # Check staged changes
gitleaks detect            # Scan entire repository

# TruffleHog:
trufflehog git file://./my-repo --json

# In .pre-commit-config.yaml:
repos:
- repo: https://github.com/gitleaks/gitleaks
  rev: v8.18.0
  hooks:
  - id: gitleaks

# CI/CD integration (GitLab):
secret_scanning:
  stage: security
  image: trufflesecurity/trufflehog:latest
  script:
    - trufflehog git file://$CI_PROJECT_DIR --only-verified
  allow_failure: false    # Pipeline fails if secrets are found!

# GitLab Secret Detection (built-in, since GitLab 13.1):
include:
  - template: Security/Secret-Detection.gitlab-ci.yml

GitHub Advanced Security

• Automatic secret scanning on all public repos
• Push protection (prevents commits containing known secrets)
• Partner program: GitHub notifies AWS/GitHub/etc. upon detection
• Private repos: Part of GitHub Advanced Security (paid)

What is detected:

• AWS access keys (AKIA...)
• Google API keys (AIza...)
• GitHub personal access tokens (ghp_...)
• Stripe API keys (sk_live_...)
• Twilio API keys
• 100+ additional provider patterns

Secret Rotation - Continuous Renewal

Rotation Strategy

Passwords:

• Critical system passwords: monthly or quarterly
• Service account passwords: Vault dynamic secrets (TTL 1–24h)
• Automated rotation via script or Vault

API keys:

• Set expiration dates (max. 1 year)
• Regular audit: Are all keys still in use?
• Revoke unused keys immediately

Certificates:

• Let's Encrypt: automatically every 60 days (certbot / cert-manager)
• Internal PKI: 1–2 years, automated renewal
• Monitor TLS certificates (expiration alerts!)

Rotation Best Practices

1. Zero-downtime rotation: set a new secret, then invalidate the old one
2. Document dependencies: who uses this key?
3. Staging test before production rotation
4. Rollback plan: in case new credentials do not work
5. Audit log of the rotation (who, when, why?)

Rotation Checklist Following a Data Breach

• Immediately: invalidate all affected credentials
• Generate new credentials with higher entropy
• Update all dependencies
• Check logs: was the key compromised? (Access logs!)
• Fix the root cause (how did the credentials leak?)

Company Checklist

Immediate Actions

# Check history for credentials:
git log --all --full-history -- "*.env" | grep -i password

# Scan all repos:
# Run TruffleHog or gitleaks

# AWS Console: IAM → Access Analyzer (public resources?)
# HIBP API: Were company credentials found on the dark web?

Medium Term

• Implement Vault or Cloud Secrets Manager
• Pre-commit hooks on all developer machines
• CI/CD: Secret scanning as a mandatory stage
• Rotate all existing credentials
• Inventory: Which secrets exist where?

Long-term

• Dynamic secrets for databases (Vault)
• Least privilege: each service only has necessary secrets
• Audit log: who accessed which secrets?
• Break-glass: emergency procedure for Vault failure
• Annual credentials inventory