Chapter 8: Secrets Management

Where Secrets Live

A personal AI assistant is a credential magnet. To do anything useful it needs an OpenAI or Anthropic API key, a Slack or Telegram bot token, an email password or OAuth refresh token, a GitHub personal access token, maybe a Stripe key and a database connection string. Each of these is a secret — a credential that grants access or carries real risk if it leaks — and each one lives inside a process that reads untrusted inbound messages and runs third-party skills, exactly the kind of process most likely to be tricked into leaking or misusing what it holds. Think of the difference between a house key that opens every door forever and a hotel keycard that stops working at checkout: both open the same door today, but only one is safe to lose.

Environment Variables: Benefits, Risks, and Leakage Paths

The most common place a developer puts a secret is an environment variable, usually loaded from a .env file (for example OPENAI_API_KEY=sk-proj-abc123... or DATABASE_URL=postgres://agent:hunter2@db:5432/assistant). Env vars are popular because they are trivial: every language reads them in one line and no library is required. That convenience is the whole appeal — and also the whole problem. Environment variables are excellent at injection but they are not a storage or lifecycle system, and using them as one is where security falls apart.

The specific deficiencies each map to a way a personal assistant gets compromised:

• No encryption at rest. Values in a .env file or a process environment block are plain text, readable by anyone with access to the host or process.
• All-or-nothing access. Any code in the process — including a compromised third-party skill — can read the entire environment. There is no per-secret scoping, so one malicious dependency reads every key you have.
• Many leakage channels. On Linux, env vars are readable at /proc/PID/environ; they are inherited by every child process the agent spawns; they can be dumped by a debugger; and they surface through docker inspect, kubectl exec, inherited shells, and CI/build logs. Crash reporters like Sentry or Datadog can silently index a password attached to an error report.
• No audit trail. Plain-text config records nothing — there is no way to know who read a secret, when, or whether an attacker got there first.
• Rotation friction. Env vars are effectively immutable for a process's lifetime, so rotating means restarting every workload that uses the secret, and that friction makes teams delay rotation.

None of this means env vars are forbidden. Harmless configuration — a log level, a feature flag, a region name — is fine in a plain env var; anything that grants access or carries risk if leaked belongs somewhere with encryption, scoping, audit, and rotation.

Figure 8.1: Leakage paths that expose environment-variable secrets from a single host

Secret Managers and Vaults

A secret manager (also called a vault) is a dedicated system whose job is to be the single source of truth for credentials — HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, Azure Key Vault, CyberArk Conjur, and hosted products like Doppler and Infisical. Instead of the secret sitting in a file next to your code, the application authenticates to the manager and fetches the secret at runtime.

What a vault adds over a bare env var:

• Centralized, encrypted storage, often backed by a Hardware Security Module (HSM) so keys are never plain text at rest.
• Immutable audit logs — every read is recorded, turning incident response from guesswork into a data-driven investigation. You can answer "did anyone read the Stripe key last Tuesday?"
• Granular, least-privilege access scoped per workload, so the email skill can fetch the email token and nothing else.
• Automated rotation without restarts, negotiating directly with the credential provider.
• Dynamic injection decoupled from the deployment artifact — the image and code carry no secrets.

The trade-off is engineering effort: you fetch secrets through an SDK or CLI, define access policies, and update pipelines. For a hobby bot on a single machine that may be overkill on day one; for an assistant that reads strangers' messages and runs third-party skills, it is exactly the isolation you want. Note that "secret files" — mounted files, tmpfs volumes, or .env files — are a middle ground: a mounted file is not in /proc/environ and is not inherited by every child, but it still lacks encryption, rotation, and audit unless a manager provides them.

The comparison below summarizes the three storage models:

Runtime Injection vs. Baked-In Secrets

The single worst place to put a secret is inside the deployment artifact — hardcoded in source, or baked into a container image via ENV OPENAI_API_KEY=sk-... or a copied .env. A baked-in secret ships wherever the image ships: to a registry, to every developer who pulls it, into every layer of the image history (deleting it in a later layer does not remove it from earlier layers), and into your git history if the Dockerfile is committed. It cannot be rotated without rebuilding and redeploying.

The alternative is runtime injection: the artifact contains no secrets, and the credential is delivered into the process only when it starts or, better, fetched on demand from the vault. The CNCF recommends that secrets be "injected at runtime through non-persistent mechanisms that are immune to leaks via logs, audit, or system dumps." In practice a sidecar or agent (for example, HashiCorp's Vault Agent) authenticates to the vault, retrieves current secrets, and writes them to a tmpfs file or the process environment at launch — with nothing sensitive persisted to the image or the repo. The distilled operating principle: store once, fetch just in time, scope narrowly, rotate automatically, and log every access.

Figure 8.2: Baked-in secrets ship everywhere vs. runtime injection keeps the artifact clean

Never Commit API Keys

How Keys Leak Into Git History, Logs, and Error Messages

The most common credential leak is embarrassingly simple: a developer hardcodes a key while testing, forgets it, and commits it — frequently to a public repository. Human error, not exotic attacks, remains the dominant cause of secret exposure. The trap that catches people is the belief that deleting the secret fixes it. It does not. Git permanently records every commit, so a secret committed once remains discoverable in the repository's history indefinitely — even after you delete the file in a later commit, make the repo private, or delete and recreate it. Anyone who cloned or forked before you noticed still has the full history. As GitGuardian puts it, leaking a secret and deleting it is "just like accidentally posting an embarrassing tweet, deleting it and just hoping no one saw it."

Worse, exploitation is fast and automated. GitHub exposes a public API that streams every commit in real time, and bots continuously scan that stream for fresh credentials. Leaked credentials from a public repo can be collected and exploited within minutes, and are typically abused within hours. For an unattended personal assistant, the window between "oops" and "someone is spending your OpenAI budget or reading your email" can be shorter than your next coffee break. Git commits are not the only leak path: the same key surfaces in logs (an assistant that logs its full outbound HTTP request logs the Authorization header), in error messages (a stack trace that prints the connection string), and in the many env-var channels above.

gitignore, Pre-Commit Scanning, and Push Protection

Defense against committed secrets is layered, moving detection as early ("left") as possible so the secret never reaches the remote:

• 1. .gitignore — the first, weakest gate. Add .env, *.pem, credentials.json, and similar patterns so they are never staged by accident. Necessary but insufficient: it does nothing about a key hardcoded inside a tracked file, and a developer can always git add -f.
• 2. Pre-commit hooks — the earliest interception. Tools like Gitleaks and TruffleHog run locally before a commit is created, scanning the staged diff for high-entropy strings and known secret patterns. Because they run before the commit exists, this is the earliest possible interception point — but a local hook can be bypassed with git commit --no-verify.
• 3. GitHub push protection — the gate at the remote. Push protection blocks a push outright when it contains a detected secret, stopping the credential before it ever reaches the remote. This layer catches the developer who skipped the local hook.
• 4. GitHub secret scanning — detection after the fact. Secret scanning examines the entire git history on all branches, plus issues, PRs, discussions, wikis, and gists. Partner notification lets GitHub tell a provider directly so they can often auto-revoke the key; validity checks confirm whether a detected secret is still live; and custom patterns cover org-specific tokens.

What To Do When a Secret Is Exposed

When a secret leaks, the instinct is to scrub the git history first. That instinct is wrong and dangerous, because rewriting history does nothing to stop an attacker who already copied the key. Any leaked secret must be treated as immediately compromised. Suppose your assistant's OpenAI key appears in a public commit at 2:14 PM. Following GitHub's recommended remediation flow — a revoke-first runbook:

1. Identify the secret: it is the OPENAI_API_KEY, hardcoded in agent/config.py, owned by your assistant's OpenAI project.
2. Assess risk: a live production key with billing attached and full model access. High risk.
3. Strategize: high risk means act immediately, not after a coordinated cleanup.
4. Revoke the secret with the provider. This is the single most important step — revoking the key neutralizes it regardless of git history, so even a copy in an attacker's hands stops working. Do this first.
5. Update affected services with a freshly issued key everywhere the old one was used — the assistant process, CI, any sidecar.
6. Check for unauthorized access by reviewing access and audit logs (provider usage dashboard, AWS CloudTrail/CloudWatch, Slack audit logs) to see whether the key was already abused.
7. Clean repository history (optional). Rewriting history is destructive, usually requires a force-push, and does not un-expose an already-discovered secret — weigh compliance needs against disruption.
8. Resolve the alert and document the incident.
9. Prevent future leaks by enabling secret scanning plus push protection and adopting the pre-commit habits above.

The order is the lesson: revoke first, clean up later. Rotation and revocation neutralize the secret; history rewriting only tidies the evidence. This is where the earlier investment in rotation pays off — if revoking and reissuing a key is a one-command operation you have practiced, step 4 takes seconds instead of an afternoon.

Figure 8.3: Revoke-first remediation runbook for a leaked secret

Credential Rotation

Why Rotation Limits the Value of a Stolen Secret

Long-lived static credentials are among the most significant vulnerabilities in modern infrastructure: once stolen, they stay valid for weeks, months, or years, giving an attacker an extended window to exploit them. Rotation attacks this directly by capping how long any single credential is usable. Return to the hotel keycard analogy. A traditional metal house key is a static credential: copy it once and you have access until the lock is physically changed. A hotel keycard is a rotating credential: it works today and is dead at checkout, so a copy made yesterday is worthless tomorrow. Short-lived (ephemeral) credentials apply the keycard model to software — they automatically expire after a brief, predefined period, typically minutes or hours. This shifts your posture from prevention (never let the key leak) to rapid containment (assume it might leak, and make sure it expires before it matters).

Automated Rotation, Short-Lived Tokens, and Dynamic Secrets

A dynamic secret is a credential generated on demand and revoked automatically when it expires, rather than a static value stored and manually rotated. HashiCorp Vault's database secrets engine is the canonical example, and its lifecycle illustrates the pattern:

1. Generation — Vault creates fresh credentials with a defined time-to-live (TTL).
2. Issuance — the credentials come with a lease ID that tracks their lifetime.
3. Usage — the client uses the temporary credentials for the lease period, renewing if needed.
4. Revocation — at lease expiration (or manual trigger), Vault automatically disables or deletes the credentials.

Figure 8.4: Lifecycle of a dynamic secret (generation to automatic revocation)

Rather than your assistant holding one permanent database password, it asks Vault for credentials when it needs them; Vault mints a unique username/password valid for, say, one hour, and tears them down afterward. A short-lived token is the same idea for API access: an OAuth access token with a short expiry plus a refresh token, or IAM temporary credentials, so no long-lived key is ever stored.

Where does the "secret zero" chain end — how does the process authenticate to Vault without yet another static key? The modern answer is OIDC / workload identity federation. In GitHub Actions, the official Vault Action authenticates via OIDC, so the pipeline proves its identity to Vault using a short-lived, GitHub-issued token instead of a stored static secret. This eliminates the last long-lived key. Similarly, Vault's AWS secrets engine dynamically generates IAM access keys scoped to a policy and rotates them automatically, and the PKI/SSH engines issue short-lived certificates instead of persistent keys.

Figure 8.5: OIDC workload-identity federation ends the "secret zero" chain

Automation is what makes short TTLs practical. If credentials expired mid-task with no renewal, legitimate work would break. Tools like Vault Agent automatically authenticate to Vault and renew short-lived secrets, writing them to a local file or environment variable, so expiration never interrupts a running workflow and you write no custom token-renewal code.

Rotation Runbooks and Blast-Radius Reduction

Rotation's payoff is blast-radius reduction — shrinking how much damage a single compromised credential can do. Short TTLs compress the attacker's usable window from calendar durations to minutes: a one-hour TTL on production database credentials means an intercepted credential is useless within sixty minutes. Netflix's "Bless" SSH-certificate system is the canonical illustration — employees get briefly-valid certificates, so any intercepted credential expires almost immediately. And because dynamic credentials are unique per request, a compromised application's credentials can be revoked individually rather than forcing a global rotation across every service.

Rotation best practices worth codifying for an assistant:

• Set TTLs on all dynamic secrets and keep them as short as the application tolerates.
• Tier TTLs by environment: production shortest, then staging, then development (~1 hour for production DB credentials, ~24 hours for staging, longer for dev).
• Rotate the root/bootstrap credential too — Vault's database engine can rotate the very root credential it uses to connect.
• Automate renewal (Vault Agent or equivalent) so expiration never causes an outage.
• Monitor lease and rotation metrics; a sudden spike in active leases (e.g., vault.expire.num_leases) can signal a leaked credential being replayed.

A rotation runbook turns all of this into a rehearsed, boring procedure so that the leaked-secret response is fast under pressure. A minimal runbook entry for a personal assistant:

Adopting short-lived credentials does not have to be a big-bang migration. A practical path is to start small with one low-risk pipeline, inventory existing static credentials using the secret-scanning tools from earlier, pilot a dynamic secrets engine against a database or PKI, integrate with CI/CD using OIDC-based authentication, and only then add monitoring and scale. Together, these turn the inevitable secret exposure from "how bad is the breach" into "the key had already expired."