0004 — Tailscale auth key in Secret Manager, fetched at boot

Status

Accepted (2026-07-02)

Context

The original design passed the Tailscale auth key through terraform.tfvars into the instance startup script. That put the key, in plaintext, in three places: tfvars on disk, Terraform state, and instance metadata (readable by anyone with compute.instances.get). The key was subsequently treated as exposed.

Decision

Store the key in GCP-Secret-Manager (tailscale-auth-key), created out-of-band. The VM's attached service account (mcp-broker-sa, holding only secretAccessor on the named secrets) fetches it at first boot via the metadata-server token + Secret Manager REST API, parsed with python3 (grep/cut dies silently on pretty-printed JSON under pipefail — a latent bug found the first time this path actually executed). The key is one-time, short-expiry, not ephemeral (ephemeral nodes are deleted when they go offline — wrong for a persistent server).

Consequences

• The key never touches git, tfvars, state or instance metadata; it exists in memory for one tailscale up.
• Rotation is a Secret Manager version bump + VM replacement (gcp-mcp-standalone/runbooks/rotate-tailscale-key).
• The VM must be able to reach Secret Manager before joining the tailnet; a bootstrap failure leaves the box unreachable except via serial console (accepted for a single-admin box).