Operations Runbook

Use this runbook from the MaidnCLI checkout. Bootstrap YAML, age identities, OpenBao recovery material, Terraform state, and generated workspaces are secret-bearing local inputs. Do not commit or print them.

Normal Reconciliation

Run after a merged GitOps migration or to recover ordinary drift:

go run . bootstrap --config <private-bootstrap-config> --mode=reconcile

This is the only regular lifecycle command. Do not use direct kubectl apply, flux reconcile, Helm upgrades, or mutating talosctl commands.

Rebuild

Use only when an authorized recovery requires recreating the Talos VM:

go run . bootstrap --config <private-bootstrap-config> --mode=rebuild --yes

The rebuild replaces Terraform-managed Talos VMs only. It does not manage or delete TrueNAS datasets or unrelated infrastructure.

OpenBao And Webhooks

After a rebuild or an OpenBao restart, refresh Kubernetes auth and reseed the encrypted operational values:

go run . bootstrap --config <private-bootstrap-config> --initialize-openbao

When the Pipeline and EventListener are already Ready, register or update the Forgejo webhook without re-running the full lifecycle:

go run . bootstrap --config <private-bootstrap-config> --register-webhook

The webhook authorization value stays in encrypted operational secrets and is never supplied on the command line.

If that authorization value is exposed, replace it and reconcile both OpenBao and the Forgejo hook in one command:

go run . bootstrap --config <private-bootstrap-config> --rotate-webhook-authorization

Read-Only Verification

Set KUBECONFIG to the generated kubeconfig for the configured cluster, then check the control plane and delivery chain:

kubectl -n flux-system get kustomizations
kubectl get clustersecretstores
kubectl -n flux-system get externalsecrets
kubectl -n tekton-pipelines get pipelines,eventlisteners,externalsecrets

Expected state:

Flux Kustomizations are READY=True.
ClusterSecretStore/openbao is READY=True.
Forgejo credential ExternalSecrets are SecretSynced.
The application Pipeline exists and the EventListener is available.

If OpenBao authentication is invalid after a rebuild, run --initialize-openbao, then allow the controllers to retry. Do not recreate the ClusterSecretStore or Secrets manually.

External DNS

Webhook delivery requires the configured tekton.<cluster-domain> hostname to resolve through Pi-hole to the Cilium Gateway address. ExternalDNS uses the Pi-hole provider with Gateway API routes, an upsert-only policy, and no ownership registry.

kubectl -n external-dns get pods,externalsecrets
kubectl -n external-dns logs deployment/external-dns --tail=100

If ExternalSecret/pihole-credentials is not SecretSynced, rerun --prompt-operational-secrets to enter the Pi-hole server and password, then run --initialize-openbao and --register-webhook. Do not create or edit the provider Secret directly.

Webhook TLS

The public Gateway terminates HTTPS with a cert-manager certificate. Its Cloudflare DNS-01 token is used only to issue the nid3.com certificate; Pi-hole remains the ExternalDNS provider. Check certificate readiness with:

kubectl -n cert-manager get externalsecret cloudflare-api-token
kubectl -n gateway-system get certificate webhook-tls

Enter the Pi-hole values, Cloudflare DNS-01 token, and Tunnel token through --prompt-operational-secrets, then run --initialize-openbao. Do not put the Cloudflare token in the cluster repository.

Internal Platform UIs

Pi-hole resolves these HTTPS names to the Cilium Gateway only on the LAN:

https://grafana.<cluster-domain>/ for Grafana. Authenticate with Grafana.
https://openbao.<cluster-domain>/ for OpenBao. Authenticate with an OpenBao token.

Hubble UI is enabled for in-cluster troubleshooting but has no LAN route because it does not provide authentication. Add an authenticated proxy before exposing it outside the cluster.

Webhook Smoke Test

Use Forgejo's hook test endpoint against an existing non-main ref. It emits a real push delivery, runs the Node build, and pushes a SHA-tagged registry image, but skips the main-only staging manifest update:

# Discover the hook ID and choose an existing non-main branch or tag.
Invoke-RestMethod -Headers @{ Authorization = "token $env:FORGEJO_TOKEN" } `
  -Uri "https://<forgejo>/api/v1/repos/<owner>/<repo>/hooks"

Invoke-WebRequest -Method Post -Headers @{ Authorization = "token $env:FORGEJO_TOKEN" } `
  -Uri "https://<forgejo>/api/v1/repos/<owner>/<repo>/hooks/<hook-id>/tests?ref=<non-main-ref>"

Forgejo returns 204 after accepting the delivery. Confirm the resulting PipelineRun instead of treating 204 as a successful build:

kubectl -n tekton-pipelines get pipelineruns
kubectl -n tekton-pipelines describe pipelinerun <name>

4.9 KiB Raw Blame History