# Operations Runbook Use this runbook from the MaidnCLI checkout. Bootstrap YAML, age identities, OpenBao recovery material, Terraform state, and generated workspaces are secret-bearing local inputs. Do not commit or print them. ## Normal Reconciliation Run after a merged GitOps migration or to recover ordinary drift: ```powershell go run . bootstrap --config --mode=reconcile ``` This is the only regular lifecycle command. Do not use direct `kubectl apply`, `flux reconcile`, Helm upgrades, or mutating `talosctl` commands. Bootstrap records the CICD, manifests, and Talos template commits in the workspace's `maidn-template-revisions.yaml` before it creates migration PRs or touches infrastructure. Later runs require the same template sources and refs and check out those commits, even when a configured branch advances. To accept new template revisions intentionally, use a new empty `workspaceDir` (and a fresh `cloneParent` when it is configured separately) and keep the prior secret-bearing workspace intact for recovery. ## Rebuild Use only when an authorized recovery requires recreating the Talos VM: ```powershell go run . bootstrap --config --mode=rebuild --yes ``` The rebuild replaces Terraform-managed Talos VMs only. It does not manage or delete TrueNAS datasets or unrelated infrastructure. ## OpenBao And Webhooks After a rebuild or an OpenBao restart, refresh Kubernetes auth and reseed the encrypted operational values: ```powershell go run . bootstrap --config --initialize-openbao ``` When the Pipeline and EventListener are already Ready, register or update the Forgejo webhook without re-running the full lifecycle: ```powershell go run . bootstrap --config --register-webhook ``` The webhook authorization value stays in encrypted operational secrets and is never supplied on the command line. If that authorization value is exposed, replace it and reconcile both OpenBao and the Forgejo hook in one command: ```powershell go run . bootstrap --config --rotate-webhook-authorization ``` ## Read-Only Verification Set `KUBECONFIG` to the generated kubeconfig for the configured cluster, then check the control plane and delivery chain: ```powershell kubectl -n flux-system get kustomizations kubectl get clustersecretstores kubectl -n flux-system get externalsecrets kubectl -n tekton-pipelines get pipelines,eventlisteners,externalsecrets ``` Expected state: - Flux Kustomizations are `READY=True`. - `ClusterSecretStore/openbao` is `READY=True`. - Forgejo credential ExternalSecrets are `SecretSynced`. - The application Pipeline exists and the EventListener is available. If OpenBao authentication is invalid after a rebuild, run `--initialize-openbao`, then allow the controllers to retry. Do not recreate the ClusterSecretStore or Secrets manually. ## External DNS Webhook delivery requires the configured `tekton.` hostname to resolve through Pi-hole to the Cilium Gateway address. ExternalDNS uses the Pi-hole provider with Gateway API routes, an `upsert-only` policy, and no ownership registry. ```powershell kubectl -n external-dns get pods,externalsecrets kubectl -n external-dns logs deployment/external-dns --tail=100 ``` If `ExternalSecret/pihole-credentials` is not `SecretSynced`, rerun `--prompt-operational-secrets` to enter the Pi-hole server and password, then run `--initialize-openbao` and `--register-webhook`. Do not create or edit the provider Secret directly. ## Webhook TLS The public Gateway terminates HTTPS with a cert-manager certificate. The Cloudflare API token issues the `nid3.com` certificate and manages explicit Tunnel CNAME records. Pi-hole remains the ExternalDNS provider. Check certificate readiness with: ```powershell kubectl -n cert-manager get externalsecret cloudflare-api-token kubectl -n gateway-system get certificate webhook-tls ``` Enter the Pi-hole values and Cloudflare API token through `--prompt-operational-secrets`, then import the user-approved local credentials file before bootstrap: ```powershell go run . cloudflare-tunnel import --config --credentials-file ``` Do not put the Cloudflare token or tunnel credentials in the cluster repository. ## Cloudflare Tunnel Routes After importing the credentials, manage one explicit proxied CNAME and ingress rule at a time: ```powershell go run . cloudflare-tunnel route add --config --hostname --service --zone-id go run . cloudflare-tunnel route remove --config --hostname --service --zone-id ``` Both commands require all three route values, preserve the terminal 404 rule, and refuse an existing CNAME that does not point to the managed tunnel. ## Internal Platform UIs Pi-hole resolves these HTTPS names to the Cilium Gateway only on the LAN: - `https://grafana./` for Grafana. Authenticate with Grafana. - `https://openbao./` for OpenBao. Authenticate with an OpenBao token. Hubble UI is enabled for in-cluster troubleshooting but has no LAN route because it does not provide authentication. Add an authenticated proxy before exposing it outside the cluster. ## Webhook Smoke Test Use Forgejo's hook test endpoint against an existing non-`main` ref. It emits a real push delivery, runs the Node build, and pushes a SHA-tagged registry image, but skips the main-only staging manifest update: ```powershell # Discover the hook ID and choose an existing non-main branch or tag. Invoke-RestMethod -Headers @{ Authorization = "token $env:FORGEJO_TOKEN" } ` -Uri "https:///api/v1/repos///hooks" Invoke-WebRequest -Method Post -Headers @{ Authorization = "token $env:FORGEJO_TOKEN" } ` -Uri "https:///api/v1/repos///hooks//tests?ref=" ``` Forgejo returns `204` after accepting the delivery. Confirm the resulting PipelineRun instead of treating `204` as a successful build: ```powershell kubectl -n tekton-pipelines get pipelineruns kubectl -n tekton-pipelines describe pipelinerun ```