# Operations Runbook Use this runbook from the MaidnCLI checkout. Bootstrap YAML, age identities, OpenBao recovery material, Terraform state, and generated workspaces are secret-bearing local inputs. Do not commit or print them. ## Normal Reconciliation Run after a merged GitOps migration or to recover ordinary drift: ```powershell go run . bootstrap --config --mode=reconcile ``` This is the only regular lifecycle command. Do not use direct `kubectl apply`, `flux reconcile`, Helm upgrades, or mutating `talosctl` commands. Bootstrap records the CICD, manifests, and Talos template commits in the workspace's `maidn-template-revisions.yaml` before it creates migration PRs or touches infrastructure. Later runs require the same template sources and refs and check out those commits, even when a configured branch advances. To accept new template revisions intentionally, use a new empty `workspaceDir` (and a fresh `cloneParent` when it is configured separately) and keep the prior secret-bearing workspace intact for recovery. ## Standard Delivery Workflow Open and merge reviewed platform-source PRs before reconciling. When a merged template source must replace an existing revision lock, run the refresh by itself. It creates migration PRs; review and merge them, then rerun normal reconciliation: ```powershell go run . bootstrap --config --mode=reconcile --refresh-template-revisions go run . bootstrap --config --mode=reconcile ``` Initialize the shared delivery platform before onboarding an application. This creates the Gateway, Tekton, and Tekton Triggers platform resources using `tekton.` and the configured Forgejo owner: ```powershell go run . bootstrap init --config --organization --create-organization ``` Use a clean checkout on `delivery.appRepoRef`. The app repository URL must be the canonical source owner, such as `Maidn/.git`; `test-org-2` is only the execution owner. Set per-app static build values with command-line overrides, not by rewriting the private default: ```powershell go run . app onboard --config --from ` --app-name --app-repo-url https:///Maidn/.git ` --image-repository // --build-strategy static ` --build-output-directory --build-configuration ``` For a runtime build, set `--build-strategy runtime`; static-only output options remain harmless. Onboarding opens, but never merges, a source delivery PR and a cluster registration PR. It registers the Forgejo hook but does not emit a test delivery. Review and merge the source delivery PR first, then the cluster registration PR. The registration is stored in `base/tekton/apps/.yaml`; Flux loads the source-owned `.tekton` path and the generic EventListener dispatches by Forgejo repository name. After Flux reports the app Kustomization Ready, use Forgejo's hook test endpoint against a non-`main` ref and inspect the resulting PipelineRun. The command and read-only checks are in [Webhook Smoke Test](#webhook-smoke-test). ### Shared Services And Add-ons Environment databases are platform-owned shared services. Staging workloads use the CNPG-generated `staging-postgres-app` Secret and production workloads use `production-postgres-app`; applications must not declare their own CNPG Cluster by default. Shared credentials are appropriate only for the shared environment database. Use a dedicated service only when isolation, lifecycle, or storage requirements demand it. An application can carry reviewed dedicated resources in `.maidn/kustomization.yaml`. Onboarding registers that path as the app's `-addons` Flux Kustomization without changing the generated `.tekton` files. Runtime secret grants also generate the app's `.maidn/secret-access.yaml` there. Add-ons must declare their namespace explicitly and contain references, never credential values. Preview namespaces do not receive staging or production runtime credentials; preview-safe configuration is the application chart's responsibility. Declare runtime secret access in the private configuration and run normal bootstrap reconciliation to create its policy and role. Set values only with `app secret set` using stdin, `--file`, or `--generate`; provision the scoped E2E identity with `bootstrap --provision-app-secret-identities --e2e-app ` only when a probe needs it. See [secret-grants.md](secret-grants.md). `delivery.productionBranch` is the application production branch. A production delivery PR targets the configured generic `/` repository semantics (for example, `test-org-2/`), never through a source checkout's former owner. ## Rebuild Use only when an authorized recovery requires recreating the Talos VM: ```powershell go run . bootstrap --config --mode=rebuild --yes ``` The rebuild replaces Terraform-managed Talos VMs only. It does not manage or delete TrueNAS datasets or unrelated infrastructure. ## Planned Cluster Removal There is currently no `maidn cluster remove` or `maidn uninstall` command. Do not substitute direct Terraform or Kubernetes deletion for a supported workflow. A future remove command must require the named-cluster confirmation, limit its Terraform scope to Maidn Talos VMs, and leave external storage and shared infrastructure under their respective operator procedures. The command must handle partial removal and retry safely: verify each VM's actual Proxmox state, accept already-absent Maidn VMs, report API timeouts as incomplete rather than successful removal, and leave a sanitized removal record in the isolated workspace. It must not delete shared network bridges, TrueNAS datasets, SOPS identities, or unrelated infrastructure. Until that command exists, an operator must use the authorized recovery procedure and resolve Proxmox API failures before retrying bootstrap. ## OpenBao And Webhooks After a rebuild or an OpenBao restart, refresh Kubernetes auth and reseed the encrypted operational values: ```powershell go run . bootstrap --config --initialize-openbao ``` When the Pipeline and EventListener are already Ready, register or update the Forgejo webhook without re-running the full lifecycle: ```powershell go run . bootstrap --config --register-webhook ``` The webhook authorization value stays in encrypted operational secrets and is never supplied on the command line. If that authorization value is exposed, replace it and reconcile both OpenBao and the Forgejo hook in one command: ```powershell go run . bootstrap --config --rotate-webhook-authorization ``` ## Read-Only Verification Set `KUBECONFIG` to the generated kubeconfig for the configured cluster, then check the control plane and delivery chain: ```powershell kubectl -n flux-system get kustomizations kubectl get clustersecretstores kubectl -n flux-system get externalsecrets kubectl -n tekton-pipelines get pipelines,eventlisteners,externalsecrets ``` Expected state: - Flux Kustomizations are `READY=True`. - `ClusterSecretStore/openbao` is `READY=True`. - Forgejo credential ExternalSecrets are `SecretSynced`. - The application Pipeline exists and the EventListener is available. If OpenBao authentication is invalid after a rebuild, run `--initialize-openbao`, then allow the controllers to retry. Do not recreate the ClusterSecretStore or Secrets manually. ## External DNS Webhook delivery requires the configured `tekton.` hostname to resolve through Pi-hole to the Cilium Gateway address. ExternalDNS uses the Pi-hole provider with Gateway API routes, an `upsert-only` policy, and no ownership registry. ```powershell kubectl -n external-dns get pods,externalsecrets kubectl -n external-dns logs deployment/external-dns --tail=100 ``` If `ExternalSecret/pihole-credentials` is not `SecretSynced`, rerun `--prompt-operational-secrets` to enter the Pi-hole server and password, then run `--initialize-openbao` and `--register-webhook`. Do not create or edit the provider Secret directly. ## Webhook TLS The public Gateway terminates HTTPS with a cert-manager certificate. The Cloudflare API token issues the `nid3.com` certificate and manages explicit Tunnel CNAME records. Pi-hole remains the ExternalDNS provider. Check certificate readiness with: ```powershell kubectl -n cert-manager get externalsecret cloudflare-api-token kubectl -n gateway-system get certificate webhook-tls ``` Enter the Pi-hole values and Cloudflare API token through `--prompt-operational-secrets`, then import the user-approved local credentials file before bootstrap: ```powershell go run . cloudflare-tunnel import --config --credentials-file ``` Do not put the Cloudflare token or tunnel credentials in the cluster repository. ## Cloudflare Tunnel Routes After importing the credentials, manage one explicit proxied CNAME and ingress rule at a time: ```powershell go run . cloudflare-tunnel route add --config --hostname --service --zone-id go run . cloudflare-tunnel route remove --config --hostname --service --zone-id ``` Both commands require all three route values, preserve the terminal 404 rule, and refuse an existing CNAME that does not point to the managed tunnel. ## Internal Platform UIs Pi-hole resolves these HTTPS names to the Cilium Gateway only on the LAN: - `https://grafana./` for Grafana. Authenticate with Grafana. - `https://openbao./` for OpenBao. Authenticate with an OpenBao token. Hubble UI is enabled for in-cluster troubleshooting but has no LAN route because it does not provide authentication. Add an authenticated proxy before exposing it outside the cluster. ## Webhook Smoke Test Use Forgejo's hook test endpoint against an existing non-`main` ref. It emits a real push delivery, runs the Node build, and pushes a SHA-tagged registry image, but skips the main-only staging manifest update: ```powershell # Discover the hook ID and choose an existing non-main branch or tag. Invoke-RestMethod -Headers @{ Authorization = "token $env:FORGEJO_TOKEN" } ` -Uri "https:///api/v1/repos///hooks" Invoke-WebRequest -Method Post -Headers @{ Authorization = "token $env:FORGEJO_TOKEN" } ` -Uri "https:///api/v1/repos///hooks//tests?ref=" ``` Forgejo returns `204` after accepting the delivery. Confirm the resulting PipelineRun instead of treating `204` as a successful build: ```powershell kubectl -n tekton-pipelines get pipelineruns kubectl -n tekton-pipelines describe pipelinerun ```