maidn-cli/docs/operations.md

Operations Runbook

Use this runbook from the MaidnCLI checkout. Bootstrap YAML, age identities, OpenBao recovery material, Terraform state, and generated workspaces are secret-bearing local inputs. Do not commit or print them.

Normal Reconciliation

Run after a merged GitOps migration or to recover ordinary drift:

go run . bootstrap --config <private-bootstrap-config> --mode=reconcile

This is the only regular lifecycle command. Do not use direct kubectl apply, flux reconcile, Helm upgrades, or mutating talosctl commands.

Bootstrap records the CICD, manifests, and Talos template commits in the workspace's maidn-template-revisions.yaml before it creates migration PRs or touches infrastructure. Later runs require the same template sources and refs and check out those commits, even when a configured branch advances. To accept new template revisions intentionally, use a new empty workspaceDir (and a fresh cloneParent when it is configured separately) and keep the prior secret-bearing workspace intact for recovery.

Standard Delivery Workflow

Open and merge reviewed platform-source PRs before reconciling. When a merged template source must replace an existing revision lock, run the refresh by itself. It creates migration PRs; review and merge them, then rerun normal reconciliation:

go run . bootstrap --config <private-bootstrap-config> --mode=reconcile --refresh-template-revisions
go run . bootstrap --config <private-bootstrap-config> --mode=reconcile

Initialize the shared delivery platform before onboarding an application. This creates the Gateway, Tekton, and Tekton Triggers platform resources using tekton.<cluster-domain> and the configured Forgejo owner:

go run . bootstrap init --config <private-bootstrap-config> --organization <owner> --create-organization

Use a clean checkout on delivery.appRepoRef. The app repository URL must be the canonical source owner, such as Maidn/<app>.git; test-org-2 is only the execution owner. Set per-app static build values with command-line overrides, not by rewriting the private default:

go run . app onboard --config <private-bootstrap-config> --from <clean-checkout> `
  --app-name <app> --app-repo-url https://<forgejo>/Maidn/<app>.git `
  --image-repository <registry>/<owner>/<app> --build-strategy static `
  --build-output-directory <output-directory> --build-configuration <configuration>

For a runtime build, set --build-strategy runtime; static-only output options remain harmless. Onboarding opens, but never merges, a source delivery PR and a cluster registration PR. It registers the Forgejo hook but does not emit a test delivery. Review and merge the source delivery PR first, then the cluster registration PR. The registration is stored in base/tekton/apps/<app>.yaml; Flux loads the source-owned .tekton path and the generic EventListener dispatches by Forgejo repository name.

Existing registrations are never overwritten. Move a legacy source registration to its canonical source in a separate reviewed cluster-repository PR before onboarding that application.

After Flux reports the app Kustomization Ready, use Forgejo's hook test endpoint against a non-main ref and inspect the resulting PipelineRun. The command and read-only checks are in Webhook Smoke Test.

Shared Services And Add-ons

Environment databases are platform-owned shared services. Staging workloads use the CNPG-generated staging-postgres-app Secret and production workloads use production-postgres-app; applications must not declare their own CNPG Cluster by default. Shared credentials are appropriate only for the shared environment database. Use a dedicated service only when isolation, lifecycle, or storage requirements demand it.

An application can carry reviewed dedicated resources in .maidn/kustomization.yaml. Onboarding registers that path as the app's <app>-addons Flux Kustomization without changing the generated .tekton files. Runtime secret grants also generate the app's .maidn/secret-access.yaml there. Add-ons must declare their namespace explicitly and contain references, never credential values. Preview namespaces do not receive staging or production runtime credentials; preview-safe configuration is the application chart's responsibility.

Declare runtime secret access in the private configuration and run normal bootstrap reconciliation to create its policy and role. Set values only with app secret set using stdin, --file, or --generate; provision the scoped E2E identity with bootstrap --provision-app-secret-identities --e2e-app <app> only when a probe needs it. See secret-grants.md.

delivery.productionBranch is the application production branch. A production delivery PR targets the configured generic <owner>/<manifests-repo> repository semantics (for example, test-org-2/<manifests-repo>), never through a source checkout's former owner.

Rebuild

Use only when an authorized recovery requires recreating the Talos VM:

go run . bootstrap --config <private-bootstrap-config> --mode=rebuild --yes

The rebuild replaces Terraform-managed Talos VMs only. It does not manage or delete TrueNAS datasets or unrelated infrastructure.

Planned Cluster Removal

There is currently no maidn cluster remove or maidn uninstall command. Do not substitute direct Terraform or Kubernetes deletion for a supported workflow. A future remove command must require the named-cluster confirmation, limit its Terraform scope to Maidn Talos VMs, and leave external storage and shared infrastructure under their respective operator procedures.

The command must handle partial removal and retry safely: verify each VM's actual Proxmox state, accept already-absent Maidn VMs, report API timeouts as incomplete rather than successful removal, and leave a sanitized removal record in the isolated workspace. It must not delete shared network bridges, TrueNAS datasets, SOPS identities, or unrelated infrastructure. Until that command exists, an operator must use the authorized recovery procedure and resolve Proxmox API failures before retrying bootstrap.

OpenBao And Webhooks

After a rebuild or an OpenBao restart, refresh Kubernetes auth and reseed the encrypted operational values:

go run . bootstrap --config <private-bootstrap-config> --initialize-openbao

When the Pipeline and EventListener are already Ready, register or update the Forgejo webhook without re-running the full lifecycle:

go run . bootstrap --config <private-bootstrap-config> --register-webhook

The webhook authorization value stays in encrypted operational secrets and is never supplied on the command line.

If that authorization value is exposed, replace it and reconcile both OpenBao and the Forgejo hook in one command:

go run . bootstrap --config <private-bootstrap-config> --rotate-webhook-authorization

Read-Only Verification

Set KUBECONFIG to the generated kubeconfig for the configured cluster, then check the control plane and delivery chain:

kubectl -n flux-system get kustomizations
kubectl get clustersecretstores
kubectl -n flux-system get externalsecrets
kubectl -n tekton-pipelines get pipelines,eventlisteners,externalsecrets

Expected state:

• Flux Kustomizations are READY=True.
• ClusterSecretStore/openbao is READY=True.
• Forgejo credential ExternalSecrets are SecretSynced.
• The application Pipeline exists and the EventListener is available.

If OpenBao authentication is invalid after a rebuild, run --initialize-openbao, then allow the controllers to retry. Do not recreate the ClusterSecretStore or Secrets manually.

External DNS

Webhook delivery requires the configured tekton.<cluster-domain> hostname to resolve through Pi-hole to the Cilium Gateway address. ExternalDNS uses the Pi-hole provider with Gateway API routes, an upsert-only policy, and no ownership registry.

kubectl -n external-dns get pods,externalsecrets
kubectl -n external-dns logs deployment/external-dns --tail=100

If ExternalSecret/pihole-credentials is not SecretSynced, rerun --prompt-operational-secrets to enter the Pi-hole server and password, then run --initialize-openbao and --register-webhook. Do not create or edit the provider Secret directly.

Webhook TLS

The public Gateway terminates HTTPS with a cert-manager certificate. The Cloudflare API token issues the nid3.com certificate and manages explicit Tunnel CNAME records. Pi-hole remains the ExternalDNS provider. Check certificate readiness with:

kubectl -n cert-manager get externalsecret cloudflare-api-token
kubectl -n gateway-system get certificate webhook-tls

Enter the Pi-hole values and Cloudflare API token through --prompt-operational-secrets, then import the user-approved local credentials file before bootstrap:

go run . cloudflare-tunnel import --config <private-bootstrap-config> --credentials-file <local-credentials-json>

Do not put the Cloudflare token or tunnel credentials in the cluster repository.

Cloudflare Tunnel Routes

After importing the credentials, manage one explicit proxied CNAME and ingress rule at a time:

go run . cloudflare-tunnel route add --config <private-bootstrap-config> --hostname <public-hostname> --service <http-or-https-upstream-url> --zone-id <cloudflare-zone-id>
go run . cloudflare-tunnel route remove --config <private-bootstrap-config> --hostname <public-hostname> --service <http-or-https-upstream-url> --zone-id <cloudflare-zone-id>

Both commands require all three route values, preserve the terminal 404 rule, and refuse an existing CNAME that does not point to the managed tunnel.

Internal Platform UIs

Pi-hole resolves these HTTPS names to the Cilium Gateway only on the LAN:

• https://grafana.<cluster-domain>/ for Grafana. Authenticate with Grafana.
• https://openbao.<cluster-domain>/ for OpenBao. Authenticate with an OpenBao token.

Hubble UI is enabled for in-cluster troubleshooting but has no LAN route because it does not provide authentication. Add an authenticated proxy before exposing it outside the cluster.

Webhook Smoke Test

Use Forgejo's hook test endpoint against an existing non-main ref. It emits a real push delivery, runs the Node build, and pushes a SHA-tagged registry image, but skips the main-only staging manifest update:

# Discover the hook ID and choose an existing non-main branch or tag.
Invoke-RestMethod -Headers @{ Authorization = "token $env:FORGEJO_TOKEN" } `
  -Uri "https://<forgejo>/api/v1/repos/<owner>/<repo>/hooks"

Invoke-WebRequest -Method Post -Headers @{ Authorization = "token $env:FORGEJO_TOKEN" } `
  -Uri "https://<forgejo>/api/v1/repos/<owner>/<repo>/hooks/<hook-id>/tests?ref=<non-main-ref>"

Forgejo returns 204 after accepting the delivery. Confirm the resulting PipelineRun instead of treating 204 as a successful build:

kubectl -n tekton-pipelines get pipelineruns
kubectl -n tekton-pipelines describe pipelinerun <name>