maidn-cli/docs/operations.md

Operations Runbook

Use this runbook from the MaidnCLI checkout. Bootstrap YAML, age identities, OpenBao recovery material, Terraform state, and generated workspaces are secret-bearing local inputs. Do not commit or print them.

Normal Reconciliation

Run after a merged GitOps migration or to recover ordinary drift:

go run . bootstrap --config <private-bootstrap-config> --mode=reconcile

This is the only regular lifecycle command. Do not use direct kubectl apply, flux reconcile, Helm upgrades, or mutating talosctl commands.

Bootstrap records the CICD, manifests, and Talos template commits in the workspace's maidn-template-revisions.yaml before it creates migration PRs or touches infrastructure. Later runs require the same template sources and refs and check out those commits, even when a configured branch advances. To accept new template revisions intentionally, use a new empty workspaceDir (and a fresh cloneParent when it is configured separately) and keep the prior secret-bearing workspace intact for recovery.

Platform Initialization And App Onboarding

Initialize the shared delivery platform before onboarding any application. This creates the Gateway, Tekton, and Tekton Triggers platform resources using tekton.<cluster-domain> and the configured Forgejo owner; it does not render an application Pipeline or register an application webhook:

bootstrap init --config <private-bootstrap-config> --organization <owner> --create-organization

Import a clean checkout into the configured owner with a separate command. The checkout may originate in another Forgejo organization, but delivery.appRepoUrl must target <owner>/<app>.git and its current branch must be delivery.appRepoRef:

app onboard --config <private-bootstrap-config> --from <clean-checkout>

Onboarding copies the current branch to the target repository, replaces only the two Maidn-generated .tekton files, protects the production branch, and auto-merges the app delivery and cluster registration PRs. The registration is stored in base/tekton/apps/<app>.yaml in the configured cluster repository; the generic EventListener dispatches by the Forgejo repository name.

delivery.productionBranch is the application production branch. A production delivery PR targets the configured generic <owner>/<manifests-repo> repository semantics (for example, test-org-2/<manifests-repo>), never through a source checkout's former owner.

Rebuild

Use only when an authorized recovery requires recreating the Talos VM:

go run . bootstrap --config <private-bootstrap-config> --mode=rebuild --yes

The rebuild replaces Terraform-managed Talos VMs only. It does not manage or delete TrueNAS datasets or unrelated infrastructure.

Planned Cluster Removal

There is currently no maidn cluster remove or maidn uninstall command. Do not substitute direct Terraform or Kubernetes deletion for a supported workflow. A future remove command must require the named-cluster confirmation, limit its Terraform scope to Maidn Talos VMs, and leave external storage and shared infrastructure under their respective operator procedures.

The command must handle partial removal and retry safely: verify each VM's actual Proxmox state, accept already-absent Maidn VMs, report API timeouts as incomplete rather than successful removal, and leave a sanitized removal record in the isolated workspace. It must not delete shared network bridges, TrueNAS datasets, SOPS identities, or unrelated infrastructure. Until that command exists, an operator must use the authorized recovery procedure and resolve Proxmox API failures before retrying bootstrap.

OpenBao And Webhooks

After a rebuild or an OpenBao restart, refresh Kubernetes auth and reseed the encrypted operational values:

go run . bootstrap --config <private-bootstrap-config> --initialize-openbao

When the Pipeline and EventListener are already Ready, register or update the Forgejo webhook without re-running the full lifecycle:

go run . bootstrap --config <private-bootstrap-config> --register-webhook

The webhook authorization value stays in encrypted operational secrets and is never supplied on the command line.

If that authorization value is exposed, replace it and reconcile both OpenBao and the Forgejo hook in one command:

go run . bootstrap --config <private-bootstrap-config> --rotate-webhook-authorization

Read-Only Verification

Set KUBECONFIG to the generated kubeconfig for the configured cluster, then check the control plane and delivery chain:

kubectl -n flux-system get kustomizations
kubectl get clustersecretstores
kubectl -n flux-system get externalsecrets
kubectl -n tekton-pipelines get pipelines,eventlisteners,externalsecrets

Expected state:

• Flux Kustomizations are READY=True.
• ClusterSecretStore/openbao is READY=True.
• Forgejo credential ExternalSecrets are SecretSynced.
• The application Pipeline exists and the EventListener is available.

If OpenBao authentication is invalid after a rebuild, run --initialize-openbao, then allow the controllers to retry. Do not recreate the ClusterSecretStore or Secrets manually.

External DNS

Webhook delivery requires the configured tekton.<cluster-domain> hostname to resolve through Pi-hole to the Cilium Gateway address. ExternalDNS uses the Pi-hole provider with Gateway API routes, an upsert-only policy, and no ownership registry.

kubectl -n external-dns get pods,externalsecrets
kubectl -n external-dns logs deployment/external-dns --tail=100

If ExternalSecret/pihole-credentials is not SecretSynced, rerun --prompt-operational-secrets to enter the Pi-hole server and password, then run --initialize-openbao and --register-webhook. Do not create or edit the provider Secret directly.

Webhook TLS

The public Gateway terminates HTTPS with a cert-manager certificate. The Cloudflare API token issues the nid3.com certificate and manages explicit Tunnel CNAME records. Pi-hole remains the ExternalDNS provider. Check certificate readiness with:

kubectl -n cert-manager get externalsecret cloudflare-api-token
kubectl -n gateway-system get certificate webhook-tls

Enter the Pi-hole values and Cloudflare API token through --prompt-operational-secrets, then import the user-approved local credentials file before bootstrap:

go run . cloudflare-tunnel import --config <private-bootstrap-config> --credentials-file <local-credentials-json>

Do not put the Cloudflare token or tunnel credentials in the cluster repository.

Cloudflare Tunnel Routes

After importing the credentials, manage one explicit proxied CNAME and ingress rule at a time:

go run . cloudflare-tunnel route add --config <private-bootstrap-config> --hostname <public-hostname> --service <http-or-https-upstream-url> --zone-id <cloudflare-zone-id>
go run . cloudflare-tunnel route remove --config <private-bootstrap-config> --hostname <public-hostname> --service <http-or-https-upstream-url> --zone-id <cloudflare-zone-id>

Both commands require all three route values, preserve the terminal 404 rule, and refuse an existing CNAME that does not point to the managed tunnel.

Internal Platform UIs

Pi-hole resolves these HTTPS names to the Cilium Gateway only on the LAN:

• https://grafana.<cluster-domain>/ for Grafana. Authenticate with Grafana.
• https://openbao.<cluster-domain>/ for OpenBao. Authenticate with an OpenBao token.

Hubble UI is enabled for in-cluster troubleshooting but has no LAN route because it does not provide authentication. Add an authenticated proxy before exposing it outside the cluster.

Webhook Smoke Test

Use Forgejo's hook test endpoint against an existing non-main ref. It emits a real push delivery, runs the Node build, and pushes a SHA-tagged registry image, but skips the main-only staging manifest update:

# Discover the hook ID and choose an existing non-main branch or tag.
Invoke-RestMethod -Headers @{ Authorization = "token $env:FORGEJO_TOKEN" } `
  -Uri "https://<forgejo>/api/v1/repos/<owner>/<repo>/hooks"

Invoke-WebRequest -Method Post -Headers @{ Authorization = "token $env:FORGEJO_TOKEN" } `
  -Uri "https://<forgejo>/api/v1/repos/<owner>/<repo>/hooks/<hook-id>/tests?ref=<non-main-ref>"

Forgejo returns 204 after accepting the delivery. Confirm the resulting PipelineRun instead of treating 204 as a successful build:

kubectl -n tekton-pipelines get pipelineruns
kubectl -n tekton-pipelines describe pipelinerun <name>