264 lines
11 KiB
Markdown
264 lines
11 KiB
Markdown
# Operations Runbook
|
|
|
|
Use this runbook from the MaidnCLI checkout. Bootstrap YAML, age identities,
|
|
OpenBao recovery material, Terraform state, and generated workspaces are
|
|
secret-bearing local inputs. Do not commit or print them.
|
|
|
|
## Normal Reconciliation
|
|
|
|
Run after a merged GitOps migration or to recover ordinary drift:
|
|
|
|
```powershell
|
|
go run . bootstrap --config <private-bootstrap-config> --mode=reconcile
|
|
```
|
|
|
|
This is the only regular lifecycle command. Do not use direct `kubectl apply`,
|
|
`flux reconcile`, Helm upgrades, or mutating `talosctl` commands.
|
|
|
|
Bootstrap records the CICD, manifests, and Talos template commits in the
|
|
workspace's `maidn-template-revisions.yaml` before it creates migration PRs or
|
|
touches infrastructure. Later runs require the same template sources and refs
|
|
and check out those commits, even when a configured branch advances. To accept
|
|
new template revisions intentionally, use a new empty `workspaceDir` (and a
|
|
fresh `cloneParent` when it is configured separately) and keep the prior
|
|
secret-bearing workspace intact for recovery.
|
|
|
|
## Standard Delivery Workflow
|
|
|
|
Open and merge reviewed platform-source PRs before reconciling. When a merged
|
|
template source must replace an existing revision lock, run the refresh by
|
|
itself. It creates migration PRs; review and merge them, then rerun normal
|
|
reconciliation:
|
|
|
|
```powershell
|
|
go run . bootstrap --config <private-bootstrap-config> --mode=reconcile --refresh-template-revisions
|
|
go run . bootstrap --config <private-bootstrap-config> --mode=reconcile
|
|
```
|
|
|
|
Initialize the shared delivery platform before onboarding an application. This
|
|
creates the Gateway, Tekton, and Tekton Triggers platform resources using
|
|
`tekton.<cluster-domain>` and the configured Forgejo owner:
|
|
|
|
```powershell
|
|
go run . bootstrap init --config <private-bootstrap-config> --organization <owner> --create-organization
|
|
```
|
|
|
|
Use a clean checkout on `delivery.appRepoRef`. The app repository URL must be
|
|
the canonical source owner, such as `Maidn/<app>.git`; `test-org-2` is only the
|
|
execution owner. Set per-app static build values with command-line overrides,
|
|
not by rewriting the private default:
|
|
|
|
```powershell
|
|
go run . app onboard --config <private-bootstrap-config> --from <clean-checkout> `
|
|
--app-name <app> --app-repo-url https://<forgejo>/Maidn/<app>.git `
|
|
--image-repository <registry>/<owner>/<app> --build-strategy static `
|
|
--build-output-directory <output-directory> --build-configuration <configuration>
|
|
```
|
|
|
|
For a runtime build, set `--build-strategy runtime`; static-only output options
|
|
remain harmless. Onboarding opens, but never merges, a source delivery PR and a
|
|
cluster registration PR. It registers the Forgejo hook but does not emit a test
|
|
delivery. Review and merge the source delivery PR first, then the cluster
|
|
registration PR. The registration is stored in
|
|
`base/tekton/apps/<app>.yaml`; Flux loads the source-owned `.tekton` path and
|
|
the generic EventListener dispatches by Forgejo repository name.
|
|
|
|
Existing registrations are never overwritten. Move a legacy source registration
|
|
to its canonical source in a separate reviewed cluster-repository PR before
|
|
onboarding that application.
|
|
|
|
After Flux reports the app Kustomization Ready, use Forgejo's hook test endpoint
|
|
against a non-`main` ref and inspect the resulting PipelineRun. The command and
|
|
read-only checks are in [Webhook Smoke Test](#webhook-smoke-test).
|
|
|
|
### Shared Services And Add-ons
|
|
|
|
Environment databases are platform-owned shared services. Staging workloads use
|
|
the CNPG-generated `staging-postgres-app` Secret and production workloads use
|
|
`production-postgres-app`; applications must not declare their own CNPG Cluster
|
|
by default. Shared credentials are appropriate only for the shared environment
|
|
database. Use a dedicated service only when isolation, lifecycle, or storage
|
|
requirements demand it.
|
|
|
|
An application can carry reviewed dedicated resources in
|
|
`.maidn/kustomization.yaml`. Onboarding registers that path as the app's
|
|
`<app>-addons` Flux Kustomization without changing the generated `.tekton`
|
|
files. Runtime secret grants also generate the app's `.maidn/secret-access.yaml`
|
|
there. Add-ons must declare their namespace explicitly and contain references,
|
|
never credential values. Preview namespaces do not receive staging or production
|
|
runtime credentials; preview-safe configuration is the application chart's
|
|
responsibility.
|
|
|
|
Declare runtime secret access in the private configuration and run normal
|
|
bootstrap reconciliation to create its policy and role. Set values only with
|
|
`app secret set` using stdin, `--file`, or `--generate`; provision the scoped
|
|
E2E identity with `bootstrap --provision-app-secret-identities --e2e-app <app>`
|
|
only when a probe needs it. See [secret-grants.md](secret-grants.md).
|
|
|
|
`delivery.productionBranch` is the application production branch. A production
|
|
delivery PR targets the configured generic
|
|
`<owner>/<manifests-repo>` repository semantics (for example,
|
|
`test-org-2/<manifests-repo>`), never through a source checkout's former owner.
|
|
|
|
## Rebuild
|
|
|
|
Use only when an authorized recovery requires recreating the Talos VM:
|
|
|
|
```powershell
|
|
go run . bootstrap --config <private-bootstrap-config> --mode=rebuild --yes
|
|
```
|
|
|
|
The rebuild replaces Terraform-managed Talos VMs only. It does not manage or
|
|
delete TrueNAS datasets or unrelated infrastructure.
|
|
|
|
## Planned Cluster Removal
|
|
|
|
There is currently no `maidn cluster remove` or `maidn uninstall` command.
|
|
Do not substitute direct Terraform or Kubernetes deletion for a supported
|
|
workflow. A future remove command must require the named-cluster confirmation,
|
|
limit its Terraform scope to Maidn Talos VMs, and leave external storage and
|
|
shared infrastructure under their respective operator procedures.
|
|
|
|
The command must handle partial removal and retry safely: verify each VM's
|
|
actual Proxmox state, accept already-absent Maidn VMs, report API timeouts as
|
|
incomplete rather than successful removal, and leave a sanitized removal
|
|
record in the isolated workspace. It must not delete shared network bridges,
|
|
TrueNAS datasets, SOPS identities, or unrelated infrastructure. Until that
|
|
command exists, an operator must use the authorized recovery procedure and
|
|
resolve Proxmox API failures before retrying bootstrap.
|
|
|
|
## OpenBao And Webhooks
|
|
|
|
After a rebuild or an OpenBao restart, refresh Kubernetes auth and reseed the
|
|
encrypted operational values:
|
|
|
|
```powershell
|
|
go run . bootstrap --config <private-bootstrap-config> --initialize-openbao
|
|
```
|
|
|
|
When the Pipeline and EventListener are already Ready, register or update the
|
|
Forgejo webhook without re-running the full lifecycle:
|
|
|
|
```powershell
|
|
go run . bootstrap --config <private-bootstrap-config> --register-webhook
|
|
```
|
|
|
|
The webhook authorization value stays in encrypted operational secrets and is
|
|
never supplied on the command line.
|
|
|
|
If that authorization value is exposed, replace it and reconcile both OpenBao
|
|
and the Forgejo hook in one command:
|
|
|
|
```powershell
|
|
go run . bootstrap --config <private-bootstrap-config> --rotate-webhook-authorization
|
|
```
|
|
|
|
## Read-Only Verification
|
|
|
|
Set `KUBECONFIG` to the generated kubeconfig for the configured cluster, then
|
|
check the control plane and delivery chain:
|
|
|
|
```powershell
|
|
kubectl -n flux-system get kustomizations
|
|
kubectl get clustersecretstores
|
|
kubectl -n flux-system get externalsecrets
|
|
kubectl -n tekton-pipelines get pipelines,eventlisteners,externalsecrets
|
|
```
|
|
|
|
Expected state:
|
|
|
|
- Flux Kustomizations are `READY=True`.
|
|
- `ClusterSecretStore/openbao` is `READY=True`.
|
|
- Forgejo credential ExternalSecrets are `SecretSynced`.
|
|
- The application Pipeline exists and the EventListener is available.
|
|
|
|
If OpenBao authentication is invalid after a rebuild, run
|
|
`--initialize-openbao`, then allow the controllers to retry. Do not recreate
|
|
the ClusterSecretStore or Secrets manually.
|
|
|
|
## External DNS
|
|
|
|
Webhook delivery requires the configured `tekton.<cluster-domain>` hostname to
|
|
resolve through Pi-hole to the Cilium Gateway address. ExternalDNS uses the
|
|
Pi-hole provider with Gateway API routes, an `upsert-only` policy, and no
|
|
ownership registry.
|
|
|
|
```powershell
|
|
kubectl -n external-dns get pods,externalsecrets
|
|
kubectl -n external-dns logs deployment/external-dns --tail=100
|
|
```
|
|
|
|
If `ExternalSecret/pihole-credentials` is not `SecretSynced`, rerun
|
|
`--prompt-operational-secrets` to enter the Pi-hole server and password, then
|
|
run `--initialize-openbao` and `--register-webhook`. Do not create or edit the
|
|
provider Secret directly.
|
|
|
|
## Webhook TLS
|
|
|
|
The public Gateway terminates HTTPS with a cert-manager certificate. The
|
|
Cloudflare API token issues the `nid3.com` certificate and manages explicit
|
|
Tunnel CNAME records. Pi-hole remains the ExternalDNS provider. Check
|
|
certificate readiness with:
|
|
|
|
```powershell
|
|
kubectl -n cert-manager get externalsecret cloudflare-api-token
|
|
kubectl -n gateway-system get certificate webhook-tls
|
|
```
|
|
|
|
Enter the Pi-hole values and Cloudflare API token through
|
|
`--prompt-operational-secrets`, then import the user-approved local credentials
|
|
file before bootstrap:
|
|
|
|
```powershell
|
|
go run . cloudflare-tunnel import --config <private-bootstrap-config> --credentials-file <local-credentials-json>
|
|
```
|
|
|
|
Do not put the Cloudflare token or tunnel credentials in the cluster repository.
|
|
|
|
## Cloudflare Tunnel Routes
|
|
|
|
After importing the credentials, manage one explicit proxied CNAME and ingress
|
|
rule at a time:
|
|
|
|
```powershell
|
|
go run . cloudflare-tunnel route add --config <private-bootstrap-config> --hostname <public-hostname> --service <http-or-https-upstream-url> --zone-id <cloudflare-zone-id>
|
|
go run . cloudflare-tunnel route remove --config <private-bootstrap-config> --hostname <public-hostname> --service <http-or-https-upstream-url> --zone-id <cloudflare-zone-id>
|
|
```
|
|
|
|
Both commands require all three route values, preserve the terminal 404 rule,
|
|
and refuse an existing CNAME that does not point to the managed tunnel.
|
|
|
|
## Internal Platform UIs
|
|
|
|
Pi-hole resolves these HTTPS names to the Cilium Gateway only on the LAN:
|
|
|
|
- `https://grafana.<cluster-domain>/` for Grafana. Authenticate with Grafana.
|
|
- `https://openbao.<cluster-domain>/` for OpenBao. Authenticate with an OpenBao token.
|
|
|
|
Hubble UI is enabled for in-cluster troubleshooting but has no LAN route because
|
|
it does not provide authentication. Add an authenticated proxy before exposing
|
|
it outside the cluster.
|
|
|
|
## Webhook Smoke Test
|
|
|
|
Use Forgejo's hook test endpoint against an existing non-`main` ref. It emits
|
|
a real push delivery, runs the Node build, and pushes a SHA-tagged registry
|
|
image, but skips the main-only staging manifest update:
|
|
|
|
```powershell
|
|
# Discover the hook ID and choose an existing non-main branch or tag.
|
|
Invoke-RestMethod -Headers @{ Authorization = "token $env:FORGEJO_TOKEN" } `
|
|
-Uri "https://<forgejo>/api/v1/repos/<owner>/<repo>/hooks"
|
|
|
|
Invoke-WebRequest -Method Post -Headers @{ Authorization = "token $env:FORGEJO_TOKEN" } `
|
|
-Uri "https://<forgejo>/api/v1/repos/<owner>/<repo>/hooks/<hook-id>/tests?ref=<non-main-ref>"
|
|
```
|
|
|
|
Forgejo returns `204` after accepting the delivery. Confirm the resulting
|
|
PipelineRun instead of treating `204` as a successful build:
|
|
|
|
```powershell
|
|
kubectl -n tekton-pipelines get pipelineruns
|
|
kubectl -n tekton-pipelines describe pipelinerun <name>
|
|
```
|