165 lines
6.1 KiB
Markdown
165 lines
6.1 KiB
Markdown
# Operations Runbook
|
|
|
|
Use this runbook from the MaidnCLI checkout. Bootstrap YAML, age identities,
|
|
OpenBao recovery material, Terraform state, and generated workspaces are
|
|
secret-bearing local inputs. Do not commit or print them.
|
|
|
|
## Normal Reconciliation
|
|
|
|
Run after a merged GitOps migration or to recover ordinary drift:
|
|
|
|
```powershell
|
|
go run . bootstrap --config <private-bootstrap-config> --mode=reconcile
|
|
```
|
|
|
|
This is the only regular lifecycle command. Do not use direct `kubectl apply`,
|
|
`flux reconcile`, Helm upgrades, or mutating `talosctl` commands.
|
|
|
|
Bootstrap records the CICD, manifests, and Talos template commits in the
|
|
workspace's `maidn-template-revisions.yaml` before it creates migration PRs or
|
|
touches infrastructure. Later runs require the same template sources and refs
|
|
and check out those commits, even when a configured branch advances. To accept
|
|
new template revisions intentionally, use a new empty `workspaceDir` (and a
|
|
fresh `cloneParent` when it is configured separately) and keep the prior
|
|
secret-bearing workspace intact for recovery.
|
|
|
|
## Rebuild
|
|
|
|
Use only when an authorized recovery requires recreating the Talos VM:
|
|
|
|
```powershell
|
|
go run . bootstrap --config <private-bootstrap-config> --mode=rebuild --yes
|
|
```
|
|
|
|
The rebuild replaces Terraform-managed Talos VMs only. It does not manage or
|
|
delete TrueNAS datasets or unrelated infrastructure.
|
|
|
|
## OpenBao And Webhooks
|
|
|
|
After a rebuild or an OpenBao restart, refresh Kubernetes auth and reseed the
|
|
encrypted operational values:
|
|
|
|
```powershell
|
|
go run . bootstrap --config <private-bootstrap-config> --initialize-openbao
|
|
```
|
|
|
|
When the Pipeline and EventListener are already Ready, register or update the
|
|
Forgejo webhook without re-running the full lifecycle:
|
|
|
|
```powershell
|
|
go run . bootstrap --config <private-bootstrap-config> --register-webhook
|
|
```
|
|
|
|
The webhook authorization value stays in encrypted operational secrets and is
|
|
never supplied on the command line.
|
|
|
|
If that authorization value is exposed, replace it and reconcile both OpenBao
|
|
and the Forgejo hook in one command:
|
|
|
|
```powershell
|
|
go run . bootstrap --config <private-bootstrap-config> --rotate-webhook-authorization
|
|
```
|
|
|
|
## Read-Only Verification
|
|
|
|
Set `KUBECONFIG` to the generated kubeconfig for the configured cluster, then
|
|
check the control plane and delivery chain:
|
|
|
|
```powershell
|
|
kubectl -n flux-system get kustomizations
|
|
kubectl get clustersecretstores
|
|
kubectl -n flux-system get externalsecrets
|
|
kubectl -n tekton-pipelines get pipelines,eventlisteners,externalsecrets
|
|
```
|
|
|
|
Expected state:
|
|
|
|
- Flux Kustomizations are `READY=True`.
|
|
- `ClusterSecretStore/openbao` is `READY=True`.
|
|
- Forgejo credential ExternalSecrets are `SecretSynced`.
|
|
- The application Pipeline exists and the EventListener is available.
|
|
|
|
If OpenBao authentication is invalid after a rebuild, run
|
|
`--initialize-openbao`, then allow the controllers to retry. Do not recreate
|
|
the ClusterSecretStore or Secrets manually.
|
|
|
|
## External DNS
|
|
|
|
Webhook delivery requires the configured `tekton.<cluster-domain>` hostname to
|
|
resolve through Pi-hole to the Cilium Gateway address. ExternalDNS uses the
|
|
Pi-hole provider with Gateway API routes, an `upsert-only` policy, and no
|
|
ownership registry.
|
|
|
|
```powershell
|
|
kubectl -n external-dns get pods,externalsecrets
|
|
kubectl -n external-dns logs deployment/external-dns --tail=100
|
|
```
|
|
|
|
If `ExternalSecret/pihole-credentials` is not `SecretSynced`, rerun
|
|
`--prompt-operational-secrets` to enter the Pi-hole server and password, then
|
|
run `--initialize-openbao` and `--register-webhook`. Do not create or edit the
|
|
provider Secret directly.
|
|
|
|
## Webhook TLS
|
|
|
|
The public Gateway terminates HTTPS with a cert-manager certificate. The
|
|
Cloudflare API token issues the `nid3.com` certificate and manages the tunnel;
|
|
Pi-hole remains the ExternalDNS provider. Check certificate readiness with:
|
|
|
|
```powershell
|
|
kubectl -n cert-manager get externalsecret cloudflare-api-token
|
|
kubectl -n gateway-system get certificate webhook-tls
|
|
```
|
|
|
|
Enter the Pi-hole values and Cloudflare API token through
|
|
`--prompt-operational-secrets`, then run `bootstrap --mode=reconcile` to create
|
|
the managed tunnel state. Do not put the Cloudflare token or tunnel credentials
|
|
in the cluster repository.
|
|
|
|
## Cloudflare Tunnel Routes
|
|
|
|
After reconcile has created the tunnel, manage one explicit proxied CNAME and
|
|
ingress rule at a time:
|
|
|
|
```powershell
|
|
go run . cloudflare-tunnel route add --config <private-bootstrap-config> --hostname <public-hostname> --service <http-or-https-upstream-url> --zone-id <cloudflare-zone-id>
|
|
go run . cloudflare-tunnel route remove --config <private-bootstrap-config> --hostname <public-hostname> --service <http-or-https-upstream-url> --zone-id <cloudflare-zone-id>
|
|
```
|
|
|
|
Both commands require all three route values, preserve the terminal 404 rule,
|
|
and refuse an existing CNAME that does not point to the managed tunnel.
|
|
|
|
## Internal Platform UIs
|
|
|
|
Pi-hole resolves these HTTPS names to the Cilium Gateway only on the LAN:
|
|
|
|
- `https://grafana.<cluster-domain>/` for Grafana. Authenticate with Grafana.
|
|
- `https://openbao.<cluster-domain>/` for OpenBao. Authenticate with an OpenBao token.
|
|
|
|
Hubble UI is enabled for in-cluster troubleshooting but has no LAN route because
|
|
it does not provide authentication. Add an authenticated proxy before exposing
|
|
it outside the cluster.
|
|
|
|
## Webhook Smoke Test
|
|
|
|
Use Forgejo's hook test endpoint against an existing non-`main` ref. It emits
|
|
a real push delivery, runs the Node build, and pushes a SHA-tagged registry
|
|
image, but skips the main-only staging manifest update:
|
|
|
|
```powershell
|
|
# Discover the hook ID and choose an existing non-main branch or tag.
|
|
Invoke-RestMethod -Headers @{ Authorization = "token $env:FORGEJO_TOKEN" } `
|
|
-Uri "https://<forgejo>/api/v1/repos/<owner>/<repo>/hooks"
|
|
|
|
Invoke-WebRequest -Method Post -Headers @{ Authorization = "token $env:FORGEJO_TOKEN" } `
|
|
-Uri "https://<forgejo>/api/v1/repos/<owner>/<repo>/hooks/<hook-id>/tests?ref=<non-main-ref>"
|
|
```
|
|
|
|
Forgejo returns `204` after accepting the delivery. Confirm the resulting
|
|
PipelineRun instead of treating `204` as a successful build:
|
|
|
|
```powershell
|
|
kubectl -n tekton-pipelines get pipelineruns
|
|
kubectl -n tekton-pipelines describe pipelinerun <name>
|
|
```
|