218 lines
8.5 KiB
Markdown
218 lines
8.5 KiB
Markdown
# Operations Runbook
|
|
|
|
Use this runbook from the MaidnCLI checkout. Bootstrap YAML, age identities,
|
|
OpenBao recovery material, Terraform state, and generated workspaces are
|
|
secret-bearing local inputs. Do not commit or print them.
|
|
|
|
## Normal Reconciliation
|
|
|
|
Run after a merged GitOps migration or to recover ordinary drift:
|
|
|
|
```powershell
|
|
go run . bootstrap --config <private-bootstrap-config> --mode=reconcile
|
|
```
|
|
|
|
This is the only regular lifecycle command. Do not use direct `kubectl apply`,
|
|
`flux reconcile`, Helm upgrades, or mutating `talosctl` commands.
|
|
|
|
Bootstrap records the CICD, manifests, and Talos template commits in the
|
|
workspace's `maidn-template-revisions.yaml` before it creates migration PRs or
|
|
touches infrastructure. Later runs require the same template sources and refs
|
|
and check out those commits, even when a configured branch advances. To accept
|
|
new template revisions intentionally, use a new empty `workspaceDir` (and a
|
|
fresh `cloneParent` when it is configured separately) and keep the prior
|
|
secret-bearing workspace intact for recovery.
|
|
|
|
## Platform Initialization And App Onboarding
|
|
|
|
Initialize the shared delivery platform before onboarding any application. This
|
|
creates the Gateway, Tekton, and Tekton Triggers platform resources using
|
|
`tekton.<cluster-domain>` and the configured Forgejo owner; it does not render
|
|
an application Pipeline or register an application webhook:
|
|
|
|
```powershell
|
|
bootstrap init --config <private-bootstrap-config> --organization <owner> --create-organization
|
|
```
|
|
|
|
Import a clean checkout into the configured owner with a separate command. The
|
|
checkout may originate in another Forgejo organization, but `delivery.appRepoUrl`
|
|
must target `<owner>/<app>.git` and its current branch must be
|
|
`delivery.appRepoRef`:
|
|
|
|
```powershell
|
|
app onboard --config <private-bootstrap-config> --from <clean-checkout>
|
|
```
|
|
|
|
Onboarding copies the current branch to the target repository, replaces only
|
|
the two Maidn-generated `.tekton` files, protects the production branch, and
|
|
auto-merges the app delivery and cluster registration PRs. The registration is
|
|
stored in `base/tekton/apps/<app>.yaml` in the configured cluster repository;
|
|
the generic EventListener dispatches by the Forgejo repository name.
|
|
|
|
`delivery.productionBranch` is the application production branch. A production
|
|
delivery PR targets the configured generic
|
|
`<owner>/<manifests-repo>` repository semantics (for example,
|
|
`test-org-2/<manifests-repo>`), never through a source checkout's former owner.
|
|
|
|
## Rebuild
|
|
|
|
Use only when an authorized recovery requires recreating the Talos VM:
|
|
|
|
```powershell
|
|
go run . bootstrap --config <private-bootstrap-config> --mode=rebuild --yes
|
|
```
|
|
|
|
The rebuild replaces Terraform-managed Talos VMs only. It does not manage or
|
|
delete TrueNAS datasets or unrelated infrastructure.
|
|
|
|
## Planned Cluster Removal
|
|
|
|
There is currently no `maidn cluster remove` or `maidn uninstall` command.
|
|
Do not substitute direct Terraform or Kubernetes deletion for a supported
|
|
workflow. A future remove command must require the named-cluster confirmation,
|
|
limit its Terraform scope to Maidn Talos VMs, and leave external storage and
|
|
shared infrastructure under their respective operator procedures.
|
|
|
|
The command must handle partial removal and retry safely: verify each VM's
|
|
actual Proxmox state, accept already-absent Maidn VMs, report API timeouts as
|
|
incomplete rather than successful removal, and leave a sanitized removal
|
|
record in the isolated workspace. It must not delete shared network bridges,
|
|
TrueNAS datasets, SOPS identities, or unrelated infrastructure. Until that
|
|
command exists, an operator must use the authorized recovery procedure and
|
|
resolve Proxmox API failures before retrying bootstrap.
|
|
|
|
## OpenBao And Webhooks
|
|
|
|
After a rebuild or an OpenBao restart, refresh Kubernetes auth and reseed the
|
|
encrypted operational values:
|
|
|
|
```powershell
|
|
go run . bootstrap --config <private-bootstrap-config> --initialize-openbao
|
|
```
|
|
|
|
When the Pipeline and EventListener are already Ready, register or update the
|
|
Forgejo webhook without re-running the full lifecycle:
|
|
|
|
```powershell
|
|
go run . bootstrap --config <private-bootstrap-config> --register-webhook
|
|
```
|
|
|
|
The webhook authorization value stays in encrypted operational secrets and is
|
|
never supplied on the command line.
|
|
|
|
If that authorization value is exposed, replace it and reconcile both OpenBao
|
|
and the Forgejo hook in one command:
|
|
|
|
```powershell
|
|
go run . bootstrap --config <private-bootstrap-config> --rotate-webhook-authorization
|
|
```
|
|
|
|
## Read-Only Verification
|
|
|
|
Set `KUBECONFIG` to the generated kubeconfig for the configured cluster, then
|
|
check the control plane and delivery chain:
|
|
|
|
```powershell
|
|
kubectl -n flux-system get kustomizations
|
|
kubectl get clustersecretstores
|
|
kubectl -n flux-system get externalsecrets
|
|
kubectl -n tekton-pipelines get pipelines,eventlisteners,externalsecrets
|
|
```
|
|
|
|
Expected state:
|
|
|
|
- Flux Kustomizations are `READY=True`.
|
|
- `ClusterSecretStore/openbao` is `READY=True`.
|
|
- Forgejo credential ExternalSecrets are `SecretSynced`.
|
|
- The application Pipeline exists and the EventListener is available.
|
|
|
|
If OpenBao authentication is invalid after a rebuild, run
|
|
`--initialize-openbao`, then allow the controllers to retry. Do not recreate
|
|
the ClusterSecretStore or Secrets manually.
|
|
|
|
## External DNS
|
|
|
|
Webhook delivery requires the configured `tekton.<cluster-domain>` hostname to
|
|
resolve through Pi-hole to the Cilium Gateway address. ExternalDNS uses the
|
|
Pi-hole provider with Gateway API routes, an `upsert-only` policy, and no
|
|
ownership registry.
|
|
|
|
```powershell
|
|
kubectl -n external-dns get pods,externalsecrets
|
|
kubectl -n external-dns logs deployment/external-dns --tail=100
|
|
```
|
|
|
|
If `ExternalSecret/pihole-credentials` is not `SecretSynced`, rerun
|
|
`--prompt-operational-secrets` to enter the Pi-hole server and password, then
|
|
run `--initialize-openbao` and `--register-webhook`. Do not create or edit the
|
|
provider Secret directly.
|
|
|
|
## Webhook TLS
|
|
|
|
The public Gateway terminates HTTPS with a cert-manager certificate. The
|
|
Cloudflare API token issues the `nid3.com` certificate and manages explicit
|
|
Tunnel CNAME records. Pi-hole remains the ExternalDNS provider. Check
|
|
certificate readiness with:
|
|
|
|
```powershell
|
|
kubectl -n cert-manager get externalsecret cloudflare-api-token
|
|
kubectl -n gateway-system get certificate webhook-tls
|
|
```
|
|
|
|
Enter the Pi-hole values and Cloudflare API token through
|
|
`--prompt-operational-secrets`, then import the user-approved local credentials
|
|
file before bootstrap:
|
|
|
|
```powershell
|
|
go run . cloudflare-tunnel import --config <private-bootstrap-config> --credentials-file <local-credentials-json>
|
|
```
|
|
|
|
Do not put the Cloudflare token or tunnel credentials in the cluster repository.
|
|
|
|
## Cloudflare Tunnel Routes
|
|
|
|
After importing the credentials, manage one explicit proxied CNAME and ingress
|
|
rule at a time:
|
|
|
|
```powershell
|
|
go run . cloudflare-tunnel route add --config <private-bootstrap-config> --hostname <public-hostname> --service <http-or-https-upstream-url> --zone-id <cloudflare-zone-id>
|
|
go run . cloudflare-tunnel route remove --config <private-bootstrap-config> --hostname <public-hostname> --service <http-or-https-upstream-url> --zone-id <cloudflare-zone-id>
|
|
```
|
|
|
|
Both commands require all three route values, preserve the terminal 404 rule,
|
|
and refuse an existing CNAME that does not point to the managed tunnel.
|
|
|
|
## Internal Platform UIs
|
|
|
|
Pi-hole resolves these HTTPS names to the Cilium Gateway only on the LAN:
|
|
|
|
- `https://grafana.<cluster-domain>/` for Grafana. Authenticate with Grafana.
|
|
- `https://openbao.<cluster-domain>/` for OpenBao. Authenticate with an OpenBao token.
|
|
|
|
Hubble UI is enabled for in-cluster troubleshooting but has no LAN route because
|
|
it does not provide authentication. Add an authenticated proxy before exposing
|
|
it outside the cluster.
|
|
|
|
## Webhook Smoke Test
|
|
|
|
Use Forgejo's hook test endpoint against an existing non-`main` ref. It emits
|
|
a real push delivery, runs the Node build, and pushes a SHA-tagged registry
|
|
image, but skips the main-only staging manifest update:
|
|
|
|
```powershell
|
|
# Discover the hook ID and choose an existing non-main branch or tag.
|
|
Invoke-RestMethod -Headers @{ Authorization = "token $env:FORGEJO_TOKEN" } `
|
|
-Uri "https://<forgejo>/api/v1/repos/<owner>/<repo>/hooks"
|
|
|
|
Invoke-WebRequest -Method Post -Headers @{ Authorization = "token $env:FORGEJO_TOKEN" } `
|
|
-Uri "https://<forgejo>/api/v1/repos/<owner>/<repo>/hooks/<hook-id>/tests?ref=<non-main-ref>"
|
|
```
|
|
|
|
Forgejo returns `204` after accepting the delivery. Confirm the resulting
|
|
PipelineRun instead of treating `204` as a successful build:
|
|
|
|
```powershell
|
|
kubectl -n tekton-pipelines get pipelineruns
|
|
kubectl -n tekton-pipelines describe pipelinerun <name>
|
|
```
|