# Talos maintenance

## Regular checks

- `talosctl --talosconfig=./generated/clusterconfig/talosconfig health`
- `kubectl get nodes`
- `flux get all -A`
- `talosctl -n <node-ip> version`

## Upgrade flow

1. Update `talos_factory_schematic_id`, `talos_version`, and `talos_iso_file` in Terraform inputs.
2. If `talos_image_update_mode = "download"`, run `terraform apply` to stage the ISO on every target Proxmox node.
3. Regenerate machine config from `generated/talconfig.yaml`:
   - `talhelper genconfig`
4. Upgrade one control plane node at a time:
   - `talosctl upgrade --nodes <ip> --image factory.talos.dev/installer/<schematic>/<version>`
   - wait for `talosctl health`
5. Upgrade workers one at a time.
6. Refresh kubeconfig if needed:
   - `talosctl kubeconfig --talosconfig=./generated/clusterconfig/talosconfig --nodes=<control-plane-ip> .`

## Certificate rotation

- Check cert expiry:
  - `talosctl -n <control-plane-ip> get kubeapicerts,k8saggregatorca,etcdcerts`
- Rotate before expiry during a maintenance window:
  - `talosctl -n <control-plane-ip> rotate-ca`
  - `talosctl -n <control-plane-ip> rotate-certs`
- After rotation, verify API and kubelet health on all nodes.

## Disaster recovery basics

- Keep `generated/talsecret.sops.yaml` encrypted and backed up.
- Keep the rendered `generated/clusterconfig/talosconfig` in a secure backup.
- Capture an etcd snapshot before major upgrades:
  - `talosctl -n <control-plane-ip> etcd snapshot ./etcd.snapshot`

## What to automate next

- PR automation that bumps Talos version + factory schematic outputs together.
- Scheduled `talosctl health` and cert expiry checks.
- Optional automated image staging to all Proxmox nodes before upgrades.
- Optional upgrade orchestration that rolls one node at a time with health gates.