1122c3f0e2
Introduces a daily Kubernetes CronJob that utilizes rclone to perform compressed backups of oCIS S3 data to a Hetzner Storage Box via SFTP. This new backup mechanism requires the manual creation of an 'ocis-storagebox-credentials' secret, which holds the Storage Box host, user, and SSH private key. A check is added to the secret initialization job to ensure this essential external secret exists.
86 lines
4.3 KiB
Markdown
86 lines
4.3 KiB
Markdown
# k8s-and-chill
|
|
|
|
## Project Overview
|
|
GitOps-managed Kubernetes cluster on Hetzner Cloud running Talos Linux. Uses [myks](https://github.com/mykso/myks) for Helm chart rendering with ytt overlays, targeting ArgoCD for continuous deployment.
|
|
|
|
## Cluster
|
|
- **3 Talos control-plane nodes** (CAX11 ARM64, Hetzner Cloud Nuremberg)
|
|
- Node IPs: `195.201.219.111`, `195.201.140.75`, `195.201.219.17`
|
|
- `allowSchedulingOnControlPlanes: true` (no dedicated workers)
|
|
- All namespaces use `pod-security.kubernetes.io/enforce: privileged`
|
|
|
|
## Domain & DNS
|
|
- **Domain**: `tr1ceracop.de` (registered at INWX)
|
|
- **DNS**: Managed at INWX with wildcard A record `*.tr1ceracop.de` pointing to node IPs
|
|
- **Forgejo**: `https://git.tr1ceracop.de`
|
|
- **ArgoCD**: `https://argocd.tr1ceracop.de`
|
|
|
|
## Deployed Applications
|
|
| App | Namespace | Notes |
|
|
|-----|-----------|-------|
|
|
| traefik | traefik | Ingress controller, DaemonSet with hostPort 80/443 |
|
|
| cert-manager | cert-manager | Let's Encrypt HTTP-01 via ClusterIssuer `letsencrypt` |
|
|
| forgejo | forgejo | Git server, SQLite, local-path PVC |
|
|
| argocd | argocd | GitOps controller |
|
|
| local-path-provisioner | local-path-storage | Default StorageClass, installed via upstream manifest |
|
|
|
|
## myks Structure
|
|
```
|
|
prototypes/ # Application templates (helm values + ytt overlays)
|
|
argocd/
|
|
traefik/
|
|
cert-manager/
|
|
forgejo/
|
|
envs/
|
|
env-data.ytt.yaml # Global ArgoCD config
|
|
_env/ # Shared overlays (annotations, secrets)
|
|
production/
|
|
env-data.ytt.yaml # App list for production
|
|
_apps/{app}/app-data.ytt.yaml # Per-app overrides
|
|
rendered/
|
|
envs/production/{app}/ # kubectl-ready manifests
|
|
argocd/production/ # ArgoCD Application resources
|
|
talos/
|
|
controlplane.yaml # Talos machine config
|
|
talosconfig # Talos client config
|
|
kubeconfig # Cluster kubeconfig
|
|
```
|
|
|
|
### Prototype Pattern
|
|
Each prototype follows this structure:
|
|
- `app-data.ytt.yaml` — namespace declaration
|
|
- `vendir/vendir-data.ytt.yaml` — chart name, version, repository URL
|
|
- `vendir/base.ytt.yaml` — vendir config template (identical across all)
|
|
- `helm/{chart}.yaml` — Helm values overrides
|
|
- `ytt/ns.ytt.yaml` — Namespace resource + namespace overlay on all resources
|
|
|
|
### Key Commands
|
|
```bash
|
|
myks render # Render all apps
|
|
myks render production <app> # Render single app
|
|
kubectl apply -f rendered/envs/production/<app>/ --server-side # Deploy
|
|
```
|
|
|
|
## Kubeconfig & Talos
|
|
- `KUBECONFIG` and `TALOSCONFIG` are already set in the user's shell environment. Do not set them in commands.
|
|
|
|
|
|
## Known Issues / TODOs
|
|
- **Namespace race condition**: First `kubectl apply` of a new app often fails because namespace isn't ready. Re-apply once.
|
|
- **Traefik DaemonSet updates**: Requires `updateStrategy.rollingUpdate.maxSurge: 0` because hostPort conflicts prevent surge.
|
|
- **Forgejo Ingress API version**: Chart renders `extensions/v1beta1`, fixed via `ytt/ingress-fix.ytt.yaml` overlay to `networking.k8s.io/v1`.
|
|
- **ArgoCD**: Fully wired to Forgejo via App of Apps. Root Application in `default` project syncs `rendered/argocd/production/`. Deploy key provisioned automatically by `argocd-deploy-key-init` Job in forgejo namespace.
|
|
|
|
## Container Images
|
|
- **Never use bitnami images.** Use `alpine/k8s` or plain `alpine` for utility Jobs instead.
|
|
|
|
## Secrets
|
|
- **Never commit secrets to git.** This is a public repository.
|
|
- **All secrets must be generated in-cluster** using init Jobs (ArgoCD PreSync hooks) that create secrets if they don't already exist. See `prototypes/ocis/ytt/s3-secret-job.ytt.yaml` for the pattern.
|
|
- **External secrets** (e.g. S3 credentials) that cannot be generated must be created manually in the cluster before deploying. The init Job should validate their existence and fail fast if missing.
|
|
- When adding a new application that uses a Helm chart generating secrets, configure all `secretRefs` to point to pre-created secret names and use an init Job to generate them.
|
|
- Known external secrets (not in git, created manually):
|
|
- `ocis/ocis-s3-credentials` — Hetzner S3 access key and secret key
|
|
- `ocis/ocis-storagebox-credentials` — Hetzner Storage Box host, user, and SSH private key (for S3 backup to Helsinki)
|
|
- `cert-manager/letsencrypt-account-key` — ACME account key (auto-generated by cert-manager)
|