Felix Wolf 1122c3f0e2 feat: Implement S3 to Storage Box backup

Introduces a daily Kubernetes CronJob that utilizes rclone to perform compressed backups of oCIS S3 data to a Hetzner Storage Box via SFTP.

This new backup mechanism requires the manual creation of an 'ocis-storagebox-credentials' secret, which holds the Storage Box host, user, and SSH private key. A check is added to the secret initialization job to ensure this essential external secret exists.

2026-04-06 15:24:14 +02:00

4.3 KiB

Raw Blame History

k8s-and-chill

Project Overview

GitOps-managed Kubernetes cluster on Hetzner Cloud running Talos Linux. Uses myks for Helm chart rendering with ytt overlays, targeting ArgoCD for continuous deployment.

Cluster

3 Talos control-plane nodes (CAX11 ARM64, Hetzner Cloud Nuremberg)
Node IPs: 195.201.219.111, 195.201.140.75, 195.201.219.17
allowSchedulingOnControlPlanes: true (no dedicated workers)
All namespaces use pod-security.kubernetes.io/enforce: privileged

Domain & DNS

Domain: tr1ceracop.de (registered at INWX)
DNS: Managed at INWX with wildcard A record *.tr1ceracop.de pointing to node IPs
Forgejo: https://git.tr1ceracop.de
ArgoCD: https://argocd.tr1ceracop.de

Deployed Applications

App	Namespace	Notes
traefik	traefik	Ingress controller, DaemonSet with hostPort 80/443
cert-manager	cert-manager	Let's Encrypt HTTP-01 via ClusterIssuer `letsencrypt`
forgejo	forgejo	Git server, SQLite, local-path PVC
argocd	argocd	GitOps controller
local-path-provisioner	local-path-storage	Default StorageClass, installed via upstream manifest

myks Structure

prototypes/         # Application templates (helm values + ytt overlays)
  argocd/
  traefik/
  cert-manager/
  forgejo/
envs/
  env-data.ytt.yaml           # Global ArgoCD config
  _env/                       # Shared overlays (annotations, secrets)
  production/
    env-data.ytt.yaml          # App list for production
    _apps/{app}/app-data.ytt.yaml  # Per-app overrides
rendered/
  envs/production/{app}/       # kubectl-ready manifests
  argocd/production/           # ArgoCD Application resources
talos/
  controlplane.yaml            # Talos machine config
  talosconfig                  # Talos client config
  kubeconfig                   # Cluster kubeconfig

Prototype Pattern

Each prototype follows this structure:

app-data.ytt.yaml — namespace declaration
vendir/vendir-data.ytt.yaml — chart name, version, repository URL
vendir/base.ytt.yaml — vendir config template (identical across all)
helm/{chart}.yaml — Helm values overrides
ytt/ns.ytt.yaml — Namespace resource + namespace overlay on all resources

Key Commands

myks render                          # Render all apps
myks render production <app>         # Render single app
kubectl apply -f rendered/envs/production/<app>/ --server-side  # Deploy

Kubeconfig & Talos

KUBECONFIG and TALOSCONFIG are already set in the user's shell environment. Do not set them in commands.

Known Issues / TODOs

Namespace race condition: First kubectl apply of a new app often fails because namespace isn't ready. Re-apply once.
Traefik DaemonSet updates: Requires updateStrategy.rollingUpdate.maxSurge: 0 because hostPort conflicts prevent surge.
Forgejo Ingress API version: Chart renders extensions/v1beta1, fixed via ytt/ingress-fix.ytt.yaml overlay to networking.k8s.io/v1.
ArgoCD: Fully wired to Forgejo via App of Apps. Root Application in default project syncs rendered/argocd/production/. Deploy key provisioned automatically by argocd-deploy-key-init Job in forgejo namespace.

Container Images

Never use bitnami images. Use alpine/k8s or plain alpine for utility Jobs instead.

Secrets

Never commit secrets to git. This is a public repository.
All secrets must be generated in-cluster using init Jobs (ArgoCD PreSync hooks) that create secrets if they don't already exist. See prototypes/ocis/ytt/s3-secret-job.ytt.yaml for the pattern.
External secrets (e.g. S3 credentials) that cannot be generated must be created manually in the cluster before deploying. The init Job should validate their existence and fail fast if missing.
When adding a new application that uses a Helm chart generating secrets, configure all secretRefs to point to pre-created secret names and use an init Job to generate them.
Known external secrets (not in git, created manually):
- ocis/ocis-s3-credentials — Hetzner S3 access key and secret key
- ocis/ocis-storagebox-credentials — Hetzner Storage Box host, user, and SSH private key (for S3 backup to Helsinki)
- cert-manager/letsencrypt-account-key — ACME account key (auto-generated by cert-manager)

4.3 KiB Raw Blame History