Integrates oCIS services into the monitoring stack by:
- Adding a new scrape configuration to VictoriaMetrics to collect metrics from oCIS services in the 'ocis' namespace.
- Introducing a new "ocis Overview" Grafana dashboard. This dashboard includes panels for user experience (proxy), service health, storage activity (uploads/downloads), and resource utilization, all leveraging the VictoriaMetrics datasource.
Replace=true alone uses kubectl replace, which rejects updates on Job
immutable fields (spec.selector, spec.template.metadata.labels) when
the cluster already has a Job with auto-generated values. Add Force=true
so ArgoCD does kubectl replace --force (delete + recreate).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Migrate ~180 LOC of openssl/kubectl init Jobs to declarative Secret
manifests reconciled by mittwald/kubernetes-secret-generator (random
strings, SSH keypair) and cert-manager Certificates (RSA private key +
self-signed CA chain). mittwald only fills empty fields, so existing
populated Secrets keep their current values across the migration.
Changes:
- New prototype kubernetes-secret-generator (chart 3.4.1, mittwald helm
repo). Cluster-wide informer reconciler, no webhook -> cold-bootstrap
safe via ArgoCD retries.
- New cert-manager selfsigned ClusterIssuer (in-cluster trust root).
letsencrypt remains for public-DNS endpoints.
- forgejo: admin-secret Job replaced with a mittwald-annotated Secret
(hex-encoded 24-char password). Deploy-key Job split: mittwald
ssh-keypair Secret + slim Job that uploads pubkey to Forgejo and
copies privkey into the argocd repo Secret.
- ocis: 13 Secrets / 16 random fields now mittwald-managed (UUIDs
replaced with opaque random hex; ocis treats user-id as opaque). IDP
RSA signing key, LDAP self-signed CA, and LDAP server cert produced
by cert-manager. Per-Deployment ytt overlay remaps volume key paths
(tls.crt -> ldap-ca.crt, tls.key -> private-key.pem, etc.) since the
ocis chart mounts Secrets raw without items support. Old multi-secret
s3-secret-job replaced with a slim external-secret precheck Job that
only validates pre-created Hetzner S3/Storage Box credentials.
- Application sync-wave -10 on cert-manager and kubernetes-secret-
generator so they install before consumers. ArgoCD selfHeal handles
any residual races.
CLAUDE.md: remove the "all namespaces use privileged PodSecurity"
convention. Existing namespaces still carry the label and will be
audited separately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Increase Traefik readTimeout from 600s to 3600s to prevent connection drops during large uploads, and enable the suspended cleanUpExpiredUploads CronJob so stale TUS sessions are automatically purged.
Configure 2GiB swap volume on system disk with LimitedSwap behavior
and zswap compression (20% max pool) to improve memory utilization
on the CAX11 nodes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Increases memory requests for the IDM and NATS services to enhance stability and performance.
Updates application, service account, and storage UUIDs in configuration maps, reflecting a re-initialization or re-rendering of OIDC settings.
Assigns specific CPU and memory requests and limits to the storageusers service to ensure stable operation and efficient resource utilization.
Introduces pod anti-affinity for storageusers to prevent it from being scheduled on the same node as victoria-metrics-single, improving resilience and preventing potential resource contention.
Introduces a daily Kubernetes CronJob that utilizes rclone to perform compressed backups of oCIS S3 data to a Hetzner Storage Box via SFTP.
This new backup mechanism requires the manual creation of an 'ocis-storagebox-credentials' secret, which holds the Storage Box host, user, and SSH private key. A check is added to the secret initialization job to ensure this essential external secret exists.
Removes the full Nextcloud stack (PostgreSQL/CNPG, Valkey, Caddy) and
deploys oCIS at drive.tr1ceracop.de. oCIS is self-contained — no
external database or cache needed.
Key design decisions:
- S3ng storage backend on Hetzner Object Storage (ocis-tr1ceracop)
- Chart fetched via vendir git source (not published to a Helm repo)
- All secrets generated in-cluster via PreSync init Job (never in git)
- Memory requests on all pods to prevent node overcommit
- Persistence on local-path for metadata (idm, nats, search, storage)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Removes the full Nextcloud stack (PostgreSQL/CNPG, Valkey, Caddy sidecar)
and replaces it with oCIS at drive.tr1ceracop.de. oCIS is self-contained
(no external DB/cache needed) with S3ng storage backend on Hetzner Object
Storage (bucket: ocis-tr1ceracop). Chart sourced from git via vendir since
it is not published to a Helm repo.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Deploys Nextcloud using an FPM-alpine image with a Caddy sidecar for web serving.
Integrates with an external CloudNativePG cluster for PostgreSQL and a dedicated Valkey instance for caching. Configures S3-compatible object storage for file data.
Includes an initialization Job to create essential admin and Valkey secrets. Sets up Ingress for external access with automated TLS provisioning via cert-manager.
Configures local-path persistence for Nextcloud's core data to ensure state is maintained across pod restarts. Centralizes hostname configuration and migrates various Nextcloud settings to environment variables for streamlined management.
Adds ArgoCD ignore rules for `batch/Job` resource selectors and template labels, preventing spurious out-of-sync states caused by Kubernetes mutations and improving synchronization stability.
Provides a comprehensive overview of the Nextcloud service setup. Explains data residency, technical architecture for reliability, data safety guarantees with multi-region backups, and the specific technologies utilized.
Also details infrastructure costs, privacy considerations, and recovery plans for different incident types to ensure user data integrity and availability.
Switch to admin.existingSecret to avoid rendering the admin password
into git. The secret must be created manually in the cluster.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Configures the Victoria Metrics single server to be scheduled on a specific host, `ubuntu-4gb-nbg1-1`. This ensures being scheduled on the same node as the pvc is bound ot since it uses local-path volume
Enables direct scraping of kubelet and cAdvisor metrics from Kubernetes nodes.
This provides more granular insights into node health and container resource utilization.
Configures secure HTTPS scraping using Kubernetes node service discovery.
Enables CPU/memory visibility in k9s and kubectl top by deploying
the Kubernetes metrics-server via the metrics.k8s.io API.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- CNPG Barman backup to Hetzner S3 (s3://k8s-and-chill-backups/forgejo/cnpg/)
- ScheduledBackup CR: daily at 2 AM, 30d retention, prefer-standby
- Git repo rclone sync to S3 (s3://k8s-and-chill-backups/forgejo/git/) via CronJob at 3 AM
- Requires secrets: forgejo-backup-s3 (S3 creds), hcloud-token (not used but created)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds victoria-metrics-single, grafana, kube-state-metrics, and
node-exporter to the cluster. Enables metrics endpoints on traefik,
argocd, and cert-manager for scraping. Grafana available at
grafana.tr1ceracop.de with VictoriaMetrics as default datasource.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Configure myks with global repoURL pointing to Forgejo, in-cluster
destination, and disabled placeholder cluster Secret. Implement App of
Apps pattern with a root Application that syncs all child apps.
Add argocd-deploy-key-init Job that generates an ed25519 SSH keypair,
registers it as a deploy key via Forgejo API, and creates the ArgoCD
repository secret with insecure host key verification (avoids
chicken-and-egg with ArgoCD managing its own known hosts ConfigMap).
Additional changes:
- Ignore /status field diffs globally (K8s 1.32 compat)
- Add Replace=true sync option on Jobs (immutable resource compat)
- Switch job images from bitnami/kubectl to alpine/k8s
- Update CLAUDE.md with ArgoCD status and no-bitnami rule
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use hostPort instead of NodePort for SSH access to avoid cross-node
asymmetric routing issues with kube-proxy nftables mode. Pin Forgejo
pod to node 3 (DNS target) and use port 222 to bypass ISP port 22
blocking.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Configures `myks` for Helm chart rendering with `ytt` overlays to manage cluster applications.
Defines prototypes and environment-specific configurations for core applications including ArgoCD, Traefik, Cert-Manager, and Forgejo.
Adds comprehensive documentation covering cluster setup, GitOps structure, and development environment.
Integrates `direnv` for environment variable management, `gitignore` for file exclusion, and `sops` for secret encryption.
Includes rendered Kubernetes manifests and ArgoCD application resources for initial deployment.
