- New headless Service matrix-synapse-metrics exposing port 9090
(Synapse's /_synapse/metrics listener), labeled matrix_metrics=enabled
- VictoriaMetrics scrape job 'matrix' targets endpoints in matrix ns
with that label + port name 'metrics'
- Grafana picks up the official Synapse dashboard from
element-hq/synapse v1.152.0 contrib/grafana/synapse.json via URL
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Personal homeserver with bridges deferred. Single host
matrix.{cluster.domain} with path-based routing: /_matrix, /_synapse,
/.well-known/matrix → Synapse; / → Element Web. Both share matrix-tls.
Stack: ananace/matrix-synapse + element-web charts, CNPG postgres
(LC_COLLATE=C), in-cluster alpine redis (no auth, replaces bitnami
subchart), mittwald-generated synapse-secrets for registration_shared/
macaroon/form_secret, custom idempotent signing-key init Job (replaces
chart's bitnami/kubectl publisher).
Sync waves:
-3 Namespace
-2 synapse-secrets (mittwald head-start), signing-key RBAC
-1 signing-key Job, CNPG Cluster, redis
0 Synapse, Element, Ingress
Synapse pod waits in extraCommands until synapse-secrets is populated,
then writes zz-overrides.yaml to override chart placeholders for the
three secret values without churning the chart-managed Secret on every
render. Resources tightened for 1-2 user scale: Synapse 256Mi/512Mi,
Postgres 64Mi/128Mi.
ArgoCD destination.namespace overridden to matrix via prototype-level
argocd overlay so both apps share the matrix ns instead of creating
unused matrix-synapse and element-web namespaces.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extract domain, ingress class, TLS issuer, storage classes, S3 endpoints,
backup toggles, and forgejo node selector into env-data values. Each
prototype's app-data declares its subdomain alongside namespace; templates
compute host as <subdomain>.<cluster.domain>.
Schema is shape-only with safe defaults; production env-data sets values
explicitly. Backup CronJobs and external-secret prechecks gate on
backups.enabled and ocis.s3.external. Adds mkcert ClusterIssuer + precheck
Job for local-dev TLS, gated on cluster.tls.issuer == "mkcert".
forgejo argocd-deploy-key Job: REPO_URL/FORGEJO_URL moved to container env
vars to keep the script ytt-templatable; runtime behavior unchanged.
Production render verified byte-identical (excluding the deploy-key Job
env-var refactor and chart-volatile UUID ConfigMaps).
Implements daily online backups for oCIS persistent volumes.
Each CronJob uses `rclone` to sync its respective PVC to a Storage Box, mounting the volume read-only to ensure zero downtime. Pod affinity is configured to schedule the backup job on the same node as the consuming application pod. This covers `idm`, `storagesystem`, and `storageusers` data volumes.
Adds `fsGroup` to the S3 backup cronjob's security context to ensure proper volume ownership. Increases the SSH key secret's `defaultMode` to grant group read access, resolving permission failures when reading the SSH key.
Integrates oCIS services into the monitoring stack by:
- Adding a new scrape configuration to VictoriaMetrics to collect metrics from oCIS services in the 'ocis' namespace.
- Introducing a new "ocis Overview" Grafana dashboard. This dashboard includes panels for user experience (proxy), service health, storage activity (uploads/downloads), and resource utilization, all leveraging the VictoriaMetrics datasource.
Replace=true alone uses kubectl replace, which rejects updates on Job
immutable fields (spec.selector, spec.template.metadata.labels) when
the cluster already has a Job with auto-generated values. Add Force=true
so ArgoCD does kubectl replace --force (delete + recreate).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Migrate ~180 LOC of openssl/kubectl init Jobs to declarative Secret
manifests reconciled by mittwald/kubernetes-secret-generator (random
strings, SSH keypair) and cert-manager Certificates (RSA private key +
self-signed CA chain). mittwald only fills empty fields, so existing
populated Secrets keep their current values across the migration.
Changes:
- New prototype kubernetes-secret-generator (chart 3.4.1, mittwald helm
repo). Cluster-wide informer reconciler, no webhook -> cold-bootstrap
safe via ArgoCD retries.
- New cert-manager selfsigned ClusterIssuer (in-cluster trust root).
letsencrypt remains for public-DNS endpoints.
- forgejo: admin-secret Job replaced with a mittwald-annotated Secret
(hex-encoded 24-char password). Deploy-key Job split: mittwald
ssh-keypair Secret + slim Job that uploads pubkey to Forgejo and
copies privkey into the argocd repo Secret.
- ocis: 13 Secrets / 16 random fields now mittwald-managed (UUIDs
replaced with opaque random hex; ocis treats user-id as opaque). IDP
RSA signing key, LDAP self-signed CA, and LDAP server cert produced
by cert-manager. Per-Deployment ytt overlay remaps volume key paths
(tls.crt -> ldap-ca.crt, tls.key -> private-key.pem, etc.) since the
ocis chart mounts Secrets raw without items support. Old multi-secret
s3-secret-job replaced with a slim external-secret precheck Job that
only validates pre-created Hetzner S3/Storage Box credentials.
- Application sync-wave -10 on cert-manager and kubernetes-secret-
generator so they install before consumers. ArgoCD selfHeal handles
any residual races.
CLAUDE.md: remove the "all namespaces use privileged PodSecurity"
convention. Existing namespaces still carry the label and will be
audited separately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Increase Traefik readTimeout from 600s to 3600s to prevent connection drops during large uploads, and enable the suspended cleanUpExpiredUploads CronJob so stale TUS sessions are automatically purged.
Increases memory requests for the IDM and NATS services to enhance stability and performance.
Updates application, service account, and storage UUIDs in configuration maps, reflecting a re-initialization or re-rendering of OIDC settings.
Assigns specific CPU and memory requests and limits to the storageusers service to ensure stable operation and efficient resource utilization.
Introduces pod anti-affinity for storageusers to prevent it from being scheduled on the same node as victoria-metrics-single, improving resilience and preventing potential resource contention.
Introduces a daily Kubernetes CronJob that utilizes rclone to perform compressed backups of oCIS S3 data to a Hetzner Storage Box via SFTP.
This new backup mechanism requires the manual creation of an 'ocis-storagebox-credentials' secret, which holds the Storage Box host, user, and SSH private key. A check is added to the secret initialization job to ensure this essential external secret exists.
Removes the full Nextcloud stack (PostgreSQL/CNPG, Valkey, Caddy) and
deploys oCIS at drive.tr1ceracop.de. oCIS is self-contained — no
external database or cache needed.
Key design decisions:
- S3ng storage backend on Hetzner Object Storage (ocis-tr1ceracop)
- Chart fetched via vendir git source (not published to a Helm repo)
- All secrets generated in-cluster via PreSync init Job (never in git)
- Memory requests on all pods to prevent node overcommit
- Persistence on local-path for metadata (idm, nats, search, storage)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Removes the full Nextcloud stack (PostgreSQL/CNPG, Valkey, Caddy sidecar)
and replaces it with oCIS at drive.tr1ceracop.de. oCIS is self-contained
(no external DB/cache needed) with S3ng storage backend on Hetzner Object
Storage (bucket: ocis-tr1ceracop). Chart sourced from git via vendir since
it is not published to a Helm repo.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Deploys Nextcloud using an FPM-alpine image with a Caddy sidecar for web serving.
Integrates with an external CloudNativePG cluster for PostgreSQL and a dedicated Valkey instance for caching. Configures S3-compatible object storage for file data.
Includes an initialization Job to create essential admin and Valkey secrets. Sets up Ingress for external access with automated TLS provisioning via cert-manager.
Configures local-path persistence for Nextcloud's core data to ensure state is maintained across pod restarts. Centralizes hostname configuration and migrates various Nextcloud settings to environment variables for streamlined management.
Adds ArgoCD ignore rules for `batch/Job` resource selectors and template labels, preventing spurious out-of-sync states caused by Kubernetes mutations and improving synchronization stability.
Switch to admin.existingSecret to avoid rendering the admin password
into git. The secret must be created manually in the cluster.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Configures the Victoria Metrics single server to be scheduled on a specific host, `ubuntu-4gb-nbg1-1`. This ensures being scheduled on the same node as the pvc is bound ot since it uses local-path volume
Enables direct scraping of kubelet and cAdvisor metrics from Kubernetes nodes.
This provides more granular insights into node health and container resource utilization.
Configures secure HTTPS scraping using Kubernetes node service discovery.
Enables CPU/memory visibility in k9s and kubectl top by deploying
the Kubernetes metrics-server via the metrics.k8s.io API.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- CNPG Barman backup to Hetzner S3 (s3://k8s-and-chill-backups/forgejo/cnpg/)
- ScheduledBackup CR: daily at 2 AM, 30d retention, prefer-standby
- Git repo rclone sync to S3 (s3://k8s-and-chill-backups/forgejo/git/) via CronJob at 3 AM
- Requires secrets: forgejo-backup-s3 (S3 creds), hcloud-token (not used but created)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds victoria-metrics-single, grafana, kube-state-metrics, and
node-exporter to the cluster. Enables metrics endpoints on traefik,
argocd, and cert-manager for scraping. Grafana available at
grafana.tr1ceracop.de with VictoriaMetrics as default datasource.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Configure myks with global repoURL pointing to Forgejo, in-cluster
destination, and disabled placeholder cluster Secret. Implement App of
Apps pattern with a root Application that syncs all child apps.
Add argocd-deploy-key-init Job that generates an ed25519 SSH keypair,
registers it as a deploy key via Forgejo API, and creates the ArgoCD
repository secret with insecure host key verification (avoids
chicken-and-egg with ArgoCD managing its own known hosts ConfigMap).
Additional changes:
- Ignore /status field diffs globally (K8s 1.32 compat)
- Add Replace=true sync option on Jobs (immutable resource compat)
- Switch job images from bitnami/kubectl to alpine/k8s
- Update CLAUDE.md with ArgoCD status and no-bitnami rule
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use hostPort instead of NodePort for SSH access to avoid cross-node
asymmetric routing issues with kube-proxy nftables mode. Pin Forgejo
pod to node 3 (DNS target) and use port 222 to bypass ISP port 22
blocking.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Configures `myks` for Helm chart rendering with `ytt` overlays to manage cluster applications.
Defines prototypes and environment-specific configurations for core applications including ArgoCD, Traefik, Cert-Manager, and Forgejo.
Adds comprehensive documentation covering cluster setup, GitOps structure, and development environment.
Integrates `direnv` for environment variable management, `gitignore` for file exclusion, and `sops` for secret encryption.
Includes rendered Kubernetes manifests and ArgoCD application resources for initial deployment.
