diff --git a/RUNBOOK-CEPH.fa.md b/RUNBOOK-CEPH.fa.md new file mode 100644 index 0000000..37b4cd0 --- /dev/null +++ b/RUNBOOK-CEPH.fa.md @@ -0,0 +1,329 @@ +# راهنمای Ceph برای ابربان (Rook) + +این سند نصب، معماری واقعی روی کلاستر **abr**، مدیریت روزمره و عیب‌یابی **Rook-Ceph** را پوشش می‌دهد. + +- چارت و اسکریپت‌ها: [`backend/helm/cloudhost-ceph/`](backend/helm/cloudhost-ceph/) +- README انگلیسی: [`backend/helm/cloudhost-ceph/README.md`](backend/helm/cloudhost-ceph/README.md) +- رجیستری: [`RUNBOOK-HARBOR.fa.md`](RUNBOOK-HARBOR.fa.md) + +--- + +## چرا Ceph؟ + +| نیاز | راه‌حل Ceph | +|------|-------------| +| PVC اپ/دیتابیس با **resize** | Block pool → StorageClass `rook-ceph-block` | +| آپلود **zip** سورس کاربر | Object store (RGW) → StorageClass `rook-ceph-bucket` | + +یک کلاستر Ceph هر دو را پوشش می‌دهد؛ zip را روی PVC نگه ندارید — از **bucket** استفاده کنید. + +--- + +## معماری روی abr (single-node) + +``` +registry.abrban.com + ├── rook/ceph:v1.20.1 → Rook operator + └── proxy-dockerhub/ceph/ceph:v19.2 → Ceph daemon (Squid) + +Node abr + ├── /dev/loop6 (15Gi) → OSD (bluestore raw) + ├── mon-a, mgr-a, osd-0, rgw → rook-ceph namespace + └── RGW: rook-ceph-rgw-ceph-objectstore.rook-ceph.svc:80 +``` + +| محدودیت | توضیح | +|---------|--------| +| **۱ OSD** | replication=1؛ بدون HA | +| **loop device** | دیسک خام نداریم؛ `/dev/loop6` از فایل `osd-loopback.img` | +| **HEALTH_WARN** | طبیعی: `OSD count 1 < default size 3`، mon low space | +| **ایمیج‌ها** | باید از قبل در Harbor mirror شده باشند (kubelet به docker.io دسترسی ندارد) | + +--- + +## پیش‌نیازها + +### پروفایل `single-node` (کلاستر فعلی abr) + +- Kubernetes 1.28+ (k3s + Traefik) +- Harbor بالا و ایمیج‌های `rook/ceph` + `ceph/ceph` mirror شده +- حداقل **۱۵ گیگ** فضا برای loop OSD (`/var/lib/rook/osd-loopback.img`) +- `helm` 3.x و `kubectl` با دسترسی cluster-admin +- Secret `registry-pull-secret` در `rook-ceph` با `harbor_registry_user` + +### پروفایل `multi-node` (production) + +- حداقل **۳ نود** + دیسک خام (raw) +- فایل values: `values-rook-cluster-multi-node.yaml` + +--- + +## نصب (اولین بار — abr) + +### ۱. آماده‌سازی loop device برای OSD + +روی نود تک‌دیسک، Rook به دیسک خام نیاز دارد. یک loop device بسازید: + +```bash +# روی نود (یا Job privileged — یک‌بار) +truncate -s 15G /var/lib/rook/osd-loopback.img +losetup --find --show /var/lib/rook/osd-loopback.img # → /dev/loop6 +``` + +### ۲. فعال‌سازی loop در Rook operator + +```bash +kubectl -n rook-ceph patch cm rook-ceph-operator-config --type merge \ + -p '{"data":{"ROOK_CEPH_ALLOW_LOOP_DEVICES":"true"}}' +kubectl -n rook-ceph rollout restart deploy/rook-ceph-operator +``` + +### ۳. mirror ایمیج‌ها (قبل از نصب cluster) + +| ایمیج | مسیر pull | +|-------|-----------| +| `rook/ceph:v1.20.1` | `registry.abrban.com/rook/ceph:v1.20.1` | +| `quay.io/ceph/ceph:v19.2` | `registry.abrban.com/proxy-dockerhub/ceph/ceph:v19.2` | + +جزئیات mirror: [`RUNBOOK-HARBOR.fa.md`](RUNBOOK-HARBOR.fa.md) + +### ۴. نصب operator + +```bash +helm repo add rook-release https://charts.rook.io/release +helm repo update rook-release + +helm upgrade --install rook-ceph rook-release/rook-ceph \ + -n rook-ceph --create-namespace \ + --set image.repository=registry.abrban.com/rook/ceph \ + --set image.tag=v1.20.1 \ + --set imagePullSecrets[0].name=registry-pull-secret +``` + +### ۵. نصب cluster + +```bash +cd backend/helm/cloudhost-ceph + +helm upgrade --install rook-ceph-cluster rook-release/rook-ceph-cluster \ + -n rook-ceph \ + -f values-rook-cluster-single-node.yaml \ + --set cephClusterSpec.cephVersion.image=registry.abrban.com/proxy-dockerhub/ceph/ceph:v19.2 +``` + +> **توجه:** `values-rook-cluster-single-node.yaml` از `devices: [{name: "/dev/loop6"}]` استفاده می‌کند (نه directory — در Rook v1.20 حذف شده). + +### ۶. extras (bucket + secret) + +```bash +kubectl create namespace cloudhost-builds 2>/dev/null || true +helm upgrade --install cloudhost-ceph . \ + -n cloudhost-builds -f values.yaml --no-hooks +``` + +اگر Job `bucket-sync` به‌خاطر `bitnami/kubectl` گیر کرد، secret را دستی بسازید: + +```bash +kubectl -n cloudhost-builds get secret app-sources -o yaml +kubectl -n cloudhost-builds get cm app-sources -o yaml # BUCKET_NAME +# → secret ceph-app-sources-credentials (کلیدهای SOURCE_STORAGE_*) +``` + +### ۷. یکپارچه‌سازی backend + +```bash +kubectl -n cloudhost set env deploy/cloudhost-backend \ + PLATFORM_STORAGE_CLASS=rook-ceph-block \ + PLATFORM_CREATE_STORAGE_CLASS=false \ + PLATFORM_STORAGE_PROVISIONER=rook-ceph.rbd.csi.ceph.com + +kubectl -n cloudhost patch deploy cloudhost-backend --type=json \ + -p '[{"op":"add","path":"/spec/template/spec/containers/0/envFrom","value":[{"secretRef":{"name":"ceph-app-sources-credentials"}}]}]' +``` + +--- + +## اسکریپت `install.sh` (نصب تمیز) + +برای نصب از صفر (بعد از آماده‌سازی loop + mirror): + +```bash +cd backend/helm/cloudhost-ceph +./scripts/install.sh single-node +./scripts/verify.sh +``` + +اسکریپت operator + cluster + extras را نصب می‌کند. روی abr حتماً **قبلش** loop device و mirror ایمیج را انجام دهید. + +--- + +## خروجی‌های مهم + +### StorageClassها + +| نام | کاربرد | +|-----|--------| +| `rook-ceph-block` | PVC اپ، DB، Redis، … | +| `rook-ceph-bucket` | claim کردن bucket برای zip | + +### Secret پلتفرم + +```bash +kubectl -n cloudhost-builds get secret ceph-app-sources-credentials -o yaml +kubectl -n cloudhost get secret ceph-app-sources-credentials -o yaml # کپی در cloudhost +``` + +کلیدها: `SOURCE_STORAGE_ENDPOINT`, `SOURCE_STORAGE_BUCKET`, `SOURCE_STORAGE_ACCESS_KEY`, `SOURCE_STORAGE_SECRET_KEY` + +### RGW endpoint + +``` +http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local:80 +``` + +--- + +## مدیریت روزمره + +### سلامت کلاستر + +```bash +kubectl -n rook-ceph get cephcluster +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph status +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd tree +kubectl get sc | grep rook-ceph +kubectl -n rook-ceph get pods +``` + +### Dashboard + +```bash +kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath='{.data.password}' | base64 -d +kubectl -n rook-ceph port-forward svc/rook-ceph-mgr-dashboard 8443:8443 +# https://localhost:8443 +``` + +### bucket و OBC + +```bash +kubectl -n cloudhost-builds get obc app-sources +kubectl -n cloudhost-builds get cm app-sources +``` + +### PVC جدید با Ceph + +```yaml +storageClassName: rook-ceph-block +``` + +فقط **اپ‌های جدید** (یا بعد از migration) از این StorageClass استفاده می‌کنند. PVCهای قدیمی روی `local-path` / `cloudhost-expandable` خودکار منتقل نمی‌شوند. + +--- + +## راه‌اندازی مجدد (reinstall) + +### ۱. حذف Helm + +```bash +cd backend/helm/cloudhost-ceph +./scripts/uninstall.sh +# تایپ: delete-ceph +``` + +### ۲. پاک‌سازی روی نود + +```bash +sudo losetup -d /dev/loop6 2>/dev/null || true +sudo rm -f /var/lib/rook/osd-loopback.img +sudo rm -rf /var/lib/rook +``` + +### ۳. نصب مجدد + +loop device + mirror + `./scripts/install.sh single-node` + +--- + +## عیب‌یابی + +### CephCluster در `Progressing` / Detecting version + +```bash +kubectl -n rook-ceph get pods | grep detect-version +kubectl -n rook-ceph describe pod -l job-name=rook-ceph-detect-version +``` + +| خطا | راه‌حل | +|-----|--------| +| `ceph/ceph:v19.2 not found` | mirror از quay.io؛ tag صحیح `v19.2` نه `v19.2.1` | +| pull timeout | اولین pull بزرگ است (~500MB)؛ صبر یا image را از قبل روی نود بکشید | +| Job `detect-version` Terminating گیر کرد | `kubectl -n rook-ceph delete job rook-ceph-detect-version --force --grace-period=0` | + +### OSD بالا نمی‌آید (OSD count 0) + +```bash +kubectl -n rook-ceph logs -l app=rook-ceph-osd-prepare --tail=50 +``` + +| خطا | راه‌حل | +|-----|--------| +| `unsupported diskType loop` | `ROOK_CEPH_ALLOW_LOOP_DEVICES=true` | +| `not picked by deviceFilter` | از `devices: [{name: "/dev/loop6"}]` استفاده کنید نه `deviceFilter` | +| `no devices matched` | `losetup -a` روی نود؛ loop6 وجود دارد؟ | +| `directories` در values | در Rook v1.20 کار نمی‌کند — loop یا raw disk | + +### Volume mount روی rook-ceph-tools + +`rook-ceph-mon-endpoints` و `rook-ceph-mon` تا قبل از بالا آمدن mon ساخته نمی‌شوند — طبیعی است؛ بعد از Ready برطرف می‌شود. + +### Helm timeout روی apiserver + +اگر `failed to download openapi` دیدید، بدون `--wait` نصب کنید و با `kubectl get cephcluster` پیگیری کنید. + +### resize PVC + +```bash +kubectl get storageclass rook-ceph-block -o yaml | grep allowVolumeExpansion +kubectl patch pvc -n --type merge \ + -p '{"spec":{"resources":{"requests":{"storage":"5Gi"}}}}' +``` + +--- + +## ارتقا (upgrade) + +```bash +helm repo update rook-release + +helm upgrade rook-ceph rook-release/rook-ceph -n rook-ceph \ + --reuse-values --timeout 10m + +helm upgrade rook-ceph-cluster rook-release/rook-ceph-cluster \ + -n rook-ceph \ + -f values-rook-cluster-single-node.yaml \ + --set cephClusterSpec.cephVersion.image=registry.abrban.com/proxy-dockerhub/ceph/ceph:v19.2 + +helm upgrade cloudhost-ceph . -n cloudhost-builds -f values.yaml +``` + +قبل از upgrade در production: [Rook upgrade guide](https://rook.io/docs/rook/latest/Upgrade/ceph-upgrade/) و snapshot. + +--- + +## نکات امنیتی + +- RGW داخل کلاستر HTTP است — برای دسترسی خارجی ingress + TLS اضافه کنید. +- Secret `ceph-app-sources-credentials` را فقط به backend بدهید. +- `single-node` + ۱ OSD فقط staging است؛ production نیاز به ۳+ نود و دیسک جدا دارد. + +--- + +## چک‌لیست بعد از نصب + +- [ ] `ceph status` — mon/mgr/osd/rgw up +- [ ] `rook-ceph-block` و `rook-ceph-bucket` در `kubectl get sc` +- [ ] `ceph-app-sources-credentials` در `cloudhost-builds` و `cloudhost` +- [ ] env بک‌اند: `PLATFORM_STORAGE_CLASS=rook-ceph-block` +- [ ] `SOURCE_STORAGE_*` در backend از secret خوانده می‌شود +- [ ] اپ تست با PVC جدید deploy شده +- [ ] ایمیج‌های Rook در Harbor موجود و pull تست شده diff --git a/RUNBOOK-HARBOR.fa.md b/RUNBOOK-HARBOR.fa.md new file mode 100644 index 0000000..d15ee9e --- /dev/null +++ b/RUNBOOK-HARBOR.fa.md @@ -0,0 +1,195 @@ +# راهنمای Harbor — `registry.abrban.com` + +این سند معماری فعلی، نصب، مدیریت روزمره و عیب‌یابی **Harbor** روی کلاستر abr را پوشش می‌دهد. + +- چارت/values: [`backend/helm/cloudhost-harbor/`](backend/helm/cloudhost-harbor/) +- اسکریپت نصب: [`backend/helm/cloudhost-harbor/scripts/install-harbor-registry.sh`](backend/helm/cloudhost-harbor/scripts/install-harbor-registry.sh) + +--- + +## معماری فعلی (خلاصه) + +``` +registry.abrban.com (Traefik + TLS wildcard) +├── / → harbor-portal (UI) +├── /api/, /service/, /c/ → harbor-core (API + auth) +├── /v2/proxy-dockerhub/ → harbor-registry (ایمیج‌های mirror‌شده Ceph/Rook) +├── /v2/rook/ → harbor-registry +└── /v2/* → registry قدیمی (ایمیج‌های platform: backend, nixpacks, …) +``` + +| کامپوننت | نقش | +|----------|-----| +| **Harbor** | UI، proxy-cache، ذخیره ایمیج‌های جدید | +| **registry قدیمی** (`Deployment/registry`) | هنوز بالاست؛ ایمیج‌های platform قبل از Harbor اینجاست | +| **registry-egress-proxy** | secret با `HTTP_PROXY` / `HTTPS_PROXY` برای pull از docker.io/quay از داخل کلاستر | +| **registry-pull-secret** | auth kubelet برای pull از `registry.abrban.com` | + +> Harbor و registry قدیمی **هم‌زمان** روی یک hostname هستند؛ مسیر `/v2/` با Ingress split می‌شود. + +--- + +## پیش‌نیازها + +- Secret `abrban-wildcard-tls` در namespace `cloudhost` +- Secret `registry-egress-proxy` در namespace `cloudhost` (پروکسی egress) +- Helm repo: `helm repo add harbor https://helm.goharbor.io` + +--- + +## نصب / ارتقا + +```bash +cd backend/helm/cloudhost-harbor +./scripts/install-harbor-registry.sh +``` + +اسکریپت: +1. Harbor را با values + proxy از `registry-egress-proxy` نصب می‌کند +2. Ingress قدیمی `registry` را حذف می‌کند (بعد از نصب باید دستی دوباره route شود — بخش Ingress) +3. `Deployment/registry` را scale به 0 می‌کند (برای rollback نگه داشته می‌شود) + +### Ingress بعد از نصب (الزامی) + +Harbor به **چند مسیر** نیاز دارد. Ingress نهایی باید شبیه این باشد: + +| Path | Service | Port | +|------|---------|------| +| `/` | `harbor-portal` | 80 | +| `/api/`, `/service/`, `/c/`, `/chartrepo/` | `harbor-core` | 80 | +| `/v2/proxy-dockerhub/`, `/v2/rook/` | `harbor-registry` | 5000 | +| `/v2/` (بقیه) | `registry` (قدیمی) | 5000 | + +بدون split روی `/v2/`، یا UI 404 می‌دهد یا kubelet ایمیج platform را پیدا نمی‌کند. + +--- + +## دسترسی و credentialها + +| کاربرد | کاربر | منبع | +|--------|-------|------| +| UI / API مدیریت | `admin` | `kubectl -n cloudhost get secret harbor-core -o jsonpath='{.data.HARBOR_ADMIN_PASSWORD}' \| base64 -d` | +| push/pull داخلی به harbor-registry | `harbor_registry_user` | secret `harbor-core` → `REGISTRY_CREDENTIAL_PASSWORD` | +| pull kubelet (ایمیج‌های platform) | `cloudhost` | secret `registry-pull-secret` (namespace `cloudhost`) | +| pull kubelet (ایمیج‌های Rook/Ceph) | `harbor_registry_user` | secret `registry-pull-secret` (namespace `rook-ceph`) | + +URL: https://registry.abrban.com/ + +--- + +## پروژه‌های proxy-cache + +| پروژه | upstream | کاربرد | +|-------|----------|--------| +| `proxy-dockerhub` | docker.io | Rook، Ceph، bitnami، … | +| `proxy-quay` | quay.io | cephcsi و … | +| `proxy-k8s` | registry.k8s.io | CSI sidecarها | + +ایجاد از UI: **Administration → Registries → New Endpoint** سپس **Projects → New Project** با نوع Proxy Cache. + +> health check بعضی endpointها (مثلاً quay) از UI timeout می‌خورد؛ از داخل `harbor-core` با curl و proxy ممکن است OK باشد. در صورت نیاز endpoint را با type `docker-registry` بسازید. + +### proxy در Harbor + +پروکسی از secret `registry-egress-proxy` در ConfigMapهای `harbor-core` و `harbor-jobservice-env` تزریق می‌شود. + +**مهم:** Go (harbor-core/jobservice) به `http_proxy` / `https_proxy` **lowercase** هم نیاز دارد. اگر health check upstream `unhealthy` ماند، هر دو حالت uppercase و lowercase را در ConfigMap بگذارید و podها را restart کنید: + +```bash +kubectl -n cloudhost rollout restart deploy/harbor-core deploy/harbor-jobservice +``` + +--- + +## mirror دستی ایمیج (وقتی proxy-cache کار نمی‌کند) + +روی abr، pull مستقیم از docker.io/quay از kubelet ممکن نیست. ایمیج‌های حیاتی را با Job داخل کلاستر mirror کنید: + +**مقصد push:** `harbor-registry.cloudhost.svc.cluster.local:5000` (HTTP، با `harbor_registry_user`) + +**مثال مسیرها در registry:** + +| ایمیج upstream | مسیر در registry | +|----------------|------------------| +| `rook/ceph:v1.20.1` | `rook/ceph:v1.20.1` | +| `quay.io/ceph/ceph:v19.2` | `proxy-dockerhub/ceph/ceph:v19.2` | + +```bash +# لیست ایمیج‌های داخل harbor-registry +kubectl -n cloudhost exec deploy/harbor-portal -- \ + curl -s -u "harbor_registry_user:$(kubectl -n cloudhost get secret harbor-core -o jsonpath='{.data.REGISTRY_CREDENTIAL_PASSWORD}' | base64 -d)" \ + http://harbor-registry:5000/v2/_catalog +``` + +> push مستقیم به `harbor-registry:5000` metadata در Harbor UI را به‌روز نمی‌کند؛ برای kubelet کافی است چون `/v2/` به harbor-registry route شده. + +--- + +## مدیریت روزمره + +### وضعیت + +```bash +kubectl -n cloudhost get pods | grep harbor +kubectl -n cloudhost get ingress registry -o yaml | grep -A3 'path:' +curl -sk -o /dev/null -w "%{http_code}\n" https://registry.abrban.com/ +curl -sk -u admin: https://registry.abrban.com/api/v2.0/systeminfo +``` + +### لاگ‌ها + +```bash +kubectl -n cloudhost logs deploy/harbor-core --tail=50 +kubectl -n cloudhost logs deploy/harbor-jobservice --tail=50 +kubectl -n cloudhost logs deploy/harbor-registry -c registry --tail=50 +``` + +### ارتقا Harbor + +```bash +helm repo update harbor +./scripts/install-harbor-registry.sh +# Ingress split را دوباره تأیید کنید +``` + +### rollback به registry قدیمی + +```bash +kubectl -n cloudhost scale deploy/registry --replicas=1 +# Ingress را فقط به service registry:5000 برگردانید +helm uninstall harbor -n cloudhost +``` + +--- + +## عیب‌یابی + +| علامت | علت محتمل | اقدام | +|-------|-----------|--------| +| `https://registry.abrban.com/` → 404 | Ingress فقط به `harbor-core` وصل است | `/` → `harbor-portal` | +| `ImagePullBackOff` برای `cloudhost-backend` | `/v2/` به Harbor رفته، ایمیج platform آنجا نیست | `/v2/` (عمومی) → `registry` قدیمی | +| `not found` برای `proxy-dockerhub/...` | ایمیج mirror نشده | Job skopeo یا proxy-cache | +| push با 499/503 | Traefik timeout | push از داخل کلاستر به `harbor-registry:5000` | +| registry endpoint `unhealthy` | proxy lowercase یا timeout health check | patch ConfigMap + restart؛ یا mirror دستی | +| `authentication required` روی pull | pull secret اشتباه namespace | `cloudhost` vs `harbor_registry_user` در `rook-ceph` | + +### تست pull + +```bash +# platform (cloudhost user) +curl -sk -u "cloudhost:" https://registry.abrban.com/v2/cloudhost-backend/tags/list + +# rook/ceph (harbor_registry_user) +curl -sk -u "harbor_registry_user:" https://registry.abrban.com/v2/rook/ceph/tags/list +``` + +--- + +## چک‌لیست بعد از نصب + +- [ ] Portal روی `/` پاسخ 200 +- [ ] `/api/v2.0/systeminfo` پاسخ JSON +- [ ] Ingress split `/v2/` درست است +- [ ] پروژه‌های `proxy-dockerhub`, `proxy-k8s`, `proxy-quay` ساخته شده +- [ ] `registry-pull-secret` در `cloudhost` و `rook-ceph` به‌روز است +- [ ] ایمیج‌های Rook/Ceph mirror شده و pull تست شده diff --git a/RUNBOOK.fa.md b/RUNBOOK.fa.md index 78c62a4..73d3840 100644 --- a/RUNBOOK.fa.md +++ b/RUNBOOK.fa.md @@ -21,7 +21,8 @@ | **Backend** | NestJS (REST `/api/v1`) | منطق کسب‌وکار، ساخت اپ، احراز هویت | | **Postgres** | postgres:16 | دیتابیس اصلی (کاربر، اپ، فاکتور، …) | | **Redis** | redis:7 | کش، صف Bull (مهاجرت اپ، دسترسی موقت)، پیشرفت بیلد | -| **Registry داخلی** | registry:2 | ایمیج‌های build‌شده | +| **Registry داخلی** | Harbor + registry:2 (legacy) | ایمیج‌های build و platform؛ جزئیات: [`RUNBOOK-HARBOR.fa.md`](RUNBOOK-HARBOR.fa.md) | +| **Storage (Ceph)** | Rook-Ceph | PVC (`rook-ceph-block`) + bucket zip (`rook-ceph-bucket`)؛ جزئیات: [`RUNBOOK-CEPH.fa.md`](RUNBOOK-CEPH.fa.md) | | **Build pipeline** | Kaniko | تبدیل سورس به ایمیج Docker داخل کلاستر (بدون Docker daemon) | | **Kubernetes** | k3s (تک‌نود) | اجرای همه‌ی موارد بالا + اپ‌های کاربر | @@ -29,7 +30,7 @@ - `abrban.com` → لندینگ - `panel.abrban.com` → پنل احرازشده - `api.abrban.com` → بک‌اند -- `registry.abrban.com` → رجیستری داخلی (pull توسط kubelet) +- `registry.abrban.com` → Harbor (UI + proxy-cache) + registry قدیمی برای ایمیج‌های platform — [`RUNBOOK-HARBOR.fa.md`](RUNBOOK-HARBOR.fa.md) - `apps.abrban.com` → دامنه‌ی پیش‌فرض اپ‌های کاربر ### ۱.۴ جریان احراز هویت diff --git a/backend/helm/cloudhost-ceph/.helmignore b/backend/helm/cloudhost-ceph/.helmignore new file mode 100644 index 0000000..c47273f --- /dev/null +++ b/backend/helm/cloudhost-ceph/.helmignore @@ -0,0 +1,3 @@ +.rook-cluster-values-ref.yaml +*.md +scripts/ diff --git a/backend/helm/cloudhost-ceph/.rook-cluster-values-ref.yaml b/backend/helm/cloudhost-ceph/.rook-cluster-values-ref.yaml new file mode 100644 index 0000000..1790c63 --- /dev/null +++ b/backend/helm/cloudhost-ceph/.rook-cluster-values-ref.yaml @@ -0,0 +1,758 @@ +# Default values for a single rook-ceph cluster +# This is a YAML-formatted file. +# Declare variables to be passed into your templates. + +# -- Namespace of the main rook operator +operatorNamespace: rook-ceph + +# -- The metadata.name of the CephCluster CR +# @default -- The same as the namespace +clusterName: + +# -- Optional override of the target kubernetes version +kubeVersion: + +# -- Cluster ceph.conf override +configOverride: +# configOverride: | +# [global] +# mon_allow_pool_delete = true +# osd_pool_default_size = 3 +# osd_pool_default_min_size = 2 + +# Installs a debugging toolbox deployment +toolbox: + # -- Enable Ceph debugging pod deployment. See [toolbox](../Troubleshooting/ceph-toolbox.md) + enabled: false + # -- Toolbox image, defaults to the image used by the Ceph cluster + image: #quay.io/ceph/ceph:v20.2.1 + # -- Toolbox tolerations + tolerations: [] + # -- Toolbox affinity + affinity: {} + # -- Toolbox labels + labels: {} + # -- Toolbox container security context + containerSecurityContext: + runAsNonRoot: true + runAsUser: 2016 + runAsGroup: 2016 + capabilities: + drop: ["ALL"] + # -- Toolbox resources + resources: + limits: + memory: "1Gi" + requests: + cpu: "100m" + memory: "128Mi" + # -- Set the priority class for the toolbox if desired + priorityClassName: + +monitoring: + # -- Enable Prometheus integration, will also create necessary RBAC rules to allow Operator to create ServiceMonitors. + # Monitoring requires Prometheus to be pre-installed + enabled: false + # -- Whether to disable the metrics reported by Ceph. If false, the prometheus mgr module and Ceph exporter are enabled + metricsDisabled: false + # -- Whether to create the Prometheus rules for Ceph alerts + createPrometheusRules: false + # -- Edit Prometheus rules for Ceph alerts + prometheusRuleOverrides: {} + # CephHealthWarning: + # disabled: true + # NVMeoFHighWriteLatency: + # for: 3m + # labels: + # severity: critical + # -- The namespace in which to create the prometheus rules, if different from the rook cluster namespace. + # If you have multiple rook-ceph clusters in the same k8s cluster, choose the same namespace (ideally, namespace with prometheus + # deployed) to set rulesNamespaceOverride for all the clusters. Otherwise, you will get duplicate alerts with multiple alert definitions. + rulesNamespaceOverride: + # Monitoring settings for external clusters: + # externalMgrEndpoints: + # externalMgrPrometheusPort: + # Scrape interval for prometheus + # interval: 10s + # allow adding custom labels and annotations to the prometheus rule + prometheusRule: + # -- Labels applied to PrometheusRule + labels: {} + # -- Annotations applied to PrometheusRule + annotations: {} + +# imagePullSecrets option allow to pull docker images from private docker registry. Option will be passed to all service accounts. +# imagePullSecrets: +# - name: my-registry-secret + +# Labels and annotations to add to the CephCluster CR +cephClusterMetadata: + annotations: {} + labels: {} + +# Specify these values to override the Ceph image in the cephClusterSpec below. +# If specifying these values, do not include the cephVersion section in the cephClusterSpec. +cephImage: + # The repository from which to pull the ceph image + repository: quay.io/ceph/ceph + # In production, use a specific version tag instead of the general v20 flag, which pulls the latest release and could result in different + # versions running within the cluster. See tags available at https://hub.docker.com/r/ceph/ceph/tags/. + # To be more precise, you can always use a timestamp tag such as quay.io/ceph/ceph:v20.2.1-20260402 + tag: v20.2.1 + # Whether to allow unsupported versions of Ceph. Currently Squid and Tentacle are supported. + # Future versions would require this to be set to `true`. + # Do not set to true in production. + allowUnsupported: false + # The image pull policy for pulling the ceph image in the ceph daemon pods, defaults to IfNotPresent + # imagePullPolicy: IfNotPresent + +# All values below are taken from the CephCluster CRD +# -- Cluster configuration. +# @default -- See [below](#ceph-cluster-spec) +cephClusterSpec: + # This cluster spec example is for a converged cluster where all the Ceph daemons are running locally, + # as in the host-based example (cluster.yaml). For a different configuration such as a + # PVC-based cluster (cluster-on-pvc.yaml), external cluster (cluster-external.yaml), + # or stretch cluster (cluster-stretched.yaml), replace this entire `cephClusterSpec` + # with the specs from those examples. + # For more details, check https://rook.io/docs/rook/v1.10/CRDs/Cluster/ceph-cluster-crd/ + + # The path on the host where configuration files will be persisted. Must be specified. If there are multiple clusters, the directory must be unique for each cluster. + # Important: if you reinstall the cluster, make sure you delete this directory from each host or else the mons will fail to start on the new cluster. + # In Minikube, the '/data' directory is configured to persist across reboots. Use "/data/rook" in Minikube environment. + dataDirHostPath: /var/lib/rook + + # Whether or not upgrade should continue even if a check fails + # This means Ceph's status could be degraded and we don't recommend upgrading but you might decide otherwise + # Use at your OWN risk + # To understand Rook's upgrade process of Ceph, read https://rook.io/docs/rook/v1.10/Upgrade/ceph-upgrade/ + skipUpgradeChecks: false + + # Whether or not continue if PGs are not clean during an upgrade + continueUpgradeAfterChecksEvenIfNotHealthy: false + + # WaitTimeoutForHealthyOSDInMinutes defines the time (in minutes) the operator would wait before an OSD can be stopped for upgrade or restart. + # If the timeout exceeds and OSD is not ok to stop, then the operator would skip upgrade for the current OSD and proceed with the next one + # if `continueUpgradeAfterChecksEvenIfNotHealthy` is `false`. If `continueUpgradeAfterChecksEvenIfNotHealthy` is `true`, then operator would + # continue with the upgrade of an OSD even if its not ok to stop after the timeout. This timeout won't be applied if `skipUpgradeChecks` is `true`. + # The default wait timeout is 10 minutes. + waitTimeoutForHealthyOSDInMinutes: 10 + + # Whether or not requires PGs are clean before an OSD upgrade. If set to `true` OSD upgrade process won't start until PGs are healthy. + # This configuration will be ignored if `skipUpgradeChecks` is `true`. + # Default is false. + upgradeOSDRequiresHealthyPGs: false + + mon: + # Set the number of mons to be started. Generally recommended to be 3. + # For highest availability, an odd number of mons should be specified. + count: 3 + # The mons should be on unique nodes. For production, at least 3 nodes are recommended for this reason. + # Mons should only be allowed on the same node for test environments where data loss is acceptable. + allowMultiplePerNode: false + + mgr: + # When higher availability of the mgr is needed, increase the count to 2. + # In that case, one mgr will be active and one in standby. When Ceph updates which + # mgr is active, Rook will update the mgr services to match the active mgr. + count: 2 + allowMultiplePerNode: false + modules: + # List of modules to optionally enable or disable. + # Note the "dashboard" and "monitoring" modules are already configured by other settings in the cluster CR. + # - name: rook + # enabled: true + + # enable the ceph dashboard for viewing cluster status + dashboard: + enabled: true + # serve the dashboard under a subpath (useful when you are accessing the dashboard via a reverse proxy) + # urlPrefix: /ceph-dashboard + # serve the dashboard at the given port. + # port: 8443 + # Serve the dashboard using SSL (if using ingress to expose the dashboard and `ssl: true` you need to set + # the corresponding "backend protocol" annotation(s) for your ingress controller of choice) + ssl: true + + # Network configuration, see: https://github.com/rook/rook/blob/master/Documentation/CRDs/Cluster/ceph-cluster-crd.md#network-configuration-settings + network: + connections: + # Whether to encrypt the data in transit across the wire to prevent eavesdropping the data on the network. + # The default is false. When encryption is enabled, all communication between clients and Ceph daemons, or between Ceph daemons will be encrypted. + # When encryption is not enabled, clients still establish a strong initial authentication and data integrity is still validated with a crc check. + # IMPORTANT: Encryption requires the 5.11 kernel for the latest nbd and cephfs drivers. Alternatively for testing only, + # you can set the "mounter: rbd-nbd" in the rbd storage class, or "mounter: fuse" in the cephfs storage class. + # The nbd and fuse drivers are *not* recommended in production since restarting the csi driver pod will disconnect the volumes. + encryption: + enabled: false + # Whether to compress the data in transit across the wire. The default is false. + # The kernel requirements above for encryption also apply to compression. + compression: + enabled: false + # Whether to require communication over msgr2. If true, the msgr v1 port (6789) will be disabled + # and clients will be required to connect to the Ceph cluster with the v2 port (3300). + # Requires a kernel that supports msgr v2 (kernel 5.11 or CentOS 8.4 or newer). + requireMsgr2: false + # # enable host networking + # provider: host + # # EXPERIMENTAL: enable the Multus network provider + # provider: multus + # selectors: + # # The selector keys are required to be `public` and `cluster`. + # # Based on the configuration, the operator will do the following: + # # 1. if only the `public` selector key is specified both public_network and cluster_network Ceph settings will listen on that interface + # # 2. if both `public` and `cluster` selector keys are specified the first one will point to 'public_network' flag and the second one to 'cluster_network' + # # + # # In order to work, each selector value must match a NetworkAttachmentDefinition object in Multus + # # + # # public: public-conf --> NetworkAttachmentDefinition object name in Multus + # # cluster: cluster-conf --> NetworkAttachmentDefinition object name in Multus + # # Provide internet protocol version. IPv6, IPv4 or empty string are valid options. Empty string would mean IPv4 + # ipFamily: "IPv6" + # # Ceph daemons to listen on both IPv4 and Ipv6 networks + # dualStack: false + + # enable the crash collector for ceph daemon crash collection + crashCollector: + disable: false + # Uncomment daysToRetain to prune ceph crash entries older than the + # specified number of days. + # daysToRetain: 30 + + # enable log collector, daemons will log on files and rotate + logCollector: + enabled: true + periodicity: daily # one of: hourly, daily, weekly, monthly + maxLogSize: 500M # SUFFIX may be 'M' or 'G'. Must be at least 1M. + + # automate [data cleanup process](https://github.com/rook/rook/blob/master/Documentation/Storage-Configuration/ceph-teardown.md#delete-the-data-on-hosts) in cluster destruction. + cleanupPolicy: + # Since cluster cleanup is destructive to data, confirmation is required. + # To destroy all Rook data on hosts during uninstall, confirmation must be set to "yes-really-destroy-data". + # This value should only be set when the cluster is about to be deleted. After the confirmation is set, + # Rook will immediately stop configuring the cluster and only wait for the delete command. + # If the empty string is set, Rook will not destroy any data on hosts during uninstall. + confirmation: "" + # sanitizeDisks represents settings for sanitizing OSD disks on cluster deletion + sanitizeDisks: + # method indicates if the entire disk should be sanitized or simply ceph's metadata + # in both case, re-install is possible + # possible choices are 'complete' or 'quick' (default) + method: quick + # dataSource indicate where to get random bytes from to write on the disk + # possible choices are 'zero' (default) or 'random' + # using random sources will consume entropy from the system and will take much more time then the zero source + dataSource: zero + # iteration overwrite N times instead of the default (1) + # takes an integer value + iteration: 1 + # allowUninstallWithVolumes defines how the uninstall should be performed + # If set to true, cephCluster deletion does not wait for the PVs to be deleted. + allowUninstallWithVolumes: false + + # To control where various services will be scheduled by kubernetes, use the placement configuration sections below. + # The example under 'all' would have all services scheduled on kubernetes nodes labeled with 'role=storage-node' and + # tolerate taints with a key of 'storage-node'. + # placement: + # all: + # nodeAffinity: + # requiredDuringSchedulingIgnoredDuringExecution: + # nodeSelectorTerms: + # - matchExpressions: + # - key: role + # operator: In + # values: + # - storage-node + # podAffinity: + # podAntiAffinity: + # topologySpreadConstraints: + # tolerations: + # - key: storage-node + # operator: Exists + # # The above placement information can also be specified for mon, osd, and mgr components + # mon: + # # Monitor deployments may contain an anti-affinity rule for avoiding monitor + # # collocation on the same node. This is a required rule when host network is used + # # or when AllowMultiplePerNode is false. Otherwise this anti-affinity rule is a + # # preferred rule with weight: 50. + # osd: + # mgr: + # cleanup: + + # annotations: + # all: + # mon: + # osd: + # cleanup: + # prepareosd: + # # If no mgr annotations are set, prometheus scrape annotations will be set by default. + # mgr: + # dashboard: + + # labels: + # all: + # mon: + # osd: + # cleanup: + # mgr: + # prepareosd: + # # monitoring is a list of key-value pairs. It is injected into all the monitoring resources created by operator. + # # These labels can be passed as LabelSelector to Prometheus + # monitoring: + # dashboard: + + resources: + mgr: + limits: + memory: "1Gi" + requests: + cpu: "500m" + memory: "512Mi" + mon: + limits: + memory: "2Gi" + requests: + cpu: "1000m" + memory: "1Gi" + osd: + limits: + memory: "4Gi" + requests: + cpu: "1000m" + memory: "4Gi" + prepareosd: + # limits: It is not recommended to set limits on the OSD prepare job + # since it's a one-time burst for memory that must be allowed to + # complete without an OOM kill. Note however that if a k8s + # limitRange guardrail is defined external to Rook, the lack of + # a limit here may result in a sync failure, in which case a + # limit should be added. 1200Mi may suffice for up to 15Ti + # OSDs ; for larger devices 2Gi may be required. + # cf. https://github.com/rook/rook/pull/11103 + requests: + cpu: "500m" + memory: "50Mi" + mgr-sidecar: + limits: + memory: "100Mi" + requests: + cpu: "100m" + memory: "40Mi" + crashcollector: + limits: + memory: "60Mi" + requests: + cpu: "100m" + memory: "60Mi" + logcollector: + limits: + memory: "1Gi" + requests: + cpu: "100m" + memory: "100Mi" + cleanup: + limits: + memory: "1Gi" + requests: + cpu: "500m" + memory: "100Mi" + exporter: + limits: + memory: "128Mi" + requests: + cpu: "50m" + memory: "50Mi" + cmd-reporter: + limits: + memory: "1Gi" + requests: + cpu: "500m" + memory: "100Mi" + + # The option to automatically remove OSDs that are out and are safe to destroy. + removeOSDsIfOutAndSafeToRemove: false + + # priority classes to apply to ceph resources + priorityClassNames: + mon: system-node-critical + osd: system-node-critical + mgr: system-cluster-critical + + storage: # cluster level storage configuration and selection + useAllNodes: true + useAllDevices: true + # deviceFilter: + # config: + # crushRoot: "custom-root" # specify a non-default root label for the CRUSH map + # metadataDevice: "md0" # specify a non-rotational storage so ceph-volume will use it as block db device of bluestore. + # databaseSizeMB: "1024" # uncomment if the disks are smaller than 100 GB + # osdsPerDevice: "1" # this value can be overridden at the node or device level + # encryptedDevice: "true" # the default value for this option is "false" + # # Individual nodes and their config can be specified as well, but 'useAllNodes' above must be set to false. Then, only the named + # # nodes below will be used as storage resources. Each node's 'name' field should match their 'kubernetes.io/hostname' label. + # nodes: + # - name: "172.17.4.201" + # devices: # specific devices to use for storage can be specified for each node + # - name: "sdb" + # - name: "nvme01" # multiple osds can be created on high performance devices + # config: + # osdsPerDevice: "5" + # - name: "/dev/disk/by-id/ata-ST4000DM004-XXXX" # devices can be specified using full udev paths + # config: # configuration can be specified at the node level which overrides the cluster level config + # - name: "172.17.4.301" + # deviceFilter: "^sd." + + # The section for configuring management of daemon disruptions during upgrade or fencing. + disruptionManagement: + # If true, the operator will create and manage PodDisruptionBudgets for OSD, Mon, RGW, and MDS daemons. OSD PDBs are managed dynamically + # via the strategy outlined in the [design](https://github.com/rook/rook/blob/master/design/ceph/ceph-managed-disruptionbudgets.md). The operator will + # block eviction of OSDs by default and unblock them safely when drains are detected. + managePodBudgets: true + # A duration in minutes that determines how long an entire failureDomain like `region/zone/host` will be held in `noout` (in addition to the + # default DOWN/OUT interval) when it is draining. This is only relevant when `managePodBudgets` is `true`. The default value is `30` minutes. + osdMaintenanceTimeout: 30 + + # Configure the healthcheck and liveness probes for ceph pods. + # Valid values for daemons are 'mon', 'osd', 'status' + healthCheck: + daemonHealth: + mon: + disabled: false + interval: 45s + osd: + disabled: false + interval: 60s + status: + disabled: false + interval: 60s + # Change pod liveness probe, it works for all mon, mgr, and osd pods. + livenessProbe: + mon: + disabled: false + mgr: + disabled: false + osd: + disabled: false + +ingress: + # -- Enable an ingress for the ceph-dashboard + dashboard: {} + # labels: + # external-dns/private: "true" + # annotations: + # external-dns.alpha.kubernetes.io/hostname: dashboard.example.com + # nginx.ingress.kubernetes.io/rewrite-target: /ceph-dashboard/$2 + # If the dashboard has ssl: true the following will make sure the NGINX Ingress controller can expose the dashboard correctly + # nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" + # nginx.ingress.kubernetes.io/server-snippet: | + # proxy_ssl_verify off; + # host: + # name: dashboard.example.com + # path: "/ceph-dashboard(/|$)(.*)" + # pathType: Prefix + # tls: + # - hosts: + # - dashboard.example.com + # secretName: testsecret-tls + ## Note: Only one of ingress class annotation or the `ingressClassName:` can be used at a time + ## to set the ingress class + # ingressClassName: nginx + +route: + # -- Enable an HTTPRoute for the ceph-dashboard + dashboard: {} + # labels: + # external-dns/private: "true" + # annotations: + # external-dns.alpha.kubernetes.io/hostname: dashboard.example.com + # nginx.ingress.kubernetes.io/rewrite-target: /ceph-dashboard/$2 + # host: + # name: dashboard.example.com + # path: "/" + # pathType: PathPrefix + # parentRefs: + # - name: internal + # namespace: kube-system + # sectionName: https + +# -- A list of CephBlockPool configurations to deploy +# @default -- See [below](#ceph-block-pools) +cephBlockPools: + - name: ceph-blockpool + # see https://github.com/rook/rook/blob/master/Documentation/CRDs/Block-Storage/ceph-block-pool-crd.md#spec for available configuration + spec: + failureDomain: host + replicated: + size: 3 + # Enables collecting RBD per-image IO statistics by enabling dynamic OSD performance counters. Defaults to false. + # For reference: https://docs.ceph.com/docs/latest/mgr/prometheus/#rbd-io-statistics + # enableRBDStats: true + storageClass: + enabled: true + name: ceph-block + annotations: {} + labels: {} + isDefault: true + reclaimPolicy: Delete + allowVolumeExpansion: true + volumeBindingMode: "Immediate" + mountOptions: [] + # see https://kubernetes.io/docs/concepts/storage/storage-classes/#allowed-topologies + allowedTopologies: [] + # - matchLabelExpressions: + # - key: rook-ceph-role + # values: + # - storage-node + # see https://github.com/rook/rook/blob/master/Documentation/Storage-Configuration/Block-Storage-RBD/block-storage.md#provision-storage for available configuration + parameters: + # (optional) mapOptions is a comma-separated list of map options. + # For krbd options refer + # https://docs.ceph.com/docs/latest/man/8/rbd/#kernel-rbd-krbd-options + # For nbd options refer + # https://docs.ceph.com/docs/latest/man/8/rbd-nbd/#options + # mapOptions: lock_on_read,queue_depth=1024 + + # (optional) unmapOptions is a comma-separated list of unmap options. + # For krbd options refer + # https://docs.ceph.com/docs/latest/man/8/rbd/#kernel-rbd-krbd-options + # For nbd options refer + # https://docs.ceph.com/docs/latest/man/8/rbd-nbd/#options + # unmapOptions: force + + # RBD image format. Defaults to "2". + imageFormat: "2" + + # RBD image features, equivalent to OR'd bitfield value: 63 + # Available for imageFormat: "2". Older releases of CSI RBD + # support only the `layering` feature. The Linux kernel (KRBD) supports the + # full feature complement as of 5.4 + imageFeatures: layering + + # These secrets contain Ceph admin credentials. + csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner + csi.storage.k8s.io/provisioner-secret-namespace: "{{ .Release.Namespace }}" + csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner + csi.storage.k8s.io/controller-expand-secret-namespace: "{{ .Release.Namespace }}" + csi.storage.k8s.io/controller-publish-secret-name: rook-csi-rbd-provisioner + csi.storage.k8s.io/controller-publish-secret-namespace: "{{ .Release.Namespace }}" + csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node + csi.storage.k8s.io/node-stage-secret-namespace: "{{ .Release.Namespace }}" + # Specify the filesystem type of the volume. If not specified, csi-provisioner + # will set default as `ext4`. Note that `xfs` is not recommended due to potential deadlock + # in hyperconverged settings where the volume is mounted on the same node as the osds. + csi.storage.k8s.io/fstype: ext4 + +# -- A list of CephFileSystem configurations to deploy +# @default -- See [below](#ceph-file-systems) +cephFileSystems: + - name: ceph-filesystem + # see https://github.com/rook/rook/blob/master/Documentation/CRDs/Shared-Filesystem/ceph-filesystem-crd.md#filesystem-settings for available configuration + spec: + metadataPool: + replicated: + size: 3 + dataPools: + - failureDomain: host + replicated: + size: 3 + # Optional and highly recommended, 'data0' by default, see https://github.com/rook/rook/blob/master/Documentation/CRDs/Shared-Filesystem/ceph-filesystem-crd.md#pools + name: data0 + metadataServer: + activeCount: 1 + activeStandby: true + resources: + limits: + memory: "4Gi" + requests: + cpu: "1000m" + memory: "4Gi" + priorityClassName: system-cluster-critical + storageClass: + enabled: true + isDefault: false + name: ceph-filesystem + # (Optional) specify a data pool to use, must be the name of one of the data pools above, 'data0' by default + pool: data0 + reclaimPolicy: Delete + allowVolumeExpansion: true + volumeBindingMode: "Immediate" + annotations: {} + labels: {} + mountOptions: [] + # see https://github.com/rook/rook/blob/master/Documentation/Storage-Configuration/Shared-Filesystem-CephFS/filesystem-storage.md#provision-storage for available configuration + parameters: + # The secrets contain Ceph admin credentials. + csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner + csi.storage.k8s.io/provisioner-secret-namespace: "{{ .Release.Namespace }}" + csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner + csi.storage.k8s.io/controller-expand-secret-namespace: "{{ .Release.Namespace }}" + csi.storage.k8s.io/controller-publish-secret-name: rook-csi-cephfs-provisioner + csi.storage.k8s.io/controller-publish-secret-namespace: "{{ .Release.Namespace }}" + csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node + csi.storage.k8s.io/node-stage-secret-namespace: "{{ .Release.Namespace }}" + # Specify the filesystem type of the volume. If not specified, csi-provisioner + # will set default as `ext4`. Note that `xfs` is not recommended due to potential deadlock + # in hyperconverged settings where the volume is mounted on the same node as the osds. + csi.storage.k8s.io/fstype: ext4 + +# -- Settings for the filesystem snapshot class +# @default -- See [CephFS Snapshots](../Storage-Configuration/Ceph-CSI/ceph-csi-snapshot.md#cephfs-snapshots) +cephFileSystemVolumeSnapshotClass: + enabled: false + name: ceph-filesystem + isDefault: true + deletionPolicy: Delete + annotations: {} + labels: {} + # see https://rook.io/docs/rook/v1.10/Storage-Configuration/Ceph-CSI/ceph-csi-snapshot/#cephfs-snapshots for available configuration + parameters: {} + +# -- Settings for the block pool snapshot class +# @default -- See [RBD Snapshots](../Storage-Configuration/Ceph-CSI/ceph-csi-snapshot.md#rbd-snapshots) +cephBlockPoolsVolumeSnapshotClass: + enabled: false + name: ceph-block + isDefault: false + deletionPolicy: Delete + annotations: {} + labels: {} + # see https://rook.io/docs/rook/v1.10/Storage-Configuration/Ceph-CSI/ceph-csi-snapshot/#rbd-snapshots for available configuration + parameters: {} + +# -- A list of CephObjectStore configurations to deploy +# @default -- See [below](#ceph-object-stores) +cephObjectStores: + - name: ceph-objectstore + # see https://github.com/rook/rook/blob/master/Documentation/CRDs/Object-Storage/ceph-object-store-crd.md#object-store-settings for available configuration + spec: + metadataPool: + failureDomain: host + replicated: + size: 3 + dataPool: + failureDomain: host + erasureCoded: + dataChunks: 2 + codingChunks: 1 + parameters: + bulk: "true" + preservePoolsOnDelete: true + gateway: + port: 80 + resources: + limits: + memory: "2Gi" + requests: + cpu: "1000m" + memory: "1Gi" + # securePort: 443 + # sslCertificateRef: + instances: 1 + priorityClassName: system-cluster-critical + # opsLogSidecar: + # resources: + # limits: + # memory: "100Mi" + # requests: + # cpu: "100m" + # memory: "40Mi" + storageClass: + enabled: true + name: ceph-bucket + reclaimPolicy: Delete + volumeBindingMode: "Immediate" + annotations: {} + labels: {} + # see https://github.com/rook/rook/blob/master/Documentation/Storage-Configuration/Object-Storage-RGW/ceph-object-bucket-claim.md#storageclass for available configuration + parameters: + # note: objectStoreNamespace and objectStoreName are configured by the chart + region: us-east-1 + ingress: + # Enable an ingress for the ceph-objectstore + enabled: false + # The ingress port by default will be the object store's "securePort" (if set), or the gateway "port". + # To override those defaults, set this ingress port to the desired port. + # port: 80 + # annotations: {} + # host: + # name: objectstore.example.com + # path: / + # pathType: Prefix + # tls: + # - hosts: + # - objectstore.example.com + # secretName: ceph-objectstore-tls + # ingressClassName: nginx + route: + # Enable an ingress for the ceph-objectstore + enabled: false + # The ingress port by default will be the object store's "securePort" (if set), or the gateway "port". + # To override those defaults, set this ingress port to the desired port. + # port: 80 + # annotations: {} + # host: + # name: objectstore.example.com + # path: / + # pathType: PathPrefix + # parentRefs: + # - name: internal + # namespace: kube-system + # sectionName: https +## cephECBlockPools are disabled by default, please remove the comments and set desired values to enable it +## For erasure coded a replicated metadata pool is required. +## https://rook.io/docs/rook/latest/CRDs/Shared-Filesystem/ceph-filesystem-crd/#erasure-coded +#cephECBlockPools: +# - name: ec-pool +# spec: +# metadataPool: +# replicated: +# size: 2 +# dataPool: +# failureDomain: osd +# erasureCoded: +# dataChunks: 2 +# codingChunks: 1 +# deviceClass: hdd +# +# parameters: +# # clusterID is the namespace where the rook cluster is running +# # If you change this namespace, also change the namespace below where the secret namespaces are defined +# clusterID: rook-ceph # namespace:cluster +# # (optional) mapOptions is a comma-separated list of map options. +# # For krbd options refer +# # https://docs.ceph.com/docs/latest/man/8/rbd/#kernel-rbd-krbd-options +# # For nbd options refer +# # https://docs.ceph.com/docs/latest/man/8/rbd-nbd/#options +# # mapOptions: lock_on_read,queue_depth=1024 +# +# # (optional) unmapOptions is a comma-separated list of unmap options. +# # For krbd options refer +# # https://docs.ceph.com/docs/latest/man/8/rbd/#kernel-rbd-krbd-options +# # For nbd options refer +# # https://docs.ceph.com/docs/latest/man/8/rbd-nbd/#options +# # unmapOptions: force +# +# # RBD image format. Defaults to "2". +# imageFormat: "2" +# +# # RBD image features, equivalent to OR'd bitfield value: 63 +# # Available for imageFormat: "2". Older releases of CSI RBD +# # support only the `layering` feature. The Linux kernel (KRBD) supports the +# # full feature complement as of 5.4 +# # imageFeatures: layering,fast-diff,object-map,deep-flatten,exclusive-lock +# imageFeatures: layering +# +# storageClass: +# provisioner: rook-ceph.rbd.csi.ceph.com # csi-provisioner-name +# enabled: true +# name: rook-ceph-block +# isDefault: false +# annotations: { } +# labels: { } +# allowVolumeExpansion: true +# reclaimPolicy: Delete + +# -- CSI driver name prefix for cephfs, rbd and nfs. +# @default -- `namespace name where rook-ceph operator is deployed` +csiDriverNamePrefix: + diff --git a/backend/helm/cloudhost-ceph/Chart.yaml b/backend/helm/cloudhost-ceph/Chart.yaml new file mode 100644 index 0000000..b8732bb --- /dev/null +++ b/backend/helm/cloudhost-ceph/Chart.yaml @@ -0,0 +1,13 @@ +apiVersion: v2 +name: cloudhost-ceph +description: CloudHost extras on top of Rook-Ceph (app source bucket, platform integration secrets) +type: application +version: 0.1.0 +appVersion: "1.0.0" +keywords: + - ceph + - rook + - storage + - s3 +maintainers: + - name: CloudHost diff --git a/backend/helm/cloudhost-ceph/README.md b/backend/helm/cloudhost-ceph/README.md new file mode 100644 index 0000000..ab65326 --- /dev/null +++ b/backend/helm/cloudhost-ceph/README.md @@ -0,0 +1,140 @@ +# CloudHost Ceph (Rook) + +Helm chart and install scripts for **Rook-Ceph** on CloudHost clusters: + +| Layer | Purpose | +|-------|---------| +| **rook-ceph-block** | Expandable PVCs for apps, databases, registry | +| **rook-ceph-bucket** | S3-compatible storage for uploaded source zip archives | + +The chart does **not** vendor Rook itself — it installs the official [`rook-release`](https://charts.rook.io/release) charts and adds CloudHost-specific **ObjectBucketClaim** + credential sync. + +--- + +## Quick install + +```bash +cd backend/helm/cloudhost-ceph +./scripts/install.sh single-node # one-node k3s (current abr cluster) +# or +./scripts/install.sh multi-node # production, 3+ nodes + raw disks +``` + +Verify: + +```bash +./scripts/verify.sh +``` + +--- + +## Profiles + +### `single-node` + +- OSD on **loop device** `/dev/loop6` (15Gi file at `/var/lib/rook/osd-loopback.img`) — no spare raw disk required +- Requires `ROOK_CEPH_ALLOW_LOOP_DEVICES=true` on the operator +- Replication **size: 1** (no HA) +- Suitable for **staging / single k3s node** +- Images must be pre-mirrored to `registry.abrban.com` (see `RUNBOOK-HARBOR.fa.md`) + +### `multi-node` + +- OSD on **raw devices** (`useAllDevices: true`) +- Replication **size: 3** for block + object metadata +- Erasure-coded object data pool +- Requires **3+ nodes** and dedicated disks + +--- + +## What gets installed + +| Step | Release | Namespace | +|------|---------|-----------| +| 1 | `rook-ceph` (operator) | `rook-ceph` | +| 2 | `rook-ceph-cluster` | `rook-ceph` | +| 3 | `cloudhost-ceph` (OBC + secrets) | `cloudhost-builds` | + +### StorageClasses (from Rook) + +| Name | Use | +|------|-----| +| `rook-ceph-block` | App PVC, DB PVC, Redis, registry, … | +| `rook-ceph-bucket` | `ObjectBucketClaim` → S3 bucket + credentials | + +### CloudHost extras + +| Resource | Description | +|----------|-------------| +| `ObjectBucketClaim/app-sources` | Bucket for user zip uploads | +| `Secret/ceph-app-sources-credentials` | Stable S3 credentials for backend | +| `ConfigMap/cloudhost-ceph-integration` | Suggested `PLATFORM_*` env values | + +--- + +## Platform integration + +After install, configure the **backend**: + +```env +PLATFORM_STORAGE_CLASS=rook-ceph-block +PLATFORM_CREATE_STORAGE_CLASS=false +PLATFORM_STORAGE_PROVISIONER=rook-ceph.rbd.csi.ceph.com +``` + +Mount or env-from secret `cloudhost-builds/ceph-app-sources-credentials`: + +```env +SOURCE_STORAGE_ENDPOINT=http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local:80 +SOURCE_STORAGE_REGION=us-east-1 +SOURCE_STORAGE_BUCKET= +SOURCE_STORAGE_ACCESS_KEY= +SOURCE_STORAGE_SECRET_KEY= +``` + +> **Note:** Existing PVCs on `local-path` / `cloudhost-expandable` are **not** migrated automatically. New apps use `rook-ceph-block` once the backend env is updated. Plan migration per workload (see `RUNBOOK-CEPH.fa.md`). + +--- + +## Uninstall (destructive) + +```bash +./scripts/uninstall.sh +# then on each node: +sudo rm -rf /var/lib/rook /var/lib/rook/osd +``` + +--- + +## Troubleshooting + +```bash +# Cluster health +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph status + +# OSD pods +kubectl -n rook-ceph get pods -l app=rook-ceph-osd + +# RGW (object store) +kubectl -n rook-ceph get pods -l app=rook-ceph-rgw + +# Bucket sync job +kubectl -n cloudhost-builds logs job -l job-name=cloudhost-ceph-bucket-sync --tail=50 +``` + +Full operational guide (Persian): [`../../../RUNBOOK-CEPH.fa.md`](../../../RUNBOOK-CEPH.fa.md) + +Registry / Harbor (Persian): [`../../../RUNBOOK-HARBOR.fa.md`](../../../RUNBOOK-HARBOR.fa.md) + +--- + +## Files + +| File | Role | +|------|------| +| `values-rook-cluster-single-node.yaml` | Rook cluster values (1 node) | +| `values-rook-cluster-multi-node.yaml` | Rook cluster values (production) | +| `values.yaml` | CloudHost OBC / secret sync | +| `scripts/install.sh` | Full install | +| `scripts/verify.sh` | Health check | +| `scripts/uninstall.sh` | Tear down | diff --git a/backend/helm/cloudhost-ceph/scripts/install.sh b/backend/helm/cloudhost-ceph/scripts/install.sh new file mode 100755 index 0000000..dd7eb7c --- /dev/null +++ b/backend/helm/cloudhost-ceph/scripts/install.sh @@ -0,0 +1,52 @@ +#!/usr/bin/env bash +# Install Rook-Ceph operator + cluster + CloudHost bucket extras. +# Usage: ./scripts/install.sh [single-node|multi-node] +set -euo pipefail + +PROFILE="${1:-single-node}" +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +CHART_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)" +ROOK_NS="rook-ceph" +EXTRAS_NS="cloudhost-builds" +CLUSTER_VALUES="${CHART_DIR}/values-rook-cluster-${PROFILE}.yaml" + +if [[ ! -f "${CLUSTER_VALUES}" ]]; then + echo "Unknown profile: ${PROFILE} (missing ${CLUSTER_VALUES})" >&2 + exit 1 +fi + +echo "==> Profile: ${PROFILE}" +echo "==> Adding rook-release helm repo" +helm repo add rook-release https://charts.rook.io/release 2>/dev/null || true +helm repo update rook-release + +echo "==> [1/4] Installing Rook operator in ${ROOK_NS}" +helm upgrade --install rook-ceph rook-release/rook-ceph \ + --namespace "${ROOK_NS}" \ + --create-namespace \ + --wait \ + --timeout 10m + +echo "==> [2/4] Waiting for Rook operator deployment" +kubectl -n "${ROOK_NS}" rollout status deploy/rook-ceph-operator --timeout=300s + +echo "==> [3/4] Installing Ceph cluster (${CLUSTER_VALUES})" +helm upgrade --install rook-ceph-cluster rook-release/rook-ceph-cluster \ + --namespace "${ROOK_NS}" \ + -f "${CLUSTER_VALUES}" \ + --wait \ + --timeout 25m + +echo "==> Waiting for CephCluster phase = Ready (up to 20 min)" +"${SCRIPT_DIR}/wait-ceph-ready.sh" 1200 + +echo "==> [4/4] Installing CloudHost Ceph extras (ObjectBucketClaim) in ${EXTRAS_NS}" +kubectl create namespace "${EXTRAS_NS}" 2>/dev/null || true +helm upgrade --install cloudhost-ceph "${CHART_DIR}" \ + --namespace "${EXTRAS_NS}" \ + -f "${CHART_DIR}/values.yaml" \ + --wait \ + --timeout 15m + +echo "" +echo "==> Done. Run ./scripts/verify.sh to confirm health and print integration hints." diff --git a/backend/helm/cloudhost-ceph/scripts/uninstall.sh b/backend/helm/cloudhost-ceph/scripts/uninstall.sh new file mode 100755 index 0000000..0defc1a --- /dev/null +++ b/backend/helm/cloudhost-ceph/scripts/uninstall.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +# Remove CloudHost extras + Rook cluster + operator (DATA LOSS). +set -euo pipefail + +read -r -p "This deletes ALL Ceph data. Type 'delete-ceph' to continue: " CONFIRM +if [[ "${CONFIRM}" != "delete-ceph" ]]; then + echo "Aborted." + exit 1 +fi + +helm uninstall cloudhost-ceph -n cloudhost-builds 2>/dev/null || true +helm uninstall rook-ceph-cluster -n rook-ceph 2>/dev/null || true +helm uninstall rook-ceph -n rook-ceph 2>/dev/null || true + +echo "Waiting for Rook resources to terminate..." +sleep 15 +kubectl -n rook-ceph get pods 2>/dev/null || true + +cat <<'EOF' + +IMPORTANT: On each node, wipe Rook state before reinstalling: + sudo rm -rf /var/lib/rook + sudo rm -rf /var/lib/rook/osd + +For raw-disk OSDs also zap disks (see RUNBOOK-CEPH.fa.md). +EOF diff --git a/backend/helm/cloudhost-ceph/scripts/verify.sh b/backend/helm/cloudhost-ceph/scripts/verify.sh new file mode 100755 index 0000000..8f66985 --- /dev/null +++ b/backend/helm/cloudhost-ceph/scripts/verify.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOK_NS="rook-ceph" +EXTRAS_NS="cloudhost-builds" + +echo "=== StorageClasses ===" +kubectl get storageclass | grep -E 'NAME|rook-ceph' || true + +echo "" +echo "=== Ceph status ===" +kubectl -n "${ROOK_NS}" exec deploy/rook-ceph-tools -- ceph status 2>/dev/null || echo "(tools pod not ready yet)" + +echo "" +echo "=== OSD / MON pods ===" +kubectl -n "${ROOK_NS}" get pods -l app=rook-ceph-osd 2>/dev/null || kubectl -n "${ROOK_NS}" get pods | grep -E 'osd|mon|mgr|rgw' || true + +echo "" +echo "=== Object bucket claim ===" +kubectl -n "${EXTRAS_NS}" get obc,app-sources 2>/dev/null || kubectl -n "${EXTRAS_NS}" get obc 2>/dev/null || true + +echo "" +echo "=== Platform credentials secret ===" +if kubectl -n "${EXTRAS_NS}" get secret ceph-app-sources-credentials >/dev/null 2>&1; then + echo "Secret ceph-app-sources-credentials exists" + kubectl -n "${EXTRAS_NS}" get secret ceph-app-sources-credentials -o jsonpath='{.data.SOURCE_STORAGE_BUCKET}' | base64 -d + echo "" +else + echo "Secret ceph-app-sources-credentials not ready — check bucket sync job:" + kubectl -n "${EXTRAS_NS}" get jobs,pods | grep bucket-sync || true +fi + +echo "" +echo "=== Suggested backend env ===" +kubectl -n "${EXTRAS_NS}" get configmap cloudhost-ceph-integration -o yaml 2>/dev/null | sed -n '/PLATFORM_/p;/SOURCE_STORAGE_ENDPOINT/p' || true diff --git a/backend/helm/cloudhost-ceph/scripts/wait-ceph-ready.sh b/backend/helm/cloudhost-ceph/scripts/wait-ceph-ready.sh new file mode 100755 index 0000000..3fbabcd --- /dev/null +++ b/backend/helm/cloudhost-ceph/scripts/wait-ceph-ready.sh @@ -0,0 +1,39 @@ +#!/usr/bin/env bash +# Wait until Ceph reports HEALTH_OK or HEALTH_WARN (single-node often stays WARN). +set -euo pipefail + +TIMEOUT="${1:-900}" +ROOK_NS="rook-ceph" +START=$(date +%s) + +echo "Waiting for rook-ceph-tools deployment..." +for _ in $(seq 1 60); do + if kubectl -n "${ROOK_NS}" get deploy rook-ceph-tools >/dev/null 2>&1; then + if kubectl -n "${ROOK_NS}" rollout status deploy/rook-ceph-tools --timeout=120s 2>/dev/null; then + break + fi + fi + sleep 5 +done + +while true; do + NOW=$(date +%s) + if (( NOW - START > TIMEOUT )); then + echo "Timed out after ${TIMEOUT}s waiting for Ceph health" >&2 + kubectl -n "${ROOK_NS}" get cephcluster,pod -o wide || true + exit 1 + fi + + if kubectl -n "${ROOK_NS}" get deploy rook-ceph-tools >/dev/null 2>&1; then + HEALTH=$(kubectl -n "${ROOK_NS}" exec deploy/rook-ceph-tools -- ceph health 2>/dev/null || echo "unknown") + echo "Ceph health: ${HEALTH}" + if [[ "${HEALTH}" == "HEALTH_OK" || "${HEALTH}" == HEALTH_WARN* ]]; then + PHASE=$(kubectl -n "${ROOK_NS}" get cephcluster rook-ceph -o jsonpath='{.status.phase}' 2>/dev/null || echo "") + echo "CephCluster phase: ${PHASE}" + if [[ "${PHASE}" == "Ready" ]]; then + exit 0 + fi + fi + fi + sleep 15 +done diff --git a/backend/helm/cloudhost-ceph/templates/NOTES.txt b/backend/helm/cloudhost-ceph/templates/NOTES.txt new file mode 100644 index 0000000..a66780b --- /dev/null +++ b/backend/helm/cloudhost-ceph/templates/NOTES.txt @@ -0,0 +1,23 @@ +CloudHost Ceph storage is ready (or still initializing). + +Profiles + single-node Directory OSD on /var/lib/rook/osd — for one-node k3s (no HA) + multi-node Raw disk OSDs with replication=3 — production + +StorageClasses created by Rook + rook-ceph-block Block volumes (app PVC, DB, registry, …) + rook-ceph-bucket S3-compatible buckets via ObjectBucketClaim + +Verify cluster health + kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph status + kubectl get storageclass | grep rook-ceph + kubectl -n cloudhost-builds get obc,secret | grep -E 'app-sources|ceph-app-sources' + +Platform backend (after bucket sync Job completes) + PLATFORM_STORAGE_CLASS=rook-ceph-block + PLATFORM_CREATE_STORAGE_CLASS=false + PLATFORM_STORAGE_PROVISIONER=rook-ceph.rbd.csi.ceph.com + + Mount secret cloudhost-builds/ceph-app-sources-credentials for zip upload S3 settings. + +Full guide: backend/helm/cloudhost-ceph/README.md and RUNBOOK-CEPH.fa.md diff --git a/backend/helm/cloudhost-ceph/templates/_helpers.tpl b/backend/helm/cloudhost-ceph/templates/_helpers.tpl new file mode 100644 index 0000000..a3d85da --- /dev/null +++ b/backend/helm/cloudhost-ceph/templates/_helpers.tpl @@ -0,0 +1,27 @@ +{{/* +CloudHost Ceph chart helpers +*/}} +{{- define "cloudhost-ceph.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{- define "cloudhost-ceph.fullname" -}} +{{- if .Values.fullnameOverride }} +{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- $name := default .Chart.Name .Values.nameOverride }} +{{- if contains $name .Release.Name }} +{{- .Release.Name | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }} +{{- end }} +{{- end }} +{{- end }} + +{{- define "cloudhost-ceph.labels" -}} +helm.sh/chart: {{ include "cloudhost-ceph.name" . }}-{{ .Chart.Version }} +app.kubernetes.io/name: {{ include "cloudhost-ceph.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +app.kubernetes.io/part-of: cloudhost +{{- end }} diff --git a/backend/helm/cloudhost-ceph/templates/integration-configmap.yaml b/backend/helm/cloudhost-ceph/templates/integration-configmap.yaml new file mode 100644 index 0000000..2c43842 --- /dev/null +++ b/backend/helm/cloudhost-ceph/templates/integration-configmap.yaml @@ -0,0 +1,20 @@ +{{- if .Values.integration.createConfigMap }} +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ .Values.integration.configMapName }} + namespace: {{ .Values.namespace }} + labels: + {{- include "cloudhost-ceph.labels" . | nindent 4 }} +data: + PLATFORM_STORAGE_CLASS: rook-ceph-block + PLATFORM_CREATE_STORAGE_CLASS: "false" + PLATFORM_STORAGE_PROVISIONER: rook-ceph.rbd.csi.ceph.com + SOURCE_STORAGE_ENDPOINT: {{ .Values.platform.endpoint | quote }} + SOURCE_STORAGE_REGION: {{ .Values.platform.region | quote }} + SOURCE_STORAGE_CREDENTIALS_SECRET: {{ .Values.platform.credentialsSecretName | quote }} + README: | + Block PVCs: set PLATFORM_STORAGE_CLASS=rook-ceph-block on the backend. + New app PVCs use rook-ceph-block; existing local-path PVCs are NOT auto-migrated. + Object storage credentials: secret {{ .Values.platform.credentialsSecretName }} in {{ .Values.namespace }}. +{{- end }} diff --git a/backend/helm/cloudhost-ceph/templates/objectbucketclaim-app-sources.yaml b/backend/helm/cloudhost-ceph/templates/objectbucketclaim-app-sources.yaml new file mode 100644 index 0000000..ad4ef39 --- /dev/null +++ b/backend/helm/cloudhost-ceph/templates/objectbucketclaim-app-sources.yaml @@ -0,0 +1,15 @@ +{{- if .Values.objectStore.claimName }} +apiVersion: objectbucket.io/v1alpha1 +kind: ObjectBucketClaim +metadata: + name: {{ .Values.objectStore.claimName }} + namespace: {{ .Values.namespace }} + labels: + {{- include "cloudhost-ceph.labels" . | nindent 4 }} + annotations: + helm.sh/hook: post-install,post-upgrade + helm.sh/hook-weight: "5" +spec: + storageClassName: {{ .Values.objectStore.bucketStorageClass | quote }} + generateBucketName: {{ .Values.objectStore.generateBucketName | quote }} +{{- end }} diff --git a/backend/helm/cloudhost-ceph/templates/platform-bucket-credentials-job.yaml b/backend/helm/cloudhost-ceph/templates/platform-bucket-credentials-job.yaml new file mode 100644 index 0000000..ffd46e3 --- /dev/null +++ b/backend/helm/cloudhost-ceph/templates/platform-bucket-credentials-job.yaml @@ -0,0 +1,113 @@ +{{- if and .Values.platform.createCredentialsSecret .Values.objectStore.claimName }} +# Stable secret name for platform workers. Populated by a post-install Job once the OBC secret exists. +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ include "cloudhost-ceph.fullname" . }}-bucket-sync + namespace: {{ .Values.namespace }} + labels: + {{- include "cloudhost-ceph.labels" . | nindent 4 }} + annotations: + helm.sh/hook: post-install,post-upgrade + helm.sh/hook-weight: "1" + helm.sh/hook-delete-policy: before-hook-creation +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: {{ include "cloudhost-ceph.fullname" . }}-bucket-sync + namespace: {{ .Values.namespace }} + annotations: + helm.sh/hook: post-install,post-upgrade + helm.sh/hook-weight: "1" + helm.sh/hook-delete-policy: before-hook-creation +rules: + - apiGroups: [""] + resources: ["secrets"] + verbs: ["get", "list", "create", "patch", "update"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: {{ include "cloudhost-ceph.fullname" . }}-bucket-sync + namespace: {{ .Values.namespace }} + annotations: + helm.sh/hook: post-install,post-upgrade + helm.sh/hook-weight: "1" + helm.sh/hook-delete-policy: before-hook-creation +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: {{ include "cloudhost-ceph.fullname" . }}-bucket-sync +subjects: + - kind: ServiceAccount + name: {{ include "cloudhost-ceph.fullname" . }}-bucket-sync + namespace: {{ .Values.namespace }} +--- +apiVersion: batch/v1 +kind: Job +metadata: + name: {{ include "cloudhost-ceph.fullname" . }}-bucket-sync + namespace: {{ .Values.namespace }} + labels: + {{- include "cloudhost-ceph.labels" . | nindent 4 }} + annotations: + helm.sh/hook: post-install,post-upgrade + helm.sh/hook-weight: "10" + helm.sh/hook-delete-policy: before-hook-creation +spec: + backoffLimit: 30 + template: + spec: + serviceAccountName: {{ include "cloudhost-ceph.fullname" . }}-bucket-sync + restartPolicy: OnFailure + containers: + - name: sync + image: registry.abrban.com/proxy-dockerhub/bitnami/kubectl:1.32 + imagePullPolicy: IfNotPresent + env: + - name: OBC_SECRET + value: {{ printf "obc-%s-%s" .Values.namespace .Values.objectStore.claimName | quote }} + - name: TARGET_SECRET + value: {{ .Values.platform.credentialsSecretName | quote }} + - name: NAMESPACE + value: {{ .Values.namespace | quote }} + - name: ENDPOINT + value: {{ .Values.platform.endpoint | quote }} + - name: REGION + value: {{ .Values.platform.region | quote }} + command: + - /bin/bash + - -ec + - | + echo "Waiting for OBC secret ${OBC_SECRET} in ${NAMESPACE}..." + for i in $(seq 1 120); do + if kubectl get secret -n "${NAMESPACE}" "${OBC_SECRET}" >/dev/null 2>&1; then + break + fi + sleep 10 + done + kubectl get secret -n "${NAMESPACE}" "${OBC_SECRET}" >/dev/null + + BUCKET=$(kubectl get secret -n "${NAMESPACE}" "${OBC_SECRET}" -o jsonpath='{.data.BUCKET_NAME}' | base64 -d) + ACCESS=$(kubectl get secret -n "${NAMESPACE}" "${OBC_SECRET}" -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 -d) + SECRET=$(kubectl get secret -n "${NAMESPACE}" "${OBC_SECRET}" -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 -d) + + cat < Ensuring harbor repo" +helm repo add harbor https://helm.goharbor.io 2>/dev/null || true +helm repo update harbor + +echo "==> [0/4] Preflight" +kubectl -n cloudhost get secret abrban-wildcard-tls >/dev/null +kubectl -n cloudhost get secret registry-egress-proxy >/dev/null + +echo "==> [1/4] Disabling old registry ingress (host registry.abrban.com)" +kubectl -n cloudhost delete ingress registry --ignore-not-found + +echo "==> [2/4] Scaling old registry deployment down (rollback-friendly)" +kubectl -n cloudhost scale deploy/registry --replicas=0 || true + +echo "==> [3/4] Installing Harbor" +HTTP_PROXY="$(kubectl -n cloudhost get secret registry-egress-proxy -o jsonpath='{.data.HTTP_PROXY}' | base64 -d)" +HTTPS_PROXY="$(kubectl -n cloudhost get secret registry-egress-proxy -o jsonpath='{.data.HTTPS_PROXY}' | base64 -d)" +NO_PROXY="$(kubectl -n cloudhost get secret registry-egress-proxy -o jsonpath='{.data.NO_PROXY}' | base64 -d)" + +TMP_PROXY_VALUES="$(mktemp)" +cat > "${TMP_PROXY_VALUES}" < [4/4] Done" +kubectl -n cloudhost get ingress | grep -n registry || true + diff --git a/backend/helm/cloudhost-harbor/values-registry.abrban.com.yaml b/backend/helm/cloudhost-harbor/values-registry.abrban.com.yaml new file mode 100644 index 0000000..bbe6be0 --- /dev/null +++ b/backend/helm/cloudhost-harbor/values-registry.abrban.com.yaml @@ -0,0 +1,75 @@ +## Harbor values to REPLACE registry.abrban.com +## Ingress controller on this cluster is Traefik (k3s). +## +## Install: +## helm upgrade --install harbor harbor/harbor -n cloudhost -f backend/helm/cloudhost-harbor/values-registry.abrban.com.yaml +## +externalURL: https://registry.abrban.com + +proxy: + # Values are injected by install script from `cloudhost/registry-egress-proxy`. + httpProxy: "" + httpsProxy: "" + noProxy: "" + +expose: + type: ingress + tls: + enabled: true + certSource: secret + secret: + secretName: abrban-wildcard-tls + ingress: + className: traefik + hosts: + core: registry.abrban.com + annotations: + traefik.ingress.kubernetes.io/router.entrypoints: websecure + # Increase timeouts for large pushes (skopeo/registry blobs) + traefik.ingress.kubernetes.io/router.tls: "true" + +# Disable components we don't need for now to reduce resources +trivy: + enabled: false +notary: + enabled: false +chartmuseum: + enabled: false + +# Single-node staging: keep resource usage modest +core: + replicas: 1 +jobservice: + replicas: 1 +registry: + replicas: 1 + +persistence: + enabled: true + persistentVolumeClaim: + # Use existing default storage (local-path) until Ceph is ready. + # After Ceph, switch to rook-ceph-block for Harbor's PVCs. + registry: + storageClass: local-path + size: 50Gi + jobservice: + storageClass: local-path + size: 5Gi + database: + storageClass: local-path + size: 10Gi + redis: + storageClass: local-path + size: 5Gi + +database: + type: internal + +redis: + type: internal + +portal: + replicas: 1 + +# We will create proxy-cache projects after install (todo: configure-proxy-cache) +