Add Harbor/Ceph runbooks and align single-node Ceph docs with abr deployment.
Document registry ingress split, image mirroring, loop OSD setup, and platform integration paths.
This commit is contained in:
keyhan
@@ -0,0 +1,329 @@
|
||||
# راهنمای Ceph برای ابربان (Rook)
|
||||
|
||||
این سند نصب، معماری واقعی روی کلاستر **abr**، مدیریت روزمره و عیبیابی **Rook-Ceph** را پوشش میدهد.
|
||||
|
||||
- چارت و اسکریپتها: [`backend/helm/cloudhost-ceph/`](backend/helm/cloudhost-ceph/)
|
||||
- README انگلیسی: [`backend/helm/cloudhost-ceph/README.md`](backend/helm/cloudhost-ceph/README.md)
|
||||
- رجیستری: [`RUNBOOK-HARBOR.fa.md`](RUNBOOK-HARBOR.fa.md)
|
||||
|
||||
---
|
||||
|
||||
## چرا Ceph؟
|
||||
|
||||
| نیاز | راهحل Ceph |
|
||||
|------|-------------|
|
||||
| PVC اپ/دیتابیس با **resize** | Block pool → StorageClass `rook-ceph-block` |
|
||||
| آپلود **zip** سورس کاربر | Object store (RGW) → StorageClass `rook-ceph-bucket` |
|
||||
|
||||
یک کلاستر Ceph هر دو را پوشش میدهد؛ zip را روی PVC نگه ندارید — از **bucket** استفاده کنید.
|
||||
|
||||
---
|
||||
|
||||
## معماری روی abr (single-node)
|
||||
|
||||
```
|
||||
registry.abrban.com
|
||||
├── rook/ceph:v1.20.1 → Rook operator
|
||||
└── proxy-dockerhub/ceph/ceph:v19.2 → Ceph daemon (Squid)
|
||||
|
||||
Node abr
|
||||
├── /dev/loop6 (15Gi) → OSD (bluestore raw)
|
||||
├── mon-a, mgr-a, osd-0, rgw → rook-ceph namespace
|
||||
└── RGW: rook-ceph-rgw-ceph-objectstore.rook-ceph.svc:80
|
||||
```
|
||||
|
||||
| محدودیت | توضیح |
|
||||
|---------|--------|
|
||||
| **۱ OSD** | replication=1؛ بدون HA |
|
||||
| **loop device** | دیسک خام نداریم؛ `/dev/loop6` از فایل `osd-loopback.img` |
|
||||
| **HEALTH_WARN** | طبیعی: `OSD count 1 < default size 3`، mon low space |
|
||||
| **ایمیجها** | باید از قبل در Harbor mirror شده باشند (kubelet به docker.io دسترسی ندارد) |
|
||||
|
||||
---
|
||||
|
||||
## پیشنیازها
|
||||
|
||||
### پروفایل `single-node` (کلاستر فعلی abr)
|
||||
|
||||
- Kubernetes 1.28+ (k3s + Traefik)
|
||||
- Harbor بالا و ایمیجهای `rook/ceph` + `ceph/ceph` mirror شده
|
||||
- حداقل **۱۵ گیگ** فضا برای loop OSD (`/var/lib/rook/osd-loopback.img`)
|
||||
- `helm` 3.x و `kubectl` با دسترسی cluster-admin
|
||||
- Secret `registry-pull-secret` در `rook-ceph` با `harbor_registry_user`
|
||||
|
||||
### پروفایل `multi-node` (production)
|
||||
|
||||
- حداقل **۳ نود** + دیسک خام (raw)
|
||||
- فایل values: `values-rook-cluster-multi-node.yaml`
|
||||
|
||||
---
|
||||
|
||||
## نصب (اولین بار — abr)
|
||||
|
||||
### ۱. آمادهسازی loop device برای OSD
|
||||
|
||||
روی نود تکدیسک، Rook به دیسک خام نیاز دارد. یک loop device بسازید:
|
||||
|
||||
```bash
|
||||
# روی نود (یا Job privileged — یکبار)
|
||||
truncate -s 15G /var/lib/rook/osd-loopback.img
|
||||
losetup --find --show /var/lib/rook/osd-loopback.img # → /dev/loop6
|
||||
```
|
||||
|
||||
### ۲. فعالسازی loop در Rook operator
|
||||
|
||||
```bash
|
||||
kubectl -n rook-ceph patch cm rook-ceph-operator-config --type merge \
|
||||
-p '{"data":{"ROOK_CEPH_ALLOW_LOOP_DEVICES":"true"}}'
|
||||
kubectl -n rook-ceph rollout restart deploy/rook-ceph-operator
|
||||
```
|
||||
|
||||
### ۳. mirror ایمیجها (قبل از نصب cluster)
|
||||
|
||||
| ایمیج | مسیر pull |
|
||||
|-------|-----------|
|
||||
| `rook/ceph:v1.20.1` | `registry.abrban.com/rook/ceph:v1.20.1` |
|
||||
| `quay.io/ceph/ceph:v19.2` | `registry.abrban.com/proxy-dockerhub/ceph/ceph:v19.2` |
|
||||
|
||||
جزئیات mirror: [`RUNBOOK-HARBOR.fa.md`](RUNBOOK-HARBOR.fa.md)
|
||||
|
||||
### ۴. نصب operator
|
||||
|
||||
```bash
|
||||
helm repo add rook-release https://charts.rook.io/release
|
||||
helm repo update rook-release
|
||||
|
||||
helm upgrade --install rook-ceph rook-release/rook-ceph \
|
||||
-n rook-ceph --create-namespace \
|
||||
--set image.repository=registry.abrban.com/rook/ceph \
|
||||
--set image.tag=v1.20.1 \
|
||||
--set imagePullSecrets[0].name=registry-pull-secret
|
||||
```
|
||||
|
||||
### ۵. نصب cluster
|
||||
|
||||
```bash
|
||||
cd backend/helm/cloudhost-ceph
|
||||
|
||||
helm upgrade --install rook-ceph-cluster rook-release/rook-ceph-cluster \
|
||||
-n rook-ceph \
|
||||
-f values-rook-cluster-single-node.yaml \
|
||||
--set cephClusterSpec.cephVersion.image=registry.abrban.com/proxy-dockerhub/ceph/ceph:v19.2
|
||||
```
|
||||
|
||||
> **توجه:** `values-rook-cluster-single-node.yaml` از `devices: [{name: "/dev/loop6"}]` استفاده میکند (نه directory — در Rook v1.20 حذف شده).
|
||||
|
||||
### ۶. extras (bucket + secret)
|
||||
|
||||
```bash
|
||||
kubectl create namespace cloudhost-builds 2>/dev/null || true
|
||||
helm upgrade --install cloudhost-ceph . \
|
||||
-n cloudhost-builds -f values.yaml --no-hooks
|
||||
```
|
||||
|
||||
اگر Job `bucket-sync` بهخاطر `bitnami/kubectl` گیر کرد، secret را دستی بسازید:
|
||||
|
||||
```bash
|
||||
kubectl -n cloudhost-builds get secret app-sources -o yaml
|
||||
kubectl -n cloudhost-builds get cm app-sources -o yaml # BUCKET_NAME
|
||||
# → secret ceph-app-sources-credentials (کلیدهای SOURCE_STORAGE_*)
|
||||
```
|
||||
|
||||
### ۷. یکپارچهسازی backend
|
||||
|
||||
```bash
|
||||
kubectl -n cloudhost set env deploy/cloudhost-backend \
|
||||
PLATFORM_STORAGE_CLASS=rook-ceph-block \
|
||||
PLATFORM_CREATE_STORAGE_CLASS=false \
|
||||
PLATFORM_STORAGE_PROVISIONER=rook-ceph.rbd.csi.ceph.com
|
||||
|
||||
kubectl -n cloudhost patch deploy cloudhost-backend --type=json \
|
||||
-p '[{"op":"add","path":"/spec/template/spec/containers/0/envFrom","value":[{"secretRef":{"name":"ceph-app-sources-credentials"}}]}]'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## اسکریپت `install.sh` (نصب تمیز)
|
||||
|
||||
برای نصب از صفر (بعد از آمادهسازی loop + mirror):
|
||||
|
||||
```bash
|
||||
cd backend/helm/cloudhost-ceph
|
||||
./scripts/install.sh single-node
|
||||
./scripts/verify.sh
|
||||
```
|
||||
|
||||
اسکریپت operator + cluster + extras را نصب میکند. روی abr حتماً **قبلش** loop device و mirror ایمیج را انجام دهید.
|
||||
|
||||
---
|
||||
|
||||
## خروجیهای مهم
|
||||
|
||||
### StorageClassها
|
||||
|
||||
| نام | کاربرد |
|
||||
|-----|--------|
|
||||
| `rook-ceph-block` | PVC اپ، DB، Redis، … |
|
||||
| `rook-ceph-bucket` | claim کردن bucket برای zip |
|
||||
|
||||
### Secret پلتفرم
|
||||
|
||||
```bash
|
||||
kubectl -n cloudhost-builds get secret ceph-app-sources-credentials -o yaml
|
||||
kubectl -n cloudhost get secret ceph-app-sources-credentials -o yaml # کپی در cloudhost
|
||||
```
|
||||
|
||||
کلیدها: `SOURCE_STORAGE_ENDPOINT`, `SOURCE_STORAGE_BUCKET`, `SOURCE_STORAGE_ACCESS_KEY`, `SOURCE_STORAGE_SECRET_KEY`
|
||||
|
||||
### RGW endpoint
|
||||
|
||||
```
|
||||
http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local:80
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## مدیریت روزمره
|
||||
|
||||
### سلامت کلاستر
|
||||
|
||||
```bash
|
||||
kubectl -n rook-ceph get cephcluster
|
||||
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph status
|
||||
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd tree
|
||||
kubectl get sc | grep rook-ceph
|
||||
kubectl -n rook-ceph get pods
|
||||
```
|
||||
|
||||
### Dashboard
|
||||
|
||||
```bash
|
||||
kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath='{.data.password}' | base64 -d
|
||||
kubectl -n rook-ceph port-forward svc/rook-ceph-mgr-dashboard 8443:8443
|
||||
# https://localhost:8443
|
||||
```
|
||||
|
||||
### bucket و OBC
|
||||
|
||||
```bash
|
||||
kubectl -n cloudhost-builds get obc app-sources
|
||||
kubectl -n cloudhost-builds get cm app-sources
|
||||
```
|
||||
|
||||
### PVC جدید با Ceph
|
||||
|
||||
```yaml
|
||||
storageClassName: rook-ceph-block
|
||||
```
|
||||
|
||||
فقط **اپهای جدید** (یا بعد از migration) از این StorageClass استفاده میکنند. PVCهای قدیمی روی `local-path` / `cloudhost-expandable` خودکار منتقل نمیشوند.
|
||||
|
||||
---
|
||||
|
||||
## راهاندازی مجدد (reinstall)
|
||||
|
||||
### ۱. حذف Helm
|
||||
|
||||
```bash
|
||||
cd backend/helm/cloudhost-ceph
|
||||
./scripts/uninstall.sh
|
||||
# تایپ: delete-ceph
|
||||
```
|
||||
|
||||
### ۲. پاکسازی روی نود
|
||||
|
||||
```bash
|
||||
sudo losetup -d /dev/loop6 2>/dev/null || true
|
||||
sudo rm -f /var/lib/rook/osd-loopback.img
|
||||
sudo rm -rf /var/lib/rook
|
||||
```
|
||||
|
||||
### ۳. نصب مجدد
|
||||
|
||||
loop device + mirror + `./scripts/install.sh single-node`
|
||||
|
||||
---
|
||||
|
||||
## عیبیابی
|
||||
|
||||
### CephCluster در `Progressing` / Detecting version
|
||||
|
||||
```bash
|
||||
kubectl -n rook-ceph get pods | grep detect-version
|
||||
kubectl -n rook-ceph describe pod -l job-name=rook-ceph-detect-version
|
||||
```
|
||||
|
||||
| خطا | راهحل |
|
||||
|-----|--------|
|
||||
| `ceph/ceph:v19.2 not found` | mirror از quay.io؛ tag صحیح `v19.2` نه `v19.2.1` |
|
||||
| pull timeout | اولین pull بزرگ است (~500MB)؛ صبر یا image را از قبل روی نود بکشید |
|
||||
| Job `detect-version` Terminating گیر کرد | `kubectl -n rook-ceph delete job rook-ceph-detect-version --force --grace-period=0` |
|
||||
|
||||
### OSD بالا نمیآید (OSD count 0)
|
||||
|
||||
```bash
|
||||
kubectl -n rook-ceph logs -l app=rook-ceph-osd-prepare --tail=50
|
||||
```
|
||||
|
||||
| خطا | راهحل |
|
||||
|-----|--------|
|
||||
| `unsupported diskType loop` | `ROOK_CEPH_ALLOW_LOOP_DEVICES=true` |
|
||||
| `not picked by deviceFilter` | از `devices: [{name: "/dev/loop6"}]` استفاده کنید نه `deviceFilter` |
|
||||
| `no devices matched` | `losetup -a` روی نود؛ loop6 وجود دارد؟ |
|
||||
| `directories` در values | در Rook v1.20 کار نمیکند — loop یا raw disk |
|
||||
|
||||
### Volume mount روی rook-ceph-tools
|
||||
|
||||
`rook-ceph-mon-endpoints` و `rook-ceph-mon` تا قبل از بالا آمدن mon ساخته نمیشوند — طبیعی است؛ بعد از Ready برطرف میشود.
|
||||
|
||||
### Helm timeout روی apiserver
|
||||
|
||||
اگر `failed to download openapi` دیدید، بدون `--wait` نصب کنید و با `kubectl get cephcluster` پیگیری کنید.
|
||||
|
||||
### resize PVC
|
||||
|
||||
```bash
|
||||
kubectl get storageclass rook-ceph-block -o yaml | grep allowVolumeExpansion
|
||||
kubectl patch pvc <name> -n <ns> --type merge \
|
||||
-p '{"spec":{"resources":{"requests":{"storage":"5Gi"}}}}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ارتقا (upgrade)
|
||||
|
||||
```bash
|
||||
helm repo update rook-release
|
||||
|
||||
helm upgrade rook-ceph rook-release/rook-ceph -n rook-ceph \
|
||||
--reuse-values --timeout 10m
|
||||
|
||||
helm upgrade rook-ceph-cluster rook-release/rook-ceph-cluster \
|
||||
-n rook-ceph \
|
||||
-f values-rook-cluster-single-node.yaml \
|
||||
--set cephClusterSpec.cephVersion.image=registry.abrban.com/proxy-dockerhub/ceph/ceph:v19.2
|
||||
|
||||
helm upgrade cloudhost-ceph . -n cloudhost-builds -f values.yaml
|
||||
```
|
||||
|
||||
قبل از upgrade در production: [Rook upgrade guide](https://rook.io/docs/rook/latest/Upgrade/ceph-upgrade/) و snapshot.
|
||||
|
||||
---
|
||||
|
||||
## نکات امنیتی
|
||||
|
||||
- RGW داخل کلاستر HTTP است — برای دسترسی خارجی ingress + TLS اضافه کنید.
|
||||
- Secret `ceph-app-sources-credentials` را فقط به backend بدهید.
|
||||
- `single-node` + ۱ OSD فقط staging است؛ production نیاز به ۳+ نود و دیسک جدا دارد.
|
||||
|
||||
---
|
||||
|
||||
## چکلیست بعد از نصب
|
||||
|
||||
- [ ] `ceph status` — mon/mgr/osd/rgw up
|
||||
- [ ] `rook-ceph-block` و `rook-ceph-bucket` در `kubectl get sc`
|
||||
- [ ] `ceph-app-sources-credentials` در `cloudhost-builds` و `cloudhost`
|
||||
- [ ] env بکاند: `PLATFORM_STORAGE_CLASS=rook-ceph-block`
|
||||
- [ ] `SOURCE_STORAGE_*` در backend از secret خوانده میشود
|
||||
- [ ] اپ تست با PVC جدید deploy شده
|
||||
- [ ] ایمیجهای Rook در Harbor موجود و pull تست شده
|
||||
@@ -0,0 +1,195 @@
|
||||
# راهنمای Harbor — `registry.abrban.com`
|
||||
|
||||
این سند معماری فعلی، نصب، مدیریت روزمره و عیبیابی **Harbor** روی کلاستر abr را پوشش میدهد.
|
||||
|
||||
- چارت/values: [`backend/helm/cloudhost-harbor/`](backend/helm/cloudhost-harbor/)
|
||||
- اسکریپت نصب: [`backend/helm/cloudhost-harbor/scripts/install-harbor-registry.sh`](backend/helm/cloudhost-harbor/scripts/install-harbor-registry.sh)
|
||||
|
||||
---
|
||||
|
||||
## معماری فعلی (خلاصه)
|
||||
|
||||
```
|
||||
registry.abrban.com (Traefik + TLS wildcard)
|
||||
├── / → harbor-portal (UI)
|
||||
├── /api/, /service/, /c/ → harbor-core (API + auth)
|
||||
├── /v2/proxy-dockerhub/ → harbor-registry (ایمیجهای mirrorشده Ceph/Rook)
|
||||
├── /v2/rook/ → harbor-registry
|
||||
└── /v2/* → registry قدیمی (ایمیجهای platform: backend, nixpacks, …)
|
||||
```
|
||||
|
||||
| کامپوننت | نقش |
|
||||
|----------|-----|
|
||||
| **Harbor** | UI، proxy-cache، ذخیره ایمیجهای جدید |
|
||||
| **registry قدیمی** (`Deployment/registry`) | هنوز بالاست؛ ایمیجهای platform قبل از Harbor اینجاست |
|
||||
| **registry-egress-proxy** | secret با `HTTP_PROXY` / `HTTPS_PROXY` برای pull از docker.io/quay از داخل کلاستر |
|
||||
| **registry-pull-secret** | auth kubelet برای pull از `registry.abrban.com` |
|
||||
|
||||
> Harbor و registry قدیمی **همزمان** روی یک hostname هستند؛ مسیر `/v2/` با Ingress split میشود.
|
||||
|
||||
---
|
||||
|
||||
## پیشنیازها
|
||||
|
||||
- Secret `abrban-wildcard-tls` در namespace `cloudhost`
|
||||
- Secret `registry-egress-proxy` در namespace `cloudhost` (پروکسی egress)
|
||||
- Helm repo: `helm repo add harbor https://helm.goharbor.io`
|
||||
|
||||
---
|
||||
|
||||
## نصب / ارتقا
|
||||
|
||||
```bash
|
||||
cd backend/helm/cloudhost-harbor
|
||||
./scripts/install-harbor-registry.sh
|
||||
```
|
||||
|
||||
اسکریپت:
|
||||
1. Harbor را با values + proxy از `registry-egress-proxy` نصب میکند
|
||||
2. Ingress قدیمی `registry` را حذف میکند (بعد از نصب باید دستی دوباره route شود — بخش Ingress)
|
||||
3. `Deployment/registry` را scale به 0 میکند (برای rollback نگه داشته میشود)
|
||||
|
||||
### Ingress بعد از نصب (الزامی)
|
||||
|
||||
Harbor به **چند مسیر** نیاز دارد. Ingress نهایی باید شبیه این باشد:
|
||||
|
||||
| Path | Service | Port |
|
||||
|------|---------|------|
|
||||
| `/` | `harbor-portal` | 80 |
|
||||
| `/api/`, `/service/`, `/c/`, `/chartrepo/` | `harbor-core` | 80 |
|
||||
| `/v2/proxy-dockerhub/`, `/v2/rook/` | `harbor-registry` | 5000 |
|
||||
| `/v2/` (بقیه) | `registry` (قدیمی) | 5000 |
|
||||
|
||||
بدون split روی `/v2/`، یا UI 404 میدهد یا kubelet ایمیج platform را پیدا نمیکند.
|
||||
|
||||
---
|
||||
|
||||
## دسترسی و credentialها
|
||||
|
||||
| کاربرد | کاربر | منبع |
|
||||
|--------|-------|------|
|
||||
| UI / API مدیریت | `admin` | `kubectl -n cloudhost get secret harbor-core -o jsonpath='{.data.HARBOR_ADMIN_PASSWORD}' \| base64 -d` |
|
||||
| push/pull داخلی به harbor-registry | `harbor_registry_user` | secret `harbor-core` → `REGISTRY_CREDENTIAL_PASSWORD` |
|
||||
| pull kubelet (ایمیجهای platform) | `cloudhost` | secret `registry-pull-secret` (namespace `cloudhost`) |
|
||||
| pull kubelet (ایمیجهای Rook/Ceph) | `harbor_registry_user` | secret `registry-pull-secret` (namespace `rook-ceph`) |
|
||||
|
||||
URL: https://registry.abrban.com/
|
||||
|
||||
---
|
||||
|
||||
## پروژههای proxy-cache
|
||||
|
||||
| پروژه | upstream | کاربرد |
|
||||
|-------|----------|--------|
|
||||
| `proxy-dockerhub` | docker.io | Rook، Ceph، bitnami، … |
|
||||
| `proxy-quay` | quay.io | cephcsi و … |
|
||||
| `proxy-k8s` | registry.k8s.io | CSI sidecarها |
|
||||
|
||||
ایجاد از UI: **Administration → Registries → New Endpoint** سپس **Projects → New Project** با نوع Proxy Cache.
|
||||
|
||||
> health check بعضی endpointها (مثلاً quay) از UI timeout میخورد؛ از داخل `harbor-core` با curl و proxy ممکن است OK باشد. در صورت نیاز endpoint را با type `docker-registry` بسازید.
|
||||
|
||||
### proxy در Harbor
|
||||
|
||||
پروکسی از secret `registry-egress-proxy` در ConfigMapهای `harbor-core` و `harbor-jobservice-env` تزریق میشود.
|
||||
|
||||
**مهم:** Go (harbor-core/jobservice) به `http_proxy` / `https_proxy` **lowercase** هم نیاز دارد. اگر health check upstream `unhealthy` ماند، هر دو حالت uppercase و lowercase را در ConfigMap بگذارید و podها را restart کنید:
|
||||
|
||||
```bash
|
||||
kubectl -n cloudhost rollout restart deploy/harbor-core deploy/harbor-jobservice
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## mirror دستی ایمیج (وقتی proxy-cache کار نمیکند)
|
||||
|
||||
روی abr، pull مستقیم از docker.io/quay از kubelet ممکن نیست. ایمیجهای حیاتی را با Job داخل کلاستر mirror کنید:
|
||||
|
||||
**مقصد push:** `harbor-registry.cloudhost.svc.cluster.local:5000` (HTTP، با `harbor_registry_user`)
|
||||
|
||||
**مثال مسیرها در registry:**
|
||||
|
||||
| ایمیج upstream | مسیر در registry |
|
||||
|----------------|------------------|
|
||||
| `rook/ceph:v1.20.1` | `rook/ceph:v1.20.1` |
|
||||
| `quay.io/ceph/ceph:v19.2` | `proxy-dockerhub/ceph/ceph:v19.2` |
|
||||
|
||||
```bash
|
||||
# لیست ایمیجهای داخل harbor-registry
|
||||
kubectl -n cloudhost exec deploy/harbor-portal -- \
|
||||
curl -s -u "harbor_registry_user:$(kubectl -n cloudhost get secret harbor-core -o jsonpath='{.data.REGISTRY_CREDENTIAL_PASSWORD}' | base64 -d)" \
|
||||
http://harbor-registry:5000/v2/_catalog
|
||||
```
|
||||
|
||||
> push مستقیم به `harbor-registry:5000` metadata در Harbor UI را بهروز نمیکند؛ برای kubelet کافی است چون `/v2/` به harbor-registry route شده.
|
||||
|
||||
---
|
||||
|
||||
## مدیریت روزمره
|
||||
|
||||
### وضعیت
|
||||
|
||||
```bash
|
||||
kubectl -n cloudhost get pods | grep harbor
|
||||
kubectl -n cloudhost get ingress registry -o yaml | grep -A3 'path:'
|
||||
curl -sk -o /dev/null -w "%{http_code}\n" https://registry.abrban.com/
|
||||
curl -sk -u admin:<pass> https://registry.abrban.com/api/v2.0/systeminfo
|
||||
```
|
||||
|
||||
### لاگها
|
||||
|
||||
```bash
|
||||
kubectl -n cloudhost logs deploy/harbor-core --tail=50
|
||||
kubectl -n cloudhost logs deploy/harbor-jobservice --tail=50
|
||||
kubectl -n cloudhost logs deploy/harbor-registry -c registry --tail=50
|
||||
```
|
||||
|
||||
### ارتقا Harbor
|
||||
|
||||
```bash
|
||||
helm repo update harbor
|
||||
./scripts/install-harbor-registry.sh
|
||||
# Ingress split را دوباره تأیید کنید
|
||||
```
|
||||
|
||||
### rollback به registry قدیمی
|
||||
|
||||
```bash
|
||||
kubectl -n cloudhost scale deploy/registry --replicas=1
|
||||
# Ingress را فقط به service registry:5000 برگردانید
|
||||
helm uninstall harbor -n cloudhost
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## عیبیابی
|
||||
|
||||
| علامت | علت محتمل | اقدام |
|
||||
|-------|-----------|--------|
|
||||
| `https://registry.abrban.com/` → 404 | Ingress فقط به `harbor-core` وصل است | `/` → `harbor-portal` |
|
||||
| `ImagePullBackOff` برای `cloudhost-backend` | `/v2/` به Harbor رفته، ایمیج platform آنجا نیست | `/v2/` (عمومی) → `registry` قدیمی |
|
||||
| `not found` برای `proxy-dockerhub/...` | ایمیج mirror نشده | Job skopeo یا proxy-cache |
|
||||
| push با 499/503 | Traefik timeout | push از داخل کلاستر به `harbor-registry:5000` |
|
||||
| registry endpoint `unhealthy` | proxy lowercase یا timeout health check | patch ConfigMap + restart؛ یا mirror دستی |
|
||||
| `authentication required` روی pull | pull secret اشتباه namespace | `cloudhost` vs `harbor_registry_user` در `rook-ceph` |
|
||||
|
||||
### تست pull
|
||||
|
||||
```bash
|
||||
# platform (cloudhost user)
|
||||
curl -sk -u "cloudhost:<pass>" https://registry.abrban.com/v2/cloudhost-backend/tags/list
|
||||
|
||||
# rook/ceph (harbor_registry_user)
|
||||
curl -sk -u "harbor_registry_user:<pass>" https://registry.abrban.com/v2/rook/ceph/tags/list
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## چکلیست بعد از نصب
|
||||
|
||||
- [ ] Portal روی `/` پاسخ 200
|
||||
- [ ] `/api/v2.0/systeminfo` پاسخ JSON
|
||||
- [ ] Ingress split `/v2/` درست است
|
||||
- [ ] پروژههای `proxy-dockerhub`, `proxy-k8s`, `proxy-quay` ساخته شده
|
||||
- [ ] `registry-pull-secret` در `cloudhost` و `rook-ceph` بهروز است
|
||||
- [ ] ایمیجهای Rook/Ceph mirror شده و pull تست شده
|
||||
+3
-2
@@ -21,7 +21,8 @@
|
||||
| **Backend** | NestJS (REST `/api/v1`) | منطق کسبوکار، ساخت اپ، احراز هویت |
|
||||
| **Postgres** | postgres:16 | دیتابیس اصلی (کاربر، اپ، فاکتور، …) |
|
||||
| **Redis** | redis:7 | کش، صف Bull (مهاجرت اپ، دسترسی موقت)، پیشرفت بیلد |
|
||||
| **Registry داخلی** | registry:2 | ایمیجهای buildشده |
|
||||
| **Registry داخلی** | Harbor + registry:2 (legacy) | ایمیجهای build و platform؛ جزئیات: [`RUNBOOK-HARBOR.fa.md`](RUNBOOK-HARBOR.fa.md) |
|
||||
| **Storage (Ceph)** | Rook-Ceph | PVC (`rook-ceph-block`) + bucket zip (`rook-ceph-bucket`)؛ جزئیات: [`RUNBOOK-CEPH.fa.md`](RUNBOOK-CEPH.fa.md) |
|
||||
| **Build pipeline** | Kaniko | تبدیل سورس به ایمیج Docker داخل کلاستر (بدون Docker daemon) |
|
||||
| **Kubernetes** | k3s (تکنود) | اجرای همهی موارد بالا + اپهای کاربر |
|
||||
|
||||
@@ -29,7 +30,7 @@
|
||||
- `abrban.com` → لندینگ
|
||||
- `panel.abrban.com` → پنل احرازشده
|
||||
- `api.abrban.com` → بکاند
|
||||
- `registry.abrban.com` → رجیستری داخلی (pull توسط kubelet)
|
||||
- `registry.abrban.com` → Harbor (UI + proxy-cache) + registry قدیمی برای ایمیجهای platform — [`RUNBOOK-HARBOR.fa.md`](RUNBOOK-HARBOR.fa.md)
|
||||
- `apps.abrban.com` → دامنهی پیشفرض اپهای کاربر
|
||||
|
||||
### ۱.۴ جریان احراز هویت
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
.rook-cluster-values-ref.yaml
|
||||
*.md
|
||||
scripts/
|
||||
@@ -0,0 +1,758 @@
|
||||
# Default values for a single rook-ceph cluster
|
||||
# This is a YAML-formatted file.
|
||||
# Declare variables to be passed into your templates.
|
||||
|
||||
# -- Namespace of the main rook operator
|
||||
operatorNamespace: rook-ceph
|
||||
|
||||
# -- The metadata.name of the CephCluster CR
|
||||
# @default -- The same as the namespace
|
||||
clusterName:
|
||||
|
||||
# -- Optional override of the target kubernetes version
|
||||
kubeVersion:
|
||||
|
||||
# -- Cluster ceph.conf override
|
||||
configOverride:
|
||||
# configOverride: |
|
||||
# [global]
|
||||
# mon_allow_pool_delete = true
|
||||
# osd_pool_default_size = 3
|
||||
# osd_pool_default_min_size = 2
|
||||
|
||||
# Installs a debugging toolbox deployment
|
||||
toolbox:
|
||||
# -- Enable Ceph debugging pod deployment. See [toolbox](../Troubleshooting/ceph-toolbox.md)
|
||||
enabled: false
|
||||
# -- Toolbox image, defaults to the image used by the Ceph cluster
|
||||
image: #quay.io/ceph/ceph:v20.2.1
|
||||
# -- Toolbox tolerations
|
||||
tolerations: []
|
||||
# -- Toolbox affinity
|
||||
affinity: {}
|
||||
# -- Toolbox labels
|
||||
labels: {}
|
||||
# -- Toolbox container security context
|
||||
containerSecurityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 2016
|
||||
runAsGroup: 2016
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
# -- Toolbox resources
|
||||
resources:
|
||||
limits:
|
||||
memory: "1Gi"
|
||||
requests:
|
||||
cpu: "100m"
|
||||
memory: "128Mi"
|
||||
# -- Set the priority class for the toolbox if desired
|
||||
priorityClassName:
|
||||
|
||||
monitoring:
|
||||
# -- Enable Prometheus integration, will also create necessary RBAC rules to allow Operator to create ServiceMonitors.
|
||||
# Monitoring requires Prometheus to be pre-installed
|
||||
enabled: false
|
||||
# -- Whether to disable the metrics reported by Ceph. If false, the prometheus mgr module and Ceph exporter are enabled
|
||||
metricsDisabled: false
|
||||
# -- Whether to create the Prometheus rules for Ceph alerts
|
||||
createPrometheusRules: false
|
||||
# -- Edit Prometheus rules for Ceph alerts
|
||||
prometheusRuleOverrides: {}
|
||||
# CephHealthWarning:
|
||||
# disabled: true
|
||||
# NVMeoFHighWriteLatency:
|
||||
# for: 3m
|
||||
# labels:
|
||||
# severity: critical
|
||||
# -- The namespace in which to create the prometheus rules, if different from the rook cluster namespace.
|
||||
# If you have multiple rook-ceph clusters in the same k8s cluster, choose the same namespace (ideally, namespace with prometheus
|
||||
# deployed) to set rulesNamespaceOverride for all the clusters. Otherwise, you will get duplicate alerts with multiple alert definitions.
|
||||
rulesNamespaceOverride:
|
||||
# Monitoring settings for external clusters:
|
||||
# externalMgrEndpoints: <list of endpoints>
|
||||
# externalMgrPrometheusPort: <port>
|
||||
# Scrape interval for prometheus
|
||||
# interval: 10s
|
||||
# allow adding custom labels and annotations to the prometheus rule
|
||||
prometheusRule:
|
||||
# -- Labels applied to PrometheusRule
|
||||
labels: {}
|
||||
# -- Annotations applied to PrometheusRule
|
||||
annotations: {}
|
||||
|
||||
# imagePullSecrets option allow to pull docker images from private docker registry. Option will be passed to all service accounts.
|
||||
# imagePullSecrets:
|
||||
# - name: my-registry-secret
|
||||
|
||||
# Labels and annotations to add to the CephCluster CR
|
||||
cephClusterMetadata:
|
||||
annotations: {}
|
||||
labels: {}
|
||||
|
||||
# Specify these values to override the Ceph image in the cephClusterSpec below.
|
||||
# If specifying these values, do not include the cephVersion section in the cephClusterSpec.
|
||||
cephImage:
|
||||
# The repository from which to pull the ceph image
|
||||
repository: quay.io/ceph/ceph
|
||||
# In production, use a specific version tag instead of the general v20 flag, which pulls the latest release and could result in different
|
||||
# versions running within the cluster. See tags available at https://hub.docker.com/r/ceph/ceph/tags/.
|
||||
# To be more precise, you can always use a timestamp tag such as quay.io/ceph/ceph:v20.2.1-20260402
|
||||
tag: v20.2.1
|
||||
# Whether to allow unsupported versions of Ceph. Currently Squid and Tentacle are supported.
|
||||
# Future versions would require this to be set to `true`.
|
||||
# Do not set to true in production.
|
||||
allowUnsupported: false
|
||||
# The image pull policy for pulling the ceph image in the ceph daemon pods, defaults to IfNotPresent
|
||||
# imagePullPolicy: IfNotPresent
|
||||
|
||||
# All values below are taken from the CephCluster CRD
|
||||
# -- Cluster configuration.
|
||||
# @default -- See [below](#ceph-cluster-spec)
|
||||
cephClusterSpec:
|
||||
# This cluster spec example is for a converged cluster where all the Ceph daemons are running locally,
|
||||
# as in the host-based example (cluster.yaml). For a different configuration such as a
|
||||
# PVC-based cluster (cluster-on-pvc.yaml), external cluster (cluster-external.yaml),
|
||||
# or stretch cluster (cluster-stretched.yaml), replace this entire `cephClusterSpec`
|
||||
# with the specs from those examples.
|
||||
# For more details, check https://rook.io/docs/rook/v1.10/CRDs/Cluster/ceph-cluster-crd/
|
||||
|
||||
# The path on the host where configuration files will be persisted. Must be specified. If there are multiple clusters, the directory must be unique for each cluster.
|
||||
# Important: if you reinstall the cluster, make sure you delete this directory from each host or else the mons will fail to start on the new cluster.
|
||||
# In Minikube, the '/data' directory is configured to persist across reboots. Use "/data/rook" in Minikube environment.
|
||||
dataDirHostPath: /var/lib/rook
|
||||
|
||||
# Whether or not upgrade should continue even if a check fails
|
||||
# This means Ceph's status could be degraded and we don't recommend upgrading but you might decide otherwise
|
||||
# Use at your OWN risk
|
||||
# To understand Rook's upgrade process of Ceph, read https://rook.io/docs/rook/v1.10/Upgrade/ceph-upgrade/
|
||||
skipUpgradeChecks: false
|
||||
|
||||
# Whether or not continue if PGs are not clean during an upgrade
|
||||
continueUpgradeAfterChecksEvenIfNotHealthy: false
|
||||
|
||||
# WaitTimeoutForHealthyOSDInMinutes defines the time (in minutes) the operator would wait before an OSD can be stopped for upgrade or restart.
|
||||
# If the timeout exceeds and OSD is not ok to stop, then the operator would skip upgrade for the current OSD and proceed with the next one
|
||||
# if `continueUpgradeAfterChecksEvenIfNotHealthy` is `false`. If `continueUpgradeAfterChecksEvenIfNotHealthy` is `true`, then operator would
|
||||
# continue with the upgrade of an OSD even if its not ok to stop after the timeout. This timeout won't be applied if `skipUpgradeChecks` is `true`.
|
||||
# The default wait timeout is 10 minutes.
|
||||
waitTimeoutForHealthyOSDInMinutes: 10
|
||||
|
||||
# Whether or not requires PGs are clean before an OSD upgrade. If set to `true` OSD upgrade process won't start until PGs are healthy.
|
||||
# This configuration will be ignored if `skipUpgradeChecks` is `true`.
|
||||
# Default is false.
|
||||
upgradeOSDRequiresHealthyPGs: false
|
||||
|
||||
mon:
|
||||
# Set the number of mons to be started. Generally recommended to be 3.
|
||||
# For highest availability, an odd number of mons should be specified.
|
||||
count: 3
|
||||
# The mons should be on unique nodes. For production, at least 3 nodes are recommended for this reason.
|
||||
# Mons should only be allowed on the same node for test environments where data loss is acceptable.
|
||||
allowMultiplePerNode: false
|
||||
|
||||
mgr:
|
||||
# When higher availability of the mgr is needed, increase the count to 2.
|
||||
# In that case, one mgr will be active and one in standby. When Ceph updates which
|
||||
# mgr is active, Rook will update the mgr services to match the active mgr.
|
||||
count: 2
|
||||
allowMultiplePerNode: false
|
||||
modules:
|
||||
# List of modules to optionally enable or disable.
|
||||
# Note the "dashboard" and "monitoring" modules are already configured by other settings in the cluster CR.
|
||||
# - name: rook
|
||||
# enabled: true
|
||||
|
||||
# enable the ceph dashboard for viewing cluster status
|
||||
dashboard:
|
||||
enabled: true
|
||||
# serve the dashboard under a subpath (useful when you are accessing the dashboard via a reverse proxy)
|
||||
# urlPrefix: /ceph-dashboard
|
||||
# serve the dashboard at the given port.
|
||||
# port: 8443
|
||||
# Serve the dashboard using SSL (if using ingress to expose the dashboard and `ssl: true` you need to set
|
||||
# the corresponding "backend protocol" annotation(s) for your ingress controller of choice)
|
||||
ssl: true
|
||||
|
||||
# Network configuration, see: https://github.com/rook/rook/blob/master/Documentation/CRDs/Cluster/ceph-cluster-crd.md#network-configuration-settings
|
||||
network:
|
||||
connections:
|
||||
# Whether to encrypt the data in transit across the wire to prevent eavesdropping the data on the network.
|
||||
# The default is false. When encryption is enabled, all communication between clients and Ceph daemons, or between Ceph daemons will be encrypted.
|
||||
# When encryption is not enabled, clients still establish a strong initial authentication and data integrity is still validated with a crc check.
|
||||
# IMPORTANT: Encryption requires the 5.11 kernel for the latest nbd and cephfs drivers. Alternatively for testing only,
|
||||
# you can set the "mounter: rbd-nbd" in the rbd storage class, or "mounter: fuse" in the cephfs storage class.
|
||||
# The nbd and fuse drivers are *not* recommended in production since restarting the csi driver pod will disconnect the volumes.
|
||||
encryption:
|
||||
enabled: false
|
||||
# Whether to compress the data in transit across the wire. The default is false.
|
||||
# The kernel requirements above for encryption also apply to compression.
|
||||
compression:
|
||||
enabled: false
|
||||
# Whether to require communication over msgr2. If true, the msgr v1 port (6789) will be disabled
|
||||
# and clients will be required to connect to the Ceph cluster with the v2 port (3300).
|
||||
# Requires a kernel that supports msgr v2 (kernel 5.11 or CentOS 8.4 or newer).
|
||||
requireMsgr2: false
|
||||
# # enable host networking
|
||||
# provider: host
|
||||
# # EXPERIMENTAL: enable the Multus network provider
|
||||
# provider: multus
|
||||
# selectors:
|
||||
# # The selector keys are required to be `public` and `cluster`.
|
||||
# # Based on the configuration, the operator will do the following:
|
||||
# # 1. if only the `public` selector key is specified both public_network and cluster_network Ceph settings will listen on that interface
|
||||
# # 2. if both `public` and `cluster` selector keys are specified the first one will point to 'public_network' flag and the second one to 'cluster_network'
|
||||
# #
|
||||
# # In order to work, each selector value must match a NetworkAttachmentDefinition object in Multus
|
||||
# #
|
||||
# # public: public-conf --> NetworkAttachmentDefinition object name in Multus
|
||||
# # cluster: cluster-conf --> NetworkAttachmentDefinition object name in Multus
|
||||
# # Provide internet protocol version. IPv6, IPv4 or empty string are valid options. Empty string would mean IPv4
|
||||
# ipFamily: "IPv6"
|
||||
# # Ceph daemons to listen on both IPv4 and Ipv6 networks
|
||||
# dualStack: false
|
||||
|
||||
# enable the crash collector for ceph daemon crash collection
|
||||
crashCollector:
|
||||
disable: false
|
||||
# Uncomment daysToRetain to prune ceph crash entries older than the
|
||||
# specified number of days.
|
||||
# daysToRetain: 30
|
||||
|
||||
# enable log collector, daemons will log on files and rotate
|
||||
logCollector:
|
||||
enabled: true
|
||||
periodicity: daily # one of: hourly, daily, weekly, monthly
|
||||
maxLogSize: 500M # SUFFIX may be 'M' or 'G'. Must be at least 1M.
|
||||
|
||||
# automate [data cleanup process](https://github.com/rook/rook/blob/master/Documentation/Storage-Configuration/ceph-teardown.md#delete-the-data-on-hosts) in cluster destruction.
|
||||
cleanupPolicy:
|
||||
# Since cluster cleanup is destructive to data, confirmation is required.
|
||||
# To destroy all Rook data on hosts during uninstall, confirmation must be set to "yes-really-destroy-data".
|
||||
# This value should only be set when the cluster is about to be deleted. After the confirmation is set,
|
||||
# Rook will immediately stop configuring the cluster and only wait for the delete command.
|
||||
# If the empty string is set, Rook will not destroy any data on hosts during uninstall.
|
||||
confirmation: ""
|
||||
# sanitizeDisks represents settings for sanitizing OSD disks on cluster deletion
|
||||
sanitizeDisks:
|
||||
# method indicates if the entire disk should be sanitized or simply ceph's metadata
|
||||
# in both case, re-install is possible
|
||||
# possible choices are 'complete' or 'quick' (default)
|
||||
method: quick
|
||||
# dataSource indicate where to get random bytes from to write on the disk
|
||||
# possible choices are 'zero' (default) or 'random'
|
||||
# using random sources will consume entropy from the system and will take much more time then the zero source
|
||||
dataSource: zero
|
||||
# iteration overwrite N times instead of the default (1)
|
||||
# takes an integer value
|
||||
iteration: 1
|
||||
# allowUninstallWithVolumes defines how the uninstall should be performed
|
||||
# If set to true, cephCluster deletion does not wait for the PVs to be deleted.
|
||||
allowUninstallWithVolumes: false
|
||||
|
||||
# To control where various services will be scheduled by kubernetes, use the placement configuration sections below.
|
||||
# The example under 'all' would have all services scheduled on kubernetes nodes labeled with 'role=storage-node' and
|
||||
# tolerate taints with a key of 'storage-node'.
|
||||
# placement:
|
||||
# all:
|
||||
# nodeAffinity:
|
||||
# requiredDuringSchedulingIgnoredDuringExecution:
|
||||
# nodeSelectorTerms:
|
||||
# - matchExpressions:
|
||||
# - key: role
|
||||
# operator: In
|
||||
# values:
|
||||
# - storage-node
|
||||
# podAffinity:
|
||||
# podAntiAffinity:
|
||||
# topologySpreadConstraints:
|
||||
# tolerations:
|
||||
# - key: storage-node
|
||||
# operator: Exists
|
||||
# # The above placement information can also be specified for mon, osd, and mgr components
|
||||
# mon:
|
||||
# # Monitor deployments may contain an anti-affinity rule for avoiding monitor
|
||||
# # collocation on the same node. This is a required rule when host network is used
|
||||
# # or when AllowMultiplePerNode is false. Otherwise this anti-affinity rule is a
|
||||
# # preferred rule with weight: 50.
|
||||
# osd:
|
||||
# mgr:
|
||||
# cleanup:
|
||||
|
||||
# annotations:
|
||||
# all:
|
||||
# mon:
|
||||
# osd:
|
||||
# cleanup:
|
||||
# prepareosd:
|
||||
# # If no mgr annotations are set, prometheus scrape annotations will be set by default.
|
||||
# mgr:
|
||||
# dashboard:
|
||||
|
||||
# labels:
|
||||
# all:
|
||||
# mon:
|
||||
# osd:
|
||||
# cleanup:
|
||||
# mgr:
|
||||
# prepareosd:
|
||||
# # monitoring is a list of key-value pairs. It is injected into all the monitoring resources created by operator.
|
||||
# # These labels can be passed as LabelSelector to Prometheus
|
||||
# monitoring:
|
||||
# dashboard:
|
||||
|
||||
resources:
|
||||
mgr:
|
||||
limits:
|
||||
memory: "1Gi"
|
||||
requests:
|
||||
cpu: "500m"
|
||||
memory: "512Mi"
|
||||
mon:
|
||||
limits:
|
||||
memory: "2Gi"
|
||||
requests:
|
||||
cpu: "1000m"
|
||||
memory: "1Gi"
|
||||
osd:
|
||||
limits:
|
||||
memory: "4Gi"
|
||||
requests:
|
||||
cpu: "1000m"
|
||||
memory: "4Gi"
|
||||
prepareosd:
|
||||
# limits: It is not recommended to set limits on the OSD prepare job
|
||||
# since it's a one-time burst for memory that must be allowed to
|
||||
# complete without an OOM kill. Note however that if a k8s
|
||||
# limitRange guardrail is defined external to Rook, the lack of
|
||||
# a limit here may result in a sync failure, in which case a
|
||||
# limit should be added. 1200Mi may suffice for up to 15Ti
|
||||
# OSDs ; for larger devices 2Gi may be required.
|
||||
# cf. https://github.com/rook/rook/pull/11103
|
||||
requests:
|
||||
cpu: "500m"
|
||||
memory: "50Mi"
|
||||
mgr-sidecar:
|
||||
limits:
|
||||
memory: "100Mi"
|
||||
requests:
|
||||
cpu: "100m"
|
||||
memory: "40Mi"
|
||||
crashcollector:
|
||||
limits:
|
||||
memory: "60Mi"
|
||||
requests:
|
||||
cpu: "100m"
|
||||
memory: "60Mi"
|
||||
logcollector:
|
||||
limits:
|
||||
memory: "1Gi"
|
||||
requests:
|
||||
cpu: "100m"
|
||||
memory: "100Mi"
|
||||
cleanup:
|
||||
limits:
|
||||
memory: "1Gi"
|
||||
requests:
|
||||
cpu: "500m"
|
||||
memory: "100Mi"
|
||||
exporter:
|
||||
limits:
|
||||
memory: "128Mi"
|
||||
requests:
|
||||
cpu: "50m"
|
||||
memory: "50Mi"
|
||||
cmd-reporter:
|
||||
limits:
|
||||
memory: "1Gi"
|
||||
requests:
|
||||
cpu: "500m"
|
||||
memory: "100Mi"
|
||||
|
||||
# The option to automatically remove OSDs that are out and are safe to destroy.
|
||||
removeOSDsIfOutAndSafeToRemove: false
|
||||
|
||||
# priority classes to apply to ceph resources
|
||||
priorityClassNames:
|
||||
mon: system-node-critical
|
||||
osd: system-node-critical
|
||||
mgr: system-cluster-critical
|
||||
|
||||
storage: # cluster level storage configuration and selection
|
||||
useAllNodes: true
|
||||
useAllDevices: true
|
||||
# deviceFilter:
|
||||
# config:
|
||||
# crushRoot: "custom-root" # specify a non-default root label for the CRUSH map
|
||||
# metadataDevice: "md0" # specify a non-rotational storage so ceph-volume will use it as block db device of bluestore.
|
||||
# databaseSizeMB: "1024" # uncomment if the disks are smaller than 100 GB
|
||||
# osdsPerDevice: "1" # this value can be overridden at the node or device level
|
||||
# encryptedDevice: "true" # the default value for this option is "false"
|
||||
# # Individual nodes and their config can be specified as well, but 'useAllNodes' above must be set to false. Then, only the named
|
||||
# # nodes below will be used as storage resources. Each node's 'name' field should match their 'kubernetes.io/hostname' label.
|
||||
# nodes:
|
||||
# - name: "172.17.4.201"
|
||||
# devices: # specific devices to use for storage can be specified for each node
|
||||
# - name: "sdb"
|
||||
# - name: "nvme01" # multiple osds can be created on high performance devices
|
||||
# config:
|
||||
# osdsPerDevice: "5"
|
||||
# - name: "/dev/disk/by-id/ata-ST4000DM004-XXXX" # devices can be specified using full udev paths
|
||||
# config: # configuration can be specified at the node level which overrides the cluster level config
|
||||
# - name: "172.17.4.301"
|
||||
# deviceFilter: "^sd."
|
||||
|
||||
# The section for configuring management of daemon disruptions during upgrade or fencing.
|
||||
disruptionManagement:
|
||||
# If true, the operator will create and manage PodDisruptionBudgets for OSD, Mon, RGW, and MDS daemons. OSD PDBs are managed dynamically
|
||||
# via the strategy outlined in the [design](https://github.com/rook/rook/blob/master/design/ceph/ceph-managed-disruptionbudgets.md). The operator will
|
||||
# block eviction of OSDs by default and unblock them safely when drains are detected.
|
||||
managePodBudgets: true
|
||||
# A duration in minutes that determines how long an entire failureDomain like `region/zone/host` will be held in `noout` (in addition to the
|
||||
# default DOWN/OUT interval) when it is draining. This is only relevant when `managePodBudgets` is `true`. The default value is `30` minutes.
|
||||
osdMaintenanceTimeout: 30
|
||||
|
||||
# Configure the healthcheck and liveness probes for ceph pods.
|
||||
# Valid values for daemons are 'mon', 'osd', 'status'
|
||||
healthCheck:
|
||||
daemonHealth:
|
||||
mon:
|
||||
disabled: false
|
||||
interval: 45s
|
||||
osd:
|
||||
disabled: false
|
||||
interval: 60s
|
||||
status:
|
||||
disabled: false
|
||||
interval: 60s
|
||||
# Change pod liveness probe, it works for all mon, mgr, and osd pods.
|
||||
livenessProbe:
|
||||
mon:
|
||||
disabled: false
|
||||
mgr:
|
||||
disabled: false
|
||||
osd:
|
||||
disabled: false
|
||||
|
||||
ingress:
|
||||
# -- Enable an ingress for the ceph-dashboard
|
||||
dashboard: {}
|
||||
# labels:
|
||||
# external-dns/private: "true"
|
||||
# annotations:
|
||||
# external-dns.alpha.kubernetes.io/hostname: dashboard.example.com
|
||||
# nginx.ingress.kubernetes.io/rewrite-target: /ceph-dashboard/$2
|
||||
# If the dashboard has ssl: true the following will make sure the NGINX Ingress controller can expose the dashboard correctly
|
||||
# nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
|
||||
# nginx.ingress.kubernetes.io/server-snippet: |
|
||||
# proxy_ssl_verify off;
|
||||
# host:
|
||||
# name: dashboard.example.com
|
||||
# path: "/ceph-dashboard(/|$)(.*)"
|
||||
# pathType: Prefix
|
||||
# tls:
|
||||
# - hosts:
|
||||
# - dashboard.example.com
|
||||
# secretName: testsecret-tls
|
||||
## Note: Only one of ingress class annotation or the `ingressClassName:` can be used at a time
|
||||
## to set the ingress class
|
||||
# ingressClassName: nginx
|
||||
|
||||
route:
|
||||
# -- Enable an HTTPRoute for the ceph-dashboard
|
||||
dashboard: {}
|
||||
# labels:
|
||||
# external-dns/private: "true"
|
||||
# annotations:
|
||||
# external-dns.alpha.kubernetes.io/hostname: dashboard.example.com
|
||||
# nginx.ingress.kubernetes.io/rewrite-target: /ceph-dashboard/$2
|
||||
# host:
|
||||
# name: dashboard.example.com
|
||||
# path: "/"
|
||||
# pathType: PathPrefix
|
||||
# parentRefs:
|
||||
# - name: internal
|
||||
# namespace: kube-system
|
||||
# sectionName: https
|
||||
|
||||
# -- A list of CephBlockPool configurations to deploy
|
||||
# @default -- See [below](#ceph-block-pools)
|
||||
cephBlockPools:
|
||||
- name: ceph-blockpool
|
||||
# see https://github.com/rook/rook/blob/master/Documentation/CRDs/Block-Storage/ceph-block-pool-crd.md#spec for available configuration
|
||||
spec:
|
||||
failureDomain: host
|
||||
replicated:
|
||||
size: 3
|
||||
# Enables collecting RBD per-image IO statistics by enabling dynamic OSD performance counters. Defaults to false.
|
||||
# For reference: https://docs.ceph.com/docs/latest/mgr/prometheus/#rbd-io-statistics
|
||||
# enableRBDStats: true
|
||||
storageClass:
|
||||
enabled: true
|
||||
name: ceph-block
|
||||
annotations: {}
|
||||
labels: {}
|
||||
isDefault: true
|
||||
reclaimPolicy: Delete
|
||||
allowVolumeExpansion: true
|
||||
volumeBindingMode: "Immediate"
|
||||
mountOptions: []
|
||||
# see https://kubernetes.io/docs/concepts/storage/storage-classes/#allowed-topologies
|
||||
allowedTopologies: []
|
||||
# - matchLabelExpressions:
|
||||
# - key: rook-ceph-role
|
||||
# values:
|
||||
# - storage-node
|
||||
# see https://github.com/rook/rook/blob/master/Documentation/Storage-Configuration/Block-Storage-RBD/block-storage.md#provision-storage for available configuration
|
||||
parameters:
|
||||
# (optional) mapOptions is a comma-separated list of map options.
|
||||
# For krbd options refer
|
||||
# https://docs.ceph.com/docs/latest/man/8/rbd/#kernel-rbd-krbd-options
|
||||
# For nbd options refer
|
||||
# https://docs.ceph.com/docs/latest/man/8/rbd-nbd/#options
|
||||
# mapOptions: lock_on_read,queue_depth=1024
|
||||
|
||||
# (optional) unmapOptions is a comma-separated list of unmap options.
|
||||
# For krbd options refer
|
||||
# https://docs.ceph.com/docs/latest/man/8/rbd/#kernel-rbd-krbd-options
|
||||
# For nbd options refer
|
||||
# https://docs.ceph.com/docs/latest/man/8/rbd-nbd/#options
|
||||
# unmapOptions: force
|
||||
|
||||
# RBD image format. Defaults to "2".
|
||||
imageFormat: "2"
|
||||
|
||||
# RBD image features, equivalent to OR'd bitfield value: 63
|
||||
# Available for imageFormat: "2". Older releases of CSI RBD
|
||||
# support only the `layering` feature. The Linux kernel (KRBD) supports the
|
||||
# full feature complement as of 5.4
|
||||
imageFeatures: layering
|
||||
|
||||
# These secrets contain Ceph admin credentials.
|
||||
csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
|
||||
csi.storage.k8s.io/provisioner-secret-namespace: "{{ .Release.Namespace }}"
|
||||
csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
|
||||
csi.storage.k8s.io/controller-expand-secret-namespace: "{{ .Release.Namespace }}"
|
||||
csi.storage.k8s.io/controller-publish-secret-name: rook-csi-rbd-provisioner
|
||||
csi.storage.k8s.io/controller-publish-secret-namespace: "{{ .Release.Namespace }}"
|
||||
csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
|
||||
csi.storage.k8s.io/node-stage-secret-namespace: "{{ .Release.Namespace }}"
|
||||
# Specify the filesystem type of the volume. If not specified, csi-provisioner
|
||||
# will set default as `ext4`. Note that `xfs` is not recommended due to potential deadlock
|
||||
# in hyperconverged settings where the volume is mounted on the same node as the osds.
|
||||
csi.storage.k8s.io/fstype: ext4
|
||||
|
||||
# -- A list of CephFileSystem configurations to deploy
|
||||
# @default -- See [below](#ceph-file-systems)
|
||||
cephFileSystems:
|
||||
- name: ceph-filesystem
|
||||
# see https://github.com/rook/rook/blob/master/Documentation/CRDs/Shared-Filesystem/ceph-filesystem-crd.md#filesystem-settings for available configuration
|
||||
spec:
|
||||
metadataPool:
|
||||
replicated:
|
||||
size: 3
|
||||
dataPools:
|
||||
- failureDomain: host
|
||||
replicated:
|
||||
size: 3
|
||||
# Optional and highly recommended, 'data0' by default, see https://github.com/rook/rook/blob/master/Documentation/CRDs/Shared-Filesystem/ceph-filesystem-crd.md#pools
|
||||
name: data0
|
||||
metadataServer:
|
||||
activeCount: 1
|
||||
activeStandby: true
|
||||
resources:
|
||||
limits:
|
||||
memory: "4Gi"
|
||||
requests:
|
||||
cpu: "1000m"
|
||||
memory: "4Gi"
|
||||
priorityClassName: system-cluster-critical
|
||||
storageClass:
|
||||
enabled: true
|
||||
isDefault: false
|
||||
name: ceph-filesystem
|
||||
# (Optional) specify a data pool to use, must be the name of one of the data pools above, 'data0' by default
|
||||
pool: data0
|
||||
reclaimPolicy: Delete
|
||||
allowVolumeExpansion: true
|
||||
volumeBindingMode: "Immediate"
|
||||
annotations: {}
|
||||
labels: {}
|
||||
mountOptions: []
|
||||
# see https://github.com/rook/rook/blob/master/Documentation/Storage-Configuration/Shared-Filesystem-CephFS/filesystem-storage.md#provision-storage for available configuration
|
||||
parameters:
|
||||
# The secrets contain Ceph admin credentials.
|
||||
csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
|
||||
csi.storage.k8s.io/provisioner-secret-namespace: "{{ .Release.Namespace }}"
|
||||
csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
|
||||
csi.storage.k8s.io/controller-expand-secret-namespace: "{{ .Release.Namespace }}"
|
||||
csi.storage.k8s.io/controller-publish-secret-name: rook-csi-cephfs-provisioner
|
||||
csi.storage.k8s.io/controller-publish-secret-namespace: "{{ .Release.Namespace }}"
|
||||
csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
|
||||
csi.storage.k8s.io/node-stage-secret-namespace: "{{ .Release.Namespace }}"
|
||||
# Specify the filesystem type of the volume. If not specified, csi-provisioner
|
||||
# will set default as `ext4`. Note that `xfs` is not recommended due to potential deadlock
|
||||
# in hyperconverged settings where the volume is mounted on the same node as the osds.
|
||||
csi.storage.k8s.io/fstype: ext4
|
||||
|
||||
# -- Settings for the filesystem snapshot class
|
||||
# @default -- See [CephFS Snapshots](../Storage-Configuration/Ceph-CSI/ceph-csi-snapshot.md#cephfs-snapshots)
|
||||
cephFileSystemVolumeSnapshotClass:
|
||||
enabled: false
|
||||
name: ceph-filesystem
|
||||
isDefault: true
|
||||
deletionPolicy: Delete
|
||||
annotations: {}
|
||||
labels: {}
|
||||
# see https://rook.io/docs/rook/v1.10/Storage-Configuration/Ceph-CSI/ceph-csi-snapshot/#cephfs-snapshots for available configuration
|
||||
parameters: {}
|
||||
|
||||
# -- Settings for the block pool snapshot class
|
||||
# @default -- See [RBD Snapshots](../Storage-Configuration/Ceph-CSI/ceph-csi-snapshot.md#rbd-snapshots)
|
||||
cephBlockPoolsVolumeSnapshotClass:
|
||||
enabled: false
|
||||
name: ceph-block
|
||||
isDefault: false
|
||||
deletionPolicy: Delete
|
||||
annotations: {}
|
||||
labels: {}
|
||||
# see https://rook.io/docs/rook/v1.10/Storage-Configuration/Ceph-CSI/ceph-csi-snapshot/#rbd-snapshots for available configuration
|
||||
parameters: {}
|
||||
|
||||
# -- A list of CephObjectStore configurations to deploy
|
||||
# @default -- See [below](#ceph-object-stores)
|
||||
cephObjectStores:
|
||||
- name: ceph-objectstore
|
||||
# see https://github.com/rook/rook/blob/master/Documentation/CRDs/Object-Storage/ceph-object-store-crd.md#object-store-settings for available configuration
|
||||
spec:
|
||||
metadataPool:
|
||||
failureDomain: host
|
||||
replicated:
|
||||
size: 3
|
||||
dataPool:
|
||||
failureDomain: host
|
||||
erasureCoded:
|
||||
dataChunks: 2
|
||||
codingChunks: 1
|
||||
parameters:
|
||||
bulk: "true"
|
||||
preservePoolsOnDelete: true
|
||||
gateway:
|
||||
port: 80
|
||||
resources:
|
||||
limits:
|
||||
memory: "2Gi"
|
||||
requests:
|
||||
cpu: "1000m"
|
||||
memory: "1Gi"
|
||||
# securePort: 443
|
||||
# sslCertificateRef:
|
||||
instances: 1
|
||||
priorityClassName: system-cluster-critical
|
||||
# opsLogSidecar:
|
||||
# resources:
|
||||
# limits:
|
||||
# memory: "100Mi"
|
||||
# requests:
|
||||
# cpu: "100m"
|
||||
# memory: "40Mi"
|
||||
storageClass:
|
||||
enabled: true
|
||||
name: ceph-bucket
|
||||
reclaimPolicy: Delete
|
||||
volumeBindingMode: "Immediate"
|
||||
annotations: {}
|
||||
labels: {}
|
||||
# see https://github.com/rook/rook/blob/master/Documentation/Storage-Configuration/Object-Storage-RGW/ceph-object-bucket-claim.md#storageclass for available configuration
|
||||
parameters:
|
||||
# note: objectStoreNamespace and objectStoreName are configured by the chart
|
||||
region: us-east-1
|
||||
ingress:
|
||||
# Enable an ingress for the ceph-objectstore
|
||||
enabled: false
|
||||
# The ingress port by default will be the object store's "securePort" (if set), or the gateway "port".
|
||||
# To override those defaults, set this ingress port to the desired port.
|
||||
# port: 80
|
||||
# annotations: {}
|
||||
# host:
|
||||
# name: objectstore.example.com
|
||||
# path: /
|
||||
# pathType: Prefix
|
||||
# tls:
|
||||
# - hosts:
|
||||
# - objectstore.example.com
|
||||
# secretName: ceph-objectstore-tls
|
||||
# ingressClassName: nginx
|
||||
route:
|
||||
# Enable an ingress for the ceph-objectstore
|
||||
enabled: false
|
||||
# The ingress port by default will be the object store's "securePort" (if set), or the gateway "port".
|
||||
# To override those defaults, set this ingress port to the desired port.
|
||||
# port: 80
|
||||
# annotations: {}
|
||||
# host:
|
||||
# name: objectstore.example.com
|
||||
# path: /
|
||||
# pathType: PathPrefix
|
||||
# parentRefs:
|
||||
# - name: internal
|
||||
# namespace: kube-system
|
||||
# sectionName: https
|
||||
## cephECBlockPools are disabled by default, please remove the comments and set desired values to enable it
|
||||
## For erasure coded a replicated metadata pool is required.
|
||||
## https://rook.io/docs/rook/latest/CRDs/Shared-Filesystem/ceph-filesystem-crd/#erasure-coded
|
||||
#cephECBlockPools:
|
||||
# - name: ec-pool
|
||||
# spec:
|
||||
# metadataPool:
|
||||
# replicated:
|
||||
# size: 2
|
||||
# dataPool:
|
||||
# failureDomain: osd
|
||||
# erasureCoded:
|
||||
# dataChunks: 2
|
||||
# codingChunks: 1
|
||||
# deviceClass: hdd
|
||||
#
|
||||
# parameters:
|
||||
# # clusterID is the namespace where the rook cluster is running
|
||||
# # If you change this namespace, also change the namespace below where the secret namespaces are defined
|
||||
# clusterID: rook-ceph # namespace:cluster
|
||||
# # (optional) mapOptions is a comma-separated list of map options.
|
||||
# # For krbd options refer
|
||||
# # https://docs.ceph.com/docs/latest/man/8/rbd/#kernel-rbd-krbd-options
|
||||
# # For nbd options refer
|
||||
# # https://docs.ceph.com/docs/latest/man/8/rbd-nbd/#options
|
||||
# # mapOptions: lock_on_read,queue_depth=1024
|
||||
#
|
||||
# # (optional) unmapOptions is a comma-separated list of unmap options.
|
||||
# # For krbd options refer
|
||||
# # https://docs.ceph.com/docs/latest/man/8/rbd/#kernel-rbd-krbd-options
|
||||
# # For nbd options refer
|
||||
# # https://docs.ceph.com/docs/latest/man/8/rbd-nbd/#options
|
||||
# # unmapOptions: force
|
||||
#
|
||||
# # RBD image format. Defaults to "2".
|
||||
# imageFormat: "2"
|
||||
#
|
||||
# # RBD image features, equivalent to OR'd bitfield value: 63
|
||||
# # Available for imageFormat: "2". Older releases of CSI RBD
|
||||
# # support only the `layering` feature. The Linux kernel (KRBD) supports the
|
||||
# # full feature complement as of 5.4
|
||||
# # imageFeatures: layering,fast-diff,object-map,deep-flatten,exclusive-lock
|
||||
# imageFeatures: layering
|
||||
#
|
||||
# storageClass:
|
||||
# provisioner: rook-ceph.rbd.csi.ceph.com # csi-provisioner-name
|
||||
# enabled: true
|
||||
# name: rook-ceph-block
|
||||
# isDefault: false
|
||||
# annotations: { }
|
||||
# labels: { }
|
||||
# allowVolumeExpansion: true
|
||||
# reclaimPolicy: Delete
|
||||
|
||||
# -- CSI driver name prefix for cephfs, rbd and nfs.
|
||||
# @default -- `namespace name where rook-ceph operator is deployed`
|
||||
csiDriverNamePrefix:
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
apiVersion: v2
|
||||
name: cloudhost-ceph
|
||||
description: CloudHost extras on top of Rook-Ceph (app source bucket, platform integration secrets)
|
||||
type: application
|
||||
version: 0.1.0
|
||||
appVersion: "1.0.0"
|
||||
keywords:
|
||||
- ceph
|
||||
- rook
|
||||
- storage
|
||||
- s3
|
||||
maintainers:
|
||||
- name: CloudHost
|
||||
@@ -0,0 +1,140 @@
|
||||
# CloudHost Ceph (Rook)
|
||||
|
||||
Helm chart and install scripts for **Rook-Ceph** on CloudHost clusters:
|
||||
|
||||
| Layer | Purpose |
|
||||
|-------|---------|
|
||||
| **rook-ceph-block** | Expandable PVCs for apps, databases, registry |
|
||||
| **rook-ceph-bucket** | S3-compatible storage for uploaded source zip archives |
|
||||
|
||||
The chart does **not** vendor Rook itself — it installs the official [`rook-release`](https://charts.rook.io/release) charts and adds CloudHost-specific **ObjectBucketClaim** + credential sync.
|
||||
|
||||
---
|
||||
|
||||
## Quick install
|
||||
|
||||
```bash
|
||||
cd backend/helm/cloudhost-ceph
|
||||
./scripts/install.sh single-node # one-node k3s (current abr cluster)
|
||||
# or
|
||||
./scripts/install.sh multi-node # production, 3+ nodes + raw disks
|
||||
```
|
||||
|
||||
Verify:
|
||||
|
||||
```bash
|
||||
./scripts/verify.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Profiles
|
||||
|
||||
### `single-node`
|
||||
|
||||
- OSD on **loop device** `/dev/loop6` (15Gi file at `/var/lib/rook/osd-loopback.img`) — no spare raw disk required
|
||||
- Requires `ROOK_CEPH_ALLOW_LOOP_DEVICES=true` on the operator
|
||||
- Replication **size: 1** (no HA)
|
||||
- Suitable for **staging / single k3s node**
|
||||
- Images must be pre-mirrored to `registry.abrban.com` (see `RUNBOOK-HARBOR.fa.md`)
|
||||
|
||||
### `multi-node`
|
||||
|
||||
- OSD on **raw devices** (`useAllDevices: true`)
|
||||
- Replication **size: 3** for block + object metadata
|
||||
- Erasure-coded object data pool
|
||||
- Requires **3+ nodes** and dedicated disks
|
||||
|
||||
---
|
||||
|
||||
## What gets installed
|
||||
|
||||
| Step | Release | Namespace |
|
||||
|------|---------|-----------|
|
||||
| 1 | `rook-ceph` (operator) | `rook-ceph` |
|
||||
| 2 | `rook-ceph-cluster` | `rook-ceph` |
|
||||
| 3 | `cloudhost-ceph` (OBC + secrets) | `cloudhost-builds` |
|
||||
|
||||
### StorageClasses (from Rook)
|
||||
|
||||
| Name | Use |
|
||||
|------|-----|
|
||||
| `rook-ceph-block` | App PVC, DB PVC, Redis, registry, … |
|
||||
| `rook-ceph-bucket` | `ObjectBucketClaim` → S3 bucket + credentials |
|
||||
|
||||
### CloudHost extras
|
||||
|
||||
| Resource | Description |
|
||||
|----------|-------------|
|
||||
| `ObjectBucketClaim/app-sources` | Bucket for user zip uploads |
|
||||
| `Secret/ceph-app-sources-credentials` | Stable S3 credentials for backend |
|
||||
| `ConfigMap/cloudhost-ceph-integration` | Suggested `PLATFORM_*` env values |
|
||||
|
||||
---
|
||||
|
||||
## Platform integration
|
||||
|
||||
After install, configure the **backend**:
|
||||
|
||||
```env
|
||||
PLATFORM_STORAGE_CLASS=rook-ceph-block
|
||||
PLATFORM_CREATE_STORAGE_CLASS=false
|
||||
PLATFORM_STORAGE_PROVISIONER=rook-ceph.rbd.csi.ceph.com
|
||||
```
|
||||
|
||||
Mount or env-from secret `cloudhost-builds/ceph-app-sources-credentials`:
|
||||
|
||||
```env
|
||||
SOURCE_STORAGE_ENDPOINT=http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local:80
|
||||
SOURCE_STORAGE_REGION=us-east-1
|
||||
SOURCE_STORAGE_BUCKET=<from secret>
|
||||
SOURCE_STORAGE_ACCESS_KEY=<from secret>
|
||||
SOURCE_STORAGE_SECRET_KEY=<from secret>
|
||||
```
|
||||
|
||||
> **Note:** Existing PVCs on `local-path` / `cloudhost-expandable` are **not** migrated automatically. New apps use `rook-ceph-block` once the backend env is updated. Plan migration per workload (see `RUNBOOK-CEPH.fa.md`).
|
||||
|
||||
---
|
||||
|
||||
## Uninstall (destructive)
|
||||
|
||||
```bash
|
||||
./scripts/uninstall.sh
|
||||
# then on each node:
|
||||
sudo rm -rf /var/lib/rook /var/lib/rook/osd
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
```bash
|
||||
# Cluster health
|
||||
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph status
|
||||
|
||||
# OSD pods
|
||||
kubectl -n rook-ceph get pods -l app=rook-ceph-osd
|
||||
|
||||
# RGW (object store)
|
||||
kubectl -n rook-ceph get pods -l app=rook-ceph-rgw
|
||||
|
||||
# Bucket sync job
|
||||
kubectl -n cloudhost-builds logs job -l job-name=cloudhost-ceph-bucket-sync --tail=50
|
||||
```
|
||||
|
||||
Full operational guide (Persian): [`../../../RUNBOOK-CEPH.fa.md`](../../../RUNBOOK-CEPH.fa.md)
|
||||
|
||||
Registry / Harbor (Persian): [`../../../RUNBOOK-HARBOR.fa.md`](../../../RUNBOOK-HARBOR.fa.md)
|
||||
|
||||
---
|
||||
|
||||
## Files
|
||||
|
||||
| File | Role |
|
||||
|------|------|
|
||||
| `values-rook-cluster-single-node.yaml` | Rook cluster values (1 node) |
|
||||
| `values-rook-cluster-multi-node.yaml` | Rook cluster values (production) |
|
||||
| `values.yaml` | CloudHost OBC / secret sync |
|
||||
| `scripts/install.sh` | Full install |
|
||||
| `scripts/verify.sh` | Health check |
|
||||
| `scripts/uninstall.sh` | Tear down |
|
||||
+52
@@ -0,0 +1,52 @@
|
||||
#!/usr/bin/env bash
|
||||
# Install Rook-Ceph operator + cluster + CloudHost bucket extras.
|
||||
# Usage: ./scripts/install.sh [single-node|multi-node]
|
||||
set -euo pipefail
|
||||
|
||||
PROFILE="${1:-single-node}"
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
CHART_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)"
|
||||
ROOK_NS="rook-ceph"
|
||||
EXTRAS_NS="cloudhost-builds"
|
||||
CLUSTER_VALUES="${CHART_DIR}/values-rook-cluster-${PROFILE}.yaml"
|
||||
|
||||
if [[ ! -f "${CLUSTER_VALUES}" ]]; then
|
||||
echo "Unknown profile: ${PROFILE} (missing ${CLUSTER_VALUES})" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "==> Profile: ${PROFILE}"
|
||||
echo "==> Adding rook-release helm repo"
|
||||
helm repo add rook-release https://charts.rook.io/release 2>/dev/null || true
|
||||
helm repo update rook-release
|
||||
|
||||
echo "==> [1/4] Installing Rook operator in ${ROOK_NS}"
|
||||
helm upgrade --install rook-ceph rook-release/rook-ceph \
|
||||
--namespace "${ROOK_NS}" \
|
||||
--create-namespace \
|
||||
--wait \
|
||||
--timeout 10m
|
||||
|
||||
echo "==> [2/4] Waiting for Rook operator deployment"
|
||||
kubectl -n "${ROOK_NS}" rollout status deploy/rook-ceph-operator --timeout=300s
|
||||
|
||||
echo "==> [3/4] Installing Ceph cluster (${CLUSTER_VALUES})"
|
||||
helm upgrade --install rook-ceph-cluster rook-release/rook-ceph-cluster \
|
||||
--namespace "${ROOK_NS}" \
|
||||
-f "${CLUSTER_VALUES}" \
|
||||
--wait \
|
||||
--timeout 25m
|
||||
|
||||
echo "==> Waiting for CephCluster phase = Ready (up to 20 min)"
|
||||
"${SCRIPT_DIR}/wait-ceph-ready.sh" 1200
|
||||
|
||||
echo "==> [4/4] Installing CloudHost Ceph extras (ObjectBucketClaim) in ${EXTRAS_NS}"
|
||||
kubectl create namespace "${EXTRAS_NS}" 2>/dev/null || true
|
||||
helm upgrade --install cloudhost-ceph "${CHART_DIR}" \
|
||||
--namespace "${EXTRAS_NS}" \
|
||||
-f "${CHART_DIR}/values.yaml" \
|
||||
--wait \
|
||||
--timeout 15m
|
||||
|
||||
echo ""
|
||||
echo "==> Done. Run ./scripts/verify.sh to confirm health and print integration hints."
|
||||
+26
@@ -0,0 +1,26 @@
|
||||
#!/usr/bin/env bash
|
||||
# Remove CloudHost extras + Rook cluster + operator (DATA LOSS).
|
||||
set -euo pipefail
|
||||
|
||||
read -r -p "This deletes ALL Ceph data. Type 'delete-ceph' to continue: " CONFIRM
|
||||
if [[ "${CONFIRM}" != "delete-ceph" ]]; then
|
||||
echo "Aborted."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
helm uninstall cloudhost-ceph -n cloudhost-builds 2>/dev/null || true
|
||||
helm uninstall rook-ceph-cluster -n rook-ceph 2>/dev/null || true
|
||||
helm uninstall rook-ceph -n rook-ceph 2>/dev/null || true
|
||||
|
||||
echo "Waiting for Rook resources to terminate..."
|
||||
sleep 15
|
||||
kubectl -n rook-ceph get pods 2>/dev/null || true
|
||||
|
||||
cat <<'EOF'
|
||||
|
||||
IMPORTANT: On each node, wipe Rook state before reinstalling:
|
||||
sudo rm -rf /var/lib/rook
|
||||
sudo rm -rf /var/lib/rook/osd
|
||||
|
||||
For raw-disk OSDs also zap disks (see RUNBOOK-CEPH.fa.md).
|
||||
EOF
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
ROOK_NS="rook-ceph"
|
||||
EXTRAS_NS="cloudhost-builds"
|
||||
|
||||
echo "=== StorageClasses ==="
|
||||
kubectl get storageclass | grep -E 'NAME|rook-ceph' || true
|
||||
|
||||
echo ""
|
||||
echo "=== Ceph status ==="
|
||||
kubectl -n "${ROOK_NS}" exec deploy/rook-ceph-tools -- ceph status 2>/dev/null || echo "(tools pod not ready yet)"
|
||||
|
||||
echo ""
|
||||
echo "=== OSD / MON pods ==="
|
||||
kubectl -n "${ROOK_NS}" get pods -l app=rook-ceph-osd 2>/dev/null || kubectl -n "${ROOK_NS}" get pods | grep -E 'osd|mon|mgr|rgw' || true
|
||||
|
||||
echo ""
|
||||
echo "=== Object bucket claim ==="
|
||||
kubectl -n "${EXTRAS_NS}" get obc,app-sources 2>/dev/null || kubectl -n "${EXTRAS_NS}" get obc 2>/dev/null || true
|
||||
|
||||
echo ""
|
||||
echo "=== Platform credentials secret ==="
|
||||
if kubectl -n "${EXTRAS_NS}" get secret ceph-app-sources-credentials >/dev/null 2>&1; then
|
||||
echo "Secret ceph-app-sources-credentials exists"
|
||||
kubectl -n "${EXTRAS_NS}" get secret ceph-app-sources-credentials -o jsonpath='{.data.SOURCE_STORAGE_BUCKET}' | base64 -d
|
||||
echo ""
|
||||
else
|
||||
echo "Secret ceph-app-sources-credentials not ready — check bucket sync job:"
|
||||
kubectl -n "${EXTRAS_NS}" get jobs,pods | grep bucket-sync || true
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "=== Suggested backend env ==="
|
||||
kubectl -n "${EXTRAS_NS}" get configmap cloudhost-ceph-integration -o yaml 2>/dev/null | sed -n '/PLATFORM_/p;/SOURCE_STORAGE_ENDPOINT/p' || true
|
||||
+39
@@ -0,0 +1,39 @@
|
||||
#!/usr/bin/env bash
|
||||
# Wait until Ceph reports HEALTH_OK or HEALTH_WARN (single-node often stays WARN).
|
||||
set -euo pipefail
|
||||
|
||||
TIMEOUT="${1:-900}"
|
||||
ROOK_NS="rook-ceph"
|
||||
START=$(date +%s)
|
||||
|
||||
echo "Waiting for rook-ceph-tools deployment..."
|
||||
for _ in $(seq 1 60); do
|
||||
if kubectl -n "${ROOK_NS}" get deploy rook-ceph-tools >/dev/null 2>&1; then
|
||||
if kubectl -n "${ROOK_NS}" rollout status deploy/rook-ceph-tools --timeout=120s 2>/dev/null; then
|
||||
break
|
||||
fi
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
|
||||
while true; do
|
||||
NOW=$(date +%s)
|
||||
if (( NOW - START > TIMEOUT )); then
|
||||
echo "Timed out after ${TIMEOUT}s waiting for Ceph health" >&2
|
||||
kubectl -n "${ROOK_NS}" get cephcluster,pod -o wide || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if kubectl -n "${ROOK_NS}" get deploy rook-ceph-tools >/dev/null 2>&1; then
|
||||
HEALTH=$(kubectl -n "${ROOK_NS}" exec deploy/rook-ceph-tools -- ceph health 2>/dev/null || echo "unknown")
|
||||
echo "Ceph health: ${HEALTH}"
|
||||
if [[ "${HEALTH}" == "HEALTH_OK" || "${HEALTH}" == HEALTH_WARN* ]]; then
|
||||
PHASE=$(kubectl -n "${ROOK_NS}" get cephcluster rook-ceph -o jsonpath='{.status.phase}' 2>/dev/null || echo "")
|
||||
echo "CephCluster phase: ${PHASE}"
|
||||
if [[ "${PHASE}" == "Ready" ]]; then
|
||||
exit 0
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
sleep 15
|
||||
done
|
||||
@@ -0,0 +1,23 @@
|
||||
CloudHost Ceph storage is ready (or still initializing).
|
||||
|
||||
Profiles
|
||||
single-node Directory OSD on /var/lib/rook/osd — for one-node k3s (no HA)
|
||||
multi-node Raw disk OSDs with replication=3 — production
|
||||
|
||||
StorageClasses created by Rook
|
||||
rook-ceph-block Block volumes (app PVC, DB, registry, …)
|
||||
rook-ceph-bucket S3-compatible buckets via ObjectBucketClaim
|
||||
|
||||
Verify cluster health
|
||||
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph status
|
||||
kubectl get storageclass | grep rook-ceph
|
||||
kubectl -n cloudhost-builds get obc,secret | grep -E 'app-sources|ceph-app-sources'
|
||||
|
||||
Platform backend (after bucket sync Job completes)
|
||||
PLATFORM_STORAGE_CLASS=rook-ceph-block
|
||||
PLATFORM_CREATE_STORAGE_CLASS=false
|
||||
PLATFORM_STORAGE_PROVISIONER=rook-ceph.rbd.csi.ceph.com
|
||||
|
||||
Mount secret cloudhost-builds/ceph-app-sources-credentials for zip upload S3 settings.
|
||||
|
||||
Full guide: backend/helm/cloudhost-ceph/README.md and RUNBOOK-CEPH.fa.md
|
||||
@@ -0,0 +1,27 @@
|
||||
{{/*
|
||||
CloudHost Ceph chart helpers
|
||||
*/}}
|
||||
{{- define "cloudhost-ceph.name" -}}
|
||||
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
|
||||
{{- end }}
|
||||
|
||||
{{- define "cloudhost-ceph.fullname" -}}
|
||||
{{- if .Values.fullnameOverride }}
|
||||
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
|
||||
{{- else }}
|
||||
{{- $name := default .Chart.Name .Values.nameOverride }}
|
||||
{{- if contains $name .Release.Name }}
|
||||
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
|
||||
{{- else }}
|
||||
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
|
||||
{{- define "cloudhost-ceph.labels" -}}
|
||||
helm.sh/chart: {{ include "cloudhost-ceph.name" . }}-{{ .Chart.Version }}
|
||||
app.kubernetes.io/name: {{ include "cloudhost-ceph.name" . }}
|
||||
app.kubernetes.io/instance: {{ .Release.Name }}
|
||||
app.kubernetes.io/managed-by: {{ .Release.Service }}
|
||||
app.kubernetes.io/part-of: cloudhost
|
||||
{{- end }}
|
||||
@@ -0,0 +1,20 @@
|
||||
{{- if .Values.integration.createConfigMap }}
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: {{ .Values.integration.configMapName }}
|
||||
namespace: {{ .Values.namespace }}
|
||||
labels:
|
||||
{{- include "cloudhost-ceph.labels" . | nindent 4 }}
|
||||
data:
|
||||
PLATFORM_STORAGE_CLASS: rook-ceph-block
|
||||
PLATFORM_CREATE_STORAGE_CLASS: "false"
|
||||
PLATFORM_STORAGE_PROVISIONER: rook-ceph.rbd.csi.ceph.com
|
||||
SOURCE_STORAGE_ENDPOINT: {{ .Values.platform.endpoint | quote }}
|
||||
SOURCE_STORAGE_REGION: {{ .Values.platform.region | quote }}
|
||||
SOURCE_STORAGE_CREDENTIALS_SECRET: {{ .Values.platform.credentialsSecretName | quote }}
|
||||
README: |
|
||||
Block PVCs: set PLATFORM_STORAGE_CLASS=rook-ceph-block on the backend.
|
||||
New app PVCs use rook-ceph-block; existing local-path PVCs are NOT auto-migrated.
|
||||
Object storage credentials: secret {{ .Values.platform.credentialsSecretName }} in {{ .Values.namespace }}.
|
||||
{{- end }}
|
||||
@@ -0,0 +1,15 @@
|
||||
{{- if .Values.objectStore.claimName }}
|
||||
apiVersion: objectbucket.io/v1alpha1
|
||||
kind: ObjectBucketClaim
|
||||
metadata:
|
||||
name: {{ .Values.objectStore.claimName }}
|
||||
namespace: {{ .Values.namespace }}
|
||||
labels:
|
||||
{{- include "cloudhost-ceph.labels" . | nindent 4 }}
|
||||
annotations:
|
||||
helm.sh/hook: post-install,post-upgrade
|
||||
helm.sh/hook-weight: "5"
|
||||
spec:
|
||||
storageClassName: {{ .Values.objectStore.bucketStorageClass | quote }}
|
||||
generateBucketName: {{ .Values.objectStore.generateBucketName | quote }}
|
||||
{{- end }}
|
||||
@@ -0,0 +1,113 @@
|
||||
{{- if and .Values.platform.createCredentialsSecret .Values.objectStore.claimName }}
|
||||
# Stable secret name for platform workers. Populated by a post-install Job once the OBC secret exists.
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: {{ include "cloudhost-ceph.fullname" . }}-bucket-sync
|
||||
namespace: {{ .Values.namespace }}
|
||||
labels:
|
||||
{{- include "cloudhost-ceph.labels" . | nindent 4 }}
|
||||
annotations:
|
||||
helm.sh/hook: post-install,post-upgrade
|
||||
helm.sh/hook-weight: "1"
|
||||
helm.sh/hook-delete-policy: before-hook-creation
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: {{ include "cloudhost-ceph.fullname" . }}-bucket-sync
|
||||
namespace: {{ .Values.namespace }}
|
||||
annotations:
|
||||
helm.sh/hook: post-install,post-upgrade
|
||||
helm.sh/hook-weight: "1"
|
||||
helm.sh/hook-delete-policy: before-hook-creation
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["secrets"]
|
||||
verbs: ["get", "list", "create", "patch", "update"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: {{ include "cloudhost-ceph.fullname" . }}-bucket-sync
|
||||
namespace: {{ .Values.namespace }}
|
||||
annotations:
|
||||
helm.sh/hook: post-install,post-upgrade
|
||||
helm.sh/hook-weight: "1"
|
||||
helm.sh/hook-delete-policy: before-hook-creation
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: Role
|
||||
name: {{ include "cloudhost-ceph.fullname" . }}-bucket-sync
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: {{ include "cloudhost-ceph.fullname" . }}-bucket-sync
|
||||
namespace: {{ .Values.namespace }}
|
||||
---
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: {{ include "cloudhost-ceph.fullname" . }}-bucket-sync
|
||||
namespace: {{ .Values.namespace }}
|
||||
labels:
|
||||
{{- include "cloudhost-ceph.labels" . | nindent 4 }}
|
||||
annotations:
|
||||
helm.sh/hook: post-install,post-upgrade
|
||||
helm.sh/hook-weight: "10"
|
||||
helm.sh/hook-delete-policy: before-hook-creation
|
||||
spec:
|
||||
backoffLimit: 30
|
||||
template:
|
||||
spec:
|
||||
serviceAccountName: {{ include "cloudhost-ceph.fullname" . }}-bucket-sync
|
||||
restartPolicy: OnFailure
|
||||
containers:
|
||||
- name: sync
|
||||
image: registry.abrban.com/proxy-dockerhub/bitnami/kubectl:1.32
|
||||
imagePullPolicy: IfNotPresent
|
||||
env:
|
||||
- name: OBC_SECRET
|
||||
value: {{ printf "obc-%s-%s" .Values.namespace .Values.objectStore.claimName | quote }}
|
||||
- name: TARGET_SECRET
|
||||
value: {{ .Values.platform.credentialsSecretName | quote }}
|
||||
- name: NAMESPACE
|
||||
value: {{ .Values.namespace | quote }}
|
||||
- name: ENDPOINT
|
||||
value: {{ .Values.platform.endpoint | quote }}
|
||||
- name: REGION
|
||||
value: {{ .Values.platform.region | quote }}
|
||||
command:
|
||||
- /bin/bash
|
||||
- -ec
|
||||
- |
|
||||
echo "Waiting for OBC secret ${OBC_SECRET} in ${NAMESPACE}..."
|
||||
for i in $(seq 1 120); do
|
||||
if kubectl get secret -n "${NAMESPACE}" "${OBC_SECRET}" >/dev/null 2>&1; then
|
||||
break
|
||||
fi
|
||||
sleep 10
|
||||
done
|
||||
kubectl get secret -n "${NAMESPACE}" "${OBC_SECRET}" >/dev/null
|
||||
|
||||
BUCKET=$(kubectl get secret -n "${NAMESPACE}" "${OBC_SECRET}" -o jsonpath='{.data.BUCKET_NAME}' | base64 -d)
|
||||
ACCESS=$(kubectl get secret -n "${NAMESPACE}" "${OBC_SECRET}" -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 -d)
|
||||
SECRET=$(kubectl get secret -n "${NAMESPACE}" "${OBC_SECRET}" -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 -d)
|
||||
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: ${TARGET_SECRET}
|
||||
namespace: ${NAMESPACE}
|
||||
labels:
|
||||
app.kubernetes.io/part-of: cloudhost
|
||||
type: Opaque
|
||||
stringData:
|
||||
SOURCE_STORAGE_ENDPOINT: "${ENDPOINT}"
|
||||
SOURCE_STORAGE_REGION: "${REGION}"
|
||||
SOURCE_STORAGE_BUCKET: "${BUCKET}"
|
||||
SOURCE_STORAGE_ACCESS_KEY: "${ACCESS}"
|
||||
SOURCE_STORAGE_SECRET_KEY: "${SECRET}"
|
||||
EOF
|
||||
echo "Synced bucket credentials to secret ${TARGET_SECRET} (bucket=${BUCKET})"
|
||||
{{- end }}
|
||||
@@ -0,0 +1,83 @@
|
||||
# Rook-Ceph cluster values — MULTI NODE (production).
|
||||
# Install: scripts/install.sh multi-node
|
||||
#
|
||||
# Prerequisites:
|
||||
# - At least 3 worker nodes (odd mon count)
|
||||
# - Raw disks available (useAllDevices) OR dedicated devices per node
|
||||
# - Taint-free nodes labeled rook-ceph-role=storage-node (optional)
|
||||
|
||||
operatorNamespace: rook-ceph
|
||||
|
||||
toolbox:
|
||||
enabled: true
|
||||
|
||||
cephClusterSpec:
|
||||
dataDirHostPath: /var/lib/rook
|
||||
|
||||
mon:
|
||||
count: 3
|
||||
allowMultiplePerNode: false
|
||||
|
||||
mgr:
|
||||
count: 2
|
||||
allowMultiplePerNode: false
|
||||
|
||||
dashboard:
|
||||
enabled: true
|
||||
ssl: true
|
||||
|
||||
storage:
|
||||
useAllNodes: false
|
||||
useAllDevices: true
|
||||
# Example: pin OSDs to storage nodes only
|
||||
# nodes:
|
||||
# - name: "node-1"
|
||||
# - name: "node-2"
|
||||
# - name: "node-3"
|
||||
|
||||
cephFileSystems: []
|
||||
|
||||
cephBlockPools:
|
||||
- name: ceph-blockpool
|
||||
spec:
|
||||
failureDomain: host
|
||||
replicated:
|
||||
size: 3
|
||||
storageClass:
|
||||
enabled: true
|
||||
name: rook-ceph-block
|
||||
isDefault: false
|
||||
reclaimPolicy: Delete
|
||||
allowVolumeExpansion: true
|
||||
volumeBindingMode: WaitForFirstConsumer
|
||||
|
||||
cephObjectStores:
|
||||
- name: ceph-objectstore
|
||||
spec:
|
||||
metadataPool:
|
||||
failureDomain: host
|
||||
replicated:
|
||||
size: 3
|
||||
dataPool:
|
||||
failureDomain: host
|
||||
erasureCoded:
|
||||
dataChunks: 2
|
||||
codingChunks: 1
|
||||
preservePoolsOnDelete: true
|
||||
gateway:
|
||||
port: 80
|
||||
instances: 2
|
||||
resources:
|
||||
limits:
|
||||
memory: "2Gi"
|
||||
requests:
|
||||
cpu: "500m"
|
||||
memory: "1Gi"
|
||||
storageClass:
|
||||
enabled: true
|
||||
name: rook-ceph-bucket
|
||||
reclaimPolicy: Delete
|
||||
parameters:
|
||||
region: us-east-1
|
||||
ingress:
|
||||
enabled: false
|
||||
@@ -0,0 +1,102 @@
|
||||
# Rook-Ceph cluster values — SINGLE NODE (k3s dev/staging).
|
||||
# Install: scripts/install.sh single-node
|
||||
#
|
||||
# Uses loop device /dev/loop6 (15Gi) on single-node clusters without a spare raw disk.
|
||||
# Replication factor = 1 (no HA). For production multi-node use values-rook-cluster-multi-node.yaml.
|
||||
# See RUNBOOK-CEPH.fa.md for loop setup and image mirroring prerequisites.
|
||||
|
||||
operatorNamespace: rook-ceph
|
||||
|
||||
toolbox:
|
||||
enabled: true
|
||||
|
||||
cephClusterSpec:
|
||||
dataDirHostPath: /var/lib/rook
|
||||
skipUpgradeChecks: true
|
||||
continueUpgradeAfterChecksEvenIfNotHealthy: true
|
||||
|
||||
mon:
|
||||
count: 1
|
||||
allowMultiplePerNode: true
|
||||
|
||||
mgr:
|
||||
count: 1
|
||||
allowMultiplePerNode: true
|
||||
|
||||
dashboard:
|
||||
enabled: true
|
||||
ssl: false
|
||||
|
||||
resources:
|
||||
mon:
|
||||
limits:
|
||||
memory: "1Gi"
|
||||
requests:
|
||||
cpu: "200m"
|
||||
memory: "512Mi"
|
||||
mgr:
|
||||
limits:
|
||||
memory: "1Gi"
|
||||
requests:
|
||||
cpu: "200m"
|
||||
memory: "512Mi"
|
||||
osd:
|
||||
limits:
|
||||
memory: "2Gi"
|
||||
requests:
|
||||
cpu: "500m"
|
||||
memory: "1Gi"
|
||||
|
||||
storage:
|
||||
useAllNodes: true
|
||||
useAllDevices: false
|
||||
devices:
|
||||
- name: "/dev/loop6"
|
||||
|
||||
# Disable CephFS to save RAM on single-node clusters.
|
||||
cephFileSystems: []
|
||||
|
||||
cephBlockPools:
|
||||
- name: ceph-blockpool
|
||||
spec:
|
||||
failureDomain: osd
|
||||
replicated:
|
||||
size: 1
|
||||
storageClass:
|
||||
enabled: true
|
||||
name: rook-ceph-block
|
||||
isDefault: false
|
||||
reclaimPolicy: Delete
|
||||
allowVolumeExpansion: true
|
||||
volumeBindingMode: WaitForFirstConsumer
|
||||
|
||||
cephObjectStores:
|
||||
- name: ceph-objectstore
|
||||
spec:
|
||||
metadataPool:
|
||||
failureDomain: osd
|
||||
replicated:
|
||||
size: 1
|
||||
dataPool:
|
||||
failureDomain: osd
|
||||
replicated:
|
||||
size: 1
|
||||
preservePoolsOnDelete: true
|
||||
gateway:
|
||||
port: 80
|
||||
instances: 1
|
||||
resources:
|
||||
limits:
|
||||
memory: "1Gi"
|
||||
requests:
|
||||
cpu: "250m"
|
||||
memory: "512Mi"
|
||||
storageClass:
|
||||
enabled: true
|
||||
name: rook-ceph-bucket
|
||||
reclaimPolicy: Delete
|
||||
volumeBindingMode: Immediate
|
||||
parameters:
|
||||
region: us-east-1
|
||||
ingress:
|
||||
enabled: false
|
||||
@@ -0,0 +1,25 @@
|
||||
# CloudHost Ceph extras (ObjectBucketClaim for zip uploads).
|
||||
# Rook operator + CephCluster are installed via scripts/install.sh using official rook-release charts.
|
||||
|
||||
namespace: cloudhost-builds
|
||||
|
||||
objectStore:
|
||||
# Must match rook-ceph-cluster cephObjectStores[].storageClass.name
|
||||
bucketStorageClass: rook-ceph-bucket
|
||||
# Claim name; Rook generates bucket + credentials secret
|
||||
claimName: app-sources
|
||||
# Prefix for generated bucket name (Rook appends random suffix)
|
||||
generateBucketName: cloudhost-app-sources
|
||||
|
||||
platform:
|
||||
# Copy S3 credentials into a stable secret name for backend/workers
|
||||
createCredentialsSecret: true
|
||||
credentialsSecretName: ceph-app-sources-credentials
|
||||
# In-cluster RGW endpoint (adjust if ingress is enabled on object store)
|
||||
endpoint: http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local:80
|
||||
region: us-east-1
|
||||
|
||||
integration:
|
||||
# Emit a ConfigMap with suggested backend env vars (non-secret)
|
||||
createConfigMap: true
|
||||
configMapName: cloudhost-ceph-integration
|
||||
@@ -0,0 +1,48 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Replaces registry.abrban.com docker distribution with Harbor.
|
||||
# WARNING: This will delete the existing `Ingress/registry` routing. The old
|
||||
# `Deployment/registry` and its PVC are left in place for rollback.
|
||||
|
||||
VALUES_FILE="${1:-/Users/keyhan/Documents/keyhan-project/cloud-host/backend/helm/cloudhost-harbor/values-registry.abrban.com.yaml}"
|
||||
|
||||
echo "==> Ensuring harbor repo"
|
||||
helm repo add harbor https://helm.goharbor.io 2>/dev/null || true
|
||||
helm repo update harbor
|
||||
|
||||
echo "==> [0/4] Preflight"
|
||||
kubectl -n cloudhost get secret abrban-wildcard-tls >/dev/null
|
||||
kubectl -n cloudhost get secret registry-egress-proxy >/dev/null
|
||||
|
||||
echo "==> [1/4] Disabling old registry ingress (host registry.abrban.com)"
|
||||
kubectl -n cloudhost delete ingress registry --ignore-not-found
|
||||
|
||||
echo "==> [2/4] Scaling old registry deployment down (rollback-friendly)"
|
||||
kubectl -n cloudhost scale deploy/registry --replicas=0 || true
|
||||
|
||||
echo "==> [3/4] Installing Harbor"
|
||||
HTTP_PROXY="$(kubectl -n cloudhost get secret registry-egress-proxy -o jsonpath='{.data.HTTP_PROXY}' | base64 -d)"
|
||||
HTTPS_PROXY="$(kubectl -n cloudhost get secret registry-egress-proxy -o jsonpath='{.data.HTTPS_PROXY}' | base64 -d)"
|
||||
NO_PROXY="$(kubectl -n cloudhost get secret registry-egress-proxy -o jsonpath='{.data.NO_PROXY}' | base64 -d)"
|
||||
|
||||
TMP_PROXY_VALUES="$(mktemp)"
|
||||
cat > "${TMP_PROXY_VALUES}" <<EOF
|
||||
proxy:
|
||||
httpProxy: "${HTTP_PROXY}"
|
||||
httpsProxy: "${HTTPS_PROXY}"
|
||||
noProxy: "${NO_PROXY}"
|
||||
EOF
|
||||
|
||||
helm upgrade --install harbor harbor/harbor \
|
||||
-n cloudhost \
|
||||
-f "$VALUES_FILE" \
|
||||
-f "${TMP_PROXY_VALUES}" \
|
||||
--wait \
|
||||
--timeout 20m
|
||||
|
||||
rm -f "${TMP_PROXY_VALUES}" || true
|
||||
|
||||
echo "==> [4/4] Done"
|
||||
kubectl -n cloudhost get ingress | grep -n registry || true
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
## Harbor values to REPLACE registry.abrban.com
|
||||
## Ingress controller on this cluster is Traefik (k3s).
|
||||
##
|
||||
## Install:
|
||||
## helm upgrade --install harbor harbor/harbor -n cloudhost -f backend/helm/cloudhost-harbor/values-registry.abrban.com.yaml
|
||||
##
|
||||
externalURL: https://registry.abrban.com
|
||||
|
||||
proxy:
|
||||
# Values are injected by install script from `cloudhost/registry-egress-proxy`.
|
||||
httpProxy: ""
|
||||
httpsProxy: ""
|
||||
noProxy: ""
|
||||
|
||||
expose:
|
||||
type: ingress
|
||||
tls:
|
||||
enabled: true
|
||||
certSource: secret
|
||||
secret:
|
||||
secretName: abrban-wildcard-tls
|
||||
ingress:
|
||||
className: traefik
|
||||
hosts:
|
||||
core: registry.abrban.com
|
||||
annotations:
|
||||
traefik.ingress.kubernetes.io/router.entrypoints: websecure
|
||||
# Increase timeouts for large pushes (skopeo/registry blobs)
|
||||
traefik.ingress.kubernetes.io/router.tls: "true"
|
||||
|
||||
# Disable components we don't need for now to reduce resources
|
||||
trivy:
|
||||
enabled: false
|
||||
notary:
|
||||
enabled: false
|
||||
chartmuseum:
|
||||
enabled: false
|
||||
|
||||
# Single-node staging: keep resource usage modest
|
||||
core:
|
||||
replicas: 1
|
||||
jobservice:
|
||||
replicas: 1
|
||||
registry:
|
||||
replicas: 1
|
||||
|
||||
persistence:
|
||||
enabled: true
|
||||
persistentVolumeClaim:
|
||||
# Use existing default storage (local-path) until Ceph is ready.
|
||||
# After Ceph, switch to rook-ceph-block for Harbor's PVCs.
|
||||
registry:
|
||||
storageClass: local-path
|
||||
size: 50Gi
|
||||
jobservice:
|
||||
storageClass: local-path
|
||||
size: 5Gi
|
||||
database:
|
||||
storageClass: local-path
|
||||
size: 10Gi
|
||||
redis:
|
||||
storageClass: local-path
|
||||
size: 5Gi
|
||||
|
||||
database:
|
||||
type: internal
|
||||
|
||||
redis:
|
||||
type: internal
|
||||
|
||||
portal:
|
||||
replicas: 1
|
||||
|
||||
# We will create proxy-cache projects after install (todo: configure-proxy-cache)
|
||||
|
||||
Reference in New Issue
Block a user