Add Harbor/Ceph runbooks and align single-node Ceph docs with abr deployment.
Document registry ingress split, image mirroring, loop OSD setup, and platform integration paths.
This commit is contained in:
keyhan
@@ -0,0 +1,140 @@
|
||||
# CloudHost Ceph (Rook)
|
||||
|
||||
Helm chart and install scripts for **Rook-Ceph** on CloudHost clusters:
|
||||
|
||||
| Layer | Purpose |
|
||||
|-------|---------|
|
||||
| **rook-ceph-block** | Expandable PVCs for apps, databases, registry |
|
||||
| **rook-ceph-bucket** | S3-compatible storage for uploaded source zip archives |
|
||||
|
||||
The chart does **not** vendor Rook itself — it installs the official [`rook-release`](https://charts.rook.io/release) charts and adds CloudHost-specific **ObjectBucketClaim** + credential sync.
|
||||
|
||||
---
|
||||
|
||||
## Quick install
|
||||
|
||||
```bash
|
||||
cd backend/helm/cloudhost-ceph
|
||||
./scripts/install.sh single-node # one-node k3s (current abr cluster)
|
||||
# or
|
||||
./scripts/install.sh multi-node # production, 3+ nodes + raw disks
|
||||
```
|
||||
|
||||
Verify:
|
||||
|
||||
```bash
|
||||
./scripts/verify.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Profiles
|
||||
|
||||
### `single-node`
|
||||
|
||||
- OSD on **loop device** `/dev/loop6` (15Gi file at `/var/lib/rook/osd-loopback.img`) — no spare raw disk required
|
||||
- Requires `ROOK_CEPH_ALLOW_LOOP_DEVICES=true` on the operator
|
||||
- Replication **size: 1** (no HA)
|
||||
- Suitable for **staging / single k3s node**
|
||||
- Images must be pre-mirrored to `registry.abrban.com` (see `RUNBOOK-HARBOR.fa.md`)
|
||||
|
||||
### `multi-node`
|
||||
|
||||
- OSD on **raw devices** (`useAllDevices: true`)
|
||||
- Replication **size: 3** for block + object metadata
|
||||
- Erasure-coded object data pool
|
||||
- Requires **3+ nodes** and dedicated disks
|
||||
|
||||
---
|
||||
|
||||
## What gets installed
|
||||
|
||||
| Step | Release | Namespace |
|
||||
|------|---------|-----------|
|
||||
| 1 | `rook-ceph` (operator) | `rook-ceph` |
|
||||
| 2 | `rook-ceph-cluster` | `rook-ceph` |
|
||||
| 3 | `cloudhost-ceph` (OBC + secrets) | `cloudhost-builds` |
|
||||
|
||||
### StorageClasses (from Rook)
|
||||
|
||||
| Name | Use |
|
||||
|------|-----|
|
||||
| `rook-ceph-block` | App PVC, DB PVC, Redis, registry, … |
|
||||
| `rook-ceph-bucket` | `ObjectBucketClaim` → S3 bucket + credentials |
|
||||
|
||||
### CloudHost extras
|
||||
|
||||
| Resource | Description |
|
||||
|----------|-------------|
|
||||
| `ObjectBucketClaim/app-sources` | Bucket for user zip uploads |
|
||||
| `Secret/ceph-app-sources-credentials` | Stable S3 credentials for backend |
|
||||
| `ConfigMap/cloudhost-ceph-integration` | Suggested `PLATFORM_*` env values |
|
||||
|
||||
---
|
||||
|
||||
## Platform integration
|
||||
|
||||
After install, configure the **backend**:
|
||||
|
||||
```env
|
||||
PLATFORM_STORAGE_CLASS=rook-ceph-block
|
||||
PLATFORM_CREATE_STORAGE_CLASS=false
|
||||
PLATFORM_STORAGE_PROVISIONER=rook-ceph.rbd.csi.ceph.com
|
||||
```
|
||||
|
||||
Mount or env-from secret `cloudhost-builds/ceph-app-sources-credentials`:
|
||||
|
||||
```env
|
||||
SOURCE_STORAGE_ENDPOINT=http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local:80
|
||||
SOURCE_STORAGE_REGION=us-east-1
|
||||
SOURCE_STORAGE_BUCKET=<from secret>
|
||||
SOURCE_STORAGE_ACCESS_KEY=<from secret>
|
||||
SOURCE_STORAGE_SECRET_KEY=<from secret>
|
||||
```
|
||||
|
||||
> **Note:** Existing PVCs on `local-path` / `cloudhost-expandable` are **not** migrated automatically. New apps use `rook-ceph-block` once the backend env is updated. Plan migration per workload (see `RUNBOOK-CEPH.fa.md`).
|
||||
|
||||
---
|
||||
|
||||
## Uninstall (destructive)
|
||||
|
||||
```bash
|
||||
./scripts/uninstall.sh
|
||||
# then on each node:
|
||||
sudo rm -rf /var/lib/rook /var/lib/rook/osd
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
```bash
|
||||
# Cluster health
|
||||
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph status
|
||||
|
||||
# OSD pods
|
||||
kubectl -n rook-ceph get pods -l app=rook-ceph-osd
|
||||
|
||||
# RGW (object store)
|
||||
kubectl -n rook-ceph get pods -l app=rook-ceph-rgw
|
||||
|
||||
# Bucket sync job
|
||||
kubectl -n cloudhost-builds logs job -l job-name=cloudhost-ceph-bucket-sync --tail=50
|
||||
```
|
||||
|
||||
Full operational guide (Persian): [`../../../RUNBOOK-CEPH.fa.md`](../../../RUNBOOK-CEPH.fa.md)
|
||||
|
||||
Registry / Harbor (Persian): [`../../../RUNBOOK-HARBOR.fa.md`](../../../RUNBOOK-HARBOR.fa.md)
|
||||
|
||||
---
|
||||
|
||||
## Files
|
||||
|
||||
| File | Role |
|
||||
|------|------|
|
||||
| `values-rook-cluster-single-node.yaml` | Rook cluster values (1 node) |
|
||||
| `values-rook-cluster-multi-node.yaml` | Rook cluster values (production) |
|
||||
| `values.yaml` | CloudHost OBC / secret sync |
|
||||
| `scripts/install.sh` | Full install |
|
||||
| `scripts/verify.sh` | Health check |
|
||||
| `scripts/uninstall.sh` | Tear down |
|
||||
Reference in New Issue
Block a user