keyhan
ee5bd0a291
Document registry ingress split, image mirroring, loop OSD setup, and platform integration paths.
141 lines
3.8 KiB
Markdown
141 lines
3.8 KiB
Markdown
# CloudHost Ceph (Rook)
|
|
|
|
Helm chart and install scripts for **Rook-Ceph** on CloudHost clusters:
|
|
|
|
| Layer | Purpose |
|
|
|-------|---------|
|
|
| **rook-ceph-block** | Expandable PVCs for apps, databases, registry |
|
|
| **rook-ceph-bucket** | S3-compatible storage for uploaded source zip archives |
|
|
|
|
The chart does **not** vendor Rook itself — it installs the official [`rook-release`](https://charts.rook.io/release) charts and adds CloudHost-specific **ObjectBucketClaim** + credential sync.
|
|
|
|
---
|
|
|
|
## Quick install
|
|
|
|
```bash
|
|
cd backend/helm/cloudhost-ceph
|
|
./scripts/install.sh single-node # one-node k3s (current abr cluster)
|
|
# or
|
|
./scripts/install.sh multi-node # production, 3+ nodes + raw disks
|
|
```
|
|
|
|
Verify:
|
|
|
|
```bash
|
|
./scripts/verify.sh
|
|
```
|
|
|
|
---
|
|
|
|
## Profiles
|
|
|
|
### `single-node`
|
|
|
|
- OSD on **loop device** `/dev/loop6` (15Gi file at `/var/lib/rook/osd-loopback.img`) — no spare raw disk required
|
|
- Requires `ROOK_CEPH_ALLOW_LOOP_DEVICES=true` on the operator
|
|
- Replication **size: 1** (no HA)
|
|
- Suitable for **staging / single k3s node**
|
|
- Images must be pre-mirrored to `registry.abrban.com` (see `RUNBOOK-HARBOR.fa.md`)
|
|
|
|
### `multi-node`
|
|
|
|
- OSD on **raw devices** (`useAllDevices: true`)
|
|
- Replication **size: 3** for block + object metadata
|
|
- Erasure-coded object data pool
|
|
- Requires **3+ nodes** and dedicated disks
|
|
|
|
---
|
|
|
|
## What gets installed
|
|
|
|
| Step | Release | Namespace |
|
|
|------|---------|-----------|
|
|
| 1 | `rook-ceph` (operator) | `rook-ceph` |
|
|
| 2 | `rook-ceph-cluster` | `rook-ceph` |
|
|
| 3 | `cloudhost-ceph` (OBC + secrets) | `cloudhost-builds` |
|
|
|
|
### StorageClasses (from Rook)
|
|
|
|
| Name | Use |
|
|
|------|-----|
|
|
| `rook-ceph-block` | App PVC, DB PVC, Redis, registry, … |
|
|
| `rook-ceph-bucket` | `ObjectBucketClaim` → S3 bucket + credentials |
|
|
|
|
### CloudHost extras
|
|
|
|
| Resource | Description |
|
|
|----------|-------------|
|
|
| `ObjectBucketClaim/app-sources` | Bucket for user zip uploads |
|
|
| `Secret/ceph-app-sources-credentials` | Stable S3 credentials for backend |
|
|
| `ConfigMap/cloudhost-ceph-integration` | Suggested `PLATFORM_*` env values |
|
|
|
|
---
|
|
|
|
## Platform integration
|
|
|
|
After install, configure the **backend**:
|
|
|
|
```env
|
|
PLATFORM_STORAGE_CLASS=rook-ceph-block
|
|
PLATFORM_CREATE_STORAGE_CLASS=false
|
|
PLATFORM_STORAGE_PROVISIONER=rook-ceph.rbd.csi.ceph.com
|
|
```
|
|
|
|
Mount or env-from secret `cloudhost-builds/ceph-app-sources-credentials`:
|
|
|
|
```env
|
|
SOURCE_STORAGE_ENDPOINT=http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local:80
|
|
SOURCE_STORAGE_REGION=us-east-1
|
|
SOURCE_STORAGE_BUCKET=<from secret>
|
|
SOURCE_STORAGE_ACCESS_KEY=<from secret>
|
|
SOURCE_STORAGE_SECRET_KEY=<from secret>
|
|
```
|
|
|
|
> **Note:** Existing PVCs on `local-path` / `cloudhost-expandable` are **not** migrated automatically. New apps use `rook-ceph-block` once the backend env is updated. Plan migration per workload (see `RUNBOOK-CEPH.fa.md`).
|
|
|
|
---
|
|
|
|
## Uninstall (destructive)
|
|
|
|
```bash
|
|
./scripts/uninstall.sh
|
|
# then on each node:
|
|
sudo rm -rf /var/lib/rook /var/lib/rook/osd
|
|
```
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
```bash
|
|
# Cluster health
|
|
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph status
|
|
|
|
# OSD pods
|
|
kubectl -n rook-ceph get pods -l app=rook-ceph-osd
|
|
|
|
# RGW (object store)
|
|
kubectl -n rook-ceph get pods -l app=rook-ceph-rgw
|
|
|
|
# Bucket sync job
|
|
kubectl -n cloudhost-builds logs job -l job-name=cloudhost-ceph-bucket-sync --tail=50
|
|
```
|
|
|
|
Full operational guide (Persian): [`../../../RUNBOOK-CEPH.fa.md`](../../../RUNBOOK-CEPH.fa.md)
|
|
|
|
Registry / Harbor (Persian): [`../../../RUNBOOK-HARBOR.fa.md`](../../../RUNBOOK-HARBOR.fa.md)
|
|
|
|
---
|
|
|
|
## Files
|
|
|
|
| File | Role |
|
|
|------|------|
|
|
| `values-rook-cluster-single-node.yaml` | Rook cluster values (1 node) |
|
|
| `values-rook-cluster-multi-node.yaml` | Rook cluster values (production) |
|
|
| `values.yaml` | CloudHost OBC / secret sync |
|
|
| `scripts/install.sh` | Full install |
|
|
| `scripts/verify.sh` | Health check |
|
|
| `scripts/uninstall.sh` | Tear down |
|