keyhan
d3bbc0c0a0
Build and Deploy Platform / build-and-deploy (push) Successful in 14m57s
User-app builds no longer pull gcr.io/docker.io directly when build.images is configured in values.yaml, fixing ImagePullBackOff on clusters without upstream registry access. Co-authored-by: Cursor <cursoragent@cursor.com>
335 lines
16 KiB
Markdown
335 lines
16 KiB
Markdown
# ☁️ CloudHost — Self-Service PaaS Platform
|
||
|
||
A self-service Platform-as-a-Service (PaaS) that lets developers deploy applications
|
||
onto Kubernetes with zero DevOps overhead. Source code is turned into a container
|
||
image **inside the cluster** with Kaniko (no Docker daemon), then rolled out with Helm —
|
||
complete with managed databases, wallet-based billing, automated lifecycle management,
|
||
live logs, and a bilingual (Persian/English) panel.
|
||
|
||
Supported runtimes — each built from a platform-maintained `Dockerfile` template:
|
||
**Node.js, Laravel, Go, PHP, Python, Django, .NET**, and **WordPress** (official image +
|
||
custom `wp-content` entrypoint).
|
||
|
||
> 🇮🇷 Production deployment on the `abrban.com` k3s cluster — including all the
|
||
> Iran-network workarounds — is documented step-by-step in **[RUNBOOK.fa.md](RUNBOOK.fa.md)** (Persian).
|
||
|
||
> 🔄 **CI/CD (Gitea Actions → Kaniko → Harbor → Argo CD):** see **[RUNBOOK-CICD.fa.md](RUNBOOK-CICD.fa.md)** (Persian) and **[gitops/README.md](gitops/README.md)** for bootstrap (`seed-ci-images`, Sealed Secrets, two-repo GitOps layout).
|
||
>
|
||
> 🚀 **Deploy from zero (any cluster):** **[RUNBOOK-DEPLOY.fa.md](RUNBOOK-DEPLOY.fa.md)** — server checklist, values, secrets, logging, greenfield reset.
|
||
|
||
---
|
||
|
||
## Architecture Overview
|
||
|
||
```
|
||
┌──────────────┐ REST ┌──────────────────┐ K8s API ┌──────────────┐
|
||
│ Next.js 16 │ /api/v1 │ NestJS 11 API │ + Helm │ Kubernetes │
|
||
│ Frontend │◄─────────►│ Backend │◄───────────►│ Cluster(s) │
|
||
└──────────────┘ └────────┬─────────┘ └──────┬───────┘
|
||
│ │ build Jobs
|
||
┌─────────────────┼─────────────────┐ ▼
|
||
▼ ▼ ▼ ┌──────────┐
|
||
PostgreSQL Redis Registry │ Kaniko │
|
||
16 (cache + Bull) (:2) └──────────┘
|
||
```
|
||
|
||
| Layer | Technology |
|
||
| ---------------- | ----------------------------------------------------------------- |
|
||
| Frontend | Next.js 16 (App Router, SSR), React 19, Tailwind CSS v4, React Query, Zustand |
|
||
| Backend API | NestJS 11, TypeORM, Passport JWT, Bull (Redis) |
|
||
| Build engine | **Kaniko** (daemon-less in-cluster builds) with platform-generated per-runtime Dockerfiles |
|
||
| Source ingestion | Uploaded archive (zip/tar.gz) streamed into a build PVC, **or** git clone |
|
||
| Deployment | Helm v3 charts, `@kubernetes/client-node` |
|
||
| Database | PostgreSQL 16 (control plane); per-app MySQL/MariaDB/PostgreSQL/MongoDB |
|
||
| Queue / cache | Redis 7 + Bull (service-access grants, app migrations) |
|
||
| Registry | In-cluster `registry:2` |
|
||
| Auth | Mobile number + **OTP** (SMS) and password, JWT access/refresh |
|
||
|
||
> 📖 See **[ARCHITECTURE.md](ARCHITECTURE.md)** for detailed system design.
|
||
> 🔼 See [UPGRADE.en.md](UPGRADE.en.md) ([فارسی](UPGRADE.md)) for dependency-upgrade notes.
|
||
> 📋 See [RUNBOOK.en.md](RUNBOOK.en.md) ([فارسی](RUNBOOK.fa.md)) for operations.
|
||
|
||
---
|
||
|
||
## Features
|
||
|
||
### For Developers
|
||
- 🚀 **Deploy from a code archive (zip/tar.gz) _or_ a git URL** (public or private via token)
|
||
- 🟢 **Multi-runtime** — Node.js, Laravel, Go, PHP, Python, Django, .NET, each built from a maintained Dockerfile template
|
||
- 🔵 **WordPress** — official image + custom entrypoint that merges your `wp-content`
|
||
- 🗄️ **Managed databases & services** — PostgreSQL, MySQL, MariaDB, MongoDB, Redis, RabbitMQ provisioned via Helm
|
||
- 💰 **Wallet system** — deposit funds, pay per plan (hourly / monthly / yearly), coupons & discounts
|
||
- 📊 **Live build & runtime logs** (Elasticsearch-backed) + deployment history with rollback
|
||
- 🔒 **Environment variables** stored as Kubernetes Secrets
|
||
- ⚙️ **Resource controls** — CPU, memory, replicas, expandable disk
|
||
- 🌐 **Custom domains** with automatic TLS
|
||
- 📸 **Snapshots** — backup & restore application state
|
||
- 🎫 **Support tickets** with technical/sales departments
|
||
|
||
### For Super Admins
|
||
- 🖥️ **Multi-cluster management** — register/remove Kubernetes clusters (kubeconfig stored encrypted)
|
||
- 👥 **User management** — activate, deactivate, change roles, per-user detail dashboard
|
||
- 📈 **Quotas & pricing** — per-cluster limits and a configurable pricing catalog
|
||
- 💳 **Billing oversight** — transactions, invoices, wallet deposits, global discount
|
||
- ⏱️ **Lifecycle settings** — grace periods per billing cycle
|
||
- 🔐 **RBAC** — role-based guards on every endpoint (`user` / `admin` / `technical` / `sales`)
|
||
|
||
---
|
||
|
||
## How the Build & Deploy Pipeline Works
|
||
|
||
When a user triggers a deploy, the backend runs the build-and-deploy pipeline and creates
|
||
the build as a **Kubernetes Job** in the `cloudhost-builds` namespace:
|
||
|
||
```
|
||
1. SOURCE
|
||
├─ uploaded archive → saved to disk (UPLOAD_DIR) → streamed into a per-build PVC
|
||
│ via a short-lived helper pod + `kubectl cp`, then unpacked (init: prepare source)
|
||
└─ git URL → cloned in-pod; private repos inject the token into the clone URL (init: git-clone)
|
||
|
||
2. DOCKERFILE
|
||
The platform detects the runtime (or uses the app's selected runtime) and generates a
|
||
Dockerfile for it — Node.js, Laravel, WordPress, Go, PHP, Python, Django, or .NET.
|
||
|
||
3. BUILD (container: kaniko)
|
||
Kaniko builds the image (layer cache per user) and pushes it to the in-cluster
|
||
registry — no Docker daemon, no privileged pod.
|
||
|
||
4. DEPLOY
|
||
Helm installs/upgrades the `cloudhost-app` chart → Deployment, Service, Ingress,
|
||
per-app DB/Redis/RabbitMQ, PVCs, Secrets, log shipper. App goes live at its subdomain.
|
||
```
|
||
|
||
The build runs inline within the deploy request and its progress/logs are tracked in
|
||
memory, then polled by the frontend. (Bull/Redis queues are used elsewhere — service-access
|
||
grants and application migrations — but not for image builds.)
|
||
|
||
---
|
||
|
||
## Project Structure
|
||
|
||
```
|
||
cloud-host/
|
||
├── README.md # This file
|
||
├── ARCHITECTURE.md # Detailed system design
|
||
├── RUNBOOK.fa.md # Persian runbook: local dev + abrban/k3s production deploy
|
||
├── RUNBOOK-DEPLOY.fa.md # Deploy platform from zero (any cluster): values, secrets, health checks
|
||
├── RUNBOOK-CICD.fa.md # CI/CD pipeline: Gitea Actions → Kaniko → Argo CD
|
||
├── CHANGELOG.md / CONTRIBUTING.md / UPGRADE.md / UPGRADE.en.md
|
||
├── docker-compose.yml # Local dev stack (Postgres + Redis + API + UI)
|
||
│
|
||
├── backend/ # NestJS 11 API (REST under /api/v1)
|
||
│ ├── Dockerfile
|
||
│ ├── helm/
|
||
│ │ ├── cloudhost-platform/ # Helm chart — control plane (API, UI, Postgres, Redis)
|
||
│ │ ├── cloudhost-app/ # Helm chart — a single user application + its services
|
||
│ │ └── cloudhost-logging/ # Helm chart — Elasticsearch / Kibana / Fluent-bit
|
||
│ ├── k8s/ # Standalone manifests (logging, mail)
|
||
│ ├── migrations/ # SQL migrations (applied via one-off Jobs in prod)
|
||
│ └── src/
|
||
│ ├── main.ts / app.module.ts
|
||
│ ├── auth/ # Mobile-OTP + password login, JWT strategies, role guards
|
||
│ ├── users/ # User CRUD, profile, phone verification
|
||
│ ├── admin/ # Super-admin user-detail dashboard & ops
|
||
│ ├── applications/ # App CRUD, code upload (→ disk), git config
|
||
│ ├── application-migrations/ # Import/migrate existing apps (Bull queue)
|
||
│ ├── deployments/ # Deploy orchestration, history, stop/restart
|
||
│ ├── build/ # Kaniko build (build.service): per-runtime Dockerfile generation
|
||
│ ├── kubernetes/ # K8s client, Helm wrapper, registry service
|
||
│ ├── clusters/ # Multi-cluster management, kubeconfig storage
|
||
│ ├── billing/ # Wallet, transactions, invoices, pricing catalog, coupons
|
||
│ ├── lifecycle/ # Scanner: auto-suspend/delete expired apps
|
||
│ ├── snapshots/ # App snapshot/restore
|
||
│ ├── tickets/ # Support tickets
|
||
│ ├── notifications/ # User notifications
|
||
│ ├── access/ # Time-limited external service access (NodePort grants, Bull queue)
|
||
│ ├── common/ # Enums, guards, decorators
|
||
│ └── config/ # Env configuration loader + TypeORM config
|
||
│
|
||
├── frontend/ # Next.js 16 App Router (bilingual fa-IR / en-US)
|
||
│ ├── Dockerfile # ARG NEXT_PUBLIC_API_URL baked at build time
|
||
│ └── src/
|
||
│ ├── middleware.ts # Locale routing + landing (abrban.com) vs panel split
|
||
│ ├── app/[lang]/
|
||
│ │ ├── page.tsx # Landing
|
||
│ │ ├── login/ register/
|
||
│ │ ├── blog/
|
||
│ │ └── dashboard/ # apps, deploy, logs, invoices, wallet, services,
|
||
│ │ │ # tickets, account, staff, admin
|
||
│ │ └── ...
|
||
│ ├── components/ hooks/ lib/ (API client, auth store) types/
|
||
│ └── i18n/ # Dictionaries, provider, language switcher
|
||
```
|
||
|
||
---
|
||
|
||
## Quick Start (Local Development)
|
||
|
||
**Prerequisites:** Node.js ≥ 20, Docker & Docker Compose, and (for actually building/deploying
|
||
user apps) a Kubernetes cluster reachable via kubeconfig.
|
||
|
||
> ℹ️ The API and UI run fine locally against Postgres + Redis. The **build/deploy pipeline
|
||
> itself runs as Kubernetes Jobs**, so triggering a real user-app build requires a cluster
|
||
> (with the in-cluster registry). For pure UI/API development you don't need one.
|
||
|
||
### 1. Clone & install
|
||
|
||
```bash
|
||
git clone <repo-url> cloud-host && cd cloud-host
|
||
cd backend && npm install && cd ..
|
||
cd frontend && npm install && cd ..
|
||
```
|
||
|
||
### 2. Environment variables
|
||
|
||
```bash
|
||
cp backend/.env.example backend/.env
|
||
cp frontend/.env.local.example frontend/.env.local
|
||
# Edit both — at minimum DB, JWT, Redis. See the Configuration table below.
|
||
```
|
||
|
||
### 3. Start Postgres + Redis
|
||
|
||
```bash
|
||
docker compose up -d postgres redis
|
||
```
|
||
|
||
### 4. Run the apps
|
||
|
||
```bash
|
||
# Terminal 1 — Backend (http://localhost:4000, prefix /api/v1, Swagger at /docs)
|
||
cd backend && npm run start:dev
|
||
|
||
# Terminal 2 — Frontend (http://localhost:3000)
|
||
cd frontend && npm run dev
|
||
```
|
||
|
||
In development `NODE_ENV=development`, so TypeORM `synchronize` builds the schema
|
||
automatically and the pricing catalog self-seeds. Set `frontend` `NEXT_PUBLIC_API_URL`
|
||
to the backend URL.
|
||
|
||
> To run the **whole** stack (API + UI + Postgres + Redis) in containers instead:
|
||
> `docker compose up --build` (backend on `:4000`, frontend on `:3000`).
|
||
|
||
---
|
||
|
||
## Deploy on Kubernetes (Helm)
|
||
|
||
> **Production GitOps (from zero):** [`RUNBOOK-DEPLOY.fa.md`](RUNBOOK-DEPLOY.fa.md) — variable table, values, Sealed Secrets, logging, health checks.
|
||
>
|
||
> **Production abrban.com specifics:** [`RUNBOOK.fa.md`](RUNBOOK.fa.md) — Iran network, Ceph, Harbor details.
|
||
>
|
||
> **CI/CD pipeline:** [`RUNBOOK-CICD.fa.md`](RUNBOOK-CICD.fa.md).
|
||
|
||
This section is the **generic Helm-only** path (Path B in RUNBOOK-DEPLOY) without Gitea/Argo.
|
||
|
||
**Prerequisites:** a Kubernetes cluster, an Ingress controller (Traefik on k3s by default,
|
||
or set `INGRESS_CLASS=nginx`), a default StorageClass for PVCs, and a container registry
|
||
reachable by the cluster.
|
||
|
||
### 1. Build & push the platform images
|
||
|
||
```bash
|
||
export REG=your-registry.example.com
|
||
docker build -t $REG/cloudhost-backend:1.0.0 ./backend
|
||
docker build -t $REG/cloudhost-frontend:1.0.0 \
|
||
--build-arg NEXT_PUBLIC_API_URL=https://api.platform.example.com ./frontend
|
||
docker push $REG/cloudhost-backend:1.0.0
|
||
docker push $REG/cloudhost-frontend:1.0.0
|
||
```
|
||
|
||
### 2. Install the control plane
|
||
|
||
```bash
|
||
cp backend/helm/cloudhost-platform/values-production.example.yaml my-values.yaml
|
||
# Edit my-values.yaml: image tags, ingress hosts, postgres password, jwtSecret, registry, SMS/OTP
|
||
|
||
helm upgrade --install cloudhost ./backend/helm/cloudhost-platform \
|
||
-n cloudhost --create-namespace \
|
||
-f my-values.yaml \
|
||
--set images.backend.tag=1.0.0 \
|
||
--set images.frontend.tag=1.0.0
|
||
```
|
||
|
||
Key values: `ingress.enabled`, `ingress.tls.*`, `ingress.frontend.host`, `ingress.api.host`,
|
||
`postgres.password`, `secrets.jwtSecret`, `migrations.enabled`. Chart defaults live in
|
||
`backend/helm/cloudhost-platform/values.yaml`; post-install notes via
|
||
`helm get notes cloudhost -n cloudhost`.
|
||
|
||
### 3. Cluster-side prerequisites for the build pipeline
|
||
|
||
Ensure the `cloudhost-builds` namespace has:
|
||
|
||
- the in-cluster **registry** (`registry:2`) reachable at `REGISTRY_URL`,
|
||
- a `kaniko-builder` ServiceAccount with an `imagePullSecret` for the registry,
|
||
- enough ephemeral storage for the per-build source PVC + helper pod.
|
||
|
||
### 4. Verify
|
||
|
||
```bash
|
||
kubectl get deploy -n cloudhost # backend & frontend 1/1
|
||
helm status cloudhost -n cloudhost # STATUS: deployed
|
||
curl -s -o /dev/null -w '%{http_code}\n' https://<frontend.host>
|
||
```
|
||
|
||
---
|
||
|
||
## Configuration (key env vars)
|
||
|
||
| Variable | Description | Default |
|
||
|----------|-------------|---------|
|
||
| `PORT` | Backend port | `4000` |
|
||
| `DB_HOST` / `DB_PORT` / `DB_USERNAME` / `DB_PASSWORD` / `DB_DATABASE` | PostgreSQL connection | `localhost` / `5432` / `cloudhost` / — / `cloudhost` |
|
||
| `JWT_SECRET` / `JWT_EXPIRES_IN` | Access token secret + TTL | — / `1h` |
|
||
| `JWT_REFRESH_SECRET` / `JWT_REFRESH_EXPIRES_IN` | Refresh token secret + TTL | — / `7d` |
|
||
| `REDIS_HOST` / `REDIS_PORT` | Redis (cache + Bull queues) | `localhost` / `6379` |
|
||
| `SMS_PROVIDER` | OTP provider (`mizbansms` \| `kavenegar`) | `mizbansms` |
|
||
| `MIZBANSMS_USERNAME` / `MIZBANSMS_PASSWORD` / `MIZBANSMS_FROM` | OTP SMS credentials (required or OTP send 503s) | — |
|
||
| `REGISTRY_URL` / `REGISTRY_PULL_URL` | In-cluster registry (push / pull) | `registry.cloudhost-builds.svc.cluster.local:5000` |
|
||
| `BUILD_NAMESPACE` / `BUILD_SERVICE_ACCOUNT` | Build Jobs namespace + SA | `cloudhost-builds` / `kaniko-builder` |
|
||
| `KANIKO_IMAGE` | Kaniko executor image | `registry.abrban.com/proxy-gcr/kaniko-project/executor:v1.23.2` |
|
||
| `BUILD_ALPINE_IMAGE` | Alpine image for build init/helper pods | `registry.abrban.com/proxy-dockerhub/library/alpine:3.19` |
|
||
| `BUILD_ALPINE_GIT_IMAGE` | Git-clone init container image | `registry.abrban.com/proxy-dockerhub/alpine/git:2.43.0` |
|
||
| `BASE_IMAGE_REGISTRY` | Harbor prefix for Docker Hub images in generated Dockerfiles | `registry.abrban.com/proxy-dockerhub/library` |
|
||
| `UPLOAD_DIR` | Disk path for uploaded source archives | `./uploads` |
|
||
| `INGRESS_CLASS` | Ingress controller for app Ingress objects | `traefik` |
|
||
| `PLATFORM_DOMAIN` / `PREVIEW_BASE_DOMAIN` | Base domain for app subdomains / previews | `apps.cloudhost.local` / — |
|
||
| `PLATFORM_STORAGE_CLASS` | StorageClass for new PVCs (needs volume expansion) | `cloudhost-expandable` |
|
||
| `ELASTICSEARCH_HOST` / `ELASTICSEARCH_PORT` | Log search backend | cluster DNS / `9200` |
|
||
| `LIFECYCLE_SCAN_INTERVAL_MS` | Lifecycle scanner tick | `60000` |
|
||
|
||
---
|
||
|
||
## Authentication
|
||
|
||
Login is **mobile-number based**: the user receives a one-time SMS code (OTP) and can also
|
||
set a password. On every request `JwtStrategy` re-reads the user's **role and active status
|
||
from the database** (not from the token), so promotions/deactivations take effect immediately.
|
||
Tokens: JWT access (`JWT_EXPIRES_IN`, default 1h) + refresh (7d).
|
||
|
||
---
|
||
|
||
## API
|
||
|
||
All endpoints are prefixed with `/api/v1`. Interactive Swagger docs at
|
||
`http://localhost:4000/api/docs`. Major route groups: `auth` (OTP request/verify, login,
|
||
refresh), `applications`, `deployments`, `clusters`, `billing` (wallet, invoices,
|
||
transactions, pricing), `snapshots`, `tickets`, `users`, `admin`, `notifications`.
|
||
|
||
---
|
||
|
||
## Security
|
||
|
||
- **JWT** access + refresh tokens; live role/active-status enforcement from DB
|
||
- **Bcrypt** password hashing
|
||
- **Helmet** HTTP security headers, **class-validator** on all DTOs
|
||
- **RBAC** role-based route guards (`@Roles(...)`)
|
||
- **Namespace isolation** — each user deploys to their own Kubernetes namespace
|
||
- **Secrets** — env vars stored as K8s Secrets
|
||
|
||
---
|
||
|
||
## License
|
||
|
||
MIT
|