feat(build): revamp app build pipeline (queue, Nixpacks, MinIO, Trivy, registry GC)
Rework the application build/deploy pipeline for scalability, reproducibility, and security: - Build queue: deploys run through a bounded-concurrency Bull queue (BUILD_CONCURRENCY, default 3) so concurrent user deploys can't flood the cluster with Kaniko jobs. Build state (progress / cancel / session) moves from in-memory Maps to Redis, so cancel + live logs work across backend replicas. - Nixpacks + BYO Dockerfile: code runtimes build via Nixpacks (or the user's own Dockerfile when present); the hand-written per-runtime Dockerfile generators and runtime auto-detection are removed. WordPress keeps its templated path. Build-time mirror env (NIXPACKS_BUILD_ENV) supports the Iran network. - Source upload to MinIO: archives stream to in-cluster MinIO; build pods pull via a presigned URL. Removes the PVC + helper pod + kubectl cp upload path. - Report-only Trivy scan after build; per-severity summary stored on the deployment and shown as a badge in the dashboard. Never gates a deploy. - Registry GC: a Redis-locked daily job keeps the newest N image tags per app (REGISTRY_KEEP_VERSIONS, default 3) and reclaims disk via garbage-collect. - Hardening: git tokens are delivered via a per-build Secret + git credential store instead of being embedded in the clone URL / Job manifest; build timeout is configurable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -125,9 +125,45 @@ export default () => ({
|
||||
password: process.env.REGISTRY_PASSWORD || '',
|
||||
},
|
||||
|
||||
// In-cluster MinIO (S3-compatible) for application source archives.
|
||||
minio: {
|
||||
endpoint: process.env.MINIO_ENDPOINT || 'minio.cloudhost-builds.svc.cluster.local',
|
||||
port: parseInt(process.env.MINIO_PORT || '9000', 10),
|
||||
useSSL: process.env.MINIO_USE_SSL === 'true',
|
||||
accessKey: process.env.MINIO_ACCESS_KEY || 'cloudhost',
|
||||
secretKey: process.env.MINIO_SECRET_KEY || 'CloudHost2024!Minio',
|
||||
bucket: process.env.MINIO_BUCKET || 'app-sources',
|
||||
},
|
||||
|
||||
build: {
|
||||
namespace: process.env.BUILD_NAMESPACE || 'cloudhost-builds',
|
||||
serviceAccount: process.env.BUILD_SERVICE_ACCOUNT || 'kaniko-builder',
|
||||
/** Max number of build+deploy pipelines processed concurrently across the queue. */
|
||||
concurrency: parseInt(process.env.BUILD_CONCURRENCY || '3', 10),
|
||||
/** Per-image-build timeout (Kaniko job) in seconds. */
|
||||
timeoutSeconds: parseInt(process.env.BUILD_TIMEOUT_SECONDS || '600', 10),
|
||||
/** Nixpacks builder image used to generate a Dockerfile for code runtimes. */
|
||||
nixpacksImage: process.env.NIXPACKS_IMAGE || 'ghcr.io/railwayapp/nixpacks:latest',
|
||||
/**
|
||||
* Build-time env baked into Nixpacks-generated images (mirrors/proxies for the
|
||||
* Iran network, e.g. "NPM_CONFIG_REGISTRY=https://registry.npmmirror.com").
|
||||
* Comma-separated KEY=VALUE pairs — set per the Phase 0 spike findings.
|
||||
*/
|
||||
nixpacksBuildEnv: (process.env.NIXPACKS_BUILD_ENV || '')
|
||||
.split(',')
|
||||
.map((s) => s.trim())
|
||||
.filter(Boolean),
|
||||
/** Report-only Trivy image scan after a successful build. */
|
||||
scanEnabled: process.env.BUILD_SCAN_ENABLED !== 'false',
|
||||
trivyImage: process.env.TRIVY_IMAGE || 'aquasec/trivy:latest',
|
||||
/** Optional mirror for Trivy's vulnerability DB (Iran network); empty = default ghcr.io. */
|
||||
trivyDbRepository: process.env.TRIVY_DB_REPOSITORY || '',
|
||||
/** Max seconds to wait for the Trivy scan job. */
|
||||
scanTimeoutSeconds: parseInt(process.env.BUILD_SCAN_TIMEOUT_SECONDS || '300', 10),
|
||||
/** Registry garbage collection: keep the N most recent image tags per app repo. */
|
||||
registryGcEnabled: process.env.REGISTRY_GC_ENABLED !== 'false',
|
||||
registryKeepVersions: parseInt(process.env.REGISTRY_KEEP_VERSIONS || '3', 10),
|
||||
registryGcIntervalMs: parseInt(process.env.REGISTRY_GC_INTERVAL_MS || '86400000', 10), // daily
|
||||
},
|
||||
|
||||
elasticsearch: {
|
||||
|
||||
Reference in New Issue
Block a user