abrban/cloud-host
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(build): revamp app build pipeline (queue, Nixpacks, MinIO, Trivy, registry GC)

Browse Source 
Rework the application build/deploy pipeline for scalability, reproducibility,
and security:

- Build queue: deploys run through a bounded-concurrency Bull queue
  (BUILD_CONCURRENCY, default 3) so concurrent user deploys can't flood the
  cluster with Kaniko jobs. Build state (progress / cancel / session) moves from
  in-memory Maps to Redis, so cancel + live logs work across backend replicas.
- Nixpacks + BYO Dockerfile: code runtimes build via Nixpacks (or the user's own
  Dockerfile when present); the hand-written per-runtime Dockerfile generators
  and runtime auto-detection are removed. WordPress keeps its templated path.
  Build-time mirror env (NIXPACKS_BUILD_ENV) supports the Iran network.
- Source upload to MinIO: archives stream to in-cluster MinIO; build pods pull
  via a presigned URL. Removes the PVC + helper pod + kubectl cp upload path.
- Report-only Trivy scan after build; per-severity summary stored on the
  deployment and shown as a badge in the dashboard. Never gates a deploy.
- Registry GC: a Redis-locked daily job keeps the newest N image tags per app
  (REGISTRY_KEEP_VERSIONS, default 3) and reclaims disk via garbage-collect.
- Hardening: git tokens are delivered via a per-build Secret + git credential
  store instead of being embedded in the clone URL / Job manifest; build timeout
  is configurable.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
keyhan
2026-06-20 22:58:58 +03:30
parent 49726f1dfd
commit 3eff38f8d2
27 changed files with 1950 additions and 1295 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/src/applications/applications.service.ts
+19 -22
View File
@@ -17,6 +17,7 @@ import {
} from '../common/enums';
import { ensureAppUrlEnv } from './app-url.util';
import { normalizeCreateApplicationDto } from './managed-service.util';
import { StorageService } from '../common/storage/storage.service';
@Injectable()
export class ApplicationsService {
@@ -27,6 +28,7 @@ export class ApplicationsService {
private appsRepository: Repository<Application>,
private clustersService: ClustersService,
private configService: ConfigService,
private storageService: StorageService,
) {}
private toDnsLabel(value: string): string {
@@ -204,18 +206,20 @@ export class ApplicationsService {
async delete(id: string, userId: string): Promise<Application> {
const app = await this.findOne(id, userId);
// Delete uploaded files
// Delete the uploaded source archive from object storage.
if (app.codePath) {
try {
const uploadDir = this.configService.get<string>('platform.uploadDir') || './uploads';
const appDir = path.join(uploadDir, app.userId, app.id);
if (fs.existsSync(appDir)) {
fs.rmSync(appDir, { recursive: true, force: true });
this.logger.log(`Deleted upload directory: ${appDir}`);
}
} catch (e: any) {
this.logger.warn(`Failed to delete upload dir for ${app.name}: ${e.message}`);
await this.storageService.removeSource(app.codePath);
}
// Remove any legacy on-disk dump/source dir (db dumps are still stored locally).
try {
const uploadDir = this.configService.get<string>('platform.uploadDir') || './uploads';
const appDir = path.join(uploadDir, app.userId, app.id);
if (fs.existsSync(appDir)) {
fs.rmSync(appDir, { recursive: true, force: true });
this.logger.log(`Deleted upload directory: ${appDir}`);
}
} catch (e: any) {
this.logger.warn(`Failed to delete upload dir for ${app.name}: ${e.message}`);
}
await this.appsRepository.remove(app);
@@ -261,21 +265,14 @@ export class ApplicationsService {
}
const app = await this.findOne(id, userId);
const uploadDir = this.configService.get<string>('platform.uploadDir') || './uploads';
const appDir = path.join(uploadDir, app.userId, app.id);
// Ensure directory exists
fs.mkdirSync(appDir, { recursive: true });
// Save the zip file
const zipPath = path.join(appDir, 'source.zip');
fs.writeFileSync(zipPath, file.buffer);
// Update app with code path
app.codePath = zipPath;
// Stream the archive to MinIO; codePath stores the object key (build pods
// pull it via a presigned URL — no local disk, no PVC, no kubectl cp).
const key = await this.storageService.putSource(app.userId, app.id, file.buffer);
app.codePath = key;
const saved = await this.appsRepository.save(app);
this.logger.log(`Uploaded code for ${app.name} → ${zipPath} (${(file.size / 1024).toFixed(1)} KB)`);
this.logger.log(`Uploaded code for ${app.name} → ${key} (${(file.size / 1024).toFixed(1)} KB)`);
return saved;
}