mirror of
https://github.com/shumaiOne/shumai.git
synced 2026-11-22 19:37:40 +00:00
* feat(workflow): migrate workflow engine from temporal to dbos Replace Temporal and LocalExecutor with DBOS Transact (@dbos-inc/dbos-sdk) as the unified workflow engine. Background tasks, media transcoding, and AI agent workflows now execute with durable steps and task queues directly against PostgreSQL, eliminating the need for an external Temporal cluster and separate polling executors. * fix(workflow): submit tasks via DBOSClient to allow enqueueing from steps Enqueuing workflows using DBOS.enqueueWorkflowWithOptions threw a DBOSInvalidWorkflowTransitionError when triggered from within a running step or activity (e.g. creating embedding or autofill tasks during video transcoding). Use DBOSClient to submit and cancel workflow tasks directly against the system database, detaching execution from the ambient workflow/step context. * feat(web): support SHUMAI_WORKERS environment variable * fix(workflow): ensure CLI tasks are enqueued to DBOS and run DBOS migrations in global setup - Initialize workflows and wait for pending DBOS submissions in shumai regenerate-embeddings CLI - Remove no-op submit bypass and enqueue tasks via DBOSClient in producer and CLI contexts - Add DBOS migration runner in test global setup to avoid concurrent migration deadlocks across test forks - Update changelog and add regression tests * fix(workflow): ensure unique per-instance executor identity for worker replicas - Derive unique per-instance DBOS executor IDs across worker replicas - Support SHUMAI_WORKER_ID, DBOS__VMID, and POD_NAME (Kubernetes Downward API) - Disambiguate multi-replica bare metal/container deployments with host, PID, and instance suffix - Prevent startup recovery from re-enqueueing workflows actively executed by other live workers - Add multi-instance startup recovery tests and update deployment documentation * fix(workflow): only register listened queues and update concurrency on change - Only register queues that the executor instance actively listens to - Use always_update conflict resolution so environment variable changes take effect on restart - Prevent dedicated workers from overwriting unmanaged queue concurrency configurations - Add tests verifying concurrency updates and queue isolation across worker roles * refactor(workflow): simplify activities to use defineStep and direct calls (#482) * refactor(workflow): simplify activities to use defineStep and direct calls - Wrap activities using defineStep from @shumai/workflow-core - Remove obsolete getActivities, registerActivities, dynamic activity proxies, and workerQueue arguments - Remove executor.ts and registry.ts, consolidating workflow registration into workflow.ts - Call activities directly across transcode and agent workflows - Update all corresponding workflow tests and developer guide documentation * fix(workflow): properly detect DBOSWorkflowCancelledError in runOnCancel fallback - DBOSWorkflowCancelledError in DBOS SDK does not set name, inheriting 'Error' - Implement isDbosWorkflowCancelledError checking instanceof, constructor.name, and dbosErrorCode - Update unit tests to verify real DBOS SDK cancellation error instances * docs(workflow): update documentation and readme for dbos migration - Rename docs/configuration/temporal.mdx to docs/configuration/distributed.mdx - Document DBOS-based distributed worker architecture and task queues - Remove obsolete docker-compose/temporal/ - Update environment variables documentation in docs/configuration/env-variables.mdx - Update references in README.md, README_CN.md, docs/docs.json, and related guides * feat(deploy): implement KEDA ScaledJob and scale-to-zero On-Demand transcoding (#483) * feat(deploy): add shumai-transcode docker image and k8s autoscaling example with garage s3 * docs(k8s): document port-forwarding and in-cluster s3 domain mapping * feat(storage): support public s3 endpoint for browser uploads and update k8s example * fix(k8s): configure s3 bucket cors rules in garage initialization * fix(k8s): increase transcode worker memory limit to 4Gi for 4K video encoding * feat(deploy): add AWS EKS deployment with Graviton3 Spot nodegroups and autoscaling * docs(deploy): redesign EKS documentation, rename k8s to eks, and add scalable deployment feature * docs: rephrase DBOS feature bullet to Durable Workflows via DBOS * docs(eks): upgrade default instances to Graviton5 c9g.large and c9g.xlarge * feat(eks): use On-Demand for general nodegroup and add spot capacity diversification * docs(eks): fix Architecture diagram to locate all transcode worker pods in Transcode Compute Pool * feat(deploy): implement KEDA ScaledJob and scale-to-zero On-Demand transcoding * feat(transcode): support idle-timeout auto-drain for KEDA scaled jobs * fix(deploy): fix EKS deployment manifests and documentation based on cluster verification
174 lines
8.8 KiB
Plaintext
174 lines
8.8 KiB
Plaintext
---
|
|
title: Deploy with Distributed Workers
|
|
description: Configure a high-scale production deployment using dedicated background workers, DBOS, and S3-compatible storage.
|
|
---
|
|
|
|
For production environments or workloads with high-volume video processing and heavy AI agent usage, running background tasks on a single server can cause bottlenecks. Shumai resolves this by decoupling the web dashboard from background processing through dedicated worker nodes orchestrated by **DBOS Transact**.
|
|
|
|
All background tasks (media transcoding, AI embeddings, agent tasks) are dispatched to durable task queues (`transcode_queue` and `agent_queue`) hosted directly in PostgreSQL. No separate external cluster (like Temporal) is required.
|
|
|
|
---
|
|
|
|
## How it Works
|
|
|
|
In a distributed deployment, the core Shumai web application handles user requests, database updates, and file uploads, while delegating durable background workflows to specialized worker instances via PostgreSQL task queues.
|
|
|
|
```mermaid
|
|
graph TD
|
|
UserUpload[User Uploads Video / Triggers Agent] --> WebApp[Shumai Web UI / API]
|
|
WebApp -->|Enqueue Task| DB[(PostgreSQL / DBOS Queues)]
|
|
DB -->|Queue: transcode_queue| TranscodeWorker[Transcode Worker Machine]
|
|
DB -->|Queue: agent_queue| AgentWorker[Agent Worker Machine]
|
|
```
|
|
|
|
- **Transcoding Workers**: Can be run on GPU-accelerated or high-CPU compute instances.
|
|
- **AI Agent Workers**: Can be run on isolated, sandboxed hosts to safely execute automation scripts.
|
|
|
|
> [!IMPORTANT]
|
|
> **Storage Requirement**: Because the core application and the background workers run on separate machines in a distributed setup, **you must configure an S3-compatible storage backend** (such as AWS S3, Cloudflare R2, or MinIO). Local storage (`STORAGE_BACKEND=local`) is not supported in this mode because the workers and the web application do not share a common filesystem.
|
|
|
|
---
|
|
|
|
## Step 1: Deploy the Core Web Application
|
|
|
|
On your primary host machine, run PostgreSQL and the main Shumai web application.
|
|
|
|
To instruct the main web application to handle only API and UI traffic without running background worker queues locally, set `SHUMAI_WORKERS=none`:
|
|
|
|
```env
|
|
DATABASE_URL=postgresql://shumai:shumai_password@localhost:5432/shumai_db?schema=public
|
|
STORAGE_BACKEND=s3
|
|
AWS_ENDPOINT_URL_S3=https://your-s3-endpoint.com
|
|
S3_BUCKET=shumai
|
|
S3_REGION=us-east-1
|
|
S3_ACCESS_KEY_ID=your-access-key-id
|
|
S3_SECRET_ACCESS_KEY=your-secret-access-key
|
|
SHUMAI_WORKERS=none
|
|
BETTER_AUTH_SECRET=your-random-32-character-secret-key
|
|
```
|
|
|
|
When started with `SHUMAI_WORKERS=none` (or `--workers=none`), the web server starts immediately without polling or consuming tasks from the transcode or agent queues.
|
|
|
|
---
|
|
|
|
## Step 2: Configure & Run Dedicated Workers
|
|
|
|
On your specialized worker machines, configure each worker with a `.env` file referencing the shared PostgreSQL database and S3 storage:
|
|
|
|
```env
|
|
DATABASE_URL=postgresql://shumai:shumai_password@<primary-host-ip>:5432/shumai_db?schema=public
|
|
STORAGE_BACKEND=s3
|
|
AWS_ENDPOINT_URL_S3=https://your-s3-endpoint.com
|
|
S3_BUCKET=shumai
|
|
S3_REGION=us-east-1
|
|
S3_ACCESS_KEY_ID=your-access-key-id
|
|
S3_SECRET_ACCESS_KEY=your-secret-access-key
|
|
CONCURRENCY_TRANSCODE=1
|
|
CONCURRENCY_AGENT=5
|
|
```
|
|
|
|
### Media Transcoding Worker
|
|
|
|
Run this on your media processing nodes (requires system installations of `ffmpeg` and `ffprobe`):
|
|
|
|
```bash
|
|
npm install -g @shumai-one/shumai-transcode
|
|
shumai-transcode
|
|
```
|
|
|
|
Alternatively, you can run the transcode worker using the dedicated lightweight Docker image:
|
|
|
|
```bash
|
|
docker run -d \
|
|
--name shumai-transcode-worker \
|
|
--env-file .env \
|
|
shumaione/shumai-transcode:latest
|
|
```
|
|
|
|
(You can also use the all-in-one image: `docker run -d --name shumai-transcode-worker --env-file .env shumaione/shumai:latest shumai-transcode`.)
|
|
|
|
### AI Agent Worker
|
|
|
|
Run this on your sandboxed agent execution nodes (requires sandbox dependencies like `bubblewrap` or `sandbox-exec`):
|
|
|
|
```bash
|
|
npm install -g @shumai-one/shumai-agent
|
|
shumai-agent
|
|
```
|
|
|
|
Alternatively, you can run the agent worker using the official Docker image:
|
|
|
|
```bash
|
|
docker run -d \
|
|
--name shumai-agent-worker \
|
|
--env-file .env \
|
|
--cap-add=SYS_ADMIN \
|
|
shumaione/shumai:latest shumai-agent
|
|
```
|
|
|
|
> [!IMPORTANT]
|
|
> **For Bun Users:** Since the worker package binaries contain a `#!/usr/bin/env node` shebang, running `shumai-transcode` or `shumai-agent` directly will run under Node.js. If you wish to run them using Bun, prefix the commands with `bun run --bun` (e.g., `bun run --bun shumai-transcode` or `bun run --bun shumai-agent`).
|
|
|
|
---
|
|
|
|
## Worker Concurrency & Multi-Instance Deployments
|
|
|
|
### Concurrency Tuning
|
|
|
|
Workers pull tasks from PostgreSQL queues based on configurable concurrency limits:
|
|
|
|
- `CONCURRENCY_TRANSCODE`: Maximum concurrent video transcoding jobs per worker (defaults to `1`).
|
|
- `CONCURRENCY_AGENT`: Maximum concurrent agent and embedding workflows per worker (defaults to `5`, recommend setting to `1` when paired with a local `embedding-server`).
|
|
|
|
### Replica Identity
|
|
|
|
When running multiple replicas of the transcode or agent worker across multiple machines or pods:
|
|
|
|
- `DbosExecutor` automatically derives a unique per-instance ID from the machine hostname and process ID to ensure startup recovery only re-enqueues tasks belonging to that specific instance.
|
|
- **Custom Worker ID**: Set `SHUMAI_WORKERS_ID` (or `SHUMAI_WORKER_ID`) if you wish to assign an explicit identifier.
|
|
- **Kubernetes**: In Kubernetes Pods, set `POD_NAME` (via Downward API `fieldRef: { fieldPath: metadata.name }`) to provide stable, unique per-pod identity across pod restarts.
|
|
|
|
---
|
|
|
|
## Scalable Deployment on Kubernetes & AWS EKS
|
|
|
|
For production cloud deployments requiring high-throughput media processing, Shumai can be deployed on Kubernetes or **AWS Elastic Kubernetes Service (AWS EKS)** with automated scale-to-zero queue-driven scaling via **KEDA (Kubernetes Event-driven Autoscaling)**.
|
|
|
|
```mermaid
|
|
graph TD
|
|
User([User / Web Browser]) -->|Port 80/3000| Server[Pod: shumai-server<br/>SHUMAI_WORKERS=agent]
|
|
Server -->|Enqueue to transcode_queue| DB[(PostgreSQL: pgvector18<br/>dbos.workflow_status)]
|
|
Server -->|Direct Uploads / Stream Proxies| Storage[(S3 / Garage Storage)]
|
|
|
|
KEDA[KEDA ScaledJob<br/>PostgreSQL Trigger<br/>min: 0, max: 10] -.->|Monitors Queue| DB
|
|
KEDA -->|Spawns Jobs for queue| Job1[Job: shumai-transcode-1<br/>Auto-drain worker]
|
|
KEDA -->|Spawns Jobs for queue| JobN[Job: shumai-transcode-N<br/>Auto-drain worker]
|
|
|
|
Job1 -->|Fetch Source / Store Proxies| Storage
|
|
JobN -->|Fetch Source / Store Proxies| Storage
|
|
Job1 -->|Mark Complete & Exit 0| DB
|
|
JobN -->|Mark Complete & Exit 0| DB
|
|
|
|
CA[Cluster Autoscaler / Karpenter] -.->|Scale-to-Zero On-Demand Nodes<br/>0 to 10 instances| JobN
|
|
```
|
|
|
|
### Key Architectural Highlights
|
|
|
|
- **Decoupled Stateless Workers**: The main web and API server, database, and storage run on reliable On-Demand compute (`c9g.large`, `SHUMAI_WORKERS=agent`), delegating media transcoding to background workers orchestrated via DBOS task queues in PostgreSQL.
|
|
- **Dedicated Graviton5 On-Demand Compute (`c9g.xlarge`)**: Media encoding runs on 4 vCPU / 8 GiB RAM AWS Graviton5 On-Demand instances (~$0.174/hr), utilizing ARM64 vector optimizations and high-speed DDR5 memory without involuntary Spot interruptions.
|
|
- **Queue-Driven Scale-to-Zero via KEDA**: KEDA monitors `dbos.workflow_status` for enqueued and pending transcode tasks and dynamically provisions Kubernetes `ScaledJob` resources (scaling 0 to 10). When the queue is empty, transcode nodes scale down to zero (`minSize: 0`), incurring **$0/hr** in worker costs.
|
|
- **Idle-Timeout Auto-Drain**: Worker pods continuously process queued tasks with concurrency `1`. When the queue is fully drained and a pod has been idle for 10 seconds (`SHUMAI_WORKER_IDLE_TIMEOUT_MS=10000`), it exits `0` cleanly so the node can scale down, processing batch uploads without cold-start pod churn.
|
|
- **Automated Stale Task Recovery**: The core web server runs an integrated recovery sweep that detects stale or orphaned tasks (e.g. from node crash) and resumes them automatically in DBOS, with a 3-attempt cap to safely handle poison-pill media.
|
|
|
|
### Complete Deployment Example & Source Code
|
|
|
|
For the full, production-ready manifests, declarative `eksctl` cluster configuration, and step-by-step walkthrough, see the **[EKS Deployment Example](https://github.com/shumaiOne/shumai/tree/main/eks)** in the Shumai repository:
|
|
|
|
👉 **[View the EKS Deployment Guide & Manifests on GitHub](https://github.com/shumaiOne/shumai/tree/main/eks)**
|
|
|
|
The repository includes:
|
|
|
|
- [`eks/eks-cluster.yaml`](https://github.com/shumaiOne/shumai/blob/main/eks/eks-cluster.yaml): Complete `eksctl` ClusterConfig with compute tiering.
|
|
- [`eks/kustomization.yaml`](https://github.com/shumaiOne/shumai/blob/main/eks/kustomization.yaml): Single-command Kustomize deployment for all services.
|
|
- Detailed instructions for setting up public endpoints, automated Cluster Autoscaler, and zero-downtime graceful shutdown.
|