--- title: Deploy with Distributed Workers description: Configure a high-scale production deployment using dedicated background workers, DBOS, and S3-compatible storage. --- For production environments or workloads with high-volume video processing and heavy AI agent usage, running background tasks on a single server can cause bottlenecks. Shumai resolves this by decoupling the web dashboard from background processing through dedicated worker nodes orchestrated by **DBOS Transact**. All background tasks (media transcoding, AI embeddings, agent tasks) are dispatched to durable task queues (`transcode_queue` and `agent_queue`) hosted directly in PostgreSQL. No separate external cluster (like Temporal) is required. --- ## How it Works In a distributed deployment, the core Shumai web application handles user requests, database updates, and file uploads, while delegating durable background workflows to specialized worker instances via PostgreSQL task queues. ```mermaid graph TD UserUpload[User Uploads Video / Triggers Agent] --> WebApp[Shumai Web UI / API] WebApp -->|Enqueue Task| DB[(PostgreSQL / DBOS Queues)] DB -->|Queue: transcode_queue| TranscodeWorker[Transcode Worker Machine] DB -->|Queue: agent_queue| AgentWorker[Agent Worker Machine] ``` - **Transcoding Workers**: Can be run on GPU-accelerated or high-CPU compute instances. - **AI Agent Workers**: Can be run on isolated, sandboxed hosts to safely execute automation scripts. > [!IMPORTANT] > **Storage Requirement**: Because the core application and the background workers run on separate machines in a distributed setup, **you must configure an S3-compatible storage backend** (such as AWS S3, Cloudflare R2, or MinIO). Local storage (`STORAGE_BACKEND=local`) is not supported in this mode because the workers and the web application do not share a common filesystem. --- ## Step 1: Deploy the Core Web Application On your primary host machine, run PostgreSQL and the main Shumai web application. To instruct the main web application to handle only API and UI traffic without running background worker queues locally, set `SHUMAI_WORKERS=none`: ```env DATABASE_URL=postgresql://shumai:shumai_password@localhost:5432/shumai_db?schema=public STORAGE_BACKEND=s3 AWS_ENDPOINT_URL_S3=https://your-s3-endpoint.com S3_BUCKET=shumai S3_REGION=us-east-1 S3_ACCESS_KEY_ID=your-access-key-id S3_SECRET_ACCESS_KEY=your-secret-access-key SHUMAI_WORKERS=none BETTER_AUTH_SECRET=your-random-32-character-secret-key ``` When started with `SHUMAI_WORKERS=none` (or `--workers=none`), the web server starts immediately without polling or consuming tasks from the transcode or agent queues. --- ## Step 2: Configure & Run Dedicated Workers On your specialized worker machines, configure each worker with a `.env` file referencing the shared PostgreSQL database and S3 storage: ```env DATABASE_URL=postgresql://shumai:shumai_password@:5432/shumai_db?schema=public STORAGE_BACKEND=s3 AWS_ENDPOINT_URL_S3=https://your-s3-endpoint.com S3_BUCKET=shumai S3_REGION=us-east-1 S3_ACCESS_KEY_ID=your-access-key-id S3_SECRET_ACCESS_KEY=your-secret-access-key CONCURRENCY_TRANSCODE=1 CONCURRENCY_AGENT=5 ``` ### Media Transcoding Worker Run this on your media processing nodes (requires system installations of `ffmpeg` and `ffprobe`): ```bash npm install -g @shumai-one/shumai-transcode shumai-transcode ``` Alternatively, you can run the transcode worker using the dedicated lightweight Docker image: ```bash docker run -d \ --name shumai-transcode-worker \ --env-file .env \ shumaione/shumai-transcode:latest ``` (You can also use the all-in-one image: `docker run -d --name shumai-transcode-worker --env-file .env shumaione/shumai:latest shumai-transcode`.) ### AI Agent Worker Run this on your sandboxed agent execution nodes (requires sandbox dependencies like `bubblewrap` or `sandbox-exec`): ```bash npm install -g @shumai-one/shumai-agent shumai-agent ``` Alternatively, you can run the agent worker using the official Docker image: ```bash docker run -d \ --name shumai-agent-worker \ --env-file .env \ --cap-add=SYS_ADMIN \ shumaione/shumai:latest shumai-agent ``` > [!IMPORTANT] > **For Bun Users:** Since the worker package binaries contain a `#!/usr/bin/env node` shebang, running `shumai-transcode` or `shumai-agent` directly will run under Node.js. If you wish to run them using Bun, prefix the commands with `bun run --bun` (e.g., `bun run --bun shumai-transcode` or `bun run --bun shumai-agent`). --- ## Worker Concurrency & Multi-Instance Deployments ### Concurrency Tuning Workers pull tasks from PostgreSQL queues based on configurable concurrency limits: - `CONCURRENCY_TRANSCODE`: Maximum concurrent video transcoding jobs per worker (defaults to `1`). - `CONCURRENCY_AGENT`: Maximum concurrent agent and embedding workflows per worker (defaults to `5`, recommend setting to `1` when paired with a local `embedding-server`). ### Replica Identity When running multiple replicas of the transcode or agent worker across multiple machines or pods: - `DbosExecutor` automatically derives a unique per-instance ID from the machine hostname and process ID to ensure startup recovery only re-enqueues tasks belonging to that specific instance. - **Custom Worker ID**: Set `SHUMAI_WORKERS_ID` (or `SHUMAI_WORKER_ID`) if you wish to assign an explicit identifier. - **Kubernetes**: In Kubernetes Pods, set `POD_NAME` (via Downward API `fieldRef: { fieldPath: metadata.name }`) to provide stable, unique per-pod identity across pod restarts. --- ## Scalable Deployment on Kubernetes & AWS EKS For production cloud deployments requiring high-throughput media processing, Shumai can be deployed on Kubernetes or **AWS Elastic Kubernetes Service (AWS EKS)** with automated scale-to-zero queue-driven scaling via **KEDA (Kubernetes Event-driven Autoscaling)**. ```mermaid graph TD User([User / Web Browser]) -->|Port 80/3000| Server[Pod: shumai-server
SHUMAI_WORKERS=agent] Server -->|Enqueue to transcode_queue| DB[(PostgreSQL: pgvector18
dbos.workflow_status)] Server -->|Direct Uploads / Stream Proxies| Storage[(S3 / Garage Storage)] KEDA[KEDA ScaledJob
PostgreSQL Trigger
min: 0, max: 10] -.->|Monitors Queue| DB KEDA -->|Spawns Jobs for queue| Job1[Job: shumai-transcode-1
Auto-drain worker] KEDA -->|Spawns Jobs for queue| JobN[Job: shumai-transcode-N
Auto-drain worker] Job1 -->|Fetch Source / Store Proxies| Storage JobN -->|Fetch Source / Store Proxies| Storage Job1 -->|Mark Complete & Exit 0| DB JobN -->|Mark Complete & Exit 0| DB CA[Cluster Autoscaler / Karpenter] -.->|Scale-to-Zero On-Demand Nodes
0 to 10 instances| JobN ``` ### Key Architectural Highlights - **Decoupled Stateless Workers**: The main web and API server, database, and storage run on reliable On-Demand compute (`c9g.large`, `SHUMAI_WORKERS=agent`), delegating media transcoding to background workers orchestrated via DBOS task queues in PostgreSQL. - **Dedicated Graviton5 On-Demand Compute (`c9g.xlarge`)**: Media encoding runs on 4 vCPU / 8 GiB RAM AWS Graviton5 On-Demand instances (~$0.174/hr), utilizing ARM64 vector optimizations and high-speed DDR5 memory without involuntary Spot interruptions. - **Queue-Driven Scale-to-Zero via KEDA**: KEDA monitors `dbos.workflow_status` for enqueued and pending transcode tasks and dynamically provisions Kubernetes `ScaledJob` resources (scaling 0 to 10). When the queue is empty, transcode nodes scale down to zero (`minSize: 0`), incurring **$0/hr** in worker costs. - **Idle-Timeout Auto-Drain**: Worker pods continuously process queued tasks with concurrency `1`. When the queue is fully drained and a pod has been idle for 10 seconds (`SHUMAI_WORKER_IDLE_TIMEOUT_MS=10000`), it exits `0` cleanly so the node can scale down, processing batch uploads without cold-start pod churn. - **Automated Stale Task Recovery**: The core web server runs an integrated recovery sweep that detects stale or orphaned tasks (e.g. from node crash) and resumes them automatically in DBOS, with a 3-attempt cap to safely handle poison-pill media. ### Complete Deployment Example & Source Code For the full, production-ready manifests, declarative `eksctl` cluster configuration, and step-by-step walkthrough, see the **[EKS Deployment Example](https://github.com/shumaiOne/shumai/tree/main/eks)** in the Shumai repository: 👉 **[View the EKS Deployment Guide & Manifests on GitHub](https://github.com/shumaiOne/shumai/tree/main/eks)** The repository includes: - [`eks/eks-cluster.yaml`](https://github.com/shumaiOne/shumai/blob/main/eks/eks-cluster.yaml): Complete `eksctl` ClusterConfig with compute tiering. - [`eks/kustomization.yaml`](https://github.com/shumaiOne/shumai/blob/main/eks/kustomization.yaml): Single-command Kustomize deployment for all services. - Detailed instructions for setting up public endpoints, automated Cluster Autoscaler, and zero-downtime graceful shutdown.