Files
Yiling-J 176b7f0eab feat(workflow): migrate workflow engine from temporal to dbos (#481)
* feat(workflow): migrate workflow engine from temporal to dbos

Replace Temporal and LocalExecutor with DBOS Transact (@dbos-inc/dbos-sdk) as the unified workflow engine. Background tasks, media transcoding, and AI agent workflows now execute with durable steps and task queues directly against PostgreSQL, eliminating the need for an external Temporal cluster and separate polling executors.

* fix(workflow): submit tasks via DBOSClient to allow enqueueing from steps

Enqueuing workflows using DBOS.enqueueWorkflowWithOptions threw a
DBOSInvalidWorkflowTransitionError when triggered from within a running
step or activity (e.g. creating embedding or autofill tasks during
video transcoding).

Use DBOSClient to submit and cancel workflow tasks directly against the
system database, detaching execution from the ambient workflow/step
context.

* feat(web): support SHUMAI_WORKERS environment variable

* fix(workflow): ensure CLI tasks are enqueued to DBOS and run DBOS migrations in global setup

- Initialize workflows and wait for pending DBOS submissions in shumai regenerate-embeddings CLI
- Remove no-op submit bypass and enqueue tasks via DBOSClient in producer and CLI contexts
- Add DBOS migration runner in test global setup to avoid concurrent migration deadlocks across test forks
- Update changelog and add regression tests

* fix(workflow): ensure unique per-instance executor identity for worker replicas

- Derive unique per-instance DBOS executor IDs across worker replicas

- Support SHUMAI_WORKER_ID, DBOS__VMID, and POD_NAME (Kubernetes Downward API)

- Disambiguate multi-replica bare metal/container deployments with host, PID, and instance suffix

- Prevent startup recovery from re-enqueueing workflows actively executed by other live workers

- Add multi-instance startup recovery tests and update deployment documentation

* fix(workflow): only register listened queues and update concurrency on change

- Only register queues that the executor instance actively listens to

- Use always_update conflict resolution so environment variable changes take effect on restart

- Prevent dedicated workers from overwriting unmanaged queue concurrency configurations

- Add tests verifying concurrency updates and queue isolation across worker roles

* refactor(workflow): simplify activities to use defineStep and direct calls (#482)

* refactor(workflow): simplify activities to use defineStep and direct calls

- Wrap activities using defineStep from @shumai/workflow-core
- Remove obsolete getActivities, registerActivities, dynamic activity proxies, and workerQueue arguments
- Remove executor.ts and registry.ts, consolidating workflow registration into workflow.ts
- Call activities directly across transcode and agent workflows
- Update all corresponding workflow tests and developer guide documentation

* fix(workflow): properly detect DBOSWorkflowCancelledError in runOnCancel fallback

- DBOSWorkflowCancelledError in DBOS SDK does not set name, inheriting 'Error'
- Implement isDbosWorkflowCancelledError checking instanceof, constructor.name, and dbosErrorCode
- Update unit tests to verify real DBOS SDK cancellation error instances

* docs(workflow): update documentation and readme for dbos migration

- Rename docs/configuration/temporal.mdx to docs/configuration/distributed.mdx
- Document DBOS-based distributed worker architecture and task queues
- Remove obsolete docker-compose/temporal/
- Update environment variables documentation in docs/configuration/env-variables.mdx
- Update references in README.md, README_CN.md, docs/docs.json, and related guides

* feat(deploy): implement KEDA ScaledJob and scale-to-zero On-Demand transcoding (#483)

* feat(deploy): add shumai-transcode docker image and k8s autoscaling example with garage s3

* docs(k8s): document port-forwarding and in-cluster s3 domain mapping

* feat(storage): support public s3 endpoint for browser uploads and update k8s example

* fix(k8s): configure s3 bucket cors rules in garage initialization

* fix(k8s): increase transcode worker memory limit to 4Gi for 4K video encoding

* feat(deploy): add AWS EKS deployment with Graviton3 Spot nodegroups and autoscaling

* docs(deploy): redesign EKS documentation, rename k8s to eks, and add scalable deployment feature

* docs: rephrase DBOS feature bullet to Durable Workflows via DBOS

* docs(eks): upgrade default instances to Graviton5 c9g.large and c9g.xlarge

* feat(eks): use On-Demand for general nodegroup and add spot capacity diversification

* docs(eks): fix Architecture diagram to locate all transcode worker pods in Transcode Compute Pool

* feat(deploy): implement KEDA ScaledJob and scale-to-zero On-Demand transcoding

* feat(transcode): support idle-timeout auto-drain for KEDA scaled jobs

* fix(deploy): fix EKS deployment manifests and documentation based on cluster verification
2026-10-19 15:35:08 +08:00

246 lines
16 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Shumai on AWS EKS with KEDA & Scale-to-Zero On-Demand Workers
This directory contains a complete, production-ready Kubernetes deployment example for running Shumai on **Amazon Elastic Kubernetes Service (Amazon EKS)**, using **KEDA (Kubernetes Event-driven Autoscaling)** and **Scale-to-Zero On-Demand AWS Graviton5 instances** for compute-intensive media processing.
The example should also be straightforward to adapt to any standard Kubernetes cluster.
To demonstrate a fully self-hosted setup, it uses Garage as an S3-compatible object storage solution. However, if you're deploying on AWS, consider using **Amazon S3** instead of running your own S3-compatible storage server, as it is managed and highly scalable.
---
## Architecture Overview
```text
[ User / Internet ]
│
┌────────────────────┴────────────────────┐
│ (Port 80/3000) │ (Port 3900)
▼ ▼
┌───────────────────────┐ ┌───────────────────────┐
│ shumai-server (ELB) │ │ garage-public (ELB) │
└───────────┬───────────┘ └───────────┬───────────┘
│ │ Direct Browser Uploads
│ ▼ (Presigned URLs)
┌───────────────────┼─────────────────────────────────────────────────────────────────┐
│ General Workload Node (c9g.large On-Demand, ~$0.087/hr) │
│ │ │
│ ▼ │
│ ┌───────────────────────┐ Enqueue DBOS Tasks ┌───────────────────┐ │
│ │ shumai-server ├────────────────────────►│ shumai-postgres │ │
│ │(SHUMAI_WORKERS=agent) │ (transcode_queue) │ (pgvector:pg18) │ │
│ │ [Stale Recovery Loop] │ │ │ │
│ └───────────┬───────────┘ └─────────┬─────────┘ │
│ │ Internal S3 Traffic │ │
│ ▼ │ │
│ ┌───────────────────────┐ │ │
│ │ garage-s3 │ │ │
│ │ (dxflrs/garage) │ │ │
│ └───────────▲───────────┘ │ │
└───────────────────┼───────────────────────────────────────────────┼─────────────────┘
│ Read Source / Write Proxies │
│ │ KEDA monitors
│ ▼ dbos.workflow_status
│ ┌───────────────────────────────────────┐
│ │ KEDA Controller: ScaledJob │
│ │ Spawns Jobs based on pending tasks │
│ └───────────────────┬───────────────────┘
│ │
│ ▼ Creates K8s Jobs
┌───────────────────┴─────────────────────────────────────────────────────────────────┐
│ Transcode Compute Pool (c9g.xlarge Graviton5 On-Demand, minSize: 0, maxSize: 5) │
│ │
│ ┌──────────────────────────┐ ┌──────────────────────────┐ ┌────────────────────┐ │
│ │ Job: transcode-job-1 │ │ Job: transcode-job-2 │ │Job: transcode-job-N│ │
│ │ (Auto-Drain Worker) │ │ (Auto-Drain Worker) │ │ (Auto-Drain Worker)│ │
│ └────────────┬─────────────┘ └────────────┬─────────────┘ └──────────┬─────────┘ │
│ ▼ ▼ ▼ │
│ ┌──────────────────────────┐ ┌──────────────────────────┐ ┌────────────────────┐ │
│ │ Node 1: c9g.xlarge │ │ Node 2: c9g.xlarge │ │ Node N: c9g.xlarge │ │
│ │ (On-Demand) │ │ (On-Demand) │ │ (On-Demand) │ │
│ └──────────────────────────┘ └──────────────────────────┘ └────────────────────┘ │
│ │
│ Scale-to-Zero: When queue is empty, 0 Jobs run and 0 EC2 instances are billed! │
└─────────────────────────────────────────────────────────────────────────────────────┘
```
---
## Core Design Principles
### 1. Scale-to-Zero On-Demand Instances (`minSize: 0`)
Media transcoding with FFmpeg is bursty: hours may pass with zero transcoding activity, followed by sudden batches of 4K video uploads.
- **Zero Idle Cost**: The transcode node group [`eks/eks-cluster.yaml`](./eks-cluster.yaml) is configured with `minSize: 0` and `desiredCapacity: 0`. When no media transcoding tasks are active, **zero transcode EC2 instances exist**, costing **$0/hr**.
- **No Spot Interruption Risk**: Transcoding jobs run on AWS Graviton5 **On-Demand** instances (`c9g.xlarge`, ~$0.174/hr). This eliminates the risk of Spot instances being reclaimed mid-encode by AWS, preventing wasted CPU cycles and avoiding unexpected transcoding delays.
### 2. KEDA `ScaledJob` vs. `Deployment` / `StatefulSet`
Standard Kubernetes Horizontal Pod Autoscaler (HPA) scales `Deployment` replicas based on metrics like CPU utilization. However, HPAs on Deployments suffer from severe limitations during video processing:
1. When a 1-minute video finishes on Pod A while a 30-minute video is still encoding on Pod B, CPU drops and HPA reduces replica count, which can arbitrarily terminate Pod B mid-encode.
2. StatefulSets scale down in strict ordinal order (`pod-4`, `pod-3`, etc.), which would terminate `pod-4` even if `pod-4` is the only pod currently transcoding!
**KEDA `ScaledJob` solves this natively**:
- KEDA queries PostgreSQL directly:
```sql
SELECT COUNT(*) FROM dbos.workflow_status WHERE queue_name = 'transcode_queue' AND status IN ('ENQUEUED', 'PENDING');
```
- KEDA spawns independent Kubernetes `Job` pods based on enqueued workload.
- Each Job runs the transcode worker with an idle timeout (`SHUMAI_WORKER_IDLE_TIMEOUT_MS=10000`).
- When the job drains its tasks and becomes idle, the pod exits `0` and Kubernetes marks the Job completed.
- Completing or terminating one job **never impacts any other job**!
### 3. Idle-Timeout Auto-Drain (10s Inactivity Shutdown)
In [`eks/shumai-transcode.yaml`](./shumai-transcode.yaml), the worker container runs with `SHUMAI_WORKER_IDLE_TIMEOUT_MS: "10000"`:
- **Concurrency Isolation**: Transcode concurrency is set to `1` per pod (`CONCURRENCY_TRANSCODE=1`). Each pod has exclusive access to the node's 4 vCPUs for FFmpeg.
- **Batch Processing Without Cold Starts**: Unlike single-task execution, a running worker continuously pulls tasks from `transcode_queue`. When a user uploads 100 images or multiple video files, a warm worker processes them back-to-back in seconds without restarting the container or control plane churn.
- **Automatic Scale-to-Zero**: When `activeWorkflows === 0` and no new task arrives within 10 consecutive seconds, the worker initiates graceful DBOS shutdown and exits with code `0`.
- The completed Job pod terminates, freeing the node so Cluster Autoscaler can terminate the EC2 instance down to zero.
---
## Step-by-Step Deployment Guide
### Prerequisites
- [AWS CLI](https://aws.amazon.com/cli/) installed and configured (`aws configure`).
- [`eksctl`](https://eksctl.io/) installed.
- [`kubectl`](https://kubernetes.io/docs/tasks/tools/) installed.
- [`helm`](https://helm.sh/) installed.
### Step 1: Provision the EKS Cluster
Create the cluster using the declarative [`eks/eks-cluster.yaml`](./eks-cluster.yaml) configuration:
```bash
eksctl create cluster -f eks/eks-cluster.yaml
```
This provisions:
- An EKS 1.31 control plane in `us-east-1` across multiple AZs. *(Note: AWS Graviton5 `c9g` instances are available in `us-east-1`. If deploying to a region where `c9g` is not yet offered, substitute `c7g.large` / `c7g.xlarge` or `c8g` in `eks/eks-cluster.yaml`.)*
- `shumai-general`: 1x `c9g.large` On-Demand instance (labeled `workload: general`) for API, database, and storage.
- `shumai-transcode`: Scale-to-zero `c9g.xlarge` On-Demand instance pool (`minSize: 0`, `maxSize: 5`, `desiredCapacity: 0`, tainted `dedicated=transcode:NoSchedule`).
- AWS EBS CSI Driver with IAM OIDC roles.
### Step 2: Install KEDA
Install KEDA using Helm:
```bash
helm repo add kedacore https://kedacore.github.io/charts
helm repo update
helm install keda kedacore/keda --namespace keda --create-namespace
```
Wait for the KEDA operator to reach `Running` state:
```bash
kubectl wait --namespace keda --for=condition=ready pod -l app.kubernetes.io/name=keda-operator --timeout=120s
```
### Step 3: Deploy the Shumai Stack
Deploy all components with Kustomize:
```bash
kubectl apply -k eks/
```
This applies:
- [`eks/storageclass.yaml`](./storageclass.yaml): Configures `gp3` as the default StorageClass using `ebs.csi.aws.com` with `WaitForFirstConsumer` binding and dynamic EBS volume expansion.
- PostgreSQL (`pgvector:pg18`) with durable EBS storage.
- Garage S3 object store with automatic bucket and CORS initialization (`garage-init` job).
- Shumai API & Web server.
- KEDA `ScaledJob` for transcode worker auto-scaling.
Wait for the core database and storage services to initialize:
```bash
# Wait for PostgreSQL and Garage S3 pods to become Ready
kubectl wait --namespace shumai --for=condition=ready pod -l app=shumai-postgres --timeout=180s
kubectl wait --namespace shumai --for=condition=ready pod -l app=garage --timeout=180s
# Wait for the bucket initialization job to complete
kubectl wait --namespace shumai --for=condition=complete job/garage-init --timeout=180s
# Verify all pods in the shumai namespace
kubectl get pods -n shumai -o wide
```
*(Note: On first startup, `shumai-server` automatically runs database migrations against PostgreSQL; this may take 30–60 seconds before the server reaches `Running (1/1)`.)*
### Step 4: Configure Public Access & S3 Endpoint
AWS automatically provisions external Elastic Load Balancers for `shumai-server` and `garage-public`. Because AWS takes 30–60 seconds to allocate external hostnames, wait for the hostnames to become available:
```bash
# Wait for AWS to assign external hostnames to the LoadBalancers
echo "Waiting for LoadBalancer external hostnames..."
until [ -n "$(kubectl get svc garage-public -n shumai -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')" ]; do
sleep 5
done
GARAGE_ELB=$(kubectl get svc garage-public -n shumai -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
SERVER_ELB=$(kubectl get svc shumai-server -n shumai -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
echo "Garage S3 Hostname: http://${GARAGE_ELB}:3900"
echo "Shumai Web UI: http://${SERVER_ELB}"
```
Inject the public Garage S3 hostname into `shumai-server` so that browsers generate correct presigned upload URLs:
```bash
kubectl set env deployment/shumai-server -n shumai AWS_PUBLIC_ENDPOINT_URL_S3="http://${GARAGE_ELB}:3900"
```
*(Note: In production with custom domains, point your DNS CNAME records to the load balancers and configure `AWS_PUBLIC_ENDPOINT_URL_S3: https://s3.mycompany.com` in [`eks/config.yaml`](./config.yaml).)*
### Step 5: Deploy Kubernetes Cluster Autoscaler
Deploy Cluster Autoscaler to dynamically provision `c9g.xlarge` EC2 nodes when KEDA jobs are created:
```bash
curl -s https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml \
| sed 's/<YOUR CLUSTER NAME>/shumai-eks/g' \
| kubectl apply -f -
# Pin autoscaler to the general workload node
kubectl patch deployment cluster-autoscaler -n kube-system -p '{"spec":{"template":{"spec":{"nodeSelector":{"workload":"general"}}}}}'
```
### Step 6: Test Scale-to-Zero Queue-Driven Autoscaling
1. Open `http://${SERVER_ELB}` in your browser and register an account via **Sign Up**. *(The first registered account automatically becomes the instance administrator).*
2. Create a project and upload a batch of 5–10 video files simultaneously.
3. Observe KEDA detecting pending DBOS tasks and spawning individual Kubernetes Jobs:
```bash
kubectl get scaledjobs -n shumai
kubectl get jobs -n shumai -w
kubectl get pods -n shumai -l app=shumai-transcode -o wide
kubectl get nodes -L role,workload
```
4. Watch Cluster Autoscaler dynamically provision dedicated `c9g.xlarge` Graviton5 EC2 instances for the Jobs.
5. As each video finishes transcoding, its worker pod exits `0` and completes immediately.
6. Once all jobs are done, KEDA holds 0 jobs, and Cluster Autoscaler terminates all transcode EC2 instances, returning the transcode pool to **0 nodes ($0/hr)**!
### Step 7: Cluster Teardown & Cost Cleanup
When you are finished testing, delete all resources to prevent ongoing cloud charges:
```bash
# 1. Delete Kubernetes namespace first to release AWS ELB load balancers and EBS volumes
# (Important: Wait for this to complete so AWS SecurityGroups can be deleted without dependency conflicts)
kubectl delete namespace shumai
# 2. Uninstall KEDA
helm uninstall keda -n keda
kubectl delete namespace keda
# 3. Delete the EKS cluster and CloudFormation stacks
eksctl delete cluster -f eks/eks-cluster.yaml --wait
# 4. Confirm all AWS resources are released
aws eks list-clusters --region us-east-1
```