mirror of
https://github.com/shumaiOne/shumai.git
synced 2026-11-19 14:16:29 +00:00
246 lines
16 KiB
Markdown
246 lines
16 KiB
Markdown
# Shumai on AWS EKS with KEDA & Scale-to-Zero On-Demand Workers
|
||||
|
|
|
|||
|
|
This directory contains a complete, production-ready Kubernetes deployment example for running Shumai on **Amazon Elastic Kubernetes Service (Amazon EKS)**, using **KEDA (Kubernetes Event-driven Autoscaling)** and **Scale-to-Zero On-Demand AWS Graviton5 instances** for compute-intensive media processing.
|
|||
|
|
|
|||
|
|
The example should also be straightforward to adapt to any standard Kubernetes cluster.
|
|||
|
|
|
|||
|
|
To demonstrate a fully self-hosted setup, it uses Garage as an S3-compatible object storage solution. However, if you're deploying on AWS, consider using **Amazon S3** instead of running your own S3-compatible storage server, as it is managed and highly scalable.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Architecture Overview
|
|||
|
|
|
|||
|
|
```text
|
|||
|
|
[ User / Internet ]
|
|||
|
|
│
|
|||
|
|
┌────────────────────┴────────────────────┐
|
|||
|
|
│ (Port 80/3000) │ (Port 3900)
|
|||
|
|
▼ ▼
|
|||
|
|
┌───────────────────────┐ ┌───────────────────────┐
|
|||
|
|
│ shumai-server (ELB) │ │ garage-public (ELB) │
|
|||
|
|
└───────────┬───────────┘ └───────────┬───────────┘
|
|||
|
|
│ │ Direct Browser Uploads
|
|||
|
|
│ ▼ (Presigned URLs)
|
|||
|
|
┌───────────────────┼─────────────────────────────────────────────────────────────────┐
|
|||
|
|
│ General Workload Node (c9g.large On-Demand, ~$0.087/hr) │
|
|||
|
|
│ │ │
|
|||
|
|
│ ▼ │
|
|||
|
|
│ ┌───────────────────────┐ Enqueue DBOS Tasks ┌───────────────────┐ │
|
|||
|
|
│ │ shumai-server ├────────────────────────►│ shumai-postgres │ │
|
|||
|
|
│ │(SHUMAI_WORKERS=agent) │ (transcode_queue) │ (pgvector:pg18) │ │
|
|||
|
|
│ │ [Stale Recovery Loop] │ │ │ │
|
|||
|
|
│ └───────────┬───────────┘ └─────────┬─────────┘ │
|
|||
|
|
│ │ Internal S3 Traffic │ │
|
|||
|
|
│ ▼ │ │
|
|||
|
|
│ ┌───────────────────────┐ │ │
|
|||
|
|
│ │ garage-s3 │ │ │
|
|||
|
|
│ │ (dxflrs/garage) │ │ │
|
|||
|
|
│ └───────────▲───────────┘ │ │
|
|||
|
|
└───────────────────┼───────────────────────────────────────────────┼─────────────────┘
|
|||
|
|
│ Read Source / Write Proxies │
|
|||
|
|
│ │ KEDA monitors
|
|||
|
|
│ ▼ dbos.workflow_status
|
|||
|
|
│ ┌───────────────────────────────────────┐
|
|||
|
|
│ │ KEDA Controller: ScaledJob │
|
|||
|
|
│ │ Spawns Jobs based on pending tasks │
|
|||
|
|
│ └───────────────────┬───────────────────┘
|
|||
|
|
│ │
|
|||
|
|
│ ▼ Creates K8s Jobs
|
|||
|
|
┌───────────────────┴─────────────────────────────────────────────────────────────────┐
|
|||
|
|
│ Transcode Compute Pool (c9g.xlarge Graviton5 On-Demand, minSize: 0, maxSize: 5) │
|
|||
|
|
│ │
|
|||
|
|
│ ┌──────────────────────────┐ ┌──────────────────────────┐ ┌────────────────────┐ │
|
|||
|
|
│ │ Job: transcode-job-1 │ │ Job: transcode-job-2 │ │Job: transcode-job-N│ │
|
|||
|
|
│ │ (Auto-Drain Worker) │ │ (Auto-Drain Worker) │ │ (Auto-Drain Worker)│ │
|
|||
|
|
│ └────────────┬─────────────┘ └────────────┬─────────────┘ └──────────┬─────────┘ │
|
|||
|
|
│ ▼ ▼ ▼ │
|
|||
|
|
│ ┌──────────────────────────┐ ┌──────────────────────────┐ ┌────────────────────┐ │
|
|||
|
|
│ │ Node 1: c9g.xlarge │ │ Node 2: c9g.xlarge │ │ Node N: c9g.xlarge │ │
|
|||
|
|
│ │ (On-Demand) │ │ (On-Demand) │ │ (On-Demand) │ │
|
|||
|
|
│ └──────────────────────────┘ └──────────────────────────┘ └────────────────────┘ │
|
|||
|
|
│ │
|
|||
|
|
│ Scale-to-Zero: When queue is empty, 0 Jobs run and 0 EC2 instances are billed! │
|
|||
|
|
└─────────────────────────────────────────────────────────────────────────────────────┘
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Core Design Principles
|
|||
|
|
|
|||
|
|
### 1. Scale-to-Zero On-Demand Instances (`minSize: 0`)
|
|||
|
|
|
|||
|
|
Media transcoding with FFmpeg is bursty: hours may pass with zero transcoding activity, followed by sudden batches of 4K video uploads.
|
|||
|
|
|
|||
|
|
- **Zero Idle Cost**: The transcode node group [`eks/eks-cluster.yaml`](./eks-cluster.yaml) is configured with `minSize: 0` and `desiredCapacity: 0`. When no media transcoding tasks are active, **zero transcode EC2 instances exist**, costing **$0/hr**.
|
|||
|
|
- **No Spot Interruption Risk**: Transcoding jobs run on AWS Graviton5 **On-Demand** instances (`c9g.xlarge`, ~$0.174/hr). This eliminates the risk of Spot instances being reclaimed mid-encode by AWS, preventing wasted CPU cycles and avoiding unexpected transcoding delays.
|
|||
|
|
|
|||
|
|
### 2. KEDA `ScaledJob` vs. `Deployment` / `StatefulSet`
|
|||
|
|
|
|||
|
|
Standard Kubernetes Horizontal Pod Autoscaler (HPA) scales `Deployment` replicas based on metrics like CPU utilization. However, HPAs on Deployments suffer from severe limitations during video processing:
|
|||
|
|
1. When a 1-minute video finishes on Pod A while a 30-minute video is still encoding on Pod B, CPU drops and HPA reduces replica count, which can arbitrarily terminate Pod B mid-encode.
|
|||
|
|
2. StatefulSets scale down in strict ordinal order (`pod-4`, `pod-3`, etc.), which would terminate `pod-4` even if `pod-4` is the only pod currently transcoding!
|
|||
|
|
|
|||
|
|
**KEDA `ScaledJob` solves this natively**:
|
|||
|
|
- KEDA queries PostgreSQL directly:
|
|||
|
|
```sql
|
|||
|
|
SELECT COUNT(*) FROM dbos.workflow_status WHERE queue_name = 'transcode_queue' AND status IN ('ENQUEUED', 'PENDING');
|
|||
|
|
```
|
|||
|
|
- KEDA spawns independent Kubernetes `Job` pods based on enqueued workload.
|
|||
|
|
- Each Job runs the transcode worker with an idle timeout (`SHUMAI_WORKER_IDLE_TIMEOUT_MS=10000`).
|
|||
|
|
- When the job drains its tasks and becomes idle, the pod exits `0` and Kubernetes marks the Job completed.
|
|||
|
|
- Completing or terminating one job **never impacts any other job**!
|
|||
|
|
|
|||
|
|
### 3. Idle-Timeout Auto-Drain (10s Inactivity Shutdown)
|
|||
|
|
|
|||
|
|
In [`eks/shumai-transcode.yaml`](./shumai-transcode.yaml), the worker container runs with `SHUMAI_WORKER_IDLE_TIMEOUT_MS: "10000"`:
|
|||
|
|
- **Concurrency Isolation**: Transcode concurrency is set to `1` per pod (`CONCURRENCY_TRANSCODE=1`). Each pod has exclusive access to the node's 4 vCPUs for FFmpeg.
|
|||
|
|
- **Batch Processing Without Cold Starts**: Unlike single-task execution, a running worker continuously pulls tasks from `transcode_queue`. When a user uploads 100 images or multiple video files, a warm worker processes them back-to-back in seconds without restarting the container or control plane churn.
|
|||
|
|
- **Automatic Scale-to-Zero**: When `activeWorkflows === 0` and no new task arrives within 10 consecutive seconds, the worker initiates graceful DBOS shutdown and exits with code `0`.
|
|||
|
|
- The completed Job pod terminates, freeing the node so Cluster Autoscaler can terminate the EC2 instance down to zero.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Step-by-Step Deployment Guide
|
|||
|
|
|
|||
|
|
### Prerequisites
|
|||
|
|
|
|||
|
|
- [AWS CLI](https://aws.amazon.com/cli/) installed and configured (`aws configure`).
|
|||
|
|
- [`eksctl`](https://eksctl.io/) installed.
|
|||
|
|
- [`kubectl`](https://kubernetes.io/docs/tasks/tools/) installed.
|
|||
|
|
- [`helm`](https://helm.sh/) installed.
|
|||
|
|
|
|||
|
|
### Step 1: Provision the EKS Cluster
|
|||
|
|
|
|||
|
|
Create the cluster using the declarative [`eks/eks-cluster.yaml`](./eks-cluster.yaml) configuration:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
eksctl create cluster -f eks/eks-cluster.yaml
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
This provisions:
|
|||
|
|
- An EKS 1.31 control plane in `us-east-1` across multiple AZs. *(Note: AWS Graviton5 `c9g` instances are available in `us-east-1`. If deploying to a region where `c9g` is not yet offered, substitute `c7g.large` / `c7g.xlarge` or `c8g` in `eks/eks-cluster.yaml`.)*
|
|||
|
|
- `shumai-general`: 1x `c9g.large` On-Demand instance (labeled `workload: general`) for API, database, and storage.
|
|||
|
|
- `shumai-transcode`: Scale-to-zero `c9g.xlarge` On-Demand instance pool (`minSize: 0`, `maxSize: 5`, `desiredCapacity: 0`, tainted `dedicated=transcode:NoSchedule`).
|
|||
|
|
- AWS EBS CSI Driver with IAM OIDC roles.
|
|||
|
|
|
|||
|
|
### Step 2: Install KEDA
|
|||
|
|
|
|||
|
|
Install KEDA using Helm:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
helm repo add kedacore https://kedacore.github.io/charts
|
|||
|
|
helm repo update
|
|||
|
|
helm install keda kedacore/keda --namespace keda --create-namespace
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Wait for the KEDA operator to reach `Running` state:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
kubectl wait --namespace keda --for=condition=ready pod -l app.kubernetes.io/name=keda-operator --timeout=120s
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### Step 3: Deploy the Shumai Stack
|
|||
|
|
|
|||
|
|
Deploy all components with Kustomize:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
kubectl apply -k eks/
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
This applies:
|
|||
|
|
- [`eks/storageclass.yaml`](./storageclass.yaml): Configures `gp3` as the default StorageClass using `ebs.csi.aws.com` with `WaitForFirstConsumer` binding and dynamic EBS volume expansion.
|
|||
|
|
- PostgreSQL (`pgvector:pg18`) with durable EBS storage.
|
|||
|
|
- Garage S3 object store with automatic bucket and CORS initialization (`garage-init` job).
|
|||
|
|
- Shumai API & Web server.
|
|||
|
|
- KEDA `ScaledJob` for transcode worker auto-scaling.
|
|||
|
|
|
|||
|
|
Wait for the core database and storage services to initialize:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
# Wait for PostgreSQL and Garage S3 pods to become Ready
|
|||
|
|
kubectl wait --namespace shumai --for=condition=ready pod -l app=shumai-postgres --timeout=180s
|
|||
|
|
kubectl wait --namespace shumai --for=condition=ready pod -l app=garage --timeout=180s
|
|||
|
|
|
|||
|
|
# Wait for the bucket initialization job to complete
|
|||
|
|
kubectl wait --namespace shumai --for=condition=complete job/garage-init --timeout=180s
|
|||
|
|
|
|||
|
|
# Verify all pods in the shumai namespace
|
|||
|
|
kubectl get pods -n shumai -o wide
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
*(Note: On first startup, `shumai-server` automatically runs database migrations against PostgreSQL; this may take 30–60 seconds before the server reaches `Running (1/1)`.)*
|
|||
|
|
|
|||
|
|
### Step 4: Configure Public Access & S3 Endpoint
|
|||
|
|
|
|||
|
|
AWS automatically provisions external Elastic Load Balancers for `shumai-server` and `garage-public`. Because AWS takes 30–60 seconds to allocate external hostnames, wait for the hostnames to become available:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
# Wait for AWS to assign external hostnames to the LoadBalancers
|
|||
|
|
echo "Waiting for LoadBalancer external hostnames..."
|
|||
|
|
until [ -n "$(kubectl get svc garage-public -n shumai -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')" ]; do
|
|||
|
|
sleep 5
|
|||
|
|
done
|
|||
|
|
|
|||
|
|
GARAGE_ELB=$(kubectl get svc garage-public -n shumai -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
|
|||
|
|
SERVER_ELB=$(kubectl get svc shumai-server -n shumai -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
|
|||
|
|
|
|||
|
|
echo "Garage S3 Hostname: http://${GARAGE_ELB}:3900"
|
|||
|
|
echo "Shumai Web UI: http://${SERVER_ELB}"
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Inject the public Garage S3 hostname into `shumai-server` so that browsers generate correct presigned upload URLs:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
kubectl set env deployment/shumai-server -n shumai AWS_PUBLIC_ENDPOINT_URL_S3="http://${GARAGE_ELB}:3900"
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
*(Note: In production with custom domains, point your DNS CNAME records to the load balancers and configure `AWS_PUBLIC_ENDPOINT_URL_S3: https://s3.mycompany.com` in [`eks/config.yaml`](./config.yaml).)*
|
|||
|
|
|
|||
|
|
### Step 5: Deploy Kubernetes Cluster Autoscaler
|
|||
|
|
|
|||
|
|
Deploy Cluster Autoscaler to dynamically provision `c9g.xlarge` EC2 nodes when KEDA jobs are created:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
curl -s https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml \
|
|||
|
|
| sed 's/<YOUR CLUSTER NAME>/shumai-eks/g' \
|
|||
|
|
| kubectl apply -f -
|
|||
|
|
|
|||
|
|
# Pin autoscaler to the general workload node
|
|||
|
|
kubectl patch deployment cluster-autoscaler -n kube-system -p '{"spec":{"template":{"spec":{"nodeSelector":{"workload":"general"}}}}}'
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### Step 6: Test Scale-to-Zero Queue-Driven Autoscaling
|
|||
|
|
|
|||
|
|
1. Open `http://${SERVER_ELB}` in your browser and register an account via **Sign Up**. *(The first registered account automatically becomes the instance administrator).*
|
|||
|
|
2. Create a project and upload a batch of 5–10 video files simultaneously.
|
|||
|
|
3. Observe KEDA detecting pending DBOS tasks and spawning individual Kubernetes Jobs:
|
|||
|
|
```bash
|
|||
|
|
kubectl get scaledjobs -n shumai
|
|||
|
|
kubectl get jobs -n shumai -w
|
|||
|
|
kubectl get pods -n shumai -l app=shumai-transcode -o wide
|
|||
|
|
kubectl get nodes -L role,workload
|
|||
|
|
```
|
|||
|
|
4. Watch Cluster Autoscaler dynamically provision dedicated `c9g.xlarge` Graviton5 EC2 instances for the Jobs.
|
|||
|
|
5. As each video finishes transcoding, its worker pod exits `0` and completes immediately.
|
|||
|
|
6. Once all jobs are done, KEDA holds 0 jobs, and Cluster Autoscaler terminates all transcode EC2 instances, returning the transcode pool to **0 nodes ($0/hr)**!
|
|||
|
|
|
|||
|
|
### Step 7: Cluster Teardown & Cost Cleanup
|
|||
|
|
|
|||
|
|
When you are finished testing, delete all resources to prevent ongoing cloud charges:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
# 1. Delete Kubernetes namespace first to release AWS ELB load balancers and EBS volumes
|
|||
|
|
# (Important: Wait for this to complete so AWS SecurityGroups can be deleted without dependency conflicts)
|
|||
|
|
kubectl delete namespace shumai
|
|||
|
|
|
|||
|
|
# 2. Uninstall KEDA
|
|||
|
|
helm uninstall keda -n keda
|
|||
|
|
kubectl delete namespace keda
|
|||
|
|
|
|||
|
|
# 3. Delete the EKS cluster and CloudFormation stacks
|
|||
|
|
eksctl delete cluster -f eks/eks-cluster.yaml --wait
|
|||
|
|
|
|||
|
|
# 4. Confirm all AWS resources are released
|
|||
|
|
aws eks list-clusters --region us-east-1
|
|||
|
|
```
|