# Shumai on AWS EKS with KEDA & Scale-to-Zero On-Demand Workers This directory contains a complete, production-ready Kubernetes deployment example for running Shumai on **Amazon Elastic Kubernetes Service (Amazon EKS)**, using **KEDA (Kubernetes Event-driven Autoscaling)** and **Scale-to-Zero On-Demand AWS Graviton5 instances** for compute-intensive media processing. The example should also be straightforward to adapt to any standard Kubernetes cluster. To demonstrate a fully self-hosted setup, it uses Garage as an S3-compatible object storage solution. However, if you're deploying on AWS, consider using **Amazon S3** instead of running your own S3-compatible storage server, as it is managed and highly scalable. --- ## Architecture Overview ```text [ User / Internet ] │ ┌────────────────────┴────────────────────┐ │ (Port 80/3000) │ (Port 3900) ▼ ▼ ┌───────────────────────┐ ┌───────────────────────┐ │ shumai-server (ELB) │ │ garage-public (ELB) │ └───────────┬───────────┘ └───────────┬───────────┘ │ │ Direct Browser Uploads │ ▼ (Presigned URLs) ┌───────────────────┼─────────────────────────────────────────────────────────────────┐ │ General Workload Node (c9g.large On-Demand, ~$0.087/hr) │ │ │ │ │ ▼ │ │ ┌───────────────────────┐ Enqueue DBOS Tasks ┌───────────────────┐ │ │ │ shumai-server ├────────────────────────►│ shumai-postgres │ │ │ │(SHUMAI_WORKERS=agent) │ (transcode_queue) │ (pgvector:pg18) │ │ │ │ [Stale Recovery Loop] │ │ │ │ │ └───────────┬───────────┘ └─────────┬─────────┘ │ │ │ Internal S3 Traffic │ │ │ ▼ │ │ │ ┌───────────────────────┐ │ │ │ │ garage-s3 │ │ │ │ │ (dxflrs/garage) │ │ │ │ └───────────▲───────────┘ │ │ └───────────────────┼───────────────────────────────────────────────┼─────────────────┘ │ Read Source / Write Proxies │ │ │ KEDA monitors │ ▼ dbos.workflow_status │ ┌───────────────────────────────────────┐ │ │ KEDA Controller: ScaledJob │ │ │ Spawns Jobs based on pending tasks │ │ └───────────────────┬───────────────────┘ │ │ │ ▼ Creates K8s Jobs ┌───────────────────┴─────────────────────────────────────────────────────────────────┐ │ Transcode Compute Pool (c9g.xlarge Graviton5 On-Demand, minSize: 0, maxSize: 5) │ │ │ │ ┌──────────────────────────┐ ┌──────────────────────────┐ ┌────────────────────┐ │ │ │ Job: transcode-job-1 │ │ Job: transcode-job-2 │ │Job: transcode-job-N│ │ │ │ (Auto-Drain Worker) │ │ (Auto-Drain Worker) │ │ (Auto-Drain Worker)│ │ │ └────────────┬─────────────┘ └────────────┬─────────────┘ └──────────┬─────────┘ │ │ ▼ ▼ ▼ │ │ ┌──────────────────────────┐ ┌──────────────────────────┐ ┌────────────────────┐ │ │ │ Node 1: c9g.xlarge │ │ Node 2: c9g.xlarge │ │ Node N: c9g.xlarge │ │ │ │ (On-Demand) │ │ (On-Demand) │ │ (On-Demand) │ │ │ └──────────────────────────┘ └──────────────────────────┘ └────────────────────┘ │ │ │ │ Scale-to-Zero: When queue is empty, 0 Jobs run and 0 EC2 instances are billed! │ └─────────────────────────────────────────────────────────────────────────────────────┘ ``` --- ## Core Design Principles ### 1. Scale-to-Zero On-Demand Instances (`minSize: 0`) Media transcoding with FFmpeg is bursty: hours may pass with zero transcoding activity, followed by sudden batches of 4K video uploads. - **Zero Idle Cost**: The transcode node group [`eks/eks-cluster.yaml`](./eks-cluster.yaml) is configured with `minSize: 0` and `desiredCapacity: 0`. When no media transcoding tasks are active, **zero transcode EC2 instances exist**, costing **$0/hr**. - **No Spot Interruption Risk**: Transcoding jobs run on AWS Graviton5 **On-Demand** instances (`c9g.xlarge`, ~$0.174/hr). This eliminates the risk of Spot instances being reclaimed mid-encode by AWS, preventing wasted CPU cycles and avoiding unexpected transcoding delays. ### 2. KEDA `ScaledJob` vs. `Deployment` / `StatefulSet` Standard Kubernetes Horizontal Pod Autoscaler (HPA) scales `Deployment` replicas based on metrics like CPU utilization. However, HPAs on Deployments suffer from severe limitations during video processing: 1. When a 1-minute video finishes on Pod A while a 30-minute video is still encoding on Pod B, CPU drops and HPA reduces replica count, which can arbitrarily terminate Pod B mid-encode. 2. StatefulSets scale down in strict ordinal order (`pod-4`, `pod-3`, etc.), which would terminate `pod-4` even if `pod-4` is the only pod currently transcoding! **KEDA `ScaledJob` solves this natively**: - KEDA queries PostgreSQL directly: ```sql SELECT COUNT(*) FROM dbos.workflow_status WHERE queue_name = 'transcode_queue' AND status IN ('ENQUEUED', 'PENDING'); ``` - KEDA spawns independent Kubernetes `Job` pods based on enqueued workload. - Each Job runs the transcode worker with an idle timeout (`SHUMAI_WORKER_IDLE_TIMEOUT_MS=10000`). - When the job drains its tasks and becomes idle, the pod exits `0` and Kubernetes marks the Job completed. - Completing or terminating one job **never impacts any other job**! ### 3. Idle-Timeout Auto-Drain (10s Inactivity Shutdown) In [`eks/shumai-transcode.yaml`](./shumai-transcode.yaml), the worker container runs with `SHUMAI_WORKER_IDLE_TIMEOUT_MS: "10000"`: - **Concurrency Isolation**: Transcode concurrency is set to `1` per pod (`CONCURRENCY_TRANSCODE=1`). Each pod has exclusive access to the node's 4 vCPUs for FFmpeg. - **Batch Processing Without Cold Starts**: Unlike single-task execution, a running worker continuously pulls tasks from `transcode_queue`. When a user uploads 100 images or multiple video files, a warm worker processes them back-to-back in seconds without restarting the container or control plane churn. - **Automatic Scale-to-Zero**: When `activeWorkflows === 0` and no new task arrives within 10 consecutive seconds, the worker initiates graceful DBOS shutdown and exits with code `0`. - The completed Job pod terminates, freeing the node so Cluster Autoscaler can terminate the EC2 instance down to zero. --- ## Step-by-Step Deployment Guide ### Prerequisites - [AWS CLI](https://aws.amazon.com/cli/) installed and configured (`aws configure`). - [`eksctl`](https://eksctl.io/) installed. - [`kubectl`](https://kubernetes.io/docs/tasks/tools/) installed. - [`helm`](https://helm.sh/) installed. ### Step 1: Provision the EKS Cluster Create the cluster using the declarative [`eks/eks-cluster.yaml`](./eks-cluster.yaml) configuration: ```bash eksctl create cluster -f eks/eks-cluster.yaml ``` This provisions: - An EKS 1.31 control plane in `us-east-1` across multiple AZs. *(Note: AWS Graviton5 `c9g` instances are available in `us-east-1`. If deploying to a region where `c9g` is not yet offered, substitute `c7g.large` / `c7g.xlarge` or `c8g` in `eks/eks-cluster.yaml`.)* - `shumai-general`: 1x `c9g.large` On-Demand instance (labeled `workload: general`) for API, database, and storage. - `shumai-transcode`: Scale-to-zero `c9g.xlarge` On-Demand instance pool (`minSize: 0`, `maxSize: 5`, `desiredCapacity: 0`, tainted `dedicated=transcode:NoSchedule`). - AWS EBS CSI Driver with IAM OIDC roles. ### Step 2: Install KEDA Install KEDA using Helm: ```bash helm repo add kedacore https://kedacore.github.io/charts helm repo update helm install keda kedacore/keda --namespace keda --create-namespace ``` Wait for the KEDA operator to reach `Running` state: ```bash kubectl wait --namespace keda --for=condition=ready pod -l app.kubernetes.io/name=keda-operator --timeout=120s ``` ### Step 3: Deploy the Shumai Stack Deploy all components with Kustomize: ```bash kubectl apply -k eks/ ``` This applies: - [`eks/storageclass.yaml`](./storageclass.yaml): Configures `gp3` as the default StorageClass using `ebs.csi.aws.com` with `WaitForFirstConsumer` binding and dynamic EBS volume expansion. - PostgreSQL (`pgvector:pg18`) with durable EBS storage. - Garage S3 object store with automatic bucket and CORS initialization (`garage-init` job). - Shumai API & Web server. - KEDA `ScaledJob` for transcode worker auto-scaling. Wait for the core database and storage services to initialize: ```bash # Wait for PostgreSQL and Garage S3 pods to become Ready kubectl wait --namespace shumai --for=condition=ready pod -l app=shumai-postgres --timeout=180s kubectl wait --namespace shumai --for=condition=ready pod -l app=garage --timeout=180s # Wait for the bucket initialization job to complete kubectl wait --namespace shumai --for=condition=complete job/garage-init --timeout=180s # Verify all pods in the shumai namespace kubectl get pods -n shumai -o wide ``` *(Note: On first startup, `shumai-server` automatically runs database migrations against PostgreSQL; this may take 30–60 seconds before the server reaches `Running (1/1)`.)* ### Step 4: Configure Public Access & S3 Endpoint AWS automatically provisions external Elastic Load Balancers for `shumai-server` and `garage-public`. Because AWS takes 30–60 seconds to allocate external hostnames, wait for the hostnames to become available: ```bash # Wait for AWS to assign external hostnames to the LoadBalancers echo "Waiting for LoadBalancer external hostnames..." until [ -n "$(kubectl get svc garage-public -n shumai -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')" ]; do sleep 5 done GARAGE_ELB=$(kubectl get svc garage-public -n shumai -o jsonpath='{.status.loadBalancer.ingress[0].hostname}') SERVER_ELB=$(kubectl get svc shumai-server -n shumai -o jsonpath='{.status.loadBalancer.ingress[0].hostname}') echo "Garage S3 Hostname: http://${GARAGE_ELB}:3900" echo "Shumai Web UI: http://${SERVER_ELB}" ``` Inject the public Garage S3 hostname into `shumai-server` so that browsers generate correct presigned upload URLs: ```bash kubectl set env deployment/shumai-server -n shumai AWS_PUBLIC_ENDPOINT_URL_S3="http://${GARAGE_ELB}:3900" ``` *(Note: In production with custom domains, point your DNS CNAME records to the load balancers and configure `AWS_PUBLIC_ENDPOINT_URL_S3: https://s3.mycompany.com` in [`eks/config.yaml`](./config.yaml).)* ### Step 5: Deploy Kubernetes Cluster Autoscaler Deploy Cluster Autoscaler to dynamically provision `c9g.xlarge` EC2 nodes when KEDA jobs are created: ```bash curl -s https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml \ | sed 's//shumai-eks/g' \ | kubectl apply -f - # Pin autoscaler to the general workload node kubectl patch deployment cluster-autoscaler -n kube-system -p '{"spec":{"template":{"spec":{"nodeSelector":{"workload":"general"}}}}}' ``` ### Step 6: Test Scale-to-Zero Queue-Driven Autoscaling 1. Open `http://${SERVER_ELB}` in your browser and register an account via **Sign Up**. *(The first registered account automatically becomes the instance administrator).* 2. Create a project and upload a batch of 5–10 video files simultaneously. 3. Observe KEDA detecting pending DBOS tasks and spawning individual Kubernetes Jobs: ```bash kubectl get scaledjobs -n shumai kubectl get jobs -n shumai -w kubectl get pods -n shumai -l app=shumai-transcode -o wide kubectl get nodes -L role,workload ``` 4. Watch Cluster Autoscaler dynamically provision dedicated `c9g.xlarge` Graviton5 EC2 instances for the Jobs. 5. As each video finishes transcoding, its worker pod exits `0` and completes immediately. 6. Once all jobs are done, KEDA holds 0 jobs, and Cluster Autoscaler terminates all transcode EC2 instances, returning the transcode pool to **0 nodes ($0/hr)**! ### Step 7: Cluster Teardown & Cost Cleanup When you are finished testing, delete all resources to prevent ongoing cloud charges: ```bash # 1. Delete Kubernetes namespace first to release AWS ELB load balancers and EBS volumes # (Important: Wait for this to complete so AWS SecurityGroups can be deleted without dependency conflicts) kubectl delete namespace shumai # 2. Uninstall KEDA helm uninstall keda -n keda kubectl delete namespace keda # 3. Delete the EKS cluster and CloudFormation stacks eksctl delete cluster -f eks/eks-cluster.yaml --wait # 4. Confirm all AWS resources are released aws eks list-clusters --region us-east-1 ```