mirror of
https://github.com/shumaiOne/shumai.git
synced 2026-11-22 04:19:29 +00:00
* feat(workflow): migrate workflow engine from temporal to dbos Replace Temporal and LocalExecutor with DBOS Transact (@dbos-inc/dbos-sdk) as the unified workflow engine. Background tasks, media transcoding, and AI agent workflows now execute with durable steps and task queues directly against PostgreSQL, eliminating the need for an external Temporal cluster and separate polling executors. * fix(workflow): submit tasks via DBOSClient to allow enqueueing from steps Enqueuing workflows using DBOS.enqueueWorkflowWithOptions threw a DBOSInvalidWorkflowTransitionError when triggered from within a running step or activity (e.g. creating embedding or autofill tasks during video transcoding). Use DBOSClient to submit and cancel workflow tasks directly against the system database, detaching execution from the ambient workflow/step context. * feat(web): support SHUMAI_WORKERS environment variable * fix(workflow): ensure CLI tasks are enqueued to DBOS and run DBOS migrations in global setup - Initialize workflows and wait for pending DBOS submissions in shumai regenerate-embeddings CLI - Remove no-op submit bypass and enqueue tasks via DBOSClient in producer and CLI contexts - Add DBOS migration runner in test global setup to avoid concurrent migration deadlocks across test forks - Update changelog and add regression tests * fix(workflow): ensure unique per-instance executor identity for worker replicas - Derive unique per-instance DBOS executor IDs across worker replicas - Support SHUMAI_WORKER_ID, DBOS__VMID, and POD_NAME (Kubernetes Downward API) - Disambiguate multi-replica bare metal/container deployments with host, PID, and instance suffix - Prevent startup recovery from re-enqueueing workflows actively executed by other live workers - Add multi-instance startup recovery tests and update deployment documentation * fix(workflow): only register listened queues and update concurrency on change - Only register queues that the executor instance actively listens to - Use always_update conflict resolution so environment variable changes take effect on restart - Prevent dedicated workers from overwriting unmanaged queue concurrency configurations - Add tests verifying concurrency updates and queue isolation across worker roles * refactor(workflow): simplify activities to use defineStep and direct calls (#482) * refactor(workflow): simplify activities to use defineStep and direct calls - Wrap activities using defineStep from @shumai/workflow-core - Remove obsolete getActivities, registerActivities, dynamic activity proxies, and workerQueue arguments - Remove executor.ts and registry.ts, consolidating workflow registration into workflow.ts - Call activities directly across transcode and agent workflows - Update all corresponding workflow tests and developer guide documentation * fix(workflow): properly detect DBOSWorkflowCancelledError in runOnCancel fallback - DBOSWorkflowCancelledError in DBOS SDK does not set name, inheriting 'Error' - Implement isDbosWorkflowCancelledError checking instanceof, constructor.name, and dbosErrorCode - Update unit tests to verify real DBOS SDK cancellation error instances * docs(workflow): update documentation and readme for dbos migration - Rename docs/configuration/temporal.mdx to docs/configuration/distributed.mdx - Document DBOS-based distributed worker architecture and task queues - Remove obsolete docker-compose/temporal/ - Update environment variables documentation in docs/configuration/env-variables.mdx - Update references in README.md, README_CN.md, docs/docs.json, and related guides * feat(deploy): implement KEDA ScaledJob and scale-to-zero On-Demand transcoding (#483) * feat(deploy): add shumai-transcode docker image and k8s autoscaling example with garage s3 * docs(k8s): document port-forwarding and in-cluster s3 domain mapping * feat(storage): support public s3 endpoint for browser uploads and update k8s example * fix(k8s): configure s3 bucket cors rules in garage initialization * fix(k8s): increase transcode worker memory limit to 4Gi for 4K video encoding * feat(deploy): add AWS EKS deployment with Graviton3 Spot nodegroups and autoscaling * docs(deploy): redesign EKS documentation, rename k8s to eks, and add scalable deployment feature * docs: rephrase DBOS feature bullet to Durable Workflows via DBOS * docs(eks): upgrade default instances to Graviton5 c9g.large and c9g.xlarge * feat(eks): use On-Demand for general nodegroup and add spot capacity diversification * docs(eks): fix Architecture diagram to locate all transcode worker pods in Transcode Compute Pool * feat(deploy): implement KEDA ScaledJob and scale-to-zero On-Demand transcoding * feat(transcode): support idle-timeout auto-drain for KEDA scaled jobs * fix(deploy): fix EKS deployment manifests and documentation based on cluster verification
359 lines
10 KiB
TypeScript
359 lines
10 KiB
TypeScript
import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest'
|
|
import { prisma, AssetStatus } from '@shumai/db'
|
|
import { setupTestDbHooks } from '@shumai/db/test'
|
|
import { workflowService, TaskQueueTranscode } from '@shumai/workflow-core'
|
|
import { initTranscodeWorkflows } from '@shumai/transcode'
|
|
import { s3Service } from '@shumai/core/src/s3/s3'
|
|
import { fileURLToPath } from 'url'
|
|
import * as path from 'path'
|
|
import * as fs from 'fs'
|
|
|
|
const currentDir = path.dirname(fileURLToPath(import.meta.url))
|
|
const fixturesDir = path.resolve(currentDir, '../fixtures')
|
|
|
|
describe('Workflow E2E - transcodeImageWorkflow', () => {
|
|
setupTestDbHooks()
|
|
|
|
beforeAll(async () => {
|
|
process.env.S3_BUCKET = 'shumai-e2e-test-bucket-transcode'
|
|
|
|
initTranscodeWorkflows()
|
|
await workflowService.startWorkers(TaskQueueTranscode)
|
|
})
|
|
|
|
afterAll(async () => {
|
|
await workflowService.shutdownWorkers()
|
|
workflowService.close()
|
|
vi.restoreAllMocks()
|
|
try {
|
|
console.log('Cleaning up local E2E storage files...')
|
|
await s3Service.deletePrefix('shumai-e2e-test-bucket-transcode', '')
|
|
} catch (err) {
|
|
console.error('Failed to clean up E2E storage folder:', err)
|
|
}
|
|
})
|
|
|
|
it('should run transcodeMedia workflow for an image asset successfully', async () => {
|
|
// 1. Seed Database
|
|
const team = await prisma.team.create({
|
|
data: { name: 'E2E Image Transcode Team' },
|
|
})
|
|
|
|
const project = await prisma.project.create({
|
|
data: { name: 'E2E Image Transcode Project', teamId: team.id },
|
|
})
|
|
|
|
const storageKey = await prisma.storageKey.create({
|
|
data: {
|
|
key: 'projects/e2e/image-trans.png',
|
|
},
|
|
})
|
|
|
|
const asset = await prisma.asset.create({
|
|
data: {
|
|
name: 'image-trans.png',
|
|
type: 'file',
|
|
status: 'uploaded',
|
|
mediaType: 'image/png',
|
|
projectId: project.id,
|
|
storageKeyId: storageKey.id,
|
|
},
|
|
})
|
|
|
|
// 2. Seed S3 Storage from Fixture
|
|
const pngPath = path.join(fixturesDir, 'small.png')
|
|
const pngBuffer = fs.readFileSync(pngPath)
|
|
await s3Service.putObject(
|
|
'shumai-e2e-test-bucket-transcode',
|
|
'projects/e2e/image-trans.png',
|
|
pngBuffer,
|
|
pngBuffer.length,
|
|
'image/png',
|
|
)
|
|
|
|
// 3. Create Workflow Task
|
|
const task = await prisma.workflowTask.create({
|
|
data: {
|
|
type: 'transcode_image',
|
|
status: 'pending',
|
|
assetId: asset.id,
|
|
projectId: project.id,
|
|
teamId: team.id,
|
|
payload: {
|
|
projectId: project.id,
|
|
transcode: {
|
|
thumbnail: true,
|
|
},
|
|
},
|
|
},
|
|
})
|
|
|
|
// 4. Wait for workflow to complete
|
|
console.log(
|
|
`Submitted E2E Image Transcode Workflow Task. ID: ${task.id}. Awaiting completion...`,
|
|
)
|
|
const completedTask = await workflowService.executeWait(task, 45000)
|
|
|
|
// 5. Verification
|
|
expect(completedTask.status).toBe('completed')
|
|
|
|
const updatedAsset = await prisma.asset.findUnique({
|
|
where: { id: asset.id },
|
|
})
|
|
expect(updatedAsset?.status).toBe(AssetStatus.processed)
|
|
|
|
const mediaInfo = updatedAsset?.media as unknown as {
|
|
proxyType: string
|
|
imageTranscodes: unknown[]
|
|
thumbnail: unknown
|
|
}
|
|
expect(mediaInfo).toBeDefined()
|
|
expect(mediaInfo.proxyType).toBe('image')
|
|
expect(mediaInfo.imageTranscodes).toBeDefined()
|
|
expect(mediaInfo.imageTranscodes.length).toBeGreaterThan(0)
|
|
expect(mediaInfo.thumbnail).toBeDefined()
|
|
}, 50000)
|
|
|
|
it('should run transcodeMedia workflow for a PSD image asset successfully', async () => {
|
|
// 1. Seed Database
|
|
const team = await prisma.team.create({
|
|
data: { name: 'E2E PSD Transcode Team' },
|
|
})
|
|
|
|
const project = await prisma.project.create({
|
|
data: { name: 'E2E PSD Transcode Project', teamId: team.id },
|
|
})
|
|
|
|
const storageKey = await prisma.storageKey.create({
|
|
data: {
|
|
key: 'projects/e2e/test.psd',
|
|
},
|
|
})
|
|
|
|
const asset = await prisma.asset.create({
|
|
data: {
|
|
name: 'test.psd',
|
|
type: 'file',
|
|
status: 'uploaded',
|
|
mediaType: 'image/vnd.adobe.photoshop',
|
|
projectId: project.id,
|
|
storageKeyId: storageKey.id,
|
|
},
|
|
})
|
|
|
|
// 2. Seed S3 Storage from fixture test.psd
|
|
const psdPath = path.join(fixturesDir, 'test.psd')
|
|
const psdBuffer = fs.readFileSync(psdPath)
|
|
await s3Service.putObject(
|
|
'shumai-e2e-test-bucket-transcode',
|
|
'projects/e2e/test.psd',
|
|
psdBuffer,
|
|
psdBuffer.length,
|
|
'image/vnd.adobe.photoshop',
|
|
)
|
|
|
|
// 3. Create Workflow Task
|
|
const task = await prisma.workflowTask.create({
|
|
data: {
|
|
type: 'transcode_image',
|
|
status: 'pending',
|
|
assetId: asset.id,
|
|
projectId: project.id,
|
|
teamId: team.id,
|
|
payload: {
|
|
projectId: project.id,
|
|
transcode: {
|
|
thumbnail: true,
|
|
},
|
|
},
|
|
},
|
|
})
|
|
|
|
// 4. Wait for workflow to complete
|
|
console.log(
|
|
`Submitted E2E PSD Image Transcode Workflow Task. ID: ${task.id}. Awaiting completion...`,
|
|
)
|
|
const completedTask = await workflowService.executeWait(task, 45000)
|
|
|
|
// 5. Verification
|
|
expect(completedTask.status).toBe('completed')
|
|
|
|
const updatedAsset = await prisma.asset.findUnique({
|
|
where: { id: asset.id },
|
|
})
|
|
expect(updatedAsset?.status).toBe(AssetStatus.processed)
|
|
|
|
const mediaInfo = updatedAsset?.media as unknown as {
|
|
proxyType: string
|
|
imageTranscodes: unknown[]
|
|
thumbnail: unknown
|
|
}
|
|
expect(mediaInfo).toBeDefined()
|
|
expect(mediaInfo.proxyType).toBe('image')
|
|
expect(mediaInfo.imageTranscodes).toBeDefined()
|
|
expect(mediaInfo.imageTranscodes.length).toBeGreaterThan(0)
|
|
expect(mediaInfo.thumbnail).toBeDefined()
|
|
}, 50000)
|
|
|
|
it('should run transcodeMedia workflow for a RAW image asset (.dng) successfully', async () => {
|
|
// 1. Seed Database
|
|
const team = await prisma.team.create({
|
|
data: { name: 'E2E RAW Transcode Team' },
|
|
})
|
|
|
|
const project = await prisma.project.create({
|
|
data: { name: 'E2E RAW Transcode Project', teamId: team.id },
|
|
})
|
|
|
|
const storageKey = await prisma.storageKey.create({
|
|
data: {
|
|
key: 'projects/e2e/test.dng',
|
|
},
|
|
})
|
|
|
|
const asset = await prisma.asset.create({
|
|
data: {
|
|
name: 'test.dng',
|
|
type: 'file',
|
|
status: 'uploaded',
|
|
mediaType: 'image/x-adobe-dng',
|
|
projectId: project.id,
|
|
storageKeyId: storageKey.id,
|
|
},
|
|
})
|
|
|
|
// 2. Seed S3 Storage from fixture test.dng
|
|
const dngPath = path.join(fixturesDir, 'test.dng')
|
|
const dngBuffer = fs.readFileSync(dngPath)
|
|
await s3Service.putObject(
|
|
'shumai-e2e-test-bucket-transcode',
|
|
'projects/e2e/test.dng',
|
|
dngBuffer,
|
|
dngBuffer.length,
|
|
'image/x-adobe-dng',
|
|
)
|
|
|
|
// 3. Create Workflow Task
|
|
const task = await prisma.workflowTask.create({
|
|
data: {
|
|
type: 'transcode_image',
|
|
status: 'pending',
|
|
assetId: asset.id,
|
|
projectId: project.id,
|
|
teamId: team.id,
|
|
payload: {
|
|
projectId: project.id,
|
|
transcode: {
|
|
thumbnail: true,
|
|
},
|
|
},
|
|
},
|
|
})
|
|
|
|
// 4. Wait for workflow to complete
|
|
console.log(
|
|
`Submitted E2E RAW Image Transcode Workflow Task. ID: ${task.id}. Awaiting completion...`,
|
|
)
|
|
const completedTask = await workflowService.executeWait(task, 45000)
|
|
|
|
// 5. Verification
|
|
expect(completedTask.status).toBe('completed')
|
|
|
|
const updatedAsset = await prisma.asset.findUnique({
|
|
where: { id: asset.id },
|
|
})
|
|
expect(updatedAsset?.status).toBe(AssetStatus.processed)
|
|
|
|
const mediaInfo = updatedAsset?.media as unknown as {
|
|
proxyType: string
|
|
imageTranscodes: unknown[]
|
|
thumbnail: unknown
|
|
}
|
|
expect(mediaInfo).toBeDefined()
|
|
expect(mediaInfo.proxyType).toBe('image')
|
|
expect(mediaInfo.imageTranscodes).toBeDefined()
|
|
expect(mediaInfo.imageTranscodes.length).toBeGreaterThan(0)
|
|
expect(mediaInfo.thumbnail).toBeDefined()
|
|
}, 50000)
|
|
|
|
it('should transcode a rotated camera photo upright', async () => {
|
|
// 1. Seed Database
|
|
const team = await prisma.team.create({
|
|
data: { name: 'E2E Rotated Transcode Team' },
|
|
})
|
|
|
|
const project = await prisma.project.create({
|
|
data: { name: 'E2E Rotated Transcode Project', teamId: team.id },
|
|
})
|
|
|
|
const storageKey = await prisma.storageKey.create({
|
|
data: {
|
|
key: 'projects/e2e/portrait-rotated.jpg',
|
|
},
|
|
})
|
|
|
|
const asset = await prisma.asset.create({
|
|
data: {
|
|
name: 'portrait-rotated.jpg',
|
|
type: 'file',
|
|
status: 'uploaded',
|
|
mediaType: 'image/jpeg',
|
|
projectId: project.id,
|
|
storageKeyId: storageKey.id,
|
|
},
|
|
})
|
|
|
|
// 2. Seed S3 Storage from fixture portrait-rotated.jpg: stored 600x400 with EXIF
|
|
// orientation 6, the way cameras save portrait shots, so it is displayed 400x600.
|
|
const jpgPath = path.join(fixturesDir, 'portrait-rotated.jpg')
|
|
const jpgBuffer = fs.readFileSync(jpgPath)
|
|
await s3Service.putObject(
|
|
'shumai-e2e-test-bucket-transcode',
|
|
'projects/e2e/portrait-rotated.jpg',
|
|
jpgBuffer,
|
|
jpgBuffer.length,
|
|
'image/jpeg',
|
|
)
|
|
|
|
// 3. Create Workflow Task
|
|
const task = await prisma.workflowTask.create({
|
|
data: {
|
|
type: 'transcode_image',
|
|
status: 'pending',
|
|
assetId: asset.id,
|
|
projectId: project.id,
|
|
teamId: team.id,
|
|
payload: {
|
|
projectId: project.id,
|
|
transcode: {
|
|
thumbnail: true,
|
|
},
|
|
},
|
|
},
|
|
})
|
|
|
|
// 4. Wait for workflow to complete
|
|
console.log(
|
|
`Submitted E2E Rotated Image Transcode Workflow Task. ID: ${task.id}. Awaiting completion...`,
|
|
)
|
|
const completedTask = await workflowService.executeWait(task, 45000)
|
|
|
|
// 5. Verification: the asset and its transcode are portrait, not the stored landscape.
|
|
expect(completedTask.status).toBe('completed')
|
|
|
|
const updatedAsset = await prisma.asset.findUnique({
|
|
where: { id: asset.id },
|
|
})
|
|
expect(updatedAsset?.status).toBe(AssetStatus.processed)
|
|
|
|
const mediaInfo = updatedAsset?.media as unknown as {
|
|
imageTranscodes: Array<{ width: number; height: number }>
|
|
metadata: { originalWidth: number; originalHeight: number }
|
|
}
|
|
expect(mediaInfo.metadata.originalWidth).toBe(400)
|
|
expect(mediaInfo.metadata.originalHeight).toBe(600)
|
|
expect(mediaInfo.imageTranscodes[0].width).toBe(400)
|
|
expect(mediaInfo.imageTranscodes[0].height).toBe(600)
|
|
}, 50000)
|
|
})
|