- Upgrade Bun to 1.4.4 across Dockerfiles, CI workflows, and release configurations. - Update @types/bun and bun-types. - Remove eslint-plugin-prettier and update format script to use native bun fmt. - Configure .prettierignore to exclude packages/webui/public and minified files.
24 KiB
Developer Guide
We use a pull request–based workflow for all tasks.
Workflow
Before starting any task, create and switch to a new feature branch:
git checkout -b <branch-name>
Complete the task on that branch.
After finishing the work and verifying that all checks pass:
- Update
CHANGELOG.mdunder## [Unreleased]for user-facing changes (see Changelog Guidelines). - Stage and commit your changes.
- Push the branch to
origin. - Open a pull request using the GitHub CLI.
git add .
git commit -m "<commit-message>"
git push origin <branch-name>
gh pr create --title "<pull-request-title>" --body "<pull-request-body>"
Commit Messages
We follow the Conventional Commits specification.
Commit messages MUST be formatted as:
<type>[optional scope]: <description>
[optional body]
[optional footer(s)]
Example:
feat(auth): add OAuth login support
The commit contains the following structural elements, to communicate intent to the consumers of your library:
fix:a commit of the typefixpatches a bug in your codebase.feat:a commit of the typefeatintroduces a new feature to the codebase.BREAKING CHANGE:a footer or!after the type/scope introduces a breaking API change.- Additional types are allowed (for example:
docs:,refactor:,test:,chore:).
Write commit messages in the imperative mood and keep descriptions concise and specific.
Reference: https://www.conventionalcommits.org/en/v1.0.0/#summary
Changelog Guidelines
We maintain CHANGELOG.md following Keep a Changelog.
Before opening a pull request, add an entry under ## [Unreleased] for any user-facing changes:
- End-User Friendly: Make entries easy to understand for end users. For new features, explain what they actually do in plain language without heavy technical jargon. For bug fixes, describe what the bug was in a friendly, relatable way rather than focusing on internal code mechanics, file paths, or private symbols.
- Formatting:
- **<scope>**: <Description>(e.g.- **asset**: Fix an issue where moving a video to trash while it was still transcoding could cause it to lose its trashed status). - No PR Numbers: Do not include PR numbers or links; the release pipeline automatically appends the detailed PR list.
- Internal Changes: Routine refactors, test additions, or internal chores without external behavior changes may omit changelog updates.
Pull Request Template
## Summary
Briefly describe what this PR changes and why.
## Changes
- List the main updates made in this branch.
- Include any important implementation details.
## Verification
- Describe the checks or tests you ran.
- Include relevant outputs if applicable.
## Manual Verification Plan
- Provide concise instructions for manually verifying this change.
- Assume familiarity with the project and existing development workflow.
- Skip boilerplate steps (e.g. starting the app, logging in, opening common pages) unless they are part of the change.
- Focus on what needs to be verified, including scenarios, edge cases, and expected results.
## Notes
Add any additional context, caveats, or follow-up work.
Submission Rules
-
Strict Requirement: A submission is considered complete only when there is a single final code state in which all of the following pass simultaneously:
bun run lintbun run formatbun run typecheckbun run testbun run test:e2e:appbun run test:e2e:webuibun run test:e2e:workflow
-
Backend Testing Mandate: Every backend feature, service method, workflow, and activity MUST be accompanied by comprehensive tests. Logic-heavy code without corresponding test coverage is considered incomplete.
-
Fixes must be iterated until no check causes any other check to fail.
-
Type Safety: The use of explicit
anyis strictly forbidden and will result in lint errors. You should useunknowninstead when the type is not known. If you absolutely must useanydue to a limitation (e.g. interacting with an untyped 3rd party library), you must add an eslint-disable comment (e.g.,// eslint-disable-next-line @typescript-eslint/no-explicit-any) and include a comment directly above it clearly explaining why we cannot be type-safe here. -
Do not submit intermediate states where some checks pass and others fail, even temporarily.
-
Cleanup Requirement: Remove all verification related files (scripts, screenshots,
verification/folder) before submit.
Backend Architecture
The backend is built with:
- Runtime: Bun
- Framework: Hono
- ORM: Prisma (with Pgvector18)
- Database: Pgvector18
Workspace Architecture
The project is a monorepo managed by Bun Workspaces. It follows a strictly decoupled architecture where each domain or layer is its own package:
- WebUI (
packages/webui): React-based frontend. - API (
packages/api): Hono-based HTTP entry point. Handles requests and calls Core services. - Core (
packages/core): Business logic, services, and infrastructure utilities. - Database (
packages/db): Prisma client, schema, and migrations. - DTOs (
packages/dtos): Shared type definitions and Zod schemas used by both API and WebUI. - Workers: Specialized packages for background task execution:
@shumai/workflow-core: Common workflow engine logic.@shumai/agent: AI agent workflows and activities.@shumai/transcode: Media processing workflows and activities.
Layered Communication Rules
- API Layer calls Core Layer. Do not access the database directly in the API layer.
- Core Layer calls Database Layer and other Core services.
- DTO Layer is imported by all layers to ensure end-to-end type safety.
- No Direct DB Leak: Do not return Prisma objects directly from the API; always map them to DTOs.
Dependency Management
We follow Bun's monorepo conventions for dependency management:
- Self-Contained Packages: Every workspace package MUST declare its own runtime
dependenciesin its localpackage.json. Do not rely on dependencies being available via the root. - Shared DevDependencies: Common development tools (e.g.,
typescript,eslint,vitest,prisma) MUST be declared in the rootpackage.jsonto ensure version consistency across the workspace. - Local DevDependencies: Tools specific to a single package (e.g.,
@vitejs/plugin-reactforwebui) should be declared in that package's localpackage.json. - Workspace Imports: Use
workspace:*for internal package dependencies. - Clean Root: The root
package.jsonmust not contain runtimedependencies. It is reserved for shareddevDependenciesand workspace-widescripts. - Adding Packages: Never edit
package.jsonmanually to add new dependencies. Always usebun add(e.g.,bun add <package> --filter <workspace>) from the root without specifying an explicit version, letting Bun resolve the correct version.
TypeScript Configuration
We use a root-level tsconfig.json to manage path mappings for the entire workspace. This ensures that imports across packages resolve correctly during typechecking and in the IDE.
- Path Mappings: All workspace packages MUST have an entry in the
pathsobject in the roottsconfig.json."paths": { "@shumai/core": ["packages/core/src/index.ts"], "@shumai/core/src/*": ["packages/core/src/*"] } - Package tsconfig: Each package should have its own
tsconfig.jsonthat extends the root configuration and defines its specificinclude/excluderules. - Cross-Package Resolution: When adding a new package, you MUST update the root
tsconfig.jsonpath mappings to ensure it can be imported by other packages.
Service Layer Patterns
- Instance Methods: Define service logic as instance methods on a class, not static methods.
- Singleton Export: Export a singleton instance of the service class.
export class MyService { async doSomething() { ... } } export const myService = new MyService() - Usage: Import and use the exported instance.
import { myService } from '@shumai/core/src/services/myService' await myService.doSomething()
Hono API Definitions
- Route Chaining: Always define API routes using method chaining on a Hono instance. This ensures that the type definitions for inputs and outputs are correctly inferred and preserved in the exported type.
- Export Pattern: Export the chained route instance as the default export.
const app = new Hono() const route = app .get('/', ...) .post('/', ...) export default route - Type Safety: Use
zValidatorfor request validation (query, json, form) to ensure full end-to-end type safety with the Hono RPC client.
Frontend Architecture
The frontend is built with:
- Runtime: Bun (Bundler & Runner)
- Framework: React
- Styling: Tailwind CSS + Shadcn UI (Components)
UI Rules
- Prefer System Colors: Always prefer using semantic system color design tokens (e.g.,
bg-background,text-foreground,text-sidebar-primary,text-muted-foreground, etc.) over hardcoded color utilities (e.g.,bg-blue-50,text-emerald-600,bg-emerald-500). This ensures seamless compatibility with light and dark themes.
Route Code-Splitting
To optimize initial bundle size, all routes must be code-split into [route].tsx (lightweight route definition and loaders) and [route].lazy.tsx (UI component and heavy imports).
Development
- Run Dev:
bun run dev(Runs on port 3000)
Testing
We use Vitest for testing (via bun run test). Do not use bun test as it uses Bun's native test runner which is not compatible with this project's test suite.
Bug Fixes
- Mandatory Reproduction: When fixing a backend bug, you MUST first write a test case that reproduces the bug (demonstrates the failure) before applying the fix. This ensures the bug is truly understood and prevents future regressions.
Test Maintenance Rule
- Do not delete tests unless the feature or code being tested is permanently removed from the codebase.
- When refactoring code, you must also refactor the corresponding tests to ensure they pass with the new implementation.
- This ensures we maintain regression coverage and do not silently break features.
Service Tests
- Located in
packages/core/src/**/*.test.ts. - For comprehensive testing against a real database instance, we use
Testcontainersalongside aPrismaTestingHelperwhich wraps each test within an automatic PostgreSQL transaction rollback boundary. - All service tests must use
setupTestDbHooks()from@shumai/dbto initialize the database boundaries for each test suite inside thedescribeblock. This automatically registersbeforeEachandafterEachhooks to manage transaction state. - Import
prismafrom@shumai/dbdirectly to interact with the database in tests. The testing helper automatically proxies it.
import { describe, it, expect } from 'vitest'
import { prisma, setupTestDbHooks } from '@shumai/db'
import { teamService } from '@shumai/core/src/team/team'
describe('TeamService', () => {
setupTestDbHooks()
it('works with db', async () => {
const user = await prisma.user.create({
data: { name: 'Test User', password: 'pw' },
})
const team = await teamService.ensureDefaultTeam()
expect(team).toBeDefined()
expect(team.name).toBe('Default Team')
})
})
The bunfig.toml is configured to run setup-tests.ts automatically as a preload step. This initializes the Testcontainers PostgreSQL instance and applies migrations before the tests begin running.
API Tests
- Located in
packages/api/src/**/*.test.ts. - Mock the Service layer using
vi.spyOn(service, 'method')orvi.mock('@shumai/core/src/team/team'). - Verify that the API calls the Service methods correctly.
import { describe, it, expect, vi } from 'vitest'
import { teamService } from '@shumai/core/src/team/team'
import app from './team'
describe('Team API', () => {
it('calls service', async () => {
const mockEnsureDefaultTeam = vi
.spyOn(teamService, 'ensureDefaultTeam')
.mockResolvedValue({ id: '1', name: 'Default Team' } as any)
// ... test logic ...
expect(mockEnsureDefaultTeam).toHaveBeenCalled()
mockEnsureDefaultTeam.mockRestore()
})
})
Workflow E2E Tests
- Located in
packages/e2e/workflow/**/*.test.ts. - Run via
bun run test:e2e:workflow. - Mandatory test coverage: When creating or updating workflows or activities, you MUST create or update the corresponding E2E tests.
- Mocking: Mock ONLY AI API calls. Do not mock S3, databases, or media/transcode extraction services.
- Mock the AI response by spying on
AgentHarness.prototype.prompt:
Web App E2E Tests
- Located in
apps/web/e2e/**/*.spec.ts, organized by domain undertests/<domain>/(e.g.auth,project). - Run via
bun run test:e2e:app. - Fixtures: Use the
owner,project, andfilefixtures for setup — they seed data through API calls/DB (no slow UI setup). You can add more fixtures if necessary. - Documentation: Every test case MUST be documented in
apps/web/e2e/README.md. Add or update the corresponding entry in the README whenever you add, rename, or remove a spec.
Prisma Configuration & Migrations
- Schema:
packages/db/prisma/schema.prisma - Generated Client:
packages/db/src/generated/prisma - Config:
prisma.config.ts(Required for Prisma 7+)
Strict JSON Typing
We use prisma-json-types-generator to enforce strict type-safety for Prisma Json columns, rather than typing them as generic JsonValue.
- Defining Types: In
packages/db/prisma/schema.prisma, annotate the JSON column with a triple-slash comment specifying the type name frompackages/db/src/prisma-json-types.ts:model WorkflowTask { /// [WorkflowTaskPayload] payload Json? } - Declaring Types: All typed JSON shapes must be defined in
packages/db/src/prisma-json-types.tsunder the globalPrismaJsonnamespace:declare global { namespace PrismaJson { export type WorkflowTaskPayload = Record<string, unknown> | TaskSpec | AiTaskPayload } } - Usage: When accessing these columns on the Prisma client (e.g.
task.payload), the field will automatically be typed as the declared shape (e.g.,PrismaJson.WorkflowTaskPayload | null). - Accessing Properties: When working with union types or custom structures in the JSON payload, use proper narrowing or type guards instead of casting to
any. Directas anycasting should be avoided.
Migration Workflow
- Development: Use
bun --bun run prisma migrate devto create and apply migrations during development. - Production: Use
bun --bun run prisma migrate deployto apply pending migrations in production environments. - No Manual DDL Creation: Never hand-write schema (DDL) migration files or directories. Always generate them with Prisma CLI (
bun --bun run prisma migrate dev --create-onlyto create a template, then edit it if needed, orbun --bun run prisma migrate devto apply) so Prisma correctly tracks migration metadata and checksums. - Data Migrations Are Allowed: Pure data migration SQL (no schema change) may be written by hand, but it MUST live inside a migration generated via
bun --bun run prisma migrate dev --create-onlyso it is tracked and applied automatically. - No Automatic Dev DB Reset: Do not run
prisma migrate reset --forceor commands that force-reset the database automatically. If migrations become out of sync or a reset is required, stop executing, report the situation to the user, and present suggested manual cleanup/reset steps for the user to execute.
Commands
- Generate Client:
bun --bun run prisma generate - Apply Migrations (Dev):
bun --bun run prisma migrate dev - Deploy Migrations (Prod):
bun --bun run prisma migrate deploy
ID Generation & Sorting
- ULID Default: All models (except key-value stores like
Setting) must use@default(ulid())for theidfield inschema.prisma.
model Example {
id String @id @default(ulid())
}
- No Manual Assignment: Do not manually generate or assign ULIDs in application code (e.g.,
packages/core/src/*.ts) for creating database records. Let Prisma handle it via the schema default.- Exception: File names generated by
FileServicemay still useulid()internally, but this is separate from the database ID.
- Exception: File names generated by
- Sorting: All list APIs must order results by
iddescending (orderBy: { id: 'desc' }) to ensure consistent, time-based sorting (since ULIDs are sortable). Do not sort bycreatedAtas it is not indexed.
Infinite Scroll & Pagination
- Frontend: For paginated APIs (which contain
pageInfoandtotalfields), the UI MUST use an auto infinite scroll style design usinguseInfiniteQueryfrom@tanstack/react-queryandreact-intersection-observer. Use thetotalcount from the API response instead of counting the results locally.- The default
limitshould be 20.
- The default
- Backend:
- Most list APIs support cursor pagination using opaque tokens (via
hyrumtokenlogic) using thepaginateQueryhelper function. - Pass
PaginationParamscontainingfirst(limit) andafter(cursor). - The default
limitshould be 20. - Always use the
paginateQueryhelper function frompackages/core/src/pagination.tsfor consistent cursor pagination.
- Most list APIs support cursor pagination using opaque tokens (via
DBOS Workflow & Activities
We use DBOS Transact (@dbos-inc/dbos-sdk) as our unified workflow engine for reliable, distributed background execution.
Architecture
- Workflows: Orchestrate multiple steps/activities reliably. Defined in worker packages (e.g.,
@shumai/agent/src/workflows,@shumai/transcode/src/workflows). - Activities (Steps): Perform the actual work (DB updates, API calls, media processing). Grouped by domain in worker packages (e.g.,
@shumai/agent/src/activities,@shumai/transcode/src/activities). - Executors:
DbosExecutor(in@shumai/workflow-core): Coordinates workflow and step dispatching via DBOS queues against PostgreSQL.
- Automatic Submission: New
WorkflowTaskrecords are automatically enqueued toWorkflowServicevia a Prisma Client Extension defined in@shumai/db.
Dynamic Registration
To avoid circular dependencies, workflows are dynamically registered onto the workflow registry at bootstrap time via initializer functions (e.g., initAgentWorkflows(), initTranscodeWorkflows()).
Non-Retryable Error Handling
To prevent the engine from indefinitely retrying fatal, expected business validation failures (e.g., missing records, invalid configurations, missing parameters), never throw standard Error objects from workflow or activity logic. Instead, throw a non-retryable ApplicationFailure (alias of WorkflowFailure).
Mandatory Requirement: You MUST set nonRetryable: true in the options object passed to ApplicationFailure.create. Standard errors or failure objects without nonRetryable: true will cause DBOS to retry the activity or workflow according to its retry policy.
- Import: Import
ApplicationFailurefrom@shumai/workflow-core:import { ApplicationFailure } from '@shumai/workflow-core' throw ApplicationFailure.create({ message: 'agentId missing in payload', nonRetryable: true })
Workflow Patterns
- Definition: Define workflows as exported async functions.
import { activityA, activityB } from '../activities/my-activities' export async function myWorkflow(task: WorkflowTask): Promise<void> { await activityA({ ... }) await activityB({ ... }) } - Activity Invocation: Workflows import and call activities directly as standard async functions.
- Queues: Workflows are dispatched to dedicated task queues (e.g.
TaskQueueTranscode,TaskQueueAgent) managed by DBOS.
Activity Patterns
- Definition: Wrap exported activity functions with
defineStepfrom@shumai/workflow-core.export const myActivity = defineStep( async function myActivity(params: MyActivityParams): Promise<MyActivityResult> { ... }, { runOnCancel: false }, // optional: set runOnCancel: true for cleanup/status handlers ) - Implementation: Activities can and should call other services (e.g.,
aiService,transcodeService) or interact with the database directly. Database (Prisma) queries or updates are allowed in all activity types (e.g., agent or transcode) for simplicity.
Worker Identity & Multi-Instance Deployments
In multi-replica deployments (e.g., multiple transcode or agent worker instances), each worker instance MUST have a unique executor ID to prevent startup recovery from re-enqueueing workflows being actively processed by other live workers.
- Identity Derivation:
DbosExecutorautomatically derives a unique per-instance ID:- If
SHUMAI_WORKER_IDorDBOS__VMIDis set, it is used as the instance identifier. - In Kubernetes, set
POD_NAME(via Downward API:fieldRef: { fieldPath: metadata.name }) to provide stable, per-pod identity across StatefulSet restarts. - In Docker/container environments,
HOSTNAME(container ID) is automatically used. - On bare metal/multi-process hosts,
process.pidand an instance suffix are appended to guarantee uniqueness.
- If
Durable Step File Exchange
Intermediate files passed across steps must be saved to durable storage (S3) by the producing step, consuming steps must use ensureLocalMediaFile (or S3 fetch) rather than assuming local /tmp presence, and any purely temporary S3 scratch files must be cleaned up in a finally block.
Import conventions
- When importing across workspace packages, always use absolute workspace imports (e.g.
import { prisma } from '@shumai/db',import { teamService } from '@shumai/core/src/team/team'). - DTO Naming: Do not append the
Dtosuffix to types or interfaces (e.g. useJoinRequestinstead ofJoinRequestDto).
Radix UI / Shadcn UI Patterns
- Modal Overlays: When triggering a dialog (e.g.,
Dialog,AlertDialog) from inside aDropdownMenu, you must setmodal={false}on the<DropdownMenu>component. Failing to do so will cause the UI to freeze due to conflicting focus management between the two modal components. - Scrollable Content: Always use the Shadcn
ScrollAreacomponent (@/ui/components/ui/scroll-area) for scrollable content containers to ensure consistent custom scrollbar UI across the application.
Logging
We use pino for logging. Always use structured logging to ensure logs are easily searchable and machine-readable.
- Import:
import { logger } from '@shumai/core/src/logger' - Usage: Pass an object as the first argument containing the metadata, and a string as the second argument for the descriptive message.
logger.info({ userId: user.id, projectId }, 'Project deleted successfully') - Levels: Use appropriate levels:
debug,info,warn,error.
Internationalization (i18n)
All user-facing text in the WebUI (packages/webui) MUST be internationalized using Paraglide JS. Hardcoded text in UI components is strictly forbidden.
Message Definition
- Define your English message strings in packages/webui/messages/en.json.
- Define the corresponding Chinese translation strings in packages/webui/messages/zh.json.
- Keys should use
snake_case.
Example (en.json):
{
"my_key": "My English Text"
}
Compilation
When you add or update translation keys, you must compile them to regenerate the type-safe functions:
# Compile once (generates type-safe TS/JS functions on disk)
bun run i18n:compile
# Watch and recompile on change during development
bun run i18n:watch
Usage in Components
Import the compiled message namespace m and call it as a function:
import { m } from '@/ui/paraglide/messages.js'
export function MyComponent() {
return (
<div>
<h1>{m.my_key()}</h1>
</div>
)
}
Always use the absolute workspace alias path @/ui/paraglide/messages.js (or runtime.js for runtime features) when importing Paraglide outputs.