49 Commits
Author SHA1 Message Date
Yiling-J 4ac274cdd4 feat(runner): replace server-side bash sandbox with remote runner (#530)
* feat(runner): replace server-side bash sandbox with remote runner

* fix(runner): address review findings for output bounds, key rotation, defaults, and generated uploads

* chore(docker): remove sandbox privileges and env vars from remaining compose files

* feat(cli): add runner start with bundled SRT sandboxing (#535)

* refactor(agent): expose runner bash without skill gating

* feat(cli): add runner start with optional SRT sandboxing

* refactor(cli): remove runner Python dependency

* feat(cli): bundle SRT and use platform-native runner shells

* refactor(cli): delegate runner processes to Execa

* chore(cli): upgrade @anthropic-ai/sandbox-runtime to 0.0.83

* feat(cli): discover Bash for Windows runners

Use Pi-style Bash discovery and launch behavior, exercise the packaged CLI shell, and wait for pending ZIP metadata reads before closing cancelled downloads.

* fix(runner): enforce ownership and safe reconnects

Gate dispatch on reconciliation, retain cancellations across disconnects, and stop replaced clients. Guard the single-server runner topology with a database lease. Restage missing generated inputs, report signal exits, and bound generated-file cleanup.

* feat(webui): add Korean translations for agent runners

* feat(cli): add runner event logging and --verbose flag (#536)

* feat(cli): add runner event logging and --verbose flag

* fix(cli): log runner capacity rejections accurately and use monotonic duration
2026-11-18 15:58:57 +08:00
yiling 2be1093b91 fix(cli): explain how to configure the installer PATH 2026-11-13 21:11:18 +08:00
yiling 83e902862c fix(cli): discover draft releases before publishing 2026-11-13 20:37:50 +08:00
Yiling-J 97c2939b5e feat(cli): add standalone downloads and installers (#528)
* feat(cli): add standalone downloads and installers

* fix(cli): silence PowerShell download progress

* fix(cli): use framework APIs for Windows verification

* fix(cli): preserve null backup path during Windows updates

* docs(cli): translate native installation quickstart into Chinese
2026-11-12 19:37:33 +08:00
Yiling-J 35a0e82284 refactor(agent)!: migrate to Pi Durable and embed the agent queue (#525)
* refactor(agent)!: migrate sessions to Pi Durable

Replace patched AgentHarness storage with per-session SQLite and PostgreSQL commit mirroring. Migrate legacy history automatically and run agent queues with the main server.

BREAKING CHANGE: Remove standalone agent workers and return native Pi entry records from chat history APIs.

* fix(agent): constrain session log dialog scrolling

Keep the header visible and wrap long output while log entries and expanded records scroll within the dialog. Cover desktop and narrow viewports through the dashboard Sessions flow.

* fix(agent): hide entry IDs from session log card headers

Retain internal entry IDs in expanded raw records for debugging. Verify that card headers omit the IDs while raw JSON preserves them.

* test(e2e): handle browser wheel limits in session log checks

Verify that a normal wheel event moves the viewport before explicitly scrolling to the final record. Keep the dialog bounds, fixed header, wrapping, and expanded-record assertions.

* fix(agent): select model providers without inheriting unrelated auth

* fix(webui): prevent resize feedback when toggling agent panel

* test(agent): reproduce concurrent comment session races

* fix(agent): serialize comment sessions and deduplicate history

* docs(changelog): consolidate Pi Durable changes into one entry
2026-11-11 16:05:18 +08:00
Yiling-J 176b7f0eab feat(workflow): migrate workflow engine from temporal to dbos (#481)
* feat(workflow): migrate workflow engine from temporal to dbos

Replace Temporal and LocalExecutor with DBOS Transact (@dbos-inc/dbos-sdk) as the unified workflow engine. Background tasks, media transcoding, and AI agent workflows now execute with durable steps and task queues directly against PostgreSQL, eliminating the need for an external Temporal cluster and separate polling executors.

* fix(workflow): submit tasks via DBOSClient to allow enqueueing from steps

Enqueuing workflows using DBOS.enqueueWorkflowWithOptions threw a
DBOSInvalidWorkflowTransitionError when triggered from within a running
step or activity (e.g. creating embedding or autofill tasks during
video transcoding).

Use DBOSClient to submit and cancel workflow tasks directly against the
system database, detaching execution from the ambient workflow/step
context.

* feat(web): support SHUMAI_WORKERS environment variable

* fix(workflow): ensure CLI tasks are enqueued to DBOS and run DBOS migrations in global setup

- Initialize workflows and wait for pending DBOS submissions in shumai regenerate-embeddings CLI
- Remove no-op submit bypass and enqueue tasks via DBOSClient in producer and CLI contexts
- Add DBOS migration runner in test global setup to avoid concurrent migration deadlocks across test forks
- Update changelog and add regression tests

* fix(workflow): ensure unique per-instance executor identity for worker replicas

- Derive unique per-instance DBOS executor IDs across worker replicas

- Support SHUMAI_WORKER_ID, DBOS__VMID, and POD_NAME (Kubernetes Downward API)

- Disambiguate multi-replica bare metal/container deployments with host, PID, and instance suffix

- Prevent startup recovery from re-enqueueing workflows actively executed by other live workers

- Add multi-instance startup recovery tests and update deployment documentation

* fix(workflow): only register listened queues and update concurrency on change

- Only register queues that the executor instance actively listens to

- Use always_update conflict resolution so environment variable changes take effect on restart

- Prevent dedicated workers from overwriting unmanaged queue concurrency configurations

- Add tests verifying concurrency updates and queue isolation across worker roles

* refactor(workflow): simplify activities to use defineStep and direct calls (#482)

* refactor(workflow): simplify activities to use defineStep and direct calls

- Wrap activities using defineStep from @shumai/workflow-core
- Remove obsolete getActivities, registerActivities, dynamic activity proxies, and workerQueue arguments
- Remove executor.ts and registry.ts, consolidating workflow registration into workflow.ts
- Call activities directly across transcode and agent workflows
- Update all corresponding workflow tests and developer guide documentation

* fix(workflow): properly detect DBOSWorkflowCancelledError in runOnCancel fallback

- DBOSWorkflowCancelledError in DBOS SDK does not set name, inheriting 'Error'
- Implement isDbosWorkflowCancelledError checking instanceof, constructor.name, and dbosErrorCode
- Update unit tests to verify real DBOS SDK cancellation error instances

* docs(workflow): update documentation and readme for dbos migration

- Rename docs/configuration/temporal.mdx to docs/configuration/distributed.mdx
- Document DBOS-based distributed worker architecture and task queues
- Remove obsolete docker-compose/temporal/
- Update environment variables documentation in docs/configuration/env-variables.mdx
- Update references in README.md, README_CN.md, docs/docs.json, and related guides

* feat(deploy): implement KEDA ScaledJob and scale-to-zero On-Demand transcoding (#483)

* feat(deploy): add shumai-transcode docker image and k8s autoscaling example with garage s3

* docs(k8s): document port-forwarding and in-cluster s3 domain mapping

* feat(storage): support public s3 endpoint for browser uploads and update k8s example

* fix(k8s): configure s3 bucket cors rules in garage initialization

* fix(k8s): increase transcode worker memory limit to 4Gi for 4K video encoding

* feat(deploy): add AWS EKS deployment with Graviton3 Spot nodegroups and autoscaling

* docs(deploy): redesign EKS documentation, rename k8s to eks, and add scalable deployment feature

* docs: rephrase DBOS feature bullet to Durable Workflows via DBOS

* docs(eks): upgrade default instances to Graviton5 c9g.large and c9g.xlarge

* feat(eks): use On-Demand for general nodegroup and add spot capacity diversification

* docs(eks): fix Architecture diagram to locate all transcode worker pods in Transcode Compute Pool

* feat(deploy): implement KEDA ScaledJob and scale-to-zero On-Demand transcoding

* feat(transcode): support idle-timeout auto-drain for KEDA scaled jobs

* fix(deploy): fix EKS deployment manifests and documentation based on cluster verification
2026-10-19 15:35:08 +08:00
Yiling-J 7ca8ba8b2a fix(transcode): resolve RAW transcoding hang in Docker deployments (#447) 2026-10-03 12:45:12 +08:00
Yiling-J 5fe22ac52c feat(transcode): add initial camera RAW image support (#435)
- Add centralized RAW file extension set and detection helper for 23 common camera RAW formats
- Update getProxyType and EXTENSION_MIME_TYPES to route RAW formats into the image pipeline
- Add exiftool-vendored runtime dependency and mark as external in build-npm script
- Implement embedded JPEG extraction to temporary file with priority JpgFromRaw2 -> JpgFromRaw -> PreviewImage
- Read and apply EXIF orientation from RAW container to prevent rotation/mirroring issues
- Integrate into TranscodeService.getImageInfo and transcodeImage with proper temporary file cleanup
- Add hasUsablePreview guard in transcodeImageWorkflow to gracefully handle RAW files without embedded previews
- Add unit, service, workflow, and E2E tests for RAW processing
2026-09-29 21:09:27 +08:00
Yiling-J 94ea4c917a feat(transcode): add VA-API hardware acceleration support (#430)
* feat(transcode): add VA-API hardware acceleration support

* fix(docker): switch to jellyfin-ffmpeg 8.1.2-5 for VA-API support

* fix(transcode): probe hardware encoder usability to skip uninitialized drivers

* ci(release): add workflow_dispatch trigger with custom tag input

* fix(transcode): use 256x256 frame dimension for encoder usability probe

* feat(docker): add hwaccel transcoding compose presets matching immich

* docs: add hardware transcoding documentation and quickstart references

* feat(transcode): prioritize h264_vaapi before h264_qsv on Linux

* fix(docker): extend device permission loop to rkmpp and wsl accelerator nodes
2026-09-26 20:55:57 +08:00
yiling 29cd5d807c test(scripts): find release by version in changelog integration test 2026-09-24 20:34:53 +08:00
Yiling-J f579947770 feat(docs): add changelog section to mintlify documentation (#422) 2026-09-22 21:02:27 +08:00
Yiling-J 3a509537fd feat(changelog): add CHANGELOG.md, release extraction script, and developer guidelines (#414)
* feat(changelog): add CHANGELOG.md, release extraction script, and developer guidelines

* docs(agents): simplify changelog guidelines for brevity
2026-09-18 11:57:25 +08:00
yiling 16d136b45a refactor(core): replace generated provider files with pi-ai dynamic methods 2026-08-03 10:00:21 +08:00
yiling c364e0e695 fix(core): guard top-level sync execution and add remote fallback for model data loading 2026-07-22 12:19:37 +08:00
yiling a4b6b0b775 refactor(core): split generated providers into per-provider files 2026-07-22 12:04:07 +08:00
yiling 3274d19574 fix(transcode): mark pdfkit as external and extract for docker runtime 2026-07-20 16:12:59 +08:00
yiling 8d19b578e7 fix(build): resolve code-splitting chunk import issue on backend and frontend 2026-07-09 15:00:23 +08:00
yiling 661e836f06 fix(docker): support non-root/restricted container environments in entrypoint
Checks if the container is running as root (UID 0) before executing
chown and gosu. If running directly as a non-root user (e.g. in hardened
environments like Kubernetes runAsNonRoot or via docker run --user),
it directly execs the command, preventing EACCES and permission errors.
2026-06-23 09:56:31 +08:00
yiling a390643458 feat(docker): add root entrypoint with privilege dropping for data volumes
This updates the Dockerfile and adds a scripts/docker-entrypoint.sh script.
Under native Linux (like Debian), when docker-compose mounts a host folder
as a volume (e.g. `./data:/app/data`), and the folder doesn't exist, the
docker daemon creates the folder as root-owned (`root:root`).

Because the container runs as a non-root user `bun`, it gets permission
denied (`EACCES`) when trying to write to the data directory.

To resolve this in a beginner-friendly way without manual host-side
permissions updates:
1. The runner stage now installs `gosu`.
2. At the end of the Dockerfile, the user switches back to `root` so the
   container launches as root.
3. On launch, the entrypoint script executes. Since it runs as root, it
   automatically changes ownership of `/app/data` to `bun:bun`.
4. The entrypoint script then uses `gosu` to drop privileges and launch the
   actual server processes as the non-root `bun` user.
2026-06-23 09:46:56 +08:00
yiling 5f1e0c1823 chore(cli): rename npm package to @shumai-one/shumai-cli 2026-06-22 14:16:47 +08:00
yiling 1c994adce9 build(cli): bundle hono and ulid directly into the CLI script for zero dependency distribution 2026-06-21 19:15:41 +08:00
yiling afd55eca85 chore(cli): add @shumai-one/cli to local release testing script 2026-06-21 19:06:12 +08:00
yiling 29e3d3ae03 refactor(cli): rename shumai-cli folder to cli and integrate packaging script 2026-06-21 19:00:38 +08:00
yiling efa0e75a06 fix(release): bundle zod for shumai webapp and keep it external for workers 2026-06-20 14:39:43 +08:00
yiling 93b4656939 feat(cli): add daemon mode and stop, logs, restart subcommands 2026-06-20 12:37:49 +08:00
yiling 982bca9383 fix(release): resolve correct workspace dependency versions and refine CI smoke tests 2026-06-20 11:43:44 +08:00
yiling d1061a1023 fix(release): exclude zod from inlined dependencies to resolve circular evaluation error 2026-06-20 11:30:22 +08:00
yiling 66d07d1fb0 feat(webui): show app version in avatar menu using local version file 2026-06-19 22:57:26 +08:00
yiling 5542538a64 feat(webui): show app version in avatar menu 2026-06-19 22:34:12 +08:00
yiling e5084cc8b3 fix(build): propagate termination signals to active child process in wrappers 2026-06-13 12:51:30 +08:00
yiling d230a1fd4a feat(npm): add dotenv loading and fix variable interpolation in wrapper 2026-06-13 10:21:36 +08:00
yiling 210ed5f95a feat(npm): auto deploy database migrations on package start 2026-06-13 09:24:54 +08:00
yiling dc63ab3eaf fix(npm): resolve binary output path mapping in build-npm script 2026-06-12 21:54:15 +08:00
yiling d06dfc0ab3 perf(npm): optimize platform binary compilation using dummy runner 2026-06-12 21:50:18 +08:00
yiling 91fedf1477 fix(cli): resolve all external module imports in global CLI package 2026-06-12 21:08:40 +08:00
yiling 1fee0f5a97 fix(cli): resolve platform binary path resolution under nested node_modules 2026-06-12 20:34:07 +08:00
yiling 12d8681cd1 fix(npm): add repository metadata for provenance validation 2026-06-12 18:47:28 +08:00
yiling 81ccc803db fix(webui): update members dialog UI and disable auto-focus 2026-06-11 10:54:41 +08:00
yiling ca4b661891 feat(npm): pin runtime dependencies in build-npm script 2026-06-11 10:36:47 +08:00
yiling 39d34b9678 fix(lint): remove unused variable in extract-runtime-deps script 2026-06-11 10:13:53 +08:00
yiling 5facd98b0d feat(docker): pin runtime dependencies using dynamic package.json 2026-06-11 10:10:55 +08:00
yiling e790df2812 fix: resolve lint, typecheck, and tsconfig deprecation errors 2026-06-07 11:10:27 +08:00
yiling 53bed7b26d feat(npm): add win-arm64 compilation target and fix win32 wrapper path mapping 2026-06-05 17:33:18 +08:00
yiling 06b276f0a7 feat(npm): add cross-platform packaging, publishing, and release automation 2026-06-05 17:01:25 +08:00
yiling 042188cc74 feat(publish): compile and package workers for npm and isolate workflow imports 2026-06-05 14:34:22 +08:00
yiling dfdf095be7 feat(publish): compile app to binary and package for npm 2026-06-05 14:16:37 +08:00
yiling 1f341c15d0 refactor(monorepo): decouple api and webui into workspaces packages 2026-06-01 23:35:00 +08:00
yiling 5173bb2a01 feat: remove all ai transcription related code and schema 2026-05-28 14:23:02 +08:00
yiling 9cad96210c initial commit 2026-05-19 16:05:09 +08:00