Commit Graph
95 Commits
Author SHA1 Message Date
yiling 0c0b887b8d Release v0.6.4 2026-11-05 22:45:25 +08:00
Yiling-J ba3c6dbfee feat(storage): add archive storage support for assets (#514)
* feat(storage): add archive storage support for assets

* fix(webui): show archive icon in front of file name in breadcrumb bar

* fix(webui): display archive icon in breadcrumbs when restoring

* refactor(transcode): simplify restore asset workflow to single while loop

* docs(configuration): add archive storage documentation and readme entries

* fix(webui): show archive icon in front of version names and prevent version card overflow

* fix(webui): fix overflow in manage versions dialog scroll area

* fix(webui): prevent manage versions dialog overflow on long file names

* docs(configuration): fix archive storage environment variables and defaults

* fix(storage): address code review findings for archive storage
2026-11-03 20:31:56 +08:00
yiling 49a2d7c380 Release v0.6.3 2026-11-01 19:52:23 +08:00
Yiling-J ff3d71ed3d feat(agent)!: suppress success comments for autofill and embedding agents (#510)
* feat(agent)!: suppress success comments for autofill and embedding agents

* fix(agent): ensure createCommentActivity runs on workflow cancellation
2026-10-31 12:02:30 +08:00
yiling f0bdccf713 Release v0.6.2 2026-10-29 20:29:50 +08:00
Yiling-J 155fb95a0e feat(asset-read): rewrite read_asset as a reader registry and Operations (#498)
Replace the monolithic agent read_asset tool with a reader registry in
@shumai/core/src/asset-read (model3d, image, video, document) behind a
shared pipeline that handles access checks, version stacks, annotations,
render tasks, and waiting.

- Add read_asset and get_asset_render Operations (agent surface only).
  Renders that outlast the caller's wait budget return "pending" with the
  workflow task ID; the read request is stored on the task payload so
  get_asset_render works across restarts and instances (requester only).
- Add workflowService.waitFor and presign expiry support.
- Keep agent output compatible via an agent tool extension (s3KeyOnly,
  image blocks, session image details); autofill keeps its team-scoped tool.
- Cap video frames at 20 per call; add the get_asset_render agent toggle.
2026-10-27 08:09:10 +08:00
yiling c2af7494ec Release v0.6.1 2026-10-24 13:21:50 +08:00
Yiling-J d87fe43b0f feat(dashboard): add workflow tasks monitoring and stopping to team dashboard (#489)
* feat(dashboard): add workflow tasks monitoring and stopping to team dashboard

* fix(asset): retain workflow tasks on asset purge and mark active tasks as failed

* style(webui): add edge padding to audit logs and workflow tasks tables

* fix(workflow): atomically update asset status to failed when task is stopped

* fix(workflow): propagate cancellation errors, guard concurrent task stop, and localize UI

* docs(changelog): keep only added log for workflow tasks dashboard

* test(provider): mock upstream catalog api calls using saved fixtures

* feat(dashboard): show file name, size, and project name in workflow tasks list
2026-10-23 16:44:52 +08:00
yiling e05a23bcc2 Release v0.6.0 2026-10-20 20:02:34 +08:00
Yiling-J 176b7f0eab feat(workflow): migrate workflow engine from temporal to dbos (#481)
* feat(workflow): migrate workflow engine from temporal to dbos

Replace Temporal and LocalExecutor with DBOS Transact (@dbos-inc/dbos-sdk) as the unified workflow engine. Background tasks, media transcoding, and AI agent workflows now execute with durable steps and task queues directly against PostgreSQL, eliminating the need for an external Temporal cluster and separate polling executors.

* fix(workflow): submit tasks via DBOSClient to allow enqueueing from steps

Enqueuing workflows using DBOS.enqueueWorkflowWithOptions threw a
DBOSInvalidWorkflowTransitionError when triggered from within a running
step or activity (e.g. creating embedding or autofill tasks during
video transcoding).

Use DBOSClient to submit and cancel workflow tasks directly against the
system database, detaching execution from the ambient workflow/step
context.

* feat(web): support SHUMAI_WORKERS environment variable

* fix(workflow): ensure CLI tasks are enqueued to DBOS and run DBOS migrations in global setup

- Initialize workflows and wait for pending DBOS submissions in shumai regenerate-embeddings CLI
- Remove no-op submit bypass and enqueue tasks via DBOSClient in producer and CLI contexts
- Add DBOS migration runner in test global setup to avoid concurrent migration deadlocks across test forks
- Update changelog and add regression tests

* fix(workflow): ensure unique per-instance executor identity for worker replicas

- Derive unique per-instance DBOS executor IDs across worker replicas

- Support SHUMAI_WORKER_ID, DBOS__VMID, and POD_NAME (Kubernetes Downward API)

- Disambiguate multi-replica bare metal/container deployments with host, PID, and instance suffix

- Prevent startup recovery from re-enqueueing workflows actively executed by other live workers

- Add multi-instance startup recovery tests and update deployment documentation

* fix(workflow): only register listened queues and update concurrency on change

- Only register queues that the executor instance actively listens to

- Use always_update conflict resolution so environment variable changes take effect on restart

- Prevent dedicated workers from overwriting unmanaged queue concurrency configurations

- Add tests verifying concurrency updates and queue isolation across worker roles

* refactor(workflow): simplify activities to use defineStep and direct calls (#482)

* refactor(workflow): simplify activities to use defineStep and direct calls

- Wrap activities using defineStep from @shumai/workflow-core
- Remove obsolete getActivities, registerActivities, dynamic activity proxies, and workerQueue arguments
- Remove executor.ts and registry.ts, consolidating workflow registration into workflow.ts
- Call activities directly across transcode and agent workflows
- Update all corresponding workflow tests and developer guide documentation

* fix(workflow): properly detect DBOSWorkflowCancelledError in runOnCancel fallback

- DBOSWorkflowCancelledError in DBOS SDK does not set name, inheriting 'Error'
- Implement isDbosWorkflowCancelledError checking instanceof, constructor.name, and dbosErrorCode
- Update unit tests to verify real DBOS SDK cancellation error instances

* docs(workflow): update documentation and readme for dbos migration

- Rename docs/configuration/temporal.mdx to docs/configuration/distributed.mdx
- Document DBOS-based distributed worker architecture and task queues
- Remove obsolete docker-compose/temporal/
- Update environment variables documentation in docs/configuration/env-variables.mdx
- Update references in README.md, README_CN.md, docs/docs.json, and related guides

* feat(deploy): implement KEDA ScaledJob and scale-to-zero On-Demand transcoding (#483)

* feat(deploy): add shumai-transcode docker image and k8s autoscaling example with garage s3

* docs(k8s): document port-forwarding and in-cluster s3 domain mapping

* feat(storage): support public s3 endpoint for browser uploads and update k8s example

* fix(k8s): configure s3 bucket cors rules in garage initialization

* fix(k8s): increase transcode worker memory limit to 4Gi for 4K video encoding

* feat(deploy): add AWS EKS deployment with Graviton3 Spot nodegroups and autoscaling

* docs(deploy): redesign EKS documentation, rename k8s to eks, and add scalable deployment feature

* docs: rephrase DBOS feature bullet to Durable Workflows via DBOS

* docs(eks): upgrade default instances to Graviton5 c9g.large and c9g.xlarge

* feat(eks): use On-Demand for general nodegroup and add spot capacity diversification

* docs(eks): fix Architecture diagram to locate all transcode worker pods in Transcode Compute Pool

* feat(deploy): implement KEDA ScaledJob and scale-to-zero On-Demand transcoding

* feat(transcode): support idle-timeout auto-drain for KEDA scaled jobs

* fix(deploy): fix EKS deployment manifests and documentation based on cluster verification
2026-10-19 15:35:08 +08:00
yiling f7ed3bafdf Release v0.5.3 2026-10-16 22:50:47 +08:00
Yiling-J 6cc90bf973 feat(search): generate query embeddings directly for semantic search (#476)
* feat(search): generate query embeddings directly for semantic search

* fix(search): resolve embedding provider from env, redact query logs, and preserve case in cache
2026-10-15 20:49:45 +08:00
yiling c302c162a0 Release v0.5.2 2026-10-13 21:03:51 +08:00
Yiling-J 43a2eced75 feat(transcode): add 3D asset support and turntable viewer (#454)
* feat(transcode): add 3D asset support and turntable viewer

* perf(turntable): stream file via Bun.file and increase upload timeout

* feat(turntable): increase upload timeout to 15m to support 5GB uploads

* fix(webui): align 3D model viewer layout and full-width control bar with video viewer

* feat(webui): unify 3D viewer bottom bar and redesign cyclic ruler slider

- Place RulerSlider in single row with playback and zoom controls, centered with max 320px width
- Flank RulerSlider with minus and plus buttons to step angle by 15°
- Redesign RulerSlider with fixed center indicator and moving cyclic track with continuous 360° wrapping
- Render major ticks longer at 60° intervals without numeric text labels on track
- Display floating degree tooltip above center indicator only during dragging

* fix(webui): prevent RulerSlider from collapsing to 0px width in control bar

- Use flex layout with flex-1 on left and right clusters to preserve center cluster width
- Enforce min-w-[140px] and w-full on RulerSlider
- Fix tooltip positioning and remove outer overflow-hidden

* feat(webui): snap RulerSlider track directly to discrete 15° ticks during drag

* fix(webui): prevent ruler snap-back caused by floating point second truncation

- Use Math.round instead of Math.floor in secondToDegree to avoid 0.999996 dropping to previous frame
- Ensure final dragDegree is committed on pointer up

* fix(webui): fix arrow key frame rounding and duplicate event bubbling

- Use Math.round instead of Math.floor for keyboard arrow frame navigation in ModelViewer
- Add e.stopPropagation() to RulerSlider keyboard events to prevent duplicate bubbling to window listener

* feat(webui): redesign RulerSlider with encapsulated step buttons and capsule pill track

* feat(webui): make RulerSlider responsive on mobile and remove bottom bar reset button

* fix(webui): load 3d video directly without poster and invert drag rotation to counterclockwise

* fix(webui): fix 3d asset viewer frame seeking and drag jitter

* feat(transcode): show 3D asset poster preview immediately after render start

* fix(viewer): synchronize 3D turntable playback via frame callback to prevent skipped degrees

* fix(viewer): handle touchpad zoom gestures and prevent page zoom in 3D viewer

* fix(viewer): eliminate 3D model rotation jitter at 345 degrees

* fix(share): watermark 3D asset turntable video on share links

* fix(share): preserve video transcode resolution on watermarked share links

* fix(viewer): download original 3D model file instead of rendered turntable MP4
2026-10-11 21:32:28 +08:00
yiling f8beb195dd Release v0.5.1 2026-10-06 22:29:50 +08:00
yiling e61641d7ec fix(workflow): increase activity startToCloseTimeout to 120 minutes 2026-10-04 10:22:31 +08:00
yiling 013c4a8156 Release v0.5.0 2026-10-02 20:28:37 +08:00
Yiling-J 20b5ebeefd fix(asset): mark asset as failed and display failure state in webui when transcoding fails (#445)
* fix(asset): mark asset as failed and display failure state in webui when transcoding fails

* fix(transcode): do not mark asset as failed if media processing already succeeded

* test(workflow): add e2e test for video transcode failure status transition

* fix(webui): update failed file card badge to top-right Transcoding Failed label with error tooltip
2026-10-02 20:09:32 +08:00
yiling f2b0415580 Release v0.4.9 2026-09-28 20:17:50 +08:00
yiling f6e27095a5 Release v0.4.8 2026-09-24 20:26:23 +08:00
yiling 657cd761bc Release v0.4.7 2026-09-19 16:01:20 +08:00
Yiling-J b52b2ce7d8 fix(transcode): preserve trashed status during background jobs and handle permanent purge (#416)
* fix(transcode): preserve trashed status during background jobs and handle permanent purge

* fix(workflow): clear heartbeat and add info logs on task cancellation

* fix(transcode): terminate media processes and guard task activities on cancellation

- Pass AbortSignal to ffmpeg and pdftoppm child processes in transcodeService to terminate processes on task cancellation
- Add local task AbortController context and cancellation guards in executeActivity to prevent subsequent workflow activities
- Forward taskId and cancellation signals in transcode, screenshot, and PDF activities
- Use updateMany instead of update in task activities to eliminate Prisma P2025 errors on purged task records

* fix(workflow): allow cleanup activities in finally blocks during task cancellation

Remove blanket cancellation check in executeActivity so that workflows can execute cleanup and failure handling activities in catch and finally blocks upon cancellation, while relying on AbortSignal in active activities to abort execution.

* docs(changelog): improve user friendliness of changelog entries and formatting example

* docs(agents): update changelog guidelines to emphasize end-user friendly descriptions

* fix(transcode): handle queued cancellation and delete races in media workflow
2026-09-19 10:40:49 +08:00
yiling 76ffb51bb2 Release v0.4.6 2026-09-17 21:58:10 +08:00
yiling ede52e5009 Release v0.4.5 2026-09-13 21:50:44 +08:00
yiling 70b81355c2 Release v0.4.4 2026-09-09 22:07:40 +08:00
yiling 62e86b703e Release v0.4.3 2026-09-06 22:22:14 +08:00
Yiling-J 576d8ebe64 chore(deps): upgrade bun to 1.4.2 and update packages (#384)
* chore(deps): upgrade bun to 1.4.2 and update packages

- Upgrade Bun to 1.4.2 in Dockerfile, CI workflows, and release workflows.
- Update dependencies across workspace with Bun 1.4.2.
- Preserve @earendil-works/pi-agent-core at 0.82.1.
- Retain stable compatible versions for database adapters and pdfjs-dist.
- Add explicit Buffer type annotation in transcode service.

* chore(deps): upgrade prisma to 7.10.0, pg to 8.23.0, and pdfjs-dist to 6.3.289

* chore(deps): revert dotenv-expand to 13.0.0
2026-09-06 11:31:05 +08:00
yiling 52330869a2 Release v0.4.2 2026-09-03 21:49:37 +08:00
yiling 6e9cf8dfb6 Release v0.4.1 2026-08-30 20:26:57 +08:00
Yiling-J 083753a96d refactor(agent): remove executeAgentToolWorkflow and invoke core services directly (#362)
* refactor(agent): remove executeAgentToolWorkflow and invoke core services directly

* fix(core): ensure creation atomicity, strict assetType filtering, and id-desc listing
2026-08-29 15:19:08 +08:00
yiling 4b62c31f29 Release v0.4.0 2026-08-28 20:40:33 +08:00
yiling 7b1428bf51 Release v0.3.11 2026-08-25 22:05:47 +08:00
yiling 6a20e06982 Release v0.3.10 2026-08-23 21:19:45 +08:00
Yiling-J 37e086d822 fix(webui): sidebar selection styling, kanban active state, pointer cursors, and owner dashboard (#344)
* fix(webui): update sidebar selection styling, kanban active state, and pointer cursors

* feat(webui): move dashboard to left sidebar and prevent mount animation

* fix(e2e): update create-project test to click Home button in sidebar
2026-08-23 16:39:56 +08:00
yiling 4c0fd29b7f Release v0.3.9 2026-08-18 18:42:59 +08:00
yiling 6d4f62d6f6 Release v0.3.8 2026-08-15 20:35:30 +08:00
yiling 0b84402c83 Release v0.3.7 2026-08-14 14:47:18 +08:00
yiling 96493a5493 Release v0.3.6 2026-08-12 14:56:54 +08:00
yiling 49ef48a74a Release v0.3.5 2026-08-09 20:58:35 +08:00
yiling 283d28bed9 Release v0.3.4 2026-08-08 20:15:25 +08:00
Yiling-J 03aea2cb36 feat(sharelink): add sharelink watermark support (#270)
* feat(sharelink): add sharelink watermark support

* fix(sharelink): harden watermark feature against races, crashes, and tenant leaks

Addresses review findings on the sharelink watermark feature:

- fix(watermark): read back sharelink via transaction client so the
  updateShareLinkWatermark response reflects the just-committed state
  (global-client reads on a pooled connection miss uncommitted changes)
- fix(watermark): use prisma upsert on the config hash instead of
  find-then-create to avoid P2002 races on concurrent enables
- fix(watermark): resolve nested symlinks when walking the share tree and
  batch the walk level-by-level instead of one query per node
- fix(watermark): keep watermark tasks in flight from being purged by
  orphan-config GC (avoids FK violations in completing activities)
- fix(transcode): re-transcode stale 'processing' watermark rows instead
  of silently skipping (crash recovery); downscale embedded block images;
  use pino structured logging; drop unused any casts
- fix(api): scope watermark-template list/create to the caller's teams,
  reject teamless template creation, and make teamless templates
  read-only; validate watermark config required when enabling (400);
  map sharelink-not-found to 404
- fix(api): resolve symlink target ids once in public share children
  endpoint instead of twice per item
- refactor(db): dedicated WatermarkFileStatus enum (migration included)
- test: activity unit tests (init crash recovery, image + video transcode
  paths), concurrent upsert, nested-symlink resolution, API scoping and
  validation, e2e video watermark path

* refactor(transcode): move sharp usage out of watermark activity into transcodeService

The transcode package does not declare sharp as a dependency (only
@shumai/core does), so importing it directly in the watermark activity
broke CI typecheck for the transcode package.

- add transcodeService.renderSvgToPng, downscaleImageToPng and
  compositeOverlayToWebpFile encapsulating the sharp logic
- watermark activity now calls these methods instead of importing sharp
- activity tests mock the new methods (no sharp/real fixture needed)
- add unit tests for the new transcodeService helpers

* feat(watermark): restrict watermark transcode to existing image and video transcodes

* refactor(transcode): reuse transcodeVideo in watermark activity

* refactor(watermark): use Bun.hash instead of node crypto for watermark config hash

* refactor(watermark): move computeWatermarkConfigHash from dtos to core watermark-hash module

* refactor(api): delegate watermark DB queries from public-share route to watermarkService

* refactor(share): replace dynamic watermarkService import with static import

* feat(watermark): add waitForWatermarkFileActivity for in-flight watermark transcoding tasks

* refactor(asset): replace dynamic watermarkService import with static import

* test(watermark): harden purgeOrphanWatermarkConfigs coverage

Replace the single happy-path GC test with a dedicated suite covering:

- orphan config with proxies: purged, watermarkFiles removed, all four
  proxy keys deleted from S3 while the original key is preserved
- config still referenced by a share link is not purged
- config with pending/processing transcode_watermark tasks is not purged
  (in-flight guard), but completed tasks do not block purging
- pending tasks of other types (transcode_video) do not protect a config
- an S3 delete failure is non-fatal: remaining keys still deleted and the
  config is still purged
- mixed run: only the orphan config is purged while a referenced one
  survives (exact count assertion)

* feat(webui): add sharelink watermark editor and sidebar controls

* style(webui): enlarge watermark editor dialog and allocate 2/3 width to preview column

* fix(webui): override sm:max-w-lg in DialogContent to allow full 1400px width

* fix(webui): stop block click propagation and remove preview mode toggles

* fix(webui): store watermark block size as fraction of canvas width

The editor stored block.size in pixels (slider 10-120px, preview
fontSize: `${block.size}px`) while the SVG generator
(generateWatermarkSvg) interprets size as a fraction (0..1) of canvas
width. A 36px block therefore rendered as a ~36x-canvas glyph covering
the entire image.

- Keep backend fraction semantics (canvasWidth * size) so watermarks
  scale proportionally with output resolution across proxies
- Editor: defaults (text 5%, image 15%), slider ranges (1-12% / 2-50%)
  and number input now work in percent, storing a fraction
- Preview: scales blocks by the measured canvas width so WYSIWYG
- DTO schema: size constrained to (0..1) to enforce the contract
- Update DTO/editor tests to fraction sizes

* feat(webui): add single-row aspect ratio selector and 5 predefined background colors to watermark editor preview

* feat(webui): update aspect ratio to multi-toggle group and enable dot-grid pattern for all background colors

* fix(webui): update background swatches to distinct colors and share PRESET_BACKGROUND_OPTIONS array

* fix(webui): hide redundant absolute close button on watermark editor dialog header

* fix(transcode): skip raw video transcode when generating watermark proxies

The raw marker entry (isRaw: true) in videoTranscodes points at the
original source file and is only used to offer the 'Original' playback
option. It is not a transcoded proxy, so watermarking it produced a
redundant full-resolution transcode (the most expensive pass) plus an
unneeded S3 upload.

Skip entries with isRaw: true in transcodeWatermarkMediaActivity, and
cover the behavior with a unit regression test plus an e2e assertion
that exactly one watermark proxy is produced for a video asset.

* feat(webui): replace watermark image asset ID with 1MB file uploader

* fix(webui): correct watermark editor preview block sizing

The preview canvas width was measured in a useEffect([open]) that ran
before the Radix Dialog had mounted its content, so canvasRef.current was
still null, canvasWidthPx stayed 0, and every block silently fell back to
the hardcoded 640px width (block.size * (canvasWidthPx || 640)). A 100%
size therefore rendered at ~640px on an ~876px canvas (~72%), making the
preview smaller than the real watermarked output for both image and text
blocks.

- Measure the canvas via a ref callback (fires exactly when the node
  mounts, regardless of dialog mount timing) and keep a ResizeObserver
  attached for aspect-ratio/window changes.
- Use clientWidth (padding-box width) instead of getBoundingClientRect(),
  which includes the dialog's zoom-in-95 animation transform.
- Drop max-w-full/max-h-full on the preview <img> so logos scale up to
  the block size like the backend SVG rendering does, instead of being
  capped at their natural pixel size.

* fix(webui): keep watermark editor preview aspect ratio and restore config scroll

- Compute the preview canvas size with a JS contain-fit (ResizeObserver on the
  frame) instead of relying on CSS aspect-ratio + w-full/max-h-full, which
  broke the selected ratio on wide-and-short (or tall-and-narrow) windows.
- Add min-h-0 to the config ScrollArea so it shrinks to the column's remaining
  height instead of growing to its content height, restoring the scroll range.

* fix(webui): remove toast notification on template load

* fix(share): serve per-file watermark proxies in public share APIs

Stop gating watermark replacement on the share-link-wide watermarkStatus.
Public share endpoints now resolve each asset's own completed watermark
proxy and:

- replace videoTranscodes/imageTranscodes with the watermarked entries
  when the watermark file is completed;
- empty those arrays when no completed proxy exists yet (still
  transcoding or failed) so original, unwatermarked media is never
  exposed while a newly added file is being processed;
- leave non-video/image media (pdf, audio, ...) untouched, since they
  never receive watermark transcodes.

This removes the previous behavior where adding a file to a ready
watermark share flipped the whole share link to processing and served
original media for every file until the new transcode finished.

Adds API tests for detail/children endpoints (ready, in-flight, pdf, and
watermark-disabled cases) and a workflow E2E test for adding an asset to
an already-ready watermark sharelink.
2026-08-07 21:36:20 +08:00
yiling 0a31659588 Release v0.3.3 2026-08-05 19:41:02 +08:00
yiling 8e223da8e3 Release v0.3.2 2026-07-30 20:54:48 +08:00
yiling 361e4f32be Release v0.3.1 2026-07-30 11:12:32 +08:00
yiling 668871b730 fix(workflow): map sizeByte BigInt to Number in getAssetActivity for Temporal serialization 2026-07-29 22:31:25 +08:00
yiling d0d29b34ed Release v0.3.0 2026-07-28 18:56:55 +08:00
yiling 72654e84b2 Release v0.2.3 2026-07-25 11:49:11 +08:00
yiling dc8c898dae Release v0.2.2 2026-07-23 20:35:22 +08:00
yiling 6ab650bc80 Release v0.2.1 2026-07-22 20:01:30 +08:00
yiling b6afcc135e Release v0.2.0 2026-07-21 21:30:00 +08:00