Commit Graph
29 Commits
Author SHA1 Message Date
Yiling-J 35a0e82284 refactor(agent)!: migrate to Pi Durable and embed the agent queue (#525)
* refactor(agent)!: migrate sessions to Pi Durable

Replace patched AgentHarness storage with per-session SQLite and PostgreSQL commit mirroring. Migrate legacy history automatically and run agent queues with the main server.

BREAKING CHANGE: Remove standalone agent workers and return native Pi entry records from chat history APIs.

* fix(agent): constrain session log dialog scrolling

Keep the header visible and wrap long output while log entries and expanded records scroll within the dialog. Cover desktop and narrow viewports through the dashboard Sessions flow.

* fix(agent): hide entry IDs from session log card headers

Retain internal entry IDs in expanded raw records for debugging. Verify that card headers omit the IDs while raw JSON preserves them.

* test(e2e): handle browser wheel limits in session log checks

Verify that a normal wheel event moves the viewport before explicitly scrolling to the final record. Keep the dialog bounds, fixed header, wrapping, and expanded-record assertions.

* fix(agent): select model providers without inheriting unrelated auth

* fix(webui): prevent resize feedback when toggling agent panel

* test(agent): reproduce concurrent comment session races

* fix(agent): serialize comment sessions and deduplicate history

* docs(changelog): consolidate Pi Durable changes into one entry
2026-11-11 16:05:18 +08:00
Yiling-J f304caea00 fix(auth,web): gracefully degrade MCP on remote HTTP and cache production static assets (#519)
* fix: gracefully handle remote HTTP URLs for MCP and cache production static assets

* test(mcp): add remote http mcp integration test and fix static etag normalization and fallback error
2026-11-05 22:21:11 +08:00
Yiling-J 2ecbd49d18 feat(storage): add team auto-archive assets policy and background job (#516)
* feat(storage): add team auto-archive assets policy and background job

* fix(storage): address code review feedback on lock safety, starvation, and permissions

* docs: update readme and expand archive storage documentation with auto-archive details

* feat(storage): log info message on auto-archive job start and finish with archived count

* style(storage): stack age threshold and grace period vertically with updated grace period description

* fix(storage): only log auto-archive info messages when candidate assets are found to process
2026-11-04 14:40:44 +08:00
Yiling-J ba3c6dbfee feat(storage): add archive storage support for assets (#514)
* feat(storage): add archive storage support for assets

* fix(webui): show archive icon in front of file name in breadcrumb bar

* fix(webui): display archive icon in breadcrumbs when restoring

* refactor(transcode): simplify restore asset workflow to single while loop

* docs(configuration): add archive storage documentation and readme entries

* fix(webui): show archive icon in front of version names and prevent version card overflow

* fix(webui): fix overflow in manage versions dialog scroll area

* fix(webui): prevent manage versions dialog overflow on long file names

* docs(configuration): fix archive storage environment variables and defaults

* fix(storage): address code review findings for archive storage
2026-11-03 20:31:56 +08:00
Yiling-J baf30730af feat(mcp): add OAuth 2.1 support (#505)
* feat(mcp): add OAuth 2.1 support with dynamic client registration

* feat(api): alias bare oauth-authorization-server discovery endpoint

* fix(auth): preserve user session on OAuth sign-in redirect to consent
2026-10-29 15:39:37 +08:00
Yiling-J b3cae4f2b0 feat: move MCP to /mcp and add teammate operations (#500)
* feat(operations): move MCP to /mcp and add search, comment, metadata, version, and upload operations

- Serve the MCP server at /mcp instead of /api/mcp
- Add search_assets, copy_assets, get_folder_instructions, list_comments,
  create_comment, resolve_comment, list_metadata_fields,
  update_asset_metadata, create_version_stack, list_versions, and
  remove_version on the agent, MCP, and CLI
- Add create_file and create_version (agent, MCP) with Creation Context
  metadata, and complete_upload (MCP) for presigned two-step uploads
- Replace the native agent create_file/create_version tools with
  Operation-backed ones
- Route the comment-complete, metadata, copy, version stack, and
  AGENTS.md read REST endpoints through their Operations

* docs(readme): add built-in MCP server and CLI to features list
2026-10-27 14:37:26 +08:00
Yiling-J 176b7f0eab feat(workflow): migrate workflow engine from temporal to dbos (#481)
* feat(workflow): migrate workflow engine from temporal to dbos

Replace Temporal and LocalExecutor with DBOS Transact (@dbos-inc/dbos-sdk) as the unified workflow engine. Background tasks, media transcoding, and AI agent workflows now execute with durable steps and task queues directly against PostgreSQL, eliminating the need for an external Temporal cluster and separate polling executors.

* fix(workflow): submit tasks via DBOSClient to allow enqueueing from steps

Enqueuing workflows using DBOS.enqueueWorkflowWithOptions threw a
DBOSInvalidWorkflowTransitionError when triggered from within a running
step or activity (e.g. creating embedding or autofill tasks during
video transcoding).

Use DBOSClient to submit and cancel workflow tasks directly against the
system database, detaching execution from the ambient workflow/step
context.

* feat(web): support SHUMAI_WORKERS environment variable

* fix(workflow): ensure CLI tasks are enqueued to DBOS and run DBOS migrations in global setup

- Initialize workflows and wait for pending DBOS submissions in shumai regenerate-embeddings CLI
- Remove no-op submit bypass and enqueue tasks via DBOSClient in producer and CLI contexts
- Add DBOS migration runner in test global setup to avoid concurrent migration deadlocks across test forks
- Update changelog and add regression tests

* fix(workflow): ensure unique per-instance executor identity for worker replicas

- Derive unique per-instance DBOS executor IDs across worker replicas

- Support SHUMAI_WORKER_ID, DBOS__VMID, and POD_NAME (Kubernetes Downward API)

- Disambiguate multi-replica bare metal/container deployments with host, PID, and instance suffix

- Prevent startup recovery from re-enqueueing workflows actively executed by other live workers

- Add multi-instance startup recovery tests and update deployment documentation

* fix(workflow): only register listened queues and update concurrency on change

- Only register queues that the executor instance actively listens to

- Use always_update conflict resolution so environment variable changes take effect on restart

- Prevent dedicated workers from overwriting unmanaged queue concurrency configurations

- Add tests verifying concurrency updates and queue isolation across worker roles

* refactor(workflow): simplify activities to use defineStep and direct calls (#482)

* refactor(workflow): simplify activities to use defineStep and direct calls

- Wrap activities using defineStep from @shumai/workflow-core
- Remove obsolete getActivities, registerActivities, dynamic activity proxies, and workerQueue arguments
- Remove executor.ts and registry.ts, consolidating workflow registration into workflow.ts
- Call activities directly across transcode and agent workflows
- Update all corresponding workflow tests and developer guide documentation

* fix(workflow): properly detect DBOSWorkflowCancelledError in runOnCancel fallback

- DBOSWorkflowCancelledError in DBOS SDK does not set name, inheriting 'Error'
- Implement isDbosWorkflowCancelledError checking instanceof, constructor.name, and dbosErrorCode
- Update unit tests to verify real DBOS SDK cancellation error instances

* docs(workflow): update documentation and readme for dbos migration

- Rename docs/configuration/temporal.mdx to docs/configuration/distributed.mdx
- Document DBOS-based distributed worker architecture and task queues
- Remove obsolete docker-compose/temporal/
- Update environment variables documentation in docs/configuration/env-variables.mdx
- Update references in README.md, README_CN.md, docs/docs.json, and related guides

* feat(deploy): implement KEDA ScaledJob and scale-to-zero On-Demand transcoding (#483)

* feat(deploy): add shumai-transcode docker image and k8s autoscaling example with garage s3

* docs(k8s): document port-forwarding and in-cluster s3 domain mapping

* feat(storage): support public s3 endpoint for browser uploads and update k8s example

* fix(k8s): configure s3 bucket cors rules in garage initialization

* fix(k8s): increase transcode worker memory limit to 4Gi for 4K video encoding

* feat(deploy): add AWS EKS deployment with Graviton3 Spot nodegroups and autoscaling

* docs(deploy): redesign EKS documentation, rename k8s to eks, and add scalable deployment feature

* docs: rephrase DBOS feature bullet to Durable Workflows via DBOS

* docs(eks): upgrade default instances to Graviton5 c9g.large and c9g.xlarge

* feat(eks): use On-Demand for general nodegroup and add spot capacity diversification

* docs(eks): fix Architecture diagram to locate all transcode worker pods in Transcode Compute Pool

* feat(deploy): implement KEDA ScaledJob and scale-to-zero On-Demand transcoding

* feat(transcode): support idle-timeout auto-drain for KEDA scaled jobs

* fix(deploy): fix EKS deployment manifests and documentation based on cluster verification
2026-10-19 15:35:08 +08:00
Yiling-J 7c297895e5 feat(embedding): add compose tiers, docs, and regenerate-embeddings CLI (#480)
* feat(embedding): update compose tiers, docs, and add regenerate-embeddings CLI

- Add Lite, Embedding, and Full Docker Compose configurations (and Aliyun mirrors)
- Set CONCURRENCY_AGENT to 1 for compose files containing embedding-server
- Update README and README_CN with compose options, recommended RAM, and semantic search links
- Add LiteRT-LM Embedding Server setup guide in docs/using-shumai/embedding-server.mdx
- Add shumai regenerate-embeddings CLI command and EmbeddingService to regenerate embeddings across teams
- Add comprehensive test coverage for EmbeddingService

* fix(docker): remove basic auth for turntable-renderer in full compose

* fix(agent): lock and disable embedding agent when ENABLE_EMBEDDING_AGENT is false

Ensure that when ENABLE_EMBEDDING_AGENT is set to 'false', existing embedding agents are automatically disabled on startup, marked as read-only, and blocked from manual API modifications or creations. Additionally guard background task creation and semantic search when the environment variable disables embedding agents.

* test(upload): remove flaky status assertion on workflow tasks
2026-10-16 21:20:25 +08:00
Yiling-J b7271777e0 feat(agent): support gemma-embedding-2-openai provider (#475)
* feat(agent): support gemma-embedding-2-openai provider

Add support for local LiteRT-LM Gemma Embedding 2 provider with 768-dimensional vector embeddings, environment-configured read-only embedding agents, 16-frame uniform sampling for video chunks, and automatic WebP-to-JPEG conversion.

* refactor(agent): remove unused legacy generateEmbeddingActivity

* chore(agent): align sharp version to 0.35.4

* refactor(agent): rename embedding server env to GEMMA_EMBEDDING_OPENAI_BASE_URL

* refactor(agent,transcode): use native Bun.Archive for video chunk frames

* fix(transcode,agent): populate buffer entries in Bun.Archive and validate frame sizes
2026-10-15 15:49:24 +08:00
Yiling-J d98f486c9d feat(notification): add email notification infrastructure, smtp configuration, and periodic background delivery job (#431)
* feat(notification): add email notification support and dedicated workflow

* feat(webui): add team email notification settings and test email dialog

* refactor(webui): remove redundant card header from email notification settings

* fix(notification): prevent sending notification emails to AI agents

* feat(notification): display project, avatar, upload time and resolve mention names in notification emails

* refactor(notification): remove uploadDebounceSeconds and naive workflow sleep

* refactor(notification): remove temporal notification workflow and keep email infrastructure

* fix(team): prevent leaking email notification settings and credentials in team settings

* feat(notification): add periodic background job for batched email notification delivery

* fix(notification): embed avatars and thumbnails as inline cid attachments
2026-09-28 09:58:59 +08:00
Yiling-J 941000d277 feat(sandbox): abstract SandboxService and hot-reload allowed domains (#398)
* feat(sandbox): abstract SandboxService and hot-reload allowed domains

Introduce SandboxService with pluggable SandboxProvider in @shumai/core. Eliminate disruptive per-session sandbox resets by hot-reloading allowed domains via updateConfig, isolate violations per commandId, and eagerly initialize sandbox at worker/server boot.

* fix(sandbox): isolate command violations and serialize concurrent domain sync

Strictly isolate getBlockedHost per commandId to prevent false-positive blocked reporting on concurrent allowed commands. Serialize syncAllowedDomains to ensure concurrent requests during initialization or updates are never silently dropped. Add comprehensive unit tests for LocalSandboxProvider and concurrent bash commands.

* fix(agent): do not synchronize sandbox when allowedDomains is omitted

Make allowedDomains optional in CreateAgentSessionParams and only invoke sandboxService.syncAllowedDomains when allowedDomains is defined. Omit allowedDomains in generateSessionNameActivity so background chat title generation never overwrites the live sandbox allowlist.
2026-09-08 20:21:29 +08:00
Yiling-J 1e60933cdb feat(agent): store preset avatars as real files in s3 storage (#381)
* feat(agent): store preset avatars as real files in s3 storage

* fix(agent): use inline base64 previews in dialog and remove static avatar files

* feat(agent): add 4 avatar presets and track avatarPreset in agent config

* refactor(agent): consolidate preset avatar base64 data in dtos
2026-09-04 16:39:19 +08:00
Yiling-J 470a4bd6f7 feat(auth): add admin password reset CLI command and reset password web UI (#347)
* feat(auth): add admin password reset CLI command and reset password web UI

* fix(auth): invalidate previous reset tokens and use AsyncLocalStorage for link capture
2026-08-25 14:35:33 +08:00
yiling 7949c1572a feat(web): update default MAX_REQUEST_BODY_SIZE to 20GB 2026-07-29 20:28:22 +08:00
yiling 6000b8e366 feat(web): register pdf.worker route manually in prod mode 2026-07-18 19:10:07 +08:00
yiling bcb17acaf7 chore(serve): set idleTimeout to 120 seconds in Bun.serve 2026-07-11 21:11:18 +08:00
yiling 13f220ef0f refactor(e2e): replace S3 bucket name check with explicit SHUMAI_E2E env flag 2026-07-10 20:11:54 +08:00
yiling 26982ee494 feat(e2e): set up fullstack app-level e2e test framework and signup test case 2026-07-10 15:21:01 +08:00
yiling d3bbd42208 fix(web): optimize serveMainHtml performance by processing HTML at startup 2026-07-09 15:50:09 +08:00
yiling 8d19b578e7 fix(build): resolve code-splitting chunk import issue on backend and frontend 2026-07-09 15:00:23 +08:00
yiling 98bbfa51ae refactor(web): dynamically register production HTML bundle routes 2026-06-21 00:20:41 +08:00
yiling 93b4656939 feat(cli): add daemon mode and stop, logs, restart subcommands 2026-06-20 12:37:49 +08:00
yiling 292307e28e test(release): add check flag to app entrypoints and simplify CI smoke tests 2026-06-20 11:54:30 +08:00
yiling a69dd6a660 feat(server): allow customizing server port via SHUMAI_SERVER_PORT 2026-06-18 10:08:31 +08:00
yiling 84aa1662be fix(docker): align build output structure and fix web 404 error 2026-06-12 21:24:59 +08:00
yiling 91fedf1477 fix(cli): resolve all external module imports in global CLI package 2026-06-12 21:08:40 +08:00
yiling 78de8e463d feat(upload): increase upload limit and fix folder upload placeholders 2026-06-07 15:53:51 +08:00
yiling 5947c3c9a7 refactor(workflow): eliminate worker-db package and simplify workflows 2026-06-02 16:06:24 +08:00
yiling 3da2cdc524 feat(monorepo): introduce apps folder and decouple api package 2026-06-02 15:30:16 +08:00