96 Commits
Author SHA1 Message Date
yiling d8c20d6a11 Release v0.7.0 2026-11-18 21:10:30 +08:00
Yiling-J 4ac274cdd4 feat(runner): replace server-side bash sandbox with remote runner (#530)
* feat(runner): replace server-side bash sandbox with remote runner

* fix(runner): address review findings for output bounds, key rotation, defaults, and generated uploads

* chore(docker): remove sandbox privileges and env vars from remaining compose files

* feat(cli): add runner start with bundled SRT sandboxing (#535)

* refactor(agent): expose runner bash without skill gating

* feat(cli): add runner start with optional SRT sandboxing

* refactor(cli): remove runner Python dependency

* feat(cli): bundle SRT and use platform-native runner shells

* refactor(cli): delegate runner processes to Execa

* chore(cli): upgrade @anthropic-ai/sandbox-runtime to 0.0.83

* feat(cli): discover Bash for Windows runners

Use Pi-style Bash discovery and launch behavior, exercise the packaged CLI shell, and wait for pending ZIP metadata reads before closing cancelled downloads.

* fix(runner): enforce ownership and safe reconnects

Gate dispatch on reconciliation, retain cancellations across disconnects, and stop replaced clients. Guard the single-server runner topology with a database lease. Restage missing generated inputs, report signal exits, and bound generated-file cleanup.

* feat(webui): add Korean translations for agent runners

* feat(cli): add runner event logging and --verbose flag (#536)

* feat(cli): add runner event logging and --verbose flag

* fix(cli): log runner capacity rejections accurately and use monotonic duration
2026-11-18 15:58:57 +08:00
yiling 0a66776067 Release v0.6.6 2026-11-13 20:01:51 +08:00
Yiling-J 35a0e82284 refactor(agent)!: migrate to Pi Durable and embed the agent queue (#525)
* refactor(agent)!: migrate sessions to Pi Durable

Replace patched AgentHarness storage with per-session SQLite and PostgreSQL commit mirroring. Migrate legacy history automatically and run agent queues with the main server.

BREAKING CHANGE: Remove standalone agent workers and return native Pi entry records from chat history APIs.

* fix(agent): constrain session log dialog scrolling

Keep the header visible and wrap long output while log entries and expanded records scroll within the dialog. Cover desktop and narrow viewports through the dashboard Sessions flow.

* fix(agent): hide entry IDs from session log card headers

Retain internal entry IDs in expanded raw records for debugging. Verify that card headers omit the IDs while raw JSON preserves them.

* test(e2e): handle browser wheel limits in session log checks

Verify that a normal wheel event moves the viewport before explicitly scrolling to the final record. Keep the dialog bounds, fixed header, wrapping, and expanded-record assertions.

* fix(agent): select model providers without inheriting unrelated auth

* fix(webui): prevent resize feedback when toggling agent panel

* test(agent): reproduce concurrent comment session races

* fix(agent): serialize comment sessions and deduplicate history

* docs(changelog): consolidate Pi Durable changes into one entry
2026-11-11 16:05:18 +08:00
yiling 1050024fff Release v0.6.5 2026-11-07 19:55:05 +08:00
yiling 0c0b887b8d Release v0.6.4 2026-11-05 22:45:25 +08:00
yiling 49a2d7c380 Release v0.6.3 2026-11-01 19:52:23 +08:00
yiling f0bdccf713 Release v0.6.2 2026-10-29 20:29:50 +08:00
Yiling-J baf30730af feat(mcp): add OAuth 2.1 support (#505)
* feat(mcp): add OAuth 2.1 support with dynamic client registration

* feat(api): alias bare oauth-authorization-server discovery endpoint

* fix(auth): preserve user session on OAuth sign-in redirect to consent
2026-10-29 15:39:37 +08:00
Yiling-J e44938cb8f feat(cli): generate shumai-cli commands from Operations with commander (#496)
Rebuild the CLI on commander. Commands for every cli-surface Operation
(project ls, ls, mkdir, rename, move/mv, delete/rm) are generated from
@shumai/operations: positionals, options, choices, numeric parsing and
help text come from the Zod input schema. They call POST /api/ops/:name
with x-shumai-client: cli.

Existing invocations keep working. Adds --json, ls --type/--first/
--after, typo suggestions, and generated help. upload and create-version
stay hand-written. Drops the unused @shumai/core dependency.
2026-10-27 08:09:10 +08:00
Yiling-J 4d61786799 feat(mcp): serve Operations over MCP and a generic /api/ops endpoint (#495)
Add a stateless Streamable HTTP MCP server at /api/mcp that exposes every
mcp-surface Operation with its input/output schemas and annotations, and
POST /api/ops/:name for the CLI. Both require an API token (session
cookies are rejected), run Operations as the token's user, honor demo
read-only mode, and record via mcp/cli/api in audit logs.

Also make list_projects toggleable in agent settings and document MCP
client setup.
2026-10-27 08:09:09 +08:00
Yiling-J 76057ccd21 feat(operations): add shared Operation layer for agent capabilities (#494)
Introduce @shumai/operations (pure definitions: Zod schemas, metadata,
CLI hints) and @shumai/core/src/operations (implementOperation and the
runOperation pipeline: validate, authorize, prepare, execute, validate
output, audit).

Migrate list_assets, create_folder, rename_asset, move_assets and
delete_asset to Operations, and add list_projects. The agent now builds
these tools from the registry, and the folder create/rename, file
rename and reparent routes call the same Operations.

Audit logs gain a nullable via column recording the surface. A lint
rule keeps the Prisma client out of Operation implementations.
2026-10-27 08:09:09 +08:00
yiling c2af7494ec Release v0.6.1 2026-10-24 13:21:50 +08:00
Yiling-J 5cf6d31794 chore(deps): upgrade bun to 1.4.4 and migrate to bun fmt (#488)
- Upgrade Bun to 1.4.4 across Dockerfiles, CI workflows, and release configurations.
- Update @types/bun and bun-types.
- Remove eslint-plugin-prettier and update format script to use native bun fmt.
- Configure .prettierignore to exclude packages/webui/public and minified files.
2026-10-22 20:25:54 +08:00
yiling e05a23bcc2 Release v0.6.0 2026-10-20 20:02:34 +08:00
Yiling-J 176b7f0eab feat(workflow): migrate workflow engine from temporal to dbos (#481)
* feat(workflow): migrate workflow engine from temporal to dbos

Replace Temporal and LocalExecutor with DBOS Transact (@dbos-inc/dbos-sdk) as the unified workflow engine. Background tasks, media transcoding, and AI agent workflows now execute with durable steps and task queues directly against PostgreSQL, eliminating the need for an external Temporal cluster and separate polling executors.

* fix(workflow): submit tasks via DBOSClient to allow enqueueing from steps

Enqueuing workflows using DBOS.enqueueWorkflowWithOptions threw a
DBOSInvalidWorkflowTransitionError when triggered from within a running
step or activity (e.g. creating embedding or autofill tasks during
video transcoding).

Use DBOSClient to submit and cancel workflow tasks directly against the
system database, detaching execution from the ambient workflow/step
context.

* feat(web): support SHUMAI_WORKERS environment variable

* fix(workflow): ensure CLI tasks are enqueued to DBOS and run DBOS migrations in global setup

- Initialize workflows and wait for pending DBOS submissions in shumai regenerate-embeddings CLI
- Remove no-op submit bypass and enqueue tasks via DBOSClient in producer and CLI contexts
- Add DBOS migration runner in test global setup to avoid concurrent migration deadlocks across test forks
- Update changelog and add regression tests

* fix(workflow): ensure unique per-instance executor identity for worker replicas

- Derive unique per-instance DBOS executor IDs across worker replicas

- Support SHUMAI_WORKER_ID, DBOS__VMID, and POD_NAME (Kubernetes Downward API)

- Disambiguate multi-replica bare metal/container deployments with host, PID, and instance suffix

- Prevent startup recovery from re-enqueueing workflows actively executed by other live workers

- Add multi-instance startup recovery tests and update deployment documentation

* fix(workflow): only register listened queues and update concurrency on change

- Only register queues that the executor instance actively listens to

- Use always_update conflict resolution so environment variable changes take effect on restart

- Prevent dedicated workers from overwriting unmanaged queue concurrency configurations

- Add tests verifying concurrency updates and queue isolation across worker roles

* refactor(workflow): simplify activities to use defineStep and direct calls (#482)

* refactor(workflow): simplify activities to use defineStep and direct calls

- Wrap activities using defineStep from @shumai/workflow-core
- Remove obsolete getActivities, registerActivities, dynamic activity proxies, and workerQueue arguments
- Remove executor.ts and registry.ts, consolidating workflow registration into workflow.ts
- Call activities directly across transcode and agent workflows
- Update all corresponding workflow tests and developer guide documentation

* fix(workflow): properly detect DBOSWorkflowCancelledError in runOnCancel fallback

- DBOSWorkflowCancelledError in DBOS SDK does not set name, inheriting 'Error'
- Implement isDbosWorkflowCancelledError checking instanceof, constructor.name, and dbosErrorCode
- Update unit tests to verify real DBOS SDK cancellation error instances

* docs(workflow): update documentation and readme for dbos migration

- Rename docs/configuration/temporal.mdx to docs/configuration/distributed.mdx
- Document DBOS-based distributed worker architecture and task queues
- Remove obsolete docker-compose/temporal/
- Update environment variables documentation in docs/configuration/env-variables.mdx
- Update references in README.md, README_CN.md, docs/docs.json, and related guides

* feat(deploy): implement KEDA ScaledJob and scale-to-zero On-Demand transcoding (#483)

* feat(deploy): add shumai-transcode docker image and k8s autoscaling example with garage s3

* docs(k8s): document port-forwarding and in-cluster s3 domain mapping

* feat(storage): support public s3 endpoint for browser uploads and update k8s example

* fix(k8s): configure s3 bucket cors rules in garage initialization

* fix(k8s): increase transcode worker memory limit to 4Gi for 4K video encoding

* feat(deploy): add AWS EKS deployment with Graviton3 Spot nodegroups and autoscaling

* docs(deploy): redesign EKS documentation, rename k8s to eks, and add scalable deployment feature

* docs: rephrase DBOS feature bullet to Durable Workflows via DBOS

* docs(eks): upgrade default instances to Graviton5 c9g.large and c9g.xlarge

* feat(eks): use On-Demand for general nodegroup and add spot capacity diversification

* docs(eks): fix Architecture diagram to locate all transcode worker pods in Transcode Compute Pool

* feat(deploy): implement KEDA ScaledJob and scale-to-zero On-Demand transcoding

* feat(transcode): support idle-timeout auto-drain for KEDA scaled jobs

* fix(deploy): fix EKS deployment manifests and documentation based on cluster verification
2026-10-19 15:35:08 +08:00
yiling f7ed3bafdf Release v0.5.3 2026-10-16 22:50:47 +08:00
Yiling-J 6cc90bf973 feat(search): generate query embeddings directly for semantic search (#476)
* feat(search): generate query embeddings directly for semantic search

* fix(search): resolve embedding provider from env, redact query logs, and preserve case in cache
2026-10-15 20:49:45 +08:00
Yiling-J b7271777e0 feat(agent): support gemma-embedding-2-openai provider (#475)
* feat(agent): support gemma-embedding-2-openai provider

Add support for local LiteRT-LM Gemma Embedding 2 provider with 768-dimensional vector embeddings, environment-configured read-only embedding agents, 16-frame uniform sampling for video chunks, and automatic WebP-to-JPEG conversion.

* refactor(agent): remove unused legacy generateEmbeddingActivity

* chore(agent): align sharp version to 0.35.4

* refactor(agent): rename embedding server env to GEMMA_EMBEDDING_OPENAI_BASE_URL

* refactor(agent,transcode): use native Bun.Archive for video chunk frames

* fix(transcode,agent): populate buffer entries in Bun.Archive and validate frame sizes
2026-10-15 15:49:24 +08:00
yiling c302c162a0 Release v0.5.2 2026-10-13 21:03:51 +08:00
yiling f8beb195dd Release v0.5.1 2026-10-06 22:29:50 +08:00
yiling 013c4a8156 Release v0.5.0 2026-10-02 20:28:37 +08:00
Yiling-J a84da8678e feat(webui): switch from video.js to hls.js and native video player (#437)
* feat(webui): switch from video.js to hls.js and native video player

* fix(webui): prevent stale seek hijacking and guard native HLS compare mode
2026-09-30 20:23:44 +08:00
Yiling-J 5fe22ac52c feat(transcode): add initial camera RAW image support (#435)
- Add centralized RAW file extension set and detection helper for 23 common camera RAW formats
- Update getProxyType and EXTENSION_MIME_TYPES to route RAW formats into the image pipeline
- Add exiftool-vendored runtime dependency and mark as external in build-npm script
- Implement embedded JPEG extraction to temporary file with priority JpgFromRaw2 -> JpgFromRaw -> PreviewImage
- Read and apply EXIF orientation from RAW container to prevent rotation/mirroring issues
- Integrate into TranscodeService.getImageInfo and transcodeImage with proper temporary file cleanup
- Add hasUsablePreview guard in transcodeImageWorkflow to gracefully handle RAW files without embedded previews
- Add unit, service, workflow, and E2E tests for RAW processing
2026-09-29 21:09:27 +08:00
yiling f2b0415580 Release v0.4.9 2026-09-28 20:17:50 +08:00
Yiling-J d98f486c9d feat(notification): add email notification infrastructure, smtp configuration, and periodic background delivery job (#431)
* feat(notification): add email notification support and dedicated workflow

* feat(webui): add team email notification settings and test email dialog

* refactor(webui): remove redundant card header from email notification settings

* fix(notification): prevent sending notification emails to AI agents

* feat(notification): display project, avatar, upload time and resolve mention names in notification emails

* refactor(notification): remove uploadDebounceSeconds and naive workflow sleep

* refactor(notification): remove temporal notification workflow and keep email infrastructure

* fix(team): prevent leaking email notification settings and credentials in team settings

* feat(notification): add periodic background job for batched email notification delivery

* fix(notification): embed avatars and thumbnails as inline cid attachments
2026-09-28 09:58:59 +08:00
yiling f6e27095a5 Release v0.4.8 2026-09-24 20:26:23 +08:00
yiling 657cd761bc Release v0.4.7 2026-09-19 16:01:20 +08:00
yiling 76ffb51bb2 Release v0.4.6 2026-09-17 21:58:10 +08:00
Yiling-J 1796de8eda feat(premiere-uxp): introduce Adobe Premiere Pro UXP plugin for Shumai (#407)
* feat(premiere-uxp): use native Bun bundler and build script instead of Vite

* feat(premiere-uxp): switch FileListView to use Search API consistent with WebUI

* feat(premiere-uxp): add on-demand jpeg preview conversion for uxp compatibility

* feat(premiere-uxp): integrate spectrum web components (swc)

* fix(premiere-uxp): align swc packages with 0.37.0 and fix auth layout

* feat(premiere-uxp): use horizontal 2-column card view for projects and files

* style(premiere-uxp): enlarge file preview by 20% and add row padding

* fix(premiere-uxp): prevent card squishing and overflow with large item counts

* fix(premiere-uxp): change host in manifest to single object

* docs(premiere-uxp): add AGENTS.md with SWC preferences and UXP layout rules

* feat(premiere-uxp): integrate spectrum web components across plugin views

* fix(premiere-uxp): remove breadcrumb icons and use slash separator

* feat(premiere-uxp): add collapsible folder and file sections with infinite scroll

* feat(premiere-uxp): update file card layout with right column info and clean preview

* fix(premiere-uxp): use raw text for file card details and fix black icons

* style(premiere-uxp): use contain object-fit and black background for file card preview

* fix(premiere-uxp): refine header logo, sticky title bars, circular badges, and controls

* fix(premiere-uxp): replace stroke icons with spectrum workflow icons

* fix(premiere-uxp): switch icon buttons to swc sp-button with icon-only and slot=icon

* fix(premiere-uxp): vertically center endpoint text in header

* feat(premiere-uxp): add asset and proxy import support into Premiere Pro

- Implement context menu on file cards with left-click, right-click, and more action button triggers
- Support raw file import for non-video assets and video assets
- Support video proxy submenu listing ready transcode resolutions
- Add UXP native file picker save dialog and binary download orchestration
- Call Premiere Pro project.importFiles API to import downloaded assets into active project
- Add non-blocking toast notification banner for progress and status reporting
- Add unit tests for premiere host service and import orchestration

* fix(premiere-uxp): anchor import menu directly to file card using sp-action-menu

- Replace custom fixed-position ContextMenu with native SWC sp-action-menu
- Anchor menu directly to the triggered file card rather than the top-left window corner
- Support programmatic opening on file card click/contextmenu via imperative handle
- Support video submenu with Original and ready proxy options inside sp-action-menu
- Lazy load video proxies on menu open or hover
- Clean up obsolete fixed-position styles and unused backdrop overlays

* fix(premiere-uxp): align SWC packages to 0.37.0 and guard custom element registry

- Downgrade action-menu and menu SWC packages to 0.37.0 to match the rest of the workspace
- Eliminate duplicate icons-ui and sp-icon-chevron100 customElements.define calls
- Add polyfill guard to customElements.define to prevent duplicate registration errors during UXP reload

* fix(premiere-uxp): polyfill window.matchMedia for Adobe UXP

- Add window.matchMedia polyfill in polyfills.ts
- Satisfies SWC MatchMediaController used by ActionMenu and PickerBase in UXP
- Prevents TypeError when opening files page

* feat(premiere-uxp): replace context menu with card download action and video import dialog

* fix(premiere-uxp): resolve download url before save, pass null targetBin, and center download icon

* fix(premiere-uxp): resolve raw download url via download-url endpoint with fallback

* feat(premiere-uxp): use native folder picker and auto-increment conflict suffixes

* feat(premiere-uxp): add sequence linking, comment-marker sync, and navigation tabs

- Add sequence-to-asset linking with persistent project metadata storage

- Support one-way comment-to-marker timeline synchronization

- Add two-tab navigation (Sequences and Browse) using Spectrum buttons

- Add LinkSequenceDialog, SequencesView, and autoSyncService background engine

- Enforce Spectrum Workflow Icons and UXP layout rules

* feat(premiere-uxp): refine UI layouts, card interactions, and search box activation

* fix(premiere-uxp): set compact fixed width for file list search box

* fix(premiere-uxp): ensure search box click activation across full width in UXP

* fix(premiere-uxp): revert search box hacks and restore simple default sp-search layout

* feat(premiere-uxp): sync non-timestamped comments as markers at time 0

* feat(premiere-uxp): change link icon to unlink when asset is linked in file card

* fix(premiere-uxp): sync linked sequences between host project and browse view

* feat(premiere-uxp): remove synced markers on sequence unlink matching Frame.io

* fix(premiere-uxp): ensure reliable marker GUID capture, normalization, and fallback on unlink

* fix(premiere-uxp): use official Shumai logo for plugin manifest icons

* fix(premiere-uxp): use official PNG icons for manifest and plugin entrypoints

* refactor(premiere-uxp): use single 114x114 icon for plugin and panel

* feat(premiere-uxp): show file version in file list page file card

* build(premiere-uxp): check in built ccx package and update packaging configuration

* fix(premiere-uxp): address code review findings for media conversion, marker sync, and link scoping

- Reset hasJpegCover on project cover update and hasJpegPreview on asset media transcode
- Only advance synced comment metadata after confirmed marker transaction
- Normalize GUID keys and lookups in AutoSyncService map
- Restrict cached sequence link lookups to project-specific keys and prevent cross-project leakage on empty sequences
- Update built CCX distribution package
2026-09-17 11:51:03 +08:00
Yiling-J def42cd000 feat(comment): add emoji reaction support for comments (#405)
* feat(comment): add emoji reaction support for comments

* fix(webui): adjust emoji picker trigger icon size to match mark-done icon

* fix(webui): prevent search icon and placeholder overlap in emoji picker

* fix(webui): use normal text color for reaction count

* feat(db): limit comment reaction code column to VarChar(32)
2026-09-14 15:01:36 +08:00
yiling ede52e5009 Release v0.4.5 2026-09-13 21:50:44 +08:00
yiling 70b81355c2 Release v0.4.4 2026-09-09 22:07:40 +08:00
Yiling-J 941000d277 feat(sandbox): abstract SandboxService and hot-reload allowed domains (#398)
* feat(sandbox): abstract SandboxService and hot-reload allowed domains

Introduce SandboxService with pluggable SandboxProvider in @shumai/core. Eliminate disruptive per-session sandbox resets by hot-reloading allowed domains via updateConfig, isolate violations per commandId, and eagerly initialize sandbox at worker/server boot.

* fix(sandbox): isolate command violations and serialize concurrent domain sync

Strictly isolate getBlockedHost per commandId to prevent false-positive blocked reporting on concurrent allowed commands. Serialize syncAllowedDomains to ensure concurrent requests during initialization or updates are never silently dropped. Add comprehensive unit tests for LocalSandboxProvider and concurrent bash commands.

* fix(agent): do not synchronize sandbox when allowedDomains is omitted

Make allowedDomains optional in CreateAgentSessionParams and only invoke sandboxService.syncAllowedDomains when allowedDomains is defined. Omit allowedDomains in generateSessionNameActivity so background chat title generation never overwrites the live sandbox allowlist.
2026-09-08 20:21:29 +08:00
Yiling-J 67b767f1e8 feat(upload): S3 multipart upload with offline resume and wake lock (#391)
* feat(upload): implement S3 multipart upload with offline resume and wake lock

* fix(upload): harden S3 sign and abort authorization, await confirmation, and clean up failed multipart uploads

* fix(webui): resume multipart upload on network recovery and log online event

* fix(webui): remove offline and online toast notifications
2026-09-07 16:49:00 +08:00
yiling 62e86b703e Release v0.4.3 2026-09-06 22:22:14 +08:00
Yiling-J b087bf73ae chore(i18n): update paraglide-js to 2.25.0 and stop checking in generated code (#386)
- Upgrade @inlang/paraglide-js to 2.25.0 in root devDependencies
- Untrack 2,700+ generated paraglide files from git index
- Ignore packages/webui/paraglide in .gitignore and .dockerignore
- Remove --no-emit-git-ignore flag from i18n:compile and i18n:watch
- Chain bun run i18n:compile into build and build-npm scripts
- Add i18n compile step in Dockerfile builder stage
- Add i18n compile step in CI workflows (.github/workflows/ci.yaml, release-npm.yaml)
- Update README.md and README_CN.md with local setup instructions for i18n compilation
2026-09-06 15:48:39 +08:00
Yiling-J a49722d5c7 feat(settings): redesign image/video settings to provider dialog with model toggles (#385) 2026-09-06 15:13:01 +08:00
Yiling-J 576d8ebe64 chore(deps): upgrade bun to 1.4.2 and update packages (#384)
* chore(deps): upgrade bun to 1.4.2 and update packages

- Upgrade Bun to 1.4.2 in Dockerfile, CI workflows, and release workflows.
- Update dependencies across workspace with Bun 1.4.2.
- Preserve @earendil-works/pi-agent-core at 0.82.1.
- Retain stable compatible versions for database adapters and pdfjs-dist.
- Add explicit Buffer type annotation in transcode service.

* chore(deps): upgrade prisma to 7.10.0, pg to 8.23.0, and pdfjs-dist to 6.3.289

* chore(deps): revert dotenv-expand to 13.0.0
2026-09-06 11:31:05 +08:00
Yiling-J 6285ff8618 feat(agent): add built-in image and video generation tools and settings (#382)
* feat(agent): add built-in image and video generation tools and settings

* fix(settings): refine media generation layout and expose media tools in agent and quota forms

* feat(media-generation): add alibaba, bytedance, minimax, prodia, and azure providers

* feat(agent): support s3 key references for media generation and s3KeyOnly in read_asset
2026-09-06 09:28:53 +08:00
Yiling-J 3c3b6880e4 feat(webui): replace legacy markdown editor with new lexical wysiwyg editor (#308)
* feat(webui): replace legacy markdown editor with new lexical wysiwyg editor

* refactor(webui): organize markdown editor files into components/markdown-editor folder
2026-08-14 16:27:14 +08:00
Yiling-J 7f185ca23c feat(agent): type-safe CREATION_CONTEXT metadata on create_file/create_version (#303)
* feat(agent): type-safe CREATION_CONTEXT metadata on create_file/create_version

Build the create_file and create_version tool metadata parameter from the
project's CREATION_CONTEXT fields at harness creation time, instead of the
schemaless key-value map that required the agent to call list_autofill_fields
first.

- Thread projectId from the chat workflow into createAgentSession and build
  a strict, all-required (nullable) metadata schema via fieldsToTypeBoxSchema
  (additionalProperties: false, option IDs enforced via enum).
- Remove the list_autofill_fields tool and its activity case.
- Reject metadata keys that are not CREATION_CONTEXT fields of the target
  project with a clear non-retryable error telling the agent to omit metadata.
- Skip null metadata values when persisting (create_file/create_version and
  the autofill workflow).
- Switch the touched schemas from @sinclair/typebox 0.34 to typebox v1 to
  match the harness validator.

* refactor(agent): migrate remaining tools from @sinclair/typebox 0.34 to typebox v1

All agent tool schemas now use the same typebox v1 package as the harness
validator; drop the @sinclair/typebox 0.34 dependency from @shumai/agent.

* test(agent): reproduce null coercion in harness tool validation

The pi-ai harness (<=0.82.1) coerces null values in [string, null] unions to
'' during tool argument validation for typebox v1 schemas. A valid create_file
call with path: null + data: {...} therefore arrives at execute with path: ''
and fails with 'Provide exactly one of path or data'. Null metadata values
(the 'unknown' sentinel) are likewise turned into ''.

This test reproduces both failures. It is expected to fail until
@earendil-works/pi-ai is upgraded to >=0.84.0 (#7328), which preserves values
that already match a union arm.

* fix(agent): patch pi-ai 0.82.1 to preserve null in nullable union tool args

Apply the upstream fix for earendil-works/pi#7328 (PR #7373) via a bun
patch instead of upgrading pi-ai. coerceWithUnionSchema now returns the
original value when it already validates against any union member,
instead of coercing null in [string, null] unions into "".

Fixes the create_file false "Provide exactly one of path or data" error
when the model passes path: null with data. Reproduced by
packages/agent/src/reproduce-tool-null-coercion.test.ts, which now passes.

* fix(docker): copy bun patch dir before frozen-lockfile install

The pi-ai patchedDependencies entry makes bun install --frozen-lockfile
fail in the builder stage because patches/ was not present in the image
yet. Copy it alongside package.json/bun.lock so the patch applies.

* refactor(agent): simplify metadata guidance in system prompt and error message

Shorten the create_file/create_version metadata instructions in the system
prompt and reword the invalid-metadata-key error to not reference
CREATION_CONTEXT internals. Update the matching test assertion.
2026-08-13 20:06:09 +08:00
Yiling-J 1b5cbf936a feat(mcp): add MCP server support with proxy tool and OAuth flow (#289)
* feat(mcp): add MCP server support with proxy tool and OAuth flow

Add URL-based MCP (Model Context Protocol) server support to the backend:

- DB: McpServer / AgentMcpServer / McpServerCredential models with
  typed JSON columns (auth config, cached tools, stored OAuth tokens,
  pending auth state) and a migration.
- Core (@shumai/core/src/mcp): DB-backed credential store, OAuth
  client provider (authorization_code + client_credentials) with a
  Hono-hosted redirect callback, connection lifecycle manager
  (Streamable HTTP with SSE fallback, lazy connect, 401 -> needs_auth),
  JSON Schema -> AgentTool parameter conversion, in-memory tool registry
  with ranked search, the single 'mcp' proxy AgentTool (search /
  describe / list / call / connect / status / auth-start), per-server
  opt-in direct tools ({server}_{tool}), and McpService (CRUD,
  discovery, OAuth, per-agent assignment via AgentMcpServer, and
  permission gating on tool execution mirroring skills).
- Agent: chat agents receive the 'mcp' proxy tool (plus direct tools
  for opted-in servers) built from their assigned, enabled servers.
- API: /teams/:teamId/mcp/servers CRUD, tools refresh/list, auth
  start/complete/status/remove, test, permission, and the public
  OAuth callback route; Admin authz via new ResourceType.McpServer;
  audit logging.
- DTOs: zod schemas + types for servers, tools, auth, and per-agent
  mcpServerIds.
- Tests: DB store, schema conversion, registry/search, service
  integration against a real in-process MCP server (v2
  @modelcontextprotocol/server), permission gating, rename
  invalidation, direct tools, per-agent assignment, API route tests,
  and workflow E2E (local + temporal) for proxy and direct modes.

* fix(mcp): persist OAuth discovery snapshot and stamp SEP-2352 issuer

The callback leg of the OAuth flow failed with
AuthorizationServerMismatchError ('discoveryState was not available on
the callback leg') because McpDbStore.savePendingAuth overwrote the
entire pendingAuth JSON column, clobbering the discoverySnapshot that
the provider's saveDiscoveryState() had stored inside it.

- McpDbStore.savePendingAuth now preserves an in-flight discovery
  snapshot so the callback leg can satisfy the SEP-2352 authorization-
  server issuer check.
- McpOauthProvider now round-trips the issuer stamp derived from the
  discovery snapshot: saveClientInformation/saveTokens persist it,
  clientInformation()/tokens() return it, and tokens() backfills a
  missing stamp for pre-existing credentials (removes the
  'stored OAuth credential has no issuer stamp' warning).
- Regression tests: savePendingAuth keeps the snapshot; the provider
  stamps/round-trips the issuer on both legs.

* fix(mcp): backfill SEP-2352 issuer stamp on stale OAuth client info

clientInformation() returned stored client info without an issuer field
when it had been dynamically registered before the stamping fix, so the
SDK logged 'stored OAuth credential has no issuer stamp' on every read.
Now the stored branch backfills the issuer from the discovery snapshot
(persisting it) exactly like the tokens() backfill.

* feat(webui): add MCP server configuration and agent assignment UI (#290)

* feat(webui): add MCP server configuration and agent assignment UI

* fix(webui): auto-detect auth for unconfigured MCP servers and show connect button

* fix(mcp): auto-generate state in mcp-oauth-provider when missing in DB

* fix(mcp): persist OAuth discovery snapshot and stamp SEP-2352 issuer

The callback leg of the OAuth flow failed with
AuthorizationServerMismatchError ('discoveryState was not available on
the callback leg') because McpDbStore.savePendingAuth overwrote the
entire pendingAuth JSON column, clobbering the discoverySnapshot that
the provider's saveDiscoveryState() had stored inside it.

- McpDbStore.savePendingAuth now preserves an in-flight discovery
  snapshot so the callback leg can satisfy the SEP-2352 authorization-
  server issuer check.
- McpOauthProvider now round-trips the issuer stamp derived from the
  discovery snapshot: saveClientInformation/saveTokens persist it,
  clientInformation()/tokens() return it, and tokens() backfills a
  missing stamp for pre-existing credentials (removes the
  'stored OAuth credential has no issuer stamp' warning).
- Regression tests: savePendingAuth keeps the snapshot; the provider
  stamps/round-trips the issuer on both legs.

* fix(mcp): backfill SEP-2352 issuer stamp on stale OAuth client info

clientInformation() returned stored client info without an issuer field
when it had been dynamically registered before the stamping fix, so the
SDK logged 'stored OAuth credential has no issuer stamp' on every read.
Now the stored branch backfills the issuer from the discovery snapshot
(persisting it) exactly like the tokens() backfill.

* fix(mcp): attach McpOauthProvider for auto-detected OAuth servers and discover tools on auth completion

* fix(webui): fix flex min-h-0 and max-h scrolling on MCP tools dialog

* fix(webui): set explicit flex height on McpToolsDialog for active scrolling

* feat(webui): make MCP server cards clickable to edit and widen server config dialog

* feat(mcp): agent-level server enablement and auto-detected server metadata (#291)

* feat(mcp): agent-level server enablement and auto-detected server metadata

- Remove the server-level `enabled` column from mcp_servers; enable/disable
  now happens per-agent via the AgentMcpServer join table (mirrors skills).
- Drop the unique (teamId, name) constraint so duplicate server names are allowed.
- Add a `description` column, auto-filled from client.getServerVersion() on
  connect; the server name is auto-filled from the reported title (fallback name)
  with a URL-derived placeholder until first connect.
- Remove manual name input: the create dialog is URL-only, and the edit dialog
  shows the auto-detected name/description read-only.
- Replace the include/exclude raw-text inputs with the tool list in the edit
  dialog, managing only excludeTools; remove includeTools support and the
  separate McpToolsDialog component.
- Sanitize server-reported names into safe direct-tool prefixes (spaces/punctuation
  become underscores) and use the same sanitization in the proxy resolver.

* fix(mcp): give the edit-dialog tool list a definite height

The Shadcn ScrollArea viewport is h-full, which requires a definite height
on the root. Using max-h-64 left the viewport at auto height, so the root
clipped the list at 256px without engaging its own scrollbar — tools past
the cutoff were unreachable. Use a fixed h-64 container (same pattern as
move-copy-dialog) so the inner scroll area scrolls independently of the
dialog's outer scroll.

* refactor(mcp): fixed header/footer with scrollable content in server config dialog

Mirror the skill config dialog layout: the edit dialog now keeps the header
and footer fixed and scrolls only the content area (ScrollArea flex-1
min-h-0 inside an h-[85vh] flex column). The tools list no longer has its
own nested ScrollArea — it flows inside the content-level scroll, so all
tools are reachable without a nested-scroll conflict. The create dialog
stays compact (URL-only) like the skill add dialog.

Verified headlessly that max-height-based ScrollArea roots do not engage
scrolling (viewport height resolves to auto), which is why the previous
nested list clipped; the flex-based definite-height pattern scrolls.

* feat(mcp): make the endpoint URL immutable after creation

Editing a server's URL effectively points it at a different server, and the
create flow is URL-only anyway — so the edit dialog no longer allows changing
it (delete + re-add covers endpoint changes). The URL is displayed read-only
next to the auto-detected name/description, and `url` was removed from the
update DTO and service; auth/transport changes still invalidate credentials
and cached tools.

* refactor(mcp): match MCP server cards to the skill card layout

The settings list cards now share the skill card structure: row 1 is the
server name (with direct-mode/transport badges) plus the three-dot menu,
row 2 is the description (falling back to 'No description provided') with
the endpoint URL and last error as small mono lines, and row 3 holds the
connection-status badge and updated date on the left with the permission
selector and the connect button at the right end.

* fix(mcp): stop card menu clicks from opening the edit dialog

The three-dot menu is portaled, so its item clicks bubble through React's
synthetic events to the card's onClick, which opened the edit dialog before
the delete confirmation appeared. Add e.stopPropagation() to every menu
item (matching the skill card pattern).

* fix(mcp): stop Refresh Tools from submitting/closing the config dialog

The Refresh Tools button lived inside the <form> and, like all buttons
without an explicit type, defaulted to type="submit" — clicking it ran the
form submit (save + close) instead of just refreshing. Set type="button".

Also per UX: while refreshing, the button is disabled with a spinning icon,
the per-tool enable/disable switches are disabled, and the success toast is
removed — the button reverting to its idle state is the success signal
(keep the error toast for failures).

* test(mcp): provide explicit redirectUri in McpOauthProvider test helper

* fix(mcp): scope tool registry to agent-assigned servers

The MCP tool registry was a process-global singleton keyed by server id,
so a chat agent could search, describe, and even call tools belonging to
MCP servers that were not assigned to it (e.g. disabled or unassigned
servers), as long as their metadata had been discovered at some point in
the process. This mirrors the disabled-skill bug: the read path never
checked per-agent enablement.

Replace the singleton with a fresh McpToolRegistry built per
buildAgentTools call and warmed from the persisted McpServer.tools DB
cache. Because the registry only ever contains the agent's assigned
servers, unassigned/disabled servers can neither be suggested, searched,
described, nor executed. Live discovery (lazy connect / connect action)
writes into the per-agent registry via ensureConnected(serverId,
registry?).

Adds test-first coverage: suggestions, search, describe and tool-call
leaks across an unassigned same-team server, staleness after
unassignment, and direct-tools scoping.

* feat(mcp): capture and surface server instructions + server-level refresh (#293)

* feat(mcp): capture and surface server instructions

- Add McpServer.instructions column, populated from getInstructions() at
  connect time (legacy initialize + modern discover results)
- Surface instructions at three levels like pi-mcp-adapter: truncated
  snippet in the mcp proxy tool description, preview in server listings,
  and full text via mcp({ instructions: "name" })
- Fix misleading mcp({ server, connect }) usage line to mcp({ connect })
- Share one renderServerListing helper across list and connect modes
- Expose instructions in McpServerInfo DTO and WebUI config card
- Test server supports instructions; service + API tests updated

* refactor(mcp): server-level refresh endpoint, drop agent connect mode

- Replace POST /mcp/servers/:id/tools/refresh with POST /mcp/servers/:id/refresh
  returning the refreshed McpServerInfo (server-level, not tools-level)
- Add McpService.refreshServer(): force-close the connection, re-discover and
  persist tools, instructions, name/description and status; used by the refresh
  endpoint, Test Connection, and post-OAuth completion (replacing discoverTools)
- Remove discoverTools entirely - refreshServer is the single discovery entry
  point (testServer and completeAuth route through it)
- Remove refreshTools from UpdateMcpServerRequest/PATCH (dead param)
- Remove mcp({ connect: "name" }) from the agent proxy tool: lazy tool calls
  still auto-connect, so agents are unaffected
- WebUI: remove the dialog 'Refresh Tools' button and create-mode background
  auto-refresh; add a server-level Refresh button at the left end of the edit
  dialog footer; i18n mcp_refresh_server (+ success message)

* feat(mcp): add in-flight request tracking and idle connection cleanup (#294)

* feat(mcp): add in-flight request tracking and idle connection cleanup

* feat(mcp): add background idle cleanup timer to McpService

* feat(mcp): update default background idle cleanup check interval to 30s

* feat(webui): improve MCP server card footer, instructions, and refresh dialog freeze (#295)

* feat(webui): improve MCP server list card, instructions dialog, and refresh freeze UI

* feat(webui): refine MCP card action menu items

* docs(webui): update direct tools mode description in i18n messages

* style(webui): refine Direct Tools Mode layout in MCP server config dialog

* feat(webui): remove tool search box from MCP server config dialog

* fix(mcp): make in-flight idle tracking test deterministic

The 'tracks in-flight requests and safely prunes idle connections' test
relied on a wall-clock sleep (10ms) to guess when the tool handler had
started executing on the server. On slow/loaded CI runners the HTTP round
trip through the real MCP client/server stack can exceed that window, so
resolveSlowTool() fired the no-op placeholder, the gated handler never
completed, and callPromise hung until the 5000ms test timeout.

The handler now signals via a deferred promise when it actually starts,
and the test awaits that signal (racing it against the call promise to
surface early failures) before asserting in-flight state and releasing
the gate. Also gives the test an explicit 10s timeout.
2026-08-11 11:01:38 +08:00
yiling dbe3420b6c feat(agent): update pi packages to 0.82.1 and implement new SessionStorage interface methods 2026-08-03 10:25:10 +08:00
yiling 16d136b45a refactor(core): replace generated provider files with pi-ai dynamic methods 2026-08-03 10:00:21 +08:00
yiling 51208d3ff0 chore(deps): update @anthropic-ai/sandbox-runtime to 0.0.66 2026-07-28 16:13:40 +08:00
yiling e214036308 fix sharp version mismatch bug 2026-07-28 13:58:52 +08:00
yiling 6226090fd1 fix(e2e): add sharp dependency to packages/e2e 2026-07-25 11:17:32 +08:00
yiling e9959bcc68 fix(agent): handle non-builtin models in getModelFromDb and upgrade pi-ai 2026-07-22 16:09:36 +08:00
yiling 1970535aa6 fix(transcode): use csv-parse for CSV parsing and pin pdfkit exact version 2026-07-19 22:46:13 +08:00