mirror of
https://github.com/shumaiOne/shumai.git
synced 2026-11-22 21:27:40 +00:00
c55804a548e48fc670a84eac0b69a413feafa54c
13
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
35a0e82284 |
refactor(agent)!: migrate to Pi Durable and embed the agent queue (#525)
* refactor(agent)!: migrate sessions to Pi Durable Replace patched AgentHarness storage with per-session SQLite and PostgreSQL commit mirroring. Migrate legacy history automatically and run agent queues with the main server. BREAKING CHANGE: Remove standalone agent workers and return native Pi entry records from chat history APIs. * fix(agent): constrain session log dialog scrolling Keep the header visible and wrap long output while log entries and expanded records scroll within the dialog. Cover desktop and narrow viewports through the dashboard Sessions flow. * fix(agent): hide entry IDs from session log card headers Retain internal entry IDs in expanded raw records for debugging. Verify that card headers omit the IDs while raw JSON preserves them. * test(e2e): handle browser wheel limits in session log checks Verify that a normal wheel event moves the viewport before explicitly scrolling to the final record. Keep the dialog bounds, fixed header, wrapping, and expanded-record assertions. * fix(agent): select model providers without inheriting unrelated auth * fix(webui): prevent resize feedback when toggling agent panel * test(agent): reproduce concurrent comment session races * fix(agent): serialize comment sessions and deduplicate history * docs(changelog): consolidate Pi Durable changes into one entry |
||
|
|
7c297895e5 |
feat(embedding): add compose tiers, docs, and regenerate-embeddings CLI (#480)
* feat(embedding): update compose tiers, docs, and add regenerate-embeddings CLI - Add Lite, Embedding, and Full Docker Compose configurations (and Aliyun mirrors) - Set CONCURRENCY_AGENT to 1 for compose files containing embedding-server - Update README and README_CN with compose options, recommended RAM, and semantic search links - Add LiteRT-LM Embedding Server setup guide in docs/using-shumai/embedding-server.mdx - Add shumai regenerate-embeddings CLI command and EmbeddingService to regenerate embeddings across teams - Add comprehensive test coverage for EmbeddingService * fix(docker): remove basic auth for turntable-renderer in full compose * fix(agent): lock and disable embedding agent when ENABLE_EMBEDDING_AGENT is false Ensure that when ENABLE_EMBEDDING_AGENT is set to 'false', existing embedding agents are automatically disabled on startup, marked as read-only, and blocked from manual API modifications or creations. Additionally guard background task creation and semantic search when the environment variable disables embedding agents. * test(upload): remove flaky status assertion on workflow tasks |
||
|
|
b7271777e0 |
feat(agent): support gemma-embedding-2-openai provider (#475)
* feat(agent): support gemma-embedding-2-openai provider Add support for local LiteRT-LM Gemma Embedding 2 provider with 768-dimensional vector embeddings, environment-configured read-only embedding agents, 16-frame uniform sampling for video chunks, and automatic WebP-to-JPEG conversion. * refactor(agent): remove unused legacy generateEmbeddingActivity * chore(agent): align sharp version to 0.35.4 * refactor(agent): rename embedding server env to GEMMA_EMBEDDING_OPENAI_BASE_URL * refactor(agent,transcode): use native Bun.Archive for video chunk frames * fix(transcode,agent): populate buffer entries in Bun.Archive and validate frame sizes |
||
|
|
55d9596835 |
feat(authz): enforce project-aware roles for agent/skill/mcp permissions (#314)
* feat(authz): enforce project-aware roles for agent/skill/mcp permissions Introduce a shared resolveEffectiveRole helper that mirrors AuthzService precedence rules (projectMember.role override -> team role for team-scoped members -> restricted), and route all agent/skill/mcp runtime permission checks through it. Previously these checks used the raw teamMember.role, which let project-only members execute team-level agents/skills/MCP tools and ignored per-project role overrides. Changes: - authz: add resolveEffectiveRole(teamId, projectId, userId, db) and refactor hasPermission to use it (behavior-preserving) - chat/asset/agent-activity: gate agent usage by the effective project role - read_skill: enforce skill permission via effective role (projectId threaded from the agent session) - mcp-service: checkPermission and buildAgentTools filter by effective role; un-permitted servers' tools are no longer advertised to the model - agent activity: only advertise skills the user's effective role permits - listProjectMembers: filter included agent bots by the requester's effective project role - new GET /projects/:projectId/chat-agents API returning chat agents scoped to the project and the user's effective role; WebUI chat selector now uses it (settings keeps the team-level API) - tests: effective-role resolver, chat/mention/MCP/skill project-role matrices, listProjectChatAgents, listProjectMembers filter, API route * fix(agent): forward projectId to getAgentChatContextActivity in agentChat workflow The agentChat workflow called getAgentChatContextActivity without passing projectId from the task payload, so project-scoped members (scope: 'project') resolved a null effective role in the activity and were denied with 'Insufficient role to use agent' even though the chat selector and chat.ts both authorized them. Forward payload.projectId so the activity enforces the user's effective project role, and assert the forwarding in the workflow unit test (fails without the fix). |
||
|
|
cb1c1c502b |
feat(agent): add permission controls for chat agents (#313)
* feat(agent): add permission controls for chat agents * refactor(agent): extract shared agent permission helpers Consolidate the duplicated role-hierarchy logic into packages/core/src/agent/permissions.ts: - getRoleLevel(): numeric level for a team role (fail closed at 0) - getAgentRequiredLevel(): required level for an agent permission (defaults to reviewer/1) - getAllowedAgentRoles(): inclusive role set for filtering The typed Record over TeamMemberRole makes the mapping exhaustive, so adding a new role fails compilation until all sites are updated. Used by ChatService.startOrContinueChat, AssetService comment mentions, getAgentChatContextActivity, AgentService.listAgents, and TeamService.getTeamMembers, replacing four inline copies of the hierarchy and the allowed-roles conditionals. Also fix the API permission-denial test to mock HTTPException(403) and assert a 403 response instead of a generic Error/500, matching real authzService behavior. * test(agent): fix flaky comment DAG tests by spacing ULID id generation ULID ids have 1ms timestamp granularity; back-to-back comment creates frequently land in the same millisecond, in which case Prisma ULIDs sort by their random suffix instead of creation order. The DAG spine in initializeAgentSessionActivity chains top-level comments by id, so a shuffled order produces paths that omit preceding comments, making the path-to-root assertions flaky (~80% of full-file runs locally). Add a 2ms sleep between consecutive comment creates in the affected tests, guaranteeing strictly increasing ULID timestamps. A 2ms delay is sufficient because the next create always fires at least 2ms after the previous ULID was generated. Production is unaffected: real comments arrive seconds apart via separate requests. |
||
|
|
1b5cbf936a |
feat(mcp): add MCP server support with proxy tool and OAuth flow (#289)
* feat(mcp): add MCP server support with proxy tool and OAuth flow
Add URL-based MCP (Model Context Protocol) server support to the backend:
- DB: McpServer / AgentMcpServer / McpServerCredential models with
typed JSON columns (auth config, cached tools, stored OAuth tokens,
pending auth state) and a migration.
- Core (@shumai/core/src/mcp): DB-backed credential store, OAuth
client provider (authorization_code + client_credentials) with a
Hono-hosted redirect callback, connection lifecycle manager
(Streamable HTTP with SSE fallback, lazy connect, 401 -> needs_auth),
JSON Schema -> AgentTool parameter conversion, in-memory tool registry
with ranked search, the single 'mcp' proxy AgentTool (search /
describe / list / call / connect / status / auth-start), per-server
opt-in direct tools ({server}_{tool}), and McpService (CRUD,
discovery, OAuth, per-agent assignment via AgentMcpServer, and
permission gating on tool execution mirroring skills).
- Agent: chat agents receive the 'mcp' proxy tool (plus direct tools
for opted-in servers) built from their assigned, enabled servers.
- API: /teams/:teamId/mcp/servers CRUD, tools refresh/list, auth
start/complete/status/remove, test, permission, and the public
OAuth callback route; Admin authz via new ResourceType.McpServer;
audit logging.
- DTOs: zod schemas + types for servers, tools, auth, and per-agent
mcpServerIds.
- Tests: DB store, schema conversion, registry/search, service
integration against a real in-process MCP server (v2
@modelcontextprotocol/server), permission gating, rename
invalidation, direct tools, per-agent assignment, API route tests,
and workflow E2E (local + temporal) for proxy and direct modes.
* fix(mcp): persist OAuth discovery snapshot and stamp SEP-2352 issuer
The callback leg of the OAuth flow failed with
AuthorizationServerMismatchError ('discoveryState was not available on
the callback leg') because McpDbStore.savePendingAuth overwrote the
entire pendingAuth JSON column, clobbering the discoverySnapshot that
the provider's saveDiscoveryState() had stored inside it.
- McpDbStore.savePendingAuth now preserves an in-flight discovery
snapshot so the callback leg can satisfy the SEP-2352 authorization-
server issuer check.
- McpOauthProvider now round-trips the issuer stamp derived from the
discovery snapshot: saveClientInformation/saveTokens persist it,
clientInformation()/tokens() return it, and tokens() backfills a
missing stamp for pre-existing credentials (removes the
'stored OAuth credential has no issuer stamp' warning).
- Regression tests: savePendingAuth keeps the snapshot; the provider
stamps/round-trips the issuer on both legs.
* fix(mcp): backfill SEP-2352 issuer stamp on stale OAuth client info
clientInformation() returned stored client info without an issuer field
when it had been dynamically registered before the stamping fix, so the
SDK logged 'stored OAuth credential has no issuer stamp' on every read.
Now the stored branch backfills the issuer from the discovery snapshot
(persisting it) exactly like the tokens() backfill.
* feat(webui): add MCP server configuration and agent assignment UI (#290)
* feat(webui): add MCP server configuration and agent assignment UI
* fix(webui): auto-detect auth for unconfigured MCP servers and show connect button
* fix(mcp): auto-generate state in mcp-oauth-provider when missing in DB
* fix(mcp): persist OAuth discovery snapshot and stamp SEP-2352 issuer
The callback leg of the OAuth flow failed with
AuthorizationServerMismatchError ('discoveryState was not available on
the callback leg') because McpDbStore.savePendingAuth overwrote the
entire pendingAuth JSON column, clobbering the discoverySnapshot that
the provider's saveDiscoveryState() had stored inside it.
- McpDbStore.savePendingAuth now preserves an in-flight discovery
snapshot so the callback leg can satisfy the SEP-2352 authorization-
server issuer check.
- McpOauthProvider now round-trips the issuer stamp derived from the
discovery snapshot: saveClientInformation/saveTokens persist it,
clientInformation()/tokens() return it, and tokens() backfills a
missing stamp for pre-existing credentials (removes the
'stored OAuth credential has no issuer stamp' warning).
- Regression tests: savePendingAuth keeps the snapshot; the provider
stamps/round-trips the issuer on both legs.
* fix(mcp): backfill SEP-2352 issuer stamp on stale OAuth client info
clientInformation() returned stored client info without an issuer field
when it had been dynamically registered before the stamping fix, so the
SDK logged 'stored OAuth credential has no issuer stamp' on every read.
Now the stored branch backfills the issuer from the discovery snapshot
(persisting it) exactly like the tokens() backfill.
* fix(mcp): attach McpOauthProvider for auto-detected OAuth servers and discover tools on auth completion
* fix(webui): fix flex min-h-0 and max-h scrolling on MCP tools dialog
* fix(webui): set explicit flex height on McpToolsDialog for active scrolling
* feat(webui): make MCP server cards clickable to edit and widen server config dialog
* feat(mcp): agent-level server enablement and auto-detected server metadata (#291)
* feat(mcp): agent-level server enablement and auto-detected server metadata
- Remove the server-level `enabled` column from mcp_servers; enable/disable
now happens per-agent via the AgentMcpServer join table (mirrors skills).
- Drop the unique (teamId, name) constraint so duplicate server names are allowed.
- Add a `description` column, auto-filled from client.getServerVersion() on
connect; the server name is auto-filled from the reported title (fallback name)
with a URL-derived placeholder until first connect.
- Remove manual name input: the create dialog is URL-only, and the edit dialog
shows the auto-detected name/description read-only.
- Replace the include/exclude raw-text inputs with the tool list in the edit
dialog, managing only excludeTools; remove includeTools support and the
separate McpToolsDialog component.
- Sanitize server-reported names into safe direct-tool prefixes (spaces/punctuation
become underscores) and use the same sanitization in the proxy resolver.
* fix(mcp): give the edit-dialog tool list a definite height
The Shadcn ScrollArea viewport is h-full, which requires a definite height
on the root. Using max-h-64 left the viewport at auto height, so the root
clipped the list at 256px without engaging its own scrollbar — tools past
the cutoff were unreachable. Use a fixed h-64 container (same pattern as
move-copy-dialog) so the inner scroll area scrolls independently of the
dialog's outer scroll.
* refactor(mcp): fixed header/footer with scrollable content in server config dialog
Mirror the skill config dialog layout: the edit dialog now keeps the header
and footer fixed and scrolls only the content area (ScrollArea flex-1
min-h-0 inside an h-[85vh] flex column). The tools list no longer has its
own nested ScrollArea — it flows inside the content-level scroll, so all
tools are reachable without a nested-scroll conflict. The create dialog
stays compact (URL-only) like the skill add dialog.
Verified headlessly that max-height-based ScrollArea roots do not engage
scrolling (viewport height resolves to auto), which is why the previous
nested list clipped; the flex-based definite-height pattern scrolls.
* feat(mcp): make the endpoint URL immutable after creation
Editing a server's URL effectively points it at a different server, and the
create flow is URL-only anyway — so the edit dialog no longer allows changing
it (delete + re-add covers endpoint changes). The URL is displayed read-only
next to the auto-detected name/description, and `url` was removed from the
update DTO and service; auth/transport changes still invalidate credentials
and cached tools.
* refactor(mcp): match MCP server cards to the skill card layout
The settings list cards now share the skill card structure: row 1 is the
server name (with direct-mode/transport badges) plus the three-dot menu,
row 2 is the description (falling back to 'No description provided') with
the endpoint URL and last error as small mono lines, and row 3 holds the
connection-status badge and updated date on the left with the permission
selector and the connect button at the right end.
* fix(mcp): stop card menu clicks from opening the edit dialog
The three-dot menu is portaled, so its item clicks bubble through React's
synthetic events to the card's onClick, which opened the edit dialog before
the delete confirmation appeared. Add e.stopPropagation() to every menu
item (matching the skill card pattern).
* fix(mcp): stop Refresh Tools from submitting/closing the config dialog
The Refresh Tools button lived inside the <form> and, like all buttons
without an explicit type, defaulted to type="submit" — clicking it ran the
form submit (save + close) instead of just refreshing. Set type="button".
Also per UX: while refreshing, the button is disabled with a spinning icon,
the per-tool enable/disable switches are disabled, and the success toast is
removed — the button reverting to its idle state is the success signal
(keep the error toast for failures).
* test(mcp): provide explicit redirectUri in McpOauthProvider test helper
* fix(mcp): scope tool registry to agent-assigned servers
The MCP tool registry was a process-global singleton keyed by server id,
so a chat agent could search, describe, and even call tools belonging to
MCP servers that were not assigned to it (e.g. disabled or unassigned
servers), as long as their metadata had been discovered at some point in
the process. This mirrors the disabled-skill bug: the read path never
checked per-agent enablement.
Replace the singleton with a fresh McpToolRegistry built per
buildAgentTools call and warmed from the persisted McpServer.tools DB
cache. Because the registry only ever contains the agent's assigned
servers, unassigned/disabled servers can neither be suggested, searched,
described, nor executed. Live discovery (lazy connect / connect action)
writes into the per-agent registry via ensureConnected(serverId,
registry?).
Adds test-first coverage: suggestions, search, describe and tool-call
leaks across an unassigned same-team server, staleness after
unassignment, and direct-tools scoping.
* feat(mcp): capture and surface server instructions + server-level refresh (#293)
* feat(mcp): capture and surface server instructions
- Add McpServer.instructions column, populated from getInstructions() at
connect time (legacy initialize + modern discover results)
- Surface instructions at three levels like pi-mcp-adapter: truncated
snippet in the mcp proxy tool description, preview in server listings,
and full text via mcp({ instructions: "name" })
- Fix misleading mcp({ server, connect }) usage line to mcp({ connect })
- Share one renderServerListing helper across list and connect modes
- Expose instructions in McpServerInfo DTO and WebUI config card
- Test server supports instructions; service + API tests updated
* refactor(mcp): server-level refresh endpoint, drop agent connect mode
- Replace POST /mcp/servers/:id/tools/refresh with POST /mcp/servers/:id/refresh
returning the refreshed McpServerInfo (server-level, not tools-level)
- Add McpService.refreshServer(): force-close the connection, re-discover and
persist tools, instructions, name/description and status; used by the refresh
endpoint, Test Connection, and post-OAuth completion (replacing discoverTools)
- Remove discoverTools entirely - refreshServer is the single discovery entry
point (testServer and completeAuth route through it)
- Remove refreshTools from UpdateMcpServerRequest/PATCH (dead param)
- Remove mcp({ connect: "name" }) from the agent proxy tool: lazy tool calls
still auto-connect, so agents are unaffected
- WebUI: remove the dialog 'Refresh Tools' button and create-mode background
auto-refresh; add a server-level Refresh button at the left end of the edit
dialog footer; i18n mcp_refresh_server (+ success message)
* feat(mcp): add in-flight request tracking and idle connection cleanup (#294)
* feat(mcp): add in-flight request tracking and idle connection cleanup
* feat(mcp): add background idle cleanup timer to McpService
* feat(mcp): update default background idle cleanup check interval to 30s
* feat(webui): improve MCP server card footer, instructions, and refresh dialog freeze (#295)
* feat(webui): improve MCP server list card, instructions dialog, and refresh freeze UI
* feat(webui): refine MCP card action menu items
* docs(webui): update direct tools mode description in i18n messages
* style(webui): refine Direct Tools Mode layout in MCP server config dialog
* feat(webui): remove tool search box from MCP server config dialog
* fix(mcp): make in-flight idle tracking test deterministic
The 'tracks in-flight requests and safely prunes idle connections' test
relied on a wall-clock sleep (10ms) to guess when the tool handler had
started executing on the server. On slow/loaded CI runners the HTTP round
trip through the real MCP client/server stack can exceed that window, so
resolveSlowTool() fired the no-op placeholder, the gated handler never
completed, and callPromise hung until the 5000ms test timeout.
The handler now signals via a deferred promise when it actually starts,
and the test awaits that signal (racing it against the call promise to
surface early failures) before asserting in-flight state and releasing
the gate. Also gives the test an explicit 10s timeout.
|
||
|
|
97069042b2 | refactor(api): enforce layered architecture by moving db queries to core services | ||
|
|
b0d8ecb589 | refactor(dtos): define AuditAction enum in dtos package to decouple from db | ||
|
|
0324b95c7e | feat(api): add missing audit log calls to API endpoints | ||
|
|
03f8ad90d2 | feat(dashboard): add agent sessions view for team owners | ||
|
|
0a7d5d1f46 | feat(agent): support disabling specific tools for agents | ||
|
|
bea81a207a | fix(settings): support type-safe thinkingLevel and propagate to AgentHarness | ||
|
|
f16a8a7069 | refactor(transcode): relocate TranscodeService to core and flatten API package |