Compare commits

..
13 Commits
Author SHA1 Message Date
yiling 52330869a2 Release v0.4.2 2026-09-03 21:49:37 +08:00
Yiling-J 0c561e9b39 refactor(webui): use semantic system colors in notification settings (#379) 2026-09-03 21:36:28 +08:00
Yiling-J a236cd3a97 feat(provider): add provider and model sync feature with vendored generator (#378)
* feat(provider): vendor and adapt pi model generator

- Vendor generate-models.ts, model-data.ts, and models-dev-reasoning-options.ts from @earendil-works/pi tag v0.82.1
- Adapt imports to use @earendil-works/pi-ai and expose generateModels programmatic function
- Add compare.test.ts to verify vendored generator against original built-in catalog
- Add packages/core/src/provider/generator to eslint ignores

* feat(provider): add provider and model sync api and ui

* feat(webui): add checkbox for existing providers and clean dialog description

* feat(webui): add expand all and collapse all to sync providers dialog

* refactor(provider): remove compare.test.ts and cli overhead from generate-models

* fix(provider): add 30s timeout and strict error handling to sync catalog fetching

* fix(e2e): eliminate drag-and-drop race condition on webkit

Allow requestAnimationFrame collision updates to settle in dragCard helper before releasing pointer, and scope fileCard locator via data-testid to avoid matching sidebar tree items.
2026-09-03 21:01:14 +08:00
Yiling-J c3ef769f51 feat(provider): optimize provider and model configuration management (#377)
* feat(provider): optimize provider and model configuration management

* style(provider): use default tabs variant in provider settings dialog

* style(provider): make tabs list full width

* feat(provider): make model card row clickable and remove pencil edit icon

* feat(settings): persist active settings tab in url hash

* feat(agent): add search box and alphabetical sorting for provider and model selectors

* refactor(providers): remove frontend pagination and display all models in scrollable list

* fix(providers): address code review findings for model editing, i18n, and api protocol propagation
2026-09-03 15:16:16 +08:00
Yiling-J e58dfaa71a feat(agent): consolidate media tools into read_asset and add S3 key support to download_asset (#376)
* feat(agent): consolidate media tools into read_asset and add S3 key support to download_asset

* fix(agent): refine parameter descriptions and remove download URL from read_asset text content

* fix(agent): resolve owning asset and enforce RBAC permissions when downloading by S3 key
2026-09-02 21:51:07 +08:00
Yiling-J 35b57cc0ab feat(agent): support dynamic annotationId resolution across chat and comments (#375)
* feat(agent): improve timestamp precision and proxy selection for video screenshots

* feat(agent): support dynamic annotationId resolution across chat and comments

* fix(agent): scope annotation resolution to asset and format has_markup consistently
2026-09-02 15:17:56 +08:00
Yiling-J 5a754bcb94 feat(agent): improve timestamp precision and proxy selection for video screenshots (#374) 2026-09-01 21:15:28 +08:00
Yiling-J 64b68e7d63 feat(agent): move thread metadata to structured context XML (#373)
* feat(agent): move thread metadata to structured context XML

* feat(core): dynamically inject thread metadata in getSessionEntries
2026-09-01 17:42:56 +08:00
Yiling-J c2f477a391 fix(agent): hide main comment sessions in dashboard sessions list (#372)
* fix(agent): hide main comment sessions in dashboard sessions list

* feat(webui): auto mention comment creator when replying
2026-09-01 16:20:58 +08:00
Yiling-J 9263971653 更新 README_CN.md 以包含 Docker Compose 部署指南
添加了使用 Docker Compose 部署 Shumai 的详细说明和提示词。
2026-09-01 10:25:57 +08:00
Yiling-J 79a479c87c fix(webui): strip styles and preserve multiline text on paste in chat input (#371)
* fix(webui): strip styles and preserve multiline text on paste in chat input

* fix(db): remove custom log wait strategy in test global setup to prevent postgres startup race condition
2026-08-31 21:29:00 +08:00
Yiling-J 134eb8b7fa fix(agent): set empty details in asset tool results to avoid data duplication (#370) 2026-08-31 20:43:05 +08:00
Yiling-J e06478df88 feat(chat): support markups, attachments, and timestamps in agent 1-to-1 chat (#368)
* feat(chat): support markups, attachments, and timestamps in agent 1-to-1 chat

* feat(chat): improve message card layout with inline badges and integrated assets

* fix(chat): enable image attachment thumbnails in 1-to-1 chat cards

* fix(chat): address code review feedback on input state, mentions, i18n, and session deletion

* fix(chat): replace ring with border color on selected message card
2026-08-31 20:11:27 +08:00
213 changed files with 13249 additions and 2592 deletions
+18
View File
@@ -69,6 +69,24 @@ Shumai 也将项目管理融入了创意工作流程。 内置的看板可以帮
以下是使用本地存储运行 Shumai 的快速上手指南。如需了解高级配置(包括 S3 兼容存储配置、Temporal 工作流编排等),请参阅我们的[官方文档](https://docs.shumai.one/getting-started/overview)。
### 方案一:使用 Docker Compose(推荐)
<details>
<summary>复制这个提示词,让agent帮你安装</summary>
```
请帮我在本机使用 Docker Compose 一键部署 Shumai。
请严格参考官方文档:
https://gh-proxy.com/https://raw.githubusercontent.com/shumaiOne/shumai/main/README_CN.md
要求:
1. 只能使用 Docker Compose local方式部署,不要使用其他部署方式。
2. 部署前请先检查本机是否已安装并可正常使用 Docker 和 Docker Compose。
3. 如果本机未安装 Docker,请立即提醒用户先安装 Docker,并停止后续部署操作。不要自行安装 Docker,也不要采用其他方式部署。
4. 获取 `docker-compose-aliyun.yaml` 文件内容时,必须通过 `https://gh-proxy.com` 加速访问,不要直接访问 GitHub Raw 地址。
```
</details>
使用 Docker Compose 是运行 Shumai 最快的方式。您无需克隆仓库,也不需要手动安装依赖包。请确保电脑上已安装 [Docker](https://docs.docker.com/get-docker/) 和 [Docker Compose](https://docs.docker.com/compose/install/),然后按照以下步骤操作:
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/agent-app",
"version": "0.4.1",
"version": "0.4.2",
"private": true,
"type": "module",
"bin": {
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/cli",
"version": "0.4.1",
"version": "0.4.2",
"private": true,
"type": "module",
"bin": {
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/transcode-app",
"version": "0.4.1",
"version": "0.4.2",
"private": true,
"type": "module",
"bin": {
+22 -4
View File
@@ -24,14 +24,32 @@ export async function dragCard(page: Page, source: Locator, target: Locator): Pr
await page.mouse.move(fromX, fromY)
await page.mouse.down()
// Move in several chunks so the dnd-kit sensor activates mid-drag
const dx = toX - fromX
const dy = toY - fromY
const dist = Math.hypot(dx, dy)
// Step past the 10px activation constraint to trigger drag activation
// and give the browser a brief moment for status to become 'dragging'
const activationRatio = dist > 0 ? Math.min(1, 15 / dist) : 1
await page.mouse.move(fromX + dx * activationRatio, fromY + dy * activationRatio)
await page.waitForTimeout(50)
// Move in several chunks so the intermediate dragmove handlers run
const chunks = 8
for (let i = 1; i <= chunks; i++) {
const x = fromX + ((toX - fromX) * i) / chunks
const y = fromY + ((toY - fromY) * i) / chunks
const x = fromX + (dx * i) / chunks
const y = fromY + (dy * i) / chunks
await page.mouse.move(x, y)
await page.waitForTimeout(15)
await page.waitForTimeout(20)
}
// Ensure pointer is directly over target center and allow requestAnimationFrame
// and collision microtasks to settle before triggering drop
await page.mouse.move(toX, toY)
await page.evaluate(
() => new Promise((resolve) => requestAnimationFrame(() => requestAnimationFrame(resolve))),
)
await page.waitForTimeout(100)
await page.mouse.up()
}
+1 -1
View File
@@ -13,7 +13,7 @@ interface CreateShareResponse {
*/
export function fileCard(page: Page, name: string): Locator {
return page
.locator('div.group')
.locator('[data-testid="file-card"]')
.filter({
has: page.locator(`input[value="${name}"]`).or(page.getByText(name, { exact: true })),
})
@@ -12,6 +12,11 @@ test('owner manages resource quotas in settings and resets usage from the dashbo
// 2. Click the Quotas sidebar tab
await page.getByRole('button', { name: /Quotas|配额/i }).click()
// Verify URL contains #quotas and remains after reload
expect(page.url()).toContain('#quotas')
await page.reload()
expect(page.url()).toContain('#quotas')
// 3. Verify Quotas settings card is displayed with empty state
await expect(page.getByRole('heading', { name: /^(Resource Quotas|资源配额)$/i })).toBeVisible()
await expect(page.getByText(/No resource quotas configured|暂未配置资源配额/i)).toBeVisible()
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/web-app",
"version": "0.4.1",
"version": "0.4.2",
"private": true,
"type": "module",
"dependencies": {
+12 -12
View File
@@ -30,36 +30,36 @@ The following tools can be individually toggled in the agent configuration dialo
- **Description:** Allows the agent to load and read skill definitions to perform specialized actions.
- **Why it matters:** Enables the agent to inspect the instructions/configurations of custom skills registered in your workspace before executing them.
### 3. Analyze Image (`analyze_image`)
### 3. Read Asset (`read_asset`)
- **Description:** Allows the agent to view and analyze the visual contents of an image file.
- **Why it matters:** Enables multimodal agents to "see" image assets uploaded to Shumai, allowing them to answer comments or write metadata about images.
- **Description:** Allows the agent to inspect images, extract still frames from video files, render PDF document pages, and read text/markdown documents.
- **Why it matters:** Enables multimodal agents to view visual media assets uploaded to Shumai, extract frame-by-frame video screenshots, inspect document layouts, and answer questions about uploaded creative assets.
### 4. Video Frame Extraction (`screenshot`)
- **Description:** Allows the agent to extract still image frames from video files for analysis.
- **Why it matters:** Enables the agent to perform frame-by-frame analysis on videos (e.g. for comments, annotation, or search).
### 5. Browse Shumai (`list_assets`)
### 4. Browse Shumai (`list_assets`)
- **Description:** Allows the agent to list folders and browse files in Shumai.
- **Why it matters:** Gives the agent visual mapping of the project structure so it can search for other files or locate related assets.
### 6. Create Folder (`create_folder`)
### 5. Create Folder (`create_folder`)
- **Description:** Allows the agent to create new folders in Shumai.
- **Why it matters:** Needed if the agent organizes outputs or places files in custom subdirectory folders.
### 7. Create File (`create_file`)
### 6. Create File (`create_file`)
- **Description:** Allows the agent to create and upload new files to Shumai.
- **Why it matters:** Allows the agent to generate new assets (e.g. text logs, transcoded clips, reports) and save them directly back into your project.
### 8. Create Version (`create_version`)
### 7. Create Version (`create_version`)
- **Description:** Allows the agent to upload and stack a new version of an existing file in Shumai.
- **Why it matters:** Allows the agent to update files in a clean, non-destructive way by stacking revisions rather than creating duplicate files.
### 8. Download Asset (`download_asset`)
- **Description:** Allows the agent to download a workspace asset or specific storage S3 key to the local `.pi` directory for processing with local scripts or bash commands.
- **Why it matters:** Enables agents executing custom skill scripts to fetch files or intermediate transcode artifacts to local disk.
---
## Restricting Tools
+1 -1
View File
@@ -53,7 +53,7 @@ Shumai supports quota enforcement across multiple layers of the AI and agent arc
| **AI Cost ($)** | Dollar expenditure accrued from AI model providers. | Team, Role, or Members |
| **MCP Server Calls** | Number of tool invocations directed to a connected Model Context Protocol (MCP) server. | Target MCP Server ID |
| **Sandboxed Bash Commands** | Shell commands executed by the agent inside the isolated sandbox. | Wildcard pattern (e.g. `npm *`, `*`) |
| **Agent Tool Calls** | Calls to specific native tools (e.g. `analyze_image`, `screenshot`, `create_file`). | Tool name |
| **Agent Tool Calls** | Calls to specific native tools (e.g. `read_asset`, `download_asset`, `create_file`). | Tool name |
### Bash Command Wildcard Matching
+1
View File
@@ -74,6 +74,7 @@ export default tseslint.config(
'**/packages/db/src/generated/**',
'**/packages/core/src/generated/**',
'**/data/**',
'**/packages/core/src/provider/generator/**',
'**/packages/webui/components/ui/**',
'**/packages/webui/paraglide/**',
'**/packages/webui/public/**',
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "shumai",
"version": "0.4.1",
"version": "0.4.2",
"private": true,
"type": "module",
"repository": {
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/agent",
"version": "0.4.1",
"version": "0.4.2",
"private": true,
"type": "module",
"main": "./src/index.ts",
+81 -5
View File
@@ -711,7 +711,7 @@ describe('Agent Database Activities Integration', () => {
expect(threadSession1?.userCommentId).toBe(mainComment1.id) // Thread Session has userCommentId = rootComment.id
})
it('should dynamically tag top-level comments with [Thread ID: id] even if thread was created after initial sync', async () => {
it('should dynamically tag top-level comments with thread metadata in context even if thread was created after initial sync', async () => {
const msg3 = await prisma.assetComment.create({
data: { assetId: asset.id, message: 'Question in msg3', creatorId: user.id },
})
@@ -756,8 +756,13 @@ describe('Agent Database Activities Integration', () => {
const msg3Entry = pathEntries.find((e) => e.id.includes(msg3.id))
expect(msg3Entry).toBeDefined()
// eslint-disable-next-line @typescript-eslint/no-explicit-any
expect((msg3Entry as any).details?.thread).toEqual({
id: msg3.id,
replyCount: 1,
})
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const text = (msg3Entry as any).content ?? (msg3Entry as any).message?.content[0]?.text
expect(text).toContain(`[Thread ID: ${msg3.id}] [Replies: 1]`)
expect(text).not.toContain(`[Thread ID: ${msg3.id}]`)
})
it('should create separate thread sessions when agent is mentioned in top-level comments (msg3 & msg5) sharing DAG entries without duplication', async () => {
@@ -979,7 +984,7 @@ describe('Agent Database Activities Integration', () => {
).resolves.not.toThrow()
})
it('should not tag Thread ID when only __CHAT__ placeholder comment exists under root comment', async () => {
it('should not tag thread when only __CHAT__ placeholder comment exists under root comment', async () => {
const rootComment = await prisma.assetComment.create({
data: { assetId: asset.id, message: 'Root comment asking agent', creatorId: user.id },
})
@@ -1006,8 +1011,7 @@ describe('Agent Database Activities Integration', () => {
expect(rootEntry).toBeDefined()
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- entry data structure parsing
const content = (rootEntry as any)?.message?.content?.[0]?.text || ''
expect(content).not.toContain(`[Thread ID: ${rootComment.id}]`)
expect((rootEntry as any)?.details?.thread).toBeUndefined()
})
it('should return user name and role for team member', async () => {
@@ -1696,5 +1700,77 @@ describe('Agent Database Activities Integration', () => {
'<project_instructions path="/subfolder/AGENTS.md">\n# Subfolder Policy\n</project_instructions>',
)
})
it('agentChatActivity injects # Comment Threads into systemPrompt when userCommentId is present', async () => {
const user = await prisma.user.create({
data: { name: 'User_Comment_Prompt', email: `comment-prompt-${Date.now()}@example.com` },
})
const team = await prisma.team.create({
data: { name: 'Team_Comment_Prompt_' + Date.now() },
})
const project = await prisma.project.create({
data: { name: 'Project_Comment_Prompt', teamId: team.id },
})
const asset = await prisma.asset.create({
data: {
name: 'Asset_1',
type: AssetType.file,
projectId: project.id,
status: AssetStatus.uploaded,
},
})
const comment = await prisma.assetComment.create({
data: {
assetId: asset.id,
message: 'Question',
creatorId: user.id,
},
})
const mockHarness = {
subscribe: vi.fn(),
prompt: vi.fn().mockResolvedValue({
content: [{ type: 'text', text: 'Chat reply' }],
usage: { input: 10, output: 20 },
}),
}
const mockSession = {
getEntries: vi.fn().mockResolvedValue([]),
getStorage: vi.fn().mockReturnValue({ sessionId: 'mock-session-id' }),
}
vi.mocked(piAgent.createAgentSession).mockResolvedValue({
session: mockSession as unknown as Session<DatabaseSessionMetadata>,
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- Mock AgentHarness instance for activity test
harness: mockHarness as unknown as AgentHarness<any, any, any, any>,
})
const context = {
agent: { id: 'agent-1', provider: { name: 'google' }, modelRef: { modelId: 'gemini' } },
dbProviders: [],
teamSkills: [],
allowedDomains: [],
} as unknown as AgentExecutionContext
await agentChatActivity({
teamId: team.id,
agentId: 'agent-1',
message: 'Hello',
imageUrls: [],
projectId: project.id,
folderId: asset.id,
assetId: asset.id,
sessionId: 'mock-session-id',
userCommentId: comment.id,
userId: user.id,
context,
})
const lastCall = vi.mocked(piAgent.createAgentSession).mock.calls.at(-1)
const systemPromptArg = lastCall?.[0].systemPrompt as string
expect(systemPromptArg).toContain('# Comment Threads')
expect(systemPromptArg).toContain('<thread id="..." reply_count="..." />')
expect(systemPromptArg).toContain('read_thread')
})
})
})
+74 -6
View File
@@ -118,7 +118,7 @@ If you need to create files in the local filesystem (for example, a temporary fi
When creating a file or version, you may attach metadata (for example, the AI model or prompt used to generate the asset). The allowed field keys and value types are declared directly in the 'metadata' parameter of the 'create_file' and 'create_version' tools.
# Message Context & Tools
User messages may contain a <context> block detailing the user, active asset location, playback position, and attachments. Use the asset IDs from <context> when invoking tools (e.g. 'analyze_image', 'screenshot', 'read_pdf_pages', 'download_asset', 'list_assets'). When an <annotation /> tag is present, it indicates the user has drawn visual markups on the asset at the specified position.`
User messages may contain a <context> block detailing the user, active asset location, playback position, and attachments. Use the asset IDs from <context> when invoking tools (e.g. 'read_asset', 'download_asset', 'list_assets'). When an <annotation id="..." /> tag is present, it indicates the user has drawn visual markups on the asset at the specified position. When calling 'read_asset', pass the 'annotationId' parameter with that ID to view the image or rendered pages with the visual markups overlaid.`
if (agent.soul) {
systemPrompt = `${systemPrompt}\n\nAgent Personality and Core Instructions:\n${agent.soul}`
@@ -163,6 +163,21 @@ User messages may contain a <context> block detailing the user, active asset loc
systemPrompt += `\n\nYour current model supports the following input types: ${modelConfig.input.join(', ')}.`
}
const isCommentMode =
!!params.userCommentId ||
(params.sessionId
? (
await prisma.agentSession.findUnique({
where: { id: params.sessionId },
select: { type: true },
})
)?.type === 'comment'
: false)
if (isCommentMode) {
systemPrompt += `\n\n# Comment Threads\nWhen operating in comment mode, previous top-level messages in the conversation history may contain a <thread id="..." reply_count="..." /> tag indicating an earlier discussion thread with replies. If a user's question refers to or depends on earlier comments or discussions, you can use the 'read_thread' tool with the thread ID to inspect all replies in that thread.`
}
const agentConfig = agent.config as PrismaJson.AgentConfig | null | undefined
const thinkingLevel = agentConfig?.thinkingLevel || 'off'
@@ -809,9 +824,15 @@ export async function initializeAgentSessionActivity(params: {
}
const commentDetails: ShumaiMessageContext = {
id: c.id,
...(userObj ? { user: userObj } : {}),
...(position ? { position } : {}),
...(annotation ? { annotation: true } : {}),
...(annotation
? {
annotation: true,
annotations: c.annotation as unknown as ShumaiMessageContext['annotations'],
}
: {}),
...(attachedFiles.length > 0 ? { attachedFiles } : {}),
}
@@ -1190,6 +1211,7 @@ export async function getAssetActivity(assetId: string) {
},
})
let resolvedAsset = asset
if (asset && asset.type === AssetType.version_stack) {
const latestVersion = await prisma.asset.findFirst({
where: { parentId: asset.id, isDeleted: false },
@@ -1205,18 +1227,36 @@ export async function getAssetActivity(assetId: string) {
})
if (latestVersion) {
return {
resolvedAsset = {
...latestVersion,
project: latestVersion.project ?? asset.project,
}
} as typeof asset
}
}
return asset
if (!resolvedAsset) return null
let url: string | undefined
if (resolvedAsset.storageKey?.key) {
try {
url = await s3Service.presign(
process.env.S3_BUCKET || 'shumai',
resolvedAsset.storageKey.key,
'GET',
)
} catch {
// Ignore presign errors
}
}
return {
...resolvedAsset,
url,
}
}
export async function getCommentActivity(commentId: string) {
return prisma.assetComment.findUnique({
const comment = await prisma.assetComment.findUnique({
where: { id: commentId },
include: {
attachments: {
@@ -1224,6 +1264,34 @@ export async function getCommentActivity(commentId: string) {
},
},
})
if (!comment) return null
const attachmentsWithUrl = await Promise.all(
comment.attachments.map(async (att) => {
let url: string | undefined
if (att.asset?.storageKey?.key) {
try {
url = await s3Service.presign(
process.env.S3_BUCKET || 'shumai',
att.asset.storageKey.key,
'GET',
)
} catch {
// Ignore
}
}
return {
...att,
asset: att.asset ? { ...att.asset, url } : null,
}
}),
)
return {
...comment,
attachments: attachmentsWithUrl,
}
}
export async function getProjectAutofillFieldsActivity(projectId: string) {
@@ -31,6 +31,25 @@ describe('serializeContextToXml', () => {
)
})
it('serializes context with user and thread tag with reply_count', () => {
const context: ShumaiMessageContext = {
user: { id: 'usr_1', name: 'Alice', role: 'editor' },
thread: { id: 'th_01JXYZ', replyCount: 3 },
}
expect(serializeContextToXml(context)).toBe(
'<context>\n <user id="usr_1" name="Alice" role="editor" />\n <thread id="th_01JXYZ" reply_count="3" />\n</context>',
)
})
it('serializes context with thread tag without reply_count', () => {
const context: ShumaiMessageContext = {
thread: { id: 'th_01JXYZ' },
}
expect(serializeContextToXml(context)).toBe(
'<context>\n <thread id="th_01JXYZ" />\n</context>',
)
})
it('serializes minimal context with only currentAsset (without ancestors)', () => {
const context: ShumaiMessageContext = {
currentAsset: {
@@ -100,7 +119,7 @@ describe('serializeContextToXml', () => {
<folder id="ast_01JC03_PARENT" name="Footage" />
</ancestors>
</current_asset>
<position type="time" seconds="14.25" />
<position type="time" seconds="14.2500" />
<annotation />
<attached_files>
<file id="ast_01JC05_ATT1" name="brand_guidelines_2026.pdf" type="file" media_type="pdf" path="Brand/brand_guidelines_2026.pdf" />
@@ -296,7 +315,7 @@ describe('serializeContextToXml', () => {
expect(refOrder).toEqual(['a_asset', 'm_asset', 'z_asset'])
})
it('normalizes float formatting with fixed .toFixed(2)', () => {
it('normalizes float formatting with fixed precision (toFixed(4) for position, toFixed(2) for duration)', () => {
const context1: ShumaiMessageContext = {
currentAsset: { id: '1', name: 'a', type: 'file', durationSeconds: 14 },
position: { type: 'time', seconds: 5.5 },
@@ -307,9 +326,29 @@ describe('serializeContextToXml', () => {
}
expect(serializeContextToXml(context1)).toContain('duration_seconds="14.00"')
expect(serializeContextToXml(context1)).toContain('seconds="5.50"')
expect(serializeContextToXml(context1)).toContain('seconds="5.5000"')
expect(serializeContextToXml(context2)).toContain('duration_seconds="14.00"')
expect(serializeContextToXml(context2)).toContain('seconds="5.50"')
expect(serializeContextToXml(context2)).toContain('seconds="5.5000"')
})
it('serializes annotation tag with id when context.id is provided', () => {
const context: ShumaiMessageContext = {
id: '01JABCDEF1234567890',
annotation: true,
}
expect(serializeContextToXml(context)).toBe(
'<context>\n <annotation id="01JABCDEF1234567890" />\n</context>',
)
})
it('escapes special XML characters in annotation id', () => {
const context: ShumaiMessageContext = {
id: 'test<id>&"123"',
annotation: true,
}
expect(serializeContextToXml(context)).toBe(
'<context>\n <annotation id="test&lt;id&gt;&amp;&quot;123&quot;" />\n</context>',
)
})
})
})
@@ -7,8 +7,7 @@ import { DatabaseSessionStorage } from './database-session-storage'
import { agentService } from '@shumai/core/src/agent/agent'
import { createAgentSession, type DbProviderInfo } from './index'
import * as sandboxedBashModule from './tools/sandboxed-bash'
import * as analyzeImageModule from './tools/analyze-image'
import * as screenshotModule from './tools/screenshot'
import * as readAssetModule from './tools/read-asset'
describe('DatabaseSessionStorage', () => {
setupTestDbHooks()
@@ -451,9 +450,8 @@ describe('DatabaseSessionStorage', () => {
})
const sessionId = initialStorage.sessionId
// 2. Spy on media tools constructors
const createAnalyzeImageToolSpy = vi.spyOn(analyzeImageModule, 'createAnalyzeImageTool')
const createScreenshotToolSpy = vi.spyOn(screenshotModule, 'createScreenshotTool')
// 2. Spy on media tool constructor
const createReadAssetToolSpy = vi.spyOn(readAssetModule, 'createReadAssetTool')
// 3. Resume the session via createAgentSession with a NEW comment ID
await createAgentSession({
@@ -470,12 +468,11 @@ describe('DatabaseSessionStorage', () => {
providers: mockProviders,
})
// 4. Assert that the tools were instantiated using the new comment ID and user ID
expect(createAnalyzeImageToolSpy).toHaveBeenCalledWith(user.id, 'new-comment-456')
expect(createScreenshotToolSpy).toHaveBeenCalledWith(user.id, 'new-comment-456')
// 4. Assert that the tool was instantiated using user ID
expect(createReadAssetToolSpy).toHaveBeenCalledWith(user.id)
})
it('should use the current comment ID for media tools in subsequent turns (video)', async () => {
it('should instantiate media tools with user ID in subsequent turns (video)', async () => {
const { agent, user, team } = await setupTestData()
// Create a project and asset to satisfy FK constraints
@@ -505,9 +502,8 @@ describe('DatabaseSessionStorage', () => {
})
const sessionId = initialStorage.sessionId
// 2. Spy on media tools constructors
const createAnalyzeImageToolSpy = vi.spyOn(analyzeImageModule, 'createAnalyzeImageTool')
const createScreenshotToolSpy = vi.spyOn(screenshotModule, 'createScreenshotTool')
// 2. Spy on media tool constructor
const createReadAssetToolSpy = vi.spyOn(readAssetModule, 'createReadAssetTool')
// 3. Resume the session via createAgentSession with a NEW comment ID
await createAgentSession({
@@ -524,9 +520,8 @@ describe('DatabaseSessionStorage', () => {
providers: mockProviders,
})
// 4. Assert that the tools were instantiated using the new comment ID and user ID
expect(createScreenshotToolSpy).toHaveBeenCalledWith(user.id, 'new-comment-456')
expect(createAnalyzeImageToolSpy).toHaveBeenCalledWith(user.id, 'new-comment-456')
// 4. Assert that the tool was instantiated using user ID
expect(createReadAssetToolSpy).toHaveBeenCalledWith(user.id)
})
describe('P1 Bug Reproductions', () => {
@@ -742,5 +737,78 @@ describe('DatabaseSessionStorage', () => {
const stats = await storage.getSessionStats()
expect(stats.messageCount).toBe(1)
})
it('should dynamically inject thread metadata into details.thread in getPathToRoot when replies exist', async () => {
const { agent, user } = await setupTestData()
const project = await prisma.project.create({
data: { name: 'Test Project', teamId: agent.teamId },
})
const asset = await prisma.asset.create({
data: {
name: 'Test Asset',
type: 'file',
status: 'uploaded',
projectId: project.id,
},
})
const storage = await DatabaseSessionStorage.create({ agentId: agent.id, assetId: asset.id })
const rootComment = await prisma.assetComment.create({
data: {
id: 'comment-root-1',
assetId: asset.id,
message: 'Top-level question',
creatorId: user.id,
},
})
// Add a reply
await prisma.assetComment.create({
data: {
id: 'comment-reply-1',
assetId: asset.id,
message: 'Reply 1',
creatorId: user.id,
replyToId: rootComment.id,
},
})
// Create session entry for root comment
await prisma.agentSessionEntry.create({
data: {
id: rootComment.id,
sessionId: storage.sessionId,
assetId: asset.id,
type: 'custom_message',
data: {
customType: 'shumai_message',
content: 'Top-level question',
display: true,
details: { user: { id: user.id, name: user.name } },
},
},
})
const path = await storage.getPathToRoot(rootComment.id)
expect(path).toHaveLength(1)
expect(path[0].type).toBe('custom_message')
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const details = (path[0] as any).details
expect(details.thread).toEqual({
id: rootComment.id,
replyCount: 1,
})
// eslint-disable-next-line @typescript-eslint/no-explicit-any
expect((path[0] as any).content).toBe('Top-level question')
const entries = await storage.getEntries()
expect(entries).toHaveLength(1)
expect(entries[0].type).toBe('custom_message')
// eslint-disable-next-line @typescript-eslint/no-explicit-any
expect((entries[0] as any).details?.thread).toEqual({
id: rootComment.id,
replyCount: 1,
})
})
})
})
+56 -54
View File
@@ -8,6 +8,7 @@ import {
type SessionTreeEntry,
} from '@earendil-works/pi-agent-core'
import { agentService } from '@shumai/core/src/agent/agent'
import type { ShumaiMessageContext } from '@shumai/dtos'
export interface DatabaseSessionMetadata extends SessionMetadata {
agentId: string
@@ -131,7 +132,10 @@ export class DatabaseSessionStorage implements SessionStorage<DatabaseSessionMet
customType: 'shumai_message',
content: contentText,
display: true,
details: this.currentMessageContext,
details: {
...this.currentMessageContext,
id: entryToProcess.id,
},
} as unknown as SessionTreeEntry
this.currentMessageContext = undefined
}
@@ -366,59 +370,7 @@ export class DatabaseSessionStorage implements SessionStorage<DatabaseSessionMet
await this.reinjectSkillContentAsync(entry)
}
// Dynamically tag comments that currently have reply threads
const entryIds = pathEntries.map((e) => e.id)
if (entryIds.length > 0) {
const threadCounts = await prisma.assetComment.groupBy({
by: ['replyToId'],
where: {
replyToId: { in: entryIds },
message: { not: '__CHAT__' },
},
_count: {
id: true,
},
})
const threadReplyCountMap = new Map<string, number>()
for (const tc of threadCounts) {
if (tc.replyToId) {
threadReplyCountMap.set(tc.replyToId, tc._count.id)
}
}
if (threadReplyCountMap.size > 0) {
for (let i = 0; i < pathEntries.length; i++) {
const entry = pathEntries[i]
const count = threadReplyCountMap.get(entry.id)
if (count !== undefined && count > 0) {
if (entry.type === 'message' && entry.message) {
const cloned = structuredClone(entry)
const msg = cloned.message as unknown as {
content?: Array<{ type: string; text: string }>
}
if (Array.isArray(msg.content) && msg.content[0] && msg.content[0].type === 'text') {
const threadTag = `[Thread ID: ${entry.id}] [Replies: ${count}]`
if (!msg.content[0].text.includes(`[Thread ID: ${entry.id}]`)) {
msg.content[0].text = `${threadTag} ${msg.content[0].text}`
}
}
pathEntries[i] = cloned
} else if (entry.type === 'custom_message') {
const cloned = structuredClone(entry)
const threadTag = `[Thread ID: ${entry.id}] [Replies: ${count}]`
if (typeof cloned.content === 'string') {
if (!cloned.content.includes(`[Thread ID: ${entry.id}]`)) {
cloned.content = `${threadTag} ${cloned.content}`
}
}
pathEntries[i] = cloned
}
}
}
}
}
return pathEntries
return this.injectThreadMetadataAsync(pathEntries)
}
async getEntries(): Promise<SessionTreeEntry[]> {
@@ -431,6 +383,56 @@ export class DatabaseSessionStorage implements SessionStorage<DatabaseSessionMet
await this.reinjectImageDataAsync(entry)
await this.reinjectSkillContentAsync(entry)
}
return this.injectThreadMetadataAsync(entries)
}
private async injectThreadMetadataAsync(
entries: SessionTreeEntry[],
): Promise<SessionTreeEntry[]> {
const entryIds = entries.map((e) => e.id)
if (entryIds.length === 0) {
return entries
}
const threadCounts = await prisma.assetComment.groupBy({
by: ['replyToId'],
where: {
replyToId: { in: entryIds },
message: { not: '__CHAT__' },
},
_count: {
id: true,
},
})
const threadReplyCountMap = new Map<string, number>()
for (const tc of threadCounts) {
if (tc.replyToId) {
threadReplyCountMap.set(tc.replyToId, tc._count.id)
}
}
if (threadReplyCountMap.size > 0) {
for (let i = 0; i < entries.length; i++) {
const entry = entries[i]
const count = threadReplyCountMap.get(entry.id)
if (count !== undefined && count > 0) {
if (entry.type === 'custom_message') {
const cloned = structuredClone(entry)
const existingDetails = ((cloned as unknown as { details?: ShumaiMessageContext })
.details || {}) as ShumaiMessageContext
;(cloned as unknown as { details: ShumaiMessageContext }).details = {
...existingDetails,
thread: {
id: entry.id,
replyCount: count,
},
}
entries[i] = cloned
}
}
}
}
return entries
}
+28
View File
@@ -1257,3 +1257,31 @@ describe('createAgentSession enabled skill filtering', () => {
}
})
})
describe('getModelFromDb', () => {
it('should prioritize provider active api protocol over model config api', () => {
const providers: DbProviderInfo[] = [
{
name: 'custom-provider',
config: { api: 'anthropic-messages', apiKey: 'KEY' },
models: [
{
modelId: 'my-model',
name: 'My Model',
config: {
api: 'openai-responses', // stale model config
reasoning: false,
input: ['text'],
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
contextWindow: 128000,
maxTokens: 4096,
},
},
],
},
]
const model = getModelFromDb(providers, 'custom-provider', 'my-model')
expect(model.api).toBe('anthropic-messages')
})
})
+3 -13
View File
@@ -19,17 +19,15 @@ import * as os from 'os'
import * as path from 'path'
import { DatabaseSessionStorage } from './database-session-storage'
import { metadataService } from '@shumai/core/src/metadata/metadata'
import { createAnalyzeImageTool } from './tools/analyze-image'
import { createCreateFileTool } from './tools/create-file'
import { createCreateFolderTool } from './tools/create-folder'
import { createCreateVersionTool } from './tools/create-version'
import { createDownloadAssetTool } from './tools/download-asset'
import { createListAssetsTool } from './tools/list-assets'
import { createReadPdfPagesTool } from './tools/read-pdf-pages'
import { createReadAssetTool } from './tools/read-asset'
import { createReadSkillTool } from './tools/read-skill'
import { createReadThreadTool } from './tools/read-thread'
import { createSandboxedBashTool } from './tools/sandboxed-bash'
import { createScreenshotTool } from './tools/screenshot'
import { serializeContextToXml } from './context/serialize-context'
import type { ShumaiMessageContext } from '@shumai/dtos'
@@ -72,10 +70,10 @@ export function getModelFromDb(providers: DbProviderInfo[], providerName: string
if (dbProvider.config.baseUrl) {
m.baseUrl = dbProvider.config.baseUrl
}
Object.assign(m, dbModel.config)
if (dbProvider.config.api) {
m.api = dbProvider.config.api
}
Object.assign(m, dbModel.config)
if (dbModel.name) {
m.name = dbModel.name
}
@@ -312,17 +310,9 @@ export async function createAgentSession(params: CreateAgentSessionParams) {
}
}
const metadata = await storage.getMetadata()
const userCommentId =
passedUserCommentId !== undefined ? passedUserCommentId : metadata.userCommentId
const mediaTools: AgentTool[] = []
if (userId) {
mediaTools.push(
createAnalyzeImageTool(userId, userCommentId),
createScreenshotTool(userId, userCommentId),
createReadPdfPagesTool(userId, userCommentId),
)
mediaTools.push(createReadAssetTool(userId))
}
const agent = await prisma.agent.findUnique({
@@ -76,7 +76,13 @@ describe('reproduce null coercion in harness tool validation', () => {
// execute must accept the valid call (data-only) without throwing
const result = await tool.execute('call-1', validated as never)
expect(result.details).toEqual({ id: 'file-1', name: 'test', type: 'file', size: 100 })
expect(result.details).toEqual({})
expect(JSON.parse((result.content[0] as { text: string }).text)).toEqual({
id: 'file-1',
name: 'test',
type: 'file',
size: 100,
})
expect(s3Service.putObject).toHaveBeenCalledTimes(1)
expect(authzService.hasPermission).toHaveBeenCalledTimes(1)
expect(assetService.createFile).toHaveBeenCalledTimes(1)
@@ -1,175 +0,0 @@
import { describe, it, expect, vi, beforeEach } from 'vitest'
import { createAnalyzeImageTool } from './analyze-image'
import {
prisma,
WorkflowTaskStatus,
type Asset,
type AssetComment,
type User,
type WorkflowTask,
} from '@shumai/db'
import { s3Service } from '@shumai/core/src/s3/s3'
import { workflowService } from '@shumai/workflow-core'
import { authzService } from '@shumai/core/src/authz/authz'
vi.mock('@shumai/db', async (importOriginal) => {
const actual = await importOriginal<typeof import('@shumai/db')>()
return {
...actual,
prisma: {
user: {
findUnique: vi.fn(),
},
asset: {
findUnique: vi.fn(),
},
assetComment: {
findUnique: vi.fn(),
},
workflowTask: {
create: vi.fn(),
},
},
}
})
vi.mock('@shumai/core/src/s3/s3', () => ({
s3Service: {
getObject: vi.fn(),
},
}))
vi.mock('@shumai/workflow-core', () => ({
workflowService: {
executeWait: vi.fn(),
},
}))
vi.mock('@shumai/core/src/authz/authz', () => ({
authzService: {
hasPermission: vi.fn(),
},
Permission: { Read: 'Read' },
ResourceType: { Asset: 'asset' },
}))
describe('analyzeImageTool', () => {
beforeEach(() => {
vi.clearAllMocks()
})
it('should check user permissions and fetch the image from S3 when comment has no annotations', async () => {
vi.mocked(prisma.user.findUnique).mockResolvedValue({ id: 'user-1' } as User)
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'asset-1',
projectId: 'project-1',
storageKey: { key: 'raw-image.png' },
media: null,
} as unknown as Asset)
vi.mocked(prisma.assetComment.findUnique).mockResolvedValue({
id: 'comment-1',
assetId: 'asset-1',
annotation: null,
} as unknown as AssetComment)
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('fake-image-bytes'),
contentType: 'image/png',
} as unknown as { buffer: Buffer; contentType: string })
const tool = createAnalyzeImageTool('user-1', 'comment-1')
const result = await tool.execute('call-1', { assetId: 'asset-1' })
expect(authzService.hasPermission).toHaveBeenCalledWith({
user: { id: 'user-1' },
permission: 'Read',
type: 'asset',
id: 'asset-1',
})
expect(result.content[0].type).toBe('image')
expect((result.content[0] as unknown as { data: string }).data).toBe(
Buffer.from('fake-image-bytes').toString('base64'),
)
expect(result.details.sourceKeys).toEqual(['raw-image.png'])
expect(prisma.workflowTask.create).not.toHaveBeenCalled()
})
it('should trigger overlay transcode workflow when comment has annotations', async () => {
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'asset-1',
projectId: 'project-1',
storageKey: { key: 'raw-image.png' },
media: {
imageTranscodes: [{ key: 'transcoded-image.webp' }],
},
} as unknown as Asset)
vi.mocked(prisma.assetComment.findUnique).mockResolvedValue({
id: 'comment-1',
assetId: 'asset-1',
annotation: [
{
type: 'box',
color: '#ff0000',
points: [
[0, 0],
[1, 1],
],
},
],
} as unknown as AssetComment)
vi.mocked(prisma.workflowTask.create).mockResolvedValue({
id: 'task-1',
} as unknown as WorkflowTask)
vi.mocked(workflowService.executeWait).mockResolvedValue({
id: 'task-1',
status: WorkflowTaskStatus.completed,
output: { key: 'annotated-image.webp' },
} as unknown as WorkflowTask)
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('fake-annotated-image-bytes'),
contentType: 'image/webp',
} as unknown as { buffer: Buffer; contentType: string })
const tool = createAnalyzeImageTool('user-1', 'comment-1')
const result = await tool.execute('call-1', { assetId: 'asset-1' })
expect(prisma.workflowTask.create).toHaveBeenCalledWith({
data: {
assetId: 'asset-1',
projectId: 'project-1',
type: 'transcode_image_annotation',
status: 'pending',
payload: {
projectId: 'project-1',
imageAnnotation: {
annotations: [
{
type: 'box',
color: '#ff0000',
points: [
[0, 0],
[1, 1],
],
},
],
},
},
},
})
expect(result.content[0].type).toBe('image')
expect((result.content[0] as unknown as { data: string }).data).toBe(
Buffer.from('fake-annotated-image-bytes').toString('base64'),
)
expect(result.details.sourceKeys).toEqual(['annotated-image.webp'])
})
})
-139
View File
@@ -1,139 +0,0 @@
import { Type } from 'typebox'
import { type AgentTool } from '@earendil-works/pi-agent-core'
import { type ImageContent } from '@earendil-works/pi-ai'
import { prisma, WorkflowTaskType, WorkflowTaskStatus, type User } from '@shumai/db'
import { s3Service } from '@shumai/core/src/s3/s3'
import { workflowService } from '@shumai/workflow-core'
import { authzService, Permission, ResourceType } from '@shumai/core/src/authz/authz'
const analyzeImageSchema = Type.Object({
assetId: Type.String({
description: 'The asset ID of the image to analyze. This parameter is required.',
}),
})
export function createAnalyzeImageTool(
userId: string,
userCommentId?: string | null,
): AgentTool<typeof analyzeImageSchema> {
return {
name: 'analyze_image',
label: 'Analyze Image',
description:
'Retrieves and views the image content. Call this tool if you need to analyze the image.',
parameters: analyzeImageSchema,
execute: async (_toolCallId, params) => {
const assetId = params.assetId
if (!userId) {
throw new Error('User ID is required for authorization.')
}
await authzService.hasPermission({
user: { id: userId } as User,
permission: Permission.Read,
type: ResourceType.Asset,
id: assetId,
})
let asset = await prisma.asset.findUnique({
where: { id: assetId },
include: { storageKey: true },
})
if (!asset) {
throw new Error(`Asset with ID ${assetId} not found.`)
}
if (asset.type === 'version_stack') {
const latestVersion = await prisma.asset.findFirst({
where: { parentId: asset.id, isDeleted: false },
orderBy: { sortIndex: 'asc' },
include: { storageKey: true },
})
if (latestVersion) {
asset = latestVersion
}
}
// Get default media key
let mediaKey = asset.storageKey?.key
if (asset.media) {
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const mediaInfo = asset.media as any
if (mediaInfo.imageTranscodes && mediaInfo.imageTranscodes.length > 0) {
mediaKey = mediaInfo.imageTranscodes[0].key || mediaKey
}
}
if (!mediaKey) {
throw new Error('No media content found for this asset.')
}
// Check if trigger comment has draw annotations for this asset
let annotations: unknown = null
if (userCommentId) {
const comment = await prisma.assetComment.findUnique({
where: { id: userCommentId },
})
if (comment && comment.assetId === assetId && comment.annotation) {
const list = comment.annotation
if (Array.isArray(list) && list.length > 0) {
annotations = list
}
}
}
let keyToUse = mediaKey
if (annotations) {
// Trigger transcode workflow to draw annotation
const task = await prisma.workflowTask.create({
data: {
assetId,
projectId: asset.projectId || 'none',
type: WorkflowTaskType.transcode_image_annotation,
status: WorkflowTaskStatus.pending,
payload: {
projectId: asset.projectId || 'none',
imageAnnotation: {
annotations: annotations as PrismaJson.AnnotationList,
},
},
},
})
const completedTask = await workflowService.executeWait(task)
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const output = completedTask.output as any
if (output?.key) {
keyToUse = output.key
} else {
throw new Error('Image annotation transcode workflow failed to output S3 key.')
}
}
// Retrieve the image content from S3
const bucket = process.env.S3_BUCKET || 'shumai'
const { buffer, contentType } = await s3Service.getObject(bucket, keyToUse)
const mimeType =
contentType && contentType !== 'application/octet-stream' ? contentType : 'image/webp'
const content: ImageContent[] = [
{
type: 'image',
data: buffer.toString('base64'),
mimeType,
},
]
return {
content,
details: {
sourceKeys: [keyToUse],
},
}
},
}
}
@@ -0,0 +1,270 @@
import { describe, it, expect, beforeEach } from 'vitest'
import { prisma } from '@shumai/db'
import { setupTestDbHooks } from '@shumai/db/test'
import { resolveAnnotationsById } from './annotation-resolver'
import { DatabaseSessionStorage } from '../database-session-storage'
import type { SessionTreeEntry } from '@earendil-works/pi-agent-core'
describe('resolveAnnotationsById', () => {
setupTestDbHooks()
beforeEach(() => {
// DB hooks automatically manage transactions
})
it('returns null annotations and null timestamp when annotationId is undefined or empty', async () => {
const result1 = await resolveAnnotationsById('asset-1', undefined)
expect(result1).toEqual({ annotations: null, timestamp: null })
const result2 = await resolveAnnotationsById('asset-1', '')
expect(result2).toEqual({ annotations: null, timestamp: null })
})
it('resolves annotations and timestamp from AssetComment (Comments Mode)', async () => {
const user = await prisma.user.create({
data: { name: 'Comment Author', email: 'author@test.com', password: 'pw' },
})
const team = await prisma.team.create({
data: { name: 'Test Team' },
})
const project = await prisma.project.create({
data: { name: 'Test Project', teamId: team.id },
})
const asset = await prisma.asset.create({
data: {
name: 'video.mp4',
type: 'file',
projectId: project.id,
status: 'uploaded',
},
})
const comment = await prisma.assetComment.create({
data: {
assetId: asset.id,
creatorId: user.id,
message: 'Look at the highlighted area',
second: 18.5,
annotation: [
{
type: 'line',
color: '#ff0000',
points: [
[10, 10],
[50, 50],
],
},
],
},
})
const result = await resolveAnnotationsById(asset.id, comment.id)
expect(result.timestamp).toBe(18.5)
expect(result.annotations).toEqual([
{
type: 'line',
color: '#ff0000',
points: [
[10, 10],
[50, 50],
],
},
])
})
it('throws an error when AssetComment belongs to a different asset', async () => {
const user = await prisma.user.create({
data: { name: 'Author', email: 'author2@test.com', password: 'pw' },
})
const team = await prisma.team.create({
data: { name: 'Team' },
})
const project = await prisma.project.create({
data: { name: 'Project', teamId: team.id },
})
const assetA = await prisma.asset.create({
data: { name: 'videoA.mp4', type: 'file', projectId: project.id, status: 'uploaded' },
})
const assetB = await prisma.asset.create({
data: { name: 'videoB.mp4', type: 'file', projectId: project.id, status: 'uploaded' },
})
const commentB = await prisma.assetComment.create({
data: {
assetId: assetB.id,
creatorId: user.id,
message: 'Comment on B',
annotation: [
{
type: 'box',
color: '#ff0000',
points: [
[0, 0],
[10, 10],
],
},
],
},
})
await expect(resolveAnnotationsById(assetA.id, commentB.id)).rejects.toThrow(
`Annotation "${commentB.id}" belongs to asset "${assetB.id}", not target asset "${assetA.id}".`,
)
})
it('resolves annotations and timestamp from AgentSessionEntry (1-on-1 Chat Mode)', async () => {
const user = await prisma.user.create({
data: { name: 'Chat User', email: 'chatuser@test.com', password: 'pw' },
})
const team = await prisma.team.create({
data: { name: 'Chat Team' },
})
const project = await prisma.project.create({
data: { name: 'Chat Project', teamId: team.id },
})
const asset = await prisma.asset.create({
data: {
name: 'chat-video.mp4',
type: 'file',
projectId: project.id,
status: 'uploaded',
},
})
const agentUser = await prisma.user.create({
data: { name: 'Chat Bot User', email: 'bot@example.com', type: 'agent' },
})
const agent = await prisma.agent.create({
data: {
id: agentUser.id,
teamId: team.id,
type: 'chat',
config: { provider: 'test-provider', model: 'test-model' },
},
})
// Create session storage
const storage = await DatabaseSessionStorage.create({
agentId: agent.id,
userId: user.id,
assetId: asset.id,
})
// Set active message context with markup annotations and position
storage.currentMessageContext = {
currentAsset: {
id: asset.id,
name: asset.name,
type: 'file',
path: asset.name,
},
position: { type: 'time', seconds: 12.34 },
annotation: true,
annotations: [
{
type: 'rectangle',
x: 100,
y: 200,
width: 50,
height: 50,
},
],
}
// User sends a message -> DatabaseSessionStorage.appendEntry is called
const userEntry: SessionTreeEntry = {
type: 'message',
id: '01JTESTENTRY1234567890',
parentId: null,
timestamp: new Date().toISOString(),
message: {
role: 'user',
content: 'Check this rectangle markup on video',
timestamp: Date.now(),
},
}
await storage.appendEntry(userEntry)
// Verify DatabaseSessionStorage persisted entry with details.id = entry.id
const dbEntry = await prisma.agentSessionEntry.findUnique({
where: { id: '01JTESTENTRY1234567890' },
})
expect(dbEntry).toBeDefined()
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const details = (dbEntry?.data as any)?.details
expect(details.id).toBe('01JTESTENTRY1234567890')
expect(details.position).toEqual({ type: 'time', seconds: 12.34 })
expect(details.annotations).toHaveLength(1)
// Now resolveAnnotationsById directly with the entry id and matching asset.id
const result = await resolveAnnotationsById(asset.id, '01JTESTENTRY1234567890')
expect(result.timestamp).toBe(12.34)
expect(result.annotations).toEqual([
{
type: 'rectangle',
x: 100,
y: 200,
width: 50,
height: 50,
},
])
})
it('throws an error when AgentSessionEntry belongs to a different asset', async () => {
const user = await prisma.user.create({
data: { name: 'User', email: 'user3@test.com', password: 'pw' },
})
const team = await prisma.team.create({
data: { name: 'Team' },
})
const project = await prisma.project.create({
data: { name: 'Project', teamId: team.id },
})
const assetA = await prisma.asset.create({
data: { name: 'chatA.mp4', type: 'file', projectId: project.id, status: 'uploaded' },
})
const assetB = await prisma.asset.create({
data: { name: 'chatB.mp4', type: 'file', projectId: project.id, status: 'uploaded' },
})
const agentUser = await prisma.user.create({
data: { name: 'Bot User', email: 'bot2@example.com', type: 'agent' },
})
const agent = await prisma.agent.create({
data: {
id: agentUser.id,
teamId: team.id,
type: 'chat',
config: { provider: 'test-provider', model: 'test-model' },
},
})
const storage = await DatabaseSessionStorage.create({
agentId: agent.id,
userId: user.id,
assetId: assetB.id,
})
storage.currentMessageContext = {
currentAsset: { id: assetB.id, name: assetB.name, type: 'file', path: assetB.name },
annotation: true,
annotations: [{ type: 'box' }],
}
const userEntry: SessionTreeEntry = {
type: 'message',
id: '01JENTRYONASSETB123456',
parentId: null,
timestamp: new Date().toISOString(),
message: { role: 'user', content: 'Markup on B', timestamp: Date.now() },
}
await storage.appendEntry(userEntry)
await expect(resolveAnnotationsById(assetA.id, '01JENTRYONASSETB123456')).rejects.toThrow(
`Annotation "01JENTRYONASSETB123456" belongs to asset "${assetB.id}", not target asset "${assetA.id}".`,
)
})
it('throws an error if annotationId does not match any record', async () => {
await expect(resolveAnnotationsById('asset-1', 'non_existent_id')).rejects.toThrow(
'Annotation with ID "non_existent_id" not found.',
)
})
})
@@ -0,0 +1,52 @@
import { prisma } from '@shumai/db'
import type { ShumaiMessageContext } from '@shumai/dtos'
export async function resolveAnnotationsById(
assetId: string,
annotationId?: string | null,
): Promise<{ annotations: PrismaJson.AnnotationList | null; timestamp: number | null }> {
if (!annotationId) {
return { annotations: null, timestamp: null }
}
// 1. Check AssetComment table (Comments Mode)
const comment = await prisma.assetComment.findUnique({
where: { id: annotationId },
select: { assetId: true, annotation: true, second: true },
})
if (comment) {
if (comment.assetId !== assetId) {
throw new Error(
`Annotation "${annotationId}" belongs to asset "${comment.assetId}", not target asset "${assetId}".`,
)
}
const list = comment.annotation
const annotations =
Array.isArray(list) && list.length > 0 ? (list as PrismaJson.AnnotationList) : null
return { annotations, timestamp: comment.second }
}
// 2. Check AgentSessionEntry table (1-on-1 Chat Mode)
const entry = await prisma.agentSessionEntry.findUnique({
where: { id: annotationId },
select: { assetId: true, data: true },
})
if (entry) {
const details = (entry.data as Record<string, unknown> | null)?.details as
| ShumaiMessageContext
| undefined
const entryAssetId = entry.assetId || details?.currentAsset?.id
if (entryAssetId && entryAssetId !== assetId) {
throw new Error(
`Annotation "${annotationId}" belongs to asset "${entryAssetId}", not target asset "${assetId}".`,
)
}
const list = details?.annotations
const annotations =
Array.isArray(list) && list.length > 0 ? (list as unknown as PrismaJson.AnnotationList) : null
const timestamp = details?.position?.type === 'time' ? details.position.seconds : null
return { annotations, timestamp }
}
throw new Error(`Annotation with ID "${annotationId}" not found.`)
}
+14 -2
View File
@@ -86,7 +86,13 @@ describe('createCreateFileTool', () => {
metadata: undefined,
})
expect(result.details).toEqual({ id: 'file-1', name: 'test', type: 'file', size: 100 })
expect(result.details).toEqual({})
expect(JSON.parse((result.content[0] as { text: string }).text)).toEqual({
id: 'file-1',
name: 'test',
type: 'file',
size: 100,
})
})
it('should forward the metadata to assetService.createFile when provided', async () => {
@@ -209,7 +215,13 @@ describe('createCreateFileTool', () => {
metadata: undefined,
})
expect(result.details).toEqual({ id: 'file-1', name: 'test', type: 'file', size: 100 })
expect(result.details).toEqual({})
expect(JSON.parse((result.content[0] as { text: string }).text)).toEqual({
id: 'file-1',
name: 'test',
type: 'file',
size: 100,
})
})
it('should default unknown extensions to text/plain when creating from content', async () => {
+1 -1
View File
@@ -133,7 +133,7 @@ export function createCreateFileTool(userId: string, metadataSchema?: TSchema):
return {
content: [{ type: 'text', text: JSON.stringify(result, null, 2) }],
details: result,
details: {},
}
},
}
@@ -45,7 +45,8 @@ describe('createCreateFolderTool', () => {
creatorId: 'user-1',
})
expect(result.details).toEqual({
expect(result.details).toEqual({})
expect(JSON.parse((result.content[0] as { text: string }).text)).toEqual({
id: 'folder-1',
name: 'New Folder',
type: 'folder',
+1 -1
View File
@@ -44,7 +44,7 @@ export function createCreateFolderTool(userId: string): AgentTool<typeof createF
return {
content: [{ type: 'text', text: JSON.stringify(result, null, 2) }],
details: result,
details: {},
}
},
}
@@ -83,7 +83,13 @@ describe('createCreateVersionTool', () => {
metadata: undefined,
})
expect(result.details).toEqual({ id: 'file-1', name: 'v2', type: 'file', size: 100 })
expect(result.details).toEqual({})
expect(JSON.parse((result.content[0] as { text: string }).text)).toEqual({
id: 'file-1',
name: 'v2',
type: 'file',
size: 100,
})
})
it('should forward the metadata to assetService.createVersion when provided', async () => {
+1 -1
View File
@@ -82,7 +82,7 @@ export function createCreateVersionTool(userId: string, metadataSchema?: TSchema
return {
content: [{ type: 'text', text: JSON.stringify(result, null, 2) }],
details: result,
details: {},
}
},
}
+272 -143
View File
@@ -1,6 +1,6 @@
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'
import { createDownloadAssetTool } from './download-asset'
import { prisma, type Asset } from '@shumai/db'
import { createDownloadAssetTool, resolveAssetIdFromKey } from './download-asset'
import { prisma, type Asset, type StorageKey } from '@shumai/db'
import { s3Service } from '@shumai/core/src/s3/s3'
import { authzService } from '@shumai/core/src/authz/authz'
import * as fs from 'fs'
@@ -15,6 +15,10 @@ vi.mock('@shumai/db', async (importOriginal) => {
findUnique: vi.fn(),
findFirst: vi.fn(),
},
storageKey: {
findFirst: vi.fn(),
findUnique: vi.fn(),
},
},
}
})
@@ -52,163 +56,288 @@ describe('downloadAssetTool', () => {
it('should throw authorization error if user ID is empty', async () => {
const tool = createDownloadAssetTool('')
await expect(tool.execute('call-1', { assetId: 'asset-1' })).rejects.toThrow(
await expect(tool.execute('call-1', { assetId: 'asset-1', key: null })).rejects.toThrow(
'User ID is required for authorization.',
)
})
it('should throw error if asset not found', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue(null)
it('should throw error if both assetId and key are null', async () => {
const tool = createDownloadAssetTool('user-1')
await expect(tool.execute('call-1', { assetId: 'asset-1' })).rejects.toThrow(
'Asset with ID asset-1 not found.',
await expect(tool.execute('call-1', { assetId: null, key: null })).rejects.toThrow(
'Provide exactly one of "assetId" (to download a workspace asset) or "key" (to download a specific S3 storage key). Set the unused parameter to null.',
)
})
it('should download proxy image to .pi directory when imageTranscodes is present', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'asset-img-1',
name: 'sample_photo.jpg',
storageKey: { key: 'raw/sample_photo.jpg' },
media: {
proxyType: 'image',
imageTranscodes: [{ key: 'proxy/sample_photo.webp' }],
},
} as unknown as Asset)
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('fake-proxy-image-bytes'),
contentType: 'image/webp',
} as unknown as { buffer: Buffer; contentType: string })
it('should throw error if both assetId and key are provided', async () => {
const tool = createDownloadAssetTool('user-1')
const result = await tool.execute('call-1', { assetId: 'asset-img-1' })
expect(authzService.hasPermission).toHaveBeenCalledWith({
user: { id: 'user-1' },
permission: 'Read',
type: 'asset',
id: 'asset-img-1',
})
expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'proxy/sample_photo.webp')
const expectedPath = path.join(piDir, 'asset-img-1_sample_photo.jpg')
createdFiles.push(expectedPath)
expect(fs.existsSync(expectedPath)).toBe(true)
expect(fs.readFileSync(expectedPath, 'utf-8')).toBe('fake-proxy-image-bytes')
expect(result.details.filePath).toBe(path.join('.pi', 'asset-img-1_sample_photo.jpg'))
expect(result.details.contentType).toBe('image/webp')
expect((result.content[0] as { type: 'text'; text: string }).text).toContain(
'Remember to delete this temporary file when finished.',
await expect(
tool.execute('call-1', { assetId: 'asset-1', key: 'files/path/file.png' }),
).rejects.toThrow(
'Provide exactly one of "assetId" (to download a workspace asset) or "key" (to download a specific S3 storage key). Set the unused parameter to null.',
)
})
it('should download proxy video to .pi directory when videoTranscodes is present', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'asset-vid-1',
name: 'video_test.mov',
storageKey: { key: 'raw/video_test.mov' },
media: {
proxyType: 'video',
videoTranscodes: [{ key: 'proxy/video_test.mp4' }],
},
} as unknown as Asset)
describe('Download by assetId', () => {
it('should throw error if asset not found', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue(null)
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('fake-video-bytes'),
contentType: 'video/mp4',
} as unknown as { buffer: Buffer; contentType: string })
const tool = createDownloadAssetTool('user-1')
const result = await tool.execute('call-1', { assetId: 'asset-vid-1' })
expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'proxy/video_test.mp4')
const expectedPath = path.join(piDir, 'asset-vid-1_video_test.mov')
createdFiles.push(expectedPath)
expect(fs.existsSync(expectedPath)).toBe(true)
expect(result.details.size).toBe(Buffer.from('fake-video-bytes').length)
})
it('should download original file to .pi directory even when pdfTranscode is present', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'asset-md-1',
name: 'doc.md',
storageKey: { key: 'raw/doc.md' },
media: {
proxyType: 'pdf',
pdfTranscode: { key: 'proxy/doc.pdf' },
original: { key: 'raw/doc.md' },
},
} as unknown as Asset)
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('# fake markdown bytes'),
contentType: 'text/markdown',
} as unknown as { buffer: Buffer; contentType: string })
const tool = createDownloadAssetTool('user-1')
const result = await tool.execute('call-1', { assetId: 'asset-md-1' })
expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'raw/doc.md')
const expectedPath = path.join(piDir, 'asset-md-1_doc.md')
createdFiles.push(expectedPath)
expect(fs.existsSync(expectedPath)).toBe(true)
expect(result.details.contentType).toBe('text/markdown')
})
it('should resolve version_stack asset to its latest version file', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'stack-1',
name: '',
type: 'version_stack',
storageKey: null,
media: null,
} as unknown as Asset)
vi.mocked(prisma.asset.findFirst).mockResolvedValue({
id: 'file-ver-2',
name: 'banana_pig.png',
type: 'file',
storageKey: { key: 'raw/banana_pig.png' },
media: {
proxyType: 'image',
imageTranscodes: [{ key: 'proxy/banana_pig.webp' }],
},
} as unknown as Asset)
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('fake-version-bytes'),
contentType: 'image/webp',
} as unknown as { buffer: Buffer; contentType: string })
const tool = createDownloadAssetTool('user-1')
const result = await tool.execute('call-1', { assetId: 'stack-1' })
expect(prisma.asset.findFirst).toHaveBeenCalledWith({
where: { parentId: 'stack-1', isDeleted: false },
orderBy: { sortIndex: 'asc' },
include: { storageKey: true },
const tool = createDownloadAssetTool('user-1')
await expect(tool.execute('call-1', { assetId: 'asset-1', key: null })).rejects.toThrow(
'Asset with ID asset-1 not found.',
)
})
expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'proxy/banana_pig.webp')
it('should download proxy image to .pi directory when downloading by assetId', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'asset-img-1',
name: 'sample_photo.jpg',
storageKey: { key: 'raw/sample_photo.jpg' },
media: {
proxyType: 'image',
imageTranscodes: [{ key: 'proxy/sample_photo.webp' }],
},
} as unknown as Asset)
const expectedPath = path.join(piDir, 'file-ver-2_banana_pig.png')
createdFiles.push(expectedPath)
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('fake-proxy-image-bytes'),
contentType: 'image/webp',
} as unknown as { buffer: Buffer; contentType: string })
expect(fs.existsSync(expectedPath)).toBe(true)
expect(result.details.name).toBe('banana_pig.png')
const tool = createDownloadAssetTool('user-1')
const result = await tool.execute('call-1', { assetId: 'asset-img-1', key: null })
expect(authzService.hasPermission).toHaveBeenCalledWith({
user: { id: 'user-1' },
permission: 'Read',
type: 'asset',
id: 'asset-img-1',
})
expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'proxy/sample_photo.webp')
const expectedPath = path.join(piDir, 'asset-img-1_sample_photo.jpg')
createdFiles.push(expectedPath)
expect(fs.existsSync(expectedPath)).toBe(true)
expect(fs.readFileSync(expectedPath, 'utf-8')).toBe('fake-proxy-image-bytes')
expect(result.details.filePath).toBe(path.join('.pi', 'asset-img-1_sample_photo.jpg'))
expect(result.details.contentType).toBe('image/webp')
expect((result.content[0] as { type: 'text'; text: string }).text).toContain(
'Remember to delete this temporary file when finished.',
)
})
it('should resolve version_stack asset to its latest version file', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'stack-1',
name: '',
type: 'version_stack',
storageKey: null,
media: null,
} as unknown as Asset)
vi.mocked(prisma.asset.findFirst).mockResolvedValue({
id: 'file-ver-2',
name: 'banana_pig.png',
type: 'file',
storageKey: { key: 'raw/banana_pig.png' },
media: {
proxyType: 'image',
imageTranscodes: [{ key: 'proxy/banana_pig.webp' }],
},
} as unknown as Asset)
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('fake-version-bytes'),
contentType: 'image/webp',
} as unknown as { buffer: Buffer; contentType: string })
const tool = createDownloadAssetTool('user-1')
const result = await tool.execute('call-1', { assetId: 'stack-1', key: null })
expect(prisma.asset.findFirst).toHaveBeenCalledWith({
where: { parentId: 'stack-1', isDeleted: false },
orderBy: { sortIndex: 'asc' },
include: { storageKey: true },
})
expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'proxy/banana_pig.webp')
const expectedPath = path.join(piDir, 'file-ver-2_banana_pig.png')
createdFiles.push(expectedPath)
expect(fs.existsSync(expectedPath)).toBe(true)
expect(result.details.name).toBe('banana_pig.png')
})
})
describe('Download by S3 key and Authorization', () => {
it('should reject keys that do not start with files/', async () => {
const tool = createDownloadAssetTool('user-1')
await expect(
tool.execute('call-1', { assetId: null, key: 'avatars/user-1.png' }),
).rejects.toThrow('Invalid storage key format: key must start with "files/"')
})
it('should reject keys with path traversal (..)', async () => {
const tool = createDownloadAssetTool('user-1')
await expect(
tool.execute('call-1', { assetId: null, key: 'files/../../etc/passwd' }),
).rejects.toThrow('cannot contain ".."')
})
it('should reject keys with insufficient path segments', async () => {
const tool = createDownloadAssetTool('user-1')
await expect(tool.execute('call-1', { assetId: null, key: 'files/invalid' })).rejects.toThrow(
'Invalid storage key format',
)
})
it('should reject keys that do not belong to any valid asset in database', async () => {
vi.mocked(prisma.asset.findUnique).mockResolvedValue(null)
vi.mocked(prisma.storageKey.findFirst).mockResolvedValue(null)
vi.mocked(prisma.storageKey.findUnique).mockResolvedValue(null)
const tool = createDownloadAssetTool('user-1')
await expect(
tool.execute('call-1', { assetId: null, key: 'files/nonexistent-id/shot.webp' }),
).rejects.toThrow('does not belong to any valid asset or the asset has been deleted')
})
it('should reject download when user lacks read permission on the owning asset (cross-project protection)', async () => {
// Key belongs to asset-in-other-project
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'asset-in-other-project',
} as unknown as Asset)
// Authz rejects because user is in a different project/team
vi.mocked(authzService.hasPermission).mockRejectedValue(
new Error('User does not have Read permission on asset asset-in-other-project'),
)
const tool = createDownloadAssetTool('user-1')
await expect(
tool.execute('call-1', {
assetId: null,
key: 'files/asset-in-other-project/screenshots/secret_frame.webp',
}),
).rejects.toThrow('User does not have Read permission on asset asset-in-other-project')
expect(authzService.hasPermission).toHaveBeenCalledWith({
user: { id: 'user-1' },
permission: 'Read',
type: 'asset',
id: 'asset-in-other-project',
})
expect(s3Service.getObject).not.toHaveBeenCalled()
})
it('should authorize and download derived artifact key (files/<assetId>/...)', async () => {
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'ast-123',
} as unknown as Asset)
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('fake-screenshot-bytes'),
contentType: 'image/webp',
} as unknown as { buffer: Buffer; contentType: string })
const tool = createDownloadAssetTool('user-1')
const result = await tool.execute('call-1', {
assetId: null,
key: 'files/ast-123/screenshots/shot_5.0s.webp',
})
expect(authzService.hasPermission).toHaveBeenCalledWith({
user: { id: 'user-1' },
permission: 'Read',
type: 'asset',
id: 'ast-123',
})
expect(s3Service.getObject).toHaveBeenCalledWith(
'shumai',
'files/ast-123/screenshots/shot_5.0s.webp',
)
const expectedPath = path.join(piDir, 'shot_5.0s.webp')
createdFiles.push(expectedPath)
expect(fs.existsSync(expectedPath)).toBe(true)
expect(fs.readFileSync(expectedPath, 'utf-8')).toBe('fake-screenshot-bytes')
expect(result.details.filePath).toBe(path.join('.pi', 'shot_5.0s.webp'))
expect(result.details.key).toBe('files/ast-123/screenshots/shot_5.0s.webp')
expect(result.details.assetId).toBe('ast-123')
expect(result.details.size).toBe(Buffer.from('fake-screenshot-bytes').length)
})
it('should authorize and download original upload key (files/<uploadUlid>/...) via StorageKey', async () => {
vi.mocked(prisma.asset.findUnique).mockResolvedValue(null)
vi.mocked(prisma.storageKey.findFirst).mockResolvedValue({
id: 'sk-1',
key: 'files/upload-ulid-456/photo.png',
assets: [{ id: 'asset-photo-1' }],
} as unknown as StorageKey & { assets: Asset[] })
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('fake-original-photo-bytes'),
contentType: 'image/png',
} as unknown as { buffer: Buffer; contentType: string })
const tool = createDownloadAssetTool('user-1')
const result = await tool.execute('call-1', {
assetId: null,
key: 'files/upload-ulid-456/photo.png',
})
expect(authzService.hasPermission).toHaveBeenCalledWith({
user: { id: 'user-1' },
permission: 'Read',
type: 'asset',
id: 'asset-photo-1',
})
expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'files/upload-ulid-456/photo.png')
const expectedPath = path.join(piDir, 'photo.png')
createdFiles.push(expectedPath)
expect(fs.existsSync(expectedPath)).toBe(true)
expect(result.details.assetId).toBe('asset-photo-1')
})
})
describe('resolveAssetIdFromKey helper unit tests', () => {
it('resolves directly by asset ID', async () => {
vi.mocked(prisma.asset.findUnique).mockResolvedValue({ id: 'ast-999' } as unknown as Asset)
const assetId = await resolveAssetIdFromKey('files/ast-999/pdf_pages/page_1.webp')
expect(assetId).toBe('ast-999')
})
it('resolves via StorageKey prefix', async () => {
vi.mocked(prisma.asset.findUnique).mockResolvedValue(null)
vi.mocked(prisma.storageKey.findFirst).mockResolvedValue({
assets: [{ id: 'ast-888' }],
} as unknown as StorageKey & { assets: Asset[] })
const assetId = await resolveAssetIdFromKey('files/ulid-777/document.pdf')
expect(assetId).toBe('ast-888')
})
it('resolves via exact StorageKey match', async () => {
vi.mocked(prisma.asset.findUnique).mockResolvedValue(null)
vi.mocked(prisma.storageKey.findFirst).mockResolvedValue(null)
vi.mocked(prisma.storageKey.findUnique).mockResolvedValue({
assets: [{ id: 'ast-777' }],
} as unknown as StorageKey & { assets: Asset[] })
const assetId = await resolveAssetIdFromKey('files/ulid-special/document.pdf')
expect(assetId).toBe('ast-777')
})
})
})
+208 -83
View File
@@ -7,82 +7,107 @@ import { sanitizeFilename } from '@shumai/core/src/utils/filename'
import * as fs from 'fs'
import * as path from 'path'
const downloadAssetSchema = Type.Object({
assetId: Type.String({
description:
'The asset ID of the workspace asset to download to local disk. This parameter is required.',
}),
})
export const downloadAssetSchema = Type.Object(
{
assetId: Type.Union([
Type.String({
description:
'The asset ID of the workspace asset to download to local disk, or null if providing "key".',
}),
Type.Null(),
]),
key: Type.Union([
Type.String({
description:
'The storage S3 key of a specific media artifact (e.g. a screenshot or transcode key from read_asset) to download to local disk, or null if providing "assetId".',
}),
Type.Null(),
]),
},
{ additionalProperties: false },
)
export async function resolveAssetIdFromKey(key: string): Promise<string> {
if (!key.startsWith('files/') || key.includes('..')) {
throw new Error(
'Invalid storage key format: key must start with "files/" and cannot contain "..".',
)
}
const segments = key.split('/')
if (segments.length < 3) {
throw new Error(
'Invalid storage key format: key must follow "files/<identifier>/<path>" structure.',
)
}
const identifier = segments[1]
// 1. Check if identifier is directly an Asset ID (Derived Artifacts: screenshots, pdf_pages, annotations, proxies)
const assetById = await prisma.asset.findUnique({
where: { id: identifier, isDeleted: false },
select: { id: true },
})
if (assetById) {
return assetById.id
}
// 2. Check if identifier is a StorageKey directory prefix (Original uploads / agent-created files)
const storageKeyByPrefix = await prisma.storageKey.findFirst({
where: { key: { startsWith: `files/${identifier}/` } },
include: {
assets: {
where: { isDeleted: false },
select: { id: true },
},
},
})
if (storageKeyByPrefix?.assets && storageKeyByPrefix.assets.length > 0) {
return storageKeyByPrefix.assets[0].id
}
// 3. Exact StorageKey fallback
const exactStorageKey = await prisma.storageKey.findUnique({
where: { key },
include: {
assets: {
where: { isDeleted: false },
select: { id: true },
},
},
})
if (exactStorageKey?.assets && exactStorageKey.assets.length > 0) {
return exactStorageKey.assets[0].id
}
throw new Error(
`Storage key "${key}" does not belong to any valid asset or the asset has been deleted.`,
)
}
export function createDownloadAssetTool(userId: string): AgentTool<typeof downloadAssetSchema> {
return {
name: 'download_asset',
label: 'Download Asset',
description:
'Downloads a workspace asset (proxy version if available) to the local .pi directory for processing with local scripts or bash commands. Remember to delete the temporary downloaded file when finished.',
'Downloads a workspace asset or specific storage S3 key to the local .pi directory for processing with local scripts or bash commands. ' +
'Provide exactly one of "assetId" (for full assets) or "key" (for intermediate artifacts like screenshots from read_asset). ' +
'Remember to delete the temporary downloaded file when finished.',
parameters: downloadAssetSchema,
execute: async (_toolCallId, params) => {
const assetId = params.assetId
const { assetId, key } = params
if (!userId) {
throw new Error('User ID is required for authorization.')
}
await authzService.hasPermission({
user: { id: userId } as User,
permission: Permission.Read,
type: ResourceType.Asset,
id: assetId,
})
const hasAssetId = assetId !== null
const hasKey = key !== null
let asset = await prisma.asset.findUnique({
where: { id: assetId },
include: { storageKey: true },
})
if (!asset) {
throw new Error(`Asset with ID ${assetId} not found.`)
}
if (asset.type === 'version_stack') {
const latestVersion = await prisma.asset.findFirst({
where: { parentId: asset.id, isDeleted: false },
orderBy: { sortIndex: 'asc' },
include: { storageKey: true },
})
if (latestVersion) {
asset = latestVersion
}
}
let mediaKey = asset.storageKey?.key
if (asset.media) {
const mediaInfo = asset.media as unknown as PrismaJson.MediaInfo
if (
mediaInfo.proxyType === 'image' ||
(mediaInfo.imageTranscodes && mediaInfo.imageTranscodes.length > 0)
) {
if (mediaInfo.imageTranscodes && mediaInfo.imageTranscodes.length > 0) {
mediaKey = mediaInfo.imageTranscodes[0].key || mediaKey
}
} else if (
mediaInfo.proxyType === 'video' ||
(mediaInfo.videoTranscodes && mediaInfo.videoTranscodes.length > 0) ||
mediaInfo.videoPreview?.key
) {
if (mediaInfo.videoTranscodes && mediaInfo.videoTranscodes.length > 0) {
mediaKey = mediaInfo.videoTranscodes[0].key || mediaKey
} else if (mediaInfo.videoPreview?.key) {
mediaKey = mediaInfo.videoPreview.key
}
} else if (mediaInfo.original?.key) {
mediaKey = mediaInfo.original.key || mediaKey
}
}
if (!mediaKey) {
throw new Error(`No media content found for asset ${assetId}.`)
if (hasAssetId === hasKey) {
throw new Error(
'Provide exactly one of "assetId" (to download a workspace asset) or "key" (to download a specific S3 storage key). Set the unused parameter to null.',
)
}
const piDir = path.join(process.cwd(), '.pi')
@@ -90,32 +115,132 @@ export function createDownloadAssetTool(userId: string): AgentTool<typeof downlo
fs.mkdirSync(piDir, { recursive: true })
}
const safeName = sanitizeFilename(asset.name || 'file')
const filename = `${asset.id}_${safeName}`
const targetFilePath = path.join(piDir, filename)
const relativePath = path.join('.pi', filename)
const bucket = process.env.S3_BUCKET || 'shumai'
const { buffer, contentType } = await s3Service.getObject(bucket, mediaKey)
fs.writeFileSync(targetFilePath, buffer)
// Branch A: Download by assetId
if (assetId) {
await authzService.hasPermission({
user: { id: userId } as User,
permission: Permission.Read,
type: ResourceType.Asset,
id: assetId,
})
return {
content: [
{
type: 'text',
text: `Downloaded asset "${asset.name}" (ID: ${assetId}) to "${relativePath}". Remember to delete this temporary file when finished.`,
let asset = await prisma.asset.findUnique({
where: { id: assetId },
include: { storageKey: true },
})
if (!asset) {
throw new Error(`Asset with ID ${assetId} not found.`)
}
if (asset.type === 'version_stack') {
const latestVersion = await prisma.asset.findFirst({
where: { parentId: asset.id, isDeleted: false },
orderBy: { sortIndex: 'asc' },
include: { storageKey: true },
})
if (latestVersion) {
asset = latestVersion
}
}
let mediaKey = asset.storageKey?.key
if (asset.media) {
const mediaInfo = asset.media as unknown as PrismaJson.MediaInfo
if (
mediaInfo.proxyType === 'image' ||
(mediaInfo.imageTranscodes && mediaInfo.imageTranscodes.length > 0)
) {
if (mediaInfo.imageTranscodes && mediaInfo.imageTranscodes.length > 0) {
mediaKey = mediaInfo.imageTranscodes[0].key || mediaKey
}
} else if (
mediaInfo.proxyType === 'video' ||
(mediaInfo.videoTranscodes && mediaInfo.videoTranscodes.length > 0) ||
mediaInfo.videoPreview?.key
) {
if (mediaInfo.videoTranscodes && mediaInfo.videoTranscodes.length > 0) {
mediaKey = mediaInfo.videoTranscodes[0].key || mediaKey
} else if (mediaInfo.videoPreview?.key) {
mediaKey = mediaInfo.videoPreview.key
}
} else if (mediaInfo.original?.key) {
mediaKey = mediaInfo.original.key || mediaKey
}
}
if (!mediaKey) {
throw new Error(`No media content found for asset ${assetId}.`)
}
const safeName = sanitizeFilename(asset.name || 'file')
const filename = `${asset.id}_${safeName}`
const targetFilePath = path.join(piDir, filename)
const relativePath = path.join('.pi', filename)
const { buffer, contentType } = await s3Service.getObject(bucket, mediaKey)
fs.writeFileSync(targetFilePath, buffer)
return {
content: [
{
type: 'text',
text: `Downloaded asset "${asset.name}" (ID: ${assetId}) to "${relativePath}". Remember to delete this temporary file when finished.`,
},
],
details: {
assetId,
name: asset.name,
filePath: relativePath,
absolutePath: targetFilePath,
contentType: contentType || 'application/octet-stream',
size: buffer.length,
},
],
details: {
assetId,
name: asset.name,
filePath: relativePath,
absolutePath: targetFilePath,
contentType: contentType || 'application/octet-stream',
size: buffer.length,
},
}
}
// Branch B: Download by S3 storage key
if (key) {
const owningAssetId = await resolveAssetIdFromKey(key)
await authzService.hasPermission({
user: { id: userId } as User,
permission: Permission.Read,
type: ResourceType.Asset,
id: owningAssetId,
})
const rawBasename = path.basename(key)
const safeBasename = sanitizeFilename(rawBasename) || 'downloaded_file'
const filename = safeBasename
const targetFilePath = path.join(piDir, filename)
const relativePath = path.join('.pi', filename)
const { buffer, contentType } = await s3Service.getObject(bucket, key)
fs.writeFileSync(targetFilePath, buffer)
return {
content: [
{
type: 'text',
text: `Downloaded storage key "${key}" to "${relativePath}". Remember to delete this temporary file when finished.`,
},
],
details: {
key,
assetId: owningAssetId,
filePath: relativePath,
absolutePath: targetFilePath,
contentType: contentType || 'application/octet-stream',
size: buffer.length,
},
}
}
throw new Error('Invalid download_asset parameters.')
},
}
}
+2 -1
View File
@@ -48,7 +48,8 @@ describe('createListAssetsTool', () => {
after: undefined,
})
expect(result.details).toEqual({
expect(result.details).toEqual({})
expect(JSON.parse((result.content[0] as { text: string }).text)).toEqual({
assets: [
{ id: '01M2FILE2', name: 'file2.txt', type: 'file', size: 200 },
{ id: '01M1FILE1', name: 'file1.txt', type: 'file', size: 100 },
+1 -1
View File
@@ -70,7 +70,7 @@ export function createListAssetsTool(userId: string): AgentTool<typeof listAsset
return {
content: [{ type: 'text', text: JSON.stringify(output, null, 2) }],
details: output,
details: {},
}
},
}
+544
View File
@@ -0,0 +1,544 @@
import { describe, it, expect, vi, beforeEach } from 'vitest'
import { createReadAssetTool } from './read-asset'
import {
prisma,
WorkflowTaskType,
WorkflowTaskStatus,
type Asset,
type WorkflowTask,
} from '@shumai/db'
import { s3Service } from '@shumai/core/src/s3/s3'
import { workflowService } from '@shumai/workflow-core'
import { authzService } from '@shumai/core/src/authz/authz'
import * as annotationResolver from './annotation-resolver'
vi.mock('@shumai/db', async (importOriginal) => {
const actual = await importOriginal<typeof import('@shumai/db')>()
return {
...actual,
prisma: {
asset: {
findUnique: vi.fn(),
findFirst: vi.fn(),
},
workflowTask: {
create: vi.fn(),
},
},
}
})
vi.mock('@shumai/core/src/s3/s3', () => ({
s3Service: {
getObject: vi.fn(),
presign: vi.fn().mockResolvedValue('https://mock-presigned-url.com'),
},
}))
vi.mock('@shumai/workflow-core', () => ({
workflowService: {
executeWait: vi.fn(),
},
}))
vi.mock('@shumai/core/src/authz/authz', () => ({
authzService: {
hasPermission: vi.fn(),
},
Permission: { Read: 'Read' },
ResourceType: { Asset: 'asset' },
}))
describe('readAssetTool', () => {
beforeEach(() => {
vi.clearAllMocks()
vi.spyOn(annotationResolver, 'resolveAnnotationsById').mockResolvedValue({
annotations: null,
timestamp: null,
})
})
it('should throw authorization error if user ID is empty', async () => {
const tool = createReadAssetTool('')
await expect(
tool.execute('call-1', {
assetId: 'asset-1',
annotationId: null,
imageConfig: null,
videoConfig: null,
docConfig: null,
}),
).rejects.toThrow('User ID is required for authorization.')
})
it('should throw error if asset not found', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue(null)
const tool = createReadAssetTool('user-1')
await expect(
tool.execute('call-1', {
assetId: 'asset-1',
annotationId: null,
imageConfig: null,
videoConfig: null,
docConfig: null,
}),
).rejects.toThrow('Asset with ID asset-1 not found.')
})
describe('Image Assets', () => {
it('should read image and return ImageContent and S3 key', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'img-1',
name: 'photo.png',
mediaType: 'image/png',
storageKey: { key: 'raw/photo.png' },
media: {
proxyType: 'image',
imageTranscodes: [{ key: 'proxy/photo.webp' }],
},
} as unknown as Asset)
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('fake-image-bytes'),
contentType: 'image/webp',
} as unknown as { buffer: Buffer; contentType: string })
const tool = createReadAssetTool('user-1')
const result = await tool.execute('call-1', {
assetId: 'img-1',
annotationId: null,
imageConfig: null,
videoConfig: null,
docConfig: null,
})
expect(authzService.hasPermission).toHaveBeenCalledWith({
user: { id: 'user-1' },
permission: 'Read',
type: 'asset',
id: 'img-1',
})
expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'proxy/photo.webp')
const textItem = result.content[0] as { type: 'text'; text: string }
const imageItem = result.content[1] as { type: 'image'; data: string; mimeType: string }
expect(textItem.text).toContain('proxy/photo.webp')
expect(imageItem.type).toBe('image')
expect(imageItem.data).toBe(Buffer.from('fake-image-bytes').toString('base64'))
expect(result.details.key).toBe('proxy/photo.webp')
})
it('should trigger image annotation transcode workflow when annotationId is provided', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'img-1',
name: 'photo.png',
mediaType: 'image/png',
projectId: 'project-1',
storageKey: { key: 'raw/photo.png' },
media: {
proxyType: 'image',
imageTranscodes: [{ key: 'proxy/photo.webp' }],
},
} as unknown as Asset)
vi.spyOn(annotationResolver, 'resolveAnnotationsById').mockResolvedValue({
annotations: [
{
type: 'arrow',
color: '#ff0000',
points: [
[0, 0],
[10, 10],
],
},
],
timestamp: null,
})
vi.mocked(prisma.workflowTask.create).mockResolvedValue({
id: 'task-anno',
} as unknown as WorkflowTask)
vi.mocked(workflowService.executeWait).mockResolvedValue({
id: 'task-anno',
output: { key: 'annotated/photo.webp' },
} as unknown as WorkflowTask)
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('annotated-bytes'),
contentType: 'image/webp',
} as unknown as { buffer: Buffer; contentType: string })
const tool = createReadAssetTool('user-1')
const result = await tool.execute('call-1', {
assetId: 'img-1',
annotationId: 'comment-1',
imageConfig: null,
videoConfig: null,
docConfig: null,
})
expect(prisma.workflowTask.create).toHaveBeenCalledWith({
data: {
assetId: 'img-1',
projectId: 'project-1',
type: WorkflowTaskType.transcode_image_annotation,
status: WorkflowTaskStatus.pending,
payload: {
projectId: 'project-1',
imageAnnotation: {
annotations: [
{
type: 'arrow',
color: '#ff0000',
points: [
[0, 0],
[10, 10],
],
},
],
},
},
},
})
expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'annotated/photo.webp')
expect(result.details.key).toBe('annotated/photo.webp')
})
})
describe('Video Assets', () => {
it('should throw error if videoConfig is missing for video assets', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'vid-1',
name: 'video.mp4',
mediaType: 'video/mp4',
media: { proxyType: 'video' },
} as unknown as Asset)
const tool = createReadAssetTool('user-1')
await expect(
tool.execute('call-1', {
assetId: 'vid-1',
annotationId: null,
imageConfig: null,
videoConfig: null,
docConfig: null,
}),
).rejects.toThrow('videoConfig with start, end, and count is required')
})
it('should throw error on invalid video time range', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'vid-1',
name: 'video.mp4',
mediaType: 'video/mp4',
media: { proxyType: 'video' },
} as unknown as Asset)
const tool = createReadAssetTool('user-1')
await expect(
tool.execute('call-1', {
assetId: 'vid-1',
annotationId: null,
imageConfig: null,
videoConfig: { start: 10, end: 5, count: 1 },
docConfig: null,
}),
).rejects.toThrow(
'Invalid video time range: start (10) must be less than or equal to end (5).',
)
})
it('should extract frames and return ImageContent array and keys', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'vid-1',
name: 'video.mp4',
mediaType: 'video/mp4',
projectId: 'project-1',
media: { proxyType: 'video' },
} as unknown as Asset)
vi.mocked(prisma.workflowTask.create).mockResolvedValue({
id: 'task-shot',
} as unknown as WorkflowTask)
vi.mocked(workflowService.executeWait).mockResolvedValue({
id: 'task-shot',
output: {
screenshots: [
{ key: 'screenshots/shot1.webp', timestamp: 2.0 },
{ key: 'screenshots/shot2.webp', timestamp: 4.0 },
],
},
} as unknown as WorkflowTask)
vi.mocked(s3Service.getObject).mockImplementation(async (_bucket, key) => ({
buffer: Buffer.from(`bytes-for-${key}`),
contentType: 'image/webp',
}))
const tool = createReadAssetTool('user-1')
const result = await tool.execute('call-1', {
assetId: 'vid-1',
annotationId: null,
imageConfig: null,
videoConfig: { start: 2.0, end: 4.0, count: 2 },
docConfig: null,
})
expect(prisma.workflowTask.create).toHaveBeenCalledWith({
data: {
assetId: 'vid-1',
projectId: 'project-1',
type: WorkflowTaskType.transcode_screenshot,
status: WorkflowTaskStatus.pending,
payload: {
projectId: 'project-1',
screenshot: {
start: 2.0,
end: 4.0,
count: 2,
commentTimestamp: null,
annotations: null,
},
},
},
})
expect(result.content.length).toBe(3) // 1 text + 2 images
expect(result.details.keys).toEqual(['screenshots/shot1.webp', 'screenshots/shot2.webp'])
})
})
describe('Document Assets', () => {
it('should throw error if docConfig is missing for document assets', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'doc-1',
name: 'doc.pdf',
mediaType: 'application/pdf',
media: { proxyType: 'pdf' },
} as unknown as Asset)
const tool = createReadAssetTool('user-1')
await expect(
tool.execute('call-1', {
assetId: 'doc-1',
annotationId: null,
imageConfig: null,
videoConfig: null,
docConfig: null,
}),
).rejects.toThrow('docConfig with mode ("pages" or "text") is required')
})
it('should render PDF pages in mode: "pages"', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'doc-1',
name: 'doc.pdf',
mediaType: 'application/pdf',
projectId: 'project-1',
media: { proxyType: 'pdf' },
} as unknown as Asset)
vi.mocked(prisma.workflowTask.create).mockResolvedValue({
id: 'task-pdf',
} as unknown as WorkflowTask)
vi.mocked(workflowService.executeWait).mockResolvedValue({
id: 'task-pdf',
output: {
pages: [
{ key: 'pdf_pages/page1.webp', page: 1 },
{ key: 'pdf_pages/page2.webp', page: 2 },
],
},
} as unknown as WorkflowTask)
vi.mocked(s3Service.getObject).mockImplementation(async (_bucket, key) => ({
buffer: Buffer.from(`bytes-for-${key}`),
contentType: 'image/webp',
}))
const tool = createReadAssetTool('user-1')
const result = await tool.execute('call-1', {
assetId: 'doc-1',
annotationId: null,
imageConfig: null,
videoConfig: null,
docConfig: { mode: 'pages', startPage: 1, endPage: 2 },
})
expect(result.content.length).toBe(3) // 1 text + 2 images
expect(result.details.keys).toEqual(['pdf_pages/page1.webp', 'pdf_pages/page2.webp'])
})
it('should throw error in mode: "pages" when page range exceeds 20 pages', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'doc-1',
name: 'doc.pdf',
mediaType: 'application/pdf',
media: { proxyType: 'pdf' },
} as unknown as Asset)
const tool = createReadAssetTool('user-1')
await expect(
tool.execute('call-1', {
assetId: 'doc-1',
annotationId: null,
imageConfig: null,
videoConfig: null,
docConfig: { mode: 'pages', startPage: 1, endPage: 25 },
}),
).rejects.toThrow('exceeds the maximum limit of 20 pages')
})
it('should throw error in mode: "text" when called on binary PDF', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'doc-1',
name: 'manual.pdf',
mediaType: 'application/pdf',
media: { proxyType: 'pdf' },
} as unknown as Asset)
const tool = createReadAssetTool('user-1')
await expect(
tool.execute('call-1', {
assetId: 'doc-1',
annotationId: null,
imageConfig: null,
videoConfig: null,
docConfig: { mode: 'text', startPage: null, endPage: null },
}),
).rejects.toThrow('is a binary PDF/document and cannot be read as raw text')
})
it('should throw error in mode: "text" when annotationId is provided', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'doc-1',
name: 'notes.md',
mediaType: 'text/markdown',
media: { proxyType: 'pdf' },
} as unknown as Asset)
const tool = createReadAssetTool('user-1')
await expect(
tool.execute('call-1', {
assetId: 'doc-1',
annotationId: 'comment-123',
imageConfig: null,
videoConfig: null,
docConfig: { mode: 'text', startPage: null, endPage: null },
}),
).rejects.toThrow('annotationId cannot be used with docConfig mode "text"')
})
it('should throw error in mode: "text" when startPage or endPage is non-null', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'doc-1',
name: 'notes.md',
mediaType: 'text/markdown',
media: { proxyType: 'pdf' },
} as unknown as Asset)
const tool = createReadAssetTool('user-1')
await expect(
tool.execute('call-1', {
assetId: 'doc-1',
annotationId: null,
imageConfig: null,
videoConfig: null,
docConfig: { mode: 'text', startPage: 1, endPage: 2 },
}),
).rejects.toThrow('startPage and endPage must be null when docConfig mode is "text"')
})
it('should read raw text from S3 in mode: "text" for markdown files', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'doc-md',
name: 'README.md',
mediaType: 'text/markdown',
storageKey: { key: 'raw/README.md' },
media: { proxyType: 'pdf' },
} as unknown as Asset)
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('# Hello Shumai\nThis is markdown text.'),
contentType: 'text/markdown',
} as unknown as { buffer: Buffer; contentType: string })
const tool = createReadAssetTool('user-1')
const result = await tool.execute('call-1', {
assetId: 'doc-md',
annotationId: null,
imageConfig: null,
videoConfig: null,
docConfig: { mode: 'text', startPage: null, endPage: null },
})
expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'raw/README.md')
expect(result.content[0].type).toBe('text')
expect((result.content[0] as { type: 'text'; text: string }).text).toContain(
'# Hello Shumai\nThis is markdown text.',
)
expect(result.details.key).toBe('raw/README.md')
})
})
describe('Version Stack Resolution', () => {
it('should resolve version_stack to latest version asset', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'stack-1',
name: '',
type: 'version_stack',
} as unknown as Asset)
vi.mocked(prisma.asset.findFirst).mockResolvedValue({
id: 'file-v2',
name: 'diagram_v2.png',
type: 'file',
mediaType: 'image/png',
storageKey: { key: 'raw/diagram_v2.png' },
media: {
proxyType: 'image',
imageTranscodes: [{ key: 'proxy/diagram_v2.webp' }],
},
} as unknown as Asset)
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('v2-image-bytes'),
contentType: 'image/webp',
} as unknown as { buffer: Buffer; contentType: string })
const tool = createReadAssetTool('user-1')
const result = await tool.execute('call-1', {
assetId: 'stack-1',
annotationId: null,
imageConfig: null,
videoConfig: null,
docConfig: null,
})
expect(prisma.asset.findFirst).toHaveBeenCalledWith({
where: { parentId: 'stack-1', isDeleted: false },
orderBy: { sortIndex: 'asc' },
include: { storageKey: true },
})
expect(result.details.name).toBe('diagram_v2.png')
expect(result.details.key).toBe('proxy/diagram_v2.webp')
})
})
})
+576
View File
@@ -0,0 +1,576 @@
import { Type } from 'typebox'
import { type AgentTool } from '@earendil-works/pi-agent-core'
import { type ImageContent, type TextContent } from '@earendil-works/pi-ai'
import { prisma, WorkflowTaskType, WorkflowTaskStatus, type User } from '@shumai/db'
import { s3Service } from '@shumai/core/src/s3/s3'
import { workflowService } from '@shumai/workflow-core'
import { authzService, Permission, ResourceType } from '@shumai/core/src/authz/authz'
import { resolveAnnotationsById } from './annotation-resolver'
export const readAssetSchema = Type.Object(
{
assetId: Type.String({
description: 'The asset ID of the workspace asset to inspect. Required.',
}),
annotationId: Type.Union([
Type.String({
description:
'The ID of the comment or message entry from <annotation id="..." /> whose visual markup should be overlaid, or null.',
}),
Type.Null(),
]),
imageConfig: Type.Union([
Type.Object(
{},
{
additionalProperties: false,
description: 'Optional configuration for image assets, or null.',
},
),
Type.Null(),
]),
videoConfig: Type.Union([
Type.Object(
{
start: Type.Number({ description: 'Start time in seconds.' }),
end: Type.Number({ description: 'End time in seconds.' }),
count: Type.Number({ description: 'Number of frames to extract (e.g. 1 to 10).' }),
},
{
additionalProperties: false,
description:
'Configuration for video assets. Required when inspecting videos, or null for other asset types.',
},
),
Type.Null(),
]),
docConfig: Type.Union([
Type.Object(
{
mode: Type.Union([
Type.Literal('pages', {
description:
'Render specific page range as images for visual layout, charts, or visual annotations (required for binary PDFs).',
}),
Type.Literal('text', {
description:
'Read raw text content directly (ideal for .md, .txt, .csv, code files).',
}),
]),
startPage: Type.Union([
Type.Number({
description:
'Start page number (1-based index, required when mode="pages"), or null when mode="text".',
}),
Type.Null(),
]),
endPage: Type.Union([
Type.Number({
description:
'End page number (1-based index, max 20 pages per call, required when mode="pages"), or null when mode="text".',
}),
Type.Null(),
]),
},
{
additionalProperties: false,
description:
'Configuration for document/PDF/text assets, or null if inspecting another asset type.',
},
),
Type.Null(),
]),
},
{ additionalProperties: false },
)
export function isPlainTextAsset(mediaType: string, filename: string): boolean {
if (
mediaType.startsWith('text/') ||
mediaType === 'application/json' ||
mediaType === 'application/javascript' ||
mediaType === 'application/typescript' ||
mediaType === 'application/xml' ||
mediaType === 'application/x-yaml'
) {
return true
}
const textExtensions = [
'.txt',
'.md',
'.markdown',
'.json',
'.csv',
'.tsv',
'.js',
'.ts',
'.tsx',
'.jsx',
'.html',
'.htm',
'.css',
'.scss',
'.yaml',
'.yml',
'.xml',
'.py',
'.rb',
'.go',
'.rs',
'.c',
'.cpp',
'.h',
'.hpp',
'.java',
'.sh',
'.bash',
'.zsh',
'.sql',
]
return textExtensions.some((ext) => filename.endsWith(ext))
}
const MAX_TEXT_BYTES = 50 * 1024
const MAX_TEXT_LINES = 1000
export function createReadAssetTool(userId: string): AgentTool<typeof readAssetSchema> {
return {
name: 'read_asset',
label: 'Read Asset',
description:
'Inspects and reads the content of an asset in the workspace. ' +
'Supports visual analysis for images, video frame extraction, PDF page rendering, and direct text reading for text/markdown files. ' +
'Pass assetId, and provide the specific config (imageConfig, videoConfig, or docConfig) while setting unused configs to null. ' +
'Optionally provide annotationId to view visual markups drawn on the asset.',
parameters: readAssetSchema,
execute: async (_toolCallId, params) => {
const { assetId, annotationId, videoConfig, docConfig } = params
if (!userId) {
throw new Error('User ID is required for authorization.')
}
await authzService.hasPermission({
user: { id: userId } as User,
permission: Permission.Read,
type: ResourceType.Asset,
id: assetId,
})
let asset = await prisma.asset.findUnique({
where: { id: assetId },
include: { storageKey: true },
})
if (!asset) {
throw new Error(`Asset with ID ${assetId} not found.`)
}
if (asset.type === 'version_stack') {
const latestVersion = await prisma.asset.findFirst({
where: { parentId: asset.id, isDeleted: false },
orderBy: { sortIndex: 'asc' },
include: { storageKey: true },
})
if (latestVersion) {
asset = latestVersion
}
}
const mediaInfo = asset.media as unknown as PrismaJson.MediaInfo | null
const proxyType = mediaInfo?.proxyType
const mediaType = asset.mediaType?.toLowerCase() || ''
const filename = asset.name?.toLowerCase() || ''
const bucket = process.env.S3_BUCKET || 'shumai'
// ----------------------------------------------------------------------
// Branch 1: Image Assets
// ----------------------------------------------------------------------
if (proxyType === 'image' || mediaType.startsWith('image/') || filename.endsWith('.psd')) {
let mediaKey = asset.storageKey?.key
if (mediaInfo?.imageTranscodes && mediaInfo.imageTranscodes.length > 0) {
mediaKey = mediaInfo.imageTranscodes[0].key || mediaKey
}
if (!mediaKey) {
throw new Error('No media content found for this asset.')
}
const { annotations } = await resolveAnnotationsById(asset.id, annotationId)
let keyToUse = mediaKey
if (annotations) {
const task = await prisma.workflowTask.create({
data: {
assetId: asset.id,
projectId: asset.projectId || 'none',
type: WorkflowTaskType.transcode_image_annotation,
status: WorkflowTaskStatus.pending,
payload: {
projectId: asset.projectId || 'none',
imageAnnotation: {
annotations,
},
},
},
})
const completedTask = await workflowService.executeWait(task)
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- transcode image annotation task output
const output = completedTask.output as any
if (output?.key) {
keyToUse = output.key
} else {
throw new Error('Image annotation transcode workflow failed to output S3 key.')
}
}
const { buffer, contentType } = await s3Service.getObject(bucket, keyToUse)
const mimeType =
contentType && contentType !== 'application/octet-stream' ? contentType : 'image/webp'
const downloadUrl = await s3Service.presign(
bucket,
keyToUse,
'GET',
true,
asset.name || undefined,
)
const content: Array<TextContent | ImageContent> = [
{
type: 'text',
text: `Image asset "${asset.name}" (ID: ${asset.id}):\n- S3 Key: "${keyToUse}"`,
},
{
type: 'image',
data: buffer.toString('base64'),
mimeType,
},
]
return {
content,
details: {
assetId: asset.id,
name: asset.name,
mediaType: asset.mediaType,
key: keyToUse,
downloadUrl,
sourceKeys: [keyToUse],
},
}
}
// ----------------------------------------------------------------------
// Branch 2: Video Assets
// ----------------------------------------------------------------------
if (proxyType === 'video' || mediaType.startsWith('video/')) {
if (!videoConfig) {
throw new Error(
'videoConfig with start, end, and count is required when inspecting video assets.',
)
}
const { start, end, count } = videoConfig
if (start < 0 || end < 0) {
throw new Error('Invalid video time range: start and end must be non-negative numbers.')
}
if (start > end) {
throw new Error(
`Invalid video time range: start (${start}) must be less than or equal to end (${end}).`,
)
}
if (count < 1) {
throw new Error(`Invalid frame count: count (${count}) must be at least 1.`)
}
const { annotations, timestamp: commentTimestamp } = await resolveAnnotationsById(
asset.id,
annotationId,
)
const task = await prisma.workflowTask.create({
data: {
assetId: asset.id,
projectId: asset.projectId || 'none',
type: WorkflowTaskType.transcode_screenshot,
status: WorkflowTaskStatus.pending,
payload: {
projectId: asset.projectId || 'none',
screenshot: {
start,
end,
count,
commentTimestamp,
annotations,
},
},
},
})
const completedTask = await workflowService.executeWait(task)
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- transcode screenshot output
const output = completedTask.output as any
const screenshots = output?.screenshots as
| Array<{ key: string; timestamp: number }>
| undefined
if (!screenshots || screenshots.length === 0) {
throw new Error('No screenshots were generated.')
}
const content: Array<TextContent | ImageContent> = []
const sourceKeys: string[] = []
const results: Array<{ key: string; timestamp: number; downloadUrl: string }> = []
const textLines: string[] = [
`Extracted ${screenshots.length} frame(s) from "${asset.name}" (${start}s - ${end}s):`,
]
for (const shot of screenshots) {
const { buffer, contentType } = await s3Service.getObject(bucket, shot.key)
const mimeType =
contentType && contentType !== 'application/octet-stream' ? contentType : 'image/webp'
const downloadUrl = await s3Service.presign(
bucket,
shot.key,
'GET',
true,
asset.name || undefined,
)
textLines.push(`- Frame at ${shot.timestamp}s: S3 Key: "${shot.key}"`)
content.push({
type: 'image',
data: buffer.toString('base64'),
mimeType,
})
sourceKeys.push(shot.key)
results.push({
key: shot.key,
timestamp: shot.timestamp,
downloadUrl,
})
}
content.unshift({
type: 'text',
text: textLines.join('\n'),
})
return {
content,
details: {
assetId: asset.id,
name: asset.name,
mediaType: asset.mediaType,
keys: sourceKeys,
results,
sourceKeys,
},
}
}
// ----------------------------------------------------------------------
// Branch 3: Documents & PDF Assets
// ----------------------------------------------------------------------
if (
proxyType === 'pdf' ||
mediaType === 'application/pdf' ||
isPlainTextAsset(mediaType, filename)
) {
if (!docConfig) {
throw new Error(
'docConfig with mode ("pages" or "text") is required when inspecting document/PDF assets.',
)
}
if (docConfig.mode === 'text') {
if (annotationId !== null) {
throw new Error(
'annotationId cannot be used with docConfig mode "text" because visual markups can only be rendered on visual pages. Use docConfig: { mode: "pages", startPage: ..., endPage: ... } instead.',
)
}
if (docConfig.startPage !== null || docConfig.endPage !== null) {
throw new Error(
'startPage and endPage must be null when docConfig mode is "text". To inspect specific page numbers, use mode: "pages".',
)
}
if (!isPlainTextAsset(mediaType, filename)) {
throw new Error(
`Asset "${asset.name}" is a binary PDF/document and cannot be read as raw text. Please call read_asset with docConfig: { mode: "pages", startPage: 1, endPage: ... } to view its visual pages.`,
)
}
const mediaKey = asset.storageKey?.key
if (!mediaKey) {
throw new Error(`No media content found for asset ${asset.id}.`)
}
const { buffer } = await s3Service.getObject(bucket, mediaKey)
const rawText = buffer.toString('utf-8')
let text = rawText
let isTruncated = false
const lines = text.split('\n')
if (lines.length > MAX_TEXT_LINES) {
text = lines.slice(0, MAX_TEXT_LINES).join('\n')
isTruncated = true
}
if (Buffer.byteLength(text, 'utf-8') > MAX_TEXT_BYTES) {
text = Buffer.from(text, 'utf-8').subarray(0, MAX_TEXT_BYTES).toString('utf-8')
isTruncated = true
}
if (isTruncated) {
text += `\n\n[Content truncated to ${MAX_TEXT_LINES} lines / ${MAX_TEXT_BYTES / 1024}KB limit. Use download_asset to process the full file.]`
}
const downloadUrl = await s3Service.presign(
bucket,
mediaKey,
'GET',
true,
asset.name || undefined,
)
return {
content: [
{
type: 'text',
text: `Document "${asset.name}" (ID: ${asset.id}, S3 Key: "${mediaKey}"):\n\n${text}`,
},
],
details: {
assetId: asset.id,
name: asset.name,
mediaType: asset.mediaType,
key: mediaKey,
downloadUrl,
sourceKeys: [mediaKey],
size: buffer.length,
isTruncated,
},
}
}
// docConfig.mode === 'pages'
if (docConfig.startPage === null || docConfig.endPage === null) {
throw new Error('startPage and endPage are required when docConfig mode is "pages".')
}
const start = docConfig.startPage
const end = docConfig.endPage
if (start < 1) {
throw new Error(`Invalid page range: start page (${start}) must be at least 1.`)
}
if (start > end) {
throw new Error(
`Invalid page range: start page (${start}) must be less than or equal to end page (${end}).`,
)
}
const MAX_PAGE_RANGE = 20
const requestedPages = end - start + 1
if (requestedPages > MAX_PAGE_RANGE) {
throw new Error(
`Page range (${requestedPages} pages requested) exceeds the maximum limit of ${MAX_PAGE_RANGE} pages per request.`,
)
}
const { annotations, timestamp: commentTimestamp } = await resolveAnnotationsById(
asset.id,
annotationId,
)
const task = await prisma.workflowTask.create({
data: {
assetId: asset.id,
projectId: asset.projectId || 'none',
type: WorkflowTaskType.transcode_pdf_pages,
status: WorkflowTaskStatus.pending,
payload: {
projectId: asset.projectId || 'none',
pdfPages: {
start,
end,
commentTimestamp,
annotations,
},
},
},
})
const completedTask = await workflowService.executeWait(task)
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- transcode pdf pages output
const output = completedTask.output as any
const pages = output?.pages as Array<{ key: string; page: number }> | undefined
if (!pages || pages.length === 0) {
throw new Error('No PDF page images were generated.')
}
const content: Array<TextContent | ImageContent> = []
const sourceKeys: string[] = []
const results: Array<{ key: string; page: number; downloadUrl: string }> = []
const textLines: string[] = [
`Rendered ${pages.length} page(s) of "${asset.name}" (Pages ${start} to ${end}):`,
]
for (const pageItem of pages) {
const { buffer, contentType } = await s3Service.getObject(bucket, pageItem.key)
const mimeType =
contentType && contentType !== 'application/octet-stream' ? contentType : 'image/webp'
const downloadUrl = await s3Service.presign(
bucket,
pageItem.key,
'GET',
true,
asset.name || undefined,
)
textLines.push(`- Page ${pageItem.page}: S3 Key: "${pageItem.key}"`)
content.push({
type: 'image',
data: buffer.toString('base64'),
mimeType,
})
sourceKeys.push(pageItem.key)
results.push({
key: pageItem.key,
page: pageItem.page,
downloadUrl,
})
}
content.unshift({
type: 'text',
text: textLines.join('\n'),
})
return {
content,
details: {
assetId: asset.id,
name: asset.name,
mediaType: asset.mediaType,
keys: sourceKeys,
results,
sourceKeys,
},
}
}
throw new Error(
`Unsupported asset type for read_asset. Asset ${asset.id} has media type "${mediaType || 'unknown'}". Use download_asset to download this file to disk.`,
)
},
}
}
@@ -1,209 +0,0 @@
import { describe, it, expect, vi, beforeEach } from 'vitest'
import { createReadPdfPagesTool } from './read-pdf-pages'
import {
prisma,
WorkflowTaskStatus,
type Asset,
type AssetComment,
type User,
type WorkflowTask,
} from '@shumai/db'
import { s3Service } from '@shumai/core/src/s3/s3'
import { workflowService } from '@shumai/workflow-core'
import { authzService } from '@shumai/core/src/authz/authz'
vi.mock('@shumai/db', async (importOriginal) => {
const actual = await importOriginal<typeof import('@shumai/db')>()
return {
...actual,
prisma: {
user: {
findUnique: vi.fn(),
},
asset: {
findUnique: vi.fn(),
},
assetComment: {
findUnique: vi.fn(),
},
workflowTask: {
create: vi.fn(),
},
},
}
})
vi.mock('@shumai/core/src/s3/s3', () => ({
s3Service: {
getObject: vi.fn(),
},
}))
vi.mock('@shumai/workflow-core', () => ({
workflowService: {
executeWait: vi.fn(),
},
}))
vi.mock('@shumai/core/src/authz/authz', () => ({
authzService: {
hasPermission: vi.fn(),
},
Permission: { Read: 'Read' },
ResourceType: { Asset: 'asset' },
}))
describe('readPdfPagesTool', () => {
beforeEach(() => {
vi.clearAllMocks()
vi.mocked(authzService.hasPermission).mockResolvedValue()
})
it('should trigger pdfPages transcode workflow and return image outputs', async () => {
vi.mocked(prisma.user.findUnique).mockResolvedValue({ id: 'user-1' } as User)
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'asset-1',
projectId: 'project-1',
media: { proxyType: 'pdf' },
} as unknown as Asset)
vi.mocked(prisma.assetComment.findUnique).mockResolvedValue({
id: 'comment-1',
assetId: 'asset-1',
second: 2,
annotation: [
{
type: 'box',
color: '#ff0000',
points: [
[0, 0],
[1, 1],
],
},
],
} as unknown as AssetComment)
vi.mocked(prisma.workflowTask.create).mockResolvedValue({
id: 'task-1',
} as unknown as WorkflowTask)
vi.mocked(workflowService.executeWait).mockResolvedValue({
id: 'task-1',
status: WorkflowTaskStatus.completed,
output: {
pages: [
{ key: 'pdf_pages/page_1.webp', page: 1 },
{ key: 'pdf_pages/page_2.webp', page: 2 },
],
},
} as unknown as WorkflowTask)
vi.mocked(s3Service.getObject).mockImplementation(async (_bucket: string, key: string) => {
return {
buffer: Buffer.from(`fake-bytes-for-${key}`),
contentType: 'image/webp',
} as unknown as { buffer: Buffer; contentType: string }
})
const tool = createReadPdfPagesTool('user-1', 'comment-1')
const result = await tool.execute('call-1', { assetId: 'asset-1', start: 1, end: 2 })
expect(authzService.hasPermission).toHaveBeenCalledWith({
user: { id: 'user-1' },
permission: 'Read',
type: 'asset',
id: 'asset-1',
})
expect(prisma.workflowTask.create).toHaveBeenCalledWith({
data: {
assetId: 'asset-1',
projectId: 'project-1',
type: 'transcode_pdf_pages',
status: 'pending',
payload: {
projectId: 'project-1',
pdfPages: {
start: 1,
end: 2,
commentTimestamp: 2,
annotations: [
{
type: 'box',
color: '#ff0000',
points: [
[0, 0],
[1, 1],
],
},
],
},
},
},
})
expect(result.content.length).toBe(2)
expect(result.content[0].type).toBe('image')
expect((result.content[0] as unknown as { data: string }).data).toBe(
Buffer.from('fake-bytes-for-pdf_pages/page_1.webp').toString('base64'),
)
expect(result.details.sourceKeys).toEqual(['pdf_pages/page_1.webp', 'pdf_pages/page_2.webp'])
})
it('should throw error if start page is less than 1', async () => {
const tool = createReadPdfPagesTool('user-1')
await expect(tool.execute('call-1', { assetId: 'asset-1', start: 0, end: 5 })).rejects.toThrow(
'Invalid page range: start page (0) must be at least 1.',
)
})
it('should throw error if start page is greater than end page', async () => {
const tool = createReadPdfPagesTool('user-1')
await expect(tool.execute('call-1', { assetId: 'asset-1', start: 5, end: 2 })).rejects.toThrow(
'Invalid page range: start page (5) must be less than or equal to end page (2).',
)
})
it('should throw error if page range exceeds maximum limit of 20', async () => {
const tool = createReadPdfPagesTool('user-1')
await expect(tool.execute('call-1', { assetId: 'asset-1', start: 1, end: 25 })).rejects.toThrow(
'Page range (25 pages requested) exceeds the maximum limit of 20 pages per request.',
)
})
it('should throw error if asset not found', async () => {
vi.mocked(prisma.asset.findUnique).mockResolvedValue(null)
const tool = createReadPdfPagesTool('user-1')
await expect(tool.execute('call-1', { assetId: 'asset-1', start: 1, end: 3 })).rejects.toThrow(
'Asset with ID asset-1 not found.',
)
})
it('should throw error if proxyType is not pdf', async () => {
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'asset-1',
projectId: 'project-1',
media: { proxyType: 'image' },
} as unknown as Asset)
const tool = createReadPdfPagesTool('user-1')
await expect(tool.execute('call-1', { assetId: 'asset-1', start: 1, end: 3 })).rejects.toThrow(
'Asset asset-1 is not a PDF or document.',
)
})
it('should throw error if proxyType is null/undefined (unprocessed)', async () => {
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'asset-1',
projectId: 'project-1',
media: null,
} as unknown as Asset)
const tool = createReadPdfPagesTool('user-1')
await expect(tool.execute('call-1', { assetId: 'asset-1', start: 1, end: 3 })).rejects.toThrow(
'Asset asset-1 is not a PDF or document.',
)
})
})
-152
View File
@@ -1,152 +0,0 @@
import { Type } from 'typebox'
import { type AgentTool } from '@earendil-works/pi-agent-core'
import { type ImageContent } from '@earendil-works/pi-ai'
import { prisma, WorkflowTaskType, WorkflowTaskStatus, type User } from '@shumai/db'
import { s3Service } from '@shumai/core/src/s3/s3'
import { workflowService } from '@shumai/workflow-core'
import { authzService, Permission, ResourceType } from '@shumai/core/src/authz/authz'
const readPdfPagesSchema = Type.Object({
assetId: Type.String({
description: 'Asset ID of the PDF document. This parameter is required.',
}),
start: Type.Number({ description: 'Start page number (1-based index)' }),
end: Type.Number({ description: 'End page number (1-based index)' }),
})
export function createReadPdfPagesTool(
userId: string,
userCommentId?: string | null,
): AgentTool<typeof readPdfPagesSchema> {
return {
name: 'read_pdf_pages',
label: 'Read PDF Pages',
description:
'Converts specific page range of a PDF document into 1080p WebP images for visual analysis. Maximum 20 pages allowed per call.',
parameters: readPdfPagesSchema,
execute: async (_toolCallId, params) => {
if (params.start < 1) {
throw new Error(`Invalid page range: start page (${params.start}) must be at least 1.`)
}
if (params.start > params.end) {
throw new Error(
`Invalid page range: start page (${params.start}) must be less than or equal to end page (${params.end}).`,
)
}
const MAX_PAGE_RANGE = 20
const requestedPages = params.end - params.start + 1
if (requestedPages > MAX_PAGE_RANGE) {
throw new Error(
`Page range (${requestedPages} pages requested) exceeds the maximum limit of ${MAX_PAGE_RANGE} pages per request.`,
)
}
const targetAssetId = params.assetId
if (!userId) {
throw new Error('User ID is required for authorization.')
}
await authzService.hasPermission({
user: { id: userId } as User,
permission: Permission.Read,
type: ResourceType.Asset,
id: targetAssetId,
})
let asset = await prisma.asset.findUnique({
where: { id: targetAssetId },
})
if (!asset) {
throw new Error(`Asset with ID ${targetAssetId} not found.`)
}
if (asset.type === 'version_stack') {
const latestVersion = await prisma.asset.findFirst({
where: { parentId: asset.id, isDeleted: false },
orderBy: { sortIndex: 'asc' },
})
if (latestVersion) {
asset = latestVersion
}
}
const proxyType = (asset.media as PrismaJson.MediaInfo | null)?.proxyType
if (proxyType !== 'pdf') {
throw new Error(`Asset ${targetAssetId} is not a PDF or document.`)
}
// Fetch trigger comment details for comment page and annotations
let commentTimestamp: number | null = null
let annotations: unknown = null
if (userCommentId) {
const comment = await prisma.assetComment.findUnique({
where: { id: userCommentId },
})
if (comment && comment.assetId === targetAssetId) {
commentTimestamp = comment.second !== null ? comment.second : null
if (comment.annotation) {
const list = comment.annotation
if (Array.isArray(list) && list.length > 0) {
annotations = list
}
}
}
}
// Trigger transcode workflow to render PDF pages
const task = await prisma.workflowTask.create({
data: {
assetId: targetAssetId,
projectId: asset.projectId || 'none',
type: WorkflowTaskType.transcode_pdf_pages,
status: WorkflowTaskStatus.pending,
payload: {
projectId: asset.projectId || 'none',
pdfPages: {
start: params.start,
end: params.end,
commentTimestamp,
annotations: annotations as PrismaJson.AnnotationList,
},
},
},
})
const completedTask = await workflowService.executeWait(task)
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const output = completedTask.output as any
const pages = output?.pages as Array<{ key: string; page: number }> | undefined
if (!pages || pages.length === 0) {
throw new Error('No PDF page images were generated.')
}
const content: ImageContent[] = []
const sourceKeys: string[] = []
const bucket = process.env.S3_BUCKET || 'shumai'
for (const pageItem of pages) {
const { buffer, contentType } = await s3Service.getObject(bucket, pageItem.key)
const mimeType =
contentType && contentType !== 'application/octet-stream' ? contentType : 'image/webp'
content.push({
type: 'image',
data: buffer.toString('base64'),
mimeType,
})
sourceKeys.push(pageItem.key)
}
return {
content,
details: {
sourceKeys,
},
}
},
}
}
+47 -4
View File
@@ -51,6 +51,8 @@ describe('createReadThreadTool', () => {
id: 'root-1',
message: 'Initial top-level question',
creator: { name: 'Alice' },
second: 12.5,
annotation: [{ type: 'box' }],
// eslint-disable-next-line @typescript-eslint/no-explicit-any
} as any)
vi.mocked(prisma.assetComment.findMany).mockResolvedValue([])
@@ -59,7 +61,9 @@ describe('createReadThreadTool', () => {
const result = await tool.execute('call-2', { threadId: 'root-1' })
const textContent = (result.content[0] as { text: string }).text
expect(textContent).toContain('Thread Root [Alice] (root-1): Initial top-level question')
expect(textContent).toContain(
'Thread Root [Alice] (id: root-1, time: 12.5s, has_markup: true): Initial top-level question',
)
expect(textContent).toContain('(No replies in this thread yet)')
})
@@ -75,6 +79,8 @@ describe('createReadThreadTool', () => {
id: 'reply-1',
message: 'First reply to thread',
creator: { name: 'Bob' },
second: 15.0,
annotation: [{ type: 'line' }],
},
{
id: 'reply-2',
@@ -88,8 +94,45 @@ describe('createReadThreadTool', () => {
const result = await tool.execute('call-3', { threadId: 'root-1' })
const textContent = (result.content[0] as { text: string }).text
expect(textContent).toContain('Thread Root [Alice] (root-1): Initial top-level question')
expect(textContent).toContain('- [Bob]: First reply to thread')
expect(textContent).toContain('- [Charlie]: Second reply to thread')
expect(textContent).toContain(
'Thread Root [Alice] (id: root-1, has_markup: false): Initial top-level question',
)
expect(textContent).toContain(
'- [Bob] (id: reply-1, time: 15s, has_markup: true): First reply to thread',
)
expect(textContent).toContain(
'- [Charlie] (id: reply-2, has_markup: false): Second reply to thread',
)
})
it('should label agent replies as [Ai Agent] when creator is null but sessionId exists', async () => {
vi.mocked(prisma.assetComment.findUnique).mockResolvedValue({
id: 'root-1',
message: 'Question to agent',
creator: { name: 'Alice' },
// eslint-disable-next-line @typescript-eslint/no-explicit-any
} as any)
vi.mocked(prisma.assetComment.findMany).mockResolvedValue([
{
id: 'reply-1',
message: 'Agent answer',
creator: null,
sessionId: 'session-123',
},
// eslint-disable-next-line @typescript-eslint/no-explicit-any
] as any)
const tool = createReadThreadTool()
const result = await tool.execute('call-agent', { threadId: 'root-1' })
const textContent = (result.content[0] as { text: string }).text
expect(textContent).toContain('- [Ai Agent] (id: reply-1, has_markup: false): Agent answer')
})
it('should describe threadId parameter as referencing <thread id="..." /> in <context>', () => {
const tool = createReadThreadTool()
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const description = (tool.parameters as any).properties.threadId.description
expect(description).toContain('<thread id="..." />')
})
})
+23 -5
View File
@@ -5,10 +5,24 @@ import { type AgentTool } from '@earendil-works/pi-agent-core'
const readThreadSchema = Type.Object({
threadId: Type.String({
description:
'The root comment ID of a thread (e.g. from [Thread ID: ...]). Do NOT pass an asset ID or project ID.',
'The root comment ID of a thread (from <thread id="..." /> in the <context> block). Do NOT pass an asset ID or project ID.',
}),
})
function formatCommentMeta(c: {
id: string
second?: number | null
annotation?: unknown
}): string {
const parts = [`id: ${c.id}`]
if (c.second !== null && c.second !== undefined) {
parts.push(`time: ${c.second}s`)
}
const hasMarkup = Boolean(c.annotation && Array.isArray(c.annotation) && c.annotation.length > 0)
parts.push(`has_markup: ${hasMarkup}`)
return `(${parts.join(', ')})`
}
export const createReadThreadTool = (): AgentTool<
typeof readThreadSchema,
{ threadId: string }
@@ -47,15 +61,19 @@ export const createReadThreadTool = (): AgentTool<
include: { creator: true },
})
const rootAuthor = rootComment.creator?.name || 'User'
let output = `Thread Root [${rootAuthor}] (${rootComment.id}): ${rootComment.message || ''}\n\nReplies (${replies.length}):`
const isRootAgent = rootComment.creator?.type === 'agent' || rootComment.sessionId !== null
const rootAuthor = rootComment.creator?.name || (isRootAgent ? 'Ai Agent' : 'User')
const rootMeta = formatCommentMeta(rootComment)
let output = `Thread Root [${rootAuthor}] ${rootMeta}: ${rootComment.message || ''}\n\nReplies (${replies.length}):`
if (replies.length === 0) {
output += '\n(No replies in this thread yet)'
} else {
for (const reply of replies) {
const author = reply.creator?.name || 'User'
output += `\n- [${author}]: ${reply.message || ''}`
const isAgent = reply.creator?.type === 'agent' || reply.sessionId !== null
const author = reply.creator?.name || (isAgent ? 'Ai Agent' : 'User')
const replyMeta = formatCommentMeta(reply)
output += `\n- [${author}] ${replyMeta}: ${reply.message || ''}`
}
}
-156
View File
@@ -1,156 +0,0 @@
import { describe, it, expect, vi, beforeEach } from 'vitest'
import { createScreenshotTool } from './screenshot'
import {
prisma,
WorkflowTaskStatus,
type Asset,
type AssetComment,
type User,
type WorkflowTask,
} from '@shumai/db'
import { s3Service } from '@shumai/core/src/s3/s3'
import { workflowService } from '@shumai/workflow-core'
import { authzService } from '@shumai/core/src/authz/authz'
vi.mock('@shumai/db', async (importOriginal) => {
const actual = await importOriginal<typeof import('@shumai/db')>()
return {
...actual,
prisma: {
user: {
findUnique: vi.fn(),
},
asset: {
findUnique: vi.fn(),
},
assetComment: {
findUnique: vi.fn(),
},
workflowTask: {
create: vi.fn(),
},
},
}
})
vi.mock('@shumai/core/src/s3/s3', () => ({
s3Service: {
getObject: vi.fn(),
},
}))
vi.mock('@shumai/workflow-core', () => ({
workflowService: {
executeWait: vi.fn(),
},
}))
vi.mock('@shumai/core/src/authz/authz', () => ({
authzService: {
hasPermission: vi.fn(),
},
Permission: { Read: 'Read' },
ResourceType: { Asset: 'asset' },
}))
describe('screenshotTool', () => {
beforeEach(() => {
vi.clearAllMocks()
})
it('should trigger screenshot transcode workflow and return image outputs', async () => {
vi.mocked(prisma.user.findUnique).mockResolvedValue({ id: 'user-1' } as User)
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'asset-1',
projectId: 'project-1',
} as unknown as Asset)
vi.mocked(prisma.assetComment.findUnique).mockResolvedValue({
id: 'comment-1',
assetId: 'asset-1',
second: 5.0,
annotation: [
{
type: 'line',
color: '#00ff00',
points: [
[0, 0],
[1, 1],
],
},
],
} as unknown as AssetComment)
vi.mocked(prisma.workflowTask.create).mockResolvedValue({
id: 'task-1',
} as unknown as WorkflowTask)
vi.mocked(workflowService.executeWait).mockResolvedValue({
id: 'task-1',
status: WorkflowTaskStatus.completed,
output: {
screenshots: [
{ key: 'screenshots/shot1.webp', timestamp: 0.0 },
{ key: 'screenshots/shot2.webp', timestamp: 5.0 },
],
},
} as unknown as WorkflowTask)
vi.mocked(s3Service.getObject).mockImplementation(async (_bucket: string, key: string) => {
return {
buffer: Buffer.from(`fake-bytes-for-${key}`),
contentType: 'image/webp',
} as unknown as { buffer: Buffer; contentType: string }
})
const tool = createScreenshotTool('user-1', 'comment-1')
const result = await tool.execute('call-1', { assetId: 'asset-1', start: 0, end: 10, count: 2 })
expect(authzService.hasPermission).toHaveBeenCalledWith({
user: { id: 'user-1' },
permission: 'Read',
type: 'asset',
id: 'asset-1',
})
expect(prisma.workflowTask.create).toHaveBeenCalledWith({
data: {
assetId: 'asset-1',
projectId: 'project-1',
type: 'transcode_screenshot',
status: 'pending',
payload: {
projectId: 'project-1',
screenshot: {
start: 0,
end: 10,
count: 2,
commentTimestamp: 5.0,
annotations: [
{
type: 'line',
color: '#00ff00',
points: [
[0, 0],
[1, 1],
],
},
],
},
},
},
})
expect(result.content.length).toBe(2)
expect(result.content[0].type).toBe('image')
expect((result.content[0] as unknown as { data: string }).data).toBe(
Buffer.from('fake-bytes-for-screenshots/shot1.webp').toString('base64'),
)
expect((result.content[1] as unknown as { data: string }).data).toBe(
Buffer.from('fake-bytes-for-screenshots/shot2.webp').toString('base64'),
)
expect(result.details.sourceKeys).toEqual(['screenshots/shot1.webp', 'screenshots/shot2.webp'])
})
})
-123
View File
@@ -1,123 +0,0 @@
import { Type } from 'typebox'
import { type AgentTool } from '@earendil-works/pi-agent-core'
import { type ImageContent } from '@earendil-works/pi-ai'
import { prisma, WorkflowTaskType, WorkflowTaskStatus, type User } from '@shumai/db'
import { s3Service } from '@shumai/core/src/s3/s3'
import { workflowService } from '@shumai/workflow-core'
import { authzService, Permission, ResourceType } from '@shumai/core/src/authz/authz'
const screenshotSchema = Type.Object({
assetId: Type.String({
description: 'The asset ID of the video asset. This parameter is required.',
}),
start: Type.Number({ description: 'Start time in seconds' }),
end: Type.Number({ description: 'End time in seconds' }),
count: Type.Number({ description: 'Number of screenshots to take' }),
})
export function createScreenshotTool(
userId: string,
userCommentId?: string | null,
): AgentTool<typeof screenshotSchema> {
return {
name: 'screenshot',
label: 'Take Video Screenshots',
description:
'Takes screenshots for a video at specified interval. It returns the images to the agent context.',
parameters: screenshotSchema,
execute: async (_toolCallId, params) => {
const assetId = params.assetId
if (!userId) {
throw new Error('User ID is required for authorization.')
}
await authzService.hasPermission({
user: { id: userId } as User,
permission: Permission.Read,
type: ResourceType.Asset,
id: assetId,
})
const asset = await prisma.asset.findUnique({
where: { id: assetId },
})
if (!asset) {
throw new Error(`Asset with ID ${assetId} not found.`)
}
// Fetch trigger comment details for timestamp and annotations
let commentTimestamp: number | null = null
let annotations: unknown = null
if (userCommentId) {
const comment = await prisma.assetComment.findUnique({
where: { id: userCommentId },
})
if (comment && comment.assetId === assetId) {
commentTimestamp = comment.second !== null ? comment.second : null
if (comment.annotation) {
const list = comment.annotation
if (Array.isArray(list) && list.length > 0) {
annotations = list
}
}
}
}
// Trigger transcode workflow to take screenshots
const task = await prisma.workflowTask.create({
data: {
assetId,
projectId: asset.projectId || 'none',
type: WorkflowTaskType.transcode_screenshot,
status: WorkflowTaskStatus.pending,
payload: {
projectId: asset.projectId || 'none',
screenshot: {
start: params.start,
end: params.end,
count: params.count,
commentTimestamp,
annotations: annotations as PrismaJson.AnnotationList,
},
},
},
})
const completedTask = await workflowService.executeWait(task)
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const output = completedTask.output as any
const screenshots = output?.screenshots as
| Array<{ key: string; timestamp: number }>
| undefined
if (!screenshots || screenshots.length === 0) {
throw new Error('No screenshots were generated.')
}
const content: ImageContent[] = []
const sourceKeys: string[] = []
const bucket = process.env.S3_BUCKET || 'shumai'
for (const shot of screenshots) {
const { buffer, contentType } = await s3Service.getObject(bucket, shot.key)
const mimeType =
contentType && contentType !== 'application/octet-stream' ? contentType : 'image/webp'
content.push({
type: 'image',
data: buffer.toString('base64'),
mimeType,
})
sourceKeys.push(shot.key)
}
return {
content,
details: {
sourceKeys,
},
}
},
}
}
@@ -735,4 +735,75 @@ describe('Agent Chat Workflow', () => {
}),
)
})
it('should include annotations and second in direct chat messageContext', async () => {
const task = await prisma.workflowTask.create({
data: {
type: 'chat',
status: 'pending',
assetId: 'a1',
payload: {
projectId: 'p1',
agent: {
prompt: 'Look at the highlighted zone',
agentId: 'b1',
sessionId: 'session-markup-1',
second: 14.2,
annotations: [{ type: 'rectangle', x: 20, y: 30, width: 40, height: 50 }],
},
},
},
})
await agentChat(task)
expect(mockActivities.agentChatActivity).toHaveBeenCalledWith(
expect.objectContaining({
messageContext: expect.objectContaining({
position: { type: 'time', seconds: 14.2 },
annotations: [{ type: 'rectangle', x: 20, y: 30, width: 40, height: 50 }],
}),
}),
)
})
it('should include userCommentId as id in comment messageContext', async () => {
mockActivities.getCommentActivity.mockResolvedValue({
id: 'comment-with-id-123',
creatorId: 'user-c1',
message: 'Check this out',
replyToId: null,
second: 5.0,
annotation: [{ type: 'arrow' }],
attachments: [],
})
const task = await prisma.workflowTask.create({
data: {
type: 'chat',
status: 'pending',
assetId: 'a1',
payload: {
projectId: 'p1',
agent: {
agentId: 'b1',
userCommentId: 'comment-with-id-123',
},
},
},
})
await agentChat(task)
expect(mockActivities.agentChatActivity).toHaveBeenCalledWith(
expect.objectContaining({
messageContext: expect.objectContaining({
id: 'comment-with-id-123',
position: { type: 'time', seconds: 5.0 },
annotation: true,
annotations: [{ type: 'arrow' }],
}),
}),
)
})
})
+24 -1
View File
@@ -61,6 +61,8 @@ export async function agentChat(task: WorkflowTask): Promise<void> {
let commentAnnotation = false
let commentCreatorId: string | undefined
let commentAnnotations: unknown[] | undefined
if (userCommentId) {
userComment = await executeActivity(agentWorkerQueue, getCommentActivity, userCommentId)
if (!userComment) {
@@ -77,6 +79,7 @@ export async function agentChat(task: WorkflowTask): Promise<void> {
userComment.annotation.length > 0
) {
commentAnnotation = true
commentAnnotations = userComment.annotation
}
// 2. Create Placeholder Comment
@@ -101,6 +104,17 @@ export async function agentChat(task: WorkflowTask): Promise<void> {
} else {
prompt = payload.agent?.prompt || ''
attachmentImageUrls = payload.agent?.imageUrls || []
if (payload.agent?.second !== null && payload.agent?.second !== undefined) {
commentTimestamp = payload.agent.second
}
if (
payload.agent?.annotations &&
Array.isArray(payload.agent.annotations) &&
payload.agent.annotations.length > 0
) {
commentAnnotation = true
commentAnnotations = payload.agent.annotations
}
}
// 3. Get Asset
@@ -181,6 +195,7 @@ export async function agentChat(task: WorkflowTask): Promise<void> {
mediaType: attProxyType as ShumaiAttachedFileContext['mediaType'],
mimeType: att.asset.mediaType || undefined,
path: filePath || undefined,
url: (att.asset as { url?: string }).url || undefined,
})
attachedAssets.push({ id: att.asset.id, name: att.asset.name, type: att.asset.type })
}
@@ -205,6 +220,7 @@ export async function agentChat(task: WorkflowTask): Promise<void> {
mediaType: fileProxyType as ShumaiAttachedFileContext['mediaType'],
mimeType: file.mediaType || undefined,
path: filePath || undefined,
url: (file as { url?: string }).url || undefined,
})
attachedAssets.push({ id: file.id, name: file.name, type: file.type })
}
@@ -233,6 +249,7 @@ export async function agentChat(task: WorkflowTask): Promise<void> {
mediaType: refProxyType as ShumaiAttachedFileContext['mediaType'],
mimeType: referencedAsset.mediaType || undefined,
path: filePath || undefined,
url: (referencedAsset as { url?: string }).url || undefined,
})
attachedAssets.push({
id: referencedAsset.id,
@@ -281,12 +298,18 @@ export async function agentChat(task: WorkflowTask): Promise<void> {
// Build structured ShumaiMessageContext
const messageContext: ShumaiMessageContext = {
...(userCommentId ? { id: userCommentId } : {}),
...(userInfo
? { user: { id: targetUserId!, name: userInfo.name, role: userInfo.role } }
: {}),
currentAsset: currentAssetContext,
...(mediaPosition ? { position: mediaPosition } : {}),
...(commentAnnotation ? { annotation: true } : {}),
...(commentAnnotation
? {
annotation: true,
annotations: commentAnnotations as ShumaiMessageContext['annotations'],
}
: {}),
...(attachedFiles.length > 0 ? { attachedFiles } : {}),
...(referencedAssets.length > 0 ? { referencedAssets } : {}),
}
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/api",
"version": "0.4.1",
"version": "0.4.2",
"private": true,
"type": "module",
"main": "./src/index.ts",
+42
View File
@@ -148,6 +148,48 @@ describe('Chat API', () => {
)
})
it('accepts chat request with annotations, second, and attachmentIds', async () => {
vi.mocked(chatService.startOrContinueChat).mockResolvedValue({
sessionId: 'session1',
taskId: 'task1',
})
vi.mocked(chatService.getNewSessionMessages).mockResolvedValue({
messages: [],
lastEntryId: null,
})
vi.mocked(chatService.getChatWorkflowStatus).mockResolvedValue({
status: 'completed',
output: {},
})
const res = await app.request('/teams/team1/chat', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
agentId: 'agent1',
textPrompt: 'Review annotation',
attachedFiles: ['att1'],
annotations: [{ type: 'rectangle', x: 10, y: 20 }],
second: 5.5,
}),
})
expect(res.status).toBe(200)
expect(chatService.startOrContinueChat).toHaveBeenCalledWith(
expect.objectContaining({ id: 'user1' }),
'team1',
expect.objectContaining({
agentId: 'agent1',
textPrompt: 'Review annotation',
attachedFiles: ['att1'],
annotations: [{ type: 'rectangle', x: 10, y: 20 }],
second: 5.5,
}),
)
})
it('sends ping keep-alive SSE events when there are no new messages', async () => {
vi.mocked(chatService.startOrContinueChat).mockResolvedValue({
sessionId: 'session1',
+192 -2
View File
@@ -163,13 +163,13 @@ describe('provider api', () => {
const res = await app.request('/providers/p1', {
method: 'PUT',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ config, models }),
body: JSON.stringify({ name: 'new-name', config, models }),
})
expect(res.status).toBe(200)
const data = await res.json()
expect(data.config.baseUrl).toBe('https://new-url.com')
expect(providerService.update).toHaveBeenCalledWith('t1', 'p1', config, models)
expect(providerService.update).toHaveBeenCalledWith('t1', 'p1', config, models, 'new-name')
expect(authzService.hasPermission).toHaveBeenCalledWith({
user: expect.anything(),
permission: Permission.Admin,
@@ -184,6 +184,111 @@ describe('provider api', () => {
})
})
it('POST /providers/:id/models creates a model', async () => {
vi.mocked(providerService.getById).mockResolvedValue({
id: 'p1',
teamId: 't1',
} as unknown as Awaited<ReturnType<typeof providerService.getById>>)
const modelData = {
modelId: 'm1',
name: 'Model 1',
config: {
reasoning: true,
input: ['text' as const],
contextWindow: 128000,
maxTokens: 4096,
cost: { input: 1, output: 2, cacheRead: 0, cacheWrite: 0 },
},
}
vi.mocked(providerService.createModel).mockResolvedValue({
id: 'm-db-1',
providerId: 'p1',
modelId: 'm1',
name: 'Model 1',
config: modelData.config,
createdAt: new Date(),
updatedAt: new Date(),
})
const res = await app.request('/providers/p1/models', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(modelData),
})
expect(res.status).toBe(201)
const data = await res.json()
expect(data.id).toBe('m-db-1')
expect(providerService.createModel).toHaveBeenCalledWith('t1', 'p1', modelData)
expect(authzService.hasPermission).toHaveBeenCalledWith({
user: expect.anything(),
permission: Permission.Admin,
type: ResourceType.Provider,
id: 'p1',
})
expect(auditLogService.logAction).toHaveBeenCalledWith({
action: 'provider_update',
teamId: 't1',
userId: 'user1',
itemId: 'p1',
})
})
it('PUT /providers/:id/models/:modelDbId updates a model', async () => {
vi.mocked(providerService.getById).mockResolvedValue({
id: 'p1',
teamId: 't1',
} as unknown as Awaited<ReturnType<typeof providerService.getById>>)
const updateData = {
name: 'Updated Model',
}
vi.mocked(providerService.updateModel).mockResolvedValue({
id: 'm-db-1',
providerId: 'p1',
modelId: 'm1',
name: 'Updated Model',
config: {
reasoning: false,
input: ['text'],
contextWindow: 128000,
maxTokens: 4096,
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
},
createdAt: new Date(),
updatedAt: new Date(),
})
const res = await app.request('/providers/p1/models/m-db-1', {
method: 'PUT',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(updateData),
})
expect(res.status).toBe(200)
const data = await res.json()
expect(data.name).toBe('Updated Model')
expect(providerService.updateModel).toHaveBeenCalledWith('t1', 'p1', 'm-db-1', updateData)
})
it('DELETE /providers/:id/models/:modelDbId deletes a model', async () => {
vi.mocked(providerService.getById).mockResolvedValue({
id: 'p1',
teamId: 't1',
} as unknown as Awaited<ReturnType<typeof providerService.getById>>)
vi.mocked(providerService.deleteModel).mockResolvedValue({
id: 'm-db-1',
} as unknown as Awaited<ReturnType<typeof providerService.deleteModel>>)
const res = await app.request('/providers/p1/models/m-db-1', {
method: 'DELETE',
})
expect(res.status).toBe(200)
const data = await res.json()
expect(data.success).toBe(true)
expect(providerService.deleteModel).toHaveBeenCalledWith('t1', 'p1', 'm-db-1')
})
it('DELETE /providers/:id deletes provider', async () => {
vi.mocked(providerService.getById).mockResolvedValue({
id: 'p1',
@@ -214,4 +319,89 @@ describe('provider api', () => {
itemId: 'p1',
})
})
it('POST /teams/:teamId/providers/sync-check checks for updates', async () => {
vi.mocked(providerService.checkUpdates).mockResolvedValue({
providers: [
{
name: 'deepseek',
isNewProvider: true,
config: { api: 'openai-responses', apiKey: 'DEEPSEEK_API_KEY' },
models: [],
},
],
totalNewProviders: 1,
totalNewModels: 0,
})
const res = await app.request('/teams/t1/providers/sync-check', {
method: 'POST',
})
expect(res.status).toBe(200)
const data = await res.json()
expect(data.totalNewProviders).toBe(1)
expect(data.providers).toHaveLength(1)
expect(providerService.checkUpdates).toHaveBeenCalledWith('t1')
expect(authzService.hasPermission).toHaveBeenCalledWith({
user: expect.anything(),
permission: Permission.Admin,
type: ResourceType.Team,
id: 't1',
})
})
it('POST /teams/:teamId/providers/sync-apply applies sync selection', async () => {
vi.mocked(providerService.applySync).mockResolvedValue({
addedProviders: 1,
addedModels: 3,
})
const payload = {
providers: [
{
name: 'deepseek',
isNewProvider: true,
config: { api: 'openai-responses', apiKey: 'KEY' },
models: [
{
modelId: 'deepseek-v4-flash',
name: 'DeepSeek V4 Flash',
config: {
api: 'openai-responses',
reasoning: true,
input: ['text'],
contextWindow: 128000,
maxTokens: 4096,
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
},
},
],
},
],
}
const res = await app.request('/teams/t1/providers/sync-apply', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(payload),
})
expect(res.status).toBe(200)
const data = await res.json()
expect(data.addedProviders).toBe(1)
expect(data.addedModels).toBe(3)
expect(providerService.applySync).toHaveBeenCalledWith('t1', payload)
expect(authzService.hasPermission).toHaveBeenCalledWith({
user: expect.anything(),
permission: Permission.Admin,
type: ResourceType.Team,
id: 't1',
})
expect(auditLogService.logAction).toHaveBeenCalledWith({
action: 'provider_update',
teamId: 't1',
userId: 'user1',
})
})
})
+130 -3
View File
@@ -2,7 +2,14 @@ import { zValidator } from '@hono/zod-validator'
import { Hono } from 'hono'
import { authzService, Permission, ResourceType } from '@shumai/core/src/authz/authz'
import { providerService } from '@shumai/core/src/provider/provider'
import { createProviderRequestSchema, updateProviderRequestSchema, AuditAction } from '@shumai/dtos'
import {
createModelRequestSchema,
createProviderRequestSchema,
syncApplyRequestSchema,
updateModelRequestSchema,
updateProviderRequestSchema,
AuditAction,
} from '@shumai/dtos'
import type { Prisma } from '@shumai/db'
import { auditLogService } from '@shumai/core/src/auditLog/auditLog'
@@ -24,6 +31,46 @@ const route = new Hono<{ Variables: { user: User } }>()
return c.json(providers)
})
.post('/teams/:teamId/providers/sync-check', async (c) => {
const teamId = c.req.param('teamId')
const user = c.get('user')
await authzService.hasPermission({
user,
permission: Permission.Admin,
type: ResourceType.Team,
id: teamId,
})
const result = await providerService.checkUpdates(teamId)
return c.json(result)
})
.post(
'/teams/:teamId/providers/sync-apply',
zValidator('json', syncApplyRequestSchema),
async (c) => {
const teamId = c.req.param('teamId')
const user = c.get('user')
const body = c.req.valid('json')
await authzService.hasPermission({
user,
permission: Permission.Admin,
type: ResourceType.Team,
id: teamId,
})
const result = await providerService.applySync(teamId, body)
await auditLogService.logAction({
action: AuditAction.provider_update,
teamId,
userId: user?.id,
})
return c.json(result)
},
)
.get('/providers/:id/models', async (c) => {
const id = c.req.param('id')
const user = c.get('user')
@@ -67,7 +114,7 @@ const route = new Hono<{ Variables: { user: User } }>()
.put('/providers/:id', zValidator('json', updateProviderRequestSchema), async (c) => {
const id = c.req.param('id')
const user = c.get('user')
const { config, models } = c.req.valid('json')
const { name, config, models } = c.req.valid('json')
await authzService.hasPermission({
user,
@@ -79,7 +126,7 @@ const route = new Hono<{ Variables: { user: User } }>()
const providerBefore = await providerService.getById(id)
if (!providerBefore) throw new Error('Provider not found')
const provider = await providerService.update(providerBefore.teamId, id, config, models)
const provider = await providerService.update(providerBefore.teamId, id, config, models, name)
await auditLogService.logAction({
action: AuditAction.provider_update,
teamId: providerBefore.teamId,
@@ -89,6 +136,86 @@ const route = new Hono<{ Variables: { user: User } }>()
return c.json(provider)
})
.post('/providers/:id/models', zValidator('json', createModelRequestSchema), async (c) => {
const id = c.req.param('id')
const user = c.get('user')
const data = c.req.valid('json')
await authzService.hasPermission({
user,
permission: Permission.Admin,
type: ResourceType.Provider,
id,
})
const provider = await providerService.getById(id)
if (!provider) throw new Error('Provider not found')
const model = await providerService.createModel(provider.teamId, id, data)
await auditLogService.logAction({
action: AuditAction.provider_update,
teamId: provider.teamId,
userId: user?.id,
itemId: id,
})
return c.json(model, 201)
})
.put(
'/providers/:id/models/:modelDbId',
zValidator('json', updateModelRequestSchema),
async (c) => {
const id = c.req.param('id')
const modelDbId = c.req.param('modelDbId')
const user = c.get('user')
const data = c.req.valid('json')
await authzService.hasPermission({
user,
permission: Permission.Admin,
type: ResourceType.Provider,
id,
})
const provider = await providerService.getById(id)
if (!provider) throw new Error('Provider not found')
const model = await providerService.updateModel(provider.teamId, id, modelDbId, data)
await auditLogService.logAction({
action: AuditAction.provider_update,
teamId: provider.teamId,
userId: user?.id,
itemId: id,
})
return c.json(model)
},
)
.delete('/providers/:id/models/:modelDbId', async (c) => {
const id = c.req.param('id')
const modelDbId = c.req.param('modelDbId')
const user = c.get('user')
await authzService.hasPermission({
user,
permission: Permission.Admin,
type: ResourceType.Provider,
id,
})
const providerBefore = await providerService.getById(id)
if (!providerBefore) throw new Error('Provider not found')
await providerService.deleteModel(providerBefore.teamId, id, modelDbId)
await auditLogService.logAction({
action: AuditAction.provider_update,
teamId: providerBefore.teamId,
userId: user?.id,
itemId: id,
})
return c.json({ success: true })
})
.delete('/providers/:id', async (c) => {
const id = c.req.param('id')
const user = c.get('user')
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/core",
"version": "0.4.1",
"version": "0.4.2",
"private": true,
"type": "module",
"main": "./src/index.ts",
+114 -13
View File
@@ -338,10 +338,89 @@ describe('AgentService', () => {
expect(entries[0].id).toBe(entry1.id)
expect(entries[1].id).toBe(entry2.id)
})
test('getSessionEntries dynamically attaches details.thread on custom_message entries with replies', async () => {
const db = prisma
const svc = new AgentService()
const { agent, team } = await setupTestData(db)
const project = await db.project.create({
data: { name: 'Test Project', teamId: team.id },
})
const asset = await db.asset.create({
data: {
name: 'video.mp4',
type: 'file',
status: 'uploaded',
projectId: project.id,
},
})
const user = await db.user.create({
data: { name: 'Alice', email: `alice-${Date.now()}@example.com` },
})
const rootComment = await db.assetComment.create({
data: {
id: 'comment-root-entries',
assetId: asset.id,
message: 'Top-level comment',
creatorId: user.id,
},
})
// Add a reply
await db.assetComment.create({
data: {
id: 'comment-reply-entries',
assetId: asset.id,
message: 'Reply to root',
creatorId: user.id,
replyToId: rootComment.id,
},
})
const session = await db.agentSession.create({
data: {
agentId: agent.id,
type: 'comment',
cwd: '/tmp',
},
})
const entry = await db.agentSessionEntry.create({
data: {
id: rootComment.id,
sessionId: session.id,
type: 'custom_message',
parentId: null,
data: {
customType: 'shumai_message',
content: 'Top-level comment',
display: true,
details: { user: { id: user.id, name: user.name } },
},
},
})
await db.agentSession.update({
where: { id: session.id },
data: { leafId: entry.id },
})
const entries = await svc.getSessionEntries({ sessionId: session.id })
expect(entries).toHaveLength(1)
expect(entries[0].id).toBe(rootComment.id)
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const entryDetails = (entries[0].entry as any).details
expect(entryDetails?.thread).toEqual({
id: rootComment.id,
replyCount: 1,
})
})
})
describe('listSessions', () => {
test('lists sessions for team and excludes pending id', async () => {
test('lists sessions for team, includes chat sessions and leaf comment sessions, hides main comment sessions and excludes pending id', async () => {
const db = prisma
const svc = new AgentService()
const { team, agent } = await setupTestData(db)
@@ -353,27 +432,43 @@ describe('AgentService', () => {
},
})
const session1 = await db.agentSession.create({
// 1. Chat session (userCommentId is null) - should be included!
const chatSession = await db.agentSession.create({
data: {
name: 'Session One',
name: 'Chat Session',
agentId: agent.id,
userId: user.id,
type: 'chat',
cwd: '/tmp',
userCommentId: null,
},
})
const session2 = await db.agentSession.create({
// 2. Main comment session (userCommentId is null) - should be HIDDEN / EXCLUDED!
const mainCommentSession = await db.agentSession.create({
data: {
name: 'Session Two',
name: 'Main Comment Session',
agentId: agent.id,
userId: user.id,
type: 'comment',
cwd: '/tmp',
userCommentId: null,
},
})
// Pending session - should be excluded!
// 3. Leaf comment session (userCommentId is not null) - should be included!
const leafCommentSession = await db.agentSession.create({
data: {
name: 'Leaf Comment Session',
agentId: agent.id,
userId: user.id,
type: 'comment',
cwd: '/tmp',
userCommentId: 'comment-123',
},
})
// 4. Pending session - should be excluded!
await db.agentSession.create({
data: {
id: 'pending',
@@ -388,15 +483,21 @@ describe('AgentService', () => {
const res = await svc.listSessions(team.id, { first: 10 })
expect(res.data).toHaveLength(2)
expect(res.data.map((s) => s.id)).not.toContain('pending')
const sessionIds = res.data.map((s) => s.id)
expect(sessionIds).toContain(session1.id)
expect(sessionIds).toContain(session2.id)
expect(sessionIds).not.toContain('pending')
expect(sessionIds).not.toContain(mainCommentSession.id)
expect(sessionIds).toContain(chatSession.id)
expect(sessionIds).toContain(leafCommentSession.id)
const found = res.data.find((s) => s.id === session1.id)
expect(found?.name).toBe('Session One')
expect(found?.type).toBe('chat')
expect(found?.creator?.name).toBe('Regular User')
const foundChat = res.data.find((s) => s.id === chatSession.id)
expect(foundChat?.name).toBe('Chat Session')
expect(foundChat?.type).toBe('chat')
expect(foundChat?.creator?.name).toBe('Regular User')
const foundLeafComment = res.data.find((s) => s.id === leafCommentSession.id)
expect(foundLeafComment?.name).toBe('Leaf Comment Session')
expect(foundLeafComment?.type).toBe('comment')
expect(foundLeafComment?.creator?.name).toBe('Regular User')
})
})
+40 -1
View File
@@ -549,21 +549,53 @@ export class AgentService {
ORDER BY depth DESC;
`
const entryIds = rows.map((r) => r.id)
const threadReplyCountMap = new Map<string, number>()
if (entryIds.length > 0) {
const threadCounts = await this.prismaClient.assetComment.groupBy({
by: ['replyToId'],
where: {
replyToId: { in: entryIds },
message: { not: '__CHAT__' },
},
_count: {
id: true,
},
})
for (const tc of threadCounts) {
if (tc.replyToId) {
threadReplyCountMap.set(tc.replyToId, tc._count.id)
}
}
}
return rows.map((r) => {
const payload = (r.data as Record<string, unknown>) || {}
const count = threadReplyCountMap.get(r.id)
let details = payload.details as Record<string, unknown> | undefined
if (count !== undefined && count > 0 && r.type === 'custom_message') {
details = {
...details,
thread: {
id: r.id,
replyCount: count,
},
}
}
const entryObj = {
id: r.id,
type: r.type || 'message',
parentId: r.parent_id,
timestamp: r.created_at.toISOString(),
...payload,
...(details ? { details } : {}),
}
return {
id: r.id,
sessionId: r.session_id || params.sessionId,
type: r.type,
parentId: r.parent_id,
data: r.data,
data: details !== payload.details ? { ...payload, details } : r.data,
createdAt: r.created_at,
entry: entryObj,
}
@@ -579,6 +611,13 @@ export class AgentService {
agent: {
teamId,
},
OR: [
{ type: { not: 'comment' } },
{
type: 'comment',
userCommentId: { not: null },
},
],
}
return await paginateQuery(
+24
View File
@@ -988,5 +988,29 @@ describe('ChatService', () => {
const msg2 = mapEntryToMessage(emptyEntry)
expect(msg2).toBeNull()
})
it('should forward annotations and second to workflow task payload', async () => {
const { user, team, project } = await setupBasicData()
const { sessionId, taskId } = await chatService.startOrContinueChat(user, team.id, {
agentId: 'test-agent-id',
textPrompt: 'Check out this area',
projectId: project.id,
annotations: [{ type: 'rectangle', x: 10, y: 20, width: 30, height: 40 }],
second: 8.5,
})
expect(sessionId).toBeDefined()
expect(taskId).toBeDefined()
const task = await prisma.workflowTask.findUnique({
where: { id: taskId },
})
expect(task).toBeDefined()
expect(task?.payload?.agent?.second).toBe(8.5)
expect(task?.payload?.agent?.annotations).toEqual([
{ type: 'rectangle', x: 10, y: 20, width: 30, height: 40 },
])
})
})
})
+4
View File
@@ -199,6 +199,8 @@ export class ChatService {
sessionId: passedSessionId,
contextAssetId,
projectId: passedProjectId,
second,
annotations,
} = req
// 1. Permissions verification
@@ -401,6 +403,8 @@ export class ChatService {
userId: user.id,
isNewChat: !passedSessionId,
hasAssetChanged,
second: second ?? undefined,
annotations: annotations ?? undefined,
},
},
},
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,42 @@
import type { ThinkingLevel, ThinkingLevelMap } from '@earendil-works/pi-ai'
export type ModelsDevReasoningOption =
| { type: 'toggle' }
| {
type: 'effort'
values: Array<
'none' | 'minimal' | 'low' | 'medium' | 'high' | 'xhigh' | 'max' | 'default' | null
>
}
| { type: 'budget_tokens'; min?: number; max?: number }
const THINKING_LEVELS: readonly ThinkingLevel[] = [
'minimal',
'low',
'medium',
'high',
'xhigh',
'max',
]
/**
* Converts models.dev verified effort values into Pi's selectable thinking levels.
* Values without a Pi equivalent (`default` and JSON `null`) are intentionally
* omitted.
*/
export function getEffortThinkingLevelMap(
options: readonly ModelsDevReasoningOption[],
): ThinkingLevelMap | undefined {
const effortValues = options.flatMap((option) => (option.type === 'effort' ? option.values : []))
if (effortValues.length === 0) return undefined
const supported = new Set(effortValues)
if (!THINKING_LEVELS.some((level) => supported.has(level)) && !supported.has('none'))
return undefined
const map: ThinkingLevelMap = { off: supported.has('none') ? 'none' : null }
for (const level of THINKING_LEVELS) {
map[level] = supported.has(level) ? level : null
}
return map
}
+335 -1
View File
@@ -1,8 +1,10 @@
import { prisma } from '@shumai/db'
import { setupTestDbHooks } from '@shumai/db/test'
import { teamService } from '@shumai/core/src/team/team'
import { agentService } from '@shumai/core/src/agent/agent'
import { providerService } from './provider'
import { describe, expect, it } from 'vitest'
import { describe, expect, it, vi } from 'vitest'
import * as generateModelsModule from './generator/generate-models'
import { ProviderConfigSerializable, providerModelSchema } from '@shumai/dtos'
import { getBuiltinProvidersMap } from '@shumai/core/src/provider/builtin'
import { z } from 'zod'
@@ -191,4 +193,336 @@ describe('ProviderService', () => {
const updatedAgent = updatedAgents.find((a) => a.id === agent!.id)
expect(updatedAgent?.modelId).toBe(initialModelId)
})
it('should support granular model CRUD operations and order by id desc', async () => {
const team = await teamService.ensureDefaultTeam()
const provider = await providerService.create(
team.id,
'test-granular-provider',
{ api: 'openai-completions', apiKey: 'key1' },
[],
)
// 1. createModel
const createdModel1 = await providerService.createModel(team.id, provider.id, {
modelId: 'custom-model-1',
name: 'Custom Model 1',
config: {
reasoning: true,
input: ['text', 'image'],
contextWindow: 200000,
maxTokens: 8192,
cost: { input: 1.5, output: 5, cacheRead: 0.1, cacheWrite: 0.5 },
},
})
expect(createdModel1.modelId).toBe('custom-model-1')
expect(createdModel1.name).toBe('Custom Model 1')
expect((createdModel1.config as { reasoning: boolean }).reasoning).toBe(true)
// Check duplicate modelId error
await expect(
providerService.createModel(team.id, provider.id, {
modelId: 'custom-model-1',
name: 'Duplicate',
config: {
reasoning: false,
input: ['text'],
contextWindow: 100000,
maxTokens: 4000,
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
},
}),
).rejects.toThrow('already exists')
// 2. createModel 2
const createdModel2 = await providerService.createModel(team.id, provider.id, {
modelId: 'custom-model-2',
name: 'Custom Model 2',
config: {
reasoning: false,
input: ['text'],
contextWindow: 100000,
maxTokens: 4000,
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
},
})
// 3. listModelsByProvider should return newest first (id desc)
const models = await providerService.listModelsByProvider(team.id, provider.id)
expect(models).toHaveLength(2)
expect(models[0].id).toBe(createdModel2.id)
expect(models[1].id).toBe(createdModel1.id)
// 4. updateModel
const updatedModel = await providerService.updateModel(team.id, provider.id, createdModel1.id, {
name: 'Updated Model 1',
config: {
reasoning: false,
input: ['text'],
contextWindow: 64000,
maxTokens: 2048,
cost: { input: 2, output: 8, cacheRead: 0.2, cacheWrite: 0.8 },
},
})
expect(updatedModel.name).toBe('Updated Model 1')
expect((updatedModel.config as { reasoning: boolean }).reasoning).toBe(false)
expect((updatedModel.config as { cost: { input: number } }).cost.input).toBe(2)
// 5. update provider config and name without touching models
const updatedProvider = await providerService.update(
team.id,
provider.id,
{ api: 'openai-completions', apiKey: 'new-key' },
undefined,
'renamed-provider',
)
expect(updatedProvider.name).toBe('renamed-provider')
expect(updatedProvider.config.apiKey).toBe('new-key')
const modelsAfterProviderUpdate = await providerService.listModelsByProvider(
team.id,
provider.id,
)
expect(modelsAfterProviderUpdate).toHaveLength(2)
// 6. deleteModel
await providerService.deleteModel(team.id, provider.id, createdModel2.id)
const modelsAfterDelete = await providerService.listModelsByProvider(team.id, provider.id)
expect(modelsAfterDelete).toHaveLength(1)
expect(modelsAfterDelete[0].id).toBe(createdModel1.id)
})
it('should propagate provider API protocol change to existing models', async () => {
const team = await teamService.ensureDefaultTeam()
// 1. Create provider with openai-responses
const provider = await providerService.create(
team.id,
'test-protocol-provider',
{ api: 'openai-responses', apiKey: 'key-1' },
[
{
modelId: 'm1',
name: 'Model 1',
config: {
api: 'openai-responses',
reasoning: false,
input: ['text'],
contextWindow: 128000,
maxTokens: 4096,
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
},
},
{
modelId: 'm2',
name: 'Model 2',
config: {
api: 'openai-responses',
reasoning: true,
input: ['text'],
contextWindow: 64000,
maxTokens: 2048,
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
},
},
],
)
const initialModels = await providerService.listModelsByProvider(team.id, provider.id)
expect(initialModels).toHaveLength(2)
expect((initialModels[0].config as { api: string }).api).toBe('openai-responses')
expect((initialModels[1].config as { api: string }).api).toBe('openai-responses')
// 2. Update provider global API protocol to anthropic-messages without passing models
const updatedProvider = await providerService.update(
team.id,
provider.id,
{ api: 'anthropic-messages', apiKey: 'key-2' },
undefined,
'test-protocol-provider',
)
expect(updatedProvider.config.api).toBe('anthropic-messages')
// 3. Verify that all existing models under this provider were updated
const modelsAfterUpdate = await providerService.listModelsByProvider(team.id, provider.id)
expect(modelsAfterUpdate).toHaveLength(2)
expect((modelsAfterUpdate[0].config as { api: string }).api).toBe('anthropic-messages')
expect((modelsAfterUpdate[1].config as { api: string }).api).toBe('anthropic-messages')
})
it(
'should check for updates and correctly segregate new vs existing providers',
{ timeout: 30000 },
async () => {
// 1. Check updates with empty team: all discovered providers are new
const emptyTeam = await prisma.team.create({
data: { name: 'Empty Provider Team' },
})
const initialCheck = await providerService.checkUpdates(emptyTeam.id)
expect(initialCheck.totalNewProviders).toBeGreaterThan(0)
expect(initialCheck.totalNewModels).toBeGreaterThan(0)
expect(initialCheck.providers.every((p) => p.isNewProvider)).toBe(true)
// 2. With default team (which has builtin providers initialized)
const teamWithBuiltins = await teamService.ensureDefaultTeam()
// 3. Check updates after init: existing providers should have isNewProvider: false
const afterInitCheck = await providerService.checkUpdates(teamWithBuiltins.id)
const existing = afterInitCheck.providers.find((p) => !p.isNewProvider)
if (existing) {
expect(existing.isNewProvider).toBe(false)
const dbModels = await providerService.listModelsByProvider(
teamWithBuiltins.id,
(await providerService.listByTeam(teamWithBuiltins.id)).find(
(p) => p.name === existing.name,
)!.id,
)
const dbModelIds = new Set(dbModels.map((m) => m.modelId))
for (const m of existing.models) {
expect(dbModelIds.has(m.modelId)).toBe(false)
}
}
},
)
it('should throw an error when upstream catalog fetch fails in checkUpdates', async () => {
const team = await teamService.ensureDefaultTeam()
const spy = vi
.spyOn(generateModelsModule, 'generateModels')
.mockRejectedValueOnce(new Error('Failed to load models.dev catalog: 502 Bad Gateway'))
await expect(providerService.checkUpdates(team.id)).rejects.toThrow(
'Failed to load models.dev catalog: 502 Bad Gateway',
)
spy.mockRestore()
})
it('should throw an error when upstream catalog is empty in checkUpdates', async () => {
const team = await teamService.ensureDefaultTeam()
const spy = vi.spyOn(generateModelsModule, 'generateModels').mockResolvedValueOnce({
providers: {},
groupedProviders: {},
sortedProviderIds: [],
})
await expect(providerService.checkUpdates(team.id)).rejects.toThrow(
'Failed to retrieve model catalog from upstream services',
)
spy.mockRestore()
})
it('should apply sync by adding selected providers and models without modifying existing ones', async () => {
const team = await teamService.ensureDefaultTeam()
// Create an existing provider with 1 model and a custom apiKey
const existingProvider = await providerService.create(
team.id,
'test-existing-provider',
{ api: 'openai-responses', apiKey: 'custom-api-key', baseUrl: 'https://example.com' },
[
{
modelId: 'existing-m1',
name: 'Existing Model 1',
config: {
api: 'openai-responses',
reasoning: false,
input: ['text'],
contextWindow: 128000,
maxTokens: 4096,
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
},
},
],
)
// Apply sync with 1 new provider and 1 new model on the existing provider
const syncResult = await providerService.applySync(team.id, {
providers: [
{
name: 'test-new-synced-provider',
isNewProvider: true,
config: {
api: 'anthropic-messages',
apiKey: 'ENV_SYNC_KEY',
baseUrl: 'https://api.anthropic.com',
},
models: [
{
modelId: 'new-p-m1',
name: 'New Provider Model 1',
config: {
api: 'anthropic-messages',
reasoning: false,
input: ['text'],
contextWindow: 200000,
maxTokens: 8192,
cost: { input: 3, output: 15, cacheRead: 0.3, cacheWrite: 3.75 },
},
},
],
},
{
name: 'test-existing-provider',
isNewProvider: false,
config: { api: 'openai-responses', apiKey: 'should-not-override' },
models: [
{
modelId: 'existing-m2',
name: 'Newly Added Model 2',
config: {
api: 'openai-responses',
reasoning: true,
input: ['text'],
contextWindow: 128000,
maxTokens: 4096,
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
},
},
// Duplicate model that already exists: should be ignored
{
modelId: 'existing-m1',
name: 'Existing Model 1 Overwrite Attempt',
config: {
api: 'openai-responses',
reasoning: false,
input: ['text'],
contextWindow: 999999,
maxTokens: 999999,
cost: { input: 9, output: 9, cacheRead: 9, cacheWrite: 9 },
},
},
],
},
],
})
expect(syncResult.addedProviders).toBe(1)
expect(syncResult.addedModels).toBe(2)
// Verify existing provider config was NOT modified
const reloadedExisting = await providerService.getById(existingProvider.id)
expect((reloadedExisting?.config as { apiKey: string }).apiKey).toBe('custom-api-key')
// Verify existing model was NOT modified
const existingModels = await providerService.listModelsByProvider(team.id, existingProvider.id)
expect(existingModels).toHaveLength(2)
const m1 = existingModels.find((m) => m.modelId === 'existing-m1')
expect(m1?.name).toBe('Existing Model 1')
expect((m1?.config as { contextWindow: number }).contextWindow).toBe(128000)
// Verify newly added model on existing provider
const m2 = existingModels.find((m) => m.modelId === 'existing-m2')
expect(m2).toBeDefined()
expect(m2?.name).toBe('Newly Added Model 2')
// Verify newly added provider
const providers = await providerService.listByTeam(team.id)
const newProvider = providers.find((p) => p.name === 'test-new-synced-provider')
expect(newProvider).toBeDefined()
expect(newProvider?.isBuiltin).toBe(true)
expect((newProvider?.config as { apiKey: string }).apiKey).toBe('ENV_SYNC_KEY')
const newProviderModels = await providerService.listModelsByProvider(team.id, newProvider!.id)
expect(newProviderModels).toHaveLength(1)
expect(newProviderModels[0].modelId).toBe('new-p-m1')
})
})
+346 -39
View File
@@ -1,6 +1,21 @@
import { prisma } from '@shumai/db'
import { ProviderConfigSerializable, providerModelSchema } from '@shumai/dtos'
import { getBuiltinProvidersMap } from '@shumai/core/src/provider/builtin'
import {
CreateModelRequest,
ProviderConfigSerializable,
providerModelSchema,
SyncApplyRequest,
SyncApplyResponse,
SyncCheckResponse,
SyncModelItem,
SyncProviderItem,
UpdateModelRequest,
} from '@shumai/dtos'
import {
ENV_MAP,
getBuiltinProvidersMap,
PRIORITY_PROVIDERS,
} from '@shumai/core/src/provider/builtin'
import { generateModels } from '@shumai/core/src/provider/generator/generate-models'
import { z } from 'zod'
type ProviderModel = z.infer<typeof providerModelSchema>
@@ -74,7 +89,7 @@ export class ProviderService {
providerId,
provider: { teamId },
},
orderBy: { id: 'asc' },
orderBy: { id: 'desc' },
})
}
@@ -82,7 +97,7 @@ export class ProviderService {
teamId: string,
name: string,
config: ProviderConfigSerializable,
models: ProviderModel[],
models: ProviderModel[] = [],
) {
return this.prismaClient.provider.create({
data: {
@@ -105,60 +120,177 @@ export class ProviderService {
teamId: string,
id: string,
config: ProviderConfigSerializable,
models: ProviderModel[],
models?: ProviderModel[],
name?: string,
) {
return this.prismaClient.$transaction(async (tx) => {
const existingModels = await tx.model.findMany({
where: { providerId: id },
})
const existingByModelId = new Map(existingModels.map((m) => [m.modelId, m]))
const incomingModelIds = new Set(models.map((m) => m.modelId))
// Delete models that are no longer in the incoming list
const modelsToDelete = existingModels.filter((m) => !incomingModelIds.has(m.modelId))
if (modelsToDelete.length > 0) {
await tx.model.deleteMany({
where: {
id: { in: modelsToDelete.map((m) => m.id) },
},
if (models !== undefined) {
const existingModels = await tx.model.findMany({
where: { providerId: id },
})
}
// Update existing models or create new ones
for (const m of models) {
const existing = existingByModelId.get(m.modelId)
if (existing) {
await tx.model.update({
where: { id: existing.id },
data: {
name: m.name,
config: m.config,
},
})
} else {
await tx.model.create({
data: {
providerId: id,
modelId: m.modelId,
name: m.name,
config: m.config,
const existingByModelId = new Map(existingModels.map((m) => [m.modelId, m]))
const incomingModelIds = new Set(models.map((m) => m.modelId))
// Delete models that are no longer in the incoming list
const modelsToDelete = existingModels.filter((m) => !incomingModelIds.has(m.modelId))
if (modelsToDelete.length > 0) {
await tx.model.deleteMany({
where: {
id: { in: modelsToDelete.map((m) => m.id) },
},
})
}
// Update existing models or create new ones
for (const m of models) {
const existing = existingByModelId.get(m.modelId)
const modelConfig = {
...m.config,
api: m.config?.api || config.api,
}
if (existing) {
await tx.model.update({
where: { id: existing.id },
data: {
name: m.name,
config: modelConfig,
},
})
} else {
await tx.model.create({
data: {
providerId: id,
modelId: m.modelId,
name: m.name,
config: modelConfig,
},
})
}
}
} else if (config.api) {
// Propagate provider API protocol change to all existing models
const existingModels = await tx.model.findMany({
where: { providerId: id },
})
for (const m of existingModels) {
const currentConfig = m.config
if (currentConfig.api !== config.api) {
await tx.model.update({
where: { id: m.id },
data: {
config: {
...currentConfig,
api: config.api,
},
},
})
}
}
}
// Update provider config
// Update provider config and optional name
return tx.provider.update({
where: { id, teamId },
data: {
config,
...(name ? { name } : {}),
},
include: { models: true },
})
})
}
async createModel(teamId: string, providerId: string, data: CreateModelRequest) {
const provider = await this.prismaClient.provider.findUnique({
where: { id: providerId, teamId },
})
if (!provider) {
throw new Error('Provider not found')
}
const existing = await this.prismaClient.model.findFirst({
where: {
providerId,
modelId: data.modelId,
},
})
if (existing) {
throw new Error(`Model with id "${data.modelId}" already exists for this provider`)
}
return this.prismaClient.model.create({
data: {
providerId,
modelId: data.modelId,
name: data.name ?? '',
config: data.config,
},
})
}
async updateModel(
teamId: string,
providerId: string,
modelDbId: string,
data: UpdateModelRequest,
) {
const provider = await this.prismaClient.provider.findUnique({
where: { id: providerId, teamId },
})
if (!provider) {
throw new Error('Provider not found')
}
const model = await this.prismaClient.model.findFirst({
where: { id: modelDbId, providerId },
})
if (!model) {
throw new Error('Model not found')
}
if (data.modelId && data.modelId !== model.modelId) {
const existingWithNewId = await this.prismaClient.model.findFirst({
where: {
providerId,
modelId: data.modelId,
},
})
if (existingWithNewId) {
throw new Error(`Model with id "${data.modelId}" already exists for this provider`)
}
}
return this.prismaClient.model.update({
where: { id: modelDbId },
data: {
...(data.modelId ? { modelId: data.modelId } : {}),
...(data.name !== undefined ? { name: data.name } : {}),
...(data.config ? { config: data.config } : {}),
},
})
}
async deleteModel(teamId: string, providerId: string, modelDbId: string) {
const provider = await this.prismaClient.provider.findUnique({
where: { id: providerId, teamId },
})
if (!provider) {
throw new Error('Provider not found')
}
const model = await this.prismaClient.model.findFirst({
where: { id: modelDbId, providerId },
})
if (!model) {
throw new Error('Model not found')
}
return this.prismaClient.model.delete({
where: { id: modelDbId },
})
}
async delete(teamId: string, id: string) {
const provider = await this.prismaClient.provider.findUnique({
where: { id, teamId },
@@ -172,6 +304,181 @@ export class ProviderService {
where: { id, teamId },
})
}
async checkUpdates(teamId: string): Promise<SyncCheckResponse> {
const catalog = await generateModels({ strict: true, timeoutMs: 30000 })
if (!catalog || catalog.sortedProviderIds.length === 0) {
throw new Error('Failed to retrieve model catalog from upstream services')
}
const existingProviders = await this.prismaClient.provider.findMany({
where: { teamId },
include: { models: true },
})
const existingProviderMap = new Map<string, (typeof existingProviders)[number]>()
for (const p of existingProviders) {
existingProviderMap.set(p.name, p)
}
const items: SyncProviderItem[] = []
let totalNewProviders = 0
let totalNewModels = 0
for (const providerId of catalog.sortedProviderIds) {
const modelsMap = catalog.providers[providerId]
if (!modelsMap) continue
const modelList = Object.values(modelsMap)
if (modelList.length === 0) continue
const existingProvider = existingProviderMap.get(providerId)
if (!existingProvider) {
const firstModel = modelList[0]
const newModels: SyncModelItem[] = modelList.map((m) => ({
modelId: m.id,
name: m.name || m.id,
config: {
api: m.api as ProviderConfigSerializable['api'],
reasoning: m.reasoning ?? false,
input: (m.input ?? ['text']) as ('text' | 'image')[],
contextWindow: m.contextWindow,
maxTokens: m.maxTokens,
cost: {
input: m.cost?.input ?? 0,
output: m.cost?.output ?? 0,
cacheRead: m.cost?.cacheRead ?? 0,
cacheWrite: m.cost?.cacheWrite ?? 0,
},
},
}))
items.push({
name: providerId,
isNewProvider: true,
config: {
api: firstModel.api as ProviderConfigSerializable['api'],
baseUrl: firstModel.baseUrl,
apiKey: ENV_MAP[providerId] || '',
},
models: newModels,
})
totalNewProviders++
totalNewModels += newModels.length
} else {
const existingModelIds = new Set(existingProvider.models.map((m) => m.modelId))
const newModels: SyncModelItem[] = []
for (const m of modelList) {
if (!existingModelIds.has(m.id)) {
newModels.push({
modelId: m.id,
name: m.name || m.id,
config: {
api: m.api as ProviderConfigSerializable['api'],
reasoning: m.reasoning ?? false,
input: (m.input ?? ['text']) as ('text' | 'image')[],
contextWindow: m.contextWindow,
maxTokens: m.maxTokens,
cost: {
input: m.cost?.input ?? 0,
output: m.cost?.output ?? 0,
cacheRead: m.cost?.cacheRead ?? 0,
cacheWrite: m.cost?.cacheWrite ?? 0,
},
},
})
}
}
if (newModels.length > 0) {
items.push({
name: providerId,
isNewProvider: false,
config: existingProvider.config as ProviderConfigSerializable,
models: newModels,
})
totalNewModels += newModels.length
}
}
}
items.sort((a, b) => {
const indexA = PRIORITY_PROVIDERS.indexOf(a.name)
const indexB = PRIORITY_PROVIDERS.indexOf(b.name)
if (indexA !== -1 && indexB !== -1) return indexA - indexB
if (indexA !== -1) return -1
if (indexB !== -1) return 1
return a.name.localeCompare(b.name)
})
return {
providers: items,
totalNewProviders,
totalNewModels,
}
}
async applySync(teamId: string, payload: SyncApplyRequest): Promise<SyncApplyResponse> {
let addedProviders = 0
let addedModels = 0
await this.prismaClient.$transaction(async (tx) => {
for (const providerItem of payload.providers) {
let providerId: string
if (providerItem.isNewProvider) {
const existing = await tx.provider.findFirst({
where: { teamId, name: providerItem.name },
})
if (!existing) {
const created = await tx.provider.create({
data: {
teamId,
name: providerItem.name,
isBuiltin: true,
config: providerItem.config,
},
})
providerId = created.id
addedProviders++
} else {
providerId = existing.id
}
} else {
const existing = await tx.provider.findFirst({
where: { teamId, name: providerItem.name },
})
if (!existing) continue
providerId = existing.id
}
for (const modelItem of providerItem.models) {
const existingModel = await tx.model.findFirst({
where: {
providerId,
modelId: modelItem.modelId,
},
})
if (!existingModel) {
await tx.model.create({
data: {
providerId,
modelId: modelItem.modelId,
name: modelItem.name || modelItem.modelId,
config: modelItem.config,
},
})
addedModels++
}
}
}
})
return { addedProviders, addedModels }
}
}
export const providerService = new ProviderService()
+10 -10
View File
@@ -21,12 +21,12 @@ describe('QuotaRule Wildcard Matchers', () => {
it('correctly matches agent tool name wildcards', () => {
const starTool = wildcardToRegex('*')
expect(starTool.test('analyze_image')).toBe(true)
expect(starTool.test('screenshot')).toBe(true)
expect(starTool.test('read_asset')).toBe(true)
expect(starTool.test('create_file')).toBe(true)
const exactTool = wildcardToRegex('analyze_image')
expect(exactTool.test('analyze_image')).toBe(true)
expect(exactTool.test('screenshot')).toBe(false)
const exactTool = wildcardToRegex('read_asset')
expect(exactTool.test('read_asset')).toBe(true)
expect(exactTool.test('create_file')).toBe(false)
})
})
@@ -405,29 +405,29 @@ describe('QuotaService', () => {
await quotaService.createRule(team.id, {
scopeMode: 'all_members',
resource: 'agent_tool_call_count',
resourceData: { name: 'analyze_image' },
resourceData: { name: 'read_asset' },
limit: 1,
period: '1hour',
enabled: true,
})
// First call to analyze_image
// First call to read_asset
await quotaService.consumeQuota(
{
teamId: team.id,
resource: 'agent_tool_call_count',
resourceData: { name: 'analyze_image' },
resourceData: { name: 'read_asset' },
},
1,
)
// 2nd call to analyze_image is blocked
// 2nd call to read_asset is blocked
await expect(
quotaService.checkQuota(
{
teamId: team.id,
resource: 'agent_tool_call_count',
resourceData: { name: 'analyze_image' },
resourceData: { name: 'read_asset' },
},
1,
),
+3 -3
View File
@@ -1225,12 +1225,12 @@ export class TranscodeService {
// 4. Extract each screenshot
for (const t of timestamps) {
const outName = `shot-${t.toFixed(3)}-${ulid()}.webp`
const outName = `shot-${t.toFixed(4)}-${ulid()}.webp`
const localShotPath = path.join(tmpDir, outName)
const args = [
'-ss',
t.toFixed(3),
t.toFixed(4),
'-i',
videoPath,
'-vframes',
@@ -1248,7 +1248,7 @@ export class TranscodeService {
const isMatch =
commentTimestamp !== undefined &&
commentTimestamp !== null &&
Math.abs(t - commentTimestamp) < 1e-4
Math.abs(t - commentTimestamp) <= 1e-3
// 5. Overlay annotations if timestamp matches commentTimestamp (within float tolerance)
if (isMatch && params.annotations && params.annotations.length > 0) {
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/db",
"version": "0.4.1",
"version": "0.4.2",
"private": true,
"type": "module",
"main": "./src/index.ts",
+2
View File
@@ -305,6 +305,8 @@ declare global {
assetIds?: string[]
isNewChat?: boolean
hasAssetChanged?: boolean
second?: number | null
annotations?: Record<string, unknown>[]
context?: import('@shumai/dtos').ShumaiMessageContext
}
+1 -8
View File
@@ -1,7 +1,6 @@
import { PostgreSqlContainer } from '@testcontainers/postgresql'
import { exec } from 'node:child_process'
import { promisify } from 'node:util'
import { Wait } from 'testcontainers'
const execAsync = promisify(exec)
@@ -9,13 +8,7 @@ let dbContainer: import('@testcontainers/postgresql').StartedPostgreSqlContainer
export async function globalTestSetup() {
console.log('Starting Testcontainers for global setup...')
dbContainer = await new PostgreSqlContainer('pgvector/pgvector:pg18')
.withWaitStrategy(
// casting to any due to testcontainers wait strategy type mismatch
// eslint-disable-next-line @typescript-eslint/no-explicit-any
Wait.forLogMessage('database system is ready to accept connections', 1) as any,
)
.start()
dbContainer = await new PostgreSqlContainer('pgvector/pgvector:pg18').start()
const databaseUrl = dbContainer.getConnectionUri()
console.log('Database started at:', databaseUrl)
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/dtos",
"version": "0.4.1",
"version": "0.4.2",
"private": true,
"type": "module",
"main": "./src/index.ts",
+17 -2
View File
@@ -10,15 +10,19 @@ export const chatRequestSchema = z
sessionId: z.string().optional(), // ID of existing session (to continue)
contextAssetId: z.string().optional(), // Active page context asset ID
projectId: z.string().optional(), // Target project ID
second: z.number().nullable().optional(),
annotations: z.array(z.record(z.string(), z.unknown())).optional(),
})
.superRefine((data, ctx) => {
const hasPrompt = data.textPrompt && data.textPrompt.trim().length > 0
const hasFiles = data.attachedFiles && data.attachedFiles.length > 0
const hasAssets = data.assetIds && data.assetIds.length > 0
if (!hasPrompt && !hasFiles && !hasAssets) {
const hasAnnotations = data.annotations && data.annotations.length > 0
if (!hasPrompt && !hasFiles && !hasAssets && !hasAnnotations) {
ctx.addIssue({
code: z.ZodIssueCode.custom,
message: 'At least one of textPrompt, attachedFiles, or assetIds must be provided.',
message:
'At least one of textPrompt, attachedFiles, assetIds, or annotations must be provided.',
path: ['textPrompt'],
})
}
@@ -63,6 +67,7 @@ export const shumaiAttachedFileContextSchema = z.object({
mediaType: z.enum(['image', 'video', 'pdf', 'audio', 'other']).optional(),
mimeType: z.string().optional(),
path: z.string().optional(),
url: z.string().optional(),
})
export type ShumaiAttachedFileContext = z.infer<typeof shumaiAttachedFileContextSchema>
@@ -74,7 +79,15 @@ export const shumaiMediaPositionSchema = z.discriminatedUnion('type', [
export type ShumaiMediaPosition = z.infer<typeof shumaiMediaPositionSchema>
export const shumaiThreadContextSchema = z.object({
id: z.string(),
replyCount: z.number().optional(),
})
export type ShumaiThreadContext = z.infer<typeof shumaiThreadContextSchema>
export const shumaiMessageContextSchema = z.object({
id: z.string().optional(),
user: z
.object({
id: z.string(),
@@ -82,9 +95,11 @@ export const shumaiMessageContextSchema = z.object({
role: z.string(),
})
.optional(),
thread: shumaiThreadContextSchema.optional(),
currentAsset: shumaiAssetContextSchema.optional(),
position: shumaiMediaPositionSchema.optional(),
annotation: z.boolean().optional(),
annotations: z.array(z.record(z.string(), z.unknown())).optional(),
attachedFiles: z.array(shumaiAttachedFileContextSchema).optional(),
referencedAssets: z.array(shumaiAttachedFileContextSchema).optional(),
})
+22 -6
View File
@@ -23,10 +23,10 @@ function formatAttr(name: string, value: string | undefined): string {
* object into a standardized, order-stable <context> XML block.
*
* Adheres strictly to the following invariants:
* 1. Tag sequence: <user>, <current_asset>, <position>, <annotation>, <attached_files>, <referenced_assets>
* 1. Tag sequence: <user>, <thread>, <current_asset>, <position>, <annotation>, <attached_files>, <referenced_assets>
* 2. Attribute sequence on every tag follows fixed, invariant order.
* 3. Arrays (attachedFiles, referencedAssets) sorted by id ascending.
* 4. Floats formatted with .toFixed(2), integers rounded.
* 4. Time position floats formatted with .toFixed(4), duration with .toFixed(2), integers rounded.
* 5. String values escaped via escapeXmlAttr.
* 6. Exact 2-space indentation and LF newlines.
*/
@@ -46,7 +46,19 @@ export function serializeContextToXml(context?: ShumaiMessageContext | null): st
}
}
// 2. <current_asset ...> ... </current_asset>
// 2. <thread ... />
if (context.thread) {
let threadAttrs = ''
if (context.thread.id !== undefined) threadAttrs += formatAttr('id', context.thread.id)
if (context.thread.replyCount !== undefined) {
threadAttrs += ` reply_count="${Math.round(context.thread.replyCount)}"`
}
if (threadAttrs) {
lines.push(` <thread${threadAttrs} />`)
}
}
// 3. <current_asset ...> ... </current_asset>
if (context.currentAsset) {
const ca = context.currentAsset
let assetAttrs = ''
@@ -89,15 +101,19 @@ export function serializeContextToXml(context?: ShumaiMessageContext | null): st
// 3. <position ... />
if (context.position) {
if (context.position.type === 'time') {
lines.push(` <position type="time" seconds="${context.position.seconds.toFixed(2)}" />`)
lines.push(` <position type="time" seconds="${context.position.seconds.toFixed(4)}" />`)
} else if (context.position.type === 'page') {
lines.push(` <position type="page" page="${Math.round(context.position.page)}" />`)
}
}
// 4. <annotation />
// 4. <annotation ... />
if (context.annotation) {
lines.push(' <annotation />')
if (context.id !== undefined) {
lines.push(` <annotation id="${escapeXmlAttr(context.id)}" />`)
} else {
lines.push(' <annotation />')
}
}
// 5. <attached_files> ... </attached_files>
+68 -2
View File
@@ -49,14 +49,35 @@ export type ProviderConfigSerializable = z.infer<typeof providerConfigSchema>
export const createProviderRequestSchema = z.object({
name: z.string().min(1, 'Provider name is required'),
config: providerConfigSchema,
models: z.array(providerModelSchema).min(1, 'At least one model is required'),
models: z.array(providerModelSchema).default([]),
})
export type CreateProviderRequest = z.infer<typeof createProviderRequestSchema>
export const updateProviderRequestSchema = z.object({
name: z.string().min(1, 'Provider name is required').optional(),
config: providerConfigSchema,
models: z.array(providerModelSchema).min(1, 'At least one model is required'),
models: z.array(providerModelSchema).optional(),
})
export type UpdateProviderRequest = z.infer<typeof updateProviderRequestSchema>
export const createModelRequestSchema = z.object({
modelId: z.string().min(1, 'Model ID is required'),
name: z.string().min(0).default(''),
config: providerModelConfigSchema,
})
export type CreateModelRequest = z.infer<typeof createModelRequestSchema>
export const updateModelRequestSchema = z.object({
modelId: z.string().min(1, 'Model ID is required').optional(),
name: z.string().min(0).optional(),
config: providerModelConfigSchema.optional(),
})
export type UpdateModelRequest = z.infer<typeof updateModelRequestSchema>
export const providerResponseSchema = z.object({
id: z.string(),
name: z.string(),
@@ -69,3 +90,48 @@ export const providerResponseSchema = z.object({
})
export type ProviderResponse = z.infer<typeof providerResponseSchema>
export const syncModelItemSchema = z.object({
modelId: z.string().min(1),
name: z.string().default(''),
config: providerModelConfigSchema,
})
export type SyncModelItem = z.infer<typeof syncModelItemSchema>
export const syncProviderItemSchema = z.object({
name: z.string(),
isNewProvider: z.boolean(),
config: providerConfigSchema,
models: z.array(syncModelItemSchema),
})
export type SyncProviderItem = z.infer<typeof syncProviderItemSchema>
export const syncCheckResponseSchema = z.object({
providers: z.array(syncProviderItemSchema),
totalNewProviders: z.number(),
totalNewModels: z.number(),
})
export type SyncCheckResponse = z.infer<typeof syncCheckResponseSchema>
export const syncApplyRequestSchema = z.object({
providers: z.array(
z.object({
name: z.string(),
isNewProvider: z.boolean(),
config: providerConfigSchema,
models: z.array(syncModelItemSchema),
}),
),
})
export type SyncApplyRequest = z.infer<typeof syncApplyRequestSchema>
export const syncApplyResponseSchema = z.object({
addedProviders: z.number(),
addedModels: z.number(),
})
export type SyncApplyResponse = z.infer<typeof syncApplyResponseSchema>
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/e2e",
"version": "0.4.1",
"version": "0.4.2",
"private": true,
"type": "module",
"dependencies": {
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/tableprinter",
"version": "0.4.1",
"version": "0.4.2",
"private": true,
"type": "module",
"main": "./src/index.ts",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/transcode",
"version": "0.4.1",
"version": "0.4.2",
"private": true,
"type": "module",
"main": "./src/index.ts",
@@ -107,4 +107,63 @@ describe('takeVideoScreenshotsWorkflow', () => {
},
})
})
it('should prefer highest-resolution video transcode proxy over master storageKey', async () => {
const task: WorkflowTask = {
id: 'task-screenshot-proxy',
assetId: 'asset-video-proxy',
type: WorkflowTaskType.transcode_screenshot,
status: WorkflowTaskStatus.pending,
sessionId: null,
output: null,
payload: {
projectId: 'proj-1',
screenshot: {
start: 1.2345,
end: 1.2345,
count: 1,
commentTimestamp: 1.2345,
annotations: [],
},
},
createdAt: new Date(),
updatedAt: new Date(),
heartbeat: null,
teamId: 'team-1',
projectId: 'proj-1',
uid: 'task-uid',
model: null,
inputTokens: 0,
outputTokens: 0,
}
mockActivities.getAssetActivity.mockResolvedValue({
id: 'asset-video-proxy',
storageKey: { key: 'raw-master.mp4' },
mediaType: 'video/mp4',
media: {
videoTranscodes: [
{ key: 'proxy-720p.mp4', height: 720, width: 1280 },
{ key: 'proxy-1080p.mp4', height: 1080, width: 1920 },
{ key: 'proxy-360p.mp4', height: 360, width: 640 },
],
},
})
mockActivities.takeScreenshotsActivity.mockResolvedValue([
{ key: 'files/asset-video-proxy/screenshots/shot1.webp', timestamp: 1.2345 },
])
await takeVideoScreenshotsWorkflow(task)
expect(mockActivities.takeScreenshotsActivity).toHaveBeenCalledWith({
assetKey: 'proxy-1080p.mp4',
assetId: 'asset-video-proxy',
start: 1.2345,
end: 1.2345,
count: 1,
commentTimestamp: 1.2345,
annotations: [],
})
})
})
@@ -17,10 +17,23 @@ export async function takeVideoScreenshotsWorkflow(task: WorkflowTask): Promise<
}
const { asset, key } = await fetchAssetWithKey(workerQueue, task.assetId)
let targetKey = key
if (asset?.media) {
const mediaInfo = asset.media as unknown as PrismaJson.MediaInfo
if (mediaInfo.videoTranscodes && mediaInfo.videoTranscodes.length > 0) {
const sorted = [...mediaInfo.videoTranscodes]
.filter((t) => t.key)
.sort((a, b) => (b.height || 0) - (a.height || 0))
if (sorted.length > 0 && sorted[0].key) {
targetKey = sorted[0].key
}
}
}
const { takeScreenshotsActivity } = getActivities()
const screenshots = await executeActivity(workerQueue, takeScreenshotsActivity, {
assetKey: key,
assetKey: targetKey,
assetId: asset.id,
start: payload.screenshot.start,
end: payload.screenshot.end,
@@ -0,0 +1,155 @@
// @vitest-environment happy-dom
import React from 'react'
import { cleanup, render, fireEvent } from '@testing-library/react'
import { afterEach, describe, expect, it, vi } from 'vitest'
import { QueryClient, QueryClientProvider } from '@tanstack/react-query'
import type { CommentInfo } from '@shumai/dtos'
import { ChatInput } from './message-input'
function createWrapper() {
const queryClient = new QueryClient({
defaultOptions: { queries: { retry: false } },
})
return ({ children }: { children: React.ReactNode }) => (
<QueryClientProvider client={queryClient}>{children}</QueryClientProvider>
)
}
describe('ChatInput auto-mention on reply', () => {
afterEach(() => {
cleanup()
vi.clearAllMocks()
})
it('automatically inserts creator mention when replyingTo is set', () => {
const mockComment: CommentInfo = {
id: 'comment-1',
assetId: 'asset-1',
message: 'Initial feedback',
annotations: null,
second: null,
creator: { id: 'user-alice', name: 'Alice Smith' },
replies: [],
attachments: [],
mentions: [],
createdAt: '2026-08-20T10:30:00.000Z',
updatedAt: '2026-08-20T10:30:00.000Z',
sessionId: null,
isCompleted: false,
completionLastChangedBy: null,
}
const onSendMessage = vi.fn()
const { rerender } = render(
<ChatInput projectId="proj-1" onSendMessage={onSendMessage} replyingTo={null} />,
{ wrapper: createWrapper() },
)
// Set replyingTo
rerender(
<ChatInput projectId="proj-1" onSendMessage={onSendMessage} replyingTo={mockComment} />,
)
// Check that mention node is inserted in contentEditable
const mentionSpan = document.querySelector('span[data-type="mention"]')
expect(mentionSpan).not.toBeNull()
expect(mentionSpan?.getAttribute('data-id')).toBe('user-alice')
expect(mentionSpan?.textContent).toBe('@Alice Smith')
// Click send and verify serialized text contains <@user-alice>
const sendButton = document.querySelector('button .lucide-arrow-up')?.closest('button')
expect(sendButton).toBeTruthy()
fireEvent.click(sendButton as HTMLButtonElement)
expect(onSendMessage).toHaveBeenCalledWith('<@user-alice>', [], [], 'comment-1', undefined, [])
})
it('does not insert mention if disableMentions is true', () => {
const mockComment: CommentInfo = {
id: 'comment-1',
assetId: 'asset-1',
message: 'Initial feedback',
annotations: null,
second: null,
creator: { id: 'user-alice', name: 'Alice Smith' },
replies: [],
attachments: [],
mentions: [],
createdAt: '2026-08-20T10:30:00.000Z',
updatedAt: '2026-08-20T10:30:00.000Z',
sessionId: null,
isCompleted: false,
completionLastChangedBy: null,
}
const onSendMessage = vi.fn()
render(
<ChatInput
projectId="proj-1"
onSendMessage={onSendMessage}
replyingTo={mockComment}
disableMentions={true}
/>,
{ wrapper: createWrapper() },
)
const mentionSpan = document.querySelector('span[data-type="mention"]')
expect(mentionSpan).toBeNull()
})
it('updates mention when replyingTo switches to a different comment author', () => {
const commentAlice: CommentInfo = {
id: 'comment-1',
assetId: 'asset-1',
message: 'Alice note',
annotations: null,
second: null,
creator: { id: 'user-alice', name: 'Alice Smith' },
replies: [],
attachments: [],
mentions: [],
createdAt: '2026-08-20T10:30:00.000Z',
updatedAt: '2026-08-20T10:30:00.000Z',
sessionId: null,
isCompleted: false,
completionLastChangedBy: null,
}
const commentBob: CommentInfo = {
id: 'comment-2',
assetId: 'asset-1',
message: 'Bob note',
annotations: null,
second: null,
creator: { id: 'user-bob', name: 'Bob Jones' },
replies: [],
attachments: [],
mentions: [],
createdAt: '2026-08-20T10:31:00.000Z',
updatedAt: '2026-08-20T10:31:00.000Z',
sessionId: null,
isCompleted: false,
completionLastChangedBy: null,
}
const onSendMessage = vi.fn()
const { rerender } = render(
<ChatInput projectId="proj-1" onSendMessage={onSendMessage} replyingTo={commentAlice} />,
{ wrapper: createWrapper() },
)
let mentionSpans = document.querySelectorAll('span[data-type="mention"]')
expect(mentionSpans).toHaveLength(1)
expect(mentionSpans[0].getAttribute('data-id')).toBe('user-alice')
// Switch to Bob's comment
rerender(<ChatInput projectId="proj-1" onSendMessage={onSendMessage} replyingTo={commentBob} />)
mentionSpans = document.querySelectorAll('span[data-type="mention"]')
expect(mentionSpans).toHaveLength(2)
expect(mentionSpans[1].getAttribute('data-id')).toBe('user-bob')
})
})
+410 -92
View File
@@ -19,6 +19,7 @@ import {
Paperclip,
Redo,
Square,
Trash2,
Undo,
XIcon,
} from 'lucide-react'
@@ -27,10 +28,12 @@ import { Button } from '../ui/button'
import { DrawAnnotation } from '../ui/icons'
import { Popover, PopoverContent, PopoverTrigger } from '../ui/popover'
import { ProgressCircle } from '../ui/progress-circle'
import { Tooltip, TooltipContent, TooltipProvider, TooltipTrigger } from '../ui/tooltip'
import { formatTimecode } from '../viewers/video/utils'
import { useUiStore } from '@/ui/stores/ui'
import { isImageFileName } from '@/ui/lib/media'
type UploadingFile = {
export type UploadingFile = {
id: string // A unique ID for the file, e.g., timestamp + name
file: File
progress: number
@@ -38,7 +41,16 @@ type UploadingFile = {
error?: string
}
interface ChatInputProps {
export interface AttachedFileMeta {
id: string
name: string
type: string
url?: string
mediaType?: string
mimeType?: string
}
export interface ChatInputProps {
projectId: string
onSendMessage: (
text: string,
@@ -46,16 +58,51 @@ interface ChatInputProps {
annotations?: Annotation[],
replyToId?: string | null,
second?: number | null,
attachedFilesMeta?: AttachedFileMeta[],
) => void
replyingTo?: CommentInfo | null
onCancelReply?: () => void
placeholder?: string
initialText?: string
hideAnnotationControl?: boolean
value?: string
onChangeText?: (text: string) => void
disabled?: boolean
// Mentions
allowMentions?: boolean
disableMentions?: boolean
// Local file attachments
allowAttachments?: boolean
// Referenced workspace assets
workspaceAssets?: Array<{ id: string; name: string; type: string }>
onRemoveWorkspaceAsset?: (id: string) => void
// Markup / Annotations
allowMarkup?: boolean
hideAnnotationControl?: boolean
canMarkup?: boolean
markupDisabledTooltip?: string
// Timestamps
allowTimestamp?: boolean
currentTime?: number
frameRate?: number
startTimecode?: string
formatTimestamp?: (second: number) => string
// Comment Thread Reply
replyingTo?: CommentInfo | null
onCancelReply?: () => void
// Bottom action bar extras (e.g. Agent Selector)
bottomLeftExtra?: React.ReactNode
// Streaming & Abort
isStreaming?: boolean
onAbort?: () => void
// Callbacks
onTyping?: () => void
}
@@ -80,13 +127,28 @@ export const ChatInput = React.forwardRef<HTMLDivElement, ChatInputProps>(
onSendMessage,
replyingTo,
onCancelReply,
placeholder,
initialText = '',
hideAnnotationControl = false,
value,
onChangeText,
disabled = false,
allowMentions,
disableMentions = false,
allowAttachments = true,
workspaceAssets,
onRemoveWorkspaceAsset,
allowMarkup,
hideAnnotationControl = false,
canMarkup = true,
markupDisabledTooltip,
allowTimestamp = true,
currentTime,
frameRate,
startTimecode,
formatTimestamp,
bottomLeftExtra,
isStreaming = false,
onAbort,
onTyping,
},
ref,
@@ -95,6 +157,10 @@ export const ChatInput = React.forwardRef<HTMLDivElement, ChatInputProps>(
const [viewingFile, setViewingFile] = useState<File | null>(null)
const [isTimestampEnabled, setIsTimestampEnabled] = useState(true)
const effectiveAllowMentions = allowMentions ?? !disableMentions
const effectiveAllowMarkup = allowMarkup ?? !hideAnnotationControl
const effectiveAllowTimestamp = allowTimestamp
const { videoTimeDisplayMode } = useUiStore()
const displayTime = React.useMemo(() => {
if (currentTime === undefined) return ''
@@ -145,6 +211,7 @@ export const ChatInput = React.forwardRef<HTMLDivElement, ChatInputProps>(
const paddingLeftClass = React.useMemo(() => {
if (
!isTimestampEnabled ||
!effectiveAllowTimestamp ||
currentTime === undefined ||
(!formatTimestamp && !frameRate) ||
hasContent
@@ -155,12 +222,21 @@ export const ChatInput = React.forwardRef<HTMLDivElement, ChatInputProps>(
return 'pl-[120px]'
}
return 'pl-[85px]'
}, [isTimestampEnabled, currentTime, frameRate, hasContent, videoTimeDisplayMode])
}, [
isTimestampEnabled,
effectiveAllowTimestamp,
currentTime,
frameRate,
hasContent,
videoTimeDisplayMode,
formatTimestamp,
])
const editorRef = useRef<HTMLDivElement>(null)
const fileInputRef = useRef<HTMLInputElement>(null)
const mentionRangeRef = useRef<Range | null>(null)
const mentionListRef = useRef<HTMLDivElement>(null)
const isInternalUpdateRef = useRef(false)
const { mutateAsync: uploadAttachment } = useMutation({
mutationFn: async ({
@@ -180,7 +256,16 @@ export const ChatInput = React.forwardRef<HTMLDivElement, ChatInputProps>(
})
useEffect(() => {
if (initialText && editorRef.current) {
if (value !== undefined && editorRef.current) {
if (isInternalUpdateRef.current) {
isInternalUpdateRef.current = false
return
}
if (editorRef.current.innerText !== value) {
editorRef.current.innerText = value
setHasContent(!!value.trim())
}
} else if (initialText && editorRef.current) {
editorRef.current.innerText = initialText
setHasContent(!!initialText.trim())
// Focus and move cursor to end
@@ -192,11 +277,66 @@ export const ChatInput = React.forwardRef<HTMLDivElement, ChatInputProps>(
sel?.removeAllRanges()
sel?.addRange(range)
}
}, [initialText])
}, [value, initialText])
// Focus editor when replyingTo becomes truthy
const prevReplyingToIdRef = useRef<string | null>(null)
const insertMention = useCallback(
(user: { id: string; name: string }) => {
if (!editorRef.current) return
const mentionNode = document.createElement('span')
mentionNode.contentEditable = 'false'
mentionNode.dataset.type = 'mention'
mentionNode.dataset.id = user.id
mentionNode.className =
'inline-block bg-blue-100 text-blue-700 dark:bg-blue-900/40 dark:text-blue-300 px-1.5 py-0.5 rounded-md text-sm font-medium align-middle select-none mx-0.5'
mentionNode.innerText = `@${user.name}`
const spaceNode = document.createTextNode('\u00A0')
editorRef.current.appendChild(mentionNode)
editorRef.current.appendChild(spaceNode)
// Move cursor after the space
const newRange = document.createRange()
const sel = window.getSelection()
newRange.setStartAfter(spaceNode)
newRange.collapse(true)
sel?.removeAllRanges()
sel?.addRange(newRange)
setHasContent(true)
const currentText = editorRef.current.innerText || ''
isInternalUpdateRef.current = true
onChangeText?.(currentText)
onTyping?.()
editorRef.current.focus()
},
[onChangeText, onTyping],
)
// Auto mention creator and focus editor when replyingTo is set or changed
useEffect(() => {
if (replyingTo && editorRef.current) {
if (replyingTo.id !== prevReplyingToIdRef.current) {
prevReplyingToIdRef.current = replyingTo.id
if (effectiveAllowMentions && replyingTo.creator?.id && replyingTo.creator?.name) {
const existingMentions = editorRef.current.querySelectorAll('span[data-type="mention"]')
const alreadyMentioned = Array.from(existingMentions).some(
(node) => (node as HTMLElement).dataset.id === replyingTo.creator.id,
)
if (!alreadyMentioned) {
insertMention({
id: replyingTo.creator.id,
name: replyingTo.creator.name,
})
return
}
}
}
editorRef.current.focus()
// Focus and move cursor to end
const range = document.createRange()
@@ -205,8 +345,10 @@ export const ChatInput = React.forwardRef<HTMLDivElement, ChatInputProps>(
range.collapse(false)
sel?.removeAllRanges()
sel?.addRange(range)
} else if (!replyingTo) {
prevReplyingToIdRef.current = null
}
}, [replyingTo])
}, [replyingTo, effectiveAllowMentions, insertMention])
// Clean up drawing mode on unmount
useEffect(() => {
@@ -254,10 +396,13 @@ export const ChatInput = React.forwardRef<HTMLDivElement, ChatInputProps>(
const handleInput = () => {
if (!editorRef.current) return
setHasContent(!!editorRef.current.innerText.trim())
const text = editorRef.current.innerText || ''
setHasContent(!!text.trim())
isInternalUpdateRef.current = true
onChangeText?.(text)
onTyping?.()
if (disableMentions) {
if (!effectiveAllowMentions) {
setShowMentionList(false)
mentionRangeRef.current = null
return
@@ -337,13 +482,63 @@ export const ChatInput = React.forwardRef<HTMLDivElement, ChatInputProps>(
setShowMentionList(false)
mentionRangeRef.current = null
setHasContent(true)
const currentText = editorRef.current.innerText || ''
isInternalUpdateRef.current = true
onChangeText?.(currentText)
onTyping?.()
editorRef.current.focus()
}
const handlePaste = (e: React.ClipboardEvent<HTMLDivElement>) => {
e.preventDefault()
const text = e.clipboardData.getData('text/plain')
if (!text) return
let selection = window.getSelection()
if (
!selection ||
selection.rangeCount === 0 ||
!editorRef.current?.contains(selection.anchorNode)
) {
if (editorRef.current) {
editorRef.current.focus()
const range = document.createRange()
range.selectNodeContents(editorRef.current)
range.collapse(false)
selection = window.getSelection()
selection?.removeAllRanges()
selection?.addRange(range)
}
}
const success =
typeof document.execCommand === 'function' &&
document.queryCommandSupported?.('insertText') &&
document.execCommand('insertText', false, text)
if (!success) {
const sel = window.getSelection()
if (sel && sel.rangeCount > 0) {
const range = sel.getRangeAt(0)
range.deleteContents()
const textNode = document.createTextNode(text)
range.insertNode(textNode)
range.setStartAfter(textNode)
range.collapse(true)
sel.removeAllRanges()
sel.addRange(range)
} else if (editorRef.current) {
editorRef.current.innerText += text
}
}
handleInput()
}
const handleKeyDown = (e: React.KeyboardEvent<HTMLDivElement>) => {
if (e.nativeEvent.isComposing) return
if (showMentionList && filteredEntities.length > 0) {
if (showMentionList && effectiveAllowMentions && filteredEntities.length > 0) {
if (e.key === 'ArrowUp') {
e.preventDefault()
setHighlightedIndex((prev) => (prev > 0 ? prev - 1 : filteredEntities.length - 1))
@@ -444,6 +639,18 @@ export const ChatInput = React.forwardRef<HTMLDivElement, ChatInputProps>(
.map((f) => f.attachmentId)
.filter((id): id is string => !!id)
const successfulAttachedFilesMeta: AttachedFileMeta[] = uploadingFiles
.filter((f) => !f.error && !!f.attachmentId)
.map((f) => ({
id: f.attachmentId!,
name: f.file.name,
type: f.file.type,
url:
f.file.type.startsWith('image/') || isImageFileName(f.file.name)
? URL.createObjectURL(f.file)
: undefined,
}))
// Serialize content
let processedText = ''
editorRef.current.childNodes.forEach((node) => {
@@ -466,20 +673,31 @@ export const ChatInput = React.forwardRef<HTMLDivElement, ChatInputProps>(
processedText = processedText.trim()
// Allow sending if annotations exist, even if text is empty
if (!processedText && successfulAttachmentIds.length === 0 && annotations.length === 0) return
const hasWorkspaceAssets = (workspaceAssets?.length ?? 0) > 0
// Allow sending if annotations, attachments, or workspace assets exist, even if text is empty
if (
!processedText &&
successfulAttachmentIds.length === 0 &&
annotations.length === 0 &&
!hasWorkspaceAssets
)
return
onSendMessage(
processedText,
successfulAttachmentIds,
annotations,
replyingTo?.id,
isTimestampEnabled ? currentTime : undefined,
effectiveAllowTimestamp && isTimestampEnabled ? currentTime : undefined,
successfulAttachedFilesMeta,
)
// Reset editor
editorRef.current.innerHTML = ''
setHasContent(false)
isInternalUpdateRef.current = true
onChangeText?.('')
setUploadingFiles([])
setViewingFile(null)
setIsDrawing(false)
@@ -490,13 +708,18 @@ export const ChatInput = React.forwardRef<HTMLDivElement, ChatInputProps>(
const isUploading = uploadingFiles.some((f) => !f.attachmentId && !f.error)
const hasSuccessfulUploads = uploadingFiles.some((f) => !!f.attachmentId)
const canSend = hasContent || hasSuccessfulUploads || annotations.length > 0
const showUpperPart = uploadingFiles.length > 0 || !!replyingTo
const hasWorkspaceAssets = (workspaceAssets?.length ?? 0) > 0
const effectiveDisabled = disabled || isStreaming
const canSend =
(hasContent || hasSuccessfulUploads || annotations.length > 0 || hasWorkspaceAssets) &&
!effectiveDisabled
const showUpperPart =
(workspaceAssets && workspaceAssets.length > 0) || uploadingFiles.length > 0 || !!replyingTo
return (
<div
ref={ref}
className="relative flex flex-col w-full border border-foreground/35 rounded-3xl shadow-sm transition-all duration-200"
className="relative flex flex-col w-full border border-foreground/35 rounded-3xl shadow-sm transition-all duration-200 bg-background"
>
<style>{`
[contenteditable]:empty:before {
@@ -506,8 +729,12 @@ export const ChatInput = React.forwardRef<HTMLDivElement, ChatInputProps>(
}
`}</style>
{showMentionList &&
effectiveAllowMentions &&
(membersLoading || filteredAgents.length > 0 || filteredHumans.length > 0) && (
<div className="absolute bottom-full left-4 mb-2 w-64 rounded-xl shadow-lg border border-border overflow-hidden z-20 bg-popover text-popover-foreground animate-in fade-in slide-in-from-bottom-2 duration-200">
<div
onMouseDown={(e) => e.preventDefault()}
className="absolute bottom-full left-4 mb-2 w-64 rounded-xl shadow-lg border border-border overflow-hidden z-20 bg-popover text-popover-foreground animate-in fade-in slide-in-from-bottom-2 duration-200"
>
<div ref={mentionListRef} className="h-64 overflow-y-auto">
{membersLoading ? (
<div className="p-2 space-y-4 animate-pulse">
@@ -545,12 +772,15 @@ export const ChatInput = React.forwardRef<HTMLDivElement, ChatInputProps>(
{filteredAgents.length > 0 && (
<>
<button
type="button"
onMouseDown={(e) => e.preventDefault()}
onClick={(e) => {
e.preventDefault()
e.stopPropagation()
setCollapsedSections((prev) => ({ ...prev, bots: !prev.bots }))
editorRef.current?.focus()
}}
className="w-full text-xs font-semibold text-muted-foreground px-3 py-2 bg-muted/50 border-b border-border flex items-center justify-between hover:bg-muted transition-colors"
className="w-full text-xs font-semibold text-muted-foreground px-3 py-2 bg-muted/50 border-b border-border flex items-center justify-between hover:bg-muted transition-colors cursor-pointer select-none"
>
<span>Agents</span>
<ChevronDown
@@ -562,11 +792,13 @@ export const ChatInput = React.forwardRef<HTMLDivElement, ChatInputProps>(
const actualIndex = index
return (
<button
type="button"
key={item.data.id}
data-index={actualIndex}
onMouseDown={(e) => e.preventDefault()}
onClick={() => handleSelectEntity(item)}
onMouseEnter={() => setHighlightedIndex(actualIndex)}
className={`w-full flex items-center gap-2 px-3 py-2 transition-colors text-left ${
className={`w-full flex items-center gap-2 px-3 py-2 transition-colors text-left cursor-pointer ${
actualIndex === highlightedIndex
? 'bg-accent text-accent-foreground'
: 'hover:bg-accent/50'
@@ -594,12 +826,15 @@ export const ChatInput = React.forwardRef<HTMLDivElement, ChatInputProps>(
{filteredHumans.length > 0 && (
<>
<button
type="button"
onMouseDown={(e) => e.preventDefault()}
onClick={(e) => {
e.preventDefault()
e.stopPropagation()
setCollapsedSections((prev) => ({ ...prev, users: !prev.users }))
editorRef.current?.focus()
}}
className="w-full text-xs font-semibold text-muted-foreground px-3 py-2 bg-muted/50 border-b border-border flex items-center justify-between hover:bg-muted transition-colors"
className="w-full text-xs font-semibold text-muted-foreground px-3 py-2 bg-muted/50 border-b border-border flex items-center justify-between hover:bg-muted transition-colors cursor-pointer select-none"
>
<span>Members</span>
<ChevronDown
@@ -608,15 +843,17 @@ export const ChatInput = React.forwardRef<HTMLDivElement, ChatInputProps>(
</button>
{!collapsedSections.users &&
filteredHumans.map((item, index) => {
const actualIndex =
(collapsedSections.bots ? 0 : filteredAgents.length) + index
const agentOffset = collapsedSections.bots ? 0 : filteredAgents.length
const actualIndex = index + agentOffset
return (
<button
type="button"
key={item.data.id}
data-index={actualIndex}
onMouseDown={(e) => e.preventDefault()}
onClick={() => handleSelectEntity(item)}
onMouseEnter={() => setHighlightedIndex(actualIndex)}
className={`w-full flex items-center gap-2 px-3 py-2 transition-colors text-left ${
className={`w-full flex items-center gap-2 px-3 py-2 transition-colors text-left cursor-pointer ${
actualIndex === highlightedIndex
? 'bg-accent text-accent-foreground'
: 'hover:bg-accent/50'
@@ -630,7 +867,7 @@ export const ChatInput = React.forwardRef<HTMLDivElement, ChatInputProps>(
className="object-cover"
/>
)}
<AvatarFallback className="text-[10px] bg-blue-400 text-black font-semibold">
<AvatarFallback className="text-[10px] bg-purple-400 text-black font-semibold">
{getInitials(item.data.name)}
</AvatarFallback>
</Avatar>
@@ -648,6 +885,35 @@ export const ChatInput = React.forwardRef<HTMLDivElement, ChatInputProps>(
{showUpperPart && (
<div className="flex flex-col gap-2 p-3 pb-0 animate-in fade-in slide-in-from-bottom-2 duration-200">
{/* Referenced Workspace Assets */}
{workspaceAssets && workspaceAssets.length > 0 && (
<div className="flex flex-col gap-1.5 min-h-0 max-h-[160px] overflow-y-auto border border-border/60 rounded-lg bg-muted/20 p-2.5 divide-y divide-border/50">
{workspaceAssets.map((asset) => (
<div
key={asset.id}
className="flex items-center justify-between py-1 text-xs group"
>
<div className="flex items-baseline gap-1.5 min-w-0 pr-2">
<span className="truncate font-medium text-foreground">{asset.name}</span>
<span className="text-[9px] text-muted-foreground uppercase flex-shrink-0">
{asset.type}
</span>
</div>
{onRemoveWorkspaceAsset && (
<button
onClick={() => onRemoveWorkspaceAsset(asset.id)}
className="text-muted-foreground hover:text-destructive transition-colors p-0.5 cursor-pointer"
title={m.remove_from_context?.() || 'Remove from context'}
>
<Trash2 className="h-3.5 w-3.5" />
</button>
)}
</div>
))}
</div>
)}
{/* Replying banner */}
{replyingTo && (
<div className="flex items-center justify-between bg-muted p-2 px-3 rounded-lg border border-border">
<div className="text-xs text-muted-foreground truncate mr-2">
@@ -662,6 +928,7 @@ export const ChatInput = React.forwardRef<HTMLDivElement, ChatInputProps>(
</div>
)}
{/* Local file uploads thumbnail strip */}
{uploadingFiles.length > 0 && (
<div className="flex gap-3 overflow-x-auto py-2 px-1 scrollbar-hide">
{uploadingFiles.map((upload) => (
@@ -687,7 +954,7 @@ export const ChatInput = React.forwardRef<HTMLDivElement, ChatInputProps>(
</div>
<button
onClick={() => removeFile(upload.id)}
className="absolute -top-1.5 -right-1.5 bg-gray-500 text-white rounded-full p-0.5 shadow-md opacity-0 group-hover:opacity-100 transition-opacity hover:bg-red-500 z-10"
className="absolute -top-1.5 -right-1.5 bg-gray-500 text-white rounded-full p-0.5 shadow-md opacity-0 group-hover:opacity-100 transition-opacity hover:bg-red-500 z-10 cursor-pointer"
>
<XIcon className="w-3 h-3" />
</button>
@@ -705,25 +972,31 @@ export const ChatInput = React.forwardRef<HTMLDivElement, ChatInputProps>(
onClick={() => editorRef.current?.focus()}
className="w-full px-4 pt-1 pb-1 cursor-text flow-root max-h-60 overflow-y-auto"
>
{isTimestampEnabled && currentTime !== undefined && frameRate !== undefined && (
<div
onClick={(e) => {
e.stopPropagation()
editorRef.current?.focus()
}}
className="float-left select-none bg-blue-100 dark:bg-blue-900/40 text-blue-600 dark:text-blue-300 px-2.5 py-0.5 mt-3 mr-2 rounded-sm text-xs font-mono font-bold flex items-center gap-1 cursor-text"
>
{displayTime}
</div>
)}
{effectiveAllowTimestamp &&
isTimestampEnabled &&
currentTime !== undefined &&
(frameRate !== undefined || formatTimestamp !== undefined) && (
<div
onClick={(e) => {
e.stopPropagation()
editorRef.current?.focus()
}}
className="float-left select-none bg-blue-100 dark:bg-blue-900/40 text-blue-600 dark:text-blue-300 px-2.5 py-0.5 mt-3 mr-2 rounded-sm text-xs font-mono font-bold flex items-center gap-1 cursor-text"
>
{displayTime}
</div>
)}
<div
ref={editorRef}
contentEditable
contentEditable={!disabled && !isStreaming}
suppressContentEditableWarning
onInput={handleInput}
onKeyDown={handleKeyDown}
data-placeholder={replyingTo ? m.reply_placeholder() : m.message_placeholder()}
className={`bg-transparent border-none focus:ring-0 resize-none min-h-[40px] leading-relaxed py-2 focus:outline-none block ${paddingLeftClass}`}
onPaste={handlePaste}
data-placeholder={
placeholder || (replyingTo ? m.reply_placeholder() : m.message_placeholder())
}
className={`bg-transparent border-none focus:ring-0 resize-none min-h-[40px] leading-relaxed py-2 focus:outline-none block whitespace-pre-wrap break-words ${paddingLeftClass}`}
/>
</div>
@@ -818,59 +1091,104 @@ export const ChatInput = React.forwardRef<HTMLDivElement, ChatInputProps>(
</button>
</div>
) : (
<div className="flex items-center gap-1">
<input
type="file"
ref={fileInputRef}
className="hidden"
multiple
onChange={handleFileSelect}
/>
{currentTime !== undefined && frameRate !== undefined && (
<Button
onClick={() => setIsTimestampEnabled(!isTimestampEnabled)}
variant={isTimestampEnabled ? 'secondary' : 'ghost'}
size="icon"
className={`p-2 rounded-full cursor-pointer ${isTimestampEnabled ? 'text-blue-600 bg-blue-50 dark:bg-blue-900/20' : ''}`}
title={isTimestampEnabled ? m.disable_timestamp() : m.enable_timestamp()}
>
<Clock className="w-4 h-4" />
</Button>
)}
<Button
onClick={() => fileInputRef.current?.click()}
className="p-2 rounded-full cursor-pointer"
variant="ghost"
size="icon"
title={m.attach_files()}
>
<Paperclip className="w-4 h-4" />
</Button>
{!hideAnnotationControl && (
<Button
onClick={() => setIsDrawing(true)}
variant="ghost"
size="icon"
className="p-2 rounded-full cursor-pointer"
title={m.toggle_annotation()}
>
<DrawAnnotation className="w-4 h-4" />
</Button>
<div className="flex items-center gap-1.5 min-w-0 max-w-[80%]">
{bottomLeftExtra}
{allowAttachments && (
<>
<input
type="file"
ref={fileInputRef}
className="hidden"
multiple
onChange={handleFileSelect}
/>
<Button
onClick={() => fileInputRef.current?.click()}
className="p-2 rounded-full cursor-pointer shrink-0"
variant="ghost"
size="icon"
title={m.attach_files()}
>
<Paperclip className="w-4 h-4" />
</Button>
</>
)}
{effectiveAllowTimestamp &&
currentTime !== undefined &&
(frameRate !== undefined || formatTimestamp !== undefined) && (
<Button
onClick={() => setIsTimestampEnabled(!isTimestampEnabled)}
variant={isTimestampEnabled ? 'secondary' : 'ghost'}
size="icon"
className={`p-2 rounded-full cursor-pointer shrink-0 ${isTimestampEnabled ? 'text-blue-600 bg-blue-50 dark:bg-blue-900/20' : ''}`}
title={isTimestampEnabled ? m.disable_timestamp() : m.enable_timestamp()}
>
<Clock className="w-4 h-4" />
</Button>
)}
{effectiveAllowMarkup &&
(canMarkup ? (
<Button
onClick={() => {
onTyping?.()
setIsDrawing(true)
}}
variant="ghost"
size="icon"
className="p-2 rounded-full cursor-pointer shrink-0"
title={m.toggle_annotation()}
>
<DrawAnnotation className="w-4 h-4" />
</Button>
) : (
<TooltipProvider>
<Tooltip>
<TooltipTrigger asChild>
<span className="inline-block cursor-not-allowed shrink-0">
<Button
disabled
variant="ghost"
size="icon"
className="p-2 rounded-full opacity-40 cursor-not-allowed pointer-events-none"
>
<DrawAnnotation className="w-4 h-4" />
</Button>
</span>
</TooltipTrigger>
<TooltipContent side="top">
<p className="text-xs">
{markupDisabledTooltip || m.markup_not_supported_for_this_file()}
</p>
</TooltipContent>
</Tooltip>
</TooltipProvider>
))}
</div>
)}
<button
onClick={handleSend}
disabled={!canSend || isUploading}
className={`p-2 rounded-full transition-all duration-200 flex items-center justify-center shrink-0 ${
canSend && !isUploading
? 'bg-blue-600 text-white shadow-md hover:bg-blue-700 hover:shadow-lg transform hover:-translate-y-0.5'
: 'bg-gray-400 cursor-not-allowed'
}`}
>
<ArrowUp className="w-4 h-4" />
</button>
{isStreaming ? (
<button
onClick={onAbort}
className="p-2 rounded-full transition-all duration-200 flex items-center justify-center shrink-0 shadow-sm bg-destructive text-destructive-foreground hover:bg-destructive/90 transform hover:-translate-y-0.5 cursor-pointer"
title="Stop generation"
>
<div className="h-4 w-4 flex items-center justify-center">
<div className="h-3 w-3 rounded-[2px] bg-white" />
</div>
</button>
) : (
<button
onClick={handleSend}
disabled={!canSend || isUploading}
className={`p-2 rounded-full transition-all duration-200 flex items-center justify-center shrink-0 ${
canSend && !isUploading
? 'bg-blue-600 text-white shadow-md hover:bg-blue-700 hover:shadow-lg transform hover:-translate-y-0.5 cursor-pointer'
: 'bg-gray-400 cursor-not-allowed opacity-50'
}`}
>
<ArrowUp className="w-4 h-4" />
</button>
)}
</div>
{viewingFile && (
@@ -6,6 +6,7 @@ import { QueryClient, QueryClientProvider } from '@tanstack/react-query'
import { ChatbotSidebar } from './chatbot-sidebar'
import { useChatbotStore, AGENT_PREFERENCE_STORAGE_KEY } from '@/ui/stores/chatbot'
import { client } from '@/ui/api/client'
import type { ChatMessage } from '@shumai/dtos'
const localStorageMock = (() => {
let store: Record<string, string> = {}
@@ -190,4 +191,281 @@ describe('ChatbotSidebar - Agent Selection & Preference Persistence', () => {
expect(localStorage.getItem(AGENT_PREFERENCE_STORAGE_KEY)).toBeNull()
})
})
it('renders user messages with markup badge, timestamp, and attachments and triggers onSelectMessage on click when matched', async () => {
vi.mocked(client.api.projects[':projectId']['chat-agents'].$get).mockResolvedValue({
ok: true,
json: async () => [{ id: 'agent-1', name: 'Agent 1', type: 'chat', enabled: true }],
} as unknown as Awaited<
ReturnType<(typeof client.api.projects)[':projectId']['chat-agents']['$get']>
>)
const mockSelectMessage = vi.fn()
const mockMessage = {
id: 'msg-1',
role: 'custom',
customType: 'shumai_message',
content: 'Fix the color in this scene',
details: {
annotation: true,
annotations: [{ type: 'rectangle', x: 5, y: 10 }],
position: { type: 'time', seconds: 15 },
currentAsset: { id: 'file-123', name: 'video.mp4', type: 'file' },
attachedFiles: [
{ id: 'att-1', name: 'color-ref.png', type: 'image', mediaType: 'image/png' },
],
},
} as unknown as ChatMessage
useChatbotStore.setState({
messages: [mockMessage],
})
const { getByText } = render(
<QueryClientProvider client={queryClient}>
<ChatbotSidebar
projectId="proj-1"
contextAssetId="file-123"
onSelectMessage={mockSelectMessage}
/>
</QueryClientProvider>,
)
expect(getByText('Fix the color in this scene')).toBeTruthy()
expect(getByText('00:15')).toBeTruthy()
expect(getByText('color-ref.png')).toBeTruthy()
const msgCardInner = getByText('Fix the color in this scene').closest('.bg-primary')
expect(msgCardInner?.className).not.toContain('ring-2')
const msgCard = getByText('Fix the color in this scene').closest('div')
msgCard?.click()
expect(mockSelectMessage).toHaveBeenCalledWith(mockMessage)
})
it('renders selected message card with border-blue-500 instead of ring when selectedMessageId matches', async () => {
const mockMessage = {
id: 'msg-selected',
role: 'custom',
customType: 'shumai_message',
content: 'Selected message content',
details: {
position: { type: 'time', seconds: 10 },
currentAsset: { id: 'file-123', name: 'video.mp4', type: 'file' },
},
} as unknown as ChatMessage
useChatbotStore.setState({
messages: [mockMessage],
})
const { getByText } = render(
<QueryClientProvider client={queryClient}>
<ChatbotSidebar
projectId="proj-1"
contextAssetId="file-123"
selectedMessageId="msg-selected"
/>
</QueryClientProvider>,
)
const cardInner = getByText('Selected message content').closest('.bg-primary')
expect(cardInner?.className).toContain('border-blue-500')
expect(cardInner?.className).not.toContain('ring')
})
it('does not trigger onSelectMessage on click when currentAsset does not match', async () => {
vi.mocked(client.api.projects[':projectId']['chat-agents'].$get).mockResolvedValue({
ok: true,
json: async () => [{ id: 'agent-1', name: 'Agent 1', type: 'chat', enabled: true }],
} as unknown as Awaited<
ReturnType<(typeof client.api.projects)[':projectId']['chat-agents']['$get']>
>)
const mockSelectMessage = vi.fn()
const mockMessage = {
id: 'msg-2',
role: 'custom',
customType: 'shumai_message',
content: 'Other file comment',
details: {
annotation: true,
annotations: [{ type: 'rectangle', x: 5, y: 10 }],
position: { type: 'time', seconds: 20 },
currentAsset: { id: 'other-file-456', name: 'other.mp4', type: 'file' },
},
} as unknown as ChatMessage
useChatbotStore.setState({
messages: [mockMessage],
})
const { getByText } = render(
<QueryClientProvider client={queryClient}>
<ChatbotSidebar
projectId="proj-1"
contextAssetId="file-123"
onSelectMessage={mockSelectMessage}
/>
</QueryClientProvider>,
)
const msgCard = getByText('Other file comment').closest('div')
msgCard?.click()
expect(mockSelectMessage).not.toHaveBeenCalled()
})
it('renders full-width message card with inline badges, attachments and assets sections', async () => {
vi.mocked(client.api.projects[':projectId']['chat-agents'].$get).mockResolvedValue({
ok: true,
json: async () => [{ id: 'agent-1', name: 'Agent 1', type: 'chat', enabled: true }],
} as unknown as Awaited<
ReturnType<(typeof client.api.projects)[':projectId']['chat-agents']['$get']>
>)
const mockMessage = {
id: 'msg-full',
role: 'custom',
customType: 'shumai_message',
content: 'Review this scene carefully',
details: {
annotation: true,
annotations: [
{
type: 'box',
color: '#ff0000',
points: [
[0, 0],
[10, 10],
],
},
],
position: { type: 'time', seconds: 45 },
currentAsset: { id: 'file-123', name: 'clip.mp4', type: 'file' },
attachedFiles: [
{ id: 'att-1', name: 'reference-mood.jpg', type: 'image', mediaType: 'image/jpeg' },
],
referencedAssets: [
{ id: 'asset-ref-1', name: 'Storyboard.pdf', type: 'file' },
{ id: 'asset-ref-2', name: 'Audio Stems', type: 'folder' },
],
},
} as unknown as ChatMessage
useChatbotStore.setState({
messages: [mockMessage],
})
const { getByText } = render(
<QueryClientProvider client={queryClient}>
<ChatbotSidebar projectId="proj-1" contextAssetId="file-123" />
</QueryClientProvider>,
)
// Inline text and timestamp
expect(getByText('Review this scene carefully')).toBeTruthy()
expect(getByText('00:45')).toBeTruthy()
// Attachments section
expect(getByText('Attachments')).toBeTruthy()
expect(getByText('reference-mood.jpg')).toBeTruthy()
// Assets section
expect(getByText('Assets')).toBeTruthy()
expect(getByText('Storyboard.pdf')).toBeTruthy()
expect(getByText('Audio Stems')).toBeTruthy()
// Card should be full width (w-full)
const cardEl = getByText('Review this scene carefully').closest('.bg-primary')
expect(cardEl?.className).toContain('w-full')
})
it('asks for confirmation before deleting a session and deletes when confirmed', async () => {
const mockDeleteSession = vi.fn()
const originalDeleteSession = useChatbotStore.getState().deleteSession
useChatbotStore.setState({
isHistoryMode: true,
historySessions: [
{
id: 'sess-1',
agentId: 'agent-1',
userId: 'user-1',
assetId: 'asset-1',
userCommentId: null,
name: 'Important Chat',
createdAt: new Date().toISOString(),
updatedAt: new Date().toISOString(),
},
],
deleteSession: mockDeleteSession,
})
const { useTeamContextStore } = await import('@/ui/stores/team-context')
useTeamContextStore.setState({ teamId: 'team-123' })
window.confirm = vi.fn()
const confirmSpy = vi.spyOn(window, 'confirm')
// 1. User cancels confirmation
confirmSpy.mockReturnValue(false)
const { getByTitle } = render(
<QueryClientProvider client={queryClient}>
<ChatbotSidebar projectId="proj-1" />
</QueryClientProvider>,
)
const deleteBtn = getByTitle('Delete Session')
deleteBtn.click()
expect(confirmSpy).toHaveBeenCalled()
expect(mockDeleteSession).not.toHaveBeenCalled()
// 2. User confirms deletion
confirmSpy.mockReturnValue(true)
deleteBtn.click()
expect(mockDeleteSession).toHaveBeenCalledWith('team-123', 'sess-1')
confirmSpy.mockRestore()
useChatbotStore.setState({ deleteSession: originalDeleteSession })
})
it('renders thinking_level_change messages properly in the chat log', () => {
const mockThinkingChangeMsg = {
id: 'think-1',
role: 'thinking_level_change',
content: 'Thinking level changed to high',
timestamp: Date.now(),
} as unknown as ChatMessage
useChatbotStore.setState({
messages: [mockThinkingChangeMsg],
})
const { getByText } = render(
<QueryClientProvider client={queryClient}>
<ChatbotSidebar projectId="proj-1" />
</QueryClientProvider>,
)
expect(getByText('Thinking level changed to high')).toBeTruthy()
})
it('disables message input when teamId is not available', async () => {
const { useTeamContextStore } = await import('@/ui/stores/team-context')
useTeamContextStore.setState({ teamId: null, projectTeamMap: {} })
useChatbotStore.setState({ selectedAgentId: 'agent-1' })
const { container } = render(
<QueryClientProvider client={queryClient}>
<ChatbotSidebar projectId="proj-1" />
</QueryClientProvider>,
)
const inputArea = container.querySelector('[contenteditable="false"]')
expect(inputArea).toBeTruthy()
expect(inputArea?.getAttribute('contenteditable')).toBe('false')
})
})
+382 -210
View File
@@ -11,31 +11,58 @@ import { cn } from '@/ui/lib/utils'
import { m } from '@/ui/paraglide/messages.js'
import { useChatbotStore } from '@/ui/stores/chatbot'
import { useTeamContextStore } from '@/ui/stores/team-context'
import { useUiStore } from '@/ui/stores/ui'
import { useDroppable } from '@dnd-kit/react'
import type { ChatMessage, ShumaiMessageContext } from '@shumai/dtos'
import type { AssetInfo, ChatMessage, ShumaiMessageContext } from '@shumai/dtos'
import { useQuery, useQueryClient } from '@tanstack/react-query'
import {
ArrowLeft,
ArrowUp,
Bot,
Brain,
ChevronDown,
Download,
FileIcon,
Folder,
History,
Loader2,
Plus,
Trash2,
Wrench,
XIcon,
} from 'lucide-react'
import { useCallback, useEffect, useRef } from 'react'
import { useCallback, useEffect, useRef, useState } from 'react'
import Markdown from 'react-markdown'
import { formatTimeAgo } from '../lib/time'
import { ChatInput } from './chat/message-input'
import { DrawAnnotation } from './ui/icons'
import { formatTimecode } from './viewers/video/utils'
import { isImageFileName } from '@/ui/lib/media'
interface ChatbotSidebarProps {
projectId: string
contextAssetId?: string
file?: AssetInfo | null
currentTime?: number
frameRate?: number
startTimecode?: string
formatTimestamp?: (second: number) => string
onTyping?: () => void
onSelectMessage?: (message: ChatMessage) => void
selectedMessageId?: string | null
}
export function ChatbotSidebar({ projectId, contextAssetId }: ChatbotSidebarProps) {
export function ChatbotSidebar({
projectId,
contextAssetId,
file,
currentTime,
frameRate,
startTimecode,
formatTimestamp,
onTyping,
onSelectMessage,
selectedMessageId,
}: ChatbotSidebarProps) {
const {
isHistoryMode,
setIsHistoryMode,
@@ -58,6 +85,8 @@ export function ChatbotSidebar({ projectId, contextAssetId }: ChatbotSidebarProp
} = useChatbotStore()
const { teamId, ensureTeamIdForProject } = useTeamContextStore()
const { videoTimeDisplayMode } = useUiStore()
const [previewImage, setPreviewImage] = useState<{ url: string; name: string } | null>(null)
useEffect(() => {
if (projectId) {
@@ -187,21 +216,6 @@ export function ChatbotSidebar({ projectId, contextAssetId }: ChatbotSidebarProp
queryClient.invalidateQueries({ queryKey: ['projects'] })
}, [queryClient])
const handleSend = () => {
if (!inputText.trim() || isStreaming || !teamId) return
sendMessage(teamId, inputText, projectId, contextAssetId, handleAssetMutation)
setInputText('')
}
const handleKeyDown = (e: React.KeyboardEvent<HTMLTextAreaElement>) => {
if (e.nativeEvent.isComposing) return
if (e.key === 'Enter' && !e.shiftKey) {
e.preventDefault()
handleSend()
}
}
const getMessageText = (content: unknown): string => {
if (typeof content === 'string') return content
if (Array.isArray(content)) {
@@ -224,29 +238,221 @@ export function ChatbotSidebar({ projectId, contextAssetId }: ChatbotSidebarProp
const renderMessage = (msg: ChatMessage) => {
switch (msg.role as string) {
case 'user': {
case 'user':
case 'custom': {
const msgObj = msg as unknown as Record<string, unknown>
return (
<div key={msg.id} className="flex justify-end w-full">
<div className="bg-primary text-primary-foreground rounded-lg px-3 py-2 max-w-[85%] text-sm whitespace-pre-wrap shadow-xs break-words">
{getMessageText(msgObj.content)}
if (msg.role === 'custom' && msgObj.customType === 'context_display_info') {
const details = msgObj.details as
| { assets?: Array<{ id: string; name: string; type: string }> }
| undefined
const assets = details?.assets || []
if (assets.length === 0) return null
return (
<div
key={msg.id}
className="text-xs text-muted-foreground bg-muted/40 p-2.5 rounded-lg border border-border/50 space-y-1 my-1"
>
<div className="font-semibold">{m.assets_added_to_context()}</div>
<ul className="list-disc list-inside space-y-0.5">
{assets.map((asset) => (
<li key={asset.id} className="truncate">
{asset.name} <span className="text-muted-foreground/70">({asset.type})</span>
</li>
))}
</ul>
</div>
)
}
if (msg.role === 'user' || msgObj.customType === 'shumai_message') {
const details = msgObj.details as ShumaiMessageContext | undefined
const referencedAssets = details?.referencedAssets || []
const attachedFiles = details?.attachedFiles || []
const position = details?.position
const hasAnnotations =
details?.annotation || (details?.annotations && details.annotations.length > 0)
const isCurrentAssetMatch =
!details?.currentAsset?.id ||
(contextAssetId && details.currentAsset.id === contextAssetId) ||
(file?.id && details.currentAsset.id === file.id)
const isSelected = selectedMessageId === msg.id
let timeStr = ''
if (position?.type === 'time') {
if (formatTimestamp) {
timeStr = formatTimestamp(position.seconds)
} else if (frameRate) {
const frameIndex = Math.round(position.seconds * frameRate)
timeStr = formatTimecode(frameIndex, frameRate, videoTimeDisplayMode, startTimecode)
} else {
const mins = Math.floor(position.seconds / 60)
const secs = Math.floor(position.seconds % 60)
timeStr = `${mins.toString().padStart(2, '0')}:${secs.toString().padStart(2, '0')}`
}
} else if (position?.type === 'page') {
timeStr = m.page_prefix({ page: position.page })
}
const messageText = getMessageText(msgObj.content)
const hasAttachmentsOrAssets = attachedFiles.length > 0 || referencedAssets.length > 0
const handleCardClick = () => {
if (isCurrentAssetMatch && onSelectMessage) {
onSelectMessage(msg)
}
}
return (
<div
key={msg.id}
onClick={handleCardClick}
className={cn(
'flex flex-col items-end w-full group transition-all duration-150',
isCurrentAssetMatch && (hasAnnotations || position) && 'cursor-pointer',
)}
>
<div
className={cn(
'bg-primary text-primary-foreground rounded-lg px-3 py-2 text-sm whitespace-pre-wrap shadow-xs break-words transition-all border',
hasAttachmentsOrAssets ? 'w-full' : 'max-w-[85%]',
isSelected ? 'border-blue-500 dark:border-blue-400' : 'border-transparent',
)}
>
<div className="text-sm leading-relaxed whitespace-pre-wrap break-words">
{hasAnnotations && (
<span
className="inline-flex items-center align-middle mr-1.5 p-0.5 rounded bg-primary-foreground/20 text-primary-foreground"
title={m.contains_drawing_annotations()}
>
<DrawAnnotation className="w-3.5 h-3.5" />
</span>
)}
{position && (
<span className="inline-flex items-center align-middle mr-1.5 px-1.5 py-0.5 rounded text-xs font-mono font-bold bg-primary-foreground/20 text-primary-foreground">
{timeStr}
</span>
)}
{messageText && <span>{messageText}</span>}
</div>
{attachedFiles.length > 0 && (
<div className="mt-2.5 w-full">
<div className="text-xs font-medium text-primary-foreground/70 mb-1">
{m.attachments()}
</div>
<div className="flex flex-col gap-1.5 w-full">
{attachedFiles.map((att) => {
const name = att.name || 'file'
const isImage =
isImageFileName(name) ||
att.mediaType === 'image' ||
att.mimeType?.startsWith('image/')
return (
<div
key={att.id}
className={cn(
'group/att relative flex items-center w-full max-w-full rounded-lg border border-primary-foreground/15 bg-primary-foreground/10 hover:bg-primary-foreground/15 transition-colors overflow-hidden',
isImage ? 'h-18 p-1.5 gap-2.5 cursor-pointer' : 'h-9 px-2.5 gap-2.5',
)}
onClick={(e) => {
e.stopPropagation()
if (att.url) {
if (isImage) {
setPreviewImage({ url: att.url, name })
} else {
window.open(att.url, '_blank', 'noreferrer')
}
}
}}
>
{isImage && att.url ? (
<div className="h-full aspect-square rounded-md overflow-hidden bg-black/20 shrink-0">
<img
src={att.url}
alt={name}
className="w-full h-full object-cover group-hover/att:scale-105 transition-transform duration-200"
/>
</div>
) : (
<FileIcon className="w-4 h-4 text-primary-foreground/70 shrink-0" />
)}
<div className="min-w-0 flex-1 overflow-hidden">
<p
className="text-xs font-medium text-primary-foreground truncate block w-full"
title={name}
>
{name}
</p>
</div>
{att.url && (
<a
href={att.url}
target="_blank"
rel="noreferrer"
download={name}
className="p-1 text-primary-foreground/70 hover:text-primary-foreground shrink-0 rounded-md hover:bg-primary-foreground/20 transition-colors opacity-0 group-hover/att:opacity-100"
onClick={(e) => e.stopPropagation()}
title={m.download()}
>
<Download className="w-3.5 h-3.5" />
</a>
)}
</div>
)
})}
</div>
</div>
)}
{referencedAssets.length > 0 && (
<div className="mt-2.5 w-full">
<div className="text-xs font-medium text-primary-foreground/70 mb-1">
{m.assets()}
</div>
<div className="flex flex-col gap-1.5 w-full">
{referencedAssets.map((asset) => (
<div
key={asset.id}
className="flex items-center justify-between w-full h-9 px-2.5 rounded-lg border border-primary-foreground/15 bg-primary-foreground/10 hover:bg-primary-foreground/15 transition-colors overflow-hidden"
>
<div className="flex items-center gap-2.5 min-w-0 flex-1">
{asset.type === 'folder' ? (
<Folder className="w-4 h-4 text-primary-foreground/70 shrink-0" />
) : (
<FileIcon className="w-4 h-4 text-primary-foreground/70 shrink-0" />
)}
<span
className="text-xs font-medium truncate text-primary-foreground"
title={asset.name}
>
{asset.name}
</span>
</div>
<span className="text-[10px] text-primary-foreground/60 uppercase shrink-0 font-mono ml-2">
{asset.type}
</span>
</div>
))}
</div>
</div>
)}
</div>
</div>
)
}
const systemContent = msgObj.content || ''
return (
<div
key={msg.id}
className="text-center text-xs text-muted-foreground italic bg-muted/20 py-1.5 rounded"
>
{getMessageText(systemContent)}
</div>
)
}
case 'assistant': {
const msgObj = msg as unknown as Record<string, unknown>
const content = msgObj.content as unknown[]
if (!content) return null
const renderedBlocks: React.ReactNode[] = []
content.forEach((block: unknown, idx: number) => {
if (!block || typeof block !== 'object') return
const b = block as unknown as Record<string, unknown>
switch (b.type) {
case 'text': {
const text = b.text
@@ -357,9 +563,7 @@ export function ChatbotSidebar({ projectId, contextAssetId }: ChatbotSidebarProp
}
}
})
if (renderedBlocks.length === 0) return null
let shouldShowFooter = false
if (content.length > 0) {
const lastBlock = content[content.length - 1]
@@ -372,7 +576,6 @@ export function ChatbotSidebar({ projectId, contextAssetId }: ChatbotSidebarProp
}
}
}
return (
<div key={msg.id} className="flex flex-col w-full space-y-1">
<div className="space-y-2 w-full">{renderedBlocks}</div>
@@ -384,7 +587,6 @@ export function ChatbotSidebar({ projectId, contextAssetId }: ChatbotSidebarProp
</div>
)
}
case 'toolResult': {
return (
<div
@@ -397,58 +599,8 @@ export function ChatbotSidebar({ projectId, contextAssetId }: ChatbotSidebarProp
</div>
)
}
case 'thinking_level_change':
case 'custom': {
case 'thinking_level_change': {
const msgObj = msg as unknown as Record<string, unknown>
if (msgObj.customType === 'shumai_message') {
const details = msgObj.details as ShumaiMessageContext | undefined
const referencedAssets = details?.referencedAssets || []
return (
<div key={msg.id} className="flex flex-col items-end w-full space-y-1">
{referencedAssets.length > 0 && (
<div className="text-xs text-muted-foreground bg-muted/40 p-2.5 rounded-lg border border-border/50 space-y-1 my-1 w-full">
<div className="font-semibold">{m.assets_added_to_context()}</div>
<ul className="list-disc list-inside space-y-0.5">
{referencedAssets.map((asset) => (
<li key={asset.id} className="truncate">
{asset.name}{' '}
<span className="text-muted-foreground/70">({asset.type})</span>
</li>
))}
</ul>
</div>
)}
<div className="bg-primary text-primary-foreground rounded-lg px-3 py-2 max-w-[85%] text-sm whitespace-pre-wrap shadow-xs break-words">
{getMessageText(msgObj.content)}
</div>
</div>
)
}
if (msgObj.customType === 'context_display_info') {
const details = msgObj.details as
| { assets?: Array<{ id: string; name: string; type: string }> }
| undefined
const assets = details?.assets || []
if (assets.length === 0) return null
return (
<div
key={msg.id}
className="text-xs text-muted-foreground bg-muted/40 p-2.5 rounded-lg border border-border/50 space-y-1 my-1"
>
<div className="font-semibold">{m.assets_added_to_context()}</div>
<ul className="list-disc list-inside space-y-0.5">
{assets.map((asset) => (
<li key={asset.id} className="truncate">
{asset.name} <span className="text-muted-foreground/70">({asset.type})</span>
</li>
))}
</ul>
</div>
)
}
const systemContent = msgObj.content || ''
return (
<div
@@ -459,7 +611,6 @@ export function ChatbotSidebar({ projectId, contextAssetId }: ChatbotSidebarProp
</div>
)
}
default:
return null
}
@@ -477,92 +628,106 @@ export function ChatbotSidebar({ projectId, contextAssetId }: ChatbotSidebarProp
<div className="bg-background/95 text-foreground px-4 py-3 rounded-xl border border-border/80 flex flex-col items-center gap-2 shadow-lg max-w-[80%] animate-in zoom-in-95 duration-150">
<Bot className="h-6 w-6 text-primary animate-bounce" />
<span className="text-sm font-semibold text-foreground">
{m.chatbot_drag_drop_hint() || 'Drop assets here'}
{m.chatbot_drag_drop_hint()}
</span>
</div>
</div>
)}
{/* Header section */}
<div className="flex items-center justify-between p-4 pt-[0.7rem] flex-shrink-0">
{isHistoryMode ? (
<>
<div className="flex items-center gap-2 font-semibold">
<span>{m.history()}</span>
</div>
{/* Header */}
<div className="h-12 border-b border-border flex items-center justify-between px-4 flex-shrink-0 bg-background/50 backdrop-blur-sm">
<div className="flex items-center gap-2">
{isHistoryMode && (
<button
onClick={() => setIsHistoryMode(false)}
className="p-1.5 hover:bg-accent rounded-md text-muted-foreground hover:text-foreground transition-colors"
title="Close History"
className="p-1 rounded-md hover:bg-muted text-muted-foreground hover:text-foreground transition-colors cursor-pointer"
title={m.back_to_chat()}
>
<ArrowLeft className="h-4 w-4" />
</button>
</>
) : (
<>
<div className="flex items-center gap-2 font-semibold">
<span>{m.shumai_agent()}</span>
</div>
<div className="flex items-center gap-1">
)}
<Bot className="h-5 w-5 text-primary" />
<span className="font-semibold text-sm">
{isHistoryMode ? m.history() : m.shumai_agent()}
</span>
</div>
<div className="flex items-center gap-1">
{!isHistoryMode ? (
<>
<button
onClick={startNewSession}
className="p-1.5 hover:bg-accent rounded-md text-muted-foreground hover:text-foreground transition-colors"
className="p-1.5 rounded-md hover:bg-muted text-muted-foreground hover:text-foreground transition-colors cursor-pointer"
title={m.new_chat()}
>
<Plus className="h-4 w-4" />
</button>
<button
onClick={() => setIsHistoryMode(true)}
className="p-1.5 hover:bg-accent rounded-md text-muted-foreground hover:text-foreground transition-colors"
className="p-1.5 rounded-md hover:bg-muted text-muted-foreground hover:text-foreground transition-colors cursor-pointer"
title={m.history()}
>
<History className="h-4 w-4" />
</button>
</div>
</>
)}
</>
) : (
<button
onClick={() => {
startNewSession()
setIsHistoryMode(false)
}}
className="flex items-center gap-1 text-xs bg-primary text-primary-foreground px-2.5 py-1 rounded-md hover:bg-primary/90 transition-colors shadow-xs cursor-pointer"
>
<Plus className="h-3.5 w-3.5" />
<span>{m.new_chat()}</span>
</button>
)}
</div>
</div>
{/* Main body view */}
{/* Body */}
{isHistoryMode ? (
<ScrollArea className="flex-1 p-4 min-h-0">
<ScrollArea className="flex-1 p-3">
{historySessions.length === 0 ? (
<div className="flex flex-col items-center justify-center h-48 text-muted-foreground p-4 text-center">
<History className="h-8 w-8 mb-2 opacity-50 text-muted-foreground" />
<p className="text-sm">{m.no_history_sessions()}</p>
<div className="text-center text-xs text-muted-foreground py-8">
{m.no_history_sessions()}
</div>
) : (
<div className="space-y-2 pb-4">
{historySessions.map((sess) => {
const isActive = sess.id === currentSessionId
<div className="space-y-1">
{historySessions.map((session) => {
const isCurrent = session.id === currentSessionId
return (
<div
key={sess.id}
key={session.id}
onClick={() => {
if (teamId) {
loadSession(teamId, session.id)
setIsHistoryMode(false)
}
}}
className={cn(
'group relative flex items-center justify-between p-3 border rounded-lg transition-all cursor-pointer bg-card',
isActive ? 'border-primary' : 'border-border hover:bg-accent/30',
'group flex items-center justify-between p-2.5 rounded-lg text-xs cursor-pointer transition-all border border-transparent',
isCurrent
? 'bg-accent/80 text-accent-foreground font-medium border-border/40'
: 'hover:bg-muted/60 text-muted-foreground hover:text-foreground',
)}
>
<div
onClick={() => teamId && loadSession(teamId, sess.id)}
className="flex-1 min-w-0 pr-6"
>
<div className="text-sm font-semibold text-foreground truncate">
{sess.name || m.new_chat()}
</div>
<div className="text-xs text-muted-foreground mt-0.5">
{formatTimeAgo(sess.createdAt)}
</div>
<div className="flex flex-col min-w-0 pr-2">
<span className="truncate text-foreground font-medium text-xs">
{session.name || m.untitled_session()}
</span>
<span className="text-[10px] text-muted-foreground">
{formatTimeAgo(session.updatedAt)}
</span>
</div>
<button
onClick={(e) => {
e.stopPropagation()
if (teamId && confirm(m.delete_session_confirm())) {
deleteSession(teamId, sess.id)
if (teamId && window.confirm(m.delete_session_confirm())) {
deleteSession(teamId, session.id)
}
}}
className="absolute right-3 opacity-0 group-hover:opacity-100 focus:opacity-100 transition-opacity text-muted-foreground hover:text-destructive p-1 rounded hover:bg-muted"
title="Delete Session"
className="opacity-0 group-hover:opacity-100 focus:opacity-100 transition-opacity text-muted-foreground hover:text-destructive p-1 rounded hover:bg-muted cursor-pointer"
title={m.delete_session()}
>
<Trash2 className="h-4 w-4" />
</button>
@@ -574,7 +739,6 @@ export function ChatbotSidebar({ projectId, contextAssetId }: ChatbotSidebarProp
</ScrollArea>
) : (
<>
{/* Active Chat view */}
<ScrollArea ref={scrollAreaRef} className="flex-1 p-4 min-h-0 [&>div>div]:!block">
{messages.length === 0 ? (
<div className="flex flex-col items-center justify-center h-48 border-2 border-dashed border-border rounded-lg text-muted-foreground p-4 text-center">
@@ -594,7 +758,7 @@ export function ChatbotSidebar({ projectId, contextAssetId }: ChatbotSidebarProp
).customType === 'shumai_message') && (
<div className="flex items-center gap-1.5 text-xs text-muted-foreground italic">
<Loader2 className="h-3 w-3 animate-spin text-primary" />
<span>Thinking...</span>
<span>{m.thinking_status()}</span>
</div>
)}
<div ref={messagesEndRef} />
@@ -602,62 +766,74 @@ export function ChatbotSidebar({ projectId, contextAssetId }: ChatbotSidebarProp
)}
</ScrollArea>
{/* Input Area (nested inside the border exactly like comment input) */}
<div className="p-2 flex-shrink-0 bg-background">
<div className="relative flex flex-col w-full border border-foreground/20 rounded-2xl shadow-xs transition-all duration-200 focus-within:ring-2 focus-within:ring-primary focus-within:border-transparent bg-background overflow-hidden">
{/* Context assets list at the top inside the border */}
{chatAssets.length > 0 && (
<div className="flex flex-col gap-1.5 min-h-0 max-h-[160px] overflow-y-auto border-b border-border bg-muted/5 p-3 divide-y divide-border/50">
{chatAssets.map((asset) => (
<div
key={asset.id}
className="flex items-center justify-between py-1.5 text-xs group"
>
<div className="flex items-baseline gap-1.5 min-w-0 pr-2">
<span className="truncate font-medium text-foreground">{asset.name}</span>
<span className="text-[9px] text-muted-foreground uppercase flex-shrink-0">
{asset.type}
</span>
</div>
<button
onClick={() => removeAsset(asset.id)}
className="text-muted-foreground hover:text-destructive transition-colors p-0.5"
title={m.remove_from_context()}
>
<Trash2 className="h-3.5 w-3.5" />
</button>
</div>
))}
</div>
)}
{/* Textarea inside the border */}
<textarea
value={inputText}
onChange={(e) => setInputText(e.target.value)}
onKeyDown={handleKeyDown}
placeholder="Type a message..."
rows={2}
className="w-full bg-transparent border-none focus:ring-0 focus:outline-hidden resize-none min-h-[44px] max-h-[120px] px-3 py-2 text-sm focus-visible:outline-hidden focus:ring-transparent focus:border-transparent focus-visible:ring-0"
disabled={isStreaming}
/>
{/* Bottom row inside the border containing the agent selector and send button */}
<div className="flex justify-between items-center px-3 pb-2 pt-1">
{/* Agent selector */}
<div className="flex items-center gap-1.5 min-w-0 max-w-[70%]">
<ChatInput
projectId={projectId}
value={inputText}
onChangeText={setInputText}
placeholder={m.type_a_message()}
disabled={!selectedAgentId || !teamId}
isStreaming={isStreaming}
onAbort={() => {
if (teamId) {
abortActiveSession(teamId)
}
}}
allowMentions={false}
allowAttachments={true}
workspaceAssets={chatAssets}
onRemoveWorkspaceAsset={removeAsset}
allowMarkup={true}
canMarkup={!!file && ['image', 'video', 'pdf'].includes(file.proxyType || '')}
markupDisabledTooltip={
!file ? m.open_file_to_add_markup() : m.markup_not_supported_for_this_file()
}
allowTimestamp={file?.proxyType === 'video' || file?.proxyType === 'pdf'}
currentTime={currentTime}
frameRate={frameRate || file?.media?.metadata?.frameRate || 30}
startTimecode={startTimecode || file?.media?.metadata?.startTimecode}
formatTimestamp={
formatTimestamp ||
(file?.proxyType === 'pdf'
? (sec: number) => m.page_prefix({ page: Math.round(sec) })
: undefined)
}
onTyping={onTyping}
onSendMessage={(
text,
attachmentIds,
annotations,
_replyToId,
second,
attachedFilesMeta,
) => {
if (!teamId) return
sendMessage(
teamId,
text,
projectId,
contextAssetId,
handleAssetMutation,
attachmentIds,
annotations,
second,
attachedFilesMeta,
)
}}
bottomLeftExtra={
<div className="flex items-center gap-1.5 min-w-0 max-w-full">
{chatAgents.length > 0 ? (
<Select
value={selectedAgentId || ''}
onValueChange={(val) => setSelectedAgentId(val)}
disabled={isStreaming}
>
<SelectTrigger className="h-8 border-none bg-transparent hover:bg-muted text-muted-foreground hover:text-foreground focus:ring-0 focus:ring-offset-0 px-2 py-0 gap-1.5 text-xs shrink-0 select-none shadow-none max-w-full">
<SelectTrigger className="h-8 border-none bg-transparent hover:bg-muted text-muted-foreground hover:text-foreground focus:ring-0 focus:ring-offset-0 px-2 py-0 gap-1.5 text-xs shrink-0 select-none shadow-none max-w-full cursor-pointer">
<SelectValue placeholder={m.select_agent()} />
</SelectTrigger>
<SelectContent>
{chatAgents.map((agent) => (
<SelectItem key={agent.id} value={agent.id}>
<SelectItem key={agent.id} value={agent.id} className="cursor-pointer">
<div className="flex items-center gap-2">
{agent.avatar ? (
<img
@@ -680,40 +856,36 @@ export function ChatbotSidebar({ projectId, contextAssetId }: ChatbotSidebarProp
</div>
)}
</div>
{isStreaming ? (
<button
onClick={() => {
if (teamId) {
abortActiveSession(teamId)
}
}}
className="p-2 rounded-full transition-all duration-200 flex items-center justify-center shrink-0 shadow-sm bg-destructive text-destructive-foreground hover:bg-destructive/90 transform hover:-translate-y-0.5"
title="Stop generation"
>
<div className="h-4 w-4 flex items-center justify-center">
<div className="h-3 w-3 bg-current rounded-[2px] bg-white" />
</div>
</button>
) : (
<button
onClick={handleSend}
disabled={!inputText.trim() || !selectedAgentId}
className={cn(
'p-2 rounded-full transition-all duration-200 flex items-center justify-center shrink-0 shadow-sm',
inputText.trim() && selectedAgentId
? 'bg-primary text-primary-foreground hover:bg-primary/95 transform hover:-translate-y-0.5'
: 'bg-muted text-muted-foreground cursor-not-allowed opacity-50',
)}
>
<ArrowUp className="h-4 w-4" />
</button>
)}
</div>
</div>
}
/>
</div>
</>
)}
{previewImage && (
<div
className="fixed inset-0 z-[100] bg-black/95 flex flex-col items-center justify-center p-4 animate-in fade-in duration-200"
onClick={() => setPreviewImage(null)}
>
<button className="absolute top-4 right-4 text-white/70 hover:text-white hover:bg-white/20 rounded-full p-2 transition-colors z-50 cursor-pointer">
<XIcon className="w-8 h-8" />
</button>
<div className="w-full h-full flex items-center justify-center p-2">
<img
src={previewImage.url}
alt={previewImage.name}
className="max-w-full max-h-[85vh] object-contain rounded shadow-2xl cursor-zoom-out"
onClick={(e) => {
e.stopPropagation()
setPreviewImage(null)
}}
/>
</div>
<div className="absolute bottom-6 text-white text-sm font-medium bg-black/50 px-4 py-2 rounded-full backdrop-blur-sm">
{previewImage.name}
</div>
</div>
)}
</div>
)
}
@@ -235,6 +235,7 @@ export function FileCard({
e.preventDefault()
}
}}
data-testid="file-card"
className={cn(
'group relative flex cursor-pointer select-none flex-col overflow-hidden rounded-xl border border-border bg-card transition-all hover:border-primary h-full m-1 outline-none focus:outline-none focus-visible:outline-none',
(isSelected || isChecked) && 'outline-1 outline-primary border-primary',
@@ -189,6 +189,7 @@ export function FileListItem({
e.preventDefault()
}
}}
data-testid="file-card"
className="group cursor-pointer select-none transition-colors whitespace-nowrap flex w-full outline-none focus:outline-none focus-visible:outline-none"
>
<div
@@ -276,6 +276,7 @@ export function FolderCard({
}
: undefined
}
data-testid="file-card"
className={cn(
'group relative w-full md:max-w-[340px] select-none cursor-pointer isolate flex flex-col items-center h-full outline-none focus:outline-none focus-visible:outline-none',
)}
@@ -12,6 +12,7 @@ import {
DialogTitle,
} from '@/ui/components/ui/dialog'
import { Field, FieldLabel, FieldError } from '@/ui/components/ui/field'
import { Popover, PopoverContent, PopoverTrigger } from '@/ui/components/ui/popover'
import {
Select,
SelectContent,
@@ -22,8 +23,8 @@ import {
import { Switch } from '@/ui/components/ui/switch'
import { useQuery, useMutation, useQueryClient } from '@tanstack/react-query'
import { useForm, useStore } from '@tanstack/react-form'
import { Loader2, Puzzle, Server, Terminal } from 'lucide-react'
import { useEffect, useMemo } from 'react'
import { Check, ChevronDown, Loader2, Puzzle, Search, Server, Terminal, Zap } from 'lucide-react'
import { useEffect, useMemo, useState } from 'react'
import { m } from '@/ui/paraglide/messages.js'
import { toast } from 'sonner'
import { AgentInfo, AgentType, ThinkingLevel, thinkingLevelSchema } from '@shumai/dtos'
@@ -70,19 +71,9 @@ const BUILTIN_TOOLS = [
description: () => m.agent_tool_read_skill_desc(),
},
{
id: 'analyze_image',
name: () => m.agent_tool_analyze_image_name(),
description: () => m.agent_tool_analyze_image_desc(),
},
{
id: 'screenshot',
name: () => m.agent_tool_screenshot_name(),
description: () => m.agent_tool_screenshot_desc(),
},
{
id: 'read_pdf_pages',
name: () => m.agent_tool_read_pdf_pages_name(),
description: () => m.agent_tool_read_pdf_pages_desc(),
id: 'read_asset',
name: () => m.agent_tool_read_asset_name(),
description: () => m.agent_tool_read_asset_desc(),
},
{
id: 'list_assets',
@@ -228,6 +219,44 @@ export function AgentFormDialog({
enabled: !!selectedProviderId && isOpen,
})
const [providerSearchQuery, setProviderSearchQuery] = useState('')
const [isProviderSelectOpen, setIsProviderSelectOpen] = useState(false)
// Sort providers alphabetically by name
const sortedProviders = useMemo(() => {
if (!providers) return []
return [...providers].sort((a, b) => a.name.toLowerCase().localeCompare(b.name.toLowerCase()))
}, [providers])
// Filter providers based on search query
const filteredProviders = useMemo(() => {
if (!providerSearchQuery.trim()) return sortedProviders
const q = providerSearchQuery.toLowerCase()
return sortedProviders.filter((p) => p.name.toLowerCase().includes(q))
}, [sortedProviders, providerSearchQuery])
const [modelSearchQuery, setModelSearchQuery] = useState('')
const [isModelSelectOpen, setIsModelSelectOpen] = useState(false)
// Sort models alphabetically by display name / modelId
const sortedModels = useMemo(() => {
if (!models) return []
return [...models].sort((a, b) => {
const nameA = (a.name || a.modelId).toLowerCase()
const nameB = (b.name || b.modelId).toLowerCase()
return nameA.localeCompare(nameB)
})
}, [models])
// Filter models based on search query
const filteredModels = useMemo(() => {
if (!modelSearchQuery.trim()) return sortedModels
const q = modelSearchQuery.toLowerCase()
return sortedModels.filter(
(m) => m.modelId.toLowerCase().includes(q) || (m.name && m.name.toLowerCase().includes(q)),
)
}, [sortedModels, modelSearchQuery])
const createMutation = useMutation({
mutationFn: async (values: AgentFormValues) => {
const res = await client.api.teams[':teamId'].agents.$post({
@@ -269,6 +298,10 @@ export function AgentFormDialog({
useEffect(() => {
if (isOpen) {
form.reset()
setProviderSearchQuery('')
setIsProviderSelectOpen(false)
setModelSearchQuery('')
setIsModelSelectOpen(false)
}
}, [isOpen, initialValues, type])
@@ -431,27 +464,109 @@ export function AgentFormDialog({
children={(field) => {
const isInvalid =
!!field.state.meta.errors.length && field.state.meta.isTouched
const selectedProvider = providers?.find((p) => p.id === field.state.value)
return (
<Field data-invalid={isInvalid}>
<FieldLabel>{m.provider()}</FieldLabel>
<Select
value={field.state.value}
onValueChange={(val) => {
field.handleChange(val)
form.setFieldValue('modelId', '')
<Popover
open={isProviderSelectOpen}
onOpenChange={(open) => {
setIsProviderSelectOpen(open)
if (!open) setProviderSearchQuery('')
}}
modal={true}
>
<SelectTrigger aria-invalid={isInvalid}>
<SelectValue placeholder={m.select_provider()} />
</SelectTrigger>
<SelectContent>
{providers?.map((p) => (
<SelectItem key={p.id} value={p.id}>
{p.name}
</SelectItem>
))}
</SelectContent>
</Select>
<PopoverTrigger asChild>
<Button
type="button"
variant="outline"
role="combobox"
aria-expanded={isProviderSelectOpen}
aria-invalid={isInvalid}
className={cn(
'w-full justify-between font-normal h-10 px-3 bg-background border-border hover:bg-accent/50',
isInvalid && 'border-destructive ring-destructive',
)}
>
<span className="truncate">
{selectedProvider ? (
<span className="font-medium text-foreground truncate">
{selectedProvider.name}
</span>
) : (
<span className="text-muted-foreground">
{m.select_provider()}
</span>
)}
</span>
<ChevronDown className="ml-2 h-4 w-4 shrink-0 opacity-50" />
</Button>
</PopoverTrigger>
<PopoverContent
className="w-[--radix-popover-trigger-width] min-w-[320px] max-h-[360px] p-0 flex flex-col overflow-hidden"
align="start"
side="bottom"
sideOffset={4}
onWheel={(e) => e.stopPropagation()}
>
<div className="relative p-2 pb-1.5 border-b border-border shrink-0">
<Search className="absolute left-4.5 top-1/2 -translate-y-1/2 w-3.5 h-3.5 text-muted-foreground" />
<Input
placeholder={m.search_providers_placeholder()}
value={providerSearchQuery}
onChange={(e) => setProviderSearchQuery(e.target.value)}
className="pl-8 h-8 text-xs bg-muted/40"
autoFocus
/>
</div>
<ScrollArea
className="w-full p-1"
style={{
height:
filteredProviders.length === 0
? '96px'
: `${Math.min(filteredProviders.length * 38 + 8, 256)}px`,
}}
onWheel={(e) => e.stopPropagation()}
>
{filteredProviders.length === 0 ? (
<div className="flex items-center justify-center h-full py-6 text-xs text-muted-foreground">
{m.no_providers_found()}
</div>
) : (
<div className="space-y-0.5 pr-2">
{filteredProviders.map((p) => {
const isSelected = field.state.value === p.id
return (
<div
key={p.id}
onClick={() => {
field.handleChange(p.id)
form.setFieldValue('modelId', '')
setModelSearchQuery('')
setIsProviderSelectOpen(false)
setProviderSearchQuery('')
}}
className={cn(
'flex items-center justify-between px-2.5 py-2 rounded-md text-xs cursor-pointer transition-colors',
isSelected
? 'bg-accent text-accent-foreground font-medium'
: 'hover:bg-muted/60 text-foreground',
)}
>
<span className="truncate flex-1 pr-2">{p.name}</span>
{isSelected && (
<Check className="w-4 h-4 text-primary shrink-0" />
)}
</div>
)
})}
</div>
)}
</ScrollArea>
</PopoverContent>
</Popover>
{isInvalid && (
<FieldError errors={mapErrors(field.state.meta.errors)} />
)}
@@ -465,28 +580,141 @@ export function AgentFormDialog({
children={(field) => {
const isInvalid =
!!field.state.meta.errors.length && field.state.meta.isTouched
const selectedModel = models?.find((m) => m.id === field.state.value)
return (
<Field data-invalid={isInvalid}>
<FieldLabel className="flex items-center gap-2">
{m.model()}
{isModelsLoading && <Loader2 className="w-3 h-3 animate-spin" />}
</FieldLabel>
<Select
value={field.state.value}
onValueChange={field.handleChange}
disabled={!selectedProviderId || isModelsLoading}
<Popover
open={isModelSelectOpen}
onOpenChange={(open) => {
setIsModelSelectOpen(open)
if (!open) setModelSearchQuery('')
}}
modal={true}
>
<SelectTrigger aria-invalid={isInvalid}>
<SelectValue placeholder={m.select_model()} />
</SelectTrigger>
<SelectContent>
{models?.map((m) => (
<SelectItem key={m.id} value={m.id}>
{m.name || m.modelId}
</SelectItem>
))}
</SelectContent>
</Select>
<PopoverTrigger asChild>
<Button
type="button"
variant="outline"
role="combobox"
aria-expanded={isModelSelectOpen}
aria-invalid={isInvalid}
disabled={!selectedProviderId || isModelsLoading}
className={cn(
'w-full justify-between font-normal h-10 px-3 bg-background border-border hover:bg-accent/50',
isInvalid && 'border-destructive ring-destructive',
)}
>
<span className="truncate flex items-center gap-2">
{selectedModel ? (
<>
<span className="truncate font-medium text-foreground">
{selectedModel.name || selectedModel.modelId}
</span>
{selectedModel.name &&
selectedModel.name !== selectedModel.modelId && (
<Badge
variant="secondary"
className="font-mono text-[10px] px-1.5 py-0 shrink-0"
>
{selectedModel.modelId}
</Badge>
)}
</>
) : (
<span className="text-muted-foreground">
{m.select_model()}
</span>
)}
</span>
<ChevronDown className="ml-2 h-4 w-4 shrink-0 opacity-50" />
</Button>
</PopoverTrigger>
<PopoverContent
className="w-[--radix-popover-trigger-width] min-w-[320px] max-h-[360px] p-0 flex flex-col overflow-hidden"
align="start"
side="bottom"
sideOffset={4}
onWheel={(e) => e.stopPropagation()}
>
<div className="relative p-2 pb-1.5 border-b border-border shrink-0">
<Search className="absolute left-4.5 top-1/2 -translate-y-1/2 w-3.5 h-3.5 text-muted-foreground" />
<Input
placeholder={m.search_models_placeholder()}
value={modelSearchQuery}
onChange={(e) => setModelSearchQuery(e.target.value)}
className="pl-8 h-8 text-xs bg-muted/40"
autoFocus
/>
</div>
<ScrollArea
className="w-full p-1"
style={{
height:
filteredModels.length === 0
? '96px'
: `${Math.min(filteredModels.length * 38 + 8, 256)}px`,
}}
onWheel={(e) => e.stopPropagation()}
>
{filteredModels.length === 0 ? (
<div className="flex items-center justify-center h-full py-6 text-xs text-muted-foreground">
{m.no_models_found()}
</div>
) : (
<div className="space-y-0.5 pr-2">
{filteredModels.map((m) => {
const isSelected = field.state.value === m.id
return (
<div
key={m.id}
onClick={() => {
field.handleChange(m.id)
setIsModelSelectOpen(false)
setModelSearchQuery('')
}}
className={cn(
'flex items-center justify-between px-2.5 py-2 rounded-md text-xs cursor-pointer transition-colors',
isSelected
? 'bg-accent text-accent-foreground font-medium'
: 'hover:bg-muted/60 text-foreground',
)}
>
<div className="min-w-0 flex-1 pr-2">
<div className="flex items-center gap-2">
<span className="truncate">
{m.name || m.modelId}
</span>
{m.name && m.name !== m.modelId && (
<span className="font-mono text-[10px] text-muted-foreground truncate">
({m.modelId})
</span>
)}
{m.config?.reasoning && (
<Badge
variant="outline"
className="text-[9px] px-1 py-0 h-4 gap-0.5 bg-primary/10 text-primary border-primary/20 shrink-0"
>
<Zap className="w-2.5 h-2.5" />
</Badge>
)}
</div>
</div>
{isSelected && (
<Check className="w-4 h-4 text-primary shrink-0" />
)}
</div>
)
})}
</div>
)}
</ScrollArea>
</PopoverContent>
</Popover>
{isInvalid && (
<FieldError errors={mapErrors(field.state.meta.errors)} />
)}
@@ -0,0 +1,92 @@
// @vitest-environment happy-dom
import { cleanup, render, screen } from '@testing-library/react'
import React from 'react'
import { afterEach, describe, expect, it, vi } from 'vitest'
import { ModelFormDialog } from './ModelFormDialog'
import type { ModelFormValues } from './ModelFormDialog'
describe('ModelFormDialog', () => {
afterEach(() => {
cleanup()
})
const model1: ModelFormValues = {
modelId: 'gpt-4o',
name: 'GPT-4o Production',
config: {
api: 'openai-responses',
reasoning: true,
input: ['text', 'image'],
contextWindow: 128000,
maxTokens: 4096,
cost: { input: 2.5, output: 10, cacheRead: 1.25, cacheWrite: 2.5 },
},
}
const model2: ModelFormValues = {
modelId: 'claude-3-5-sonnet',
name: 'Claude 3.5 Sonnet',
config: {
api: 'anthropic-messages',
reasoning: false,
input: ['text'],
contextWindow: 200000,
maxTokens: 8192,
cost: { input: 3, output: 15, cacheRead: 0.3, cacheWrite: 3.75 },
},
}
it('populates fields with initialValues when opened for editing', () => {
render(
<ModelFormDialog
isOpen={true}
onClose={vi.fn()}
onSubmit={vi.fn()}
title="Edit Model"
initialValues={model1}
defaultApi="openai-responses"
/>,
)
const modelIdInput = screen.getByLabelText(/model id/i) as HTMLInputElement
const nameInput = screen.getByLabelText(/display name/i) as HTMLInputElement
expect(modelIdInput.value).toBe('gpt-4o')
expect(nameInput.value).toBe('GPT-4o Production')
})
it('updates fields when switching between different models', () => {
const { rerender } = render(
<ModelFormDialog
key="model-1"
isOpen={true}
onClose={vi.fn()}
onSubmit={vi.fn()}
title="Edit Model"
initialValues={model1}
defaultApi="openai-responses"
/>,
)
let modelIdInput = screen.getByLabelText(/model id/i) as HTMLInputElement
expect(modelIdInput.value).toBe('gpt-4o')
rerender(
<ModelFormDialog
key="model-2"
isOpen={true}
onClose={vi.fn()}
onSubmit={vi.fn()}
title="Edit Model"
initialValues={model2}
defaultApi="anthropic-messages"
/>,
)
modelIdInput = screen.getByLabelText(/model id/i) as HTMLInputElement
const nameInput = screen.getByLabelText(/display name/i) as HTMLInputElement
expect(modelIdInput.value).toBe('claude-3-5-sonnet')
expect(nameInput.value).toBe('Claude 3.5 Sonnet')
})
})
@@ -0,0 +1,398 @@
import { ScrollArea } from '@/ui/components/ui/scroll-area'
import { Button } from '@/ui/components/ui/button'
import {
Dialog,
DialogContent,
DialogFooter,
DialogHeader,
DialogTitle,
} from '@/ui/components/ui/dialog'
import { Field, FieldError, FieldLabel } from '@/ui/components/ui/field'
import { Input } from '@/ui/components/ui/input'
import { Switch } from '@/ui/components/ui/switch'
import { createModelRequestSchema } from '@shumai/dtos'
import { useForm } from '@tanstack/react-form'
import { DollarSign, Loader2 } from 'lucide-react'
import { useEffect, useMemo } from 'react'
import { z } from 'zod'
import { m } from '@/ui/paraglide/messages.js'
export type ModelFormValues = z.infer<typeof createModelRequestSchema>
export interface ModelFormDialogProps {
isOpen: boolean
onClose: () => void
onSubmit: (values: ModelFormValues) => void
initialValues?: ModelFormValues | null
title: string
isLoading?: boolean
defaultApi?: string
existingModelIds?: string[]
}
export function ModelFormDialog({
isOpen,
onClose,
onSubmit,
initialValues,
title,
isLoading = false,
defaultApi,
existingModelIds = [],
}: ModelFormDialogProps) {
const schema = useMemo(() => {
return createModelRequestSchema.extend({
modelId: z
.string()
.min(1, m.model_id())
.refine(
(id) => {
if (initialValues?.modelId === id) return true
return !existingModelIds.includes(id)
},
{ message: m.model_id_already_exists() },
),
})
}, [existingModelIds, initialValues])
const getDefaultValues = () =>
(initialValues || {
modelId: '',
name: '',
config: {
api: defaultApi,
reasoning: false,
input: ['text'],
contextWindow: 128000,
maxTokens: 4096,
cost: {
input: 0,
output: 0,
cacheRead: 0,
cacheWrite: 0,
},
},
}) as z.input<typeof schema>
const form = useForm({
defaultValues: getDefaultValues(),
validators: {
onSubmit: schema,
},
onSubmit: async ({ value }) => {
onSubmit(value as ModelFormValues)
},
})
useEffect(() => {
if (isOpen) {
form.reset(getDefaultValues())
}
}, [isOpen, initialValues, defaultApi, form])
const mapErrors = (errors: unknown[]) => {
return errors.map((e) => {
if (typeof e === 'string') return { message: e }
if (e && typeof e === 'object' && 'message' in e) {
return { message: String(e.message) }
}
return { message: String(e || '') }
})
}
return (
<Dialog open={isOpen} onOpenChange={onClose}>
<DialogContent className="sm:max-w-xl h-[85vh] max-h-[700px] flex flex-col p-0 overflow-hidden">
<DialogHeader className="p-6 pb-2 shrink-0">
<DialogTitle className="text-xl">{title}</DialogTitle>
</DialogHeader>
<form
onSubmit={(e) => {
e.preventDefault()
e.stopPropagation()
form.handleSubmit()
}}
className="flex-1 flex flex-col min-h-0 overflow-hidden"
>
<div className="flex-1 min-h-0">
<ScrollArea className="h-full">
<div className="p-6 pt-2 space-y-6 pr-6">
{/* Basic Model Info */}
<div className="space-y-4">
<form.Field
name="modelId"
children={(field) => {
const isInvalid =
!!field.state.meta.errors.length && field.state.meta.isTouched
return (
<Field data-invalid={isInvalid}>
<FieldLabel htmlFor={field.name}>{m.model_id()}</FieldLabel>
<Input
id={field.name}
name={field.name}
value={field.state.value}
onBlur={field.handleBlur}
onChange={(e) => field.handleChange(e.target.value)}
placeholder={m.model_id_placeholder()}
aria-invalid={isInvalid}
/>
{isInvalid && <FieldError errors={mapErrors(field.state.meta.errors)} />}
</Field>
)
}}
/>
<form.Field
name="name"
children={(field) => {
const isInvalid =
!!field.state.meta.errors.length && field.state.meta.isTouched
return (
<Field data-invalid={isInvalid}>
<FieldLabel htmlFor={field.name}>{m.display_name_optional()}</FieldLabel>
<Input
id={field.name}
name={field.name}
value={field.state.value || ''}
onBlur={field.handleBlur}
onChange={(e) => field.handleChange(e.target.value)}
placeholder={m.model_name_placeholder()}
aria-invalid={isInvalid}
/>
{isInvalid && <FieldError errors={mapErrors(field.state.meta.errors)} />}
</Field>
)
}}
/>
</div>
{/* Parameters & Reasoning */}
<div className="space-y-4 pt-2 border-t border-border">
<h4 className="text-sm font-semibold text-foreground">{m.parameters()}</h4>
<form.Field
name="config.reasoning"
children={(field) => (
<Field
orientation="horizontal"
className="rounded-lg border border-border p-3 shadow-sm bg-muted/40 justify-between flex items-center"
>
<FieldLabel htmlFor={field.name} className="cursor-pointer">
{m.reasoning_support()}
</FieldLabel>
<Switch
id={field.name}
checked={field.state.value}
onCheckedChange={field.handleChange}
/>
</Field>
)}
/>
<div className="grid grid-cols-1 sm:grid-cols-2 gap-4">
<form.Field
name="config.contextWindow"
children={(field) => {
const isInvalid =
!!field.state.meta.errors.length && field.state.meta.isTouched
return (
<Field data-invalid={isInvalid}>
<FieldLabel htmlFor={field.name} className="text-xs">
{m.context_window_tokens()}
</FieldLabel>
<Input
id={field.name}
name={field.name}
type="number"
value={field.state.value}
onBlur={field.handleBlur}
onChange={(e) => field.handleChange(parseInt(e.target.value) || 0)}
aria-invalid={isInvalid}
/>
{isInvalid && (
<FieldError errors={mapErrors(field.state.meta.errors)} />
)}
</Field>
)
}}
/>
<form.Field
name="config.maxTokens"
children={(field) => {
const isInvalid =
!!field.state.meta.errors.length && field.state.meta.isTouched
return (
<Field data-invalid={isInvalid}>
<FieldLabel htmlFor={field.name} className="text-xs">
{m.max_output_tokens()}
</FieldLabel>
<Input
id={field.name}
name={field.name}
type="number"
value={field.state.value}
onBlur={field.handleBlur}
onChange={(e) => field.handleChange(parseInt(e.target.value) || 0)}
aria-invalid={isInvalid}
/>
{isInvalid && (
<FieldError errors={mapErrors(field.state.meta.errors)} />
)}
</Field>
)
}}
/>
</div>
</div>
{/* Pricing Section */}
<div className="space-y-4 pt-2 border-t border-border">
<h4 className="text-sm font-semibold text-foreground flex items-center gap-1.5">
<DollarSign className="w-4 h-4 text-primary" />
{m.pricing()}
</h4>
<div className="grid grid-cols-1 sm:grid-cols-2 gap-4">
<form.Field
name="config.cost.input"
children={(field) => {
const isInvalid =
!!field.state.meta.errors.length && field.state.meta.isTouched
return (
<Field data-invalid={isInvalid}>
<FieldLabel htmlFor={field.name} className="text-xs">
{m.input_cost_1m()}
</FieldLabel>
<Input
id={field.name}
name={field.name}
type="number"
step="0.0001"
value={field.state.value}
onBlur={field.handleBlur}
onChange={(e) => field.handleChange(parseFloat(e.target.value) || 0)}
aria-invalid={isInvalid}
/>
{isInvalid && (
<FieldError errors={mapErrors(field.state.meta.errors)} />
)}
</Field>
)
}}
/>
<form.Field
name="config.cost.output"
children={(field) => {
const isInvalid =
!!field.state.meta.errors.length && field.state.meta.isTouched
return (
<Field data-invalid={isInvalid}>
<FieldLabel htmlFor={field.name} className="text-xs">
{m.output_cost_1m()}
</FieldLabel>
<Input
id={field.name}
name={field.name}
type="number"
step="0.0001"
value={field.state.value}
onBlur={field.handleBlur}
onChange={(e) => field.handleChange(parseFloat(e.target.value) || 0)}
aria-invalid={isInvalid}
/>
{isInvalid && (
<FieldError errors={mapErrors(field.state.meta.errors)} />
)}
</Field>
)
}}
/>
<form.Field
name="config.cost.cacheRead"
children={(field) => {
const isInvalid =
!!field.state.meta.errors.length && field.state.meta.isTouched
return (
<Field data-invalid={isInvalid}>
<FieldLabel htmlFor={field.name} className="text-xs">
{m.cache_read_cost_1m()}
</FieldLabel>
<Input
id={field.name}
name={field.name}
type="number"
step="0.0001"
value={field.state.value}
onBlur={field.handleBlur}
onChange={(e) => field.handleChange(parseFloat(e.target.value) || 0)}
aria-invalid={isInvalid}
/>
{isInvalid && (
<FieldError errors={mapErrors(field.state.meta.errors)} />
)}
</Field>
)
}}
/>
<form.Field
name="config.cost.cacheWrite"
children={(field) => {
const isInvalid =
!!field.state.meta.errors.length && field.state.meta.isTouched
return (
<Field data-invalid={isInvalid}>
<FieldLabel htmlFor={field.name} className="text-xs">
{m.cache_write_cost_1m()}
</FieldLabel>
<Input
id={field.name}
name={field.name}
type="number"
step="0.0001"
value={field.state.value}
onBlur={field.handleBlur}
onChange={(e) => field.handleChange(parseFloat(e.target.value) || 0)}
aria-invalid={isInvalid}
/>
{isInvalid && (
<FieldError errors={mapErrors(field.state.meta.errors)} />
)}
</Field>
)
}}
/>
</div>
</div>
</div>
</ScrollArea>
</div>
<DialogFooter className="p-6 pt-4 border-t border-border gap-2 shrink-0">
<Button type="button" variant="outline" onClick={onClose} disabled={isLoading}>
{m.cancel()}
</Button>
<form.Subscribe
selector={(state) => [state.canSubmit, state.isSubmitting]}
children={([canSubmit, isSubmitting]) => (
<Button
type="submit"
className="min-w-[100px]"
disabled={!canSubmit || isSubmitting || isLoading}
>
{isSubmitting || isLoading ? (
<Loader2 className="w-4 h-4 animate-spin" />
) : (
m.save_changes()
)}
</Button>
)}
/>
</DialogFooter>
</form>
</DialogContent>
</Dialog>
)
}
@@ -0,0 +1,124 @@
// @vitest-environment happy-dom
import { cleanup, render, screen, fireEvent, waitFor } from '@testing-library/react'
import React from 'react'
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import { QueryClient, QueryClientProvider } from '@tanstack/react-query'
import { NotificationSettings } from './NotificationSettings'
import type { NotificationSettings as Settings } from '@shumai/dtos'
const mockGetSettings = vi.fn()
const mockPostSettings = vi.fn()
vi.mock('@/ui/api/client', () => ({
client: {
api: {
teams: {
':teamId': {
notifications: {
settings: {
$get: () => mockGetSettings(),
$post: (args: unknown) => mockPostSettings(args),
},
},
},
},
},
},
}))
vi.mock('sonner', () => ({
toast: {
success: vi.fn(),
error: vi.fn(),
},
}))
describe('NotificationSettings', () => {
const initialSettings: Settings = {
comments: true,
replies: true,
mentions: true,
yourUploads: false,
otherUploads: true,
statusUpdates: true,
kanbanTasks: true,
kanbanComments: true,
}
beforeEach(() => {
vi.clearAllMocks()
mockGetSettings.mockResolvedValue({
ok: true,
json: async () => initialSettings,
})
mockPostSettings.mockResolvedValue({
ok: true,
json: async () => ({ ...initialSettings, yourUploads: true }),
})
})
afterEach(() => {
cleanup()
})
function renderComponent() {
const queryClient = new QueryClient({
defaultOptions: {
queries: { retry: false },
},
})
return render(
<QueryClientProvider client={queryClient}>
<NotificationSettings teamId="team-1" />
</QueryClientProvider>,
)
}
it('renders all notification sections and switches', async () => {
const { container } = renderComponent()
await waitFor(() => {
expect(screen.getByText(/comments & replies/i)).toBeDefined()
})
expect(screen.getByText(/assets & statuses/i)).toBeDefined()
expect(screen.getByText(/kanban notifications/i)).toBeDefined()
// Verify switches are rendered
const switches = screen.getAllByRole('switch')
expect(switches.length).toBe(8)
// Verify no hardcoded blue/purple color classes exist in the markup
const html = container.innerHTML
expect(html).not.toMatch(/text-blue-500/)
expect(html).not.toMatch(/bg-blue-50/)
expect(html).not.toMatch(/text-purple-600/)
expect(html).not.toMatch(/bg-purple-50/)
expect(html).not.toMatch(/text-indigo-500/)
expect(html).not.toMatch(/border-slate-/)
})
it('handles toggling a notification setting', async () => {
renderComponent()
await waitFor(() => {
expect(screen.getByText(/comments & replies/i)).toBeDefined()
})
const switches = screen.getAllByRole('switch')
// Click the first switch (comments)
fireEvent.click(switches[0])
await waitFor(() => {
expect(mockPostSettings).toHaveBeenCalledTimes(1)
expect(mockPostSettings).toHaveBeenCalledWith({
param: { teamId: 'team-1' },
json: {
...initialSettings,
comments: false,
},
})
})
})
})
@@ -71,28 +71,24 @@ export function NotificationSettings({ teamId }: { teamId: string }) {
<ScrollArea className="h-full">
<div className="space-y-6 pr-4 pb-8">
{/* Comments Settings */}
<Card className="border border-slate-200 dark:border-slate-800">
<CardHeader className="border-b border-slate-100 dark:border-slate-800 bg-slate-50/50 dark:bg-slate-900/50">
<Card className="overflow-hidden gap-0 py-0">
<CardHeader className="border-b border-border bg-muted/30 py-5">
<div className="flex items-center gap-2">
<MessageSquare className="w-5 h-5 text-blue-500" />
<MessageSquare className="w-5 h-5 text-primary" />
<CardTitle className="text-lg">{m.comments_and_replies()}</CardTitle>
</div>
<CardDescription>{m.comments_notification_description()}</CardDescription>
</CardHeader>
<CardContent className="divide-y divide-slate-100 dark:divide-slate-800 p-0">
<CardContent className="divide-y divide-border p-0">
{/* General Comments */}
<div className="flex items-center justify-between p-6">
<div className="flex gap-4">
<div className="mt-1 flex items-center justify-center w-8 h-8 rounded-lg bg-blue-50 dark:bg-blue-900/20 text-blue-600 dark:text-blue-400">
<div className="mt-1 flex items-center justify-center w-8 h-8 rounded-lg bg-muted text-foreground">
<MessageSquare className="w-4 h-4" />
</div>
<div className="space-y-0.5">
<h4 className="text-sm font-semibold text-slate-900 dark:text-white">
{m.general_comments()}
</h4>
<p className="text-sm text-slate-500 dark:text-slate-400">
{m.when_someone_comments()}
</p>
<h4 className="text-sm font-semibold text-foreground">{m.general_comments()}</h4>
<p className="text-sm text-muted-foreground">{m.when_someone_comments()}</p>
</div>
</div>
<Switch
@@ -105,16 +101,12 @@ export function NotificationSettings({ teamId }: { teamId: string }) {
{/* Comment Replies */}
<div className="flex items-center justify-between p-6">
<div className="flex gap-4">
<div className="mt-1 flex items-center justify-center w-8 h-8 rounded-lg bg-purple-50 dark:bg-purple-900/20 text-purple-600 dark:text-purple-400">
<div className="mt-1 flex items-center justify-center w-8 h-8 rounded-lg bg-muted text-foreground">
<MessageCircle className="w-4 h-4" />
</div>
<div className="space-y-0.5">
<h4 className="text-sm font-semibold text-slate-900 dark:text-white">
{m.comment_replies()}
</h4>
<p className="text-sm text-slate-500 dark:text-slate-400">
{m.when_someone_replies()}
</p>
<h4 className="text-sm font-semibold text-foreground">{m.comment_replies()}</h4>
<p className="text-sm text-muted-foreground">{m.when_someone_replies()}</p>
</div>
</div>
<Switch
@@ -127,16 +119,12 @@ export function NotificationSettings({ teamId }: { teamId: string }) {
{/* Mentions */}
<div className="flex items-center justify-between p-6">
<div className="flex gap-4">
<div className="mt-1 flex items-center justify-center w-8 h-8 rounded-lg bg-pink-50 dark:bg-pink-900/20 text-pink-600 dark:text-pink-400">
<div className="mt-1 flex items-center justify-center w-8 h-8 rounded-lg bg-muted text-foreground">
<AtSign className="w-4 h-4" />
</div>
<div className="space-y-0.5">
<h4 className="text-sm font-semibold text-slate-900 dark:text-white">
{m.at_mentions()}
</h4>
<p className="text-sm text-slate-500 dark:text-slate-400">
{m.when_someone_mentions()}
</p>
<h4 className="text-sm font-semibold text-foreground">{m.at_mentions()}</h4>
<p className="text-sm text-muted-foreground">{m.when_someone_mentions()}</p>
</div>
</div>
<Switch
@@ -149,28 +137,24 @@ export function NotificationSettings({ teamId }: { teamId: string }) {
</Card>
{/* Assets Settings */}
<Card className="border border-slate-200 dark:border-slate-800">
<CardHeader className="border-b border-slate-100 dark:border-slate-800 bg-slate-50/50 dark:bg-slate-900/50">
<Card className="overflow-hidden gap-0 py-0">
<CardHeader className="border-b border-border bg-muted/30 py-5">
<div className="flex items-center gap-2">
<UploadCloud className="w-5 h-5 text-emerald-500" />
<UploadCloud className="w-5 h-5 text-primary" />
<CardTitle className="text-lg">{m.assets_and_statuses()}</CardTitle>
</div>
<CardDescription>{m.assets_notification_description()}</CardDescription>
</CardHeader>
<CardContent className="divide-y divide-slate-100 dark:divide-slate-800 p-0">
<CardContent className="divide-y divide-border p-0">
{/* Your Uploads */}
<div className="flex items-center justify-between p-6">
<div className="flex gap-4">
<div className="mt-1 flex items-center justify-center w-8 h-8 rounded-lg bg-slate-100 dark:bg-slate-800 text-slate-600 dark:text-slate-400">
<div className="mt-1 flex items-center justify-center w-8 h-8 rounded-lg bg-muted text-foreground">
<Bell className="w-4 h-4" />
</div>
<div className="space-y-0.5">
<h4 className="text-sm font-semibold text-slate-900 dark:text-white">
{m.your_uploads()}
</h4>
<p className="text-sm text-slate-500 dark:text-slate-400">
{m.when_you_upload_assets()}
</p>
<h4 className="text-sm font-semibold text-foreground">{m.your_uploads()}</h4>
<p className="text-sm text-muted-foreground">{m.when_you_upload_assets()}</p>
</div>
</div>
<Switch
@@ -183,14 +167,12 @@ export function NotificationSettings({ teamId }: { teamId: string }) {
{/* Other Uploads */}
<div className="flex items-center justify-between p-6">
<div className="flex gap-4">
<div className="mt-1 flex items-center justify-center w-8 h-8 rounded-lg bg-emerald-50 dark:bg-emerald-900/20 text-emerald-600 dark:text-emerald-400">
<div className="mt-1 flex items-center justify-center w-8 h-8 rounded-lg bg-muted text-foreground">
<UploadCloud className="w-4 h-4" />
</div>
<div className="space-y-0.5">
<h4 className="text-sm font-semibold text-slate-900 dark:text-white">
{m.other_uploads()}
</h4>
<p className="text-sm text-slate-500 dark:text-slate-400">
<h4 className="text-sm font-semibold text-foreground">{m.other_uploads()}</h4>
<p className="text-sm text-muted-foreground">
{m.when_other_users_upload_assets()}
</p>
</div>
@@ -205,16 +187,12 @@ export function NotificationSettings({ teamId }: { teamId: string }) {
{/* Status Updates */}
<div className="flex items-center justify-between p-6">
<div className="flex gap-4">
<div className="mt-1 flex items-center justify-center w-8 h-8 rounded-lg bg-amber-50 dark:bg-amber-900/20 text-amber-600 dark:text-amber-400">
<div className="mt-1 flex items-center justify-center w-8 h-8 rounded-lg bg-muted text-foreground">
<Activity className="w-4 h-4" />
</div>
<div className="space-y-0.5">
<h4 className="text-sm font-semibold text-slate-900 dark:text-white">
{m.status_updates()}
</h4>
<p className="text-sm text-slate-500 dark:text-slate-400">
{m.when_someone_changes_status()}
</p>
<h4 className="text-sm font-semibold text-foreground">{m.status_updates()}</h4>
<p className="text-sm text-muted-foreground">{m.when_someone_changes_status()}</p>
</div>
</div>
<Switch
@@ -227,28 +205,24 @@ export function NotificationSettings({ teamId }: { teamId: string }) {
</Card>
{/* Kanban Settings */}
<Card className="border border-slate-200 dark:border-slate-800">
<CardHeader className="border-b border-slate-100 dark:border-slate-800 bg-slate-50/50 dark:bg-slate-900/50">
<Card className="overflow-hidden gap-0 py-0">
<CardHeader className="border-b border-border bg-muted/30 py-5">
<div className="flex items-center gap-2">
<LayoutDashboard className="w-5 h-5 text-indigo-500" />
<LayoutDashboard className="w-5 h-5 text-primary" />
<CardTitle className="text-lg">{m.kanban_notifications()}</CardTitle>
</div>
<CardDescription>{m.kanban_notification_description()}</CardDescription>
</CardHeader>
<CardContent className="divide-y divide-slate-100 dark:divide-slate-800 p-0">
<CardContent className="divide-y divide-border p-0">
{/* Kanban Tasks */}
<div className="flex items-center justify-between p-6">
<div className="flex gap-4">
<div className="mt-1 flex items-center justify-center w-8 h-8 rounded-lg bg-indigo-50 dark:bg-indigo-900/20 text-indigo-600 dark:text-indigo-400">
<div className="mt-1 flex items-center justify-center w-8 h-8 rounded-lg bg-muted text-foreground">
<LayoutDashboard className="w-4 h-4" />
</div>
<div className="space-y-0.5">
<h4 className="text-sm font-semibold text-slate-900 dark:text-white">
{m.kanban_tasks()}
</h4>
<p className="text-sm text-slate-500 dark:text-slate-400">
{m.when_kanban_task_events()}
</p>
<h4 className="text-sm font-semibold text-foreground">{m.kanban_tasks()}</h4>
<p className="text-sm text-muted-foreground">{m.when_kanban_task_events()}</p>
</div>
</div>
<Switch
@@ -261,16 +235,12 @@ export function NotificationSettings({ teamId }: { teamId: string }) {
{/* Kanban Comments */}
<div className="flex items-center justify-between p-6">
<div className="flex gap-4">
<div className="mt-1 flex items-center justify-center w-8 h-8 rounded-lg bg-cyan-50 dark:bg-cyan-900/20 text-cyan-600 dark:text-cyan-400">
<div className="mt-1 flex items-center justify-center w-8 h-8 rounded-lg bg-muted text-foreground">
<MessageSquare className="w-4 h-4" />
</div>
<div className="space-y-0.5">
<h4 className="text-sm font-semibold text-slate-900 dark:text-white">
{m.kanban_comments()}
</h4>
<p className="text-sm text-slate-500 dark:text-slate-400">
{m.when_kanban_task_commented()}
</p>
<h4 className="text-sm font-semibold text-foreground">{m.kanban_comments()}</h4>
<p className="text-sm text-muted-foreground">{m.when_kanban_task_commented()}</p>
</div>
</div>
<Switch
File diff suppressed because it is too large Load Diff
@@ -69,9 +69,7 @@ interface QuotaRuleDialogProps {
}
const STANDARD_TOOLS = [
{ value: 'analyze_image', label: () => m.agent_tool_analyze_image_name() },
{ value: 'screenshot', label: () => m.agent_tool_screenshot_name() },
{ value: 'read_pdf_pages', label: () => m.agent_tool_read_pdf_pages_name() },
{ value: 'read_asset', label: () => m.agent_tool_read_asset_name() },
{ value: 'list_assets', label: () => m.agent_tool_list_assets_name() },
{ value: 'create_folder', label: () => m.agent_tool_create_folder_name() },
{ value: 'create_file', label: () => m.agent_tool_create_file_name() },
@@ -129,7 +127,7 @@ export const QuotaRuleDialog: React.FC<QuotaRuleDialogProps> = ({
const [resource, setResource] = useState<QuotaResourceTypeEnum>('agent_total_tokens')
const [mcpServerId, setMcpServerId] = useState<string>('')
const [bashMatch, setBashMatch] = useState<string>('*')
const [toolName, setToolName] = useState<string>('analyze_image')
const [toolName, setToolName] = useState<string>('read_asset')
const [limit, setLimit] = useState<number>(100000)
const [period, setPeriod] = useState<QuotaPeriodEnum>('1day')
const [enabled, setEnabled] = useState<boolean>(true)
@@ -177,7 +175,7 @@ export const QuotaRuleDialog: React.FC<QuotaRuleDialogProps> = ({
? resData.name
: typeof resData.toolName === 'string'
? resData.toolName
: 'analyze_image',
: 'read_asset',
)
setLimit(rule.limit)
setPeriod(formatQuotaPeriod(rule.period) as QuotaPeriodEnum)
@@ -189,7 +187,7 @@ export const QuotaRuleDialog: React.FC<QuotaRuleDialogProps> = ({
setResource('agent_total_tokens')
setMcpServerId('')
setBashMatch('*')
setToolName('analyze_image')
setToolName('read_asset')
setLimit(RESOURCE_CONFIG.agent_total_tokens.defaultLimit)
setPeriod('1day')
setEnabled(true)
@@ -213,7 +211,7 @@ export const QuotaRuleDialog: React.FC<QuotaRuleDialogProps> = ({
} else if (resource === 'agent_bash_call_count') {
resourceData.match = bashMatch.trim() || '*'
} else if (resource === 'agent_tool_call_count') {
resourceData.name = toolName || 'analyze_image'
resourceData.name = toolName || 'read_asset'
}
const res = await client.api.teams[':teamId'].quotas.$post({
@@ -261,7 +259,7 @@ export const QuotaRuleDialog: React.FC<QuotaRuleDialogProps> = ({
} else if (resource === 'agent_bash_call_count') {
resourceData.match = bashMatch.trim() || '*'
} else if (resource === 'agent_tool_call_count') {
resourceData.name = toolName || 'analyze_image'
resourceData.name = toolName || 'read_asset'
}
const res = await client.api.teams[':teamId'].quotas[':id'].$put({
@@ -0,0 +1,275 @@
// @vitest-environment happy-dom
import { cleanup, render, screen, fireEvent } from '@testing-library/react'
import React from 'react'
import { afterEach, describe, expect, it, vi } from 'vitest'
import { QueryClient, QueryClientProvider } from '@tanstack/react-query'
import { SyncProvidersDialog } from './SyncProvidersDialog'
import type { SyncCheckResponse } from '@shumai/dtos'
describe('SyncProvidersDialog', () => {
afterEach(() => {
cleanup()
})
const mockSyncData: SyncCheckResponse = {
totalNewProviders: 1,
totalNewModels: 3,
providers: [
{
name: 'test-new-provider',
isNewProvider: true,
config: { api: 'openai-responses', apiKey: 'NEW_KEY' },
models: [
{
modelId: 'm-new-1',
name: 'New Model 1',
config: {
api: 'openai-responses',
reasoning: true,
input: ['text'],
contextWindow: 128000,
maxTokens: 4096,
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
},
},
{
modelId: 'm-new-2',
name: 'New Model 2',
config: {
api: 'openai-responses',
reasoning: false,
input: ['text'],
contextWindow: 64000,
maxTokens: 2048,
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
},
},
],
},
{
name: 'test-existing-provider',
isNewProvider: false,
config: { api: 'anthropic-messages', apiKey: 'EXISTING_KEY' },
models: [
{
modelId: 'm-existing-add-1',
name: 'Existing Provider New Model 1',
config: {
api: 'anthropic-messages',
reasoning: false,
input: ['text'],
contextWindow: 200000,
maxTokens: 8192,
cost: { input: 3, output: 15, cacheRead: 0.3, cacheWrite: 3.75 },
},
},
],
},
],
}
function renderDialog(props?: Partial<React.ComponentProps<typeof SyncProvidersDialog>>) {
const queryClient = new QueryClient({
defaultOptions: { queries: { retry: false } },
})
return render(
<QueryClientProvider client={queryClient}>
<SyncProvidersDialog
isOpen={true}
onClose={vi.fn()}
teamId="team-1"
syncData={mockSyncData}
onSuccess={vi.fn()}
{...props}
/>
</QueryClientProvider>,
)
}
it('renders the dialog with description, providers and models pre-selected by default', () => {
renderDialog()
// Title and description
expect(screen.getByRole('heading', { name: /sync providers & models/i })).toBeDefined()
expect(
screen.getAllByText(
/existing providers, custom endpoints, api keys, and model configurations will never be deleted or modified/i,
).length,
).toBeGreaterThan(0)
// Check provider names
expect(screen.getByText('test-new-provider')).toBeDefined()
expect(screen.getByText('test-existing-provider')).toBeDefined()
// Badges
expect(screen.getByText(/^new provider$/i)).toBeDefined()
expect(screen.getByText(/^existing provider$/i)).toBeDefined()
// Check that both new and existing providers have checkboxes
const newProviderCheckbox = document.getElementById('provider-check-test-new-provider')
expect(newProviderCheckbox).toBeDefined()
expect(newProviderCheckbox?.getAttribute('data-state')).toBe('checked')
const existingProviderCheckbox = document.getElementById(
'provider-check-test-existing-provider',
)
expect(existingProviderCheckbox).toBeDefined()
expect(existingProviderCheckbox?.getAttribute('data-state')).toBe('checked')
// Model checkboxes are checked by default
const mNew1Check = document.getElementById('model-check-test-new-provider-m-new-1')
expect(mNew1Check).toBeDefined()
expect(mNew1Check?.getAttribute('data-state')).toBe('checked')
})
it('toggles all models for an existing provider when its checkbox is clicked', () => {
renderDialog()
const existingProviderCheckbox = document.getElementById(
'provider-check-test-existing-provider',
)!
const existingModelCheck = document.getElementById(
'model-check-test-existing-provider-m-existing-add-1',
)!
expect(existingModelCheck.getAttribute('data-state')).toBe('checked')
// Click to uncheck all
fireEvent.click(existingProviderCheckbox)
expect(existingModelCheck.getAttribute('data-state')).toBe('unchecked')
expect(existingProviderCheckbox.getAttribute('data-state')).toBe('unchecked')
// Click to check all
fireEvent.click(existingProviderCheckbox)
expect(existingModelCheck.getAttribute('data-state')).toBe('checked')
expect(existingProviderCheckbox.getAttribute('data-state')).toBe('checked')
})
it('supports select all and deselect all buttons', () => {
renderDialog()
const deselectAllBtn = screen.getByRole('button', { name: /^deselect all$/i })
fireEvent.click(deselectAllBtn)
const mNew1Check = document.getElementById('model-check-test-new-provider-m-new-1')
expect(mNew1Check?.getAttribute('data-state')).toBe('unchecked')
const selectAllBtn = screen.getByRole('button', { name: /^select all$/i })
fireEvent.click(selectAllBtn)
expect(mNew1Check?.getAttribute('data-state')).toBe('checked')
})
it('unchecks new provider when all its models are unchecked', () => {
renderDialog()
const mNew1Check = document.getElementById('model-check-test-new-provider-m-new-1')!
const mNew2Check = document.getElementById('model-check-test-new-provider-m-new-2')!
const newProviderCheckbox = document.getElementById('provider-check-test-new-provider')!
// Uncheck both models
fireEvent.click(mNew1Check)
expect(newProviderCheckbox.getAttribute('data-state')).toBe('indeterminate')
fireEvent.click(mNew2Check)
expect(newProviderCheckbox.getAttribute('data-state')).toBe('unchecked')
// Re-check one model -> provider should automatically check
fireEvent.click(mNew1Check)
expect(newProviderCheckbox.getAttribute('data-state')).not.toBe('unchecked')
})
it('filters models when user types in search input', () => {
renderDialog()
const searchInput = screen.getByPlaceholderText(/filter new providers or models/i)
fireEvent.change(searchInput, { target: { value: 'm-existing-add-1' } })
// Only existing provider model should be visible
expect(screen.queryByText('m-new-1')).toBeNull()
expect(screen.getByText('Existing Provider New Model 1')).toBeDefined()
})
it('supports expand all and collapse all buttons', () => {
renderDialog()
// Models should be visible initially (all expanded)
expect(document.getElementById('model-check-test-new-provider-m-new-1')).toBeDefined()
// Click Collapse All
const collapseAllBtn = screen.getByRole('button', { name: /^collapse all$/i })
fireEvent.click(collapseAllBtn)
// Child models should be collapsed (not in DOM)
expect(document.getElementById('model-check-test-new-provider-m-new-1')).toBeNull()
// Click Expand All
const expandAllBtn = screen.getByRole('button', { name: /^expand all$/i })
fireEvent.click(expandAllBtn)
// Child models should be visible again
expect(document.getElementById('model-check-test-new-provider-m-new-1')).toBeDefined()
})
it('updates selection and expansion state when syncData prop changes while mounted', () => {
const { rerender } = renderDialog()
// Initially, test-new-provider is present
expect(screen.getByText('test-new-provider')).toBeDefined()
// Deselect all models to change internal state
const deselectAllBtn = screen.getByRole('button', { name: /^deselect all$/i })
fireEvent.click(deselectAllBtn)
const mNew1Check = document.getElementById('model-check-test-new-provider-m-new-1')
expect(mNew1Check?.getAttribute('data-state')).toBe('unchecked')
// Now supply new syncData prop with different providers and models
const updatedSyncData: SyncCheckResponse = {
totalNewProviders: 1,
totalNewModels: 1,
providers: [
{
name: 'updated-provider',
isNewProvider: true,
config: { api: 'openai-responses' },
models: [
{
modelId: 'm-updated-1',
name: 'Updated Model 1',
config: {
api: 'openai-responses',
reasoning: false,
input: ['text'],
contextWindow: 32000,
maxTokens: 1024,
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
},
},
],
},
],
}
const queryClient = new QueryClient({
defaultOptions: { queries: { retry: false } },
})
rerender(
<QueryClientProvider client={queryClient}>
<SyncProvidersDialog
isOpen={true}
onClose={vi.fn()}
teamId="team-1"
syncData={updatedSyncData}
onSuccess={vi.fn()}
/>
</QueryClientProvider>,
)
// New provider should be rendered, expanded by default, and model should be pre-selected
expect(screen.getByText('updated-provider')).toBeDefined()
const updatedModelCheck = document.getElementById('model-check-updated-provider-m-updated-1')
expect(updatedModelCheck).toBeDefined()
expect(updatedModelCheck?.getAttribute('data-state')).toBe('checked')
})
})
@@ -0,0 +1,474 @@
import { useState, useMemo, useEffect } from 'react'
import {
Dialog,
DialogContent,
DialogDescription,
DialogFooter,
DialogHeader,
DialogTitle,
} from '@/ui/components/ui/dialog'
import { Button } from '@/ui/components/ui/button'
import { Checkbox } from '@/ui/components/ui/checkbox'
import { Badge } from '@/ui/components/ui/badge'
import { Input } from '@/ui/components/ui/input'
import { ScrollArea } from '@/ui/components/ui/scroll-area'
import { ChevronDown, ChevronRight, Cpu, Loader2, Search, Sparkles } from 'lucide-react'
import { useMutation } from '@tanstack/react-query'
import { client } from '@/ui/api/client'
import { toast } from 'sonner'
import { m } from '@/ui/paraglide/messages.js'
import type { SyncCheckResponse, SyncApplyRequest } from '@shumai/dtos'
interface SyncProvidersDialogProps {
isOpen: boolean
onClose: () => void
teamId: string
syncData: SyncCheckResponse | null
onSuccess: () => void
}
export function SyncProvidersDialog({
isOpen,
onClose,
teamId,
syncData,
onSuccess,
}: SyncProvidersDialogProps) {
const [filterQuery, setFilterQuery] = useState('')
const [expandedProviders, setExpandedProviders] = useState<Set<string>>(() => {
return new Set(syncData?.providers.map((p) => p.name) || [])
})
// Pre-selected by default (opt-out)
const [selectedModels, setSelectedModels] = useState<Set<string>>(() => {
const set = new Set<string>()
syncData?.providers.forEach((p) => {
p.models.forEach((mItem) => {
set.add(`${p.name}:${mItem.modelId}`)
})
})
return set
})
// Sync state if syncData changes
useEffect(() => {
if (syncData) {
setExpandedProviders(new Set(syncData.providers.map((p) => p.name)))
const modelSet = new Set<string>()
syncData.providers.forEach((p) => {
p.models.forEach((mItem) => {
modelSet.add(`${p.name}:${mItem.modelId}`)
})
})
setSelectedModels(modelSet)
}
}, [syncData])
// Filtered tree data based on filterQuery
const filteredTree = useMemo(() => {
if (!syncData) return []
const q = filterQuery.trim().toLowerCase()
if (!q) return syncData.providers
return syncData.providers
.map((p) => {
const matchesProvider = p.name.toLowerCase().includes(q)
const matchingModels = p.models.filter(
(mItem) =>
matchesProvider ||
mItem.modelId.toLowerCase().includes(q) ||
mItem.name.toLowerCase().includes(q),
)
if (matchingModels.length > 0) {
return { ...p, models: matchingModels }
}
return null
})
.filter((p): p is NonNullable<typeof p> => p !== null)
}, [syncData, filterQuery])
// Total selected models & providers
const totalSelectedModelsCount = selectedModels.size
const totalSelectedProvidersCount = useMemo(() => {
if (!syncData) return 0
let count = 0
for (const p of syncData.providers) {
const hasSelectedModel = p.models.some((mItem) =>
selectedModels.has(`${p.name}:${mItem.modelId}`),
)
if (hasSelectedModel) count++
}
return count
}, [syncData, selectedModels])
const toggleExpand = (providerName: string) => {
setExpandedProviders((prev) => {
const next = new Set(prev)
if (next.has(providerName)) {
next.delete(providerName)
} else {
next.add(providerName)
}
return next
})
}
const handleExpandAll = () => {
if (!syncData) return
setExpandedProviders(new Set(syncData.providers.map((p) => p.name)))
}
const handleCollapseAll = () => {
setExpandedProviders(new Set())
}
const toggleModel = (providerName: string, modelId: string) => {
const key = `${providerName}:${modelId}`
setSelectedModels((prevModels) => {
const nextModels = new Set(prevModels)
if (nextModels.has(key)) {
nextModels.delete(key)
} else {
nextModels.add(key)
}
return nextModels
})
}
const toggleProvider = (providerName: string) => {
const providerData = syncData?.providers.find((p) => p.name === providerName)
if (!providerData || providerData.models.length === 0) return
const allModelsChecked = providerData.models.every((mItem) =>
selectedModels.has(`${providerName}:${mItem.modelId}`),
)
setSelectedModels((prev) => {
const next = new Set(prev)
providerData.models.forEach((mItem) => {
const key = `${providerName}:${mItem.modelId}`
if (allModelsChecked) {
next.delete(key)
} else {
next.add(key)
}
})
return next
})
}
const handleSelectAll = () => {
if (!syncData) return
const modelSet = new Set<string>()
syncData.providers.forEach((p) => {
p.models.forEach((mItem) => {
modelSet.add(`${p.name}:${mItem.modelId}`)
})
})
setSelectedModels(modelSet)
}
const handleDeselectAll = () => {
setSelectedModels(new Set())
}
// Apply mutation
const applyMutation = useMutation({
mutationFn: async (payload: SyncApplyRequest) => {
const res = await client.api.teams[':teamId'].providers['sync-apply'].$post({
param: { teamId },
json: payload,
})
if (!res.ok) {
const err = await res.json().catch(() => ({ message: 'Failed to apply sync' }))
throw new Error((err as { message?: string }).message || 'Failed to apply sync')
}
return await res.json()
},
onSuccess: (data) => {
toast.success(
m.sync_providers_success({
modelCount: data.addedModels,
providerCount: data.addedProviders,
}),
)
onSuccess()
onClose()
},
onError: (error: Error) => {
toast.error(error.message)
},
})
const handleConfirm = async () => {
if (!syncData || totalSelectedModelsCount === 0) return
const providersToSync: SyncApplyRequest['providers'] = []
for (const p of syncData.providers) {
const selectedModelsForProvider = p.models.filter((mItem) =>
selectedModels.has(`${p.name}:${mItem.modelId}`),
)
if (selectedModelsForProvider.length > 0) {
providersToSync.push({
name: p.name,
isNewProvider: p.isNewProvider,
config: p.config,
models: selectedModelsForProvider,
})
}
}
if (providersToSync.length === 0) return
await applyMutation.mutateAsync({
providers: providersToSync,
})
}
return (
<Dialog open={isOpen} onOpenChange={(open) => !open && onClose()}>
<DialogContent className="sm:max-w-3xl md:max-w-4xl w-full h-[85vh] max-h-[85vh] flex flex-col p-0 overflow-hidden gap-0">
<DialogHeader className="p-6 pb-4 shrink-0">
<DialogTitle className="flex items-center gap-2 text-xl font-semibold">
<Sparkles className="w-5 h-5 text-primary" />
{m.sync_providers_dialog_title()}
</DialogTitle>
<DialogDescription className="text-xs text-muted-foreground leading-relaxed mt-1.5">
{m.sync_providers_dialog_description()}
</DialogDescription>
</DialogHeader>
{/* Controls Toolbar */}
<div className="px-6 pb-3 shrink-0">
<div className="flex items-center justify-between gap-3">
<div className="relative flex-1">
<Search className="absolute left-2.5 top-1/2 -translate-y-1/2 w-3.5 h-3.5 text-muted-foreground" />
<Input
value={filterQuery}
onChange={(e) => setFilterQuery(e.target.value)}
placeholder={m.sync_providers_search_placeholder()}
className="pl-8 h-8 text-xs bg-background"
/>
</div>
<div className="flex items-center gap-1.5">
<Button
type="button"
variant="outline"
size="sm"
onClick={handleExpandAll}
className="h-8 text-xs px-2.5"
>
{m.sync_providers_expand_all()}
</Button>
<Button
type="button"
variant="outline"
size="sm"
onClick={handleCollapseAll}
className="h-8 text-xs px-2.5"
>
{m.sync_providers_collapse_all()}
</Button>
<div className="w-px h-4 bg-border mx-1" />
<Button
type="button"
variant="outline"
size="sm"
onClick={handleSelectAll}
className="h-8 text-xs px-2.5"
>
{m.sync_providers_select_all()}
</Button>
<Button
type="button"
variant="outline"
size="sm"
onClick={handleDeselectAll}
className="h-8 text-xs px-2.5"
>
{m.sync_providers_deselect_all()}
</Button>
</div>
</div>
</div>
{/* Tree List Container */}
<div className="flex-1 min-h-0 px-6 overflow-hidden">
<ScrollArea className="h-full border border-border/60 rounded-md bg-muted/20">
<div className="p-3 space-y-3">
{filteredTree.length === 0 ? (
<div className="py-8 text-center text-sm text-muted-foreground">
{m.no_matches_found()}
</div>
) : (
filteredTree.map((provider) => {
const isExpanded = expandedProviders.has(provider.name)
const isNew = provider.isNewProvider
// Determine provider check state
const checkedCount = provider.models.filter((mItem) =>
selectedModels.has(`${provider.name}:${mItem.modelId}`),
).length
const allChecked = checkedCount === provider.models.length
const someChecked = checkedCount > 0 && !allChecked
return (
<div
key={provider.name}
className="border border-border/50 rounded-lg bg-background overflow-hidden"
>
{/* Provider Header Row */}
<div className="flex items-center justify-between px-3 py-2 bg-card hover:bg-muted/40 transition-colors">
<div className="flex items-center gap-2.5">
<button
type="button"
onClick={() => toggleExpand(provider.name)}
className="p-0.5 text-muted-foreground hover:text-foreground rounded transition-colors"
>
{isExpanded ? (
<ChevronDown className="w-4 h-4" />
) : (
<ChevronRight className="w-4 h-4" />
)}
</button>
<Checkbox
checked={allChecked ? true : someChecked ? 'indeterminate' : false}
onCheckedChange={() => toggleProvider(provider.name)}
id={`provider-check-${provider.name}`}
/>
<label
htmlFor={`provider-check-${provider.name}`}
className="font-medium text-sm text-foreground flex items-center gap-2 cursor-pointer select-none"
>
<Cpu className="w-4 h-4 text-muted-foreground" />
<span>{provider.name}</span>
</label>
</div>
<div className="flex items-center gap-2">
{isNew ? (
<Badge
variant="secondary"
className="text-[10px] px-1.5 py-0 bg-primary/10 text-primary border-primary/20"
>
{m.sync_providers_new_provider()}
</Badge>
) : (
<Badge
variant="outline"
className="text-[10px] px-1.5 py-0 text-muted-foreground"
>
{m.sync_providers_existing_provider()}
</Badge>
)}
<span className="text-xs text-muted-foreground">
{m.sync_providers_new_models_count({
count: provider.models.length,
})}
</span>
</div>
</div>
{/* Model Children Rows */}
{isExpanded && (
<div className="border-t border-border/40 divide-y divide-border/30 bg-muted/10">
{provider.models.map((model) => {
const isModelChecked = selectedModels.has(
`${provider.name}:${model.modelId}`,
)
return (
<div
key={model.modelId}
className="flex items-center justify-between pl-9 pr-3 py-2 hover:bg-muted/30 transition-colors"
>
<div className="flex items-center gap-2.5 min-w-0">
<Checkbox
checked={isModelChecked}
onCheckedChange={() =>
toggleModel(provider.name, model.modelId)
}
id={`model-check-${provider.name}-${model.modelId}`}
/>
<label
htmlFor={`model-check-${provider.name}-${model.modelId}`}
className="text-xs font-medium text-foreground cursor-pointer select-none truncate"
>
{model.name || model.modelId}
<span className="ml-1.5 text-[11px] text-muted-foreground font-mono font-normal">
{model.modelId}
</span>
</label>
</div>
<div className="flex items-center gap-1.5 shrink-0 ml-2">
{model.config.reasoning && (
<Badge
variant="outline"
className="text-[10px] px-1.5 py-0 font-normal border-amber-500/30 text-amber-600 dark:text-amber-400"
>
{m.sync_providers_reasoning()}
</Badge>
)}
<span className="text-[11px] text-muted-foreground font-mono">
{m.sync_providers_context_window({
count: Math.round(model.config.contextWindow / 1000),
})}
</span>
</div>
</div>
)
})}
</div>
)}
</div>
)
})
)}
</div>
</ScrollArea>
</div>
{/* Selected Count & Footer */}
<DialogFooter className="p-6 pt-3 shrink-0 border-t border-border flex items-center justify-between sm:justify-between w-full">
<div className="text-xs text-muted-foreground">
{m.sync_providers_selected_count({
modelCount: totalSelectedModelsCount,
providerCount: totalSelectedProvidersCount,
})}
</div>
<div className="flex items-center gap-2">
<Button
type="button"
variant="outline"
onClick={onClose}
disabled={applyMutation.isPending}
>
{m.cancel()}
</Button>
<Button
type="button"
onClick={handleConfirm}
disabled={totalSelectedModelsCount === 0 || applyMutation.isPending}
className="gap-2"
>
{applyMutation.isPending ? (
<>
<Loader2 className="w-4 h-4 animate-spin" />
{m.sync_providers_adding()}
</>
) : (
m.sync_providers_add_selected({
count: totalSelectedModelsCount,
})
)}
</Button>
</div>
</DialogFooter>
</DialogContent>
</Dialog>
)
}
+54 -7
View File
@@ -311,7 +311,27 @@
"providers_description": "Configure AI providers and their models for this team.",
"configured_ai_providers": "Configured AI Providers",
"add_provider": "Add Provider",
"sync_providers": "Sync Providers",
"sync_providers_checking": "Checking for updates...",
"sync_providers_dialog_title": "Sync Providers & Models",
"sync_providers_dialog_description": "Select the new providers and models you want to add. Existing providers, custom endpoints, API keys, and model configurations will never be deleted or modified.",
"sync_providers_up_to_date": "All providers and models are already up to date.",
"sync_providers_select_all": "Select All",
"sync_providers_deselect_all": "Deselect All",
"sync_providers_expand_all": "Expand All",
"sync_providers_collapse_all": "Collapse All",
"sync_providers_search_placeholder": "Filter new providers or models...",
"sync_providers_new_provider": "New Provider",
"sync_providers_existing_provider": "Existing Provider",
"sync_providers_new_models_count": "{count} new models",
"sync_providers_selected_count": "{modelCount} models selected across {providerCount} providers",
"sync_providers_reasoning": "reasoning",
"sync_providers_context_window": "{count}k ctx",
"sync_providers_add_selected": "Add Selected ({count})",
"sync_providers_adding": "Adding...",
"sync_providers_success": "Successfully added {modelCount} models across {providerCount} providers.",
"search_providers_placeholder": "Search providers...",
"no_providers_found": "No providers found",
"built_in": "Built-in",
"models_count": "Models",
"n_models_count": "{count} Models",
@@ -340,6 +360,20 @@
"api_key_description": "You can provide a literal value (e.g. sk-...) or an Environment variable name (e.g. MY_API_KEY).",
"models_configuration": "Models Configuration",
"add_model": "Add Model",
"edit_model": "Edit Model",
"delete_model": "Delete Model",
"delete_model_confirmation": "This action cannot be undone. This will permanently delete the model \"{name}\".",
"model_created_successfully": "Model created successfully",
"model_updated_successfully": "Model updated successfully",
"model_deleted_successfully": "Model deleted successfully",
"models": "Models",
"search_models_placeholder": "Search models by ID or name...",
"no_models_found": "No models found",
"page_info": "Page {current} of {total}",
"previous": "Previous",
"next": "Next",
"pricing": "Pricing ($ per 1M tokens)",
"parameters": "Parameters",
"model_id": "Model ID",
"display_name_optional": "Display Name (Optional)",
"reasoning_support": "Reasoning Support",
@@ -347,8 +381,16 @@
"max_output_tokens": "Max Output Tokens",
"input_cost_1m": "Input Cost (1M)",
"output_cost_1m": "Output Cost (1M)",
"tokens": "tokens",
"cache_read_cost_1m": "Cache Read Cost (1M)",
"cache_write_cost_1m": "Cache Write Cost (1M)",
"save_configuration": "Save Configuration",
"at_least_one_model_required": "At least one model is required",
"model_id_already_exists": "Model ID already exists for this provider",
"model_id_placeholder": "e.g., gpt-4o, claude-3-5-sonnet-20241022",
"model_name_placeholder": "e.g., GPT-4o",
"cost_per_million": "${input}/1M in · ${output}/1M out",
"max_tokens_label": "max {count}",
"provider": "Provider",
"model": "Model",
"select_provider": "Select Provider",
@@ -477,12 +519,8 @@
"agent_tool_bash_desc": "Allows the agent to run terminal/shell commands in a highly isolated environment.",
"agent_tool_read_skill_name": "Read Skills",
"agent_tool_read_skill_desc": "Allows the agent to load and read skill definitions to perform specialized actions.",
"agent_tool_analyze_image_name": "Analyze Image",
"agent_tool_analyze_image_desc": "Allows the agent to view and analyze the visual contents of an image file.",
"agent_tool_screenshot_name": "Video Frame Extraction",
"agent_tool_screenshot_desc": "Allows the agent to extract still image frames from video files for analysis.",
"agent_tool_read_pdf_pages_name": "Read PDF Pages",
"agent_tool_read_pdf_pages_desc": "Allows the agent to convert and view specific pages of a PDF document as images.",
"agent_tool_read_asset_name": "Read Asset",
"agent_tool_read_asset_desc": "Allows the agent to view and inspect images, video frames, and document pages.",
"agent_tool_list_assets_name": "Browse Shumai",
"agent_tool_list_assets_desc": "Allows the agent to list folders and browse files in Shumai.",
"agent_tool_create_folder_name": "Create Folder",
@@ -662,6 +700,7 @@
"select_source_folder": "Select Source Folder",
"assets": "Assets",
"attachments": "Attachments",
"filter": "Filter",
"project_members": "Project Members",
@@ -916,6 +955,9 @@
"enable_timestamp": "Enable Timestamp",
"attach_files": "Attach files",
"toggle_annotation": "Toggle Annotation",
"markup_not_supported_for_this_file": "Markup is not supported for this file type",
"open_file_to_add_markup": "Open a supported file to add markup",
"type_a_message": "Type a message...",
"loop": "Loop",
"click_toggle_time_frames": "Click to toggle Time/Frames",
@@ -1313,5 +1355,10 @@
"playback_speed": "Playback Speed",
"settings": "Settings",
"failed_to_post_comment": "Failed to post comment",
"entry_id": "Entry ID"
"entry_id": "Entry ID",
"back_to_chat": "Back to Chat",
"delete_session": "Delete Session",
"thinking_status": "Thinking...",
"contains_drawing_annotations": "Contains drawing annotations",
"page_prefix": "Page {page}"
}
+54 -7
View File
@@ -311,7 +311,27 @@
"providers_description": "为此团队配置 AI 提供商及其模型。",
"configured_ai_providers": "已配置的 AI 提供商",
"add_provider": "添加提供商",
"sync_providers": "同步提供商",
"sync_providers_checking": "正在检查更新...",
"sync_providers_dialog_title": "同步提供商与模型",
"sync_providers_dialog_description": "选择您希望添加的新提供商和模型。现有的提供商、自定义端点、API 密钥和模型配置绝不会被删除或修改。",
"sync_providers_up_to_date": "所有提供商和模型均已是最新状态。",
"sync_providers_select_all": "全选",
"sync_providers_deselect_all": "取消全选",
"sync_providers_expand_all": "全部展开",
"sync_providers_collapse_all": "全部折叠",
"sync_providers_search_placeholder": "筛选新提供商或模型...",
"sync_providers_new_provider": "新提供商",
"sync_providers_existing_provider": "现有提供商",
"sync_providers_new_models_count": "{count} 个新模型",
"sync_providers_selected_count": "已选 {modelCount} 个模型,分布在 {providerCount} 个提供商中",
"sync_providers_reasoning": "深度思考",
"sync_providers_context_window": "{count}k 上下文",
"sync_providers_add_selected": "添加所选 ({count})",
"sync_providers_adding": "正在添加...",
"sync_providers_success": "已成功添加 {providerCount} 个提供商中的 {modelCount} 个模型。",
"search_providers_placeholder": "搜索提供商...",
"no_providers_found": "未找到提供商",
"built_in": "内置",
"models_count": "模型",
"n_models_count": "{count} 个模型",
@@ -340,6 +360,20 @@
"api_key_description": "您可以提供一个字面值(例如 sk-...)或环境变量名称(例如 MY_API_KEY)。",
"models_configuration": "模型配置",
"add_model": "添加模型",
"edit_model": "编辑模型",
"delete_model": "删除模型",
"delete_model_confirmation": "此操作无法撤销。这将永久删除模型 \"{name}\"。",
"model_created_successfully": "模型创建成功",
"model_updated_successfully": "模型更新成功",
"model_deleted_successfully": "模型删除成功",
"models": "模型列表",
"search_models_placeholder": "按 ID 或名称搜索模型...",
"no_models_found": "未找到模型",
"page_info": "第 {current} 页,共 {total} 页",
"previous": "上一页",
"next": "下一页",
"pricing": "计费(每百万令牌美元)",
"parameters": "参数",
"model_id": "模型 ID",
"display_name_optional": "显示名称(可选)",
"reasoning_support": "推理支持",
@@ -347,8 +381,16 @@
"max_output_tokens": "最大输出令牌数",
"input_cost_1m": "输入成本(每百万)",
"output_cost_1m": "输出成本(每百万)",
"tokens": "令牌",
"cache_read_cost_1m": "缓存读取成本(每百万)",
"cache_write_cost_1m": "缓存写入成本(每百万)",
"save_configuration": "保存配置",
"at_least_one_model_required": "至少需要一个模型",
"model_id_already_exists": "此提供商已存在该模型 ID",
"model_id_placeholder": "例如:gpt-4o, claude-3-5-sonnet-20241022",
"model_name_placeholder": "例如:GPT-4o",
"cost_per_million": "${input}/1M 输入 · ${output}/1M 输出",
"max_tokens_label": "最大 {count}",
"provider": "提供商",
"model": "模型",
"select_provider": "选择提供商",
@@ -477,12 +519,8 @@
"agent_tool_bash_desc": "允许智能体在高度隔离的环境中运行终端/命令行命令。",
"agent_tool_read_skill_name": "读取技能",
"agent_tool_read_skill_desc": "允许智能体加载和读取技能定义以执行特定操作。",
"agent_tool_analyze_image_name": "分析图片",
"agent_tool_analyze_image_desc": "允许智能体查看和分析图片文件的视觉内容。",
"agent_tool_screenshot_name": "提取视频帧",
"agent_tool_screenshot_desc": "允许智能体从视频文件中提取静态画面进行分析。",
"agent_tool_read_pdf_pages_name": "读取 PDF 页面",
"agent_tool_read_pdf_pages_desc": "允许智能体将 PDF 文件的指定页面转换为图片并查看。",
"agent_tool_read_asset_name": "读取资源",
"agent_tool_read_asset_desc": "允许智能体查看和分析图片、视频画面和文档页面。",
"agent_tool_list_assets_name": "浏览 Shumai",
"agent_tool_list_assets_desc": "允许智能体列出文件夹并浏览 Shumai 中的文件。",
"agent_tool_create_folder_name": "创建文件夹",
@@ -662,6 +700,7 @@
"select_source_folder": "选择来源文件夹",
"assets": "资源",
"attachments": "附件",
"filter": "筛选",
"project_members": "项目成员",
@@ -916,6 +955,9 @@
"enable_timestamp": "启用时间戳",
"attach_files": "附加文件",
"toggle_annotation": "切换标注",
"markup_not_supported_for_this_file": "此文件类型不支持标注",
"open_file_to_add_markup": "打开支持的文件以添加标注",
"type_a_message": "输入消息...",
"loop": "循环",
"click_toggle_time_frames": "点击切换时间/帧",
@@ -1313,5 +1355,10 @@
"playback_speed": "播放速度",
"settings": "设置",
"failed_to_post_comment": "发送评论失败",
"entry_id": "条目 ID"
"entry_id": "条目 ID",
"back_to_chat": "返回聊天",
"delete_session": "删除会话",
"thinking_status": "思考中...",
"contains_drawing_annotations": "包含标注绘图",
"page_prefix": "第 {page} 页"
}
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/webui",
"version": "0.4.1",
"version": "0.4.2",
"private": true,
"type": "module",
"main": "./index.html",
+52 -6
View File
@@ -295,7 +295,27 @@ export * from "./providers.js";
export * from "./providers_description.js";
export * from "./configured_ai_providers.js";
export * from "./add_provider.js";
export * from "./sync_providers.js";
export * from "./sync_providers_checking.js";
export * from "./sync_providers_dialog_title.js";
export * from "./sync_providers_dialog_description.js";
export * from "./sync_providers_up_to_date.js";
export * from "./sync_providers_select_all.js";
export * from "./sync_providers_deselect_all.js";
export * from "./sync_providers_expand_all.js";
export * from "./sync_providers_collapse_all.js";
export * from "./sync_providers_search_placeholder.js";
export * from "./sync_providers_new_provider.js";
export * from "./sync_providers_existing_provider.js";
export * from "./sync_providers_new_models_count.js";
export * from "./sync_providers_selected_count.js";
export * from "./sync_providers_reasoning.js";
export * from "./sync_providers_context_window.js";
export * from "./sync_providers_add_selected.js";
export * from "./sync_providers_adding.js";
export * from "./sync_providers_success.js";
export * from "./search_providers_placeholder.js";
export * from "./no_providers_found.js";
export * from "./built_in.js";
export * from "./models_count.js";
export * from "./n_models_count.js";
@@ -324,6 +344,20 @@ export * from "./api_key_placeholder.js";
export * from "./api_key_description.js";
export * from "./models_configuration.js";
export * from "./add_model.js";
export * from "./edit_model.js";
export * from "./delete_model.js";
export * from "./delete_model_confirmation.js";
export * from "./model_created_successfully.js";
export * from "./model_updated_successfully.js";
export * from "./model_deleted_successfully.js";
export * from "./models.js";
export * from "./search_models_placeholder.js";
export * from "./no_models_found.js";
export * from "./page_info.js";
export * from "./previous.js";
export * from "./next.js";
export * from "./pricing.js";
export * from "./parameters.js";
export * from "./model_id.js";
export * from "./display_name_optional.js";
export * from "./reasoning_support.js";
@@ -331,8 +365,16 @@ export * from "./context_window_tokens.js";
export * from "./max_output_tokens.js";
export * from "./input_cost_1m.js";
export * from "./output_cost_1m.js";
export * from "./tokens.js";
export * from "./cache_read_cost_1m.js";
export * from "./cache_write_cost_1m.js";
export * from "./save_configuration.js";
export * from "./at_least_one_model_required.js";
export * from "./model_id_already_exists.js";
export * from "./model_id_placeholder.js";
export * from "./model_name_placeholder.js";
export * from "./cost_per_million.js";
export * from "./max_tokens_label.js";
export * from "./provider.js";
export * from "./model.js";
export * from "./select_provider.js";
@@ -458,12 +500,8 @@ export * from "./agent_tool_bash_name.js";
export * from "./agent_tool_bash_desc.js";
export * from "./agent_tool_read_skill_name.js";
export * from "./agent_tool_read_skill_desc.js";
export * from "./agent_tool_analyze_image_name.js";
export * from "./agent_tool_analyze_image_desc.js";
export * from "./agent_tool_screenshot_name.js";
export * from "./agent_tool_screenshot_desc.js";
export * from "./agent_tool_read_pdf_pages_name.js";
export * from "./agent_tool_read_pdf_pages_desc.js";
export * from "./agent_tool_read_asset_name.js";
export * from "./agent_tool_read_asset_desc.js";
export * from "./agent_tool_list_assets_name.js";
export * from "./agent_tool_list_assets_desc.js";
export * from "./agent_tool_create_folder_name.js";
@@ -632,6 +670,7 @@ export * from "./count_overflow_assets.js";
export * from "./count_overflow_folders.js";
export * from "./select_source_folder.js";
export * from "./assets.js";
export * from "./attachments.js";
export * from "./filter.js";
export * from "./project_members.js";
export * from "./upload_file.js";
@@ -868,6 +907,9 @@ export * from "./disable_timestamp.js";
export * from "./enable_timestamp.js";
export * from "./attach_files.js";
export * from "./toggle_annotation.js";
export * from "./markup_not_supported_for_this_file.js";
export * from "./open_file_to_add_markup.js";
export * from "./type_a_message.js";
export * from "./loop.js";
export * from "./click_toggle_time_frames.js";
export * from "./quality.js";
@@ -1251,4 +1293,8 @@ export * from "./timecode.js";
export * from "./playback_speed.js";
export * from "./failed_to_post_comment.js";
export * from "./entry_id.js";
export * from "./back_to_chat.js";
export * from "./delete_session.js";
export * from "./thinking_status.js";
export * from "./page_prefix.js";
export type LocalizedString = import("../runtime.js").LocalizedString;
+53 -7
View File
@@ -297,7 +297,27 @@ export * from './providers.js'
export * from './providers_description.js'
export * from './configured_ai_providers.js'
export * from './add_provider.js'
export * from './sync_providers.js'
export * from './sync_providers_checking.js'
export * from './sync_providers_dialog_title.js'
export * from './sync_providers_dialog_description.js'
export * from './sync_providers_up_to_date.js'
export * from './sync_providers_select_all.js'
export * from './sync_providers_deselect_all.js'
export * from './sync_providers_expand_all.js'
export * from './sync_providers_collapse_all.js'
export * from './sync_providers_search_placeholder.js'
export * from './sync_providers_new_provider.js'
export * from './sync_providers_existing_provider.js'
export * from './sync_providers_new_models_count.js'
export * from './sync_providers_selected_count.js'
export * from './sync_providers_reasoning.js'
export * from './sync_providers_context_window.js'
export * from './sync_providers_add_selected.js'
export * from './sync_providers_adding.js'
export * from './sync_providers_success.js'
export * from './search_providers_placeholder.js'
export * from './no_providers_found.js'
export * from './built_in.js'
export * from './models_count.js'
export * from './n_models_count.js'
@@ -326,6 +346,20 @@ export * from './api_key_placeholder.js'
export * from './api_key_description.js'
export * from './models_configuration.js'
export * from './add_model.js'
export * from './edit_model.js'
export * from './delete_model.js'
export * from './delete_model_confirmation.js'
export * from './model_created_successfully.js'
export * from './model_updated_successfully.js'
export * from './model_deleted_successfully.js'
export * from './models.js'
export * from './search_models_placeholder.js'
export * from './no_models_found.js'
export * from './page_info.js'
export * from './previous.js'
export * from './next.js'
export * from './pricing.js'
export * from './parameters.js'
export * from './model_id.js'
export * from './display_name_optional.js'
export * from './reasoning_support.js'
@@ -333,8 +367,16 @@ export * from './context_window_tokens.js'
export * from './max_output_tokens.js'
export * from './input_cost_1m.js'
export * from './output_cost_1m.js'
export * from './tokens.js'
export * from './cache_read_cost_1m.js'
export * from './cache_write_cost_1m.js'
export * from './save_configuration.js'
export * from './at_least_one_model_required.js'
export * from './model_id_already_exists.js'
export * from './model_id_placeholder.js'
export * from './model_name_placeholder.js'
export * from './cost_per_million.js'
export * from './max_tokens_label.js'
export * from './provider.js'
export * from './model.js'
export * from './select_provider.js'
@@ -460,12 +502,8 @@ export * from './agent_tool_bash_name.js'
export * from './agent_tool_bash_desc.js'
export * from './agent_tool_read_skill_name.js'
export * from './agent_tool_read_skill_desc.js'
export * from './agent_tool_analyze_image_name.js'
export * from './agent_tool_analyze_image_desc.js'
export * from './agent_tool_screenshot_name.js'
export * from './agent_tool_screenshot_desc.js'
export * from './agent_tool_read_pdf_pages_name.js'
export * from './agent_tool_read_pdf_pages_desc.js'
export * from './agent_tool_read_asset_name.js'
export * from './agent_tool_read_asset_desc.js'
export * from './agent_tool_list_assets_name.js'
export * from './agent_tool_list_assets_desc.js'
export * from './agent_tool_create_folder_name.js'
@@ -634,6 +672,7 @@ export * from './count_overflow_assets.js'
export * from './count_overflow_folders.js'
export * from './select_source_folder.js'
export * from './assets.js'
export * from './attachments.js'
export * from './filter.js'
export * from './project_members.js'
export * from './upload_file.js'
@@ -870,6 +909,9 @@ export * from './disable_timestamp.js'
export * from './enable_timestamp.js'
export * from './attach_files.js'
export * from './toggle_annotation.js'
export * from './markup_not_supported_for_this_file.js'
export * from './open_file_to_add_markup.js'
export * from './type_a_message.js'
export * from './loop.js'
export * from './click_toggle_time_frames.js'
export * from './quality.js'
@@ -1252,4 +1294,8 @@ export * from './frames.js'
export * from './timecode.js'
export * from './playback_speed.js'
export * from './failed_to_post_comment.js'
export * from './entry_id.js'
export * from './entry_id.js'
export * from './back_to_chat.js'
export * from './delete_session.js'
export * from './thinking_status.js'
export * from './page_prefix.js'
-16
View File
@@ -1,16 +0,0 @@
/**
* | output |
* | --- |
* | "Allows the agent to view and analyze the visual contents of an image file." |
*
* @param {Agent_Tool_Analyze_Image_DescInputs} inputs
* @param {{ locale?: "en" | "zh" }} options
* @returns {LocalizedString}
*/
export const agent_tool_analyze_image_desc: ((inputs?: Agent_Tool_Analyze_Image_DescInputs, options?: {
locale?: "en" | "zh";
}) => LocalizedString) & import("../runtime.js").MessageMetadata<Agent_Tool_Analyze_Image_DescInputs, {
locale?: "en" | "zh";
}, {}>;
export type LocalizedString = import("../runtime.js").LocalizedString;
export type Agent_Tool_Analyze_Image_DescInputs = {};
@@ -1,29 +0,0 @@
/* eslint-disable */
import { getLocale, experimentalStaticLocale } from '../runtime.js';
/** @typedef {import('../runtime.js').LocalizedString} LocalizedString */
/** @typedef {{}} Agent_Tool_Analyze_Image_DescInputs */
const en_agent_tool_analyze_image_desc = /** @type {(inputs: Agent_Tool_Analyze_Image_DescInputs) => LocalizedString} */ () => {
return /** @type {LocalizedString} */ (`Allows the agent to view and analyze the visual contents of an image file.`)
};
const zh_agent_tool_analyze_image_desc = /** @type {(inputs: Agent_Tool_Analyze_Image_DescInputs) => LocalizedString} */ () => {
return /** @type {LocalizedString} */ (`允许智能体查看和分析图片文件的视觉内容。`)
};
/**
* | output |
* | --- |
* | "Allows the agent to view and analyze the visual contents of an image file." |
*
* @param {Agent_Tool_Analyze_Image_DescInputs} inputs
* @param {{ locale?: "en" | "zh" }} options
* @returns {LocalizedString}
*/
export const agent_tool_analyze_image_desc = /** @type {((inputs?: Agent_Tool_Analyze_Image_DescInputs, options?: { locale?: "en" | "zh" }) => LocalizedString) & import('../runtime.js').MessageMetadata<Agent_Tool_Analyze_Image_DescInputs, { locale?: "en" | "zh" }, {}>} */ ((inputs = {}, options = {}) => {
const locale = experimentalStaticLocale ?? options.locale ?? getLocale()
if (locale === "en") return en_agent_tool_analyze_image_desc(inputs)
return zh_agent_tool_analyze_image_desc(inputs)
});
@@ -1,29 +0,0 @@
/* eslint-disable */
import { getLocale, experimentalStaticLocale } from '../runtime.js';
/** @typedef {import('../runtime.js').LocalizedString} LocalizedString */
/** @typedef {{}} Agent_Tool_Analyze_Image_NameInputs */
const en_agent_tool_analyze_image_name = /** @type {(inputs: Agent_Tool_Analyze_Image_NameInputs) => LocalizedString} */ () => {
return /** @type {LocalizedString} */ (`Analyze Image`)
};
const zh_agent_tool_analyze_image_name = /** @type {(inputs: Agent_Tool_Analyze_Image_NameInputs) => LocalizedString} */ () => {
return /** @type {LocalizedString} */ (`分析图片`)
};
/**
* | output |
* | --- |
* | "Analyze Image" |
*
* @param {Agent_Tool_Analyze_Image_NameInputs} inputs
* @param {{ locale?: "en" | "zh" }} options
* @returns {LocalizedString}
*/
export const agent_tool_analyze_image_name = /** @type {((inputs?: Agent_Tool_Analyze_Image_NameInputs, options?: { locale?: "en" | "zh" }) => LocalizedString) & import('../runtime.js').MessageMetadata<Agent_Tool_Analyze_Image_NameInputs, { locale?: "en" | "zh" }, {}>} */ ((inputs = {}, options = {}) => {
const locale = experimentalStaticLocale ?? options.locale ?? getLocale()
if (locale === "en") return en_agent_tool_analyze_image_name(inputs)
return zh_agent_tool_analyze_image_name(inputs)
});
+16
View File
@@ -0,0 +1,16 @@
/**
* | output |
* | --- |
* | "Allows the agent to view and inspect images, video frames, and document pages." |
*
* @param {Agent_Tool_Read_Asset_DescInputs} inputs
* @param {{ locale?: "en" | "zh" }} options
* @returns {LocalizedString}
*/
export const agent_tool_read_asset_desc: ((inputs?: Agent_Tool_Read_Asset_DescInputs, options?: {
locale?: "en" | "zh";
}) => LocalizedString) & import("../runtime.js").MessageMetadata<Agent_Tool_Read_Asset_DescInputs, {
locale?: "en" | "zh";
}, {}>;
export type LocalizedString = import("../runtime.js").LocalizedString;
export type Agent_Tool_Read_Asset_DescInputs = {};
@@ -0,0 +1,29 @@
/* eslint-disable */
import { getLocale, experimentalStaticLocale } from '../runtime.js';
/** @typedef {import('../runtime.js').LocalizedString} LocalizedString */
/** @typedef {{}} Agent_Tool_Read_Asset_DescInputs */
const en_agent_tool_read_asset_desc = /** @type {(inputs: Agent_Tool_Read_Asset_DescInputs) => LocalizedString} */ () => {
return /** @type {LocalizedString} */ (`Allows the agent to view and inspect images, video frames, and document pages.`)
};
const zh_agent_tool_read_asset_desc = /** @type {(inputs: Agent_Tool_Read_Asset_DescInputs) => LocalizedString} */ () => {
return /** @type {LocalizedString} */ (`允许智能体查看和分析图片、视频画面和文档页面。`)
};
/**
* | output |
* | --- |
* | "Allows the agent to view and inspect images, video frames, and document pages." |
*
* @param {Agent_Tool_Read_Asset_DescInputs} inputs
* @param {{ locale?: "en" | "zh" }} options
* @returns {LocalizedString}
*/
export const agent_tool_read_asset_desc = /** @type {((inputs?: Agent_Tool_Read_Asset_DescInputs, options?: { locale?: "en" | "zh" }) => LocalizedString) & import('../runtime.js').MessageMetadata<Agent_Tool_Read_Asset_DescInputs, { locale?: "en" | "zh" }, {}>} */ ((inputs = {}, options = {}) => {
const locale = experimentalStaticLocale ?? options.locale ?? getLocale()
if (locale === "en") return en_agent_tool_read_asset_desc(inputs)
return zh_agent_tool_read_asset_desc(inputs)
});
@@ -1,16 +1,16 @@
/**
* | output |
* | --- |
* | "Video Frame Extraction" |
* | "Read Asset" |
*
* @param {Agent_Tool_Screenshot_NameInputs} inputs
* @param {Agent_Tool_Read_Asset_NameInputs} inputs
* @param {{ locale?: "en" | "zh" }} options
* @returns {LocalizedString}
*/
export const agent_tool_screenshot_name: ((inputs?: Agent_Tool_Screenshot_NameInputs, options?: {
export const agent_tool_read_asset_name: ((inputs?: Agent_Tool_Read_Asset_NameInputs, options?: {
locale?: "en" | "zh";
}) => LocalizedString) & import("../runtime.js").MessageMetadata<Agent_Tool_Screenshot_NameInputs, {
}) => LocalizedString) & import("../runtime.js").MessageMetadata<Agent_Tool_Read_Asset_NameInputs, {
locale?: "en" | "zh";
}, {}>;
export type LocalizedString = import("../runtime.js").LocalizedString;
export type Agent_Tool_Screenshot_NameInputs = {};
export type Agent_Tool_Read_Asset_NameInputs = {};
@@ -0,0 +1,29 @@
/* eslint-disable */
import { getLocale, experimentalStaticLocale } from '../runtime.js';
/** @typedef {import('../runtime.js').LocalizedString} LocalizedString */
/** @typedef {{}} Agent_Tool_Read_Asset_NameInputs */
const en_agent_tool_read_asset_name = /** @type {(inputs: Agent_Tool_Read_Asset_NameInputs) => LocalizedString} */ () => {
return /** @type {LocalizedString} */ (`Read Asset`)
};
const zh_agent_tool_read_asset_name = /** @type {(inputs: Agent_Tool_Read_Asset_NameInputs) => LocalizedString} */ () => {
return /** @type {LocalizedString} */ (`读取资源`)
};
/**
* | output |
* | --- |
* | "Read Asset" |
*
* @param {Agent_Tool_Read_Asset_NameInputs} inputs
* @param {{ locale?: "en" | "zh" }} options
* @returns {LocalizedString}
*/
export const agent_tool_read_asset_name = /** @type {((inputs?: Agent_Tool_Read_Asset_NameInputs, options?: { locale?: "en" | "zh" }) => LocalizedString) & import('../runtime.js').MessageMetadata<Agent_Tool_Read_Asset_NameInputs, { locale?: "en" | "zh" }, {}>} */ ((inputs = {}, options = {}) => {
const locale = experimentalStaticLocale ?? options.locale ?? getLocale()
if (locale === "en") return en_agent_tool_read_asset_name(inputs)
return zh_agent_tool_read_asset_name(inputs)
});

Some files were not shown because too many files have changed in this diff Show More