diff --git a/docs/using-shumai/agent-tools.mdx b/docs/using-shumai/agent-tools.mdx index b50baf4a..c2e2e741 100644 --- a/docs/using-shumai/agent-tools.mdx +++ b/docs/using-shumai/agent-tools.mdx @@ -30,36 +30,36 @@ The following tools can be individually toggled in the agent configuration dialo - **Description:** Allows the agent to load and read skill definitions to perform specialized actions. - **Why it matters:** Enables the agent to inspect the instructions/configurations of custom skills registered in your workspace before executing them. -### 3. Analyze Image (`analyze_image`) +### 3. Read Asset (`read_asset`) -- **Description:** Allows the agent to view and analyze the visual contents of an image file. -- **Why it matters:** Enables multimodal agents to "see" image assets uploaded to Shumai, allowing them to answer comments or write metadata about images. +- **Description:** Allows the agent to inspect images, extract still frames from video files, render PDF document pages, and read text/markdown documents. +- **Why it matters:** Enables multimodal agents to view visual media assets uploaded to Shumai, extract frame-by-frame video screenshots, inspect document layouts, and answer questions about uploaded creative assets. -### 4. Video Frame Extraction (`screenshot`) - -- **Description:** Allows the agent to extract still image frames from video files for analysis. -- **Why it matters:** Enables the agent to perform frame-by-frame analysis on videos (e.g. for comments, annotation, or search). - -### 5. Browse Shumai (`list_assets`) +### 4. Browse Shumai (`list_assets`) - **Description:** Allows the agent to list folders and browse files in Shumai. - **Why it matters:** Gives the agent visual mapping of the project structure so it can search for other files or locate related assets. -### 6. Create Folder (`create_folder`) +### 5. Create Folder (`create_folder`) - **Description:** Allows the agent to create new folders in Shumai. - **Why it matters:** Needed if the agent organizes outputs or places files in custom subdirectory folders. -### 7. Create File (`create_file`) +### 6. Create File (`create_file`) - **Description:** Allows the agent to create and upload new files to Shumai. - **Why it matters:** Allows the agent to generate new assets (e.g. text logs, transcoded clips, reports) and save them directly back into your project. -### 8. Create Version (`create_version`) +### 7. Create Version (`create_version`) - **Description:** Allows the agent to upload and stack a new version of an existing file in Shumai. - **Why it matters:** Allows the agent to update files in a clean, non-destructive way by stacking revisions rather than creating duplicate files. +### 8. Download Asset (`download_asset`) + +- **Description:** Allows the agent to download a workspace asset or specific storage S3 key to the local `.pi` directory for processing with local scripts or bash commands. +- **Why it matters:** Enables agents executing custom skill scripts to fetch files or intermediate transcode artifacts to local disk. + --- ## Restricting Tools diff --git a/docs/using-shumai/quotas-settings.mdx b/docs/using-shumai/quotas-settings.mdx index 4ce19560..1c192dac 100644 --- a/docs/using-shumai/quotas-settings.mdx +++ b/docs/using-shumai/quotas-settings.mdx @@ -53,7 +53,7 @@ Shumai supports quota enforcement across multiple layers of the AI and agent arc | **AI Cost ($)** | Dollar expenditure accrued from AI model providers. | Team, Role, or Members | | **MCP Server Calls** | Number of tool invocations directed to a connected Model Context Protocol (MCP) server. | Target MCP Server ID | | **Sandboxed Bash Commands** | Shell commands executed by the agent inside the isolated sandbox. | Wildcard pattern (e.g. `npm *`, `*`) | -| **Agent Tool Calls** | Calls to specific native tools (e.g. `analyze_image`, `screenshot`, `create_file`). | Tool name | +| **Agent Tool Calls** | Calls to specific native tools (e.g. `read_asset`, `download_asset`, `create_file`). | Tool name | ### Bash Command Wildcard Matching diff --git a/packages/agent/src/activities/agent.ts b/packages/agent/src/activities/agent.ts index d93b341c..6d8a0eba 100644 --- a/packages/agent/src/activities/agent.ts +++ b/packages/agent/src/activities/agent.ts @@ -118,7 +118,7 @@ If you need to create files in the local filesystem (for example, a temporary fi When creating a file or version, you may attach metadata (for example, the AI model or prompt used to generate the asset). The allowed field keys and value types are declared directly in the 'metadata' parameter of the 'create_file' and 'create_version' tools. # Message Context & Tools -User messages may contain a block detailing the user, active asset location, playback position, and attachments. Use the asset IDs from when invoking tools (e.g. 'analyze_image', 'screenshot', 'read_pdf_pages', 'download_asset', 'list_assets'). When an tag is present, it indicates the user has drawn visual markups on the asset at the specified position. When calling 'screenshot', 'analyze_image', or 'read_pdf_pages', pass the 'annotationId' parameter with that ID to view the image with the visual markups overlaid.` +User messages may contain a block detailing the user, active asset location, playback position, and attachments. Use the asset IDs from when invoking tools (e.g. 'read_asset', 'download_asset', 'list_assets'). When an tag is present, it indicates the user has drawn visual markups on the asset at the specified position. When calling 'read_asset', pass the 'annotationId' parameter with that ID to view the image or rendered pages with the visual markups overlaid.` if (agent.soul) { systemPrompt = `${systemPrompt}\n\nAgent Personality and Core Instructions:\n${agent.soul}` diff --git a/packages/agent/src/database-session-storage.test.ts b/packages/agent/src/database-session-storage.test.ts index e3bb27aa..6527f726 100644 --- a/packages/agent/src/database-session-storage.test.ts +++ b/packages/agent/src/database-session-storage.test.ts @@ -7,8 +7,7 @@ import { DatabaseSessionStorage } from './database-session-storage' import { agentService } from '@shumai/core/src/agent/agent' import { createAgentSession, type DbProviderInfo } from './index' import * as sandboxedBashModule from './tools/sandboxed-bash' -import * as analyzeImageModule from './tools/analyze-image' -import * as screenshotModule from './tools/screenshot' +import * as readAssetModule from './tools/read-asset' describe('DatabaseSessionStorage', () => { setupTestDbHooks() @@ -451,9 +450,8 @@ describe('DatabaseSessionStorage', () => { }) const sessionId = initialStorage.sessionId - // 2. Spy on media tools constructors - const createAnalyzeImageToolSpy = vi.spyOn(analyzeImageModule, 'createAnalyzeImageTool') - const createScreenshotToolSpy = vi.spyOn(screenshotModule, 'createScreenshotTool') + // 2. Spy on media tool constructor + const createReadAssetToolSpy = vi.spyOn(readAssetModule, 'createReadAssetTool') // 3. Resume the session via createAgentSession with a NEW comment ID await createAgentSession({ @@ -470,9 +468,8 @@ describe('DatabaseSessionStorage', () => { providers: mockProviders, }) - // 4. Assert that the tools were instantiated using user ID - expect(createAnalyzeImageToolSpy).toHaveBeenCalledWith(user.id) - expect(createScreenshotToolSpy).toHaveBeenCalledWith(user.id) + // 4. Assert that the tool was instantiated using user ID + expect(createReadAssetToolSpy).toHaveBeenCalledWith(user.id) }) it('should instantiate media tools with user ID in subsequent turns (video)', async () => { @@ -505,9 +502,8 @@ describe('DatabaseSessionStorage', () => { }) const sessionId = initialStorage.sessionId - // 2. Spy on media tools constructors - const createAnalyzeImageToolSpy = vi.spyOn(analyzeImageModule, 'createAnalyzeImageTool') - const createScreenshotToolSpy = vi.spyOn(screenshotModule, 'createScreenshotTool') + // 2. Spy on media tool constructor + const createReadAssetToolSpy = vi.spyOn(readAssetModule, 'createReadAssetTool') // 3. Resume the session via createAgentSession with a NEW comment ID await createAgentSession({ @@ -524,9 +520,8 @@ describe('DatabaseSessionStorage', () => { providers: mockProviders, }) - // 4. Assert that the tools were instantiated using user ID - expect(createScreenshotToolSpy).toHaveBeenCalledWith(user.id) - expect(createAnalyzeImageToolSpy).toHaveBeenCalledWith(user.id) + // 4. Assert that the tool was instantiated using user ID + expect(createReadAssetToolSpy).toHaveBeenCalledWith(user.id) }) describe('P1 Bug Reproductions', () => { diff --git a/packages/agent/src/index.ts b/packages/agent/src/index.ts index e999e384..0593ecdc 100644 --- a/packages/agent/src/index.ts +++ b/packages/agent/src/index.ts @@ -19,17 +19,15 @@ import * as os from 'os' import * as path from 'path' import { DatabaseSessionStorage } from './database-session-storage' import { metadataService } from '@shumai/core/src/metadata/metadata' -import { createAnalyzeImageTool } from './tools/analyze-image' import { createCreateFileTool } from './tools/create-file' import { createCreateFolderTool } from './tools/create-folder' import { createCreateVersionTool } from './tools/create-version' import { createDownloadAssetTool } from './tools/download-asset' import { createListAssetsTool } from './tools/list-assets' -import { createReadPdfPagesTool } from './tools/read-pdf-pages' +import { createReadAssetTool } from './tools/read-asset' import { createReadSkillTool } from './tools/read-skill' import { createReadThreadTool } from './tools/read-thread' import { createSandboxedBashTool } from './tools/sandboxed-bash' -import { createScreenshotTool } from './tools/screenshot' import { serializeContextToXml } from './context/serialize-context' import type { ShumaiMessageContext } from '@shumai/dtos' @@ -314,11 +312,7 @@ export async function createAgentSession(params: CreateAgentSessionParams) { const mediaTools: AgentTool[] = [] if (userId) { - mediaTools.push( - createAnalyzeImageTool(userId), - createScreenshotTool(userId), - createReadPdfPagesTool(userId), - ) + mediaTools.push(createReadAssetTool(userId)) } const agent = await prisma.agent.findUnique({ diff --git a/packages/agent/src/tools/analyze-image.test.ts b/packages/agent/src/tools/analyze-image.test.ts deleted file mode 100644 index d9045592..00000000 --- a/packages/agent/src/tools/analyze-image.test.ts +++ /dev/null @@ -1,262 +0,0 @@ -import { describe, it, expect, vi, beforeEach } from 'vitest' -import { createAnalyzeImageTool } from './analyze-image' -import { - prisma, - WorkflowTaskStatus, - type Asset, - type AssetComment, - type AgentSessionEntry, - type User, - type WorkflowTask, -} from '@shumai/db' -import { s3Service } from '@shumai/core/src/s3/s3' -import { workflowService } from '@shumai/workflow-core' -import { authzService } from '@shumai/core/src/authz/authz' - -vi.mock('@shumai/db', async (importOriginal) => { - const actual = await importOriginal() - return { - ...actual, - prisma: { - user: { - findUnique: vi.fn(), - }, - asset: { - findUnique: vi.fn(), - }, - assetComment: { - findUnique: vi.fn(), - }, - agentSessionEntry: { - findUnique: vi.fn(), - }, - workflowTask: { - create: vi.fn(), - }, - }, - } -}) - -vi.mock('@shumai/core/src/s3/s3', () => ({ - s3Service: { - getObject: vi.fn(), - }, -})) - -vi.mock('@shumai/workflow-core', () => ({ - workflowService: { - executeWait: vi.fn(), - }, -})) - -vi.mock('@shumai/core/src/authz/authz', () => ({ - authzService: { - hasPermission: vi.fn(), - }, - Permission: { Read: 'Read' }, - ResourceType: { Asset: 'asset' }, -})) - -describe('analyzeImageTool', () => { - beforeEach(() => { - vi.clearAllMocks() - }) - - it('should check user permissions and fetch the image from S3 when annotationId is omitted', async () => { - vi.mocked(prisma.user.findUnique).mockResolvedValue({ id: 'user-1' } as User) - vi.mocked(authzService.hasPermission).mockResolvedValue() - - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'asset-1', - projectId: 'project-1', - storageKey: { key: 'raw-image.png' }, - media: null, - } as unknown as Asset) - - vi.mocked(s3Service.getObject).mockResolvedValue({ - buffer: Buffer.from('fake-image-bytes'), - contentType: 'image/png', - } as unknown as { buffer: Buffer; contentType: string }) - - const tool = createAnalyzeImageTool('user-1') - const result = await tool.execute('call-1', { assetId: 'asset-1' }) - - expect(authzService.hasPermission).toHaveBeenCalledWith({ - user: { id: 'user-1' }, - permission: 'Read', - type: 'asset', - id: 'asset-1', - }) - - expect(result.content[0].type).toBe('image') - expect((result.content[0] as unknown as { data: string }).data).toBe( - Buffer.from('fake-image-bytes').toString('base64'), - ) - expect(result.details.sourceKeys).toEqual(['raw-image.png']) - - expect(prisma.workflowTask.create).not.toHaveBeenCalled() - }) - - it('should trigger overlay transcode workflow when annotationId matches AssetComment with annotations', async () => { - vi.mocked(prisma.user.findUnique).mockResolvedValue({ id: 'user-1' } as User) - vi.mocked(authzService.hasPermission).mockResolvedValue() - - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'asset-1', - projectId: 'project-1', - storageKey: { key: 'raw-image.png' }, - media: { - imageTranscodes: [{ key: 'transcoded-image.webp' }], - }, - } as unknown as Asset) - - vi.mocked(prisma.assetComment.findUnique).mockResolvedValue({ - id: 'comment-1', - assetId: 'asset-1', - annotation: [ - { - type: 'box', - color: '#ff0000', - points: [ - [0, 0], - [1, 1], - ], - }, - ], - } as unknown as AssetComment) - - vi.mocked(prisma.workflowTask.create).mockResolvedValue({ - id: 'task-1', - } as unknown as WorkflowTask) - - vi.mocked(workflowService.executeWait).mockResolvedValue({ - id: 'task-1', - status: WorkflowTaskStatus.completed, - output: { key: 'annotated-image.webp' }, - } as unknown as WorkflowTask) - - vi.mocked(s3Service.getObject).mockResolvedValue({ - buffer: Buffer.from('fake-annotated-image-bytes'), - contentType: 'image/webp', - } as unknown as { buffer: Buffer; contentType: string }) - - const tool = createAnalyzeImageTool('user-1') - const result = await tool.execute('call-1', { assetId: 'asset-1', annotationId: 'comment-1' }) - - expect(prisma.workflowTask.create).toHaveBeenCalledWith({ - data: { - assetId: 'asset-1', - projectId: 'project-1', - type: 'transcode_image_annotation', - status: 'pending', - payload: { - projectId: 'project-1', - imageAnnotation: { - annotations: [ - { - type: 'box', - color: '#ff0000', - points: [ - [0, 0], - [1, 1], - ], - }, - ], - }, - }, - }, - }) - - expect(result.content[0].type).toBe('image') - expect((result.content[0] as unknown as { data: string }).data).toBe( - Buffer.from('fake-annotated-image-bytes').toString('base64'), - ) - expect(result.details.sourceKeys).toEqual(['annotated-image.webp']) - }) - - it('should trigger overlay transcode workflow when annotationId matches AgentSessionEntry (1-on-1 chat mode)', async () => { - vi.mocked(prisma.user.findUnique).mockResolvedValue({ id: 'user-1' } as User) - vi.mocked(authzService.hasPermission).mockResolvedValue() - - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'asset-1', - projectId: 'project-1', - storageKey: { key: 'raw-image.png' }, - media: null, - } as unknown as Asset) - - vi.mocked(prisma.assetComment.findUnique).mockResolvedValue(null) - vi.mocked(prisma.agentSessionEntry.findUnique).mockResolvedValue({ - id: 'entry-1', - assetId: 'asset-1', - data: { - details: { - annotations: [{ type: 'arrow', x1: 5, y1: 5, x2: 20, y2: 20 }], - }, - }, - } as unknown as AgentSessionEntry) - - vi.mocked(prisma.workflowTask.create).mockResolvedValue({ - id: 'task-1', - } as unknown as WorkflowTask) - - vi.mocked(workflowService.executeWait).mockResolvedValue({ - id: 'task-1', - status: WorkflowTaskStatus.completed, - output: { key: 'chat-annotated-image.webp' }, - } as unknown as WorkflowTask) - - vi.mocked(s3Service.getObject).mockResolvedValue({ - buffer: Buffer.from('fake-chat-annotated-bytes'), - contentType: 'image/webp', - } as unknown as { buffer: Buffer; contentType: string }) - - const tool = createAnalyzeImageTool('user-1') - const result = await tool.execute('call-2', { assetId: 'asset-1', annotationId: 'entry-1' }) - - expect(prisma.workflowTask.create).toHaveBeenCalledWith({ - data: { - assetId: 'asset-1', - projectId: 'project-1', - type: 'transcode_image_annotation', - status: 'pending', - payload: { - projectId: 'project-1', - imageAnnotation: { - annotations: [{ type: 'arrow', x1: 5, y1: 5, x2: 20, y2: 20 }], - }, - }, - }, - }) - - expect(result.details.sourceKeys).toEqual(['chat-annotated-image.webp']) - }) - - it('should throw an error when annotationId belongs to a different asset', async () => { - vi.mocked(prisma.user.findUnique).mockResolvedValue({ id: 'user-1' } as User) - vi.mocked(authzService.hasPermission).mockResolvedValue() - - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'asset-1', - projectId: 'project-1', - storageKey: { key: 'raw-image.png' }, - media: null, - } as unknown as Asset) - - vi.mocked(prisma.assetComment.findUnique).mockResolvedValue({ - id: 'comment-on-other-asset', - assetId: 'asset-999', - annotation: [{ type: 'arrow' }], - } as unknown as AssetComment) - - const tool = createAnalyzeImageTool('user-1') - await expect( - tool.execute('call-mismatch', { - assetId: 'asset-1', - annotationId: 'comment-on-other-asset', - }), - ).rejects.toThrow( - 'Annotation "comment-on-other-asset" belongs to asset "asset-999", not target asset "asset-1".', - ) - }) -}) diff --git a/packages/agent/src/tools/analyze-image.ts b/packages/agent/src/tools/analyze-image.ts deleted file mode 100644 index 472d277d..00000000 --- a/packages/agent/src/tools/analyze-image.ts +++ /dev/null @@ -1,132 +0,0 @@ -import { Type } from 'typebox' -import { type AgentTool } from '@earendil-works/pi-agent-core' -import { type ImageContent } from '@earendil-works/pi-ai' -import { prisma, WorkflowTaskType, WorkflowTaskStatus, type User } from '@shumai/db' -import { s3Service } from '@shumai/core/src/s3/s3' -import { workflowService } from '@shumai/workflow-core' -import { authzService, Permission, ResourceType } from '@shumai/core/src/authz/authz' -import { resolveAnnotationsById } from './annotation-resolver' - -const analyzeImageSchema = Type.Object({ - assetId: Type.String({ - description: 'The asset ID of the image to analyze. This parameter is required.', - }), - annotationId: Type.Optional( - Type.String({ - description: - 'The ID of the comment or message entry from whose visual markup should be overlaid on the image.', - }), - ), -}) - -export function createAnalyzeImageTool(userId: string): AgentTool { - return { - name: 'analyze_image', - label: 'Analyze Image', - description: - 'Retrieves and views the image content. Call this tool if you need to analyze the image.', - parameters: analyzeImageSchema, - execute: async (_toolCallId, params) => { - const assetId = params.assetId - - if (!userId) { - throw new Error('User ID is required for authorization.') - } - - await authzService.hasPermission({ - user: { id: userId } as User, - permission: Permission.Read, - type: ResourceType.Asset, - id: assetId, - }) - - let asset = await prisma.asset.findUnique({ - where: { id: assetId }, - include: { storageKey: true }, - }) - - if (!asset) { - throw new Error(`Asset with ID ${assetId} not found.`) - } - - if (asset.type === 'version_stack') { - const latestVersion = await prisma.asset.findFirst({ - where: { parentId: asset.id, isDeleted: false }, - orderBy: { sortIndex: 'asc' }, - include: { storageKey: true }, - }) - if (latestVersion) { - asset = latestVersion - } - } - - // Get default media key - let mediaKey = asset.storageKey?.key - if (asset.media) { - // eslint-disable-next-line @typescript-eslint/no-explicit-any - const mediaInfo = asset.media as any - if (mediaInfo.imageTranscodes && mediaInfo.imageTranscodes.length > 0) { - mediaKey = mediaInfo.imageTranscodes[0].key || mediaKey - } - } - - if (!mediaKey) { - throw new Error('No media content found for this asset.') - } - - // Resolve annotations dynamically by annotationId - const { annotations } = await resolveAnnotationsById(assetId, params.annotationId) - - let keyToUse = mediaKey - - if (annotations) { - // Trigger transcode workflow to draw annotation - const task = await prisma.workflowTask.create({ - data: { - assetId, - projectId: asset.projectId || 'none', - type: WorkflowTaskType.transcode_image_annotation, - status: WorkflowTaskStatus.pending, - payload: { - projectId: asset.projectId || 'none', - imageAnnotation: { - annotations, - }, - }, - }, - }) - - const completedTask = await workflowService.executeWait(task) - // eslint-disable-next-line @typescript-eslint/no-explicit-any - const output = completedTask.output as any - if (output?.key) { - keyToUse = output.key - } else { - throw new Error('Image annotation transcode workflow failed to output S3 key.') - } - } - - // Retrieve the image content from S3 - const bucket = process.env.S3_BUCKET || 'shumai' - const { buffer, contentType } = await s3Service.getObject(bucket, keyToUse) - - const mimeType = - contentType && contentType !== 'application/octet-stream' ? contentType : 'image/webp' - - const content: ImageContent[] = [ - { - type: 'image', - data: buffer.toString('base64'), - mimeType, - }, - ] - - return { - content, - details: { - sourceKeys: [keyToUse], - }, - } - }, - } -} diff --git a/packages/agent/src/tools/annotation-resolver.ts b/packages/agent/src/tools/annotation-resolver.ts index b4201700..97128e1f 100644 --- a/packages/agent/src/tools/annotation-resolver.ts +++ b/packages/agent/src/tools/annotation-resolver.ts @@ -3,7 +3,7 @@ import type { ShumaiMessageContext } from '@shumai/dtos' export async function resolveAnnotationsById( assetId: string, - annotationId?: string, + annotationId?: string | null, ): Promise<{ annotations: PrismaJson.AnnotationList | null; timestamp: number | null }> { if (!annotationId) { return { annotations: null, timestamp: null } diff --git a/packages/agent/src/tools/download-asset.test.ts b/packages/agent/src/tools/download-asset.test.ts index a811d050..30e9e07a 100644 --- a/packages/agent/src/tools/download-asset.test.ts +++ b/packages/agent/src/tools/download-asset.test.ts @@ -1,6 +1,6 @@ import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest' -import { createDownloadAssetTool } from './download-asset' -import { prisma, type Asset } from '@shumai/db' +import { createDownloadAssetTool, resolveAssetIdFromKey } from './download-asset' +import { prisma, type Asset, type StorageKey } from '@shumai/db' import { s3Service } from '@shumai/core/src/s3/s3' import { authzService } from '@shumai/core/src/authz/authz' import * as fs from 'fs' @@ -15,6 +15,10 @@ vi.mock('@shumai/db', async (importOriginal) => { findUnique: vi.fn(), findFirst: vi.fn(), }, + storageKey: { + findFirst: vi.fn(), + findUnique: vi.fn(), + }, }, } }) @@ -52,163 +56,288 @@ describe('downloadAssetTool', () => { it('should throw authorization error if user ID is empty', async () => { const tool = createDownloadAssetTool('') - await expect(tool.execute('call-1', { assetId: 'asset-1' })).rejects.toThrow( + await expect(tool.execute('call-1', { assetId: 'asset-1', key: null })).rejects.toThrow( 'User ID is required for authorization.', ) }) - it('should throw error if asset not found', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue(null) - + it('should throw error if both assetId and key are null', async () => { const tool = createDownloadAssetTool('user-1') - await expect(tool.execute('call-1', { assetId: 'asset-1' })).rejects.toThrow( - 'Asset with ID asset-1 not found.', + await expect(tool.execute('call-1', { assetId: null, key: null })).rejects.toThrow( + 'Provide exactly one of "assetId" (to download a workspace asset) or "key" (to download a specific S3 storage key). Set the unused parameter to null.', ) }) - it('should download proxy image to .pi directory when imageTranscodes is present', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'asset-img-1', - name: 'sample_photo.jpg', - storageKey: { key: 'raw/sample_photo.jpg' }, - media: { - proxyType: 'image', - imageTranscodes: [{ key: 'proxy/sample_photo.webp' }], - }, - } as unknown as Asset) - - vi.mocked(s3Service.getObject).mockResolvedValue({ - buffer: Buffer.from('fake-proxy-image-bytes'), - contentType: 'image/webp', - } as unknown as { buffer: Buffer; contentType: string }) - + it('should throw error if both assetId and key are provided', async () => { const tool = createDownloadAssetTool('user-1') - const result = await tool.execute('call-1', { assetId: 'asset-img-1' }) - - expect(authzService.hasPermission).toHaveBeenCalledWith({ - user: { id: 'user-1' }, - permission: 'Read', - type: 'asset', - id: 'asset-img-1', - }) - - expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'proxy/sample_photo.webp') - - const expectedPath = path.join(piDir, 'asset-img-1_sample_photo.jpg') - createdFiles.push(expectedPath) - - expect(fs.existsSync(expectedPath)).toBe(true) - expect(fs.readFileSync(expectedPath, 'utf-8')).toBe('fake-proxy-image-bytes') - - expect(result.details.filePath).toBe(path.join('.pi', 'asset-img-1_sample_photo.jpg')) - expect(result.details.contentType).toBe('image/webp') - expect((result.content[0] as { type: 'text'; text: string }).text).toContain( - 'Remember to delete this temporary file when finished.', + await expect( + tool.execute('call-1', { assetId: 'asset-1', key: 'files/path/file.png' }), + ).rejects.toThrow( + 'Provide exactly one of "assetId" (to download a workspace asset) or "key" (to download a specific S3 storage key). Set the unused parameter to null.', ) }) - it('should download proxy video to .pi directory when videoTranscodes is present', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'asset-vid-1', - name: 'video_test.mov', - storageKey: { key: 'raw/video_test.mov' }, - media: { - proxyType: 'video', - videoTranscodes: [{ key: 'proxy/video_test.mp4' }], - }, - } as unknown as Asset) + describe('Download by assetId', () => { + it('should throw error if asset not found', async () => { + vi.mocked(authzService.hasPermission).mockResolvedValue() + vi.mocked(prisma.asset.findUnique).mockResolvedValue(null) - vi.mocked(s3Service.getObject).mockResolvedValue({ - buffer: Buffer.from('fake-video-bytes'), - contentType: 'video/mp4', - } as unknown as { buffer: Buffer; contentType: string }) - - const tool = createDownloadAssetTool('user-1') - const result = await tool.execute('call-1', { assetId: 'asset-vid-1' }) - - expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'proxy/video_test.mp4') - - const expectedPath = path.join(piDir, 'asset-vid-1_video_test.mov') - createdFiles.push(expectedPath) - - expect(fs.existsSync(expectedPath)).toBe(true) - expect(result.details.size).toBe(Buffer.from('fake-video-bytes').length) - }) - - it('should download original file to .pi directory even when pdfTranscode is present', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'asset-md-1', - name: 'doc.md', - storageKey: { key: 'raw/doc.md' }, - media: { - proxyType: 'pdf', - pdfTranscode: { key: 'proxy/doc.pdf' }, - original: { key: 'raw/doc.md' }, - }, - } as unknown as Asset) - - vi.mocked(s3Service.getObject).mockResolvedValue({ - buffer: Buffer.from('# fake markdown bytes'), - contentType: 'text/markdown', - } as unknown as { buffer: Buffer; contentType: string }) - - const tool = createDownloadAssetTool('user-1') - const result = await tool.execute('call-1', { assetId: 'asset-md-1' }) - - expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'raw/doc.md') - - const expectedPath = path.join(piDir, 'asset-md-1_doc.md') - createdFiles.push(expectedPath) - - expect(fs.existsSync(expectedPath)).toBe(true) - expect(result.details.contentType).toBe('text/markdown') - }) - - it('should resolve version_stack asset to its latest version file', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'stack-1', - name: '', - type: 'version_stack', - storageKey: null, - media: null, - } as unknown as Asset) - - vi.mocked(prisma.asset.findFirst).mockResolvedValue({ - id: 'file-ver-2', - name: 'banana_pig.png', - type: 'file', - storageKey: { key: 'raw/banana_pig.png' }, - media: { - proxyType: 'image', - imageTranscodes: [{ key: 'proxy/banana_pig.webp' }], - }, - } as unknown as Asset) - - vi.mocked(s3Service.getObject).mockResolvedValue({ - buffer: Buffer.from('fake-version-bytes'), - contentType: 'image/webp', - } as unknown as { buffer: Buffer; contentType: string }) - - const tool = createDownloadAssetTool('user-1') - const result = await tool.execute('call-1', { assetId: 'stack-1' }) - - expect(prisma.asset.findFirst).toHaveBeenCalledWith({ - where: { parentId: 'stack-1', isDeleted: false }, - orderBy: { sortIndex: 'asc' }, - include: { storageKey: true }, + const tool = createDownloadAssetTool('user-1') + await expect(tool.execute('call-1', { assetId: 'asset-1', key: null })).rejects.toThrow( + 'Asset with ID asset-1 not found.', + ) }) - expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'proxy/banana_pig.webp') + it('should download proxy image to .pi directory when downloading by assetId', async () => { + vi.mocked(authzService.hasPermission).mockResolvedValue() + vi.mocked(prisma.asset.findUnique).mockResolvedValue({ + id: 'asset-img-1', + name: 'sample_photo.jpg', + storageKey: { key: 'raw/sample_photo.jpg' }, + media: { + proxyType: 'image', + imageTranscodes: [{ key: 'proxy/sample_photo.webp' }], + }, + } as unknown as Asset) - const expectedPath = path.join(piDir, 'file-ver-2_banana_pig.png') - createdFiles.push(expectedPath) + vi.mocked(s3Service.getObject).mockResolvedValue({ + buffer: Buffer.from('fake-proxy-image-bytes'), + contentType: 'image/webp', + } as unknown as { buffer: Buffer; contentType: string }) - expect(fs.existsSync(expectedPath)).toBe(true) - expect(result.details.name).toBe('banana_pig.png') + const tool = createDownloadAssetTool('user-1') + const result = await tool.execute('call-1', { assetId: 'asset-img-1', key: null }) + + expect(authzService.hasPermission).toHaveBeenCalledWith({ + user: { id: 'user-1' }, + permission: 'Read', + type: 'asset', + id: 'asset-img-1', + }) + + expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'proxy/sample_photo.webp') + + const expectedPath = path.join(piDir, 'asset-img-1_sample_photo.jpg') + createdFiles.push(expectedPath) + + expect(fs.existsSync(expectedPath)).toBe(true) + expect(fs.readFileSync(expectedPath, 'utf-8')).toBe('fake-proxy-image-bytes') + + expect(result.details.filePath).toBe(path.join('.pi', 'asset-img-1_sample_photo.jpg')) + expect(result.details.contentType).toBe('image/webp') + expect((result.content[0] as { type: 'text'; text: string }).text).toContain( + 'Remember to delete this temporary file when finished.', + ) + }) + + it('should resolve version_stack asset to its latest version file', async () => { + vi.mocked(authzService.hasPermission).mockResolvedValue() + vi.mocked(prisma.asset.findUnique).mockResolvedValue({ + id: 'stack-1', + name: '', + type: 'version_stack', + storageKey: null, + media: null, + } as unknown as Asset) + + vi.mocked(prisma.asset.findFirst).mockResolvedValue({ + id: 'file-ver-2', + name: 'banana_pig.png', + type: 'file', + storageKey: { key: 'raw/banana_pig.png' }, + media: { + proxyType: 'image', + imageTranscodes: [{ key: 'proxy/banana_pig.webp' }], + }, + } as unknown as Asset) + + vi.mocked(s3Service.getObject).mockResolvedValue({ + buffer: Buffer.from('fake-version-bytes'), + contentType: 'image/webp', + } as unknown as { buffer: Buffer; contentType: string }) + + const tool = createDownloadAssetTool('user-1') + const result = await tool.execute('call-1', { assetId: 'stack-1', key: null }) + + expect(prisma.asset.findFirst).toHaveBeenCalledWith({ + where: { parentId: 'stack-1', isDeleted: false }, + orderBy: { sortIndex: 'asc' }, + include: { storageKey: true }, + }) + + expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'proxy/banana_pig.webp') + + const expectedPath = path.join(piDir, 'file-ver-2_banana_pig.png') + createdFiles.push(expectedPath) + + expect(fs.existsSync(expectedPath)).toBe(true) + expect(result.details.name).toBe('banana_pig.png') + }) + }) + + describe('Download by S3 key and Authorization', () => { + it('should reject keys that do not start with files/', async () => { + const tool = createDownloadAssetTool('user-1') + await expect( + tool.execute('call-1', { assetId: null, key: 'avatars/user-1.png' }), + ).rejects.toThrow('Invalid storage key format: key must start with "files/"') + }) + + it('should reject keys with path traversal (..)', async () => { + const tool = createDownloadAssetTool('user-1') + await expect( + tool.execute('call-1', { assetId: null, key: 'files/../../etc/passwd' }), + ).rejects.toThrow('cannot contain ".."') + }) + + it('should reject keys with insufficient path segments', async () => { + const tool = createDownloadAssetTool('user-1') + await expect(tool.execute('call-1', { assetId: null, key: 'files/invalid' })).rejects.toThrow( + 'Invalid storage key format', + ) + }) + + it('should reject keys that do not belong to any valid asset in database', async () => { + vi.mocked(prisma.asset.findUnique).mockResolvedValue(null) + vi.mocked(prisma.storageKey.findFirst).mockResolvedValue(null) + vi.mocked(prisma.storageKey.findUnique).mockResolvedValue(null) + + const tool = createDownloadAssetTool('user-1') + await expect( + tool.execute('call-1', { assetId: null, key: 'files/nonexistent-id/shot.webp' }), + ).rejects.toThrow('does not belong to any valid asset or the asset has been deleted') + }) + + it('should reject download when user lacks read permission on the owning asset (cross-project protection)', async () => { + // Key belongs to asset-in-other-project + vi.mocked(prisma.asset.findUnique).mockResolvedValue({ + id: 'asset-in-other-project', + } as unknown as Asset) + + // Authz rejects because user is in a different project/team + vi.mocked(authzService.hasPermission).mockRejectedValue( + new Error('User does not have Read permission on asset asset-in-other-project'), + ) + + const tool = createDownloadAssetTool('user-1') + await expect( + tool.execute('call-1', { + assetId: null, + key: 'files/asset-in-other-project/screenshots/secret_frame.webp', + }), + ).rejects.toThrow('User does not have Read permission on asset asset-in-other-project') + + expect(authzService.hasPermission).toHaveBeenCalledWith({ + user: { id: 'user-1' }, + permission: 'Read', + type: 'asset', + id: 'asset-in-other-project', + }) + expect(s3Service.getObject).not.toHaveBeenCalled() + }) + + it('should authorize and download derived artifact key (files//...)', async () => { + vi.mocked(prisma.asset.findUnique).mockResolvedValue({ + id: 'ast-123', + } as unknown as Asset) + vi.mocked(authzService.hasPermission).mockResolvedValue() + + vi.mocked(s3Service.getObject).mockResolvedValue({ + buffer: Buffer.from('fake-screenshot-bytes'), + contentType: 'image/webp', + } as unknown as { buffer: Buffer; contentType: string }) + + const tool = createDownloadAssetTool('user-1') + const result = await tool.execute('call-1', { + assetId: null, + key: 'files/ast-123/screenshots/shot_5.0s.webp', + }) + + expect(authzService.hasPermission).toHaveBeenCalledWith({ + user: { id: 'user-1' }, + permission: 'Read', + type: 'asset', + id: 'ast-123', + }) + + expect(s3Service.getObject).toHaveBeenCalledWith( + 'shumai', + 'files/ast-123/screenshots/shot_5.0s.webp', + ) + + const expectedPath = path.join(piDir, 'shot_5.0s.webp') + createdFiles.push(expectedPath) + + expect(fs.existsSync(expectedPath)).toBe(true) + expect(fs.readFileSync(expectedPath, 'utf-8')).toBe('fake-screenshot-bytes') + + expect(result.details.filePath).toBe(path.join('.pi', 'shot_5.0s.webp')) + expect(result.details.key).toBe('files/ast-123/screenshots/shot_5.0s.webp') + expect(result.details.assetId).toBe('ast-123') + expect(result.details.size).toBe(Buffer.from('fake-screenshot-bytes').length) + }) + + it('should authorize and download original upload key (files//...) via StorageKey', async () => { + vi.mocked(prisma.asset.findUnique).mockResolvedValue(null) + vi.mocked(prisma.storageKey.findFirst).mockResolvedValue({ + id: 'sk-1', + key: 'files/upload-ulid-456/photo.png', + assets: [{ id: 'asset-photo-1' }], + } as unknown as StorageKey & { assets: Asset[] }) + vi.mocked(authzService.hasPermission).mockResolvedValue() + + vi.mocked(s3Service.getObject).mockResolvedValue({ + buffer: Buffer.from('fake-original-photo-bytes'), + contentType: 'image/png', + } as unknown as { buffer: Buffer; contentType: string }) + + const tool = createDownloadAssetTool('user-1') + const result = await tool.execute('call-1', { + assetId: null, + key: 'files/upload-ulid-456/photo.png', + }) + + expect(authzService.hasPermission).toHaveBeenCalledWith({ + user: { id: 'user-1' }, + permission: 'Read', + type: 'asset', + id: 'asset-photo-1', + }) + + expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'files/upload-ulid-456/photo.png') + + const expectedPath = path.join(piDir, 'photo.png') + createdFiles.push(expectedPath) + + expect(fs.existsSync(expectedPath)).toBe(true) + expect(result.details.assetId).toBe('asset-photo-1') + }) + }) + + describe('resolveAssetIdFromKey helper unit tests', () => { + it('resolves directly by asset ID', async () => { + vi.mocked(prisma.asset.findUnique).mockResolvedValue({ id: 'ast-999' } as unknown as Asset) + const assetId = await resolveAssetIdFromKey('files/ast-999/pdf_pages/page_1.webp') + expect(assetId).toBe('ast-999') + }) + + it('resolves via StorageKey prefix', async () => { + vi.mocked(prisma.asset.findUnique).mockResolvedValue(null) + vi.mocked(prisma.storageKey.findFirst).mockResolvedValue({ + assets: [{ id: 'ast-888' }], + } as unknown as StorageKey & { assets: Asset[] }) + const assetId = await resolveAssetIdFromKey('files/ulid-777/document.pdf') + expect(assetId).toBe('ast-888') + }) + + it('resolves via exact StorageKey match', async () => { + vi.mocked(prisma.asset.findUnique).mockResolvedValue(null) + vi.mocked(prisma.storageKey.findFirst).mockResolvedValue(null) + vi.mocked(prisma.storageKey.findUnique).mockResolvedValue({ + assets: [{ id: 'ast-777' }], + } as unknown as StorageKey & { assets: Asset[] }) + const assetId = await resolveAssetIdFromKey('files/ulid-special/document.pdf') + expect(assetId).toBe('ast-777') + }) }) }) diff --git a/packages/agent/src/tools/download-asset.ts b/packages/agent/src/tools/download-asset.ts index 67bf41a8..8266f817 100644 --- a/packages/agent/src/tools/download-asset.ts +++ b/packages/agent/src/tools/download-asset.ts @@ -7,82 +7,107 @@ import { sanitizeFilename } from '@shumai/core/src/utils/filename' import * as fs from 'fs' import * as path from 'path' -const downloadAssetSchema = Type.Object({ - assetId: Type.String({ - description: - 'The asset ID of the workspace asset to download to local disk. This parameter is required.', - }), -}) +export const downloadAssetSchema = Type.Object( + { + assetId: Type.Union([ + Type.String({ + description: + 'The asset ID of the workspace asset to download to local disk, or null if providing "key".', + }), + Type.Null(), + ]), + key: Type.Union([ + Type.String({ + description: + 'The storage S3 key of a specific media artifact (e.g. a screenshot or transcode key from read_asset) to download to local disk, or null if providing "assetId".', + }), + Type.Null(), + ]), + }, + { additionalProperties: false }, +) + +export async function resolveAssetIdFromKey(key: string): Promise { + if (!key.startsWith('files/') || key.includes('..')) { + throw new Error( + 'Invalid storage key format: key must start with "files/" and cannot contain "..".', + ) + } + + const segments = key.split('/') + if (segments.length < 3) { + throw new Error( + 'Invalid storage key format: key must follow "files//" structure.', + ) + } + + const identifier = segments[1] + + // 1. Check if identifier is directly an Asset ID (Derived Artifacts: screenshots, pdf_pages, annotations, proxies) + const assetById = await prisma.asset.findUnique({ + where: { id: identifier, isDeleted: false }, + select: { id: true }, + }) + if (assetById) { + return assetById.id + } + + // 2. Check if identifier is a StorageKey directory prefix (Original uploads / agent-created files) + const storageKeyByPrefix = await prisma.storageKey.findFirst({ + where: { key: { startsWith: `files/${identifier}/` } }, + include: { + assets: { + where: { isDeleted: false }, + select: { id: true }, + }, + }, + }) + if (storageKeyByPrefix?.assets && storageKeyByPrefix.assets.length > 0) { + return storageKeyByPrefix.assets[0].id + } + + // 3. Exact StorageKey fallback + const exactStorageKey = await prisma.storageKey.findUnique({ + where: { key }, + include: { + assets: { + where: { isDeleted: false }, + select: { id: true }, + }, + }, + }) + if (exactStorageKey?.assets && exactStorageKey.assets.length > 0) { + return exactStorageKey.assets[0].id + } + + throw new Error( + `Storage key "${key}" does not belong to any valid asset or the asset has been deleted.`, + ) +} export function createDownloadAssetTool(userId: string): AgentTool { return { name: 'download_asset', label: 'Download Asset', description: - 'Downloads a workspace asset (proxy version if available) to the local .pi directory for processing with local scripts or bash commands. Remember to delete the temporary downloaded file when finished.', + 'Downloads a workspace asset or specific storage S3 key to the local .pi directory for processing with local scripts or bash commands. ' + + 'Provide exactly one of "assetId" (for full assets) or "key" (for intermediate artifacts like screenshots from read_asset). ' + + 'Remember to delete the temporary downloaded file when finished.', parameters: downloadAssetSchema, execute: async (_toolCallId, params) => { - const assetId = params.assetId + const { assetId, key } = params if (!userId) { throw new Error('User ID is required for authorization.') } - await authzService.hasPermission({ - user: { id: userId } as User, - permission: Permission.Read, - type: ResourceType.Asset, - id: assetId, - }) + const hasAssetId = assetId !== null + const hasKey = key !== null - let asset = await prisma.asset.findUnique({ - where: { id: assetId }, - include: { storageKey: true }, - }) - - if (!asset) { - throw new Error(`Asset with ID ${assetId} not found.`) - } - - if (asset.type === 'version_stack') { - const latestVersion = await prisma.asset.findFirst({ - where: { parentId: asset.id, isDeleted: false }, - orderBy: { sortIndex: 'asc' }, - include: { storageKey: true }, - }) - if (latestVersion) { - asset = latestVersion - } - } - - let mediaKey = asset.storageKey?.key - - if (asset.media) { - const mediaInfo = asset.media as unknown as PrismaJson.MediaInfo - if ( - mediaInfo.proxyType === 'image' || - (mediaInfo.imageTranscodes && mediaInfo.imageTranscodes.length > 0) - ) { - if (mediaInfo.imageTranscodes && mediaInfo.imageTranscodes.length > 0) { - mediaKey = mediaInfo.imageTranscodes[0].key || mediaKey - } - } else if ( - mediaInfo.proxyType === 'video' || - (mediaInfo.videoTranscodes && mediaInfo.videoTranscodes.length > 0) || - mediaInfo.videoPreview?.key - ) { - if (mediaInfo.videoTranscodes && mediaInfo.videoTranscodes.length > 0) { - mediaKey = mediaInfo.videoTranscodes[0].key || mediaKey - } else if (mediaInfo.videoPreview?.key) { - mediaKey = mediaInfo.videoPreview.key - } - } else if (mediaInfo.original?.key) { - mediaKey = mediaInfo.original.key || mediaKey - } - } - - if (!mediaKey) { - throw new Error(`No media content found for asset ${assetId}.`) + if (hasAssetId === hasKey) { + throw new Error( + 'Provide exactly one of "assetId" (to download a workspace asset) or "key" (to download a specific S3 storage key). Set the unused parameter to null.', + ) } const piDir = path.join(process.cwd(), '.pi') @@ -90,32 +115,132 @@ export function createDownloadAssetTool(userId: string): AgentTool 0) + ) { + if (mediaInfo.imageTranscodes && mediaInfo.imageTranscodes.length > 0) { + mediaKey = mediaInfo.imageTranscodes[0].key || mediaKey + } + } else if ( + mediaInfo.proxyType === 'video' || + (mediaInfo.videoTranscodes && mediaInfo.videoTranscodes.length > 0) || + mediaInfo.videoPreview?.key + ) { + if (mediaInfo.videoTranscodes && mediaInfo.videoTranscodes.length > 0) { + mediaKey = mediaInfo.videoTranscodes[0].key || mediaKey + } else if (mediaInfo.videoPreview?.key) { + mediaKey = mediaInfo.videoPreview.key + } + } else if (mediaInfo.original?.key) { + mediaKey = mediaInfo.original.key || mediaKey + } + } + + if (!mediaKey) { + throw new Error(`No media content found for asset ${assetId}.`) + } + + const safeName = sanitizeFilename(asset.name || 'file') + const filename = `${asset.id}_${safeName}` + const targetFilePath = path.join(piDir, filename) + const relativePath = path.join('.pi', filename) + + const { buffer, contentType } = await s3Service.getObject(bucket, mediaKey) + fs.writeFileSync(targetFilePath, buffer) + + return { + content: [ + { + type: 'text', + text: `Downloaded asset "${asset.name}" (ID: ${assetId}) to "${relativePath}". Remember to delete this temporary file when finished.`, + }, + ], + details: { + assetId, + name: asset.name, + filePath: relativePath, + absolutePath: targetFilePath, + contentType: contentType || 'application/octet-stream', + size: buffer.length, }, - ], - details: { - assetId, - name: asset.name, - filePath: relativePath, - absolutePath: targetFilePath, - contentType: contentType || 'application/octet-stream', - size: buffer.length, - }, + } } + + // Branch B: Download by S3 storage key + if (key) { + const owningAssetId = await resolveAssetIdFromKey(key) + + await authzService.hasPermission({ + user: { id: userId } as User, + permission: Permission.Read, + type: ResourceType.Asset, + id: owningAssetId, + }) + + const rawBasename = path.basename(key) + const safeBasename = sanitizeFilename(rawBasename) || 'downloaded_file' + const filename = safeBasename + const targetFilePath = path.join(piDir, filename) + const relativePath = path.join('.pi', filename) + + const { buffer, contentType } = await s3Service.getObject(bucket, key) + fs.writeFileSync(targetFilePath, buffer) + + return { + content: [ + { + type: 'text', + text: `Downloaded storage key "${key}" to "${relativePath}". Remember to delete this temporary file when finished.`, + }, + ], + details: { + key, + assetId: owningAssetId, + filePath: relativePath, + absolutePath: targetFilePath, + contentType: contentType || 'application/octet-stream', + size: buffer.length, + }, + } + } + + throw new Error('Invalid download_asset parameters.') }, } } diff --git a/packages/agent/src/tools/read-asset.test.ts b/packages/agent/src/tools/read-asset.test.ts new file mode 100644 index 00000000..23c067e8 --- /dev/null +++ b/packages/agent/src/tools/read-asset.test.ts @@ -0,0 +1,544 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest' +import { createReadAssetTool } from './read-asset' +import { + prisma, + WorkflowTaskType, + WorkflowTaskStatus, + type Asset, + type WorkflowTask, +} from '@shumai/db' +import { s3Service } from '@shumai/core/src/s3/s3' +import { workflowService } from '@shumai/workflow-core' +import { authzService } from '@shumai/core/src/authz/authz' +import * as annotationResolver from './annotation-resolver' + +vi.mock('@shumai/db', async (importOriginal) => { + const actual = await importOriginal() + return { + ...actual, + prisma: { + asset: { + findUnique: vi.fn(), + findFirst: vi.fn(), + }, + workflowTask: { + create: vi.fn(), + }, + }, + } +}) + +vi.mock('@shumai/core/src/s3/s3', () => ({ + s3Service: { + getObject: vi.fn(), + presign: vi.fn().mockResolvedValue('https://mock-presigned-url.com'), + }, +})) + +vi.mock('@shumai/workflow-core', () => ({ + workflowService: { + executeWait: vi.fn(), + }, +})) + +vi.mock('@shumai/core/src/authz/authz', () => ({ + authzService: { + hasPermission: vi.fn(), + }, + Permission: { Read: 'Read' }, + ResourceType: { Asset: 'asset' }, +})) + +describe('readAssetTool', () => { + beforeEach(() => { + vi.clearAllMocks() + vi.spyOn(annotationResolver, 'resolveAnnotationsById').mockResolvedValue({ + annotations: null, + timestamp: null, + }) + }) + + it('should throw authorization error if user ID is empty', async () => { + const tool = createReadAssetTool('') + await expect( + tool.execute('call-1', { + assetId: 'asset-1', + annotationId: null, + imageConfig: null, + videoConfig: null, + docConfig: null, + }), + ).rejects.toThrow('User ID is required for authorization.') + }) + + it('should throw error if asset not found', async () => { + vi.mocked(authzService.hasPermission).mockResolvedValue() + vi.mocked(prisma.asset.findUnique).mockResolvedValue(null) + + const tool = createReadAssetTool('user-1') + await expect( + tool.execute('call-1', { + assetId: 'asset-1', + annotationId: null, + imageConfig: null, + videoConfig: null, + docConfig: null, + }), + ).rejects.toThrow('Asset with ID asset-1 not found.') + }) + + describe('Image Assets', () => { + it('should read image and return ImageContent and S3 key', async () => { + vi.mocked(authzService.hasPermission).mockResolvedValue() + vi.mocked(prisma.asset.findUnique).mockResolvedValue({ + id: 'img-1', + name: 'photo.png', + mediaType: 'image/png', + storageKey: { key: 'raw/photo.png' }, + media: { + proxyType: 'image', + imageTranscodes: [{ key: 'proxy/photo.webp' }], + }, + } as unknown as Asset) + + vi.mocked(s3Service.getObject).mockResolvedValue({ + buffer: Buffer.from('fake-image-bytes'), + contentType: 'image/webp', + } as unknown as { buffer: Buffer; contentType: string }) + + const tool = createReadAssetTool('user-1') + const result = await tool.execute('call-1', { + assetId: 'img-1', + annotationId: null, + imageConfig: null, + videoConfig: null, + docConfig: null, + }) + + expect(authzService.hasPermission).toHaveBeenCalledWith({ + user: { id: 'user-1' }, + permission: 'Read', + type: 'asset', + id: 'img-1', + }) + expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'proxy/photo.webp') + + const textItem = result.content[0] as { type: 'text'; text: string } + const imageItem = result.content[1] as { type: 'image'; data: string; mimeType: string } + + expect(textItem.text).toContain('proxy/photo.webp') + expect(imageItem.type).toBe('image') + expect(imageItem.data).toBe(Buffer.from('fake-image-bytes').toString('base64')) + expect(result.details.key).toBe('proxy/photo.webp') + }) + + it('should trigger image annotation transcode workflow when annotationId is provided', async () => { + vi.mocked(authzService.hasPermission).mockResolvedValue() + vi.mocked(prisma.asset.findUnique).mockResolvedValue({ + id: 'img-1', + name: 'photo.png', + mediaType: 'image/png', + projectId: 'project-1', + storageKey: { key: 'raw/photo.png' }, + media: { + proxyType: 'image', + imageTranscodes: [{ key: 'proxy/photo.webp' }], + }, + } as unknown as Asset) + + vi.spyOn(annotationResolver, 'resolveAnnotationsById').mockResolvedValue({ + annotations: [ + { + type: 'arrow', + color: '#ff0000', + points: [ + [0, 0], + [10, 10], + ], + }, + ], + timestamp: null, + }) + + vi.mocked(prisma.workflowTask.create).mockResolvedValue({ + id: 'task-anno', + } as unknown as WorkflowTask) + vi.mocked(workflowService.executeWait).mockResolvedValue({ + id: 'task-anno', + output: { key: 'annotated/photo.webp' }, + } as unknown as WorkflowTask) + + vi.mocked(s3Service.getObject).mockResolvedValue({ + buffer: Buffer.from('annotated-bytes'), + contentType: 'image/webp', + } as unknown as { buffer: Buffer; contentType: string }) + + const tool = createReadAssetTool('user-1') + const result = await tool.execute('call-1', { + assetId: 'img-1', + annotationId: 'comment-1', + imageConfig: null, + videoConfig: null, + docConfig: null, + }) + + expect(prisma.workflowTask.create).toHaveBeenCalledWith({ + data: { + assetId: 'img-1', + projectId: 'project-1', + type: WorkflowTaskType.transcode_image_annotation, + status: WorkflowTaskStatus.pending, + payload: { + projectId: 'project-1', + imageAnnotation: { + annotations: [ + { + type: 'arrow', + color: '#ff0000', + points: [ + [0, 0], + [10, 10], + ], + }, + ], + }, + }, + }, + }) + + expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'annotated/photo.webp') + expect(result.details.key).toBe('annotated/photo.webp') + }) + }) + + describe('Video Assets', () => { + it('should throw error if videoConfig is missing for video assets', async () => { + vi.mocked(authzService.hasPermission).mockResolvedValue() + vi.mocked(prisma.asset.findUnique).mockResolvedValue({ + id: 'vid-1', + name: 'video.mp4', + mediaType: 'video/mp4', + media: { proxyType: 'video' }, + } as unknown as Asset) + + const tool = createReadAssetTool('user-1') + await expect( + tool.execute('call-1', { + assetId: 'vid-1', + annotationId: null, + imageConfig: null, + videoConfig: null, + docConfig: null, + }), + ).rejects.toThrow('videoConfig with start, end, and count is required') + }) + + it('should throw error on invalid video time range', async () => { + vi.mocked(authzService.hasPermission).mockResolvedValue() + vi.mocked(prisma.asset.findUnique).mockResolvedValue({ + id: 'vid-1', + name: 'video.mp4', + mediaType: 'video/mp4', + media: { proxyType: 'video' }, + } as unknown as Asset) + + const tool = createReadAssetTool('user-1') + await expect( + tool.execute('call-1', { + assetId: 'vid-1', + annotationId: null, + imageConfig: null, + videoConfig: { start: 10, end: 5, count: 1 }, + docConfig: null, + }), + ).rejects.toThrow( + 'Invalid video time range: start (10) must be less than or equal to end (5).', + ) + }) + + it('should extract frames and return ImageContent array and keys', async () => { + vi.mocked(authzService.hasPermission).mockResolvedValue() + vi.mocked(prisma.asset.findUnique).mockResolvedValue({ + id: 'vid-1', + name: 'video.mp4', + mediaType: 'video/mp4', + projectId: 'project-1', + media: { proxyType: 'video' }, + } as unknown as Asset) + + vi.mocked(prisma.workflowTask.create).mockResolvedValue({ + id: 'task-shot', + } as unknown as WorkflowTask) + vi.mocked(workflowService.executeWait).mockResolvedValue({ + id: 'task-shot', + output: { + screenshots: [ + { key: 'screenshots/shot1.webp', timestamp: 2.0 }, + { key: 'screenshots/shot2.webp', timestamp: 4.0 }, + ], + }, + } as unknown as WorkflowTask) + + vi.mocked(s3Service.getObject).mockImplementation(async (_bucket, key) => ({ + buffer: Buffer.from(`bytes-for-${key}`), + contentType: 'image/webp', + })) + + const tool = createReadAssetTool('user-1') + const result = await tool.execute('call-1', { + assetId: 'vid-1', + annotationId: null, + imageConfig: null, + videoConfig: { start: 2.0, end: 4.0, count: 2 }, + docConfig: null, + }) + + expect(prisma.workflowTask.create).toHaveBeenCalledWith({ + data: { + assetId: 'vid-1', + projectId: 'project-1', + type: WorkflowTaskType.transcode_screenshot, + status: WorkflowTaskStatus.pending, + payload: { + projectId: 'project-1', + screenshot: { + start: 2.0, + end: 4.0, + count: 2, + commentTimestamp: null, + annotations: null, + }, + }, + }, + }) + + expect(result.content.length).toBe(3) // 1 text + 2 images + expect(result.details.keys).toEqual(['screenshots/shot1.webp', 'screenshots/shot2.webp']) + }) + }) + + describe('Document Assets', () => { + it('should throw error if docConfig is missing for document assets', async () => { + vi.mocked(authzService.hasPermission).mockResolvedValue() + vi.mocked(prisma.asset.findUnique).mockResolvedValue({ + id: 'doc-1', + name: 'doc.pdf', + mediaType: 'application/pdf', + media: { proxyType: 'pdf' }, + } as unknown as Asset) + + const tool = createReadAssetTool('user-1') + await expect( + tool.execute('call-1', { + assetId: 'doc-1', + annotationId: null, + imageConfig: null, + videoConfig: null, + docConfig: null, + }), + ).rejects.toThrow('docConfig with mode ("pages" or "text") is required') + }) + + it('should render PDF pages in mode: "pages"', async () => { + vi.mocked(authzService.hasPermission).mockResolvedValue() + vi.mocked(prisma.asset.findUnique).mockResolvedValue({ + id: 'doc-1', + name: 'doc.pdf', + mediaType: 'application/pdf', + projectId: 'project-1', + media: { proxyType: 'pdf' }, + } as unknown as Asset) + + vi.mocked(prisma.workflowTask.create).mockResolvedValue({ + id: 'task-pdf', + } as unknown as WorkflowTask) + vi.mocked(workflowService.executeWait).mockResolvedValue({ + id: 'task-pdf', + output: { + pages: [ + { key: 'pdf_pages/page1.webp', page: 1 }, + { key: 'pdf_pages/page2.webp', page: 2 }, + ], + }, + } as unknown as WorkflowTask) + + vi.mocked(s3Service.getObject).mockImplementation(async (_bucket, key) => ({ + buffer: Buffer.from(`bytes-for-${key}`), + contentType: 'image/webp', + })) + + const tool = createReadAssetTool('user-1') + const result = await tool.execute('call-1', { + assetId: 'doc-1', + annotationId: null, + imageConfig: null, + videoConfig: null, + docConfig: { mode: 'pages', startPage: 1, endPage: 2 }, + }) + + expect(result.content.length).toBe(3) // 1 text + 2 images + expect(result.details.keys).toEqual(['pdf_pages/page1.webp', 'pdf_pages/page2.webp']) + }) + + it('should throw error in mode: "pages" when page range exceeds 20 pages', async () => { + vi.mocked(authzService.hasPermission).mockResolvedValue() + vi.mocked(prisma.asset.findUnique).mockResolvedValue({ + id: 'doc-1', + name: 'doc.pdf', + mediaType: 'application/pdf', + media: { proxyType: 'pdf' }, + } as unknown as Asset) + + const tool = createReadAssetTool('user-1') + await expect( + tool.execute('call-1', { + assetId: 'doc-1', + annotationId: null, + imageConfig: null, + videoConfig: null, + docConfig: { mode: 'pages', startPage: 1, endPage: 25 }, + }), + ).rejects.toThrow('exceeds the maximum limit of 20 pages') + }) + + it('should throw error in mode: "text" when called on binary PDF', async () => { + vi.mocked(authzService.hasPermission).mockResolvedValue() + vi.mocked(prisma.asset.findUnique).mockResolvedValue({ + id: 'doc-1', + name: 'manual.pdf', + mediaType: 'application/pdf', + media: { proxyType: 'pdf' }, + } as unknown as Asset) + + const tool = createReadAssetTool('user-1') + await expect( + tool.execute('call-1', { + assetId: 'doc-1', + annotationId: null, + imageConfig: null, + videoConfig: null, + docConfig: { mode: 'text', startPage: null, endPage: null }, + }), + ).rejects.toThrow('is a binary PDF/document and cannot be read as raw text') + }) + + it('should throw error in mode: "text" when annotationId is provided', async () => { + vi.mocked(authzService.hasPermission).mockResolvedValue() + vi.mocked(prisma.asset.findUnique).mockResolvedValue({ + id: 'doc-1', + name: 'notes.md', + mediaType: 'text/markdown', + media: { proxyType: 'pdf' }, + } as unknown as Asset) + + const tool = createReadAssetTool('user-1') + await expect( + tool.execute('call-1', { + assetId: 'doc-1', + annotationId: 'comment-123', + imageConfig: null, + videoConfig: null, + docConfig: { mode: 'text', startPage: null, endPage: null }, + }), + ).rejects.toThrow('annotationId cannot be used with docConfig mode "text"') + }) + + it('should throw error in mode: "text" when startPage or endPage is non-null', async () => { + vi.mocked(authzService.hasPermission).mockResolvedValue() + vi.mocked(prisma.asset.findUnique).mockResolvedValue({ + id: 'doc-1', + name: 'notes.md', + mediaType: 'text/markdown', + media: { proxyType: 'pdf' }, + } as unknown as Asset) + + const tool = createReadAssetTool('user-1') + await expect( + tool.execute('call-1', { + assetId: 'doc-1', + annotationId: null, + imageConfig: null, + videoConfig: null, + docConfig: { mode: 'text', startPage: 1, endPage: 2 }, + }), + ).rejects.toThrow('startPage and endPage must be null when docConfig mode is "text"') + }) + + it('should read raw text from S3 in mode: "text" for markdown files', async () => { + vi.mocked(authzService.hasPermission).mockResolvedValue() + vi.mocked(prisma.asset.findUnique).mockResolvedValue({ + id: 'doc-md', + name: 'README.md', + mediaType: 'text/markdown', + storageKey: { key: 'raw/README.md' }, + media: { proxyType: 'pdf' }, + } as unknown as Asset) + + vi.mocked(s3Service.getObject).mockResolvedValue({ + buffer: Buffer.from('# Hello Shumai\nThis is markdown text.'), + contentType: 'text/markdown', + } as unknown as { buffer: Buffer; contentType: string }) + + const tool = createReadAssetTool('user-1') + const result = await tool.execute('call-1', { + assetId: 'doc-md', + annotationId: null, + imageConfig: null, + videoConfig: null, + docConfig: { mode: 'text', startPage: null, endPage: null }, + }) + + expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'raw/README.md') + expect(result.content[0].type).toBe('text') + expect((result.content[0] as { type: 'text'; text: string }).text).toContain( + '# Hello Shumai\nThis is markdown text.', + ) + expect(result.details.key).toBe('raw/README.md') + }) + }) + + describe('Version Stack Resolution', () => { + it('should resolve version_stack to latest version asset', async () => { + vi.mocked(authzService.hasPermission).mockResolvedValue() + vi.mocked(prisma.asset.findUnique).mockResolvedValue({ + id: 'stack-1', + name: '', + type: 'version_stack', + } as unknown as Asset) + + vi.mocked(prisma.asset.findFirst).mockResolvedValue({ + id: 'file-v2', + name: 'diagram_v2.png', + type: 'file', + mediaType: 'image/png', + storageKey: { key: 'raw/diagram_v2.png' }, + media: { + proxyType: 'image', + imageTranscodes: [{ key: 'proxy/diagram_v2.webp' }], + }, + } as unknown as Asset) + + vi.mocked(s3Service.getObject).mockResolvedValue({ + buffer: Buffer.from('v2-image-bytes'), + contentType: 'image/webp', + } as unknown as { buffer: Buffer; contentType: string }) + + const tool = createReadAssetTool('user-1') + const result = await tool.execute('call-1', { + assetId: 'stack-1', + annotationId: null, + imageConfig: null, + videoConfig: null, + docConfig: null, + }) + + expect(prisma.asset.findFirst).toHaveBeenCalledWith({ + where: { parentId: 'stack-1', isDeleted: false }, + orderBy: { sortIndex: 'asc' }, + include: { storageKey: true }, + }) + expect(result.details.name).toBe('diagram_v2.png') + expect(result.details.key).toBe('proxy/diagram_v2.webp') + }) + }) +}) diff --git a/packages/agent/src/tools/read-asset.ts b/packages/agent/src/tools/read-asset.ts new file mode 100644 index 00000000..2e53d798 --- /dev/null +++ b/packages/agent/src/tools/read-asset.ts @@ -0,0 +1,576 @@ +import { Type } from 'typebox' +import { type AgentTool } from '@earendil-works/pi-agent-core' +import { type ImageContent, type TextContent } from '@earendil-works/pi-ai' +import { prisma, WorkflowTaskType, WorkflowTaskStatus, type User } from '@shumai/db' +import { s3Service } from '@shumai/core/src/s3/s3' +import { workflowService } from '@shumai/workflow-core' +import { authzService, Permission, ResourceType } from '@shumai/core/src/authz/authz' +import { resolveAnnotationsById } from './annotation-resolver' + +export const readAssetSchema = Type.Object( + { + assetId: Type.String({ + description: 'The asset ID of the workspace asset to inspect. Required.', + }), + annotationId: Type.Union([ + Type.String({ + description: + 'The ID of the comment or message entry from whose visual markup should be overlaid, or null.', + }), + Type.Null(), + ]), + imageConfig: Type.Union([ + Type.Object( + {}, + { + additionalProperties: false, + description: 'Optional configuration for image assets, or null.', + }, + ), + Type.Null(), + ]), + videoConfig: Type.Union([ + Type.Object( + { + start: Type.Number({ description: 'Start time in seconds.' }), + end: Type.Number({ description: 'End time in seconds.' }), + count: Type.Number({ description: 'Number of frames to extract (e.g. 1 to 10).' }), + }, + { + additionalProperties: false, + description: + 'Configuration for video assets. Required when inspecting videos, or null for other asset types.', + }, + ), + Type.Null(), + ]), + docConfig: Type.Union([ + Type.Object( + { + mode: Type.Union([ + Type.Literal('pages', { + description: + 'Render specific page range as images for visual layout, charts, or visual annotations (required for binary PDFs).', + }), + Type.Literal('text', { + description: + 'Read raw text content directly (ideal for .md, .txt, .csv, code files).', + }), + ]), + startPage: Type.Union([ + Type.Number({ + description: + 'Start page number (1-based index, required when mode="pages"), or null when mode="text".', + }), + Type.Null(), + ]), + endPage: Type.Union([ + Type.Number({ + description: + 'End page number (1-based index, max 20 pages per call, required when mode="pages"), or null when mode="text".', + }), + Type.Null(), + ]), + }, + { + additionalProperties: false, + description: + 'Configuration for document/PDF/text assets, or null if inspecting another asset type.', + }, + ), + Type.Null(), + ]), + }, + { additionalProperties: false }, +) + +export function isPlainTextAsset(mediaType: string, filename: string): boolean { + if ( + mediaType.startsWith('text/') || + mediaType === 'application/json' || + mediaType === 'application/javascript' || + mediaType === 'application/typescript' || + mediaType === 'application/xml' || + mediaType === 'application/x-yaml' + ) { + return true + } + const textExtensions = [ + '.txt', + '.md', + '.markdown', + '.json', + '.csv', + '.tsv', + '.js', + '.ts', + '.tsx', + '.jsx', + '.html', + '.htm', + '.css', + '.scss', + '.yaml', + '.yml', + '.xml', + '.py', + '.rb', + '.go', + '.rs', + '.c', + '.cpp', + '.h', + '.hpp', + '.java', + '.sh', + '.bash', + '.zsh', + '.sql', + ] + return textExtensions.some((ext) => filename.endsWith(ext)) +} + +const MAX_TEXT_BYTES = 50 * 1024 +const MAX_TEXT_LINES = 1000 + +export function createReadAssetTool(userId: string): AgentTool { + return { + name: 'read_asset', + label: 'Read Asset', + description: + 'Inspects and reads the content of an asset in the workspace. ' + + 'Supports visual analysis for images, video frame extraction, PDF page rendering, and direct text reading for text/markdown files. ' + + 'Pass assetId, and provide the specific config (imageConfig, videoConfig, or docConfig) while setting unused configs to null. ' + + 'Optionally provide annotationId to view visual markups drawn on the asset.', + parameters: readAssetSchema, + execute: async (_toolCallId, params) => { + const { assetId, annotationId, videoConfig, docConfig } = params + + if (!userId) { + throw new Error('User ID is required for authorization.') + } + + await authzService.hasPermission({ + user: { id: userId } as User, + permission: Permission.Read, + type: ResourceType.Asset, + id: assetId, + }) + + let asset = await prisma.asset.findUnique({ + where: { id: assetId }, + include: { storageKey: true }, + }) + + if (!asset) { + throw new Error(`Asset with ID ${assetId} not found.`) + } + + if (asset.type === 'version_stack') { + const latestVersion = await prisma.asset.findFirst({ + where: { parentId: asset.id, isDeleted: false }, + orderBy: { sortIndex: 'asc' }, + include: { storageKey: true }, + }) + if (latestVersion) { + asset = latestVersion + } + } + + const mediaInfo = asset.media as unknown as PrismaJson.MediaInfo | null + const proxyType = mediaInfo?.proxyType + const mediaType = asset.mediaType?.toLowerCase() || '' + const filename = asset.name?.toLowerCase() || '' + const bucket = process.env.S3_BUCKET || 'shumai' + + // ---------------------------------------------------------------------- + // Branch 1: Image Assets + // ---------------------------------------------------------------------- + if (proxyType === 'image' || mediaType.startsWith('image/') || filename.endsWith('.psd')) { + let mediaKey = asset.storageKey?.key + if (mediaInfo?.imageTranscodes && mediaInfo.imageTranscodes.length > 0) { + mediaKey = mediaInfo.imageTranscodes[0].key || mediaKey + } + + if (!mediaKey) { + throw new Error('No media content found for this asset.') + } + + const { annotations } = await resolveAnnotationsById(asset.id, annotationId) + let keyToUse = mediaKey + + if (annotations) { + const task = await prisma.workflowTask.create({ + data: { + assetId: asset.id, + projectId: asset.projectId || 'none', + type: WorkflowTaskType.transcode_image_annotation, + status: WorkflowTaskStatus.pending, + payload: { + projectId: asset.projectId || 'none', + imageAnnotation: { + annotations, + }, + }, + }, + }) + + const completedTask = await workflowService.executeWait(task) + // eslint-disable-next-line @typescript-eslint/no-explicit-any -- transcode image annotation task output + const output = completedTask.output as any + if (output?.key) { + keyToUse = output.key + } else { + throw new Error('Image annotation transcode workflow failed to output S3 key.') + } + } + + const { buffer, contentType } = await s3Service.getObject(bucket, keyToUse) + const mimeType = + contentType && contentType !== 'application/octet-stream' ? contentType : 'image/webp' + const downloadUrl = await s3Service.presign( + bucket, + keyToUse, + 'GET', + true, + asset.name || undefined, + ) + + const content: Array = [ + { + type: 'text', + text: `Image asset "${asset.name}" (ID: ${asset.id}):\n- S3 Key: "${keyToUse}"`, + }, + { + type: 'image', + data: buffer.toString('base64'), + mimeType, + }, + ] + + return { + content, + details: { + assetId: asset.id, + name: asset.name, + mediaType: asset.mediaType, + key: keyToUse, + downloadUrl, + sourceKeys: [keyToUse], + }, + } + } + + // ---------------------------------------------------------------------- + // Branch 2: Video Assets + // ---------------------------------------------------------------------- + if (proxyType === 'video' || mediaType.startsWith('video/')) { + if (!videoConfig) { + throw new Error( + 'videoConfig with start, end, and count is required when inspecting video assets.', + ) + } + + const { start, end, count } = videoConfig + + if (start < 0 || end < 0) { + throw new Error('Invalid video time range: start and end must be non-negative numbers.') + } + + if (start > end) { + throw new Error( + `Invalid video time range: start (${start}) must be less than or equal to end (${end}).`, + ) + } + + if (count < 1) { + throw new Error(`Invalid frame count: count (${count}) must be at least 1.`) + } + + const { annotations, timestamp: commentTimestamp } = await resolveAnnotationsById( + asset.id, + annotationId, + ) + + const task = await prisma.workflowTask.create({ + data: { + assetId: asset.id, + projectId: asset.projectId || 'none', + type: WorkflowTaskType.transcode_screenshot, + status: WorkflowTaskStatus.pending, + payload: { + projectId: asset.projectId || 'none', + screenshot: { + start, + end, + count, + commentTimestamp, + annotations, + }, + }, + }, + }) + + const completedTask = await workflowService.executeWait(task) + // eslint-disable-next-line @typescript-eslint/no-explicit-any -- transcode screenshot output + const output = completedTask.output as any + const screenshots = output?.screenshots as + | Array<{ key: string; timestamp: number }> + | undefined + + if (!screenshots || screenshots.length === 0) { + throw new Error('No screenshots were generated.') + } + + const content: Array = [] + const sourceKeys: string[] = [] + const results: Array<{ key: string; timestamp: number; downloadUrl: string }> = [] + const textLines: string[] = [ + `Extracted ${screenshots.length} frame(s) from "${asset.name}" (${start}s - ${end}s):`, + ] + + for (const shot of screenshots) { + const { buffer, contentType } = await s3Service.getObject(bucket, shot.key) + const mimeType = + contentType && contentType !== 'application/octet-stream' ? contentType : 'image/webp' + const downloadUrl = await s3Service.presign( + bucket, + shot.key, + 'GET', + true, + asset.name || undefined, + ) + + textLines.push(`- Frame at ${shot.timestamp}s: S3 Key: "${shot.key}"`) + content.push({ + type: 'image', + data: buffer.toString('base64'), + mimeType, + }) + sourceKeys.push(shot.key) + results.push({ + key: shot.key, + timestamp: shot.timestamp, + downloadUrl, + }) + } + + content.unshift({ + type: 'text', + text: textLines.join('\n'), + }) + + return { + content, + details: { + assetId: asset.id, + name: asset.name, + mediaType: asset.mediaType, + keys: sourceKeys, + results, + sourceKeys, + }, + } + } + + // ---------------------------------------------------------------------- + // Branch 3: Documents & PDF Assets + // ---------------------------------------------------------------------- + if ( + proxyType === 'pdf' || + mediaType === 'application/pdf' || + isPlainTextAsset(mediaType, filename) + ) { + if (!docConfig) { + throw new Error( + 'docConfig with mode ("pages" or "text") is required when inspecting document/PDF assets.', + ) + } + + if (docConfig.mode === 'text') { + if (annotationId !== null) { + throw new Error( + 'annotationId cannot be used with docConfig mode "text" because visual markups can only be rendered on visual pages. Use docConfig: { mode: "pages", startPage: ..., endPage: ... } instead.', + ) + } + + if (docConfig.startPage !== null || docConfig.endPage !== null) { + throw new Error( + 'startPage and endPage must be null when docConfig mode is "text". To inspect specific page numbers, use mode: "pages".', + ) + } + + if (!isPlainTextAsset(mediaType, filename)) { + throw new Error( + `Asset "${asset.name}" is a binary PDF/document and cannot be read as raw text. Please call read_asset with docConfig: { mode: "pages", startPage: 1, endPage: ... } to view its visual pages.`, + ) + } + + const mediaKey = asset.storageKey?.key + if (!mediaKey) { + throw new Error(`No media content found for asset ${asset.id}.`) + } + + const { buffer } = await s3Service.getObject(bucket, mediaKey) + const rawText = buffer.toString('utf-8') + let text = rawText + let isTruncated = false + + const lines = text.split('\n') + if (lines.length > MAX_TEXT_LINES) { + text = lines.slice(0, MAX_TEXT_LINES).join('\n') + isTruncated = true + } + + if (Buffer.byteLength(text, 'utf-8') > MAX_TEXT_BYTES) { + text = Buffer.from(text, 'utf-8').subarray(0, MAX_TEXT_BYTES).toString('utf-8') + isTruncated = true + } + + if (isTruncated) { + text += `\n\n[Content truncated to ${MAX_TEXT_LINES} lines / ${MAX_TEXT_BYTES / 1024}KB limit. Use download_asset to process the full file.]` + } + + const downloadUrl = await s3Service.presign( + bucket, + mediaKey, + 'GET', + true, + asset.name || undefined, + ) + + return { + content: [ + { + type: 'text', + text: `Document "${asset.name}" (ID: ${asset.id}, S3 Key: "${mediaKey}"):\n\n${text}`, + }, + ], + details: { + assetId: asset.id, + name: asset.name, + mediaType: asset.mediaType, + key: mediaKey, + downloadUrl, + sourceKeys: [mediaKey], + size: buffer.length, + isTruncated, + }, + } + } + + // docConfig.mode === 'pages' + if (docConfig.startPage === null || docConfig.endPage === null) { + throw new Error('startPage and endPage are required when docConfig mode is "pages".') + } + + const start = docConfig.startPage + const end = docConfig.endPage + + if (start < 1) { + throw new Error(`Invalid page range: start page (${start}) must be at least 1.`) + } + + if (start > end) { + throw new Error( + `Invalid page range: start page (${start}) must be less than or equal to end page (${end}).`, + ) + } + + const MAX_PAGE_RANGE = 20 + const requestedPages = end - start + 1 + if (requestedPages > MAX_PAGE_RANGE) { + throw new Error( + `Page range (${requestedPages} pages requested) exceeds the maximum limit of ${MAX_PAGE_RANGE} pages per request.`, + ) + } + + const { annotations, timestamp: commentTimestamp } = await resolveAnnotationsById( + asset.id, + annotationId, + ) + + const task = await prisma.workflowTask.create({ + data: { + assetId: asset.id, + projectId: asset.projectId || 'none', + type: WorkflowTaskType.transcode_pdf_pages, + status: WorkflowTaskStatus.pending, + payload: { + projectId: asset.projectId || 'none', + pdfPages: { + start, + end, + commentTimestamp, + annotations, + }, + }, + }, + }) + + const completedTask = await workflowService.executeWait(task) + // eslint-disable-next-line @typescript-eslint/no-explicit-any -- transcode pdf pages output + const output = completedTask.output as any + const pages = output?.pages as Array<{ key: string; page: number }> | undefined + + if (!pages || pages.length === 0) { + throw new Error('No PDF page images were generated.') + } + + const content: Array = [] + const sourceKeys: string[] = [] + const results: Array<{ key: string; page: number; downloadUrl: string }> = [] + const textLines: string[] = [ + `Rendered ${pages.length} page(s) of "${asset.name}" (Pages ${start} to ${end}):`, + ] + + for (const pageItem of pages) { + const { buffer, contentType } = await s3Service.getObject(bucket, pageItem.key) + const mimeType = + contentType && contentType !== 'application/octet-stream' ? contentType : 'image/webp' + const downloadUrl = await s3Service.presign( + bucket, + pageItem.key, + 'GET', + true, + asset.name || undefined, + ) + + textLines.push(`- Page ${pageItem.page}: S3 Key: "${pageItem.key}"`) + content.push({ + type: 'image', + data: buffer.toString('base64'), + mimeType, + }) + sourceKeys.push(pageItem.key) + results.push({ + key: pageItem.key, + page: pageItem.page, + downloadUrl, + }) + } + + content.unshift({ + type: 'text', + text: textLines.join('\n'), + }) + + return { + content, + details: { + assetId: asset.id, + name: asset.name, + mediaType: asset.mediaType, + keys: sourceKeys, + results, + sourceKeys, + }, + } + } + + throw new Error( + `Unsupported asset type for read_asset. Asset ${asset.id} has media type "${mediaType || 'unknown'}". Use download_asset to download this file to disk.`, + ) + }, + } +} diff --git a/packages/agent/src/tools/read-pdf-pages.test.ts b/packages/agent/src/tools/read-pdf-pages.test.ts deleted file mode 100644 index 9f8743d9..00000000 --- a/packages/agent/src/tools/read-pdf-pages.test.ts +++ /dev/null @@ -1,301 +0,0 @@ -import { describe, it, expect, vi, beforeEach } from 'vitest' -import { createReadPdfPagesTool } from './read-pdf-pages' -import { - prisma, - WorkflowTaskStatus, - type Asset, - type AssetComment, - type AgentSessionEntry, - type User, - type WorkflowTask, -} from '@shumai/db' -import { s3Service } from '@shumai/core/src/s3/s3' -import { workflowService } from '@shumai/workflow-core' -import { authzService } from '@shumai/core/src/authz/authz' - -vi.mock('@shumai/db', async (importOriginal) => { - const actual = await importOriginal() - return { - ...actual, - prisma: { - user: { - findUnique: vi.fn(), - }, - asset: { - findUnique: vi.fn(), - }, - assetComment: { - findUnique: vi.fn(), - }, - agentSessionEntry: { - findUnique: vi.fn(), - }, - workflowTask: { - create: vi.fn(), - }, - }, - } -}) - -vi.mock('@shumai/core/src/s3/s3', () => ({ - s3Service: { - getObject: vi.fn(), - }, -})) - -vi.mock('@shumai/workflow-core', () => ({ - workflowService: { - executeWait: vi.fn(), - }, -})) - -vi.mock('@shumai/core/src/authz/authz', () => ({ - authzService: { - hasPermission: vi.fn(), - }, - Permission: { Read: 'Read' }, - ResourceType: { Asset: 'asset' }, -})) - -describe('readPdfPagesTool', () => { - beforeEach(() => { - vi.clearAllMocks() - vi.mocked(authzService.hasPermission).mockResolvedValue() - }) - - it('should trigger pdfPages transcode workflow and return image outputs with annotation from AssetComment', async () => { - vi.mocked(prisma.user.findUnique).mockResolvedValue({ id: 'user-1' } as User) - - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'asset-1', - projectId: 'project-1', - media: { proxyType: 'pdf' }, - } as unknown as Asset) - - vi.mocked(prisma.assetComment.findUnique).mockResolvedValue({ - id: 'comment-1', - assetId: 'asset-1', - second: 2, - annotation: [ - { - type: 'box', - color: '#ff0000', - points: [ - [0, 0], - [1, 1], - ], - }, - ], - } as unknown as AssetComment) - - vi.mocked(prisma.workflowTask.create).mockResolvedValue({ - id: 'task-1', - } as unknown as WorkflowTask) - - vi.mocked(workflowService.executeWait).mockResolvedValue({ - id: 'task-1', - status: WorkflowTaskStatus.completed, - output: { - pages: [ - { key: 'pdf_pages/page_1.webp', page: 1 }, - { key: 'pdf_pages/page_2.webp', page: 2 }, - ], - }, - } as unknown as WorkflowTask) - - vi.mocked(s3Service.getObject).mockImplementation(async (_bucket: string, key: string) => { - return { - buffer: Buffer.from(`fake-bytes-for-${key}`), - contentType: 'image/webp', - } as unknown as { buffer: Buffer; contentType: string } - }) - - const tool = createReadPdfPagesTool('user-1') - const result = await tool.execute('call-1', { - assetId: 'asset-1', - start: 1, - end: 2, - annotationId: 'comment-1', - }) - - expect(authzService.hasPermission).toHaveBeenCalledWith({ - user: { id: 'user-1' }, - permission: 'Read', - type: 'asset', - id: 'asset-1', - }) - - expect(prisma.workflowTask.create).toHaveBeenCalledWith({ - data: { - assetId: 'asset-1', - projectId: 'project-1', - type: 'transcode_pdf_pages', - status: 'pending', - payload: { - projectId: 'project-1', - pdfPages: { - start: 1, - end: 2, - commentTimestamp: 2, - annotations: [ - { - type: 'box', - color: '#ff0000', - points: [ - [0, 0], - [1, 1], - ], - }, - ], - }, - }, - }, - }) - - expect(result.content.length).toBe(2) - expect(result.content[0].type).toBe('image') - expect((result.content[0] as unknown as { data: string }).data).toBe( - Buffer.from('fake-bytes-for-pdf_pages/page_1.webp').toString('base64'), - ) - expect(result.details.sourceKeys).toEqual(['pdf_pages/page_1.webp', 'pdf_pages/page_2.webp']) - }) - - it('should trigger pdfPages transcode workflow with annotations from AgentSessionEntry (1-on-1 chat mode)', async () => { - vi.mocked(prisma.user.findUnique).mockResolvedValue({ id: 'user-1' } as User) - - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'asset-1', - projectId: 'project-1', - media: { proxyType: 'pdf' }, - } as unknown as Asset) - - vi.mocked(prisma.assetComment.findUnique).mockResolvedValue(null) - vi.mocked(prisma.agentSessionEntry.findUnique).mockResolvedValue({ - id: 'entry-1', - assetId: 'asset-1', - data: { - details: { - position: { type: 'page', page: 3 }, - annotations: [{ type: 'highlight', page: 3 }], - }, - }, - } as unknown as AgentSessionEntry) - - vi.mocked(prisma.workflowTask.create).mockResolvedValue({ - id: 'task-1', - } as unknown as WorkflowTask) - - vi.mocked(workflowService.executeWait).mockResolvedValue({ - id: 'task-1', - status: WorkflowTaskStatus.completed, - output: { - pages: [{ key: 'pdf_pages/page_3.webp', page: 3 }], - }, - } as unknown as WorkflowTask) - - vi.mocked(s3Service.getObject).mockImplementation(async (_bucket: string, key: string) => { - return { - buffer: Buffer.from(`fake-bytes-for-${key}`), - contentType: 'image/webp', - } as unknown as { buffer: Buffer; contentType: string } - }) - - const tool = createReadPdfPagesTool('user-1') - await tool.execute('call-2', { - assetId: 'asset-1', - start: 1, - end: 3, - annotationId: 'entry-1', - }) - - expect(prisma.workflowTask.create).toHaveBeenCalledWith({ - data: { - assetId: 'asset-1', - projectId: 'project-1', - type: 'transcode_pdf_pages', - status: 'pending', - payload: { - projectId: 'project-1', - pdfPages: { - start: 1, - end: 3, - commentTimestamp: null, - annotations: [{ type: 'highlight', page: 3 }], - }, - }, - }, - }) - }) - - it('should throw an error when annotationId belongs to a different asset', async () => { - vi.mocked(prisma.user.findUnique).mockResolvedValue({ id: 'user-1' } as User) - - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'asset-1', - projectId: 'project-1', - media: { proxyType: 'pdf' }, - } as unknown as Asset) - - vi.mocked(prisma.assetComment.findUnique).mockResolvedValue({ - id: 'comment-on-other-asset', - assetId: 'asset-pdf-2', - annotation: [{ type: 'highlight' }], - } as unknown as AssetComment) - - const tool = createReadPdfPagesTool('user-1') - await expect( - tool.execute('call-mismatch', { - assetId: 'asset-1', - start: 1, - end: 3, - annotationId: 'comment-on-other-asset', - }), - ).rejects.toThrow( - 'Annotation "comment-on-other-asset" belongs to asset "asset-pdf-2", not target asset "asset-1".', - ) - }) - - it('should throw error if start page is less than 1', async () => { - const tool = createReadPdfPagesTool('user-1') - await expect(tool.execute('call-1', { assetId: 'asset-1', start: 0, end: 5 })).rejects.toThrow( - 'Invalid page range: start page (0) must be at least 1.', - ) - }) - - it('should throw error if start page is greater than end page', async () => { - const tool = createReadPdfPagesTool('user-1') - await expect(tool.execute('call-1', { assetId: 'asset-1', start: 6, end: 5 })).rejects.toThrow( - 'Invalid page range: start page (6) must be less than or equal to end page (5).', - ) - }) - - it('should throw error if requested page range exceeds maximum limit of 20', async () => { - const tool = createReadPdfPagesTool('user-1') - await expect(tool.execute('call-1', { assetId: 'asset-1', start: 1, end: 22 })).rejects.toThrow( - 'Page range (22 pages requested) exceeds the maximum limit of 20 pages per request.', - ) - }) - - it('should throw error if asset is not found', async () => { - vi.mocked(prisma.user.findUnique).mockResolvedValue({ id: 'user-1' } as User) - vi.mocked(prisma.asset.findUnique).mockResolvedValue(null) - - const tool = createReadPdfPagesTool('user-1') - await expect( - tool.execute('call-1', { assetId: 'non-existent', start: 1, end: 2 }), - ).rejects.toThrow('Asset with ID non-existent not found.') - }) - - it('should throw error if asset is not a PDF proxyType', async () => { - vi.mocked(prisma.user.findUnique).mockResolvedValue({ id: 'user-1' } as User) - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'asset-video', - media: { proxyType: 'video' }, - } as unknown as Asset) - - const tool = createReadPdfPagesTool('user-1') - await expect( - tool.execute('call-1', { assetId: 'asset-video', start: 1, end: 2 }), - ).rejects.toThrow('Asset asset-video is not a PDF or document.') - }) -}) diff --git a/packages/agent/src/tools/read-pdf-pages.ts b/packages/agent/src/tools/read-pdf-pages.ts deleted file mode 100644 index f1ba9c09..00000000 --- a/packages/agent/src/tools/read-pdf-pages.ts +++ /dev/null @@ -1,144 +0,0 @@ -import { Type } from 'typebox' -import { type AgentTool } from '@earendil-works/pi-agent-core' -import { type ImageContent } from '@earendil-works/pi-ai' -import { prisma, WorkflowTaskType, WorkflowTaskStatus, type User } from '@shumai/db' -import { s3Service } from '@shumai/core/src/s3/s3' -import { workflowService } from '@shumai/workflow-core' -import { authzService, Permission, ResourceType } from '@shumai/core/src/authz/authz' -import { resolveAnnotationsById } from './annotation-resolver' - -const readPdfPagesSchema = Type.Object({ - assetId: Type.String({ - description: 'Asset ID of the PDF document. This parameter is required.', - }), - start: Type.Number({ description: 'Start page number (1-based index)' }), - end: Type.Number({ description: 'End page number (1-based index)' }), - annotationId: Type.Optional( - Type.String({ - description: - 'The ID of the comment or message entry from whose visual markup should be overlaid on the rendered PDF pages.', - }), - ), -}) - -export function createReadPdfPagesTool(userId: string): AgentTool { - return { - name: 'read_pdf_pages', - label: 'Read PDF Pages', - description: - 'Converts specific page range of a PDF document into 1080p WebP images for visual analysis. Maximum 20 pages allowed per call.', - parameters: readPdfPagesSchema, - execute: async (_toolCallId, params) => { - if (params.start < 1) { - throw new Error(`Invalid page range: start page (${params.start}) must be at least 1.`) - } - - if (params.start > params.end) { - throw new Error( - `Invalid page range: start page (${params.start}) must be less than or equal to end page (${params.end}).`, - ) - } - - const MAX_PAGE_RANGE = 20 - const requestedPages = params.end - params.start + 1 - if (requestedPages > MAX_PAGE_RANGE) { - throw new Error( - `Page range (${requestedPages} pages requested) exceeds the maximum limit of ${MAX_PAGE_RANGE} pages per request.`, - ) - } - - const targetAssetId = params.assetId - - if (!userId) { - throw new Error('User ID is required for authorization.') - } - - await authzService.hasPermission({ - user: { id: userId } as User, - permission: Permission.Read, - type: ResourceType.Asset, - id: targetAssetId, - }) - - let asset = await prisma.asset.findUnique({ - where: { id: targetAssetId }, - }) - - if (!asset) { - throw new Error(`Asset with ID ${targetAssetId} not found.`) - } - - if (asset.type === 'version_stack') { - const latestVersion = await prisma.asset.findFirst({ - where: { parentId: asset.id, isDeleted: false }, - orderBy: { sortIndex: 'asc' }, - }) - if (latestVersion) { - asset = latestVersion - } - } - - const proxyType = (asset.media as PrismaJson.MediaInfo | null)?.proxyType - if (proxyType !== 'pdf') { - throw new Error(`Asset ${targetAssetId} is not a PDF or document.`) - } - - // Resolve annotations dynamically by annotationId - const { annotations, timestamp: commentTimestamp } = await resolveAnnotationsById( - targetAssetId, - params.annotationId, - ) - - // Trigger transcode workflow to render PDF pages - const task = await prisma.workflowTask.create({ - data: { - assetId: targetAssetId, - projectId: asset.projectId || 'none', - type: WorkflowTaskType.transcode_pdf_pages, - status: WorkflowTaskStatus.pending, - payload: { - projectId: asset.projectId || 'none', - pdfPages: { - start: params.start, - end: params.end, - commentTimestamp, - annotations, - }, - }, - }, - }) - - const completedTask = await workflowService.executeWait(task) - // eslint-disable-next-line @typescript-eslint/no-explicit-any - const output = completedTask.output as any - const pages = output?.pages as Array<{ key: string; page: number }> | undefined - - if (!pages || pages.length === 0) { - throw new Error('No PDF page images were generated.') - } - - const content: ImageContent[] = [] - const sourceKeys: string[] = [] - const bucket = process.env.S3_BUCKET || 'shumai' - - for (const pageItem of pages) { - const { buffer, contentType } = await s3Service.getObject(bucket, pageItem.key) - const mimeType = - contentType && contentType !== 'application/octet-stream' ? contentType : 'image/webp' - content.push({ - type: 'image', - data: buffer.toString('base64'), - mimeType, - }) - sourceKeys.push(pageItem.key) - } - - return { - content, - details: { - sourceKeys, - }, - } - }, - } -} diff --git a/packages/agent/src/tools/screenshot.test.ts b/packages/agent/src/tools/screenshot.test.ts deleted file mode 100644 index 1fab9864..00000000 --- a/packages/agent/src/tools/screenshot.test.ts +++ /dev/null @@ -1,320 +0,0 @@ -import { describe, it, expect, vi, beforeEach } from 'vitest' -import { createScreenshotTool } from './screenshot' -import { - prisma, - WorkflowTaskStatus, - type Asset, - type AssetComment, - type AgentSessionEntry, - type User, - type WorkflowTask, -} from '@shumai/db' -import { s3Service } from '@shumai/core/src/s3/s3' -import { workflowService } from '@shumai/workflow-core' -import { authzService } from '@shumai/core/src/authz/authz' - -vi.mock('@shumai/db', async (importOriginal) => { - const actual = await importOriginal() - return { - ...actual, - prisma: { - user: { - findUnique: vi.fn(), - }, - asset: { - findUnique: vi.fn(), - }, - assetComment: { - findUnique: vi.fn(), - }, - agentSessionEntry: { - findUnique: vi.fn(), - }, - workflowTask: { - create: vi.fn(), - }, - }, - } -}) - -vi.mock('@shumai/core/src/s3/s3', () => ({ - s3Service: { - getObject: vi.fn(), - }, -})) - -vi.mock('@shumai/workflow-core', () => ({ - workflowService: { - executeWait: vi.fn(), - }, -})) - -vi.mock('@shumai/core/src/authz/authz', () => ({ - authzService: { - hasPermission: vi.fn(), - }, - Permission: { Read: 'Read' }, - ResourceType: { Asset: 'asset' }, -})) - -describe('screenshotTool', () => { - beforeEach(() => { - vi.clearAllMocks() - }) - - it('should trigger screenshot transcode workflow with annotations when annotationId matches AssetComment', async () => { - vi.mocked(prisma.user.findUnique).mockResolvedValue({ id: 'user-1' } as User) - vi.mocked(authzService.hasPermission).mockResolvedValue() - - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'asset-1', - projectId: 'project-1', - } as unknown as Asset) - - vi.mocked(prisma.assetComment.findUnique).mockResolvedValue({ - id: 'comment-1', - assetId: 'asset-1', - second: 5.0, - annotation: [ - { - type: 'line', - color: '#00ff00', - points: [ - [0, 0], - [1, 1], - ], - }, - ], - } as unknown as AssetComment) - - vi.mocked(prisma.workflowTask.create).mockResolvedValue({ - id: 'task-1', - } as unknown as WorkflowTask) - - vi.mocked(workflowService.executeWait).mockResolvedValue({ - id: 'task-1', - status: WorkflowTaskStatus.completed, - output: { - screenshots: [ - { key: 'screenshots/shot1.webp', timestamp: 0.0 }, - { key: 'screenshots/shot2.webp', timestamp: 5.0 }, - ], - }, - } as unknown as WorkflowTask) - - vi.mocked(s3Service.getObject).mockImplementation(async (_bucket: string, key: string) => { - return { - buffer: Buffer.from(`fake-bytes-for-${key}`), - contentType: 'image/webp', - } as unknown as { buffer: Buffer; contentType: string } - }) - - const tool = createScreenshotTool('user-1') - const result = await tool.execute('call-1', { - assetId: 'asset-1', - start: 0, - end: 10, - count: 2, - annotationId: 'comment-1', - }) - - expect(authzService.hasPermission).toHaveBeenCalledWith({ - user: { id: 'user-1' }, - permission: 'Read', - type: 'asset', - id: 'asset-1', - }) - - expect(prisma.workflowTask.create).toHaveBeenCalledWith({ - data: { - assetId: 'asset-1', - projectId: 'project-1', - type: 'transcode_screenshot', - status: 'pending', - payload: { - projectId: 'project-1', - screenshot: { - start: 0, - end: 10, - count: 2, - commentTimestamp: 5.0, - annotations: [ - { - type: 'line', - color: '#00ff00', - points: [ - [0, 0], - [1, 1], - ], - }, - ], - }, - }, - }, - }) - - expect(result.content.length).toBe(2) - expect(result.content[0].type).toBe('image') - expect((result.content[0] as unknown as { data: string }).data).toBe( - Buffer.from('fake-bytes-for-screenshots/shot1.webp').toString('base64'), - ) - expect((result.content[1] as unknown as { data: string }).data).toBe( - Buffer.from('fake-bytes-for-screenshots/shot2.webp').toString('base64'), - ) - expect(result.details.sourceKeys).toEqual(['screenshots/shot1.webp', 'screenshots/shot2.webp']) - }) - - it('should trigger screenshot transcode workflow with annotations when annotationId matches AgentSessionEntry (1-on-1 chat)', async () => { - vi.mocked(prisma.user.findUnique).mockResolvedValue({ id: 'user-1' } as User) - vi.mocked(authzService.hasPermission).mockResolvedValue() - - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'asset-1', - projectId: 'project-1', - } as unknown as Asset) - - vi.mocked(prisma.assetComment.findUnique).mockResolvedValue(null) - vi.mocked(prisma.agentSessionEntry.findUnique).mockResolvedValue({ - id: 'entry-1', - assetId: 'asset-1', - data: { - details: { - position: { type: 'time', seconds: 7.25 }, - annotations: [{ type: 'rectangle', x: 10, y: 20, width: 30, height: 40 }], - }, - }, - } as unknown as AgentSessionEntry) - - vi.mocked(prisma.workflowTask.create).mockResolvedValue({ - id: 'task-1', - } as unknown as WorkflowTask) - - vi.mocked(workflowService.executeWait).mockResolvedValue({ - id: 'task-1', - status: WorkflowTaskStatus.completed, - output: { - screenshots: [{ key: 'screenshots/shot-chat.webp', timestamp: 7.25 }], - }, - } as unknown as WorkflowTask) - - vi.mocked(s3Service.getObject).mockImplementation(async (_bucket: string, key: string) => { - return { - buffer: Buffer.from(`fake-bytes-for-${key}`), - contentType: 'image/webp', - } as unknown as { buffer: Buffer; contentType: string } - }) - - const tool = createScreenshotTool('user-1') - await tool.execute('call-2', { - assetId: 'asset-1', - start: 5, - end: 10, - count: 1, - annotationId: 'entry-1', - }) - - expect(prisma.workflowTask.create).toHaveBeenCalledWith({ - data: { - assetId: 'asset-1', - projectId: 'project-1', - type: 'transcode_screenshot', - status: 'pending', - payload: { - projectId: 'project-1', - screenshot: { - start: 5, - end: 10, - count: 1, - commentTimestamp: 7.25, - annotations: [{ type: 'rectangle', x: 10, y: 20, width: 30, height: 40 }], - }, - }, - }, - }) - }) - - it('should throw an error when annotationId belongs to a different asset', async () => { - vi.mocked(prisma.user.findUnique).mockResolvedValue({ id: 'user-1' } as User) - vi.mocked(authzService.hasPermission).mockResolvedValue() - - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'asset-1', - projectId: 'project-1', - } as unknown as Asset) - - vi.mocked(prisma.assetComment.findUnique).mockResolvedValue({ - id: 'comment-on-other-asset', - assetId: 'asset-2', - annotation: [{ type: 'arrow' }], - } as unknown as AssetComment) - - const tool = createScreenshotTool('user-1') - await expect( - tool.execute('call-mismatch', { - assetId: 'asset-1', - start: 0, - end: 5, - count: 1, - annotationId: 'comment-on-other-asset', - }), - ).rejects.toThrow( - 'Annotation "comment-on-other-asset" belongs to asset "asset-2", not target asset "asset-1".', - ) - }) - - it('should take unannotated screenshots when annotationId is omitted', async () => { - vi.mocked(prisma.user.findUnique).mockResolvedValue({ id: 'user-1' } as User) - vi.mocked(authzService.hasPermission).mockResolvedValue() - - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'asset-1', - projectId: 'project-1', - } as unknown as Asset) - - vi.mocked(prisma.workflowTask.create).mockResolvedValue({ - id: 'task-1', - } as unknown as WorkflowTask) - - vi.mocked(workflowService.executeWait).mockResolvedValue({ - id: 'task-1', - status: WorkflowTaskStatus.completed, - output: { - screenshots: [{ key: 'screenshots/shot-clean.webp', timestamp: 0.0 }], - }, - } as unknown as WorkflowTask) - - vi.mocked(s3Service.getObject).mockImplementation(async (_bucket: string, key: string) => { - return { - buffer: Buffer.from(`fake-bytes-for-${key}`), - contentType: 'image/webp', - } as unknown as { buffer: Buffer; contentType: string } - }) - - const tool = createScreenshotTool('user-1') - await tool.execute('call-3', { - assetId: 'asset-1', - start: 0, - end: 5, - count: 1, - }) - - expect(prisma.workflowTask.create).toHaveBeenCalledWith({ - data: { - assetId: 'asset-1', - projectId: 'project-1', - type: 'transcode_screenshot', - status: 'pending', - payload: { - projectId: 'project-1', - screenshot: { - start: 0, - end: 5, - count: 1, - commentTimestamp: null, - annotations: null, - }, - }, - }, - }) - }) -}) diff --git a/packages/agent/src/tools/screenshot.ts b/packages/agent/src/tools/screenshot.ts deleted file mode 100644 index 9bf750c8..00000000 --- a/packages/agent/src/tools/screenshot.ts +++ /dev/null @@ -1,115 +0,0 @@ -import { Type } from 'typebox' -import { type AgentTool } from '@earendil-works/pi-agent-core' -import { type ImageContent } from '@earendil-works/pi-ai' -import { prisma, WorkflowTaskType, WorkflowTaskStatus, type User } from '@shumai/db' -import { s3Service } from '@shumai/core/src/s3/s3' -import { workflowService } from '@shumai/workflow-core' -import { authzService, Permission, ResourceType } from '@shumai/core/src/authz/authz' -import { resolveAnnotationsById } from './annotation-resolver' - -const screenshotSchema = Type.Object({ - assetId: Type.String({ - description: 'The asset ID of the video asset. This parameter is required.', - }), - start: Type.Number({ description: 'Start time in seconds' }), - end: Type.Number({ description: 'End time in seconds' }), - count: Type.Number({ description: 'Number of screenshots to take' }), - annotationId: Type.Optional( - Type.String({ - description: - 'The ID of the comment or message entry from whose visual markup should be overlaid on the screenshot.', - }), - ), -}) - -export function createScreenshotTool(userId: string): AgentTool { - return { - name: 'screenshot', - label: 'Take Video Screenshots', - description: - 'Takes screenshots for a video at specified interval. It returns the images to the agent context.', - parameters: screenshotSchema, - execute: async (_toolCallId, params) => { - const assetId = params.assetId - - if (!userId) { - throw new Error('User ID is required for authorization.') - } - - await authzService.hasPermission({ - user: { id: userId } as User, - permission: Permission.Read, - type: ResourceType.Asset, - id: assetId, - }) - - const asset = await prisma.asset.findUnique({ - where: { id: assetId }, - }) - - if (!asset) { - throw new Error(`Asset with ID ${assetId} not found.`) - } - - // Resolve annotations dynamically by annotationId - const { annotations, timestamp: commentTimestamp } = await resolveAnnotationsById( - assetId, - params.annotationId, - ) - - // Trigger transcode workflow to take screenshots - const task = await prisma.workflowTask.create({ - data: { - assetId, - projectId: asset.projectId || 'none', - type: WorkflowTaskType.transcode_screenshot, - status: WorkflowTaskStatus.pending, - payload: { - projectId: asset.projectId || 'none', - screenshot: { - start: params.start, - end: params.end, - count: params.count, - commentTimestamp, - annotations, - }, - }, - }, - }) - - const completedTask = await workflowService.executeWait(task) - // eslint-disable-next-line @typescript-eslint/no-explicit-any - const output = completedTask.output as any - const screenshots = output?.screenshots as - | Array<{ key: string; timestamp: number }> - | undefined - - if (!screenshots || screenshots.length === 0) { - throw new Error('No screenshots were generated.') - } - - const content: ImageContent[] = [] - const sourceKeys: string[] = [] - const bucket = process.env.S3_BUCKET || 'shumai' - - for (const shot of screenshots) { - const { buffer, contentType } = await s3Service.getObject(bucket, shot.key) - const mimeType = - contentType && contentType !== 'application/octet-stream' ? contentType : 'image/webp' - content.push({ - type: 'image', - data: buffer.toString('base64'), - mimeType, - }) - sourceKeys.push(shot.key) - } - - return { - content, - details: { - sourceKeys, - }, - } - }, - } -} diff --git a/packages/core/src/quota/quota-service.test.ts b/packages/core/src/quota/quota-service.test.ts index 3ec0aa7a..5841a009 100644 --- a/packages/core/src/quota/quota-service.test.ts +++ b/packages/core/src/quota/quota-service.test.ts @@ -21,12 +21,12 @@ describe('QuotaRule Wildcard Matchers', () => { it('correctly matches agent tool name wildcards', () => { const starTool = wildcardToRegex('*') - expect(starTool.test('analyze_image')).toBe(true) - expect(starTool.test('screenshot')).toBe(true) + expect(starTool.test('read_asset')).toBe(true) + expect(starTool.test('create_file')).toBe(true) - const exactTool = wildcardToRegex('analyze_image') - expect(exactTool.test('analyze_image')).toBe(true) - expect(exactTool.test('screenshot')).toBe(false) + const exactTool = wildcardToRegex('read_asset') + expect(exactTool.test('read_asset')).toBe(true) + expect(exactTool.test('create_file')).toBe(false) }) }) @@ -405,29 +405,29 @@ describe('QuotaService', () => { await quotaService.createRule(team.id, { scopeMode: 'all_members', resource: 'agent_tool_call_count', - resourceData: { name: 'analyze_image' }, + resourceData: { name: 'read_asset' }, limit: 1, period: '1hour', enabled: true, }) - // First call to analyze_image + // First call to read_asset await quotaService.consumeQuota( { teamId: team.id, resource: 'agent_tool_call_count', - resourceData: { name: 'analyze_image' }, + resourceData: { name: 'read_asset' }, }, 1, ) - // 2nd call to analyze_image is blocked + // 2nd call to read_asset is blocked await expect( quotaService.checkQuota( { teamId: team.id, resource: 'agent_tool_call_count', - resourceData: { name: 'analyze_image' }, + resourceData: { name: 'read_asset' }, }, 1, ), diff --git a/packages/webui/components/settings/AgentFormDialog.tsx b/packages/webui/components/settings/AgentFormDialog.tsx index 961f7f9a..3e7f3623 100644 --- a/packages/webui/components/settings/AgentFormDialog.tsx +++ b/packages/webui/components/settings/AgentFormDialog.tsx @@ -70,19 +70,9 @@ const BUILTIN_TOOLS = [ description: () => m.agent_tool_read_skill_desc(), }, { - id: 'analyze_image', - name: () => m.agent_tool_analyze_image_name(), - description: () => m.agent_tool_analyze_image_desc(), - }, - { - id: 'screenshot', - name: () => m.agent_tool_screenshot_name(), - description: () => m.agent_tool_screenshot_desc(), - }, - { - id: 'read_pdf_pages', - name: () => m.agent_tool_read_pdf_pages_name(), - description: () => m.agent_tool_read_pdf_pages_desc(), + id: 'read_asset', + name: () => m.agent_tool_read_asset_name(), + description: () => m.agent_tool_read_asset_desc(), }, { id: 'list_assets', diff --git a/packages/webui/components/settings/QuotaRuleDialog.tsx b/packages/webui/components/settings/QuotaRuleDialog.tsx index 1e0d05f1..306972ca 100644 --- a/packages/webui/components/settings/QuotaRuleDialog.tsx +++ b/packages/webui/components/settings/QuotaRuleDialog.tsx @@ -69,9 +69,7 @@ interface QuotaRuleDialogProps { } const STANDARD_TOOLS = [ - { value: 'analyze_image', label: () => m.agent_tool_analyze_image_name() }, - { value: 'screenshot', label: () => m.agent_tool_screenshot_name() }, - { value: 'read_pdf_pages', label: () => m.agent_tool_read_pdf_pages_name() }, + { value: 'read_asset', label: () => m.agent_tool_read_asset_name() }, { value: 'list_assets', label: () => m.agent_tool_list_assets_name() }, { value: 'create_folder', label: () => m.agent_tool_create_folder_name() }, { value: 'create_file', label: () => m.agent_tool_create_file_name() }, @@ -129,7 +127,7 @@ export const QuotaRuleDialog: React.FC = ({ const [resource, setResource] = useState('agent_total_tokens') const [mcpServerId, setMcpServerId] = useState('') const [bashMatch, setBashMatch] = useState('*') - const [toolName, setToolName] = useState('analyze_image') + const [toolName, setToolName] = useState('read_asset') const [limit, setLimit] = useState(100000) const [period, setPeriod] = useState('1day') const [enabled, setEnabled] = useState(true) @@ -177,7 +175,7 @@ export const QuotaRuleDialog: React.FC = ({ ? resData.name : typeof resData.toolName === 'string' ? resData.toolName - : 'analyze_image', + : 'read_asset', ) setLimit(rule.limit) setPeriod(formatQuotaPeriod(rule.period) as QuotaPeriodEnum) @@ -189,7 +187,7 @@ export const QuotaRuleDialog: React.FC = ({ setResource('agent_total_tokens') setMcpServerId('') setBashMatch('*') - setToolName('analyze_image') + setToolName('read_asset') setLimit(RESOURCE_CONFIG.agent_total_tokens.defaultLimit) setPeriod('1day') setEnabled(true) @@ -213,7 +211,7 @@ export const QuotaRuleDialog: React.FC = ({ } else if (resource === 'agent_bash_call_count') { resourceData.match = bashMatch.trim() || '*' } else if (resource === 'agent_tool_call_count') { - resourceData.name = toolName || 'analyze_image' + resourceData.name = toolName || 'read_asset' } const res = await client.api.teams[':teamId'].quotas.$post({ @@ -261,7 +259,7 @@ export const QuotaRuleDialog: React.FC = ({ } else if (resource === 'agent_bash_call_count') { resourceData.match = bashMatch.trim() || '*' } else if (resource === 'agent_tool_call_count') { - resourceData.name = toolName || 'analyze_image' + resourceData.name = toolName || 'read_asset' } const res = await client.api.teams[':teamId'].quotas[':id'].$put({ diff --git a/packages/webui/messages/en.json b/packages/webui/messages/en.json index 8398ffe8..555a59f3 100644 --- a/packages/webui/messages/en.json +++ b/packages/webui/messages/en.json @@ -477,12 +477,8 @@ "agent_tool_bash_desc": "Allows the agent to run terminal/shell commands in a highly isolated environment.", "agent_tool_read_skill_name": "Read Skills", "agent_tool_read_skill_desc": "Allows the agent to load and read skill definitions to perform specialized actions.", - "agent_tool_analyze_image_name": "Analyze Image", - "agent_tool_analyze_image_desc": "Allows the agent to view and analyze the visual contents of an image file.", - "agent_tool_screenshot_name": "Video Frame Extraction", - "agent_tool_screenshot_desc": "Allows the agent to extract still image frames from video files for analysis.", - "agent_tool_read_pdf_pages_name": "Read PDF Pages", - "agent_tool_read_pdf_pages_desc": "Allows the agent to convert and view specific pages of a PDF document as images.", + "agent_tool_read_asset_name": "Read Asset", + "agent_tool_read_asset_desc": "Allows the agent to view and inspect images, video frames, and document pages.", "agent_tool_list_assets_name": "Browse Shumai", "agent_tool_list_assets_desc": "Allows the agent to list folders and browse files in Shumai.", "agent_tool_create_folder_name": "Create Folder", diff --git a/packages/webui/messages/zh.json b/packages/webui/messages/zh.json index d04e32a6..fbfd10e1 100644 --- a/packages/webui/messages/zh.json +++ b/packages/webui/messages/zh.json @@ -477,12 +477,8 @@ "agent_tool_bash_desc": "允许智能体在高度隔离的环境中运行终端/命令行命令。", "agent_tool_read_skill_name": "读取技能", "agent_tool_read_skill_desc": "允许智能体加载和读取技能定义以执行特定操作。", - "agent_tool_analyze_image_name": "分析图片", - "agent_tool_analyze_image_desc": "允许智能体查看和分析图片文件的视觉内容。", - "agent_tool_screenshot_name": "提取视频帧", - "agent_tool_screenshot_desc": "允许智能体从视频文件中提取静态画面进行分析。", - "agent_tool_read_pdf_pages_name": "读取 PDF 页面", - "agent_tool_read_pdf_pages_desc": "允许智能体将 PDF 文件的指定页面转换为图片并查看。", + "agent_tool_read_asset_name": "读取资源", + "agent_tool_read_asset_desc": "允许智能体查看和分析图片、视频画面和文档页面。", "agent_tool_list_assets_name": "浏览 Shumai", "agent_tool_list_assets_desc": "允许智能体列出文件夹并浏览 Shumai 中的文件。", "agent_tool_create_folder_name": "创建文件夹", diff --git a/packages/webui/paraglide/messages/_index.d.ts b/packages/webui/paraglide/messages/_index.d.ts index 52c1ee78..fc07b45e 100644 --- a/packages/webui/paraglide/messages/_index.d.ts +++ b/packages/webui/paraglide/messages/_index.d.ts @@ -458,12 +458,8 @@ export * from "./agent_tool_bash_name.js"; export * from "./agent_tool_bash_desc.js"; export * from "./agent_tool_read_skill_name.js"; export * from "./agent_tool_read_skill_desc.js"; -export * from "./agent_tool_analyze_image_name.js"; -export * from "./agent_tool_analyze_image_desc.js"; -export * from "./agent_tool_screenshot_name.js"; -export * from "./agent_tool_screenshot_desc.js"; -export * from "./agent_tool_read_pdf_pages_name.js"; -export * from "./agent_tool_read_pdf_pages_desc.js"; +export * from "./agent_tool_read_asset_name.js"; +export * from "./agent_tool_read_asset_desc.js"; export * from "./agent_tool_list_assets_name.js"; export * from "./agent_tool_list_assets_desc.js"; export * from "./agent_tool_create_folder_name.js"; diff --git a/packages/webui/paraglide/messages/_index.js b/packages/webui/paraglide/messages/_index.js index 70885e69..a70afb54 100644 --- a/packages/webui/paraglide/messages/_index.js +++ b/packages/webui/paraglide/messages/_index.js @@ -460,12 +460,8 @@ export * from './agent_tool_bash_name.js' export * from './agent_tool_bash_desc.js' export * from './agent_tool_read_skill_name.js' export * from './agent_tool_read_skill_desc.js' -export * from './agent_tool_analyze_image_name.js' -export * from './agent_tool_analyze_image_desc.js' -export * from './agent_tool_screenshot_name.js' -export * from './agent_tool_screenshot_desc.js' -export * from './agent_tool_read_pdf_pages_name.js' -export * from './agent_tool_read_pdf_pages_desc.js' +export * from './agent_tool_read_asset_name.js' +export * from './agent_tool_read_asset_desc.js' export * from './agent_tool_list_assets_name.js' export * from './agent_tool_list_assets_desc.js' export * from './agent_tool_create_folder_name.js' diff --git a/packages/webui/paraglide/messages/agent_tool_analyze_image_desc.d.ts b/packages/webui/paraglide/messages/agent_tool_analyze_image_desc.d.ts deleted file mode 100644 index 08ac466b..00000000 --- a/packages/webui/paraglide/messages/agent_tool_analyze_image_desc.d.ts +++ /dev/null @@ -1,16 +0,0 @@ -/** -* | output | -* | --- | -* | "Allows the agent to view and analyze the visual contents of an image file." | -* -* @param {Agent_Tool_Analyze_Image_DescInputs} inputs -* @param {{ locale?: "en" | "zh" }} options -* @returns {LocalizedString} -*/ -export const agent_tool_analyze_image_desc: ((inputs?: Agent_Tool_Analyze_Image_DescInputs, options?: { - locale?: "en" | "zh"; -}) => LocalizedString) & import("../runtime.js").MessageMetadata; -export type LocalizedString = import("../runtime.js").LocalizedString; -export type Agent_Tool_Analyze_Image_DescInputs = {}; diff --git a/packages/webui/paraglide/messages/agent_tool_analyze_image_desc.js b/packages/webui/paraglide/messages/agent_tool_analyze_image_desc.js deleted file mode 100644 index bf028bf1..00000000 --- a/packages/webui/paraglide/messages/agent_tool_analyze_image_desc.js +++ /dev/null @@ -1,29 +0,0 @@ -/* eslint-disable */ -import { getLocale, experimentalStaticLocale } from '../runtime.js'; - -/** @typedef {import('../runtime.js').LocalizedString} LocalizedString */ - -/** @typedef {{}} Agent_Tool_Analyze_Image_DescInputs */ - -const en_agent_tool_analyze_image_desc = /** @type {(inputs: Agent_Tool_Analyze_Image_DescInputs) => LocalizedString} */ () => { - return /** @type {LocalizedString} */ (`Allows the agent to view and analyze the visual contents of an image file.`) -}; - -const zh_agent_tool_analyze_image_desc = /** @type {(inputs: Agent_Tool_Analyze_Image_DescInputs) => LocalizedString} */ () => { - return /** @type {LocalizedString} */ (`允许智能体查看和分析图片文件的视觉内容。`) -}; - -/** -* | output | -* | --- | -* | "Allows the agent to view and analyze the visual contents of an image file." | -* -* @param {Agent_Tool_Analyze_Image_DescInputs} inputs -* @param {{ locale?: "en" | "zh" }} options -* @returns {LocalizedString} -*/ -export const agent_tool_analyze_image_desc = /** @type {((inputs?: Agent_Tool_Analyze_Image_DescInputs, options?: { locale?: "en" | "zh" }) => LocalizedString) & import('../runtime.js').MessageMetadata} */ ((inputs = {}, options = {}) => { - const locale = experimentalStaticLocale ?? options.locale ?? getLocale() - if (locale === "en") return en_agent_tool_analyze_image_desc(inputs) - return zh_agent_tool_analyze_image_desc(inputs) -}); \ No newline at end of file diff --git a/packages/webui/paraglide/messages/agent_tool_analyze_image_name.d.ts b/packages/webui/paraglide/messages/agent_tool_analyze_image_name.d.ts deleted file mode 100644 index c61ac6bb..00000000 --- a/packages/webui/paraglide/messages/agent_tool_analyze_image_name.d.ts +++ /dev/null @@ -1,16 +0,0 @@ -/** -* | output | -* | --- | -* | "Analyze Image" | -* -* @param {Agent_Tool_Analyze_Image_NameInputs} inputs -* @param {{ locale?: "en" | "zh" }} options -* @returns {LocalizedString} -*/ -export const agent_tool_analyze_image_name: ((inputs?: Agent_Tool_Analyze_Image_NameInputs, options?: { - locale?: "en" | "zh"; -}) => LocalizedString) & import("../runtime.js").MessageMetadata; -export type LocalizedString = import("../runtime.js").LocalizedString; -export type Agent_Tool_Analyze_Image_NameInputs = {}; diff --git a/packages/webui/paraglide/messages/agent_tool_analyze_image_name.js b/packages/webui/paraglide/messages/agent_tool_analyze_image_name.js deleted file mode 100644 index 36e1ef2a..00000000 --- a/packages/webui/paraglide/messages/agent_tool_analyze_image_name.js +++ /dev/null @@ -1,29 +0,0 @@ -/* eslint-disable */ -import { getLocale, experimentalStaticLocale } from '../runtime.js'; - -/** @typedef {import('../runtime.js').LocalizedString} LocalizedString */ - -/** @typedef {{}} Agent_Tool_Analyze_Image_NameInputs */ - -const en_agent_tool_analyze_image_name = /** @type {(inputs: Agent_Tool_Analyze_Image_NameInputs) => LocalizedString} */ () => { - return /** @type {LocalizedString} */ (`Analyze Image`) -}; - -const zh_agent_tool_analyze_image_name = /** @type {(inputs: Agent_Tool_Analyze_Image_NameInputs) => LocalizedString} */ () => { - return /** @type {LocalizedString} */ (`分析图片`) -}; - -/** -* | output | -* | --- | -* | "Analyze Image" | -* -* @param {Agent_Tool_Analyze_Image_NameInputs} inputs -* @param {{ locale?: "en" | "zh" }} options -* @returns {LocalizedString} -*/ -export const agent_tool_analyze_image_name = /** @type {((inputs?: Agent_Tool_Analyze_Image_NameInputs, options?: { locale?: "en" | "zh" }) => LocalizedString) & import('../runtime.js').MessageMetadata} */ ((inputs = {}, options = {}) => { - const locale = experimentalStaticLocale ?? options.locale ?? getLocale() - if (locale === "en") return en_agent_tool_analyze_image_name(inputs) - return zh_agent_tool_analyze_image_name(inputs) -}); \ No newline at end of file diff --git a/packages/webui/paraglide/messages/agent_tool_read_asset_desc.d.ts b/packages/webui/paraglide/messages/agent_tool_read_asset_desc.d.ts new file mode 100644 index 00000000..44f41fd9 --- /dev/null +++ b/packages/webui/paraglide/messages/agent_tool_read_asset_desc.d.ts @@ -0,0 +1,16 @@ +/** +* | output | +* | --- | +* | "Allows the agent to view and inspect images, video frames, and document pages." | +* +* @param {Agent_Tool_Read_Asset_DescInputs} inputs +* @param {{ locale?: "en" | "zh" }} options +* @returns {LocalizedString} +*/ +export const agent_tool_read_asset_desc: ((inputs?: Agent_Tool_Read_Asset_DescInputs, options?: { + locale?: "en" | "zh"; +}) => LocalizedString) & import("../runtime.js").MessageMetadata; +export type LocalizedString = import("../runtime.js").LocalizedString; +export type Agent_Tool_Read_Asset_DescInputs = {}; diff --git a/packages/webui/paraglide/messages/agent_tool_read_asset_desc.js b/packages/webui/paraglide/messages/agent_tool_read_asset_desc.js new file mode 100644 index 00000000..044be7d0 --- /dev/null +++ b/packages/webui/paraglide/messages/agent_tool_read_asset_desc.js @@ -0,0 +1,29 @@ +/* eslint-disable */ +import { getLocale, experimentalStaticLocale } from '../runtime.js'; + +/** @typedef {import('../runtime.js').LocalizedString} LocalizedString */ + +/** @typedef {{}} Agent_Tool_Read_Asset_DescInputs */ + +const en_agent_tool_read_asset_desc = /** @type {(inputs: Agent_Tool_Read_Asset_DescInputs) => LocalizedString} */ () => { + return /** @type {LocalizedString} */ (`Allows the agent to view and inspect images, video frames, and document pages.`) +}; + +const zh_agent_tool_read_asset_desc = /** @type {(inputs: Agent_Tool_Read_Asset_DescInputs) => LocalizedString} */ () => { + return /** @type {LocalizedString} */ (`允许智能体查看和分析图片、视频画面和文档页面。`) +}; + +/** +* | output | +* | --- | +* | "Allows the agent to view and inspect images, video frames, and document pages." | +* +* @param {Agent_Tool_Read_Asset_DescInputs} inputs +* @param {{ locale?: "en" | "zh" }} options +* @returns {LocalizedString} +*/ +export const agent_tool_read_asset_desc = /** @type {((inputs?: Agent_Tool_Read_Asset_DescInputs, options?: { locale?: "en" | "zh" }) => LocalizedString) & import('../runtime.js').MessageMetadata} */ ((inputs = {}, options = {}) => { + const locale = experimentalStaticLocale ?? options.locale ?? getLocale() + if (locale === "en") return en_agent_tool_read_asset_desc(inputs) + return zh_agent_tool_read_asset_desc(inputs) +}); \ No newline at end of file diff --git a/packages/webui/paraglide/messages/agent_tool_screenshot_name.d.ts b/packages/webui/paraglide/messages/agent_tool_read_asset_name.d.ts similarity index 52% rename from packages/webui/paraglide/messages/agent_tool_screenshot_name.d.ts rename to packages/webui/paraglide/messages/agent_tool_read_asset_name.d.ts index 4e0bf436..98b2f490 100644 --- a/packages/webui/paraglide/messages/agent_tool_screenshot_name.d.ts +++ b/packages/webui/paraglide/messages/agent_tool_read_asset_name.d.ts @@ -1,16 +1,16 @@ /** * | output | * | --- | -* | "Video Frame Extraction" | +* | "Read Asset" | * -* @param {Agent_Tool_Screenshot_NameInputs} inputs +* @param {Agent_Tool_Read_Asset_NameInputs} inputs * @param {{ locale?: "en" | "zh" }} options * @returns {LocalizedString} */ -export const agent_tool_screenshot_name: ((inputs?: Agent_Tool_Screenshot_NameInputs, options?: { +export const agent_tool_read_asset_name: ((inputs?: Agent_Tool_Read_Asset_NameInputs, options?: { locale?: "en" | "zh"; -}) => LocalizedString) & import("../runtime.js").MessageMetadata LocalizedString) & import("../runtime.js").MessageMetadata; export type LocalizedString = import("../runtime.js").LocalizedString; -export type Agent_Tool_Screenshot_NameInputs = {}; +export type Agent_Tool_Read_Asset_NameInputs = {}; diff --git a/packages/webui/paraglide/messages/agent_tool_read_asset_name.js b/packages/webui/paraglide/messages/agent_tool_read_asset_name.js new file mode 100644 index 00000000..4cf80078 --- /dev/null +++ b/packages/webui/paraglide/messages/agent_tool_read_asset_name.js @@ -0,0 +1,29 @@ +/* eslint-disable */ +import { getLocale, experimentalStaticLocale } from '../runtime.js'; + +/** @typedef {import('../runtime.js').LocalizedString} LocalizedString */ + +/** @typedef {{}} Agent_Tool_Read_Asset_NameInputs */ + +const en_agent_tool_read_asset_name = /** @type {(inputs: Agent_Tool_Read_Asset_NameInputs) => LocalizedString} */ () => { + return /** @type {LocalizedString} */ (`Read Asset`) +}; + +const zh_agent_tool_read_asset_name = /** @type {(inputs: Agent_Tool_Read_Asset_NameInputs) => LocalizedString} */ () => { + return /** @type {LocalizedString} */ (`读取资源`) +}; + +/** +* | output | +* | --- | +* | "Read Asset" | +* +* @param {Agent_Tool_Read_Asset_NameInputs} inputs +* @param {{ locale?: "en" | "zh" }} options +* @returns {LocalizedString} +*/ +export const agent_tool_read_asset_name = /** @type {((inputs?: Agent_Tool_Read_Asset_NameInputs, options?: { locale?: "en" | "zh" }) => LocalizedString) & import('../runtime.js').MessageMetadata} */ ((inputs = {}, options = {}) => { + const locale = experimentalStaticLocale ?? options.locale ?? getLocale() + if (locale === "en") return en_agent_tool_read_asset_name(inputs) + return zh_agent_tool_read_asset_name(inputs) +}); \ No newline at end of file diff --git a/packages/webui/paraglide/messages/agent_tool_read_pdf_pages_desc.d.ts b/packages/webui/paraglide/messages/agent_tool_read_pdf_pages_desc.d.ts deleted file mode 100644 index 3393e629..00000000 --- a/packages/webui/paraglide/messages/agent_tool_read_pdf_pages_desc.d.ts +++ /dev/null @@ -1,16 +0,0 @@ -/** -* | output | -* | --- | -* | "Allows the agent to convert and view specific pages of a PDF document as images." | -* -* @param {Agent_Tool_Read_Pdf_Pages_DescInputs} inputs -* @param {{ locale?: "en" | "zh" }} options -* @returns {LocalizedString} -*/ -export const agent_tool_read_pdf_pages_desc: ((inputs?: Agent_Tool_Read_Pdf_Pages_DescInputs, options?: { - locale?: "en" | "zh"; -}) => LocalizedString) & import("../runtime.js").MessageMetadata; -export type LocalizedString = import("../runtime.js").LocalizedString; -export type Agent_Tool_Read_Pdf_Pages_DescInputs = {}; diff --git a/packages/webui/paraglide/messages/agent_tool_read_pdf_pages_desc.js b/packages/webui/paraglide/messages/agent_tool_read_pdf_pages_desc.js deleted file mode 100644 index 78a91df3..00000000 --- a/packages/webui/paraglide/messages/agent_tool_read_pdf_pages_desc.js +++ /dev/null @@ -1,29 +0,0 @@ -/* eslint-disable */ -import { getLocale, experimentalStaticLocale } from '../runtime.js'; - -/** @typedef {import('../runtime.js').LocalizedString} LocalizedString */ - -/** @typedef {{}} Agent_Tool_Read_Pdf_Pages_DescInputs */ - -const en_agent_tool_read_pdf_pages_desc = /** @type {(inputs: Agent_Tool_Read_Pdf_Pages_DescInputs) => LocalizedString} */ () => { - return /** @type {LocalizedString} */ (`Allows the agent to convert and view specific pages of a PDF document as images.`) -}; - -const zh_agent_tool_read_pdf_pages_desc = /** @type {(inputs: Agent_Tool_Read_Pdf_Pages_DescInputs) => LocalizedString} */ () => { - return /** @type {LocalizedString} */ (`允许智能体将 PDF 文件的指定页面转换为图片并查看。`) -}; - -/** -* | output | -* | --- | -* | "Allows the agent to convert and view specific pages of a PDF document as images." | -* -* @param {Agent_Tool_Read_Pdf_Pages_DescInputs} inputs -* @param {{ locale?: "en" | "zh" }} options -* @returns {LocalizedString} -*/ -export const agent_tool_read_pdf_pages_desc = /** @type {((inputs?: Agent_Tool_Read_Pdf_Pages_DescInputs, options?: { locale?: "en" | "zh" }) => LocalizedString) & import('../runtime.js').MessageMetadata} */ ((inputs = {}, options = {}) => { - const locale = experimentalStaticLocale ?? options.locale ?? getLocale() - if (locale === "en") return en_agent_tool_read_pdf_pages_desc(inputs) - return zh_agent_tool_read_pdf_pages_desc(inputs) -}); \ No newline at end of file diff --git a/packages/webui/paraglide/messages/agent_tool_read_pdf_pages_name.d.ts b/packages/webui/paraglide/messages/agent_tool_read_pdf_pages_name.d.ts deleted file mode 100644 index ca1bacbc..00000000 --- a/packages/webui/paraglide/messages/agent_tool_read_pdf_pages_name.d.ts +++ /dev/null @@ -1,16 +0,0 @@ -/** -* | output | -* | --- | -* | "Read PDF Pages" | -* -* @param {Agent_Tool_Read_Pdf_Pages_NameInputs} inputs -* @param {{ locale?: "en" | "zh" }} options -* @returns {LocalizedString} -*/ -export const agent_tool_read_pdf_pages_name: ((inputs?: Agent_Tool_Read_Pdf_Pages_NameInputs, options?: { - locale?: "en" | "zh"; -}) => LocalizedString) & import("../runtime.js").MessageMetadata; -export type LocalizedString = import("../runtime.js").LocalizedString; -export type Agent_Tool_Read_Pdf_Pages_NameInputs = {}; diff --git a/packages/webui/paraglide/messages/agent_tool_read_pdf_pages_name.js b/packages/webui/paraglide/messages/agent_tool_read_pdf_pages_name.js deleted file mode 100644 index afffea8f..00000000 --- a/packages/webui/paraglide/messages/agent_tool_read_pdf_pages_name.js +++ /dev/null @@ -1,29 +0,0 @@ -/* eslint-disable */ -import { getLocale, experimentalStaticLocale } from '../runtime.js'; - -/** @typedef {import('../runtime.js').LocalizedString} LocalizedString */ - -/** @typedef {{}} Agent_Tool_Read_Pdf_Pages_NameInputs */ - -const en_agent_tool_read_pdf_pages_name = /** @type {(inputs: Agent_Tool_Read_Pdf_Pages_NameInputs) => LocalizedString} */ () => { - return /** @type {LocalizedString} */ (`Read PDF Pages`) -}; - -const zh_agent_tool_read_pdf_pages_name = /** @type {(inputs: Agent_Tool_Read_Pdf_Pages_NameInputs) => LocalizedString} */ () => { - return /** @type {LocalizedString} */ (`读取 PDF 页面`) -}; - -/** -* | output | -* | --- | -* | "Read PDF Pages" | -* -* @param {Agent_Tool_Read_Pdf_Pages_NameInputs} inputs -* @param {{ locale?: "en" | "zh" }} options -* @returns {LocalizedString} -*/ -export const agent_tool_read_pdf_pages_name = /** @type {((inputs?: Agent_Tool_Read_Pdf_Pages_NameInputs, options?: { locale?: "en" | "zh" }) => LocalizedString) & import('../runtime.js').MessageMetadata} */ ((inputs = {}, options = {}) => { - const locale = experimentalStaticLocale ?? options.locale ?? getLocale() - if (locale === "en") return en_agent_tool_read_pdf_pages_name(inputs) - return zh_agent_tool_read_pdf_pages_name(inputs) -}); \ No newline at end of file diff --git a/packages/webui/paraglide/messages/agent_tool_screenshot_desc.d.ts b/packages/webui/paraglide/messages/agent_tool_screenshot_desc.d.ts deleted file mode 100644 index 93b0c295..00000000 --- a/packages/webui/paraglide/messages/agent_tool_screenshot_desc.d.ts +++ /dev/null @@ -1,16 +0,0 @@ -/** -* | output | -* | --- | -* | "Allows the agent to extract still image frames from video files for analysis." | -* -* @param {Agent_Tool_Screenshot_DescInputs} inputs -* @param {{ locale?: "en" | "zh" }} options -* @returns {LocalizedString} -*/ -export const agent_tool_screenshot_desc: ((inputs?: Agent_Tool_Screenshot_DescInputs, options?: { - locale?: "en" | "zh"; -}) => LocalizedString) & import("../runtime.js").MessageMetadata; -export type LocalizedString = import("../runtime.js").LocalizedString; -export type Agent_Tool_Screenshot_DescInputs = {}; diff --git a/packages/webui/paraglide/messages/agent_tool_screenshot_desc.js b/packages/webui/paraglide/messages/agent_tool_screenshot_desc.js deleted file mode 100644 index ca20dda5..00000000 --- a/packages/webui/paraglide/messages/agent_tool_screenshot_desc.js +++ /dev/null @@ -1,29 +0,0 @@ -/* eslint-disable */ -import { getLocale, experimentalStaticLocale } from '../runtime.js'; - -/** @typedef {import('../runtime.js').LocalizedString} LocalizedString */ - -/** @typedef {{}} Agent_Tool_Screenshot_DescInputs */ - -const en_agent_tool_screenshot_desc = /** @type {(inputs: Agent_Tool_Screenshot_DescInputs) => LocalizedString} */ () => { - return /** @type {LocalizedString} */ (`Allows the agent to extract still image frames from video files for analysis.`) -}; - -const zh_agent_tool_screenshot_desc = /** @type {(inputs: Agent_Tool_Screenshot_DescInputs) => LocalizedString} */ () => { - return /** @type {LocalizedString} */ (`允许智能体从视频文件中提取静态画面进行分析。`) -}; - -/** -* | output | -* | --- | -* | "Allows the agent to extract still image frames from video files for analysis." | -* -* @param {Agent_Tool_Screenshot_DescInputs} inputs -* @param {{ locale?: "en" | "zh" }} options -* @returns {LocalizedString} -*/ -export const agent_tool_screenshot_desc = /** @type {((inputs?: Agent_Tool_Screenshot_DescInputs, options?: { locale?: "en" | "zh" }) => LocalizedString) & import('../runtime.js').MessageMetadata} */ ((inputs = {}, options = {}) => { - const locale = experimentalStaticLocale ?? options.locale ?? getLocale() - if (locale === "en") return en_agent_tool_screenshot_desc(inputs) - return zh_agent_tool_screenshot_desc(inputs) -}); \ No newline at end of file diff --git a/packages/webui/paraglide/messages/agent_tool_screenshot_name.js b/packages/webui/paraglide/messages/agent_tool_screenshot_name.js deleted file mode 100644 index 07292470..00000000 --- a/packages/webui/paraglide/messages/agent_tool_screenshot_name.js +++ /dev/null @@ -1,29 +0,0 @@ -/* eslint-disable */ -import { getLocale, experimentalStaticLocale } from '../runtime.js'; - -/** @typedef {import('../runtime.js').LocalizedString} LocalizedString */ - -/** @typedef {{}} Agent_Tool_Screenshot_NameInputs */ - -const en_agent_tool_screenshot_name = /** @type {(inputs: Agent_Tool_Screenshot_NameInputs) => LocalizedString} */ () => { - return /** @type {LocalizedString} */ (`Video Frame Extraction`) -}; - -const zh_agent_tool_screenshot_name = /** @type {(inputs: Agent_Tool_Screenshot_NameInputs) => LocalizedString} */ () => { - return /** @type {LocalizedString} */ (`提取视频帧`) -}; - -/** -* | output | -* | --- | -* | "Video Frame Extraction" | -* -* @param {Agent_Tool_Screenshot_NameInputs} inputs -* @param {{ locale?: "en" | "zh" }} options -* @returns {LocalizedString} -*/ -export const agent_tool_screenshot_name = /** @type {((inputs?: Agent_Tool_Screenshot_NameInputs, options?: { locale?: "en" | "zh" }) => LocalizedString) & import('../runtime.js').MessageMetadata} */ ((inputs = {}, options = {}) => { - const locale = experimentalStaticLocale ?? options.locale ?? getLocale() - if (locale === "en") return en_agent_tool_screenshot_name(inputs) - return zh_agent_tool_screenshot_name(inputs) -}); \ No newline at end of file