diff --git a/AGENTS.md b/AGENTS.md index 2ee08f98..77427597 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -234,6 +234,19 @@ An **Operation** is a capability agents can use, defined once and exposed automa - **Register** the definition in `packages/operations/src/registry.ts` and the implementation in `packages/core/src/operations/registry.ts`. A definition without an implementation fails typecheck. - **Run** Operations only through `runOperation(op, { userId, via, agentId? }, input)`. `via` (`api`, `agent`, `mcp`, `cli`) is recorded in audit logs. - **Tests**: real-database tests per Operation covering success, permission denial (no side effects), and audit entries. +- **Agent presentation**: by default an agent tool returns the Operation output as JSON. To add agent-only parameters or render output differently (e.g. image blocks), register an `AgentToolExtension` in `packages/agent/src/operation-extensions/`. + +### Asset Readers + +`read_asset` is backed by a registry of readers in `packages/core/src/asset-read/readers/`, one per asset family (`model3d`, `image`, `video`, `document`). The shared pipeline (`assetReadService`) handles access checks, version stacks, annotations, render tasks, and waiting; a reader only contains type-specific logic. + +To add a reader: + +1. Add its kind to `READER_KINDS` and its options to `readerOptionSchemas` in `packages/operations/src/asset/read-asset.ts` (one optional `Config` field on `read_asset`). +2. Implement `AssetReader` (`matches`, synchronous `validate`, `plan`, and `collect` if `plan` can return `render`), and add it to `readers` in `packages/core/src/asset-read/registry.ts`. Order matters: the first matching reader wins. A declared kind without a reader fails typecheck. +3. `plan` returns either `ready` with the result, or `render` with a workflow task spec and optional `context`. The pipeline stores the request on the task payload (`assetRead`), so `collect` and `get_asset_render` work across restarts and instances. +4. Use `badRequest(...)` for invalid input. Return storage keys in `ReadResult`; the pipeline turns them into short-lived links (and only gives keys to Shumai's built-in agent). +5. Add reader unit tests (`readers.test.ts`), pipeline cases in `asset-read.test.ts`, and a workflow E2E case if it renders. ## Frontend Architecture diff --git a/CHANGELOG.md b/CHANGELOG.md index 38344a95..10c09b17 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). * **cli**: `shumai-cli` has clearer help for every command (`shumai-cli --help`), suggests the right command when you mistype one, and can print raw JSON with `--json`. `ls` can now filter by type and page through very large folders. Existing commands work as before; the npm package now needs Node.js 22.12 or later * **agent**: When the agent lists a folder's contents, large folders are now paged with a "next page" marker instead of page numbers, and items appear in the same order as in the file browser +* **agent**: When the agent looks at a long video or a large document that takes more than about a minute to render, it no longer fails with a timeout. The agent is told the render is still in progress and picks up the frames or pages when they're ready, using the new "Get Asset Render" tool. Each call can now read at most 20 video frames, the same limit as PDF pages * **api-token**: API tokens are now stored securely, so they can no longer be read back from the database. A new token is shown only once when you create it, so copy it right away; the token list now shows just the last few characters. Existing tokens keep working without any changes. ## [0.6.1] - 2026-10-24 diff --git a/docs/using-shumai/agent-tools.mdx b/docs/using-shumai/agent-tools.mdx index 0f3acf18..ddc6629e 100644 --- a/docs/using-shumai/agent-tools.mdx +++ b/docs/using-shumai/agent-tools.mdx @@ -34,60 +34,66 @@ The following tools can be individually toggled in the agent configuration dialo - **Description:** Allows the agent to inspect images, extract still frames from video files, render PDF document pages, and read text/markdown documents. - **Why it matters:** Enables multimodal agents to view visual media assets uploaded to Shumai, extract frame-by-frame video screenshots, inspect document layouts, and answer questions about uploaded creative assets. +- **Limits:** Up to 20 video frames or PDF pages per call. If rendering takes longer than about a minute, the tool tells the agent the render is still in progress instead of failing. -### 4. Browse Shumai (`list_assets`) +### 4. Get Asset Render (`get_asset_render`) + +- **Description:** Allows the agent to pick up video frames or document pages that were still rendering when `read_asset` returned. +- **Why it matters:** Long videos and large documents can take a while to render. With this tool the agent can come back for the result instead of starting over. It only returns renders that `read_asset` started for the same user. + +### 5. Browse Shumai (`list_assets`) - **Description:** Allows the agent to list folders and browse files in Shumai. Large folders are returned page by page. - **Why it matters:** Gives the agent visual mapping of the project structure so it can search for other files or locate related assets. -### 5. List Projects (`list_projects`) +### 6. List Projects (`list_projects`) - **Description:** Allows the agent to list the projects you have access to across your teams, including each project's root folder. - **Why it matters:** Lets the agent find the right project and its root folder on its own before browsing or creating files. -### 6. Create Folder (`create_folder`) +### 7. Create Folder (`create_folder`) - **Description:** Allows the agent to create new folders in Shumai. - **Why it matters:** Needed if the agent organizes outputs or places files in custom subdirectory folders. -### 7. Create File (`create_file`) +### 8. Create File (`create_file`) - **Description:** Allows the agent to create and upload new files to Shumai. - **Why it matters:** Allows the agent to generate new assets (e.g. text logs, transcoded clips, reports) and save them directly back into your project. -### 8. Create Version (`create_version`) +### 9. Create Version (`create_version`) - **Description:** Allows the agent to upload and stack a new version of an existing file in Shumai. - **Why it matters:** Allows the agent to update files in a clean, non-destructive way by stacking revisions rather than creating duplicate files. -### 9. Generate Image (`generate_image`) +### 10. Generate Image (`generate_image`) - **Description:** Allows the agent to generate still images from text prompts and reference images using configured AI image models. - **Why it matters:** Enables agents to create concept art, mockups, textures, and illustrations directly inside Shumai. - **Prerequisite:** This tool requires at least one image model to be configured and enabled in [Image / Video Generation](/using-shumai/image-video-generation). If no image models are enabled under a configured provider, this tool is omitted from the agent session even if enabled in the agent settings dialog. -### 10. Generate Video (`generate_video`) +### 11. Generate Video (`generate_video`) - **Description:** Allows the agent to generate video clips from text prompts, image animation, or transition keyframes using configured AI video models. - **Why it matters:** Enables agents to produce short animated clips, camera motion shots, and video transitions. - **Prerequisite:** This tool requires at least one video model to be configured and enabled in [Image / Video Generation](/using-shumai/image-video-generation). If no video models are enabled under a configured provider, this tool is omitted from the agent session even if enabled in the agent settings dialog. -### 11. Download Asset (`download_asset`) +### 12. Download Asset (`download_asset`) - **Description:** Allows the agent to download a workspace asset or specific storage S3 key to the local `.pi` directory for processing with local scripts or bash commands. - **Why it matters:** Enables agents executing custom skill scripts to fetch files or intermediate transcode artifacts to local disk. -### 12. Rename Asset (`rename_asset`) +### 13. Rename Asset (`rename_asset`) - **Description:** Allows the agent to rename an existing file or folder in Shumai. - **Why it matters:** Enables agents to keep project files neatly named and organized according to workspace naming conventions. Enabled by default. -### 13. Move Assets (`move_assets`) +### 14. Move Assets (`move_assets`) - **Description:** Allows the agent to move one or more assets to a destination parent folder. - **Why it matters:** Enables agents to reorganize workspace structures, relocate generated outputs, or tidy up project folders. Enabled by default. -### 14. Delete Asset (`delete_asset`) +### 15. Delete Asset (`delete_asset`) - **Description:** Allows the agent to delete a single file or folder by moving it to the Trash bin. - **Why it matters:** Provides agents with the ability to clean up outdated or redundant assets. To prevent accidental data loss, this tool is **disabled by default** and strictly limited to deleting one asset at a time. Team Owners must explicitly enable this tool for an agent. diff --git a/packages/agent/src/tools/annotation-resolver.test.ts b/packages/agent/src/annotation-resolver.test.ts similarity index 98% rename from packages/agent/src/tools/annotation-resolver.test.ts rename to packages/agent/src/annotation-resolver.test.ts index 5452944b..c6942111 100644 --- a/packages/agent/src/tools/annotation-resolver.test.ts +++ b/packages/agent/src/annotation-resolver.test.ts @@ -1,8 +1,8 @@ import { describe, it, expect, beforeEach } from 'vitest' import { prisma } from '@shumai/db' import { setupTestDbHooks } from '@shumai/db/test' -import { resolveAnnotationsById } from './annotation-resolver' -import { DatabaseSessionStorage } from '../database-session-storage' +import { resolveAnnotationsById } from '@shumai/core/src/asset-read/annotations' +import { DatabaseSessionStorage } from './database-session-storage' import type { SessionTreeEntry } from '@earendil-works/pi-agent-core' import { degreeToSecond } from '@shumai/core/src/utils/model3d' diff --git a/packages/agent/src/database-session-storage.test.ts b/packages/agent/src/database-session-storage.test.ts index 6527f726..e2e91b68 100644 --- a/packages/agent/src/database-session-storage.test.ts +++ b/packages/agent/src/database-session-storage.test.ts @@ -7,7 +7,7 @@ import { DatabaseSessionStorage } from './database-session-storage' import { agentService } from '@shumai/core/src/agent/agent' import { createAgentSession, type DbProviderInfo } from './index' import * as sandboxedBashModule from './tools/sandboxed-bash' -import * as readAssetModule from './tools/read-asset' +import * as operationToolsModule from './operation-tools' describe('DatabaseSessionStorage', () => { setupTestDbHooks() @@ -450,8 +450,8 @@ describe('DatabaseSessionStorage', () => { }) const sessionId = initialStorage.sessionId - // 2. Spy on media tool constructor - const createReadAssetToolSpy = vi.spyOn(readAssetModule, 'createReadAssetTool') + // 2. Spy on the Operation tools builder (read_asset comes from the Operation registry) + const buildOperationToolsSpy = vi.spyOn(operationToolsModule, 'buildOperationTools') // 3. Resume the session via createAgentSession with a NEW comment ID await createAgentSession({ @@ -468,8 +468,10 @@ describe('DatabaseSessionStorage', () => { providers: mockProviders, }) - // 4. Assert that the tool was instantiated using user ID - expect(createReadAssetToolSpy).toHaveBeenCalledWith(user.id) + // 4. Assert that the tools were built for the user + expect(buildOperationToolsSpy).toHaveBeenCalledWith( + expect.objectContaining({ userId: user.id }), + ) }) it('should instantiate media tools with user ID in subsequent turns (video)', async () => { @@ -502,8 +504,8 @@ describe('DatabaseSessionStorage', () => { }) const sessionId = initialStorage.sessionId - // 2. Spy on media tool constructor - const createReadAssetToolSpy = vi.spyOn(readAssetModule, 'createReadAssetTool') + // 2. Spy on the Operation tools builder (read_asset comes from the Operation registry) + const buildOperationToolsSpy = vi.spyOn(operationToolsModule, 'buildOperationTools') // 3. Resume the session via createAgentSession with a NEW comment ID await createAgentSession({ @@ -520,8 +522,10 @@ describe('DatabaseSessionStorage', () => { providers: mockProviders, }) - // 4. Assert that the tool was instantiated using user ID - expect(createReadAssetToolSpy).toHaveBeenCalledWith(user.id) + // 4. Assert that the tools were built for the user + expect(buildOperationToolsSpy).toHaveBeenCalledWith( + expect.objectContaining({ userId: user.id }), + ) }) describe('P1 Bug Reproductions', () => { diff --git a/packages/agent/src/index.ts b/packages/agent/src/index.ts index faffed13..141c6b67 100644 --- a/packages/agent/src/index.ts +++ b/packages/agent/src/index.ts @@ -21,13 +21,13 @@ import { metadataService } from '@shumai/core/src/metadata/metadata' import { createCreateFileTool } from './tools/create-file' import { createCreateVersionTool } from './tools/create-version' import { createDownloadAssetTool } from './tools/download-asset' -import { createReadAssetTool } from './tools/read-asset' import { createReadSkillTool } from './tools/read-skill' import { createReadThreadTool } from './tools/read-thread' import { createSandboxedBashTool } from './tools/sandboxed-bash' import { createGenerateImageTool } from './tools/generate-image' import { createGenerateVideoTool } from './tools/generate-video' import { buildOperationTools } from './operation-tools' +import { createAutofillReadAssetTool } from './operation-extensions' import { mediaGenerationService } from '@shumai/core/src/media-generation/media-generation' import { serializeContextToXml } from './context/serialize-context' import { DEFAULT_DENIED_TOOLS, type ShumaiMessageContext } from '@shumai/dtos' @@ -267,13 +267,12 @@ export async function createAgentSession(params: CreateAgentSessionParams) { const deniedTools = agentConfig?.deniedTools ?? [...DEFAULT_DENIED_TOOLS] const mediaTools: AgentTool[] = [] - if (userId) { - mediaTools.push(createReadAssetTool(userId)) - } else if (agent?.type === 'autofill' && assetId) { + // Agents acting for a user get read_asset / get_asset_render from the Operation registry + // (see systemTools below); the autofill system agent has its own team-scoped read_asset. + if (!userId && agent?.type === 'autofill' && assetId) { mediaTools.push( - createReadAssetTool({ + createAutofillReadAssetTool({ teamId, - agentType: 'autofill', targetAssetId: assetId, }), ) @@ -337,7 +336,8 @@ export async function createAgentSession(params: CreateAgentSessionParams) { const metadataSchema = await buildProjectMetadataSchema(projectId) const agentContext = { teamId, agentId } systemTools.push( - // Shared Operations (also exposed via MCP and the CLI): list/create/rename/move/delete + // Shared Operations (also exposed via MCP and the CLI): list/create/rename/move/delete, + // read_asset and get_asset_render ...buildOperationTools({ userId, agentId, teamId, projectId }), createCreateFileTool(userId, metadataSchema, agentContext), createCreateVersionTool(userId, metadataSchema, agentContext), diff --git a/packages/agent/src/operation-extensions/index.ts b/packages/agent/src/operation-extensions/index.ts new file mode 100644 index 00000000..a59cebd1 --- /dev/null +++ b/packages/agent/src/operation-extensions/index.ts @@ -0,0 +1,2 @@ +// Registers agent-specific presentation for Operations. Import once before building tools. +export { createAutofillReadAssetTool, presentReadOutput } from './read-asset' diff --git a/packages/agent/src/operation-extensions/read-asset.test.ts b/packages/agent/src/operation-extensions/read-asset.test.ts new file mode 100644 index 00000000..e3e3e26c --- /dev/null +++ b/packages/agent/src/operation-extensions/read-asset.test.ts @@ -0,0 +1,1058 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest' +import { + prisma, + WorkflowTaskType, + WorkflowTaskStatus, + type Asset, + type WorkflowTask, +} from '@shumai/db' +import { s3Service } from '@shumai/core/src/s3/s3' +import { workflowService } from '@shumai/workflow-core' +import { authzService } from '@shumai/core/src/authz/authz' +import { resolveAnnotationsById } from '@shumai/core/src/asset-read/annotations' +import { getOperation } from '@shumai/core/src/operations' +import { degreeToSecond } from '@shumai/core/src/utils/model3d' +import { toAgentTool } from '../operation-tools' +import { createAutofillReadAssetTool } from './read-asset' + +vi.mock('@shumai/db', async (importOriginal) => { + const actual = await importOriginal() + return { + ...actual, + prisma: { + asset: { + findUnique: vi.fn(), + findFirst: vi.fn(), + }, + workflowTask: { + create: vi.fn(), + findUnique: vi.fn(), + }, + }, + } +}) + +vi.mock('@shumai/core/src/s3/s3', async (importOriginal) => { + const actual = await importOriginal() + return { + ...actual, + s3Service: { + getObject: vi.fn(), + presign: vi.fn().mockResolvedValue('https://mock-presigned-url.com'), + }, + } +}) + +vi.mock('@shumai/workflow-core', async (importOriginal) => { + const actual = await importOriginal() + return { + ...actual, + workflowService: { + waitFor: vi.fn(), + }, + } +}) + +vi.mock('@shumai/core/src/authz/authz', async (importOriginal) => { + const actual = await importOriginal() + return { + ...actual, + authzService: { + hasPermission: vi.fn(), + }, + } +}) + +vi.mock('@shumai/core/src/asset-read/annotations', () => ({ + resolveAnnotationsById: vi.fn(), +})) + +type TextBlock = { type: 'text'; text: string } +type ImageBlock = { type: 'image'; data: string; mimeType: string } + +/** The agent's read_asset tool, as the chat agent gets it from the Operation registry. */ +function readAssetTool(userId = 'user-1') { + return toAgentTool(getOperation('read_asset')!, { userId }) +} + +function getAssetRenderTool(userId = 'user-1') { + return toAgentTool(getOperation('get_asset_render')!, { userId }) +} + +// The harness passes untyped tool details; tests look at the read_asset details shape. +type Details = { + assetId: string + name: string + key?: string + keys: string[] + sourceKeys: string[] + status?: string + taskId?: string + isTruncated?: boolean +} +function details(result: { details: unknown }): Details { + return result.details as Details +} + +function textOf(result: { content: unknown[] }): string { + return (result.content[0] as TextBlock).text +} + +function mockAsset(asset: Record) { + vi.mocked(prisma.asset.findUnique).mockResolvedValue(asset as unknown as Asset) +} + +function mockTaskCreated(id: string) { + // Echo the created row back (with an ID) so the pipeline sees the payload it wrote. + vi.mocked(prisma.workflowTask.create).mockImplementation(((args: { + data: Record + }) => + Promise.resolve({ + id, + ...args.data, + } as unknown as WorkflowTask)) as unknown as typeof prisma.workflowTask.create) +} + +function mockTaskFinished(id: string, output: Record) { + vi.mocked(workflowService.waitFor).mockResolvedValue({ + task: { id, status: WorkflowTaskStatus.completed, output } as unknown as WorkflowTask, + done: true, + }) +} + +function mockObjects() { + vi.mocked(s3Service.getObject).mockImplementation(async (_bucket, key) => ({ + buffer: Buffer.from(`bytes-for-${key}`), + contentType: 'image/webp', + })) +} + +const imageAsset = { + id: 'img-1', + name: 'photo.png', + mediaType: 'image/png', + projectId: 'project-1', + storageKey: { key: 'raw/photo.png' }, + media: { + proxyType: 'image', + imageTranscodes: [{ key: 'proxy/photo.webp' }], + }, +} + +const modelAsset = { + id: 'model-1', + name: 'robot.glb', + mediaType: 'model/gltf-binary', + projectId: 'proj-1', + storageKey: { key: 'raw/robot.glb' }, + media: { + proxyType: '3d', + videoTranscodes: [{ key: 'proxy/turntable.mp4' }], + imageTranscodes: [{ key: 'proxy/views.webp' }], + }, +} + +const videoAsset = { + id: 'vid-1', + name: 'video.mp4', + mediaType: 'video/mp4', + projectId: 'project-1', + media: { proxyType: 'video' }, +} + +const pdfAsset = { + id: 'doc-1', + name: 'doc.pdf', + mediaType: 'application/pdf', + projectId: 'project-1', + media: { proxyType: 'pdf' }, +} + +const markdownAsset = { + id: 'doc-md', + name: 'README.md', + mediaType: 'text/markdown', + storageKey: { key: 'raw/README.md' }, + media: { proxyType: 'pdf' }, +} + +const arrow = [ + { + type: 'arrow', + color: '#ff0000', + points: [ + [0, 0], + [10, 10], + ], + }, +] as unknown as PrismaJson.AnnotationList + +describe('read_asset agent tool', () => { + beforeEach(() => { + vi.clearAllMocks() + vi.mocked(authzService.hasPermission).mockResolvedValue() + vi.mocked(resolveAnnotationsById).mockResolvedValue({ annotations: null, timestamp: null }) + }) + + it('exposes the agent-only s3KeyOnly parameter alongside the Operation input', () => { + const tool = readAssetTool() + const properties = (tool.parameters as { properties: Record }).properties + expect(Object.keys(properties)).toEqual( + expect.arrayContaining([ + 'assetId', + 'annotationId', + 'config3d', + 'imageConfig', + 'videoConfig', + 'docConfig', + 'maxWaitSeconds', + 's3KeyOnly', + ]), + ) + }) + + it('throws an authorization error if the user ID is empty', async () => { + await expect(readAssetTool('').execute('call-1', { assetId: 'asset-1' })).rejects.toThrow( + 'User ID is required for authorization.', + ) + }) + + it('throws if the asset is not found', async () => { + vi.mocked(prisma.asset.findUnique).mockResolvedValue(null) + await expect( + readAssetTool().execute('call-1', { + assetId: 'asset-1', + annotationId: null, + imageConfig: null, + videoConfig: null, + docConfig: null, + }), + ).rejects.toThrow('Asset with ID asset-1 not found.') + }) + + it('does not plan or render anything when Read is denied', async () => { + vi.mocked(authzService.hasPermission).mockRejectedValue(new Error('Forbidden')) + await expect( + readAssetTool().execute('call-1', { + assetId: 'vid-1', + videoConfig: { start: 0, end: 1, count: 1 }, + }), + ).rejects.toThrow('Forbidden') + expect(prisma.asset.findUnique).not.toHaveBeenCalled() + expect(prisma.workflowTask.create).not.toHaveBeenCalled() + }) + + it('rejects unsupported asset types', async () => { + mockAsset({ id: 'zip-1', name: 'bundle.zip', mediaType: 'application/zip', media: null }) + await expect(readAssetTool().execute('call-1', { assetId: 'zip-1' })).rejects.toThrow( + 'Unsupported asset type for read_asset. Asset zip-1 has media type "application/zip".', + ) + }) + + describe('Image Assets', () => { + it('reads the image proxy and returns an image block and S3 key', async () => { + mockAsset(imageAsset) + mockObjects() + + const result = await readAssetTool().execute('call-1', { + assetId: 'img-1', + annotationId: null, + imageConfig: null, + videoConfig: null, + docConfig: null, + }) + + expect(authzService.hasPermission).toHaveBeenCalledWith({ + user: { id: 'user-1' }, + permission: 'Read', + type: 'asset', + id: 'img-1', + }) + expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'proxy/photo.webp') + + expect(textOf(result)).toBe( + 'Image asset "photo.png" (ID: img-1):\n- Image (S3 Key: "proxy/photo.webp", MIME: image/webp)', + ) + const image = result.content[1] as ImageBlock + expect(image.type).toBe('image') + expect(image.data).toBe(Buffer.from('bytes-for-proxy/photo.webp').toString('base64')) + expect(image.mimeType).toBe('image/webp') + expect(details(result)).toMatchObject({ + assetId: 'img-1', + name: 'photo.png', + key: 'proxy/photo.webp', + keys: ['proxy/photo.webp'], + sourceKeys: ['proxy/photo.webp'], + }) + }) + + it('ignores an empty imageConfig', async () => { + mockAsset(imageAsset) + mockObjects() + const result = await readAssetTool().execute('call-1', { assetId: 'img-1', imageConfig: {} }) + expect(details(result).key).toBe('proxy/photo.webp') + }) + + it('rejects a config meant for another asset type', async () => { + mockAsset(imageAsset) + await expect( + readAssetTool().execute('call-1', { + assetId: 'img-1', + docConfig: { mode: 'pages', startPage: 1, endPage: 1 }, + }), + ).rejects.toThrow( + "Asset img-1 is a image. Pass 'imageConfig: {}' or no config; do not pass docConfig.", + ) + }) + + it('renders the annotation overlay in a workflow task when annotationId is provided', async () => { + mockAsset(imageAsset) + vi.mocked(resolveAnnotationsById).mockResolvedValue({ annotations: arrow, timestamp: null }) + mockTaskCreated('task-anno') + mockTaskFinished('task-anno', { key: 'annotated/photo.webp' }) + mockObjects() + + const result = await readAssetTool().execute('call-1', { + assetId: 'img-1', + annotationId: 'comment-1', + }) + + expect(resolveAnnotationsById).toHaveBeenCalledWith('img-1', 'comment-1') + expect(prisma.workflowTask.create).toHaveBeenCalledWith({ + data: { + assetId: 'img-1', + projectId: 'project-1', + type: WorkflowTaskType.transcode_image_annotation, + status: WorkflowTaskStatus.pending, + payload: { + projectId: 'project-1', + imageAnnotation: { annotations: arrow }, + assetRead: { + reader: 'image', + annotationId: 'comment-1', + requestedBy: 'user-1', + }, + }, + }, + }) + expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'annotated/photo.webp') + expect(textOf(result)).toBe( + 'Image asset "photo.png" (ID: img-1) with visual markups overlaid:\n- Image with markups (S3 Key: "annotated/photo.webp", MIME: image/webp)', + ) + expect(details(result).key).toBe('annotated/photo.webp') + }) + }) + + describe('3D Assets', () => { + it('returns the 4-view composite with the quadrant legend', async () => { + mockAsset(modelAsset) + mockObjects() + + const result = await readAssetTool().execute('call-3d', { assetId: 'model-1' }) + + expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'proxy/views.webp') + const text = textOf(result) + expect(text).toContain('3D asset "robot.glb" (ID: model-1):') + expect(text).toContain('Orthogonal 4-view composite image') + expect(text).toContain('Top-Left quadrant: Front view (0°)') + expect(text).toContain('Top-Right quadrant: Right side view (90°)') + expect(text).toContain('Bottom-Left quadrant: Back view (180°)') + expect(text).toContain('Bottom-Right quadrant: Left side view (270°)') + expect(text).toContain('- 4-view composite (S3 Key: "proxy/views.webp", MIME: image/webp)') + expect((result.content[1] as ImageBlock).data).toBe( + Buffer.from('bytes-for-proxy/views.webp').toString('base64'), + ) + expect(details(result).key).toBe('proxy/views.webp') + }) + + it('supports s3KeyOnly for 3D assets', async () => { + mockAsset(modelAsset) + const result = await readAssetTool().execute('call-3d-s3', { + assetId: 'model-1', + s3KeyOnly: true, + }) + expect(s3Service.getObject).not.toHaveBeenCalled() + expect(result.content).toHaveLength(1) + expect(textOf(result)).toContain('proxy/views.webp') + expect(textOf(result)).toContain('Front view (0°)') + expect(details(result).key).toBe('proxy/views.webp') + }) + + it('rejects videoConfig for 3D assets', async () => { + mockAsset(modelAsset) + await expect( + readAssetTool().execute('call-3d-err', { + assetId: 'model-1', + videoConfig: { start: 0, end: 4, count: 4 }, + }), + ).rejects.toThrow( + "To inspect specific angles pass 'config3d: { degree: ... }'; do not pass videoConfig", + ) + }) + + it('throws if the 4-views image is missing when the overview is requested', async () => { + mockAsset({ ...modelAsset, media: { proxyType: '3d', imageTranscodes: [] } }) + await expect(readAssetTool().execute('call-3d-err2', { assetId: 'model-1' })).rejects.toThrow( + '4-views image (views.webp) is missing', + ) + }) + + it('renders a turntable frame when config3d.degree is provided', async () => { + mockAsset(modelAsset) + mockTaskCreated('task-shot-3d') + mockTaskFinished('task-shot-3d', { + screenshots: [{ key: 'proxy/shot-45.webp', timestamp: 0.5833 }], + }) + mockObjects() + + const result = await readAssetTool().execute('call-3d-degree', { + assetId: 'model-1', + config3d: { degree: 45 }, + }) + + const second = degreeToSecond(45) + expect(prisma.workflowTask.create).toHaveBeenCalledWith({ + data: expect.objectContaining({ + type: WorkflowTaskType.transcode_screenshot, + assetId: 'model-1', + payload: { + projectId: 'proj-1', + screenshot: { + start: second, + end: second, + count: 1, + commentTimestamp: second, + annotations: undefined, + }, + assetRead: { + reader: 'model3d', + options: { degree: 45 }, + context: { degree: 45, hasMarkup: false }, + requestedBy: 'user-1', + }, + }, + }), + }) + expect(textOf(result)).toBe( + '3D asset "robot.glb" (ID: model-1) view at 45°:\n- View at 45° (S3 Key: "proxy/shot-45.webp", MIME: image/webp)', + ) + expect((result.content[1] as ImageBlock).data).toBe( + Buffer.from('bytes-for-proxy/shot-45.webp').toString('base64'), + ) + expect(details(result).key).toBe('proxy/shot-45.webp') + }) + + it('rejects degrees that are not a multiple of 15 or out of range', async () => { + mockAsset(modelAsset) + const tool = readAssetTool() + await expect( + tool.execute('call-1', { assetId: 'model-1', config3d: { degree: 40 } }), + ).rejects.toThrow('Invalid turntable degree: 40') + await expect( + tool.execute('call-2', { assetId: 'model-1', config3d: { degree: 360 } }), + ).rejects.toThrow('Invalid turntable degree: 360') + expect(resolveAnnotationsById).not.toHaveBeenCalled() + }) + + it('renders the frame with markups at the annotation angle', async () => { + mockAsset(modelAsset) + vi.mocked(resolveAnnotationsById).mockResolvedValue({ annotations: arrow, timestamp: 0.5 }) + mockTaskCreated('task-shot-ann') + mockTaskFinished('task-shot-ann', { + screenshots: [{ key: 'proxy/shot-annotated.webp', timestamp: 0.5 }], + }) + mockObjects() + + const result = await readAssetTool().execute('call-3d-ann', { + assetId: 'model-1', + annotationId: 'comm-1', + config3d: null, + }) + + expect(prisma.workflowTask.create).toHaveBeenCalledWith({ + data: expect.objectContaining({ + type: WorkflowTaskType.transcode_screenshot, + payload: expect.objectContaining({ + screenshot: { + start: 0.5, + end: 0.5, + count: 1, + commentTimestamp: 0.5, + annotations: arrow, + }, + }), + }), + }) + expect(textOf(result)).toContain('view at 45° with visual markups overlaid:') + expect(details(result).key).toBe('proxy/shot-annotated.webp') + }) + + it('renders session annotations at their resolved degree', async () => { + mockAsset(modelAsset) + const expectedTime = degreeToSecond(90) + vi.mocked(resolveAnnotationsById).mockResolvedValue({ + annotations: arrow, + timestamp: expectedTime, + }) + mockTaskCreated('task-shot-3d-session') + mockTaskFinished('task-shot-3d-session', { + screenshots: [{ key: 'proxy/shot-annotated-90.webp', timestamp: expectedTime }], + }) + mockObjects() + + const result = await readAssetTool().execute('call-3d-session-ann', { + assetId: 'model-1', + annotationId: 'session-entry-1', + }) + + expect(prisma.workflowTask.create).toHaveBeenCalledWith({ + data: expect.objectContaining({ + payload: expect.objectContaining({ + screenshot: expect.objectContaining({ + start: expectedTime, + end: expectedTime, + commentTimestamp: expectedTime, + }), + }), + }), + }) + expect(textOf(result)).toContain('view at 90° with visual markups overlaid:') + expect(details(result).key).toBe('proxy/shot-annotated-90.webp') + }) + + it('rejects a degree that conflicts with the annotation angle', async () => { + mockAsset(modelAsset) + vi.mocked(resolveAnnotationsById).mockResolvedValue({ annotations: arrow, timestamp: 0.5 }) + await expect( + readAssetTool().execute('call-3d-conflict', { + assetId: 'model-1', + annotationId: 'comm-1', + config3d: { degree: 90 }, + }), + ).rejects.toThrow( + 'Annotation "comm-1" was drawn at 45°, but degree 90° was specified. Do not pass a conflicting degree with annotationId.', + ) + expect(prisma.workflowTask.create).not.toHaveBeenCalled() + }) + + it('throws if the turntable video is missing when an angle is requested', async () => { + mockAsset({ ...modelAsset, media: { proxyType: '3d', videoTranscodes: [] } }) + await expect( + readAssetTool().execute('call-1', { assetId: 'model-1', config3d: { degree: 45 } }), + ).rejects.toThrow('turntable video (turntable.mp4) is missing') + }) + }) + + describe('Video Assets', () => { + it('requires videoConfig', async () => { + mockAsset(videoAsset) + await expect(readAssetTool().execute('call-1', { assetId: 'vid-1' })).rejects.toThrow( + 'videoConfig with start, end, and count is required', + ) + }) + + it('rejects an invalid time range', async () => { + mockAsset(videoAsset) + await expect( + readAssetTool().execute('call-1', { + assetId: 'vid-1', + videoConfig: { start: 10, end: 5, count: 1 }, + }), + ).rejects.toThrow( + 'Invalid video time range: start (10) must be less than or equal to end (5).', + ) + }) + + it('rejects more than 20 frames', async () => { + mockAsset(videoAsset) + await expect( + readAssetTool().execute('call-1', { + assetId: 'vid-1', + videoConfig: { start: 0, end: 60, count: 21 }, + }), + ).rejects.toThrow('exceeds the maximum of 20 frames per request') + }) + + it('extracts frames and returns image blocks and keys', async () => { + mockAsset(videoAsset) + mockTaskCreated('task-shot') + mockTaskFinished('task-shot', { + screenshots: [ + { key: 'screenshots/shot1.webp', timestamp: 2.0 }, + { key: 'screenshots/shot2.webp', timestamp: 4.0 }, + ], + }) + mockObjects() + + const result = await readAssetTool().execute('call-1', { + assetId: 'vid-1', + annotationId: null, + imageConfig: null, + videoConfig: { start: 2.0, end: 4.0, count: 2 }, + docConfig: null, + }) + + expect(prisma.workflowTask.create).toHaveBeenCalledWith({ + data: { + assetId: 'vid-1', + projectId: 'project-1', + type: WorkflowTaskType.transcode_screenshot, + status: WorkflowTaskStatus.pending, + payload: { + projectId: 'project-1', + screenshot: { + start: 2.0, + end: 4.0, + count: 2, + commentTimestamp: null, + annotations: null, + }, + assetRead: { + reader: 'video', + options: { start: 2.0, end: 4.0, count: 2 }, + requestedBy: 'user-1', + }, + }, + }, + }) + expect(workflowService.waitFor).toHaveBeenCalledWith( + expect.objectContaining({ id: 'task-shot' }), + 60_000, + ) + expect(result.content.length).toBe(3) // 1 text + 2 images + expect(textOf(result)).toBe( + [ + 'Extracted 2 frame(s) from "video.mp4" (2s - 4s):', + '- Frame at 2s (S3 Key: "screenshots/shot1.webp", MIME: image/webp)', + '- Frame at 4s (S3 Key: "screenshots/shot2.webp", MIME: image/webp)', + ].join('\n'), + ) + expect(details(result).keys).toEqual(['screenshots/shot1.webp', 'screenshots/shot2.webp']) + }) + + it('returns pending with the task ID when rendering outlasts the wait budget', async () => { + mockAsset(videoAsset) + mockTaskCreated('task-slow') + vi.mocked(workflowService.waitFor).mockImplementation(async (task) => ({ + task, + done: false, + })) + + const result = await readAssetTool().execute('call-1', { + assetId: 'vid-1', + videoConfig: { start: 0, end: 10, count: 5 }, + maxWaitSeconds: 0, + }) + + expect(workflowService.waitFor).toHaveBeenCalledWith( + expect.objectContaining({ id: 'task-slow' }), + 0, + ) + expect(result.content).toHaveLength(1) + expect(textOf(result)).toBe( + 'Rendering is still in progress for "video.mp4". Call get_asset_render with taskId "task-slow" in about 5 seconds to get the result.', + ) + expect(details(result)).toEqual({ taskId: 'task-slow', status: 'pending' }) + }) + + it('never waits longer than the agent budget', async () => { + mockAsset(videoAsset) + mockTaskCreated('task-shot') + mockTaskFinished('task-shot', { screenshots: [{ key: 's/1.webp', timestamp: 0 }] }) + mockObjects() + + await readAssetTool().execute('call-1', { + assetId: 'vid-1', + videoConfig: { start: 0, end: 0, count: 1 }, + maxWaitSeconds: 600, + }) + expect(workflowService.waitFor).toHaveBeenCalledWith(expect.anything(), 60_000) + }) + }) + + describe('Document Assets', () => { + it('requires docConfig', async () => { + mockAsset(pdfAsset) + await expect(readAssetTool().execute('call-1', { assetId: 'doc-1' })).rejects.toThrow( + 'docConfig with mode ("pages" or "text") is required', + ) + }) + + it('renders PDF pages in mode "pages"', async () => { + mockAsset(pdfAsset) + mockTaskCreated('task-pdf') + mockTaskFinished('task-pdf', { + pages: [ + { key: 'pdf_pages/page1.webp', page: 1 }, + { key: 'pdf_pages/page2.webp', page: 2 }, + ], + }) + mockObjects() + + const result = await readAssetTool().execute('call-1', { + assetId: 'doc-1', + docConfig: { mode: 'pages', startPage: 1, endPage: 2 }, + }) + + expect(prisma.workflowTask.create).toHaveBeenCalledWith({ + data: expect.objectContaining({ + type: WorkflowTaskType.transcode_pdf_pages, + payload: expect.objectContaining({ + pdfPages: { start: 1, end: 2, commentTimestamp: null, annotations: null }, + }), + }), + }) + expect(result.content.length).toBe(3) // 1 text + 2 images + expect(textOf(result)).toBe( + [ + 'Rendered 2 page(s) of "doc.pdf" (Pages 1 to 2):', + '- Page 1 (S3 Key: "pdf_pages/page1.webp", MIME: image/webp)', + '- Page 2 (S3 Key: "pdf_pages/page2.webp", MIME: image/webp)', + ].join('\n'), + ) + expect(details(result).keys).toEqual(['pdf_pages/page1.webp', 'pdf_pages/page2.webp']) + }) + + it('rejects page ranges over 20 pages', async () => { + mockAsset(pdfAsset) + await expect( + readAssetTool().execute('call-1', { + assetId: 'doc-1', + docConfig: { mode: 'pages', startPage: 1, endPage: 25 }, + }), + ).rejects.toThrow('exceeds the maximum limit of 20 pages') + }) + + it('rejects mode "text" on a binary PDF', async () => { + mockAsset({ ...pdfAsset, name: 'manual.pdf' }) + await expect( + readAssetTool().execute('call-1', { + assetId: 'doc-1', + docConfig: { mode: 'text', startPage: null, endPage: null }, + }), + ).rejects.toThrow('is a binary PDF/document and cannot be read as raw text') + }) + + it('rejects mode "text" with annotationId', async () => { + mockAsset(markdownAsset) + await expect( + readAssetTool().execute('call-1', { + assetId: 'doc-md', + annotationId: 'comment-123', + docConfig: { mode: 'text', startPage: null, endPage: null }, + }), + ).rejects.toThrow('annotationId cannot be used with docConfig mode "text"') + expect(resolveAnnotationsById).not.toHaveBeenCalled() + }) + + it('rejects page numbers in mode "text"', async () => { + mockAsset(markdownAsset) + await expect( + readAssetTool().execute('call-1', { + assetId: 'doc-md', + docConfig: { mode: 'text', startPage: 1, endPage: 2 }, + }), + ).rejects.toThrow('startPage and endPage must be null when docConfig mode is "text"') + }) + + it('reads raw text in mode "text"', async () => { + mockAsset(markdownAsset) + vi.mocked(s3Service.getObject).mockResolvedValue({ + buffer: Buffer.from('# Hello Shumai\nThis is markdown text.'), + contentType: 'text/markdown', + }) + + const result = await readAssetTool().execute('call-1', { + assetId: 'doc-md', + docConfig: { mode: 'text', startPage: null, endPage: null }, + }) + + expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'raw/README.md') + expect(result.content).toHaveLength(1) + expect(textOf(result)).toBe( + [ + 'Document "README.md" (ID: doc-md):', + '- Source file (S3 Key: "raw/README.md", MIME: text/markdown)', + '', + '# Hello Shumai\nThis is markdown text.', + ].join('\n'), + ) + expect(details(result)).toMatchObject({ + key: 'raw/README.md', + keys: [], + sourceKeys: ['raw/README.md'], + isTruncated: false, + }) + }) + + it('truncates long text and tells the agent how to get the rest', async () => { + mockAsset(markdownAsset) + const lines = Array.from({ length: 1500 }, (_, i) => `line ${i + 1}`).join('\n') + vi.mocked(s3Service.getObject).mockResolvedValue({ + buffer: Buffer.from(lines), + contentType: 'text/markdown', + }) + + const result = await readAssetTool().execute('call-1', { + assetId: 'doc-md', + docConfig: { mode: 'text' }, + }) + + const text = textOf(result) + expect(text).toContain('line 1000') + expect(text).not.toContain('line 1001') + expect(text).toMatch( + /\[Content truncated to 1000 lines \/ 50KB limit\. Use download_asset to process the full file\.\]$/, + ) + expect(details(result).isTruncated).toBe(true) + }) + }) + + describe('Version Stack Resolution', () => { + it('reads the latest version of a version stack', async () => { + mockAsset({ id: 'stack-1', name: '', type: 'version_stack' }) + vi.mocked(prisma.asset.findFirst).mockResolvedValue({ + id: 'file-v2', + name: 'diagram_v2.png', + type: 'file', + mediaType: 'image/png', + storageKey: { key: 'raw/diagram_v2.png' }, + media: { + proxyType: 'image', + imageTranscodes: [{ key: 'proxy/diagram_v2.webp' }], + }, + } as unknown as Asset) + mockObjects() + + const result = await readAssetTool().execute('call-1', { assetId: 'stack-1' }) + + expect(prisma.asset.findFirst).toHaveBeenCalledWith({ + where: { parentId: 'stack-1', isDeleted: false }, + orderBy: { sortIndex: 'asc' }, + include: { storageKey: true }, + }) + expect(details(result).name).toBe('diagram_v2.png') + expect(details(result).key).toBe('proxy/diagram_v2.webp') + }) + }) + + describe('s3KeyOnly mode', () => { + it('returns only the S3 key and MIME for image assets', async () => { + mockAsset(imageAsset) + const result = await readAssetTool().execute('call-1', { + assetId: 'img-1', + s3KeyOnly: true, + }) + + expect(s3Service.getObject).not.toHaveBeenCalled() + expect(result.content).toHaveLength(1) + expect(textOf(result)).toBe( + 'Image asset "photo.png" (ID: img-1):\n- Image (S3 Key: "proxy/photo.webp", MIME: image/webp)', + ) + expect(details(result).key).toBe('proxy/photo.webp') + }) + + it('returns only frame S3 keys and MIME for video assets', async () => { + mockAsset({ ...videoAsset, name: 'clip.mp4' }) + mockTaskCreated('task-shot') + mockTaskFinished('task-shot', { + screenshots: [{ key: 'screenshots/shot1.webp', timestamp: 1.5 }], + }) + + const result = await readAssetTool().execute('call-1', { + assetId: 'vid-1', + s3KeyOnly: true, + videoConfig: { start: 1.5, end: 1.5, count: 1 }, + }) + + expect(s3Service.getObject).not.toHaveBeenCalled() + expect(result.content).toHaveLength(1) + expect(textOf(result)).toContain( + '- Frame at 1.5s (S3 Key: "screenshots/shot1.webp", MIME: image/webp)', + ) + expect(details(result).keys).toEqual(['screenshots/shot1.webp']) + }) + + it('returns only the document S3 key and MIME without reading the text', async () => { + mockAsset({ ...markdownAsset, name: 'notes.md', storageKey: { key: 'raw/notes.md' } }) + const result = await readAssetTool().execute('call-1', { + assetId: 'doc-md', + s3KeyOnly: true, + docConfig: { mode: 'text', startPage: null, endPage: null }, + }) + + expect(s3Service.getObject).not.toHaveBeenCalled() + expect(result.content).toHaveLength(1) + expect(textOf(result)).toBe( + 'Document "notes.md" (ID: doc-md):\n- Source file (S3 Key: "raw/notes.md", MIME: text/markdown)', + ) + }) + }) +}) + +describe('get_asset_render agent tool', () => { + beforeEach(() => { + vi.clearAllMocks() + vi.mocked(authzService.hasPermission).mockResolvedValue() + }) + + function mockPendingTask(requestedBy: string) { + vi.mocked(prisma.workflowTask.findUnique).mockResolvedValue({ + id: 'task-1', + assetId: 'vid-1', + status: WorkflowTaskStatus.processing, + payload: { + projectId: 'project-1', + assetRead: { reader: 'video', options: { start: 0, end: 4, count: 2 }, requestedBy }, + }, + } as unknown as WorkflowTask) + mockAsset(videoAsset) + } + + it('returns the frames once the render has finished', async () => { + mockPendingTask('user-1') + mockTaskFinished('task-1', { + screenshots: [ + { key: 'screenshots/a.webp', timestamp: 0 }, + { key: 'screenshots/b.webp', timestamp: 4 }, + ], + }) + mockObjects() + + const result = await getAssetRenderTool().execute('call-1', { taskId: 'task-1' }) + + expect(authzService.hasPermission).toHaveBeenCalledWith({ + user: { id: 'user-1' }, + permission: 'Read', + type: 'asset', + id: 'vid-1', + }) + expect(result.content).toHaveLength(3) + expect(textOf(result)).toContain('Extracted 2 frame(s) from "video.mp4" (0s - 4s):') + expect(details(result).keys).toEqual(['screenshots/a.webp', 'screenshots/b.webp']) + }) + + it('returns pending again while the render is still running', async () => { + mockPendingTask('user-1') + vi.mocked(workflowService.waitFor).mockImplementation(async (task) => ({ task, done: false })) + + const result = await getAssetRenderTool().execute('call-1', { taskId: 'task-1' }) + expect(textOf(result)).toContain('Call get_asset_render with taskId "task-1"') + expect(details(result)).toEqual({ taskId: 'task-1', status: 'pending' }) + }) + + it('hides renders requested by another user', async () => { + mockPendingTask('someone-else') + await expect(getAssetRenderTool().execute('call-1', { taskId: 'task-1' })).rejects.toThrow( + 'Render not found', + ) + expect(workflowService.waitFor).not.toHaveBeenCalled() + }) + + it('hides renders once the user loses access to the asset', async () => { + mockPendingTask('user-1') + vi.mocked(authzService.hasPermission).mockRejectedValue(new Error('Forbidden')) + await expect(getAssetRenderTool().execute('call-1', { taskId: 'task-1' })).rejects.toThrow( + 'Render not found', + ) + }) + + it('hides workflow tasks that are not asset reads', async () => { + vi.mocked(prisma.workflowTask.findUnique).mockResolvedValue({ + id: 'task-2', + assetId: 'vid-1', + payload: { projectId: 'project-1' }, + } as unknown as WorkflowTask) + await expect(getAssetRenderTool().execute('call-1', { taskId: 'task-2' })).rejects.toThrow( + 'Render not found', + ) + }) +}) + +describe('autofill read_asset tool', () => { + beforeEach(() => { + vi.clearAllMocks() + }) + + it('requires targetAssetId', async () => { + const tool = createAutofillReadAssetTool({ teamId: 'team-1' }) + await expect(tool.execute('call-1', { assetId: 'asset-1' })).rejects.toThrow( + 'Target asset ID is required for autofill agent authorization.', + ) + }) + + it('requires teamId', async () => { + const tool = createAutofillReadAssetTool({ targetAssetId: 'asset-1' }) + await expect(tool.execute('call-1', { assetId: 'asset-1' })).rejects.toThrow( + 'Team ID is required for autofill agent authorization.', + ) + }) + + it('denies assets from another team', async () => { + mockAsset({ id: 'asset-1', project: { teamId: 'other-team' } }) + const tool = createAutofillReadAssetTool({ teamId: 'team-1', targetAssetId: 'asset-1' }) + await expect(tool.execute('call-1', { assetId: 'asset-1' })).rejects.toThrow( + 'Access denied: asset asset-1 does not belong to team team-1.', + ) + }) + + it('denies assets other than the target asset', async () => { + mockAsset({ id: 'other-asset', parentId: null, project: { teamId: 'team-1' } }) + const tool = createAutofillReadAssetTool({ + teamId: 'team-1', + targetAssetId: 'target-asset-id', + }) + await expect(tool.execute('call-1', { assetId: 'other-asset' })).rejects.toThrow( + 'Access denied: autofill agent can only read target asset target-asset-id.', + ) + }) + + it('reads the target asset without a user', async () => { + mockAsset({ + id: 'target-asset-id', + name: 'test.png', + mediaType: 'image/png', + storageKey: { key: 'raw/test.png' }, + media: { proxyType: 'image' }, + project: { teamId: 'team-1' }, + }) + const tool = createAutofillReadAssetTool({ + teamId: 'team-1', + targetAssetId: 'target-asset-id', + }) + + const result = await tool.execute('call-1', { + assetId: 'target-asset-id', + annotationId: null, + s3KeyOnly: true, + imageConfig: null, + }) + + expect(authzService.hasPermission).not.toHaveBeenCalled() + expect(result.content).toHaveLength(1) + expect(textOf(result)).toBe( + 'Image asset "test.png" (ID: target-asset-id):\n- Image (S3 Key: "raw/test.png", MIME: image/png)', + ) + }) + + it('waits up to two minutes for renders', async () => { + mockAsset({ ...videoAsset, project: { teamId: 'team-1' } }) + mockTaskCreated('task-shot') + mockTaskFinished('task-shot', { screenshots: [{ key: 's/1.webp', timestamp: 0 }] }) + + const tool = createAutofillReadAssetTool({ teamId: 'team-1', targetAssetId: 'vid-1' }) + await tool.execute('call-1', { + assetId: 'vid-1', + s3KeyOnly: true, + videoConfig: { start: 0, end: 0, count: 1 }, + }) + expect(workflowService.waitFor).toHaveBeenCalledWith(expect.anything(), 120_000) + expect(prisma.workflowTask.create).toHaveBeenCalledWith({ + data: expect.objectContaining({ + payload: expect.objectContaining({ + assetRead: expect.objectContaining({ reader: 'video', requestedBy: undefined }), + }), + }), + }) + }) +}) diff --git a/packages/agent/src/operation-extensions/read-asset.ts b/packages/agent/src/operation-extensions/read-asset.ts new file mode 100644 index 00000000..d1968988 --- /dev/null +++ b/packages/agent/src/operation-extensions/read-asset.ts @@ -0,0 +1,155 @@ +import { z } from 'zod' +import { type AgentTool, type AgentToolResult } from '@earendil-works/pi-agent-core' +import { type ImageContent, type TextContent } from '@earendil-works/pi-ai' +import { readAsset, type AssetReadOutput } from '@shumai/operations' +import { assetReadService, toAssetReadOutput } from '@shumai/core/src/asset-read' +import { getOperation, READ_WAIT_BUDGET_MS } from '@shumai/core/src/operations' +import { s3Service } from '@shumai/core/src/s3/s3' +import { getFileMimeType } from '@shumai/core/src/utils/file-mime' +import { MAX_TEXT_BYTES, MAX_TEXT_LINES } from '@shumai/core/src/asset-read' +import { + registerAgentToolExtension, + splitAgentParams, + toToolParameters, + type AgentToolExtension, +} from '../operation-tools' + +const extraParameters = z.object({ + s3KeyOnly: z + .boolean() + .optional() + .describe( + 'When true, returns only the storage S3 key(s) and MIME types of the asset/frames without attaching base64 image or text content to the agent context. Use this to obtain S3 keys for generate_image or generate_video references without consuming context tokens.', + ), +}) + +/** Details kept compatible with the previous tool; `sourceKeys` drives session image storage. */ +interface ReadAssetDetails { + assetId: string + name: string + mediaType: string | null + key?: string + keys: string[] + downloadUrl?: string + results: Array<{ key: string; label: string; downloadUrl?: string }> + sourceKeys: string[] + size?: number + isTruncated?: boolean +} + +/** Render a read_asset / get_asset_render output as agent content (text plus image blocks). */ +export async function presentReadOutput( + output: AssetReadOutput, + opts: { s3KeyOnly: boolean }, +): Promise> { + if (output.status === 'pending') { + const retry = output.retryAfterSeconds ?? 5 + return { + content: [ + { + type: 'text', + text: `${output.summary} Call get_asset_render with taskId "${output.taskId}" in about ${retry} seconds to get the result.`, + }, + ], + details: { taskId: output.taskId, status: 'pending' }, + } + } + + const bucket = process.env.S3_BUCKET || 'shumai' + const imageBlocks: ImageContent[] = [] + const imageLines: string[] = [] + for (const image of output.images) { + const key = image.key ?? '' + let mimeType = image.mimeType + if (!opts.s3KeyOnly && key) { + const { buffer, contentType } = await s3Service.getObject(bucket, key) + mimeType = + contentType && contentType !== 'application/octet-stream' + ? contentType + : getFileMimeType(buffer, key, image.mimeType) + imageBlocks.push({ type: 'image', data: buffer.toString('base64'), mimeType }) + } + imageLines.push(`- ${image.label} (S3 Key: "${key}", MIME: ${mimeType})`) + } + + const lines = [output.summary, ...output.details, ...imageLines] + if (output.text) { + lines.push(`- Source file (S3 Key: "${output.text.key ?? ''}", MIME: ${output.text.mimeType})`) + } + let text = lines.join('\n') + if (output.text && !opts.s3KeyOnly) { + text += `\n\n${output.text.content}` + if (output.text.truncated) { + text += `\n\n[Content truncated to ${MAX_TEXT_LINES} lines / ${MAX_TEXT_BYTES / 1024}KB limit. Use download_asset to process the full file.]` + } + } + + const imageKeys = output.images.map((i) => i.key ?? '') + const content: Array = [{ type: 'text', text }, ...imageBlocks] + return { + content, + details: { + assetId: output.asset.id, + name: output.asset.name, + mediaType: output.asset.mediaType, + key: imageKeys[0] ?? output.text?.key, + keys: imageKeys, + downloadUrl: output.images[0]?.url ?? output.text?.url, + results: output.images.map((i) => ({ key: i.key ?? '', label: i.label, downloadUrl: i.url })), + sourceKeys: output.text?.key && imageKeys.length === 0 ? [output.text.key] : imageKeys, + ...(output.text ? { size: output.text.size, isTruncated: output.text.truncated } : {}), + }, + } +} + +const readAssetExtension: AgentToolExtension = { + extraParameters, + // s3KeyOnly also means "don't read text content", as before. + toInput: (input, extra) => (extra.s3KeyOnly ? { ...input, includeContent: false } : input), + present: (output, extra) => + presentReadOutput(output as AssetReadOutput, { s3KeyOnly: extra.s3KeyOnly === true }), +} + +const getAssetRenderExtension: AgentToolExtension = { + present: (output) => presentReadOutput(output as AssetReadOutput, { s3KeyOnly: false }), +} + +registerAgentToolExtension('read_asset', readAssetExtension) +registerAgentToolExtension('get_asset_render', getAssetRenderExtension) + +/** Longer than the chat budget: autofill can't poll get_asset_render (it has no user). */ +const AUTOFILL_WAIT_BUDGET_MS = 2 * READ_WAIT_BUDGET_MS.agent + +/** + * `read_asset` for the autofill system agent, which runs without a user. It uses the same + * pipeline and presentation, but access is confined to the agent's team and target asset. + */ +export function createAutofillReadAssetTool(access: { + teamId?: string + targetAssetId?: string +}): AgentTool { + const op = getOperation(readAsset.name)! + return { + name: readAsset.name, + label: readAsset.title, + description: readAsset.description, + parameters: toToolParameters(op, readAssetExtension), + execute: async (_toolCallId, params) => { + const { input, extra } = splitAgentParams(op, readAssetExtension, params) + const parsed = readAsset.input.safeParse(input) + if (!parsed.success) { + throw new Error(`Invalid input for read_asset: ${z.prettifyError(parsed.error)}`) + } + const { maxWaitSeconds, ...request } = parsed.data + const budget = + maxWaitSeconds === undefined + ? AUTOFILL_WAIT_BUDGET_MS + : Math.min(AUTOFILL_WAIT_BUDGET_MS, maxWaitSeconds * 1000) + const outcome = await assetReadService.read({ kind: 'autofill', ...access }, request, { + waitBudgetMs: budget, + }) + const output = await toAssetReadOutput(outcome, { includeKeys: true }) + return presentReadOutput(output, { s3KeyOnly: extra.s3KeyOnly === true }) + }, + } +} diff --git a/packages/agent/src/operation-tools.test.ts b/packages/agent/src/operation-tools.test.ts index 03d1e0f3..615ba4fa 100644 --- a/packages/agent/src/operation-tools.test.ts +++ b/packages/agent/src/operation-tools.test.ts @@ -61,7 +61,7 @@ describe('operation-backed agent tools', () => { }) describe('tool definitions', () => { - it('keeps the existing tool names and adds list_projects', () => { + it('keeps the existing tool names and adds list_projects and get_asset_render', () => { expect( buildOperationTools(ctx) .map((t) => t.name) @@ -69,9 +69,11 @@ describe('operation-backed agent tools', () => { ).toEqual([ 'create_folder', 'delete_asset', + 'get_asset_render', 'list_assets', 'list_projects', 'move_assets', + 'read_asset', 'rename_asset', ]) }) diff --git a/packages/agent/src/operation-tools.ts b/packages/agent/src/operation-tools.ts index 70fc57da..4555ac90 100644 --- a/packages/agent/src/operation-tools.ts +++ b/packages/agent/src/operation-tools.ts @@ -1,7 +1,7 @@ import { z } from 'zod' import type { TSchema } from 'typebox' import { Compile } from 'typebox/compile' -import { type AgentTool } from '@earendil-works/pi-agent-core' +import { type AgentTool, type AgentToolResult } from '@earendil-works/pi-agent-core' import { getOperationsForSurface, runOperation, @@ -12,13 +12,41 @@ import { /** Context for agent-invoked Operations; `via` is always `agent`. */ export type AgentOperationContext = Omit +/** + * Agent-specific presentation for an Operation. By default a tool returns the Operation output + * as JSON text; an extension can add agent-only parameters, map them into the Operation input, + * and render the output (e.g. as image blocks). + */ +export interface AgentToolExtension { + /** Extra parameters only the agent sees; they are removed before the Operation runs. */ + extraParameters?: z.ZodObject + /** Adjust the Operation input using the extra parameters. */ + toInput?(input: Record, extra: Record): Record + present(output: unknown, extra: Record): Promise> +} + +const extensions = new Map() + +/** Register agent presentation for an Operation (see `./operation-extensions`). */ +export function registerAgentToolExtension(name: string, extension: AgentToolExtension): void { + extensions.set(name, extension) +} + +function toolInputSchema(op: Operation, extension?: AgentToolExtension): z.ZodObject { + return extension?.extraParameters + ? op.definition.input.extend(extension.extraParameters.shape) + : op.definition.input +} + /** * Convert an Operation's Zod input schema into the JSON Schema the agent harness expects. * TypeBox schemas are plain JSON Schema at runtime, so the converted schema is used as-is. * Throws when the schema cannot be compiled, so a bad definition fails loudly in tests. */ -export function toToolParameters(op: Operation): TSchema { - const schema: Record = z.toJSONSchema(op.definition.input, { io: 'input' }) +export function toToolParameters(op: Operation, extension?: AgentToolExtension): TSchema { + const schema: Record = z.toJSONSchema(toolInputSchema(op, extension), { + io: 'input', + }) // `$schema` is not understood by the harness validator or every model provider. delete schema.$schema Compile(schema) @@ -27,11 +55,11 @@ export function toToolParameters(op: Operation): TSchema { /** * Models sometimes send `null` for optional parameters they don't want to set. Drop those - * so the Operation's input schema treats them as omitted; nulls for required fields still fail. + * so the input schema treats them as omitted; nulls for required fields still fail. */ -function dropNullOptionalFields(op: Operation, params: unknown): unknown { +export function dropNullOptionalFields(schema: z.ZodObject, params: unknown): unknown { if (!params || typeof params !== 'object' || Array.isArray(params)) return params - const shape = op.definition.input.shape + const shape = schema.shape return Object.fromEntries( Object.entries(params).filter( ([key, value]) => !(value === null && shape[key]?.safeParse(undefined).success), @@ -39,20 +67,40 @@ function dropNullOptionalFields(op: Operation, params: unknown): unknown { ) } +/** Split agent-only parameters from the Operation input. */ +export function splitAgentParams( + op: Operation, + extension: AgentToolExtension | undefined, + params: unknown, +): { input: unknown; extra: Record } { + const cleaned = dropNullOptionalFields(toolInputSchema(op, extension), params) + if (!extension?.extraParameters || !cleaned || typeof cleaned !== 'object') { + return { input: cleaned, extra: {} } + } + const extraKeys = new Set(Object.keys(extension.extraParameters.shape)) + const input: Record = {} + const extra: Record = {} + for (const [key, value] of Object.entries(cleaned)) { + ;(extraKeys.has(key) ? extra : input)[key] = value + } + return { input: extension.toInput ? extension.toInput(input, extra) : input, extra } +} + /** Expose an Operation as a native agent tool that runs through `runOperation`. */ export function toAgentTool(op: Operation, ctx: AgentOperationContext): AgentTool { const { definition } = op + const extension = extensions.get(definition.name) return { name: definition.name, label: definition.title, description: definition.description, - parameters: toToolParameters(op), + parameters: toToolParameters(op, extension), execute: async (_toolCallId, params) => { - const output = await runOperation( - op, - { ...ctx, via: 'agent' }, - dropNullOptionalFields(op, params), - ) + const { input, extra } = splitAgentParams(op, extension, params) + const output = await runOperation(op, { ...ctx, via: 'agent' }, input) + if (extension) { + return extension.present(output, extra) + } return { content: [{ type: 'text', text: JSON.stringify(output, null, 2) }], details: {}, diff --git a/packages/agent/src/tools/read-asset.test.ts b/packages/agent/src/tools/read-asset.test.ts deleted file mode 100644 index 1073e640..00000000 --- a/packages/agent/src/tools/read-asset.test.ts +++ /dev/null @@ -1,1246 +0,0 @@ -import { describe, it, expect, vi, beforeEach } from 'vitest' -import { createReadAssetTool, type ReadAssetAuthContext } from './read-asset' -import { - prisma, - WorkflowTaskType, - WorkflowTaskStatus, - type Asset, - type WorkflowTask, -} from '@shumai/db' -import { s3Service } from '@shumai/core/src/s3/s3' -import { workflowService } from '@shumai/workflow-core' -import { authzService } from '@shumai/core/src/authz/authz' -import * as annotationResolver from './annotation-resolver' -import { degreeToSecond } from '@shumai/core/src/utils/model3d' - -vi.mock('@shumai/db', async (importOriginal) => { - const actual = await importOriginal() - return { - ...actual, - prisma: { - asset: { - findUnique: vi.fn(), - findFirst: vi.fn(), - }, - workflowTask: { - create: vi.fn(), - }, - }, - } -}) - -vi.mock('@shumai/core/src/s3/s3', () => ({ - s3Service: { - getObject: vi.fn(), - presign: vi.fn().mockResolvedValue('https://mock-presigned-url.com'), - }, -})) - -vi.mock('@shumai/workflow-core', () => ({ - workflowService: { - executeWait: vi.fn(), - }, -})) - -vi.mock('@shumai/core/src/authz/authz', () => ({ - authzService: { - hasPermission: vi.fn(), - }, - Permission: { Read: 'Read' }, - ResourceType: { Asset: 'asset' }, -})) - -describe('readAssetTool', () => { - beforeEach(() => { - vi.clearAllMocks() - vi.spyOn(annotationResolver, 'resolveAnnotationsById').mockResolvedValue({ - annotations: null, - timestamp: null, - }) - }) - - it('should throw authorization error if user ID is empty', async () => { - const tool = createReadAssetTool('') - await expect( - tool.execute('call-1', { - assetId: 'asset-1', - annotationId: null, - imageConfig: null, - videoConfig: null, - docConfig: null, - }), - ).rejects.toThrow('User ID is required for authorization.') - }) - - it('should throw error if asset not found', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue(null) - - const tool = createReadAssetTool('user-1') - await expect( - tool.execute('call-1', { - assetId: 'asset-1', - annotationId: null, - imageConfig: null, - videoConfig: null, - docConfig: null, - }), - ).rejects.toThrow('Asset with ID asset-1 not found.') - }) - - describe('Image Assets', () => { - it('should read image and return ImageContent and S3 key', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'img-1', - name: 'photo.png', - mediaType: 'image/png', - storageKey: { key: 'raw/photo.png' }, - media: { - proxyType: 'image', - imageTranscodes: [{ key: 'proxy/photo.webp' }], - }, - } as unknown as Asset) - - vi.mocked(s3Service.getObject).mockResolvedValue({ - buffer: Buffer.from('fake-image-bytes'), - contentType: 'image/webp', - } as unknown as { buffer: Buffer; contentType: string }) - - const tool = createReadAssetTool('user-1') - const result = await tool.execute('call-1', { - assetId: 'img-1', - annotationId: null, - imageConfig: null, - videoConfig: null, - docConfig: null, - }) - - expect(authzService.hasPermission).toHaveBeenCalledWith({ - user: { id: 'user-1' }, - permission: 'Read', - type: 'asset', - id: 'img-1', - }) - expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'proxy/photo.webp') - - const textItem = result.content[0] as { type: 'text'; text: string } - const imageItem = result.content[1] as { type: 'image'; data: string; mimeType: string } - - expect(textItem.text).toContain('proxy/photo.webp') - expect(imageItem.type).toBe('image') - expect(imageItem.data).toBe(Buffer.from('fake-image-bytes').toString('base64')) - expect(result.details.key).toBe('proxy/photo.webp') - }) - - it('should trigger image annotation transcode workflow when annotationId is provided', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'img-1', - name: 'photo.png', - mediaType: 'image/png', - projectId: 'project-1', - storageKey: { key: 'raw/photo.png' }, - media: { - proxyType: 'image', - imageTranscodes: [{ key: 'proxy/photo.webp' }], - }, - } as unknown as Asset) - - vi.spyOn(annotationResolver, 'resolveAnnotationsById').mockResolvedValue({ - annotations: [ - { - type: 'arrow', - color: '#ff0000', - points: [ - [0, 0], - [10, 10], - ], - }, - ], - timestamp: null, - }) - - vi.mocked(prisma.workflowTask.create).mockResolvedValue({ - id: 'task-anno', - } as unknown as WorkflowTask) - vi.mocked(workflowService.executeWait).mockResolvedValue({ - id: 'task-anno', - output: { key: 'annotated/photo.webp' }, - } as unknown as WorkflowTask) - - vi.mocked(s3Service.getObject).mockResolvedValue({ - buffer: Buffer.from('annotated-bytes'), - contentType: 'image/webp', - } as unknown as { buffer: Buffer; contentType: string }) - - const tool = createReadAssetTool('user-1') - const result = await tool.execute('call-1', { - assetId: 'img-1', - annotationId: 'comment-1', - imageConfig: null, - videoConfig: null, - docConfig: null, - }) - - expect(prisma.workflowTask.create).toHaveBeenCalledWith({ - data: { - assetId: 'img-1', - projectId: 'project-1', - type: WorkflowTaskType.transcode_image_annotation, - status: WorkflowTaskStatus.pending, - payload: { - projectId: 'project-1', - imageAnnotation: { - annotations: [ - { - type: 'arrow', - color: '#ff0000', - points: [ - [0, 0], - [10, 10], - ], - }, - ], - }, - }, - }, - }) - - expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'annotated/photo.webp') - expect(result.details.key).toBe('annotated/photo.webp') - }) - }) - - describe('3D Assets', () => { - it('should read 3D asset 4-views image and return image with quadrant layout description', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'model-1', - name: 'robot.glb', - mediaType: 'model/gltf-binary', - storageKey: { key: 'raw/robot.glb' }, - media: { - proxyType: '3d', - imageTranscodes: [{ key: 'proxy/views.webp' }], - }, - } as unknown as Asset) - - vi.mocked(s3Service.getObject).mockResolvedValue({ - buffer: Buffer.from('fake-views-bytes'), - contentType: 'image/webp', - } as unknown as { buffer: Buffer; contentType: string }) - - const tool = createReadAssetTool('user-1') - const result = await tool.execute('call-3d', { - assetId: 'model-1', - annotationId: null, - imageConfig: null, - videoConfig: null, - docConfig: null, - }) - - expect(authzService.hasPermission).toHaveBeenCalledWith({ - user: { id: 'user-1' }, - permission: 'Read', - type: 'asset', - id: 'model-1', - }) - expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'proxy/views.webp') - - const textItem = result.content[0] as { type: 'text'; text: string } - const imageItem = result.content[1] as { type: 'image'; data: string; mimeType: string } - - expect(textItem.text).toContain('Orthogonal 4-view composite image') - expect(textItem.text).toContain('Top-Left quadrant: Front view (0°)') - expect(textItem.text).toContain('Top-Right quadrant: Right side view (90°)') - expect(textItem.text).toContain('Bottom-Left quadrant: Back view (180°)') - expect(textItem.text).toContain('Bottom-Right quadrant: Left side view (270°)') - expect(imageItem.type).toBe('image') - expect(imageItem.data).toBe(Buffer.from('fake-views-bytes').toString('base64')) - expect(result.details.key).toBe('proxy/views.webp') - }) - - it('should support s3KeyOnly mode for 3D assets', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'model-1', - name: 'robot.glb', - mediaType: 'model/gltf-binary', - media: { - proxyType: '3d', - imageTranscodes: [{ key: 'proxy/views.webp' }], - }, - } as unknown as Asset) - - const tool = createReadAssetTool('user-1') - const result = await tool.execute('call-3d-s3', { - assetId: 'model-1', - annotationId: null, - s3KeyOnly: true, - imageConfig: null, - videoConfig: null, - docConfig: null, - }) - - expect(result.content).toHaveLength(1) - const textItem = result.content[0] as { type: 'text'; text: string } - expect(textItem.text).toContain('proxy/views.webp') - expect(textItem.text).toContain('Front view (0°)') - expect(result.details.key).toBe('proxy/views.webp') - }) - - it('should throw error if videoConfig is passed for 3D assets', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'model-1', - name: 'robot.glb', - mediaType: 'model/gltf-binary', - media: { - proxyType: '3d', - imageTranscodes: [{ key: 'proxy/views.webp' }], - }, - } as unknown as Asset) - - const tool = createReadAssetTool('user-1') - await expect( - tool.execute('call-3d-err', { - assetId: 'model-1', - annotationId: null, - imageConfig: null, - videoConfig: { start: 0, end: 4, count: 4 }, - docConfig: null, - }), - ).rejects.toThrow( - "To inspect specific angles pass 'config3d: { degree: ... }'; do not pass videoConfig", - ) - }) - - it('should throw error if 3D asset imageTranscodes is missing when overview is requested', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'model-1', - name: 'robot.glb', - mediaType: 'model/gltf-binary', - media: { - proxyType: '3d', - imageTranscodes: [], - }, - } as unknown as Asset) - - const tool = createReadAssetTool('user-1') - await expect( - tool.execute('call-3d-err2', { - assetId: 'model-1', - annotationId: null, - imageConfig: null, - videoConfig: null, - docConfig: null, - }), - ).rejects.toThrow('4-views image (views.webp) is missing') - }) - - it('should extract specific turntable frame when config3d.degree is provided', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'model-1', - name: 'robot.glb', - mediaType: 'model/gltf-binary', - projectId: 'proj-1', - media: { - proxyType: '3d', - videoTranscodes: [{ key: 'proxy/turntable.mp4' }], - imageTranscodes: [{ key: 'proxy/views.webp' }], - }, - } as unknown as Asset) - - vi.mocked(prisma.workflowTask.create).mockResolvedValue({ - id: 'task-shot-3d', - } as unknown as WorkflowTask) - - vi.mocked(workflowService.executeWait).mockResolvedValue({ - id: 'task-shot-3d', - status: WorkflowTaskStatus.completed, - output: { - screenshots: [{ key: 'proxy/shot-45.webp', timestamp: 0.5833 }], - }, - } as unknown as WorkflowTask) - - vi.mocked(s3Service.getObject).mockResolvedValue({ - buffer: Buffer.from('fake-shot-45-bytes'), - contentType: 'image/webp', - } as unknown as { buffer: Buffer; contentType: string }) - - const tool = createReadAssetTool('user-1') - const result = await tool.execute('call-3d-degree', { - assetId: 'model-1', - annotationId: null, - config3d: { degree: 45 }, - imageConfig: null, - videoConfig: null, - docConfig: null, - }) - - expect(prisma.workflowTask.create).toHaveBeenCalledWith({ - data: expect.objectContaining({ - type: WorkflowTaskType.transcode_screenshot, - assetId: 'model-1', - payload: { - projectId: 'proj-1', - screenshot: { - start: expect.any(Number), - end: expect.any(Number), - count: 1, - commentTimestamp: expect.any(Number), - annotations: undefined, - }, - }, - }), - }) - - const textItem = result.content[0] as { type: 'text'; text: string } - const imageItem = result.content[1] as { type: 'image'; data: string; mimeType: string } - - expect(textItem.text).toContain('view at 45°:') - expect(imageItem.data).toBe(Buffer.from('fake-shot-45-bytes').toString('base64')) - expect(result.details.key).toBe('proxy/shot-45.webp') - }) - - it('should throw error when config3d.degree is not a multiple of 15 or out of range', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'model-1', - name: 'robot.glb', - mediaType: 'model/gltf-binary', - media: { - proxyType: '3d', - videoTranscodes: [{ key: 'proxy/turntable.mp4' }], - }, - } as unknown as Asset) - - const tool = createReadAssetTool('user-1') - - await expect( - tool.execute('call-3d-invalid-deg', { - assetId: 'model-1', - annotationId: null, - config3d: { degree: 40 }, - imageConfig: null, - videoConfig: null, - docConfig: null, - }), - ).rejects.toThrow('Invalid turntable degree: 40') - - await expect( - tool.execute('call-3d-out-of-range', { - assetId: 'model-1', - annotationId: null, - config3d: { degree: 360 }, - imageConfig: null, - videoConfig: null, - docConfig: null, - }), - ).rejects.toThrow('Invalid turntable degree: 360') - }) - - it('should extract frame with overlaid markups when annotationId is provided on 3D asset', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'model-1', - name: 'robot.glb', - mediaType: 'model/gltf-binary', - projectId: 'proj-1', - media: { - proxyType: '3d', - videoTranscodes: [{ key: 'proxy/turntable.mp4' }], - }, - } as unknown as Asset) - - vi.spyOn(annotationResolver, 'resolveAnnotationsById').mockResolvedValue({ - annotations: [ - { - type: 'freehand', - color: '#ff0000', - points: [ - [0, 0], - [10, 10], - ], - }, - ], - timestamp: 0.5, // 0.5s = 45° - }) - - vi.mocked(prisma.workflowTask.create).mockResolvedValue({ - id: 'task-shot-ann', - } as unknown as WorkflowTask) - - vi.mocked(workflowService.executeWait).mockResolvedValue({ - id: 'task-shot-ann', - status: WorkflowTaskStatus.completed, - output: { - screenshots: [{ key: 'proxy/shot-annotated.webp', timestamp: 0.5 }], - }, - } as unknown as WorkflowTask) - - vi.mocked(s3Service.getObject).mockResolvedValue({ - buffer: Buffer.from('fake-shot-ann-bytes'), - contentType: 'image/webp', - } as unknown as { buffer: Buffer; contentType: string }) - - const tool = createReadAssetTool('user-1') - const result = await tool.execute('call-3d-ann', { - assetId: 'model-1', - annotationId: 'comm-1', - config3d: null, - imageConfig: null, - videoConfig: null, - docConfig: null, - }) - - expect(prisma.workflowTask.create).toHaveBeenCalledWith({ - data: expect.objectContaining({ - type: WorkflowTaskType.transcode_screenshot, - assetId: 'model-1', - payload: { - projectId: 'proj-1', - screenshot: { - start: 0.5, - end: 0.5, - count: 1, - commentTimestamp: 0.5, - annotations: [ - { - type: 'freehand', - color: '#ff0000', - points: [ - [0, 0], - [10, 10], - ], - }, - ], - }, - }, - }), - }) - - const textItem = result.content[0] as { type: 'text'; text: string } - expect(textItem.text).toContain('view at 45° with visual markups overlaid:') - expect(result.details.key).toBe('proxy/shot-annotated.webp') - }) - - it('should extract turntable frame at resolved degree angle for 3D session annotations', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'model-1', - name: 'robot.glb', - mediaType: 'model/gltf-binary', - projectId: 'proj-1', - media: { - proxyType: '3d', - videoTranscodes: [{ key: 'proxy/turntable.mp4' }], - }, - } as unknown as Asset) - - const expectedTime = degreeToSecond(90) - vi.spyOn(annotationResolver, 'resolveAnnotationsById').mockResolvedValue({ - annotations: [ - { - type: 'box', - color: '#00ff00', - points: [ - [10, 20], - [40, 60], - ], - }, - ], - timestamp: expectedTime, - }) - - vi.mocked(prisma.workflowTask.create).mockResolvedValue({ - id: 'task-shot-3d-session', - } as unknown as WorkflowTask) - - vi.mocked(workflowService.executeWait).mockResolvedValue({ - id: 'task-shot-3d-session', - status: WorkflowTaskStatus.completed, - output: { - screenshots: [{ key: 'proxy/shot-annotated-90.webp', timestamp: expectedTime }], - }, - } as unknown as WorkflowTask) - - vi.mocked(s3Service.getObject).mockResolvedValue({ - buffer: Buffer.from('fake-shot-90-bytes'), - contentType: 'image/webp', - } as unknown as { buffer: Buffer; contentType: string }) - - const tool = createReadAssetTool('user-1') - const result = await tool.execute('call-3d-session-ann', { - assetId: 'model-1', - annotationId: 'session-entry-1', - config3d: null, - imageConfig: null, - videoConfig: null, - docConfig: null, - }) - - expect(prisma.workflowTask.create).toHaveBeenCalledWith({ - data: expect.objectContaining({ - type: WorkflowTaskType.transcode_screenshot, - assetId: 'model-1', - payload: { - projectId: 'proj-1', - screenshot: { - start: expectedTime, - end: expectedTime, - count: 1, - commentTimestamp: expectedTime, - annotations: [ - { - type: 'box', - color: '#00ff00', - points: [ - [10, 20], - [40, 60], - ], - }, - ], - }, - }, - }), - }) - - const textItem = result.content[0] as { type: 'text'; text: string } - expect(textItem.text).toContain('view at 90° with visual markups overlaid:') - expect(result.details.key).toBe('proxy/shot-annotated-90.webp') - }) - - it('should throw error if config3d.degree conflicts with annotation angle', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'model-1', - name: 'robot.glb', - mediaType: 'model/gltf-binary', - projectId: 'proj-1', - media: { - proxyType: '3d', - videoTranscodes: [{ key: 'proxy/turntable.mp4' }], - }, - } as unknown as Asset) - - vi.spyOn(annotationResolver, 'resolveAnnotationsById').mockResolvedValue({ - annotations: [ - { - type: 'box', - color: '#ff0000', - points: [ - [0, 0], - [10, 10], - ], - }, - ], - timestamp: 0.5, // 45° - }) - - const tool = createReadAssetTool('user-1') - await expect( - tool.execute('call-3d-conflict', { - assetId: 'model-1', - annotationId: 'comm-1', - config3d: { degree: 90 }, // Conflicts with 45° - imageConfig: null, - videoConfig: null, - docConfig: null, - }), - ).rejects.toThrow( - 'Annotation "comm-1" was drawn at 45°, but degree 90° was specified. Do not pass a conflicting degree with annotationId.', - ) - }) - - it('should throw error if 3D asset turntable video is missing when angle inspection is requested', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'model-1', - name: 'robot.glb', - mediaType: 'model/gltf-binary', - media: { - proxyType: '3d', - videoTranscodes: [], - }, - } as unknown as Asset) - - const tool = createReadAssetTool('user-1') - await expect( - tool.execute('call-3d-missing-turntable', { - assetId: 'model-1', - annotationId: null, - config3d: { degree: 45 }, - imageConfig: null, - videoConfig: null, - docConfig: null, - }), - ).rejects.toThrow('turntable video (turntable.mp4) is missing') - }) - }) - - describe('Video Assets', () => { - it('should throw error if videoConfig is missing for video assets', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'vid-1', - name: 'video.mp4', - mediaType: 'video/mp4', - media: { proxyType: 'video' }, - } as unknown as Asset) - - const tool = createReadAssetTool('user-1') - await expect( - tool.execute('call-1', { - assetId: 'vid-1', - annotationId: null, - imageConfig: null, - videoConfig: null, - docConfig: null, - }), - ).rejects.toThrow('videoConfig with start, end, and count is required') - }) - - it('should throw error on invalid video time range', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'vid-1', - name: 'video.mp4', - mediaType: 'video/mp4', - media: { proxyType: 'video' }, - } as unknown as Asset) - - const tool = createReadAssetTool('user-1') - await expect( - tool.execute('call-1', { - assetId: 'vid-1', - annotationId: null, - imageConfig: null, - videoConfig: { start: 10, end: 5, count: 1 }, - docConfig: null, - }), - ).rejects.toThrow( - 'Invalid video time range: start (10) must be less than or equal to end (5).', - ) - }) - - it('should extract frames and return ImageContent array and keys', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'vid-1', - name: 'video.mp4', - mediaType: 'video/mp4', - projectId: 'project-1', - media: { proxyType: 'video' }, - } as unknown as Asset) - - vi.mocked(prisma.workflowTask.create).mockResolvedValue({ - id: 'task-shot', - } as unknown as WorkflowTask) - vi.mocked(workflowService.executeWait).mockResolvedValue({ - id: 'task-shot', - output: { - screenshots: [ - { key: 'screenshots/shot1.webp', timestamp: 2.0 }, - { key: 'screenshots/shot2.webp', timestamp: 4.0 }, - ], - }, - } as unknown as WorkflowTask) - - vi.mocked(s3Service.getObject).mockImplementation(async (_bucket, key) => ({ - buffer: Buffer.from(`bytes-for-${key}`), - contentType: 'image/webp', - })) - - const tool = createReadAssetTool('user-1') - const result = await tool.execute('call-1', { - assetId: 'vid-1', - annotationId: null, - imageConfig: null, - videoConfig: { start: 2.0, end: 4.0, count: 2 }, - docConfig: null, - }) - - expect(prisma.workflowTask.create).toHaveBeenCalledWith({ - data: { - assetId: 'vid-1', - projectId: 'project-1', - type: WorkflowTaskType.transcode_screenshot, - status: WorkflowTaskStatus.pending, - payload: { - projectId: 'project-1', - screenshot: { - start: 2.0, - end: 4.0, - count: 2, - commentTimestamp: null, - annotations: null, - }, - }, - }, - }) - - expect(result.content.length).toBe(3) // 1 text + 2 images - expect(result.details.keys).toEqual(['screenshots/shot1.webp', 'screenshots/shot2.webp']) - }) - }) - - describe('Document Assets', () => { - it('should throw error if docConfig is missing for document assets', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'doc-1', - name: 'doc.pdf', - mediaType: 'application/pdf', - media: { proxyType: 'pdf' }, - } as unknown as Asset) - - const tool = createReadAssetTool('user-1') - await expect( - tool.execute('call-1', { - assetId: 'doc-1', - annotationId: null, - imageConfig: null, - videoConfig: null, - docConfig: null, - }), - ).rejects.toThrow('docConfig with mode ("pages" or "text") is required') - }) - - it('should render PDF pages in mode: "pages"', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'doc-1', - name: 'doc.pdf', - mediaType: 'application/pdf', - projectId: 'project-1', - media: { proxyType: 'pdf' }, - } as unknown as Asset) - - vi.mocked(prisma.workflowTask.create).mockResolvedValue({ - id: 'task-pdf', - } as unknown as WorkflowTask) - vi.mocked(workflowService.executeWait).mockResolvedValue({ - id: 'task-pdf', - output: { - pages: [ - { key: 'pdf_pages/page1.webp', page: 1 }, - { key: 'pdf_pages/page2.webp', page: 2 }, - ], - }, - } as unknown as WorkflowTask) - - vi.mocked(s3Service.getObject).mockImplementation(async (_bucket, key) => ({ - buffer: Buffer.from(`bytes-for-${key}`), - contentType: 'image/webp', - })) - - const tool = createReadAssetTool('user-1') - const result = await tool.execute('call-1', { - assetId: 'doc-1', - annotationId: null, - imageConfig: null, - videoConfig: null, - docConfig: { mode: 'pages', startPage: 1, endPage: 2 }, - }) - - expect(result.content.length).toBe(3) // 1 text + 2 images - expect(result.details.keys).toEqual(['pdf_pages/page1.webp', 'pdf_pages/page2.webp']) - }) - - it('should throw error in mode: "pages" when page range exceeds 20 pages', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'doc-1', - name: 'doc.pdf', - mediaType: 'application/pdf', - media: { proxyType: 'pdf' }, - } as unknown as Asset) - - const tool = createReadAssetTool('user-1') - await expect( - tool.execute('call-1', { - assetId: 'doc-1', - annotationId: null, - imageConfig: null, - videoConfig: null, - docConfig: { mode: 'pages', startPage: 1, endPage: 25 }, - }), - ).rejects.toThrow('exceeds the maximum limit of 20 pages') - }) - - it('should throw error in mode: "text" when called on binary PDF', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'doc-1', - name: 'manual.pdf', - mediaType: 'application/pdf', - media: { proxyType: 'pdf' }, - } as unknown as Asset) - - const tool = createReadAssetTool('user-1') - await expect( - tool.execute('call-1', { - assetId: 'doc-1', - annotationId: null, - imageConfig: null, - videoConfig: null, - docConfig: { mode: 'text', startPage: null, endPage: null }, - }), - ).rejects.toThrow('is a binary PDF/document and cannot be read as raw text') - }) - - it('should throw error in mode: "text" when annotationId is provided', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'doc-1', - name: 'notes.md', - mediaType: 'text/markdown', - media: { proxyType: 'pdf' }, - } as unknown as Asset) - - const tool = createReadAssetTool('user-1') - await expect( - tool.execute('call-1', { - assetId: 'doc-1', - annotationId: 'comment-123', - imageConfig: null, - videoConfig: null, - docConfig: { mode: 'text', startPage: null, endPage: null }, - }), - ).rejects.toThrow('annotationId cannot be used with docConfig mode "text"') - }) - - it('should throw error in mode: "text" when startPage or endPage is non-null', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'doc-1', - name: 'notes.md', - mediaType: 'text/markdown', - media: { proxyType: 'pdf' }, - } as unknown as Asset) - - const tool = createReadAssetTool('user-1') - await expect( - tool.execute('call-1', { - assetId: 'doc-1', - annotationId: null, - imageConfig: null, - videoConfig: null, - docConfig: { mode: 'text', startPage: 1, endPage: 2 }, - }), - ).rejects.toThrow('startPage and endPage must be null when docConfig mode is "text"') - }) - - it('should read raw text from S3 in mode: "text" for markdown files', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'doc-md', - name: 'README.md', - mediaType: 'text/markdown', - storageKey: { key: 'raw/README.md' }, - media: { proxyType: 'pdf' }, - } as unknown as Asset) - - vi.mocked(s3Service.getObject).mockResolvedValue({ - buffer: Buffer.from('# Hello Shumai\nThis is markdown text.'), - contentType: 'text/markdown', - } as unknown as { buffer: Buffer; contentType: string }) - - const tool = createReadAssetTool('user-1') - const result = await tool.execute('call-1', { - assetId: 'doc-md', - annotationId: null, - imageConfig: null, - videoConfig: null, - docConfig: { mode: 'text', startPage: null, endPage: null }, - }) - - expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'raw/README.md') - expect(result.content[0].type).toBe('text') - expect((result.content[0] as { type: 'text'; text: string }).text).toContain( - '# Hello Shumai\nThis is markdown text.', - ) - expect(result.details.key).toBe('raw/README.md') - }) - }) - - describe('Version Stack Resolution', () => { - it('should resolve version_stack to latest version asset', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'stack-1', - name: '', - type: 'version_stack', - } as unknown as Asset) - - vi.mocked(prisma.asset.findFirst).mockResolvedValue({ - id: 'file-v2', - name: 'diagram_v2.png', - type: 'file', - mediaType: 'image/png', - storageKey: { key: 'raw/diagram_v2.png' }, - media: { - proxyType: 'image', - imageTranscodes: [{ key: 'proxy/diagram_v2.webp' }], - }, - } as unknown as Asset) - - vi.mocked(s3Service.getObject).mockResolvedValue({ - buffer: Buffer.from('v2-image-bytes'), - contentType: 'image/webp', - } as unknown as { buffer: Buffer; contentType: string }) - - const tool = createReadAssetTool('user-1') - const result = await tool.execute('call-1', { - assetId: 'stack-1', - annotationId: null, - imageConfig: null, - videoConfig: null, - docConfig: null, - }) - - expect(prisma.asset.findFirst).toHaveBeenCalledWith({ - where: { parentId: 'stack-1', isDeleted: false }, - orderBy: { sortIndex: 'asc' }, - include: { storageKey: true }, - }) - expect(result.details.name).toBe('diagram_v2.png') - expect(result.details.key).toBe('proxy/diagram_v2.webp') - }) - }) - - describe('s3KeyOnly mode', () => { - it('returns only S3 key and MIME in text for image assets without fetching S3 buffer or attaching base64 image', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'img-1', - name: 'photo.png', - mediaType: 'image/png', - storageKey: { key: 'raw/photo.png' }, - media: { - proxyType: 'image', - imageTranscodes: [{ key: 'proxy/photo.webp' }], - }, - } as unknown as Asset) - - const tool = createReadAssetTool('user-1') - const result = await tool.execute('call-1', { - assetId: 'img-1', - annotationId: null, - s3KeyOnly: true, - imageConfig: null, - videoConfig: null, - docConfig: null, - }) - - expect(s3Service.getObject).not.toHaveBeenCalled() - expect(result.content.length).toBe(1) - expect(result.content[0].type).toBe('text') - expect((result.content[0] as { type: 'text'; text: string }).text).toBe( - 'Image asset "photo.png" (ID: img-1, MIME: image/webp, S3 Key: "proxy/photo.webp")', - ) - expect(result.details.key).toBe('proxy/photo.webp') - }) - - it('returns only frame S3 keys and MIME in text for video assets without fetching S3 buffers or attaching base64 images', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'vid-1', - name: 'clip.mp4', - mediaType: 'video/mp4', - projectId: 'project-1', - media: { proxyType: 'video' }, - } as unknown as Asset) - - vi.mocked(prisma.workflowTask.create).mockResolvedValue({ - id: 'task-shot', - } as unknown as WorkflowTask) - vi.mocked(workflowService.executeWait).mockResolvedValue({ - id: 'task-shot', - output: { - screenshots: [{ key: 'screenshots/shot1.webp', timestamp: 1.5 }], - }, - } as unknown as WorkflowTask) - - const tool = createReadAssetTool('user-1') - const result = await tool.execute('call-1', { - assetId: 'vid-1', - annotationId: null, - s3KeyOnly: true, - imageConfig: null, - videoConfig: { start: 1.5, end: 1.5, count: 1 }, - docConfig: null, - }) - - expect(s3Service.getObject).not.toHaveBeenCalled() - expect(result.content.length).toBe(1) - expect(result.content[0].type).toBe('text') - const text = (result.content[0] as { type: 'text'; text: string }).text - expect(text).toContain('- Frame at 1.5s (S3 Key: "screenshots/shot1.webp", MIME: image/webp)') - expect(result.details.keys).toEqual(['screenshots/shot1.webp']) - }) - - it('returns only document S3 key and MIME in text for text mode without fetching S3 buffer', async () => { - vi.mocked(authzService.hasPermission).mockResolvedValue() - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'doc-md', - name: 'notes.md', - mediaType: 'text/markdown', - storageKey: { key: 'raw/notes.md' }, - media: { proxyType: 'pdf' }, - } as unknown as Asset) - - const tool = createReadAssetTool('user-1') - const result = await tool.execute('call-1', { - assetId: 'doc-md', - annotationId: null, - s3KeyOnly: true, - imageConfig: null, - videoConfig: null, - docConfig: { mode: 'text', startPage: null, endPage: null }, - }) - - expect(s3Service.getObject).not.toHaveBeenCalled() - expect(result.content.length).toBe(1) - expect(result.content[0].type).toBe('text') - expect((result.content[0] as { type: 'text'; text: string }).text).toBe( - 'Document "notes.md" (ID: doc-md, MIME: text/markdown, S3 Key: "raw/notes.md")', - ) - }) - }) - - describe('ReadAssetAuthContext & Security Boundaries', () => { - it('throws authorization error if chat agent lacks userId', async () => { - const tool = createReadAssetTool({ - agentType: 'chat', - teamId: 'team-1', - }) - await expect( - tool.execute('call-1', { - assetId: 'asset-1', - annotationId: null, - imageConfig: null, - videoConfig: null, - docConfig: null, - }), - ).rejects.toThrow('User ID is required for authorization.') - }) - - it('throws error if autofill agent lacks targetAssetId', async () => { - const tool = createReadAssetTool({ - agentType: 'autofill', - teamId: 'team-1', - } as unknown as ReadAssetAuthContext) - - await expect( - tool.execute('call-1', { - assetId: 'asset-1', - annotationId: null, - imageConfig: null, - videoConfig: null, - docConfig: null, - }), - ).rejects.toThrow('Target asset ID is required for autofill agent authorization.') - }) - - it('throws error if autofill agent lacks teamId', async () => { - const tool = createReadAssetTool({ - agentType: 'autofill', - targetAssetId: 'asset-1', - } as unknown as ReadAssetAuthContext) - - await expect( - tool.execute('call-1', { - assetId: 'asset-1', - annotationId: null, - imageConfig: null, - videoConfig: null, - docConfig: null, - }), - ).rejects.toThrow('Team ID is required for autofill agent authorization.') - }) - - it('denies autofill agent if asset belongs to another team', async () => { - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'asset-1', - project: { teamId: 'other-team' }, - } as unknown as Asset) - - const tool = createReadAssetTool({ - agentType: 'autofill', - teamId: 'team-1', - targetAssetId: 'asset-1', - }) - - await expect( - tool.execute('call-1', { - assetId: 'asset-1', - annotationId: null, - imageConfig: null, - videoConfig: null, - docConfig: null, - }), - ).rejects.toThrow('Access denied: asset asset-1 does not belong to team team-1.') - }) - - it('denies autofill agent if reading an unrelated asset not matching targetAssetId', async () => { - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'other-asset', - parentId: null, - project: { teamId: 'team-1' }, - } as unknown as Asset) - - const tool = createReadAssetTool({ - agentType: 'autofill', - teamId: 'team-1', - targetAssetId: 'target-asset-id', - }) - - await expect( - tool.execute('call-1', { - assetId: 'other-asset', - annotationId: null, - imageConfig: null, - videoConfig: null, - docConfig: null, - }), - ).rejects.toThrow('Access denied: autofill agent can only read target asset target-asset-id.') - }) - - it('allows autofill agent to read target asset without userId', async () => { - vi.mocked(prisma.asset.findUnique).mockResolvedValue({ - id: 'target-asset-id', - name: 'test.png', - mediaType: 'image/png', - storageKey: { key: 'raw/test.png' }, - media: { proxyType: 'image' }, - project: { teamId: 'team-1' }, - } as unknown as Asset) - - const tool = createReadAssetTool({ - agentType: 'autofill', - teamId: 'team-1', - targetAssetId: 'target-asset-id', - }) - - const result = await tool.execute('call-1', { - assetId: 'target-asset-id', - annotationId: null, - s3KeyOnly: true, - imageConfig: null, - videoConfig: null, - docConfig: null, - }) - - expect(authzService.hasPermission).not.toHaveBeenCalled() - expect(result.content.length).toBe(1) - expect((result.content[0] as { type: 'text'; text: string }).text).toContain( - 'target-asset-id', - ) - }) - }) -}) diff --git a/packages/agent/src/tools/read-asset.ts b/packages/agent/src/tools/read-asset.ts deleted file mode 100644 index 78905dbe..00000000 --- a/packages/agent/src/tools/read-asset.ts +++ /dev/null @@ -1,922 +0,0 @@ -import { Type } from 'typebox' -import { type AgentTool } from '@earendil-works/pi-agent-core' -import { type ImageContent, type TextContent } from '@earendil-works/pi-ai' -import { prisma, WorkflowTaskType, WorkflowTaskStatus, type User } from '@shumai/db' -import { s3Service } from '@shumai/core/src/s3/s3' -import { workflowService } from '@shumai/workflow-core' -import { authzService, Permission, ResourceType } from '@shumai/core/src/authz/authz' -import { resolveAnnotationsById } from './annotation-resolver' -import { getFileMimeType } from '@shumai/core/src/utils/file-mime' -import { is3dModel } from '@shumai/core/src/utils/mime' -import { - isValidTurntableDegree, - secondToDegree, - degreeToSecond, -} from '@shumai/core/src/utils/model3d' - -export const readAssetSchema = Type.Object( - { - assetId: Type.String({ - description: 'The asset ID of the workspace asset to inspect. Required.', - }), - annotationId: Type.Union([ - Type.String({ - description: - 'The ID of the comment or message entry from whose visual markup should be overlaid, or null.', - }), - Type.Null(), - ]), - s3KeyOnly: Type.Optional( - Type.Boolean({ - description: - 'When true, returns only the storage S3 key(s) and MIME types of the asset/frames without attaching base64 image or text content to the agent context. Use this to obtain S3 keys for generate_image or generate_video references without consuming context tokens.', - default: false, - }), - ), - imageConfig: Type.Union([ - Type.Object( - {}, - { - additionalProperties: false, - description: 'Optional configuration for image assets, or null.', - }, - ), - Type.Null(), - ]), - config3d: Type.Optional( - Type.Union([ - Type.Object( - { - degree: Type.Optional( - Type.Union([ - Type.Integer({ - minimum: 0, - maximum: 345, - description: - 'Turntable rotation degree (integer: 0, 15, 30, ..., 345 in 15° increments) to inspect a specific angle of a 3D model. Leave null/omitted to view the 4-view composite overview.', - }), - Type.Null(), - ]), - ), - }, - { - additionalProperties: false, - description: - 'Configuration for 3D model assets. Pass null if inspecting another asset type. Optionally pass degree to inspect a specific turntable angle, or leave null/omitted to view the 4-view composite overview.', - }, - ), - Type.Null(), - ]), - ), - videoConfig: Type.Union([ - Type.Object( - { - start: Type.Number({ description: 'Start time in seconds.' }), - end: Type.Number({ description: 'End time in seconds.' }), - count: Type.Number({ description: 'Number of frames to extract (e.g. 1 to 10).' }), - }, - { - additionalProperties: false, - description: - 'Configuration for video assets. Required when inspecting videos, or null for other asset types.', - }, - ), - Type.Null(), - ]), - docConfig: Type.Union([ - Type.Object( - { - mode: Type.Union([ - Type.Literal('pages', { - description: - 'Render specific page range as images for visual layout, charts, or visual annotations (required for binary PDFs).', - }), - Type.Literal('text', { - description: - 'Read raw text content directly (ideal for .md, .txt, .csv, code files).', - }), - ]), - startPage: Type.Union([ - Type.Number({ - description: - 'Start page number (1-based index, required when mode="pages"), or null when mode="text".', - }), - Type.Null(), - ]), - endPage: Type.Union([ - Type.Number({ - description: - 'End page number (1-based index, max 20 pages per call, required when mode="pages"), or null when mode="text".', - }), - Type.Null(), - ]), - }, - { - additionalProperties: false, - description: - 'Configuration for document/PDF/text assets, or null if inspecting another asset type.', - }, - ), - Type.Null(), - ]), - }, - { additionalProperties: false }, -) - -export function isPlainTextAsset(mediaType: string, filename: string): boolean { - if ( - mediaType.startsWith('text/') || - mediaType === 'application/json' || - mediaType === 'application/javascript' || - mediaType === 'application/typescript' || - mediaType === 'application/xml' || - mediaType === 'application/x-yaml' - ) { - return true - } - const textExtensions = [ - '.txt', - '.md', - '.markdown', - '.json', - '.csv', - '.tsv', - '.js', - '.ts', - '.tsx', - '.jsx', - '.html', - '.htm', - '.css', - '.scss', - '.yaml', - '.yml', - '.xml', - '.py', - '.rb', - '.go', - '.rs', - '.c', - '.cpp', - '.h', - '.hpp', - '.java', - '.sh', - '.bash', - '.zsh', - '.sql', - ] - return textExtensions.some((ext) => filename.endsWith(ext)) -} - -const MAX_TEXT_BYTES = 50 * 1024 -const MAX_TEXT_LINES = 1000 - -export type ReadAssetAuthContext = - | { - userId?: string - teamId?: string - agentType?: 'chat' | 'embedding' - targetAssetId?: string - } - | { - userId?: undefined - teamId: string - agentType: 'autofill' - targetAssetId: string - } - -export function createReadAssetTool( - auth: string | ReadAssetAuthContext, -): AgentTool { - const authContext: ReadAssetAuthContext = - typeof auth === 'string' ? { userId: auth, agentType: 'chat' } : auth - const { userId, teamId, agentType = 'chat', targetAssetId } = authContext - - return { - name: 'read_asset', - label: 'Read Asset', - description: - 'Inspects and reads the content of an asset in the workspace. ' + - 'Supports visual analysis for images, 3D models (overview or specific degree angle), video frame extraction, PDF page rendering, and direct text reading for text/markdown files. ' + - 'Pass assetId, and provide the specific config (imageConfig, config3d, videoConfig, or docConfig) while setting unused configs to null. ' + - 'Optionally provide annotationId to view visual markups drawn on the asset.', - parameters: readAssetSchema, - execute: async (_toolCallId, params) => { - const { assetId, annotationId, s3KeyOnly, config3d, videoConfig, docConfig } = params - - if (agentType === 'autofill' && !userId) { - if (!targetAssetId) { - throw new Error('Target asset ID is required for autofill agent authorization.') - } - if (!teamId) { - throw new Error('Team ID is required for autofill agent authorization.') - } - - // System background execution for Autofill Agent - const assetRecord = await prisma.asset.findUnique({ - where: { id: assetId }, - include: { project: true, teamRootFolder: true }, - }) - if (!assetRecord) { - throw new Error(`Asset with ID ${assetId} not found.`) - } - - // Strict team boundary check - const assetTeamId = assetRecord.project?.teamId || assetRecord.teamRootFolder?.id - if (!assetTeamId || assetTeamId !== teamId) { - throw new Error(`Access denied: asset ${assetId} does not belong to team ${teamId}.`) - } - - // Strict target asset boundary check (allows the target asset or its version stack children) - if (assetRecord.id !== targetAssetId && assetRecord.parentId !== targetAssetId) { - throw new Error( - `Access denied: autofill agent can only read target asset ${targetAssetId}.`, - ) - } - } else { - // Chat agents or user-driven executions strictly require a userId and full ACL validation - if (!userId) { - throw new Error('User ID is required for authorization.') - } - - await authzService.hasPermission({ - user: { id: userId } as User, - permission: Permission.Read, - type: ResourceType.Asset, - id: assetId, - }) - } - - let asset = await prisma.asset.findUnique({ - where: { id: assetId }, - include: { storageKey: true }, - }) - - if (!asset) { - throw new Error(`Asset with ID ${assetId} not found.`) - } - - if (asset.type === 'version_stack') { - const latestVersion = await prisma.asset.findFirst({ - where: { parentId: asset.id, isDeleted: false }, - orderBy: { sortIndex: 'asc' }, - include: { storageKey: true }, - }) - if (latestVersion) { - asset = latestVersion - } - } - - const mediaInfo = asset.media as unknown as PrismaJson.MediaInfo | null - const proxyType = mediaInfo?.proxyType - const mediaType = asset.mediaType?.toLowerCase() || '' - const filename = asset.name?.toLowerCase() || '' - const bucket = process.env.S3_BUCKET || 'shumai' - - // ---------------------------------------------------------------------- - // Branch 0: 3D Assets (Orthogonal 4-View Composite Image or Degree Angle) - // ---------------------------------------------------------------------- - const is3d = proxyType === '3d' || is3dModel(mediaType, filename) - if (is3d) { - if (videoConfig) { - throw new Error( - `Asset ${asset.id} is a 3D model. To inspect specific angles pass 'config3d: { degree: ... }'; do not pass videoConfig.`, - ) - } - - if (config3d?.degree !== undefined && config3d?.degree !== null) { - if (!isValidTurntableDegree(config3d.degree)) { - throw new Error( - `Invalid turntable degree: ${config3d.degree}. Degree must be an integer multiple of 15 between 0 and 345 (e.g. 0, 15, 30, 45, ..., 345).`, - ) - } - } - - let annotations: PrismaJson.AnnotationList | null = null - let commentTimestamp: number | null = null - if (annotationId) { - const resolved = await resolveAnnotationsById(asset.id, annotationId) - annotations = resolved.annotations - commentTimestamp = resolved.timestamp - if (commentTimestamp !== null && commentTimestamp !== undefined) { - const commentDegree = secondToDegree(commentTimestamp) - if ( - config3d?.degree !== undefined && - config3d?.degree !== null && - config3d.degree !== commentDegree - ) { - throw new Error( - `Annotation "${annotationId}" was drawn at ${commentDegree}°, but degree ${config3d.degree}° was specified. Do not pass a conflicting degree with annotationId.`, - ) - } - } - } - - if ((config3d?.degree !== undefined && config3d?.degree !== null) || annotationId) { - if (!mediaInfo?.videoTranscodes || mediaInfo.videoTranscodes.length === 0) { - throw new Error( - `Asset ${asset.id} is a 3D model, but its turntable video (turntable.mp4) is missing.`, - ) - } - - let targetDegree: number - let targetSecond: number - - if (config3d?.degree !== undefined && config3d?.degree !== null) { - targetDegree = config3d.degree - targetSecond = degreeToSecond(targetDegree) - } else { - targetSecond = - commentTimestamp !== null && commentTimestamp !== undefined ? commentTimestamp : 0 - targetDegree = secondToDegree(targetSecond) - } - - const task = await prisma.workflowTask.create({ - data: { - assetId: asset.id, - projectId: asset.projectId || 'none', - type: WorkflowTaskType.transcode_screenshot, - status: WorkflowTaskStatus.pending, - payload: { - projectId: asset.projectId || 'none', - screenshot: { - start: targetSecond, - end: targetSecond, - count: 1, - commentTimestamp: targetSecond, - annotations: annotations ?? undefined, - }, - }, - }, - }) - - const completedTask = await workflowService.executeWait(task) - // eslint-disable-next-line @typescript-eslint/no-explicit-any -- transcode screenshot output - const output = completedTask.output as any - const screenshots = output?.screenshots as - Array<{ key: string; timestamp: number }> | undefined - - if (!screenshots || screenshots.length === 0) { - throw new Error( - `Failed to extract 3D turntable frame for asset ${asset.id} at ${targetDegree}°.`, - ) - } - - const shotKey = screenshots[0].key - const downloadUrl = await s3Service.presign( - bucket, - shotKey, - 'GET', - true, - asset.name || undefined, - ) - - const content: Array = [] - const hasMarkup = Boolean(annotations && annotations.length > 0) - const textDescription = hasMarkup - ? `3D asset "${asset.name}" (ID: ${asset.id}, S3 Key: "${shotKey}") view at ${targetDegree}° with visual markups overlaid:` - : `3D asset "${asset.name}" (ID: ${asset.id}, S3 Key: "${shotKey}") view at ${targetDegree}°:` - - if (s3KeyOnly) { - content.push({ - type: 'text', - text: textDescription, - }) - } else { - const { buffer, contentType } = await s3Service.getObject(bucket, shotKey) - const actualMimeType = - contentType && contentType !== 'application/octet-stream' ? contentType : 'image/webp' - content.push({ - type: 'text', - text: textDescription, - }) - content.push({ - type: 'image', - data: buffer.toString('base64'), - mimeType: actualMimeType, - }) - } - - return { - content, - details: { - assetId: asset.id, - name: asset.name, - mediaType: asset.mediaType, - key: shotKey, - downloadUrl, - sourceKeys: [shotKey], - }, - } - } - - if (!mediaInfo?.imageTranscodes || mediaInfo.imageTranscodes.length === 0) { - throw new Error( - `Asset ${asset.id} is a 3D model, but its 4-views image (views.webp) is missing.`, - ) - } - - const mediaKey = mediaInfo.imageTranscodes[0].key - if (!mediaKey) { - throw new Error(`Asset ${asset.id} is a 3D model, but its 4-views image key is missing.`) - } - - const keyToUse = mediaKey - const mimeType = 'image/webp' - - const downloadUrl = await s3Service.presign( - bucket, - keyToUse, - 'GET', - true, - asset.name || undefined, - ) - - const content: Array = [] - const textDescription = [ - `3D asset "${asset.name}" (ID: ${asset.id}, MIME: ${mimeType}, S3 Key: "${keyToUse}"):`, - 'Orthogonal 4-view composite image (2x2 grid across 360° rotation):', - '- Top-Left quadrant: Front view (0°)', - '- Top-Right quadrant: Right side view (90°)', - '- Bottom-Left quadrant: Back view (180°)', - '- Bottom-Right quadrant: Left side view (270°)', - ].join('\n') - - if (s3KeyOnly) { - content.push({ - type: 'text', - text: textDescription, - }) - } else { - const { buffer, contentType } = await s3Service.getObject(bucket, keyToUse) - const actualMimeType = - contentType && contentType !== 'application/octet-stream' ? contentType : mimeType - content.push({ - type: 'text', - text: textDescription, - }) - content.push({ - type: 'image', - data: buffer.toString('base64'), - mimeType: actualMimeType, - }) - } - - return { - content, - details: { - assetId: asset.id, - name: asset.name, - mediaType: asset.mediaType, - key: keyToUse, - downloadUrl, - sourceKeys: [keyToUse], - }, - } - } - - // ---------------------------------------------------------------------- - // Branch 1: Image Assets - // ---------------------------------------------------------------------- - if (proxyType === 'image' || mediaType.startsWith('image/') || filename.endsWith('.psd')) { - let mediaKey = asset.storageKey?.key - if (mediaInfo?.imageTranscodes && mediaInfo.imageTranscodes.length > 0) { - mediaKey = mediaInfo.imageTranscodes[0].key || mediaKey - } - - if (!mediaKey) { - throw new Error('No media content found for this asset.') - } - - let keyToUse = mediaKey - let mimeType = getFileMimeType(null, keyToUse, asset.mediaType || 'image/png') - - if (annotationId) { - const { annotations } = await resolveAnnotationsById(asset.id, annotationId) - if (annotations && annotations.length > 0) { - const task = await prisma.workflowTask.create({ - data: { - assetId: asset.id, - projectId: asset.projectId || 'none', - type: WorkflowTaskType.transcode_image_annotation, - status: WorkflowTaskStatus.pending, - payload: { - projectId: asset.projectId || 'none', - imageAnnotation: { - annotations, - }, - }, - }, - }) - - const completedTask = await workflowService.executeWait(task) - // eslint-disable-next-line @typescript-eslint/no-explicit-any -- transcode image annotation task output - const output = completedTask.output as any - if (output?.key) { - keyToUse = output.key - mimeType = 'image/webp' - } else { - throw new Error('Image annotation transcode workflow failed to output S3 key.') - } - } - } - - const downloadUrl = await s3Service.presign( - bucket, - keyToUse, - 'GET', - true, - asset.name || undefined, - ) - - const content: Array = [] - - if (s3KeyOnly) { - content.push({ - type: 'text', - text: `Image asset "${asset.name}" (ID: ${asset.id}, MIME: ${mimeType}, S3 Key: "${keyToUse}")`, - }) - } else { - const { buffer, contentType } = await s3Service.getObject(bucket, keyToUse) - const actualMimeType = - contentType && contentType !== 'application/octet-stream' ? contentType : mimeType - content.push({ - type: 'text', - text: `Image asset "${asset.name}" (ID: ${asset.id}, MIME: ${actualMimeType}, S3 Key: "${keyToUse}")`, - }) - content.push({ - type: 'image', - data: buffer.toString('base64'), - mimeType: actualMimeType, - }) - } - - return { - content, - details: { - assetId: asset.id, - name: asset.name, - mediaType: asset.mediaType, - key: keyToUse, - downloadUrl, - sourceKeys: [keyToUse], - }, - } - } - - // ---------------------------------------------------------------------- - // Branch 2: Video Assets - // ---------------------------------------------------------------------- - if (proxyType === 'video' || mediaType.startsWith('video/')) { - if (!videoConfig) { - throw new Error( - 'videoConfig with start, end, and count is required when inspecting video assets.', - ) - } - - const { start, end, count } = videoConfig - - if (start < 0 || end < 0) { - throw new Error('Invalid video time range: start and end must be non-negative numbers.') - } - - if (start > end) { - throw new Error( - `Invalid video time range: start (${start}) must be less than or equal to end (${end}).`, - ) - } - - if (count < 1) { - throw new Error(`Invalid frame count: count (${count}) must be at least 1.`) - } - - const { annotations, timestamp: commentTimestamp } = await resolveAnnotationsById( - asset.id, - annotationId, - ) - - const task = await prisma.workflowTask.create({ - data: { - assetId: asset.id, - projectId: asset.projectId || 'none', - type: WorkflowTaskType.transcode_screenshot, - status: WorkflowTaskStatus.pending, - payload: { - projectId: asset.projectId || 'none', - screenshot: { - start, - end, - count, - commentTimestamp, - annotations, - }, - }, - }, - }) - - const completedTask = await workflowService.executeWait(task) - // eslint-disable-next-line @typescript-eslint/no-explicit-any -- transcode screenshot output - const output = completedTask.output as any - const screenshots = output?.screenshots as - Array<{ key: string; timestamp: number }> | undefined - - if (!screenshots || screenshots.length === 0) { - throw new Error('No screenshots were generated.') - } - - const content: Array = [] - const sourceKeys: string[] = [] - const results: Array<{ key: string; timestamp: number; downloadUrl: string }> = [] - const textLines: string[] = [ - `Extracted ${screenshots.length} frame(s) from "${asset.name}" (${start}s - ${end}s):`, - ] - - for (const shot of screenshots) { - const downloadUrl = await s3Service.presign( - bucket, - shot.key, - 'GET', - true, - asset.name || undefined, - ) - - sourceKeys.push(shot.key) - results.push({ - key: shot.key, - timestamp: shot.timestamp, - downloadUrl, - }) - - if (s3KeyOnly) { - const mimeType = getFileMimeType(null, shot.key, 'image/webp') - textLines.push( - `- Frame at ${shot.timestamp}s (S3 Key: "${shot.key}", MIME: ${mimeType})`, - ) - } else { - const { buffer, contentType } = await s3Service.getObject(bucket, shot.key) - const mimeType = - contentType && contentType !== 'application/octet-stream' - ? contentType - : getFileMimeType(buffer, shot.key, 'image/webp') - textLines.push( - `- Frame at ${shot.timestamp}s (S3 Key: "${shot.key}", MIME: ${mimeType})`, - ) - content.push({ - type: 'image', - data: buffer.toString('base64'), - mimeType, - }) - } - } - - content.unshift({ - type: 'text', - text: textLines.join('\n'), - }) - - return { - content, - details: { - assetId: asset.id, - name: asset.name, - mediaType: asset.mediaType, - keys: sourceKeys, - results, - sourceKeys, - }, - } - } - - // ---------------------------------------------------------------------- - // Branch 3: Documents & PDF Assets - // ---------------------------------------------------------------------- - if ( - proxyType === 'pdf' || - mediaType === 'application/pdf' || - isPlainTextAsset(mediaType, filename) - ) { - if (!docConfig) { - throw new Error( - 'docConfig with mode ("pages" or "text") is required when inspecting document/PDF assets.', - ) - } - - if (docConfig.mode === 'text') { - if (annotationId !== null) { - throw new Error( - 'annotationId cannot be used with docConfig mode "text" because visual markups can only be rendered on visual pages. Use docConfig: { mode: "pages", startPage: ..., endPage: ... } instead.', - ) - } - - if (docConfig.startPage !== null || docConfig.endPage !== null) { - throw new Error( - 'startPage and endPage must be null when docConfig mode is "text". To inspect specific page numbers, use mode: "pages".', - ) - } - - if (!isPlainTextAsset(mediaType, filename)) { - throw new Error( - `Asset "${asset.name}" is a binary PDF/document and cannot be read as raw text. Please call read_asset with docConfig: { mode: "pages", startPage: 1, endPage: ... } to view its visual pages.`, - ) - } - - const mediaKey = asset.storageKey?.key - if (!mediaKey) { - throw new Error(`No media content found for asset ${asset.id}.`) - } - - const docMimeType = - asset.mediaType || getFileMimeType(null, asset.name || mediaKey, 'text/plain') - const downloadUrl = await s3Service.presign( - bucket, - mediaKey, - 'GET', - true, - asset.name || undefined, - ) - - if (s3KeyOnly) { - return { - content: [ - { - type: 'text', - text: `Document "${asset.name}" (ID: ${asset.id}, MIME: ${docMimeType}, S3 Key: "${mediaKey}")`, - }, - ], - details: { - assetId: asset.id, - name: asset.name, - mediaType: asset.mediaType, - key: mediaKey, - downloadUrl, - sourceKeys: [mediaKey], - size: 0, - isTruncated: false, - }, - } - } - - const { buffer } = await s3Service.getObject(bucket, mediaKey) - const rawText = buffer.toString('utf-8') - let text = rawText - let isTruncated = false - - const lines = text.split('\n') - if (lines.length > MAX_TEXT_LINES) { - text = lines.slice(0, MAX_TEXT_LINES).join('\n') - isTruncated = true - } - - if (Buffer.byteLength(text, 'utf-8') > MAX_TEXT_BYTES) { - text = Buffer.from(text, 'utf-8').subarray(0, MAX_TEXT_BYTES).toString('utf-8') - isTruncated = true - } - - if (isTruncated) { - text += `\n\n[Content truncated to ${MAX_TEXT_LINES} lines / ${MAX_TEXT_BYTES / 1024}KB limit. Use download_asset to process the full file.]` - } - - return { - content: [ - { - type: 'text', - text: `Document "${asset.name}" (ID: ${asset.id}, MIME: ${docMimeType}, S3 Key: "${mediaKey}"):\n\n${text}`, - }, - ], - details: { - assetId: asset.id, - name: asset.name, - mediaType: asset.mediaType, - key: mediaKey, - downloadUrl, - sourceKeys: [mediaKey], - size: buffer.length, - isTruncated, - }, - } - } - - // docConfig.mode === 'pages' - if (docConfig.startPage === null || docConfig.endPage === null) { - throw new Error('startPage and endPage are required when docConfig mode is "pages".') - } - - const start = docConfig.startPage - const end = docConfig.endPage - - if (start < 1) { - throw new Error(`Invalid page range: start page (${start}) must be at least 1.`) - } - - if (start > end) { - throw new Error( - `Invalid page range: start page (${start}) must be less than or equal to end page (${end}).`, - ) - } - - const MAX_PAGE_RANGE = 20 - const requestedPages = end - start + 1 - if (requestedPages > MAX_PAGE_RANGE) { - throw new Error( - `Page range (${requestedPages} pages requested) exceeds the maximum limit of ${MAX_PAGE_RANGE} pages per request.`, - ) - } - - const { annotations, timestamp: commentTimestamp } = await resolveAnnotationsById( - asset.id, - annotationId, - ) - - const task = await prisma.workflowTask.create({ - data: { - assetId: asset.id, - projectId: asset.projectId || 'none', - type: WorkflowTaskType.transcode_pdf_pages, - status: WorkflowTaskStatus.pending, - payload: { - projectId: asset.projectId || 'none', - pdfPages: { - start, - end, - commentTimestamp, - annotations, - }, - }, - }, - }) - - const completedTask = await workflowService.executeWait(task) - // eslint-disable-next-line @typescript-eslint/no-explicit-any -- transcode pdf pages output - const output = completedTask.output as any - const pages = output?.pages as Array<{ key: string; page: number }> | undefined - - if (!pages || pages.length === 0) { - throw new Error('No PDF page images were generated.') - } - - const content: Array = [] - const sourceKeys: string[] = [] - const results: Array<{ key: string; page: number; downloadUrl: string }> = [] - const textLines: string[] = [ - `Rendered ${pages.length} page(s) of "${asset.name}" (Pages ${start} to ${end}):`, - ] - - for (const pageItem of pages) { - const downloadUrl = await s3Service.presign( - bucket, - pageItem.key, - 'GET', - true, - asset.name || undefined, - ) - - sourceKeys.push(pageItem.key) - results.push({ - key: pageItem.key, - page: pageItem.page, - downloadUrl, - }) - - if (s3KeyOnly) { - const mimeType = getFileMimeType(null, pageItem.key, 'image/webp') - textLines.push(`- Page ${pageItem.page} (S3 Key: "${pageItem.key}", MIME: ${mimeType})`) - } else { - const { buffer, contentType } = await s3Service.getObject(bucket, pageItem.key) - const mimeType = - contentType && contentType !== 'application/octet-stream' - ? contentType - : getFileMimeType(buffer, pageItem.key, 'image/webp') - textLines.push(`- Page ${pageItem.page} (S3 Key: "${pageItem.key}", MIME: ${mimeType})`) - content.push({ - type: 'image', - data: buffer.toString('base64'), - mimeType, - }) - } - } - - content.unshift({ - type: 'text', - text: textLines.join('\n'), - }) - - return { - content, - details: { - assetId: asset.id, - name: asset.name, - mediaType: asset.mediaType, - keys: sourceKeys, - results, - sourceKeys, - }, - } - } - - throw new Error( - `Unsupported asset type for read_asset. Asset ${asset.id} has media type "${mediaType || 'unknown'}". Use download_asset to download this file to disk.`, - ) - }, - } -} diff --git a/packages/agent/src/tools/annotation-resolver.ts b/packages/core/src/asset-read/annotations.ts similarity index 74% rename from packages/agent/src/tools/annotation-resolver.ts rename to packages/core/src/asset-read/annotations.ts index 92d5aab4..a2b02538 100644 --- a/packages/agent/src/tools/annotation-resolver.ts +++ b/packages/core/src/asset-read/annotations.ts @@ -1,7 +1,12 @@ import { prisma } from '@shumai/db' import type { ShumaiMessageContext } from '@shumai/dtos' +import { HTTPException } from 'hono/http-exception' import { degreeToSecond } from '@shumai/core/src/utils/model3d' +/** + * Resolve the visual markup referenced by an annotation ID: a comment (comments mode) or an + * agent session entry (1-on-1 chat mode). The annotation must belong to `assetId`. + */ export async function resolveAnnotationsById( assetId: string, annotationId?: string | null, @@ -17,9 +22,9 @@ export async function resolveAnnotationsById( }) if (comment) { if (comment.assetId !== assetId) { - throw new Error( - `Annotation "${annotationId}" belongs to asset "${comment.assetId}", not target asset "${assetId}".`, - ) + throw new HTTPException(400, { + message: `Annotation "${annotationId}" belongs to asset "${comment.assetId}", not target asset "${assetId}".`, + }) } const list = comment.annotation const annotations = @@ -37,9 +42,9 @@ export async function resolveAnnotationsById( ShumaiMessageContext | undefined const entryAssetId = entry.assetId || details?.currentAsset?.id if (entryAssetId && entryAssetId !== assetId) { - throw new Error( - `Annotation "${annotationId}" belongs to asset "${entryAssetId}", not target asset "${assetId}".`, - ) + throw new HTTPException(400, { + message: `Annotation "${annotationId}" belongs to asset "${entryAssetId}", not target asset "${assetId}".`, + }) } const list = details?.annotations const annotations = @@ -55,5 +60,5 @@ export async function resolveAnnotationsById( return { annotations, timestamp } } - throw new Error(`Annotation with ID "${annotationId}" not found.`) + throw new HTTPException(404, { message: `Annotation with ID "${annotationId}" not found.` }) } diff --git a/packages/core/src/asset-read/asset-read.test.ts b/packages/core/src/asset-read/asset-read.test.ts new file mode 100644 index 00000000..e6a1bb11 --- /dev/null +++ b/packages/core/src/asset-read/asset-read.test.ts @@ -0,0 +1,601 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { + AssetType, + prisma, + WorkflowTaskStatus, + WorkflowTaskType, + type Prisma, + type User, +} from '@shumai/db' +import { setupTestDbHooks } from '@shumai/db/test' +import { workflowService } from '@shumai/workflow-core' +import { projectService } from '@shumai/core/src/project/project' +import { s3Service } from '@shumai/core/src/s3/s3' +import { getAssetRenderOp, readAssetOp, runOperation } from '@shumai/core/src/operations' +import type { OperationContext } from '@shumai/core/src/operations' +import { assetReadService, READ_URL_EXPIRES_SECONDS, toAssetReadOutput } from './asset-read' + +describe('AssetReadService', () => { + setupTestDbHooks() + + let owner: User + let reviewer: User + let outsider: User + let teamId: string + let projectId: string + let rootFolderId: string + + const user = (u: User) => ({ kind: 'user' as const, userId: u.id }) + const now = { waitBudgetMs: 0 } + + beforeEach(async () => { + owner = await prisma.user.create({ + data: { name: 'Owner', email: 'read-owner@test.com', password: 'pw' }, + }) + reviewer = await prisma.user.create({ + data: { name: 'Reviewer', email: 'read-reviewer@test.com', password: 'pw' }, + }) + outsider = await prisma.user.create({ + data: { name: 'Outsider', email: 'read-outsider@test.com', password: 'pw' }, + }) + const team = await prisma.team.create({ data: { name: 'Read Team' } }) + teamId = team.id + await prisma.teamMember.create({ data: { teamId, userId: owner.id, role: 'owner' } }) + await prisma.teamMember.create({ data: { teamId, userId: reviewer.id, role: 'reviewer' } }) + const project = await projectService.createProject(owner, { teamId, name: 'Read Project' }) + projectId = project.id + rootFolderId = project.rootFolder! + + vi.spyOn(s3Service, 'presign').mockImplementation( + async (_bucket, key) => `https://signed.example/${key}`, + ) + vi.spyOn(s3Service, 'getObject').mockImplementation(async (_bucket, key) => ({ + buffer: Buffer.from(`content of ${key}`), + contentType: 'text/plain', + })) + }) + + afterEach(() => { + vi.restoreAllMocks() + }) + + async function createFile( + name: string, + data: Partial & { key?: string } = {}, + ) { + const { key, ...rest } = data + const storageKey = key ? await prisma.storageKey.create({ data: { key } }) : null + return prisma.asset.create({ + data: { + name, + type: AssetType.file, + status: 'uploaded', + projectId, + parentId: rootFolderId, + storageKeyId: storageKey?.id, + ...rest, + }, + }) + } + + const imageMedia = { + proxyType: 'image', + imageTranscodes: [{ key: 'proxy/photo.webp' }], + } as unknown as PrismaJson.MediaInfo + + async function createVideo() { + return createFile('clip.mp4', { + mediaType: 'video/mp4', + media: { proxyType: 'video' } as unknown as PrismaJson.MediaInfo, + key: 'raw/clip.mp4', + }) + } + + describe('access', () => { + it('lets a reviewer read an image without creating a render task', async () => { + const image = await createFile('photo.png', { mediaType: 'image/png', media: imageMedia }) + + const outcome = await assetReadService.read(user(reviewer), { assetId: image.id }, now) + + expect(outcome).toMatchObject({ + status: 'ready', + kind: 'image', + asset: { id: image.id, name: 'photo.png' }, + result: { + summary: `Image asset "photo.png" (ID: ${image.id}):`, + images: [{ key: 'proxy/photo.webp', label: 'Image', mimeType: 'image/webp' }], + }, + }) + expect(await prisma.workflowTask.count({ where: { assetId: image.id } })).toBe(0) + }) + + it('rejects users without Read access before creating anything', async () => { + const video = await createVideo() + await expect( + assetReadService.read( + user(outsider), + { assetId: video.id, videoConfig: { start: 0, end: 1, count: 1 } }, + now, + ), + ).rejects.toMatchObject({ status: 403 }) + expect(await prisma.workflowTask.count({ where: { assetId: video.id } })).toBe(0) + }) + + it('confines the autofill agent to its team and target asset', async () => { + const image = await createFile('photo.png', { mediaType: 'image/png', media: imageMedia }) + const other = await createFile('other.png', { mediaType: 'image/png', media: imageMedia }) + + const ok = await assetReadService.read( + { kind: 'autofill', teamId, targetAssetId: image.id }, + { assetId: image.id }, + now, + ) + expect(ok.status).toBe('ready') + + await expect( + assetReadService.read( + { kind: 'autofill', teamId, targetAssetId: image.id }, + { assetId: other.id }, + now, + ), + ).rejects.toThrow(`autofill agent can only read target asset ${image.id}`) + await expect( + assetReadService.read( + { kind: 'autofill', teamId: 'another-team', targetAssetId: image.id }, + { assetId: image.id }, + now, + ), + ).rejects.toThrow(`does not belong to team another-team`) + }) + }) + + it('reads the latest version of a version stack', async () => { + const stack = await prisma.asset.create({ + data: { + name: 'diagram', + type: AssetType.version_stack, + status: 'uploaded', + projectId, + parentId: rootFolderId, + }, + }) + await createFile('diagram_v1.png', { + parentId: stack.id, + sortIndex: 'b', + mediaType: 'image/png', + key: 'raw/v1.png', + }) + const v2 = await createFile('diagram_v2.png', { + parentId: stack.id, + sortIndex: 'a', + mediaType: 'image/png', + key: 'raw/v2.png', + }) + + const outcome = await assetReadService.read(user(owner), { assetId: stack.id }, now) + expect(outcome.asset.id).toBe(v2.id) + expect(outcome.status === 'ready' && outcome.result.images[0].key).toBe('raw/v2.png') + }) + + describe('reader selection and options', () => { + it.each([ + ['model.glb', 'model/gltf-binary', 'model3d'], + ['photo.psd', 'image/vnd.adobe.photoshop', 'image'], + ['clip.mov', 'video/quicktime', 'video'], + ['doc.pdf', 'application/pdf', 'document'], + ['notes.md', 'text/markdown', 'document'], + ['script.ts', null, 'document'], + ])('reads %s with the %s reader', async (name, mediaType, kind) => { + const asset = await createFile(name, { mediaType, key: `raw/${name}` }) + // An empty-object config of the wrong type is ignored; the error names the right reader. + const error = await assetReadService + .read(user(owner), { assetId: asset.id, imageConfig: {}, config3d: { degree: 7 } }, now) + .catch((err: unknown) => err) + if (kind === 'model3d') { + expect(error).toMatchObject({ + message: expect.stringContaining('Invalid turntable degree'), + }) + } else { + expect(error).toMatchObject({ + status: 400, + message: expect.stringContaining(`Asset ${asset.id} is a`), + }) + expect((error as Error).message).toContain('do not pass config3d') + } + }) + + it('rejects unsupported files', async () => { + const zip = await createFile('bundle.zip', { mediaType: 'application/zip' }) + await expect(assetReadService.read(user(owner), { assetId: zip.id }, now)).rejects.toThrow( + 'Unsupported asset type for read_asset', + ) + }) + + it('reads text content and reports truncation', async () => { + const doc = await createFile('notes.md', { mediaType: 'text/markdown', key: 'raw/notes.md' }) + const outcome = await assetReadService.read( + user(owner), + { assetId: doc.id, docConfig: { mode: 'text' } }, + now, + ) + expect(outcome).toMatchObject({ + status: 'ready', + kind: 'document', + result: { + images: [], + text: { + content: 'content of raw/notes.md', + truncated: false, + size: 'content of raw/notes.md'.length, + key: 'raw/notes.md', + mimeType: 'text/markdown', + }, + }, + }) + }) + + it('skips reading text when includeContent is false', async () => { + const doc = await createFile('notes.md', { mediaType: 'text/markdown', key: 'raw/notes.md' }) + const outcome = await assetReadService.read( + user(owner), + { assetId: doc.id, docConfig: { mode: 'text' }, includeContent: false }, + now, + ) + expect(s3Service.getObject).not.toHaveBeenCalled() + expect(outcome.status === 'ready' && outcome.result.text?.content).toBe('') + }) + }) + + describe('render tasks', () => { + it('stores the read request on the task and returns pending when the budget runs out', async () => { + const video = await createVideo() + const outcome = await assetReadService.read( + user(reviewer), + { assetId: video.id, videoConfig: { start: 0, end: 4, count: 2 } }, + now, + ) + + expect(outcome).toMatchObject({ status: 'pending', kind: 'video', retryAfterSeconds: 5 }) + const taskId = outcome.status === 'pending' ? outcome.taskId : '' + const task = await prisma.workflowTask.findUniqueOrThrow({ where: { id: taskId } }) + expect(task).toMatchObject({ + assetId: video.id, + projectId, + type: WorkflowTaskType.transcode_screenshot, + status: WorkflowTaskStatus.pending, + }) + expect(task.payload).toEqual({ + projectId, + screenshot: { start: 0, end: 4, count: 2, commentTimestamp: null, annotations: null }, + assetRead: { + reader: 'video', + options: { start: 0, end: 4, count: 2 }, + requestedBy: reviewer.id, + }, + }) + }) + + it('overlays annotations from a comment on the image', async () => { + const image = await createFile('photo.png', { mediaType: 'image/png', media: imageMedia }) + const annotation = [ + { + type: 'box', + color: '#ff0000', + points: [ + [0, 0], + [10, 10], + ], + }, + ] as unknown as PrismaJson.AnnotationList + const comment = await prisma.assetComment.create({ + data: { assetId: image.id, creatorId: owner.id, message: 'here', annotation }, + }) + + const outcome = await assetReadService.read( + user(owner), + { assetId: image.id, annotationId: comment.id }, + now, + ) + + expect(outcome).toMatchObject({ status: 'pending', kind: 'image', retryAfterSeconds: 3 }) + const task = await prisma.workflowTask.findFirstOrThrow({ where: { assetId: image.id } }) + expect(task.type).toBe(WorkflowTaskType.transcode_image_annotation) + expect(task.payload?.imageAnnotation).toEqual({ annotations: annotation }) + expect(task.payload?.assetRead).toMatchObject({ reader: 'image', annotationId: comment.id }) + }) + + it('rejects annotations from another asset', async () => { + const image = await createFile('photo.png', { mediaType: 'image/png', media: imageMedia }) + const other = await createFile('other.png', { mediaType: 'image/png', media: imageMedia }) + const comment = await prisma.assetComment.create({ + data: { assetId: other.id, creatorId: owner.id, message: 'elsewhere' }, + }) + await expect( + assetReadService.read(user(owner), { assetId: image.id, annotationId: comment.id }, now), + ).rejects.toMatchObject({ status: 400 }) + }) + + it('stores reader context for 3D angles and collects it later', async () => { + const model = await createFile('robot.glb', { + mediaType: 'model/gltf-binary', + media: { + proxyType: '3d', + videoTranscodes: [{ key: 'proxy/turntable.mp4' }], + } as unknown as PrismaJson.MediaInfo, + }) + const pending = await assetReadService.read( + user(owner), + { assetId: model.id, config3d: { degree: 90 } }, + now, + ) + const taskId = pending.status === 'pending' ? pending.taskId : '' + const task = await prisma.workflowTask.findUniqueOrThrow({ where: { id: taskId } }) + expect(task.payload?.assetRead?.context).toEqual({ degree: 90, hasMarkup: false }) + + await prisma.workflowTask.update({ + where: { id: taskId }, + data: { + status: WorkflowTaskStatus.completed, + output: { screenshots: [{ key: 'shots/90.webp', timestamp: 1 }] }, + }, + }) + const ready = await assetReadService.getRender(owner.id, taskId, now) + expect(ready).toMatchObject({ + status: 'ready', + kind: 'model3d', + result: { + summary: `3D asset "robot.glb" (ID: ${model.id}) view at 90°:`, + images: [{ key: 'shots/90.webp', label: 'View at 90°' }], + }, + }) + }) + }) + + describe('getRender', () => { + async function pendingVideoRead(by: User) { + const video = await createVideo() + const outcome = await assetReadService.read( + user(by), + { assetId: video.id, videoConfig: { start: 0, end: 4, count: 2 } }, + now, + ) + if (outcome.status !== 'pending') throw new Error('expected pending') + return { video, taskId: outcome.taskId } + } + + it('stays pending until the task completes, then returns the frames', async () => { + const { taskId } = await pendingVideoRead(reviewer) + + expect(await assetReadService.getRender(reviewer.id, taskId, now)).toMatchObject({ + status: 'pending', + taskId, + }) + + await prisma.workflowTask.update({ + where: { id: taskId }, + data: { + status: WorkflowTaskStatus.completed, + output: { + screenshots: [ + { key: 'shots/0.webp', timestamp: 0 }, + { key: 'shots/4.webp', timestamp: 4 }, + ], + }, + }, + }) + + const ready = await assetReadService.getRender(reviewer.id, taskId, now) + expect(ready).toMatchObject({ + status: 'ready', + result: { + summary: 'Extracted 2 frame(s) from "clip.mp4" (0s - 4s):', + images: [ + { key: 'shots/0.webp', label: 'Frame at 0s', mimeType: 'image/webp' }, + { key: 'shots/4.webp', label: 'Frame at 4s', mimeType: 'image/webp' }, + ], + }, + }) + }) + + it('surfaces a failed render', async () => { + const { taskId } = await pendingVideoRead(owner) + await prisma.workflowTask.update({ + where: { id: taskId }, + data: { status: WorkflowTaskStatus.failed, output: { error: 'ffmpeg exploded' } }, + }) + await expect(assetReadService.getRender(owner.id, taskId, now)).rejects.toThrow( + 'ffmpeg exploded', + ) + }) + + it('returns 404 to anyone but the requester', async () => { + const { taskId } = await pendingVideoRead(reviewer) + await expect(assetReadService.getRender(owner.id, taskId, now)).rejects.toMatchObject({ + status: 404, + message: 'Render not found', + }) + }) + + it('returns 404 once the requester loses access', async () => { + const { taskId } = await pendingVideoRead(reviewer) + await prisma.teamMember.deleteMany({ where: { userId: reviewer.id } }) + await expect(assetReadService.getRender(reviewer.id, taskId, now)).rejects.toMatchObject({ + status: 404, + }) + }) + + it('returns 404 for unknown tasks and tasks that are not asset reads', async () => { + const video = await createVideo() + const other = await prisma.workflowTask.create({ + data: { + assetId: video.id, + projectId, + type: WorkflowTaskType.transcode_video, + status: WorkflowTaskStatus.completed, + payload: { projectId }, + }, + }) + await expect(assetReadService.getRender(owner.id, other.id, now)).rejects.toMatchObject({ + status: 404, + }) + await expect(assetReadService.getRender(owner.id, 'nope', now)).rejects.toMatchObject({ + status: 404, + }) + }) + }) + + describe('toAssetReadOutput', () => { + it('returns short-lived links and only includes storage keys when asked', async () => { + const image = await createFile('photo.png', { mediaType: 'image/png', media: imageMedia }) + const outcome = await assetReadService.read(user(owner), { assetId: image.id }, now) + + const external = await toAssetReadOutput(outcome, { includeKeys: false }) + expect(external.images).toEqual([ + { label: 'Image', mimeType: 'image/webp', url: 'https://signed.example/proxy/photo.webp' }, + ]) + expect(s3Service.presign).toHaveBeenCalledWith( + expect.any(String), + 'proxy/photo.webp', + 'GET', + false, + undefined, + false, + READ_URL_EXPIRES_SECONDS, + ) + + const internal = await toAssetReadOutput(outcome, { includeKeys: true }) + expect(internal.images[0].key).toBe('proxy/photo.webp') + }) + + it('hides the text storage key and link from external callers', async () => { + const doc = await createFile('notes.md', { mediaType: 'text/markdown', key: 'raw/notes.md' }) + const outcome = await assetReadService.read( + user(owner), + { assetId: doc.id, docConfig: { mode: 'text' } }, + now, + ) + const external = await toAssetReadOutput(outcome, { includeKeys: false }) + expect(external.text).toEqual({ + content: 'content of raw/notes.md', + truncated: false, + size: 23, + mimeType: 'text/markdown', + }) + const internal = await toAssetReadOutput(outcome, { includeKeys: true }) + expect(internal.text).toMatchObject({ + key: 'raw/notes.md', + url: 'https://signed.example/raw/notes.md', + }) + }) + }) +}) + +describe('read_asset and get_asset_render Operations', () => { + setupTestDbHooks() + + let owner: User + let videoId: string + let imageId: string + + const as = (via: OperationContext['via']): OperationContext => ({ userId: owner.id, via }) + + beforeEach(async () => { + owner = await prisma.user.create({ + data: { name: 'Owner', email: 'read-op-owner@test.com', password: 'pw' }, + }) + const team = await prisma.team.create({ data: { name: 'Read Op Team' } }) + await prisma.teamMember.create({ data: { teamId: team.id, userId: owner.id, role: 'owner' } }) + const project = await projectService.createProject(owner, { + teamId: team.id, + name: 'Read Op Project', + }) + const base = { + type: AssetType.file, + status: 'uploaded' as const, + projectId: project.id, + parentId: project.rootFolder!, + } + videoId = ( + await prisma.asset.create({ + data: { + ...base, + name: 'clip.mp4', + mediaType: 'video/mp4', + media: { proxyType: 'video' } as unknown as PrismaJson.MediaInfo, + }, + }) + ).id + imageId = ( + await prisma.asset.create({ + data: { + ...base, + name: 'photo.png', + mediaType: 'image/png', + media: { + proxyType: 'image', + imageTranscodes: [{ key: 'proxy/photo.webp' }], + } as unknown as PrismaJson.MediaInfo, + }, + }) + ).id + vi.spyOn(s3Service, 'presign').mockResolvedValue('https://signed.example/x') + }) + + afterEach(() => { + vi.restoreAllMocks() + }) + + it.each([ + ['agent', true], + ['mcp', false], + ['cli', false], + ['api', false], + ] as const)('via %s includes storage keys: %s', async (via, includesKey) => { + const output = await runOperation(readAssetOp, as(via), { assetId: imageId }) + expect(output.status).toBe('ready') + expect(output.images[0].url).toBe('https://signed.example/x') + expect('key' in output.images[0]).toBe(includesKey) + }) + + it.each([ + ['agent', undefined, 60_000], + ['mcp', undefined, 20_000], + ['cli', undefined, 120_000], + ['api', undefined, 20_000], + ['cli', 5, 5_000], + ['mcp', 600, 20_000], + ['agent', 0, 0], + ] as const)('via %s with maxWaitSeconds %s waits %sms', async (via, maxWaitSeconds, budget) => { + const waitFor = vi + .spyOn(workflowService, 'waitFor') + .mockImplementation(async (task) => ({ task, done: false })) + + const output = await runOperation(readAssetOp, as(via), { + assetId: videoId, + videoConfig: { start: 0, end: 1, count: 1 }, + maxWaitSeconds, + }) + expect(waitFor).toHaveBeenCalledWith(expect.anything(), budget) + expect(output).toMatchObject({ + status: 'pending', + summary: 'Rendering is still in progress for "clip.mp4".', + asset: { id: videoId, kind: 'video' }, + images: [], + taskId: expect.any(String), + retryAfterSeconds: 5, + }) + + await runOperation(getAssetRenderOp, as(via), { taskId: output.taskId!, maxWaitSeconds }) + expect(waitFor).toHaveBeenLastCalledWith(expect.anything(), budget) + }) + + it('rejects invalid input before touching the asset', async () => { + await expect( + runOperation(readAssetOp, as('mcp'), { + assetId: videoId, + videoConfig: { start: 0, end: 1 }, + }), + ).rejects.toMatchObject({ status: 400 }) + expect(await prisma.workflowTask.count({ where: { assetId: videoId } })).toBe(0) + }) +}) diff --git a/packages/core/src/asset-read/asset-read.ts b/packages/core/src/asset-read/asset-read.ts new file mode 100644 index 00000000..fe0d1240 --- /dev/null +++ b/packages/core/src/asset-read/asset-read.ts @@ -0,0 +1,342 @@ +import { + readerOptionSchemas, + type AssetReadOutput, + type ReaderConfigKey, + type ReaderKind, +} from '@shumai/operations' +import { AssetType, prisma, WorkflowTaskStatus, type Prisma, type WorkflowTask } from '@shumai/db' +import { HTTPException } from 'hono/http-exception' +import { workflowService } from '@shumai/workflow-core' +import { authzService, Permission, ResourceType } from '@shumai/core/src/authz/authz' +import { s3Service } from '@shumai/core/src/s3/s3' +import { resolveAnnotationsById } from './annotations' +import { findReader, getReader, readers } from './registry' +import { badRequest, storageBucket, type ReadableAsset, type ReadResult } from './types' + +/** Who is reading: a user (Read permission required) or the autofill system agent. */ +export type ReadAccess = + { kind: 'user'; userId: string } | { kind: 'autofill'; teamId?: string; targetAssetId?: string } + +export type ReadAssetRequest = { + assetId: string + annotationId?: string + includeContent?: boolean +} & { [K in ReaderConfigKey]?: Record } + +export type ReadOutcome = + | { status: 'ready'; asset: ReadableAsset; kind: ReaderKind; result: ReadResult } + | { + status: 'pending' + asset: ReadableAsset + kind: ReaderKind + taskId: string + retryAfterSeconds: number + } + +/** Lifetime of image links handed to callers outside the server. */ +export const READ_URL_EXPIRES_SECONDS = 3600 + +const assetWithStorage = { storageKey: true } satisfies Prisma.AssetInclude +type AssetRecord = Prisma.AssetGetPayload<{ include: typeof assetWithStorage }> + +function toReadable(asset: AssetRecord): ReadableAsset { + return { + id: asset.id, + name: asset.name, + projectId: asset.projectId, + mediaType: asset.mediaType, + mediaTypeLower: asset.mediaType?.toLowerCase() || '', + filenameLower: asset.name?.toLowerCase() || '', + storageKey: asset.storageKey?.key ?? null, + media: asset.media as PrismaJson.MediaInfo | null, + } +} + +function hasContent(value: unknown): boolean { + return ( + value !== null && + value !== undefined && + typeof value === 'object' && + Object.values(value).some((v) => v !== null && v !== undefined) + ) +} + +function notFoundRender(): HTTPException { + return new HTTPException(404, { message: 'Render not found' }) +} + +/** + * Shared pipeline behind `read_asset` and `get_asset_render`: access checks, version stacks, + * reader selection, option validation, annotations, render tasks, and waiting. Readers in + * `./readers` only contain the type-specific logic. + */ +export class AssetReadService { + async read( + access: ReadAccess, + request: ReadAssetRequest, + opts: { waitBudgetMs: number }, + ): Promise { + await this.checkAccess(access, request.assetId) + const asset = await this.loadLatestVersion(request.assetId) + + const reader = findReader(asset) + if (!reader) { + throw badRequest( + `Unsupported asset type for read_asset. Asset ${asset.id} has media type "${asset.mediaTypeLower || 'unknown'}". Use download_asset to download this file to disk.`, + ) + } + + // Reject configs meant for another type (empty objects like `imageConfig: {}` are ignored). + const configKey = readerOptionSchemas[reader.kind].key + for (const other of readers) { + const otherKey = readerOptionSchemas[other.kind].key + if (otherKey !== configKey && hasContent(request[otherKey])) { + throw badRequest( + `Asset ${asset.id} is a ${reader.label}. ${reader.configHint}; do not pass ${otherKey}.`, + ) + } + } + + const options = request[configKey] + reader.validate(asset, options as never, { hasAnnotation: Boolean(request.annotationId) }) + + const annotation = request.annotationId + ? { + annotationId: request.annotationId, + ...(await resolveAnnotationsById(asset.id, request.annotationId)), + } + : null + + const plan = await reader.plan(asset, options as never, { + annotation, + includeContent: request.includeContent !== false, + }) + if (plan.status === 'ready') { + return { status: 'ready', asset, kind: reader.kind, result: plan.result } + } + + const projectId = asset.projectId || 'none' + const task = await prisma.workflowTask.create({ + data: { + assetId: asset.id, + projectId, + type: plan.task.type, + status: WorkflowTaskStatus.pending, + payload: { + projectId, + ...plan.task.spec, + assetRead: { + reader: reader.kind, + options: options ?? undefined, + annotationId: request.annotationId, + context: plan.context, + requestedBy: access.kind === 'user' ? access.userId : undefined, + }, + }, + }, + }) + return this.finish(task, asset, opts.waitBudgetMs) + } + + /** Completes a read that returned `pending`. Only the requesting user can fetch it. */ + async getRender( + userId: string, + taskId: string, + opts: { waitBudgetMs: number }, + ): Promise { + const task = await prisma.workflowTask.findUnique({ where: { id: taskId } }) + const request = task?.payload?.assetRead + if (!task || !request || !task.assetId || !userId || request.requestedBy !== userId) { + throw notFoundRender() + } + try { + await authzService.hasPermission({ + user: { id: userId }, + permission: Permission.Read, + type: ResourceType.Asset, + id: task.assetId, + }) + } catch { + // Don't reveal that the task exists to someone who can no longer see the asset. + throw notFoundRender() + } + const record = await prisma.asset.findUnique({ + where: { id: task.assetId }, + include: assetWithStorage, + }) + if (!record) throw notFoundRender() + return this.finish(task, toReadable(record), opts.waitBudgetMs) + } + + private async finish( + task: WorkflowTask, + asset: ReadableAsset, + waitBudgetMs: number, + ): Promise { + const request = task.payload!.assetRead! + const reader = getReader(request.reader) + const { task: latest, done } = await workflowService.waitFor(task, waitBudgetMs) + if (!done) { + return { + status: 'pending', + asset, + kind: reader.kind, + taskId: task.id, + retryAfterSeconds: reader.kind === 'video' || reader.kind === 'document' ? 5 : 3, + } + } + if (!reader.collect) { + throw new Error(`Reader "${reader.kind}" cannot collect render results`) + } + const result = reader.collect( + asset, + request.options as never, + latest.output ?? {}, + request.context ?? {}, + ) + return { status: 'ready', asset, kind: reader.kind, result } + } + + private async checkAccess(access: ReadAccess, assetId: string): Promise { + if (access.kind === 'user') { + if (!access.userId) { + throw new HTTPException(401, { message: 'User ID is required for authorization.' }) + } + await authzService.hasPermission({ + user: { id: access.userId }, + permission: Permission.Read, + type: ResourceType.Asset, + id: assetId, + }) + return + } + + // Autofill system agent: no user; confined to its team and target asset. + if (!access.targetAssetId) { + throw new HTTPException(403, { + message: 'Target asset ID is required for autofill agent authorization.', + }) + } + if (!access.teamId) { + throw new HTTPException(403, { + message: 'Team ID is required for autofill agent authorization.', + }) + } + const asset = await prisma.asset.findUnique({ + where: { id: assetId }, + include: { project: true, teamRootFolder: true }, + }) + if (!asset) { + throw new HTTPException(404, { message: `Asset with ID ${assetId} not found.` }) + } + const assetTeamId = asset.project?.teamId || asset.teamRootFolder?.id + if (!assetTeamId || assetTeamId !== access.teamId) { + throw new HTTPException(403, { + message: `Access denied: asset ${assetId} does not belong to team ${access.teamId}.`, + }) + } + if (asset.id !== access.targetAssetId && asset.parentId !== access.targetAssetId) { + throw new HTTPException(403, { + message: `Access denied: autofill agent can only read target asset ${access.targetAssetId}.`, + }) + } + } + + private async loadLatestVersion(assetId: string): Promise { + let asset = await prisma.asset.findUnique({ + where: { id: assetId }, + include: assetWithStorage, + }) + if (!asset) { + throw new HTTPException(404, { message: `Asset with ID ${assetId} not found.` }) + } + if (asset.type === AssetType.version_stack) { + const latest = await prisma.asset.findFirst({ + where: { parentId: asset.id, isDeleted: false }, + orderBy: { sortIndex: 'asc' }, + include: assetWithStorage, + }) + if (latest) asset = latest + } + return toReadable(asset) + } +} + +export const assetReadService = new AssetReadService() + +/** + * Converts a read outcome into the Operation output. Storage keys are included only for + * Shumai's built-in agent; every image gets a short-lived link. + */ +export async function toAssetReadOutput( + outcome: ReadOutcome, + opts: { includeKeys: boolean }, +): Promise { + const asset = { + id: outcome.asset.id, + name: outcome.asset.name, + kind: outcome.kind, + mediaType: outcome.asset.mediaType, + } + if (outcome.status === 'pending') { + return { + status: 'pending', + summary: `Rendering is still in progress for "${outcome.asset.name}".`, + asset, + details: [], + images: [], + taskId: outcome.taskId, + retryAfterSeconds: outcome.retryAfterSeconds, + } + } + + const bucket = storageBucket() + const { result } = outcome + const images = await Promise.all( + result.images.map(async (image) => ({ + label: image.label, + mimeType: image.mimeType, + url: await s3Service.presign( + bucket, + image.key, + 'GET', + false, + undefined, + false, + READ_URL_EXPIRES_SECONDS, + ), + ...(opts.includeKeys ? { key: image.key } : {}), + })), + ) + const text = result.text + ? { + content: result.text.content, + truncated: result.text.truncated, + size: result.text.size, + mimeType: result.text.mimeType, + ...(opts.includeKeys + ? { + key: result.text.key, + url: await s3Service.presign( + bucket, + result.text.key, + 'GET', + true, + outcome.asset.name || undefined, + false, + READ_URL_EXPIRES_SECONDS, + ), + } + : {}), + } + : undefined + + return { + status: 'ready', + summary: result.summary, + asset, + details: result.details, + images, + ...(text ? { text } : {}), + } +} diff --git a/packages/core/src/asset-read/index.ts b/packages/core/src/asset-read/index.ts new file mode 100644 index 00000000..528511ae --- /dev/null +++ b/packages/core/src/asset-read/index.ts @@ -0,0 +1,13 @@ +export { + assetReadService, + AssetReadService, + READ_URL_EXPIRES_SECONDS, + toAssetReadOutput, + type ReadAccess, + type ReadAssetRequest, + type ReadOutcome, +} from './asset-read' +export { resolveAnnotationsById } from './annotations' +export { findReader, getReader, readers } from './registry' +export { isPlainTextAsset, MAX_TEXT_BYTES, MAX_TEXT_LINES } from './readers/document' +export type { AssetReader, ReadableAsset, ReadImage, ReadPlan, ReadResult } from './types' diff --git a/packages/core/src/asset-read/readers/document.ts b/packages/core/src/asset-read/readers/document.ts new file mode 100644 index 00000000..50e57bf3 --- /dev/null +++ b/packages/core/src/asset-read/readers/document.ts @@ -0,0 +1,188 @@ +import { MAX_RENDER_ITEMS } from '@shumai/operations' +import { WorkflowTaskType } from '@shumai/db' +import { getFileMimeType } from '@shumai/core/src/utils/file-mime' +import { s3Service } from '@shumai/core/src/s3/s3' +import { badRequest, storageBucket, type AssetReader } from '../types' + +export const MAX_TEXT_BYTES = 50 * 1024 +export const MAX_TEXT_LINES = 1000 + +const TEXT_MEDIA_TYPES = new Set([ + 'application/json', + 'application/javascript', + 'application/typescript', + 'application/xml', + 'application/x-yaml', +]) + +const TEXT_EXTENSIONS = [ + '.txt', + '.md', + '.markdown', + '.json', + '.csv', + '.tsv', + '.js', + '.ts', + '.tsx', + '.jsx', + '.html', + '.htm', + '.css', + '.scss', + '.yaml', + '.yml', + '.xml', + '.py', + '.rb', + '.go', + '.rs', + '.c', + '.cpp', + '.h', + '.hpp', + '.java', + '.sh', + '.bash', + '.zsh', + '.sql', +] + +export function isPlainTextAsset(mediaType: string, filename: string): boolean { + if (mediaType.startsWith('text/') || TEXT_MEDIA_TYPES.has(mediaType)) { + return true + } + return TEXT_EXTENSIONS.some((ext) => filename.endsWith(ext)) +} + +/** Truncate to the line and byte limits; returns the kept text and whether anything was cut. */ +export function truncateText(raw: string): { content: string; truncated: boolean } { + let content = raw + let truncated = false + const lines = content.split('\n') + if (lines.length > MAX_TEXT_LINES) { + content = lines.slice(0, MAX_TEXT_LINES).join('\n') + truncated = true + } + if (Buffer.byteLength(content, 'utf-8') > MAX_TEXT_BYTES) { + content = Buffer.from(content, 'utf-8').subarray(0, MAX_TEXT_BYTES).toString('utf-8') + truncated = true + } + return { content, truncated } +} + +/** PDFs and text files: rendered page images ("pages") or raw text content ("text"). */ +export const documentReader: AssetReader<'document'> = { + kind: 'document', + label: 'document', + configHint: "Pass 'docConfig: { mode, startPage, endPage }'", + + matches: (asset) => + asset.media?.proxyType === 'pdf' || + asset.mediaTypeLower === 'application/pdf' || + isPlainTextAsset(asset.mediaTypeLower, asset.filenameLower), + + validate(asset, options, { hasAnnotation }) { + if (!options) { + throw badRequest( + 'docConfig with mode ("pages" or "text") is required when inspecting document/PDF assets.', + ) + } + const startPage = options.startPage ?? null + const endPage = options.endPage ?? null + + if (options.mode === 'text') { + if (hasAnnotation) { + throw badRequest( + 'annotationId cannot be used with docConfig mode "text" because visual markups can only be rendered on visual pages. Use docConfig: { mode: "pages", startPage: ..., endPage: ... } instead.', + ) + } + if (startPage !== null || endPage !== null) { + throw badRequest( + 'startPage and endPage must be null when docConfig mode is "text". To inspect specific page numbers, use mode: "pages".', + ) + } + if (!isPlainTextAsset(asset.mediaTypeLower, asset.filenameLower)) { + throw badRequest( + `Asset "${asset.name}" is a binary PDF/document and cannot be read as raw text. Please call read_asset with docConfig: { mode: "pages", startPage: 1, endPage: ... } to view its visual pages.`, + ) + } + return + } + + if (startPage === null || endPage === null) { + throw badRequest('startPage and endPage are required when docConfig mode is "pages".') + } + if (startPage < 1) { + throw badRequest(`Invalid page range: start page (${startPage}) must be at least 1.`) + } + if (startPage > endPage) { + throw badRequest( + `Invalid page range: start page (${startPage}) must be less than or equal to end page (${endPage}).`, + ) + } + const requested = endPage - startPage + 1 + if (requested > MAX_RENDER_ITEMS) { + throw badRequest( + `Page range (${requested} pages requested) exceeds the maximum limit of ${MAX_RENDER_ITEMS} pages per request.`, + ) + } + }, + + async plan(asset, options, { annotation, includeContent }) { + // validate() guarantees options are present and consistent + const opts = options! + if (opts.mode === 'text') { + const key = asset.storageKey + if (!key) { + throw new Error(`No media content found for asset ${asset.id}.`) + } + const mimeType = asset.mediaType || getFileMimeType(null, asset.name || key, 'text/plain') + let text = { content: '', truncated: false, size: 0, key, mimeType } + if (includeContent) { + const { buffer } = await s3Service.getObject(storageBucket(), key) + text = { ...truncateText(buffer.toString('utf-8')), size: buffer.length, key, mimeType } + } + return { + status: 'ready', + result: { + summary: `Document "${asset.name}" (ID: ${asset.id}):`, + details: [], + images: [], + text, + }, + } + } + + return { + status: 'render', + task: { + type: WorkflowTaskType.transcode_pdf_pages, + spec: { + pdfPages: { + start: opts.startPage!, + end: opts.endPage!, + commentTimestamp: annotation?.timestamp ?? null, + annotations: annotation?.annotations ?? null, + }, + }, + }, + } + }, + + collect(asset, options, output) { + const pages = output.pages as Array<{ key: string; page: number }> | undefined + if (!pages || pages.length === 0) { + throw new Error('No PDF page images were generated.') + } + return { + summary: `Rendered ${pages.length} page(s) of "${asset.name}" (Pages ${options!.startPage} to ${options!.endPage}):`, + details: [], + images: pages.map((page) => ({ + key: page.key, + label: `Page ${page.page}`, + mimeType: getFileMimeType(null, page.key, 'image/webp'), + })), + } + }, +} diff --git a/packages/core/src/asset-read/readers/image.ts b/packages/core/src/asset-read/readers/image.ts new file mode 100644 index 00000000..b8dbc903 --- /dev/null +++ b/packages/core/src/asset-read/readers/image.ts @@ -0,0 +1,61 @@ +import { WorkflowTaskType } from '@shumai/db' +import { getFileMimeType } from '@shumai/core/src/utils/file-mime' +import type { AssetReader } from '../types' + +/** Images (including PSD): the web proxy, or a copy with the annotation drawn on top. */ +export const imageReader: AssetReader<'image'> = { + kind: 'image', + label: 'image', + configHint: "Pass 'imageConfig: {}' or no config", + + matches: (asset) => + asset.media?.proxyType === 'image' || + asset.mediaTypeLower.startsWith('image/') || + asset.filenameLower.endsWith('.psd'), + + validate() {}, + + async plan(asset, _options, { annotation }) { + const key = asset.media?.imageTranscodes?.[0]?.key || asset.storageKey + if (!key) { + throw new Error('No media content found for this asset.') + } + + if (annotation?.annotations && annotation.annotations.length > 0) { + return { + status: 'render', + task: { + type: WorkflowTaskType.transcode_image_annotation, + spec: { imageAnnotation: { annotations: annotation.annotations } }, + }, + } + } + + return { + status: 'ready', + result: { + summary: `Image asset "${asset.name}" (ID: ${asset.id}):`, + details: [], + images: [ + { + key, + label: 'Image', + mimeType: getFileMimeType(null, key, asset.mediaType || 'image/png'), + }, + ], + }, + } + }, + + collect(asset, _options, output) { + const key = typeof output.key === 'string' ? output.key : null + if (!key) { + throw new Error('Image annotation transcode workflow failed to output S3 key.') + } + return { + summary: `Image asset "${asset.name}" (ID: ${asset.id}) with visual markups overlaid:`, + details: [], + images: [{ key, label: 'Image with markups', mimeType: 'image/webp' }], + } + }, +} diff --git a/packages/core/src/asset-read/readers/model3d.ts b/packages/core/src/asset-read/readers/model3d.ts new file mode 100644 index 00000000..e9c2c5b5 --- /dev/null +++ b/packages/core/src/asset-read/readers/model3d.ts @@ -0,0 +1,112 @@ +import { WorkflowTaskType } from '@shumai/db' +import { is3dModel } from '@shumai/core/src/utils/mime' +import { + degreeToSecond, + isValidTurntableDegree, + secondToDegree, +} from '@shumai/core/src/utils/model3d' +import { badRequest, type AssetReader, type ReadResult } from '../types' + +const QUADRANT_LEGEND = [ + 'Orthogonal 4-view composite image (2x2 grid across 360° rotation):', + '- Top-Left quadrant: Front view (0°)', + '- Top-Right quadrant: Right side view (90°)', + '- Bottom-Left quadrant: Back view (180°)', + '- Bottom-Right quadrant: Left side view (270°)', +] + +function angleSummary(name: string, id: string, degree: number, hasMarkup: boolean): string { + return hasMarkup + ? `3D asset "${name}" (ID: ${id}) view at ${degree}° with visual markups overlaid:` + : `3D asset "${name}" (ID: ${id}) view at ${degree}°:` +} + +/** 3D models: the 4-view composite, or one turntable angle rendered from the turntable video. */ +export const model3dReader: AssetReader<'model3d'> = { + kind: 'model3d', + label: '3D model', + configHint: "To inspect specific angles pass 'config3d: { degree: ... }'", + + matches: (asset) => + asset.media?.proxyType === '3d' || is3dModel(asset.mediaTypeLower, asset.filenameLower), + + validate(_asset, options) { + const degree = options?.degree + if (degree !== undefined && degree !== null && !isValidTurntableDegree(degree)) { + throw badRequest( + `Invalid turntable degree: ${degree}. Degree must be an integer multiple of 15 between 0 and 345 (e.g. 0, 15, 30, 45, ..., 345).`, + ) + } + }, + + async plan(asset, options, { annotation }) { + const requestedDegree = options?.degree ?? null + const annotations = annotation?.annotations ?? null + const commentTimestamp = annotation?.timestamp ?? null + + if (annotation && commentTimestamp !== null) { + const commentDegree = secondToDegree(commentTimestamp) + if (requestedDegree !== null && requestedDegree !== commentDegree) { + throw badRequest( + `Annotation "${annotation.annotationId}" was drawn at ${commentDegree}°, but degree ${requestedDegree}° was specified. Do not pass a conflicting degree with annotationId.`, + ) + } + } + + if (requestedDegree !== null || annotation) { + if (!asset.media?.videoTranscodes || asset.media.videoTranscodes.length === 0) { + throw new Error( + `Asset ${asset.id} is a 3D model, but its turntable video (turntable.mp4) is missing.`, + ) + } + const second = + requestedDegree !== null ? degreeToSecond(requestedDegree) : (commentTimestamp ?? 0) + const degree = requestedDegree ?? secondToDegree(second) + return { + status: 'render', + task: { + type: WorkflowTaskType.transcode_screenshot, + spec: { + screenshot: { + start: second, + end: second, + count: 1, + commentTimestamp: second, + annotations: annotations ?? undefined, + }, + }, + }, + context: { degree, hasMarkup: Boolean(annotations && annotations.length > 0) }, + } + } + + const key = asset.media?.imageTranscodes?.[0]?.key + if (!asset.media?.imageTranscodes || asset.media.imageTranscodes.length === 0) { + throw new Error( + `Asset ${asset.id} is a 3D model, but its 4-views image (views.webp) is missing.`, + ) + } + if (!key) { + throw new Error(`Asset ${asset.id} is a 3D model, but its 4-views image key is missing.`) + } + const result: ReadResult = { + summary: `3D asset "${asset.name}" (ID: ${asset.id}):`, + details: QUADRANT_LEGEND, + images: [{ key, label: '4-view composite', mimeType: 'image/webp' }], + } + return { status: 'ready', result } + }, + + collect(asset, _options, output, context) { + const degree = Number(context.degree) + const screenshots = output.screenshots as Array<{ key: string }> | undefined + if (!screenshots || screenshots.length === 0) { + throw new Error(`Failed to extract 3D turntable frame for asset ${asset.id} at ${degree}°.`) + } + return { + summary: angleSummary(asset.name, asset.id, degree, context.hasMarkup === true), + details: [], + images: [{ key: screenshots[0].key, label: `View at ${degree}°`, mimeType: 'image/webp' }], + } + }, +} diff --git a/packages/core/src/asset-read/readers/readers.test.ts b/packages/core/src/asset-read/readers/readers.test.ts new file mode 100644 index 00000000..ecc9ad95 --- /dev/null +++ b/packages/core/src/asset-read/readers/readers.test.ts @@ -0,0 +1,223 @@ +import { describe, expect, it } from 'vitest' +import { READER_KINDS, readerOptionSchemas } from '@shumai/operations' +import { findReader, getReader, readers } from '../registry' +import type { ReadableAsset } from '../types' +import { documentReader, isPlainTextAsset, MAX_TEXT_BYTES, truncateText } from './document' +import { imageReader } from './image' +import { model3dReader } from './model3d' +import { videoReader } from './video' + +function asset(overrides: Partial & { name: string }): ReadableAsset { + const mediaType = overrides.mediaType ?? null + return { + id: 'asset-1', + projectId: 'project-1', + storageKey: null, + media: null, + mediaType, + mediaTypeLower: mediaType?.toLowerCase() ?? '', + filenameLower: overrides.name.toLowerCase(), + ...overrides, + } +} + +const noAnnotation = { annotation: null, includeContent: true } + +describe('reader registry', () => { + it('has exactly one reader per declared kind, each with a config field', () => { + expect(readers.map((r) => r.kind).sort()).toEqual([...READER_KINDS].sort()) + for (const kind of READER_KINDS) { + expect(getReader(kind).kind).toBe(kind) + expect(readerOptionSchemas[kind].key).toMatch(/Config$|^config3d$/) + } + }) + + it.each([ + [{ name: 'robot.glb', mediaType: 'model/gltf-binary' }, 'model3d'], + // 3D models also have image transcodes; the 3D reader must win. + [{ name: 'robot.png', mediaType: 'image/png', media: { proxyType: '3d' } }, 'model3d'], + [{ name: 'photo.JPG', mediaType: 'IMAGE/JPEG' }, 'image'], + [{ name: 'layers.psd', mediaType: 'application/octet-stream' }, 'image'], + [{ name: 'clip.webm', mediaType: 'video/webm' }, 'video'], + [{ name: 'slides.pdf', mediaType: 'application/pdf' }, 'document'], + [{ name: 'data.csv', mediaType: null }, 'document'], + [{ name: 'config.json', mediaType: 'application/json' }, 'document'], + ] as const)('%o is read by %s', (overrides, kind) => { + expect(findReader(asset(overrides as Partial & { name: string }))?.kind).toBe( + kind, + ) + }) + + it('has no reader for archives', () => { + expect(findReader(asset({ name: 'bundle.zip', mediaType: 'application/zip' }))).toBeUndefined() + }) +}) + +describe('imageReader', () => { + it('prefers the web proxy over the original', async () => { + const plan = await imageReader.plan( + asset({ + name: 'photo.png', + mediaType: 'image/png', + storageKey: 'raw/photo.png', + media: { imageTranscodes: [{ key: 'proxy/photo.webp' }] } as PrismaJson.MediaInfo, + }), + undefined, + noAnnotation, + ) + expect(plan).toMatchObject({ + status: 'ready', + result: { images: [{ key: 'proxy/photo.webp', mimeType: 'image/webp' }] }, + }) + }) + + it('fails without any stored content', async () => { + await expect( + imageReader.plan(asset({ name: 'photo.png' }), undefined, noAnnotation), + ).rejects.toThrow('No media content found') + }) + + it('requires the annotated key from the render', () => { + expect(() => imageReader.collect!(asset({ name: 'photo.png' }), undefined, {}, {})).toThrow( + 'failed to output S3 key', + ) + }) +}) + +describe('videoReader', () => { + const video = asset({ name: 'clip.mp4', mediaType: 'video/mp4' }) + + it.each([ + [{ start: -1, end: 2, count: 1 }, 'must be non-negative'], + [{ start: 3, end: 2, count: 1 }, 'must be less than or equal to end'], + [{ start: 0, end: 2, count: 0 }, 'must be at least 1'], + [{ start: 0, end: 2, count: 21 }, 'exceeds the maximum of 20 frames'], + ])('rejects %o', (options, message) => { + expect(() => videoReader.validate(video, options, { hasAnnotation: false })).toThrow(message) + }) + + it('accepts exactly 20 frames', () => { + expect(() => + videoReader.validate(video, { start: 0, end: 60, count: 20 }, { hasAnnotation: false }), + ).not.toThrow() + }) + + it('renders frames through the screenshot workflow with the annotation', async () => { + const plan = await videoReader.plan( + video, + { start: 1, end: 3, count: 2 }, + { + annotation: { annotationId: 'c1', annotations: [], timestamp: 2 }, + includeContent: true, + }, + ) + expect(plan).toEqual({ + status: 'render', + task: { + type: 'transcode_screenshot', + spec: { + screenshot: { start: 1, end: 3, count: 2, commentTimestamp: 2, annotations: [] }, + }, + }, + }) + }) + + it('fails when the render produced no frames', () => { + expect(() => + videoReader.collect!(video, { start: 0, end: 1, count: 1 }, { screenshots: [] }, {}), + ).toThrow('No screenshots were generated.') + }) +}) + +describe('model3dReader', () => { + const model = asset({ + name: 'robot.glb', + mediaType: 'model/gltf-binary', + media: { + proxyType: '3d', + videoTranscodes: [{ key: 'proxy/turntable.mp4' }], + imageTranscodes: [{ key: 'proxy/views.webp' }], + } as unknown as PrismaJson.MediaInfo, + }) + + it('accepts null as "show the overview"', async () => { + expect(() => + model3dReader.validate(model, { degree: null }, { hasAnnotation: false }), + ).not.toThrow() + const plan = await model3dReader.plan(model, { degree: null }, noAnnotation) + expect(plan.status).toBe('ready') + }) + + it('carries the angle to collect through the task context', async () => { + const plan = await model3dReader.plan(model, { degree: 180 }, noAnnotation) + expect(plan).toMatchObject({ + status: 'render', + context: { degree: 180, hasMarkup: false }, + }) + const result = model3dReader.collect!( + model, + { degree: 180 }, + { screenshots: [{ key: 'shots/180.webp' }] }, + { degree: 180, hasMarkup: false }, + ) + expect(result.images).toEqual([ + { key: 'shots/180.webp', label: 'View at 180°', mimeType: 'image/webp' }, + ]) + }) +}) + +describe('documentReader', () => { + const pdf = asset({ name: 'slides.pdf', mediaType: 'application/pdf' }) + + it.each([ + [undefined, 'docConfig with mode'], + [{ mode: 'pages' as const }, 'startPage and endPage are required'], + [{ mode: 'pages' as const, startPage: 0, endPage: 1 }, 'must be at least 1'], + [{ mode: 'pages' as const, startPage: 3, endPage: 2 }, 'less than or equal to end page'], + [{ mode: 'pages' as const, startPage: 1, endPage: 21 }, 'exceeds the maximum limit of 20'], + [{ mode: 'text' as const }, 'cannot be read as raw text'], + ])('rejects %o for PDFs', (options, message) => { + expect(() => documentReader.validate(pdf, options, { hasAnnotation: false })).toThrow(message) + }) + + it('renders a page range with annotations', async () => { + const plan = await documentReader.plan( + pdf, + { mode: 'pages', startPage: 2, endPage: 3 }, + { annotation: { annotationId: 'c1', annotations: null, timestamp: 2 }, includeContent: true }, + ) + expect(plan).toMatchObject({ + status: 'render', + task: { + type: 'transcode_pdf_pages', + spec: { pdfPages: { start: 2, end: 3, commentTimestamp: 2, annotations: null } }, + }, + }) + }) + + it('labels rendered pages by page number', () => { + const result = documentReader.collect!( + pdf, + { mode: 'pages', startPage: 2, endPage: 3 }, + { pages: [{ key: 'p/2.webp', page: 2 }] }, + {}, + ) + expect(result.summary).toBe('Rendered 1 page(s) of "slides.pdf" (Pages 2 to 3):') + expect(result.images).toEqual([{ key: 'p/2.webp', label: 'Page 2', mimeType: 'image/webp' }]) + }) + + it('detects plain text by media type or extension', () => { + expect(isPlainTextAsset('text/plain', 'a.bin')).toBe(true) + expect(isPlainTextAsset('application/x-yaml', 'a')).toBe(true) + expect(isPlainTextAsset('', 'main.rs')).toBe(true) + expect(isPlainTextAsset('application/pdf', 'a.pdf')).toBe(false) + }) + + it('truncates by bytes as well as lines', () => { + const big = 'x'.repeat(MAX_TEXT_BYTES + 10) + const { content, truncated } = truncateText(big) + expect(truncated).toBe(true) + expect(Buffer.byteLength(content)).toBe(MAX_TEXT_BYTES) + expect(truncateText('short')).toEqual({ content: 'short', truncated: false }) + }) +}) diff --git a/packages/core/src/asset-read/readers/video.ts b/packages/core/src/asset-read/readers/video.ts new file mode 100644 index 00000000..a40a660c --- /dev/null +++ b/packages/core/src/asset-read/readers/video.ts @@ -0,0 +1,75 @@ +import { MAX_RENDER_ITEMS } from '@shumai/operations' +import { WorkflowTaskType } from '@shumai/db' +import { getFileMimeType } from '@shumai/core/src/utils/file-mime' +import { badRequest, type AssetReader } from '../types' + +/** Videos: evenly spaced frames between `start` and `end`, rendered by the screenshot workflow. */ +export const videoReader: AssetReader<'video'> = { + kind: 'video', + label: 'video', + configHint: "Pass 'videoConfig: { start, end, count }'", + + matches: (asset) => + asset.media?.proxyType === 'video' || asset.mediaTypeLower.startsWith('video/'), + + validate(_asset, options) { + if (!options) { + throw badRequest( + 'videoConfig with start, end, and count is required when inspecting video assets.', + ) + } + const { start, end, count } = options + if (start < 0 || end < 0) { + throw badRequest('Invalid video time range: start and end must be non-negative numbers.') + } + if (start > end) { + throw badRequest( + `Invalid video time range: start (${start}) must be less than or equal to end (${end}).`, + ) + } + if (count < 1) { + throw badRequest(`Invalid frame count: count (${count}) must be at least 1.`) + } + if (count > MAX_RENDER_ITEMS) { + throw badRequest( + `Invalid frame count: count (${count}) exceeds the maximum of ${MAX_RENDER_ITEMS} frames per request. Read a shorter range or fewer frames per call.`, + ) + } + }, + + async plan(_asset, options, { annotation }) { + // validate() guarantees options are present + const { start, end, count } = options! + return { + status: 'render', + task: { + type: WorkflowTaskType.transcode_screenshot, + spec: { + screenshot: { + start, + end, + count, + commentTimestamp: annotation?.timestamp ?? null, + annotations: annotation?.annotations ?? null, + }, + }, + }, + } + }, + + collect(asset, options, output) { + const screenshots = output.screenshots as Array<{ key: string; timestamp: number }> | undefined + if (!screenshots || screenshots.length === 0) { + throw new Error('No screenshots were generated.') + } + return { + summary: `Extracted ${screenshots.length} frame(s) from "${asset.name}" (${options!.start}s - ${options!.end}s):`, + details: [], + images: screenshots.map((shot) => ({ + key: shot.key, + label: `Frame at ${shot.timestamp}s`, + mimeType: getFileMimeType(null, shot.key, 'image/webp'), + })), + } + }, +} diff --git a/packages/core/src/asset-read/registry.ts b/packages/core/src/asset-read/registry.ts new file mode 100644 index 00000000..0ef97529 --- /dev/null +++ b/packages/core/src/asset-read/registry.ts @@ -0,0 +1,42 @@ +import type { ReaderKind } from '@shumai/operations' +import { documentReader } from './readers/document' +import { imageReader } from './readers/image' +import { model3dReader } from './readers/model3d' +import { videoReader } from './readers/video' +import type { AssetReader, ReadableAsset } from './types' + +type MissingKinds = Exclude< + ReaderKind, + TReaders[number]['kind'] +> + +/** Compile-time guard: every reader kind declared in `@shumai/operations` needs a reader. */ +function defineReaderRegistry( + readers: TReaders & + ([MissingKinds] extends [never] + ? unknown + : { missingReaders: MissingKinds }), +): TReaders { + return readers +} + +/** + * Readers in matching order: the first reader whose `matches` returns true handles the asset. + * 3D comes before image (3D models also have image transcodes); documents cover PDFs and text. + */ +export const readers = defineReaderRegistry([ + model3dReader, + imageReader, + videoReader, + documentReader, +] as const) + +export function findReader(asset: ReadableAsset): AssetReader | undefined { + return readers.find((reader) => reader.matches(asset)) as AssetReader | undefined +} + +export function getReader(kind: ReaderKind): AssetReader { + const reader = readers.find((r) => r.kind === kind) + if (!reader) throw new Error(`No reader registered for "${kind}"`) + return reader as AssetReader +} diff --git a/packages/core/src/asset-read/types.ts b/packages/core/src/asset-read/types.ts new file mode 100644 index 00000000..b568935a --- /dev/null +++ b/packages/core/src/asset-read/types.ts @@ -0,0 +1,100 @@ +import type { ReaderKind, ReaderOptions } from '@shumai/operations' +import type { WorkflowTaskType } from '@shumai/db' +import { HTTPException } from 'hono/http-exception' + +/** The asset being read, after version stacks are resolved to their latest version. */ +export interface ReadableAsset { + id: string + name: string + projectId: string | null + /** Media type as stored (may be null). */ + mediaType: string | null + /** Lowercased media type ('' when unknown), for matching. */ + mediaTypeLower: string + /** Lowercased file name, for extension matching. */ + filenameLower: string + storageKey: string | null + media: PrismaJson.MediaInfo | null +} + +export interface ReadAnnotation { + annotationId: string + annotations: PrismaJson.AnnotationList | null + /** Seconds (video, 3D turntable) or page number (PDF) where the markup was drawn. */ + timestamp: number | null +} + +export interface ReadImage { + key: string + /** What the image shows, e.g. "Frame at 30s", "Page 3", "View at 90°". */ + label: string + mimeType: string +} + +export interface ReadResult { + /** One-line description, e.g. `Extracted 4 frame(s) from "clip.mp4" (0s - 60s):`. */ + summary: string + /** Extra description lines (e.g. the 3D quadrant legend). */ + details: string[] + images: ReadImage[] + text?: { content: string; truncated: boolean; size: number; key: string; mimeType: string } +} + +export interface PlanContext { + annotation: ReadAnnotation | null + /** False when the caller only wants a description (skip reading text content). */ + includeContent: boolean +} + +export type RenderSpec = Pick< + PrismaJson.WorkflowTaskPayload, + 'screenshot' | 'imageAnnotation' | 'pdfPages' +> + +export type ReadPlan = + | { status: 'ready'; result: ReadResult } + | { + status: 'render' + task: { type: WorkflowTaskType; spec: RenderSpec } + /** Reader-private values stored on the task and passed back to `collect`. */ + context?: Record + } + +/** + * Reads one family of assets. The shared pipeline (`assetReadService`) handles access checks, + * version stacks, annotations, workflow tasks, and waiting; a reader only decides what to show. + */ +export interface AssetReader { + kind: TKind + /** Shown in errors, e.g. "Asset X is a 3D model." */ + label: string + /** How to pass this reader's options, shown when another reader's config is passed. */ + configHint: string + matches(asset: ReadableAsset): boolean + /** Synchronous argument checks, run before annotations are resolved or anything is rendered. */ + validate( + asset: ReadableAsset, + options: ReaderOptions | undefined, + ctx: { hasAnnotation: boolean }, + ): void + plan( + asset: ReadableAsset, + options: ReaderOptions | undefined, + ctx: PlanContext, + ): Promise + /** Builds the result from a finished render. Required when `plan` can return `render`. */ + collect?( + asset: ReadableAsset, + options: ReaderOptions | undefined, + renderOutput: PrismaJson.WorkflowTaskOutput, + context: Record, + ): ReadResult +} + +export function badRequest(message: string): HTTPException { + return new HTTPException(400, { message }) +} + +export function storageBucket(): string { + return process.env.S3_BUCKET || 'shumai' +} diff --git a/packages/core/src/operations/asset/read-asset.ts b/packages/core/src/operations/asset/read-asset.ts new file mode 100644 index 00000000..30f6c187 --- /dev/null +++ b/packages/core/src/operations/asset/read-asset.ts @@ -0,0 +1,43 @@ +import { getAssetRender, readAsset, type Via } from '@shumai/operations' +import { assetReadService, toAssetReadOutput } from '@shumai/core/src/asset-read' +import { implementOperation } from '../implement' + +/** + * How long each surface waits for a render before returning `pending`. MCP clients commonly + * time out around 60s, so MCP and generic API callers get a short budget. + */ +export const READ_WAIT_BUDGET_MS: Record = { + agent: 60_000, + mcp: 20_000, + cli: 120_000, + api: 20_000, +} + +/** `maxWaitSeconds` can only shorten the surface budget. */ +export function waitBudgetMs(via: Via, maxWaitSeconds: number | undefined): number { + const budget = READ_WAIT_BUDGET_MS[via] + return maxWaitSeconds === undefined ? budget : Math.min(budget, maxWaitSeconds * 1000) +} + +export const readAssetOp = implementOperation(readAsset, { + // assetReadService checks Read on the asset before planning or creating any render task. + authz: () => [], + execute: async (input, ctx) => { + const { maxWaitSeconds, ...request } = input + const outcome = await assetReadService.read({ kind: 'user', userId: ctx.userId }, request, { + waitBudgetMs: waitBudgetMs(ctx.via, maxWaitSeconds), + }) + return toAssetReadOutput(outcome, { includeKeys: ctx.via === 'agent' }) + }, +}) + +export const getAssetRenderOp = implementOperation(getAssetRender, { + // Only the requesting user, with Read on the asset, can fetch a render (checked by the service). + authz: () => [], + execute: async (input, ctx) => { + const outcome = await assetReadService.getRender(ctx.userId, input.taskId, { + waitBudgetMs: waitBudgetMs(ctx.via, input.maxWaitSeconds), + }) + return toAssetReadOutput(outcome, { includeKeys: ctx.via === 'agent' }) + }, +}) diff --git a/packages/core/src/operations/index.ts b/packages/core/src/operations/index.ts index ca80c6cc..b0e9b0d5 100644 --- a/packages/core/src/operations/index.ts +++ b/packages/core/src/operations/index.ts @@ -14,5 +14,11 @@ export { createFolderOp } from './asset/create-folder' export { deleteAssetOp } from './asset/delete-asset' export { listAssetsOp } from './asset/list-assets' export { moveAssetsOp } from './asset/move-assets' +export { + getAssetRenderOp, + READ_WAIT_BUDGET_MS, + readAssetOp, + waitBudgetMs, +} from './asset/read-asset' export { renameAssetOp } from './asset/rename-asset' export { listProjectsOp } from './project/list-projects' diff --git a/packages/core/src/operations/registry.ts b/packages/core/src/operations/registry.ts index 3e0a995e..d839788a 100644 --- a/packages/core/src/operations/registry.ts +++ b/packages/core/src/operations/registry.ts @@ -3,6 +3,7 @@ import { createFolderOp } from './asset/create-folder' import { deleteAssetOp } from './asset/delete-asset' import { listAssetsOp } from './asset/list-assets' import { moveAssetsOp } from './asset/move-assets' +import { getAssetRenderOp, readAssetOp } from './asset/read-asset' import { renameAssetOp } from './asset/rename-asset' import type { Operation } from './implement' import { listProjectsOp } from './project/list-projects' @@ -31,6 +32,8 @@ export const operations = defineRegistry([ renameAssetOp, moveAssetsOp, deleteAssetOp, + readAssetOp, + getAssetRenderOp, ] as const) const operationsByName = new Map() diff --git a/packages/core/src/s3/s3.test.ts b/packages/core/src/s3/s3.test.ts index 165b5d55..6eac56c9 100644 --- a/packages/core/src/s3/s3.test.ts +++ b/packages/core/src/s3/s3.test.ts @@ -94,6 +94,43 @@ describe('S3Service implementations', () => { expect(getSignedUrlSpy).toHaveBeenCalledTimes(1) }) + it('signs with an explicit lifetime and caches it separately from the default', async () => { + const s3 = new S3StorageService('http://localhost:9000', 'key', 'secret', 'test-bucket') + getSignedUrlSpy.mockClear() + getSignedUrlSpy.mockResolvedValueOnce('http://default-url') + getSignedUrlSpy.mockResolvedValueOnce('http://short-url') + + const defaultUrl = await s3.presign('bucket', 'expiry-key', 'GET') + const shortUrl = await s3.presign( + 'bucket', + 'expiry-key', + 'GET', + false, + undefined, + false, + 3600, + ) + const shortAgain = await s3.presign( + 'bucket', + 'expiry-key', + 'GET', + false, + undefined, + false, + 3600, + ) + + expect(defaultUrl).toBe('http://default-url') + expect(shortUrl).toBe('http://short-url') + expect(shortAgain).toBe('http://short-url') + expect(getSignedUrlSpy).toHaveBeenCalledTimes(2) + expect(getSignedUrlSpy).toHaveBeenLastCalledWith( + expect.anything(), + expect.any(GetObjectCommand), + expect.objectContaining({ expiresIn: 3600 }), + ) + }) + it('should NOT cache PUT presign URLs', async () => { const s3 = new S3StorageService('http://localhost:9000', 'key', 'secret', 'test-bucket') diff --git a/packages/core/src/s3/s3.ts b/packages/core/src/s3/s3.ts index bdb5ca6a..12c78331 100644 --- a/packages/core/src/s3/s3.ts +++ b/packages/core/src/s3/s3.ts @@ -112,6 +112,8 @@ export interface S3Service { download?: boolean, filename?: string, internal?: boolean, + /** URL lifetime; defaults to PRESIGNED_URL_EXPIRES_IN hours. Ignored by local storage. */ + expiresInSeconds?: number, ) => Promise presignMultipart: ( bucket: string, @@ -360,12 +362,15 @@ export class S3StorageService implements S3Service { download?: boolean, filename?: string, internal?: boolean, + expiresInSeconds?: number, ): Promise { const expireHours = parseInt(process.env.PRESIGNED_URL_EXPIRES_IN || '5', 10) - const expiresInSeconds = expireHours * 3600 - const cacheTtlMs = expiresInSeconds * 1000 + const lifetimeSeconds = expiresInSeconds ?? expireHours * 3600 + // With an explicit (usually short) lifetime, only reuse a cached URL while it has at + // least half of its lifetime left, so callers never receive a nearly expired link. + const cacheTtlMs = expiresInSeconds ? (lifetimeSeconds * 1000) / 2 : lifetimeSeconds * 1000 - const cacheKey = `${bucket}/${key}?download=${download || false}&filename=${filename || ''}&internal=${internal || false}` + const cacheKey = `${bucket}/${key}?download=${download || false}&filename=${filename || ''}&internal=${internal || false}&expires=${expiresInSeconds ?? 'default'}` if (method === 'GET' && !download) { const cached = this.presignCache.get(cacheKey) if (cached) { @@ -382,19 +387,19 @@ export class S3StorageService implements S3Service { Key: key, ResponseContentDisposition: download ? buildContentDisposition(filename) : undefined, }) - url = await getSignedUrl(signingClient, command, { expiresIn: expiresInSeconds }) + url = await getSignedUrl(signingClient, command, { expiresIn: lifetimeSeconds }) } else if (method === 'PUT') { const command = new PutObjectCommand({ Bucket: bucket, Key: key, }) - url = await getSignedUrl(signingClient, command, { expiresIn: expiresInSeconds }) + url = await getSignedUrl(signingClient, command, { expiresIn: lifetimeSeconds }) } else if (method === 'DELETE') { const command = new DeleteObjectCommand({ Bucket: bucket, Key: key, }) - url = await getSignedUrl(signingClient, command, { expiresIn: expiresInSeconds }) + url = await getSignedUrl(signingClient, command, { expiresIn: lifetimeSeconds }) } else { throw new Error(`Unsupported method for presign: ${method}`) } diff --git a/packages/db/src/prisma-json-types.ts b/packages/db/src/prisma-json-types.ts index bb0e987b..753a317b 100644 --- a/packages/db/src/prisma-json-types.ts +++ b/packages/db/src/prisma-json-types.ts @@ -359,10 +359,27 @@ declare global { screenshot?: ScreenshotSpec imageAnnotation?: ImageAnnotationSpec pdfPages?: PdfPagesSpec + /** Render that produced this result, if any (set by the read_asset pipeline). */ + assetRead?: AssetReadRequest watermark?: WatermarkTaskPayload recoveryAttempts?: number } + /** + * Records which reader started a render so the read can be completed later from the task + * alone (see `get_asset_render`), on any server instance. + */ + export interface AssetReadRequest { + reader: 'model3d' | 'image' | 'video' | 'document' + /** The reader's validated options. */ + options?: Record + annotationId?: string + /** Reader-private values computed while planning (e.g. the resolved 3D angle). */ + context?: Record + /** User who requested the read; undefined for system agents (autofill). */ + requestedBy?: string + } + export interface WorkflowTaskOutput { error?: string recoveryAttempts?: number diff --git a/packages/e2e/workflow/read-asset.test.ts b/packages/e2e/workflow/read-asset.test.ts new file mode 100644 index 00000000..dc6af328 --- /dev/null +++ b/packages/e2e/workflow/read-asset.test.ts @@ -0,0 +1,292 @@ +import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest' +import { prisma, type User } from '@shumai/db' +import { setupTestDbHooks } from '@shumai/db/test' +import { workflowService, TaskQueueAgent, TaskQueueTranscode } from '@shumai/workflow-core' +import { initAgentWorkflows } from '@shumai/agent' +import { initTranscodeWorkflows } from '@shumai/transcode' +import { s3Service } from '@shumai/core/src/s3/s3' +import { projectService } from '@shumai/core/src/project/project' +import { getAssetRenderOp, readAssetOp, runOperation } from '@shumai/core/src/operations' +import { AgentHarness } from '@earendil-works/pi-agent-core' +import { fileURLToPath } from 'url' +import * as path from 'path' +import * as fs from 'fs' + +const BUCKET = 'shumai-e2e-test-bucket-read-asset' +const fixturesDir = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../fixtures') + +type ToolResult = { content: Array<{ type: string; text?: string; data?: string }> } +type ExecutableTool = { + execute: (id: string, params: Record) => Promise +} + +/** + * read_asset end to end: real render workflows (frames, PDF pages) through the reader + * pipeline, the pending → get_asset_render path, and the agent tool in a chat workflow. + * Only the model call is mocked. + */ +describe('Workflow E2E - read_asset', () => { + setupTestDbHooks() + + let videoAssetId = '' + + beforeAll(async () => { + process.env.S3_BUCKET = BUCKET + process.env.GEMINI_API_KEY = 'dummy-key' + + initTranscodeWorkflows() + initAgentWorkflows() + + vi.spyOn(AgentHarness.prototype, 'prompt').mockImplementation(async function ( + this: AgentHarness, + ) { + const readAsset = this.getTools().find((t) => t.name === 'read_asset') as unknown as + ExecutableTool | undefined + const hasRenderTool = this.getTools().some((t) => t.name === 'get_asset_render') + if (!readAsset) { + return { + content: [{ type: 'text', text: 'MISSING_READ_ASSET' }], + usage: { input: 1, output: 1 }, + stopReason: 'stop', + } as unknown as Awaited> + } + const result = await readAsset.execute('call-1', { + assetId: videoAssetId, + videoConfig: { start: 0, end: 1, count: 2 }, + }) + const text = result.content.find((c) => c.type === 'text')?.text ?? '' + const images = result.content.filter((c) => c.type === 'image' && c.data) + const summary = [ + `images:${images.length}`, + `frames:${(text.match(/- Frame at /g) ?? []).length}`, + `render_tool:${hasRenderTool ? 'present' : 'absent'}`, + ].join('|') + return { + content: [{ type: 'text', text: summary }], + usage: { input: 10, output: 20 }, + stopReason: 'stop', + } as unknown as Awaited> + }) + + await workflowService.startWorkers([TaskQueueTranscode, TaskQueueAgent]) + }) + + afterAll(async () => { + await workflowService.shutdownWorkers() + workflowService.close() + vi.restoreAllMocks() + try { + await s3Service.deletePrefix(BUCKET, '') + } catch (err) { + console.error('Failed to clean up E2E storage folder:', err) + } + }) + + async function seed(prefix: string) { + const team = await prisma.team.create({ data: { name: `E2E Read ${prefix} Team` } }) + const user = await prisma.user.create({ + data: { name: 'Reader', email: `read-${prefix}@example.com`, type: 'human' }, + }) + await prisma.teamMember.create({ data: { teamId: team.id, userId: user.id, role: 'owner' } }) + const project = await projectService.createProject(user, { + teamId: team.id, + name: `E2E Read ${prefix} Project`, + }) + return { team, user, project } + } + + async function createFileAsset( + projectId: string, + parentId: string, + fixture: string, + name: string, + mediaType: string, + proxyType: string, + ) { + const key = `projects/${projectId}/${name}` + const buffer = fs.readFileSync(path.join(fixturesDir, fixture)) + await s3Service.putObject(BUCKET, key, buffer, buffer.length, mediaType) + const storageKey = await prisma.storageKey.create({ data: { key } }) + return prisma.asset.create({ + data: { + name, + type: 'file', + status: 'uploaded', + mediaType, + projectId, + parentId, + storageKeyId: storageKey.id, + media: { proxyType } as unknown as PrismaJson.MediaInfo, + }, + }) + } + + const cli = (user: User) => ({ userId: user.id, via: 'cli' as const }) + + it('extracts real video frames and returns links without storage keys', async () => { + const { user, project } = await seed('frames') + const video = await createFileAsset( + project.id, + project.rootFolder!, + 'small.mp4', + 'clip.mp4', + 'video/mp4', + 'video', + ) + + const output = await runOperation(readAssetOp, cli(user), { + assetId: video.id, + videoConfig: { start: 0, end: 1, count: 2 }, + }) + + expect(output.status).toBe('ready') + expect(output.summary).toBe('Extracted 2 frame(s) from "clip.mp4" (0s - 1s):') + expect(output.images).toHaveLength(2) + for (const image of output.images) { + expect(image.label).toMatch(/^Frame at [\d.]+s$/) + expect(image.url).toEqual(expect.any(String)) + expect(image.key).toBeUndefined() + } + }, 60000) + + it('returns pending and completes the read through get_asset_render', async () => { + const { user, project } = await seed('pending') + const pdf = await createFileAsset( + project.id, + project.rootFolder!, + 'test.pdf', + 'deck.pdf', + 'application/pdf', + 'pdf', + ) + + const pending = await runOperation(readAssetOp, cli(user), { + assetId: pdf.id, + docConfig: { mode: 'pages', startPage: 1, endPage: 1 }, + maxWaitSeconds: 0, + }) + expect(pending).toMatchObject({ + status: 'pending', + asset: { id: pdf.id, kind: 'document' }, + taskId: expect.any(String), + retryAfterSeconds: 5, + }) + + const ready = await runOperation(getAssetRenderOp, cli(user), { taskId: pending.taskId! }) + expect(ready.status).toBe('ready') + expect(ready.summary).toBe('Rendered 1 page(s) of "deck.pdf" (Pages 1 to 1):') + expect(ready.images).toEqual([ + { label: 'Page 1', mimeType: 'image/webp', url: expect.any(String) }, + ]) + + // Only the requester can pick up the render. + const other = await prisma.user.create({ + data: { name: 'Other', email: 'read-other@example.com', type: 'human' }, + }) + await expect( + runOperation(getAssetRenderOp, cli(other), { taskId: pending.taskId! }), + ).rejects.toMatchObject({ status: 404 }) + }, 60000) + + it('gives the chat agent video frames as images', async () => { + const { team, user, project } = await seed('agent') + const video = await createFileAsset( + project.id, + project.rootFolder!, + 'small.mp4', + 'agent-clip.mp4', + 'video/mp4', + 'video', + ) + videoAssetId = video.id + + const agentUser = await prisma.user.create({ + data: { name: 'E2E Read Agent', email: 'e2e-read-agent@shumai.ai', type: 'agent' }, + }) + await prisma.teamMember.create({ + data: { teamId: team.id, userId: agentUser.id, role: 'editor' }, + }) + const provider = await prisma.provider.create({ + data: { + name: 'google', + teamId: team.id, + config: { api: 'google-generative-ai', apiKey: 'dummy-google-api-key' }, + }, + }) + const model = await prisma.model.create({ + data: { + modelId: 'gemini', + name: 'Gemini', + providerId: provider.id, + config: { + input: ['text', 'image'], + cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }, + contextWindow: 8192, + maxTokens: 2048, + reasoning: false, + }, + }, + }) + await prisma.agent.create({ + data: { + id: agentUser.id, + teamId: team.id, + type: 'chat', + enabled: true, + providerId: provider.id, + modelId: model.id, + config: { provider: 'google', model: 'gemini' }, + }, + }) + await prisma.agentSession.create({ + data: { + id: 'read-asset-session', + agentId: agentUser.id, + userId: user.id, + cwd: process.cwd(), + }, + }) + const userComment = await prisma.assetComment.create({ + data: { + assetId: video.id, + message: 'What happens in the first second?', + creatorId: user.id, + sessionId: 'read-asset-session', + }, + }) + + const task = await prisma.workflowTask.create({ + data: { + type: 'chat', + status: 'pending', + assetId: video.id, + projectId: project.id, + teamId: team.id, + payload: { + projectId: project.id, + agent: { + agentId: agentUser.id, + sessionId: 'read-asset-session', + userCommentId: userComment.id, + userId: user.id, + }, + }, + }, + }) + + const completed = await workflowService.executeWait(task, 90000) + expect(completed.status).toBe('completed') + + const [reply] = await prisma.assetComment.findMany({ + where: { assetId: video.id, replyToId: userComment.id }, + }) + expect(reply?.message).toContain('images:2') + expect(reply?.message).toContain('frames:2') + expect(reply?.message).toContain('render_tool:present') + + const render = await prisma.workflowTask.findFirstOrThrow({ + where: { assetId: video.id, type: 'transcode_screenshot' }, + }) + expect(render.payload?.assetRead).toMatchObject({ reader: 'video', requestedBy: user.id }) + }, 100000) +}) diff --git a/packages/operations/src/asset/read-asset.ts b/packages/operations/src/asset/read-asset.ts new file mode 100644 index 00000000..af6f752e --- /dev/null +++ b/packages/operations/src/asset/read-asset.ts @@ -0,0 +1,176 @@ +import { z } from 'zod' +import { defineOperation } from '../define' + +/** + * Asset readers, in matching order. Each reader handles one family of assets and contributes + * one optional config field to `read_asset`. Implementations live in + * `@shumai/core/src/asset-read/readers`. + */ +export const READER_KINDS = ['model3d', 'image', 'video', 'document'] as const +export type ReaderKind = (typeof READER_KINDS)[number] + +/** Most frames or pages a single read may render. */ +export const MAX_RENDER_ITEMS = 20 + +export const readerOptionSchemas = { + model3d: { + key: 'config3d', + schema: z + .object({ + degree: z + .number() + .int() + .nullable() + .optional() + .describe( + 'Turntable rotation degree (integer: 0, 15, 30, ..., 345 in 15° increments) to inspect a specific angle of a 3D model. Leave null/omitted to view the 4-view composite overview.', + ), + }) + .describe( + 'Configuration for 3D model assets. Optionally pass degree to inspect a specific turntable angle, or leave it out to view the 4-view composite overview.', + ), + }, + image: { + key: 'imageConfig', + schema: z.object({}).describe('Optional configuration for image assets.'), + }, + video: { + key: 'videoConfig', + schema: z + .object({ + start: z.number().describe('Start time in seconds.'), + end: z.number().describe('End time in seconds.'), + count: z + .number() + .describe(`Number of frames to extract, evenly spaced (1 to ${MAX_RENDER_ITEMS}).`), + }) + .describe('Configuration for video assets. Required when inspecting videos.'), + }, + document: { + key: 'docConfig', + schema: z + .object({ + mode: z + .enum(['pages', 'text']) + .describe( + '"pages" renders a page range as images (visual layout, charts, markups; required for binary PDFs). "text" reads raw text (ideal for .md, .txt, .csv, code files).', + ), + startPage: z + .number() + .nullable() + .optional() + .describe('Start page number (1-based, required when mode="pages"; omit for "text").'), + endPage: z + .number() + .nullable() + .optional() + .describe( + `End page number (1-based, max ${MAX_RENDER_ITEMS} pages per call, required when mode="pages"; omit for "text").`, + ), + }) + .describe('Configuration for document, PDF, and text assets.'), + }, +} as const satisfies Record + +export type ReaderOptions = z.output< + (typeof readerOptionSchemas)[TKind]['schema'] +> + +/** The config field name each reader contributes, e.g. `videoConfig`. */ +export type ReaderConfigKey = (typeof readerOptionSchemas)[ReaderKind]['key'] + +const readImageSchema = z.object({ + label: z.string().describe('What the image shows, e.g. "Frame at 30s" or "Page 3".'), + mimeType: z.string(), + url: z + .string() + .optional() + .describe('Short-lived link to the image (about 1 hour). Download it to look at the image.'), + /** Storage key; only returned to Shumai's built-in agent. */ + key: z.string().optional(), +}) + +/** Shared by `read_asset` and `get_asset_render`. */ +export const assetReadOutputSchema = z.object({ + status: z + .enum(['ready', 'pending']) + .describe('"pending" means a render is still running; call get_asset_render with taskId.'), + summary: z.string(), + asset: z.object({ + id: z.string(), + name: z.string(), + kind: z.enum(READER_KINDS), + mediaType: z.string().nullable(), + }), + details: z.array(z.string()), + images: z.array(readImageSchema), + text: z + .object({ + content: z.string(), + truncated: z.boolean(), + size: z.number(), + mimeType: z.string(), + url: z.string().optional(), + key: z.string().optional(), + }) + .optional(), + taskId: z.string().optional(), + retryAfterSeconds: z.number().optional(), +}) +export type AssetReadOutput = z.infer + +const maxWaitSeconds = z + .number() + .int() + .min(0) + .optional() + .describe( + 'Longest time to wait for rendering, in seconds. If rendering takes longer, the result is "pending" with a taskId to pass to get_asset_render. 0 returns immediately.', + ) + +export const readAsset = defineOperation({ + name: 'read_asset', + title: 'Read Asset', + description: + 'Inspects and reads the content of an asset without downloading it. ' + + 'Supports images, 3D models (4-view overview or a specific degree angle), video frame extraction, PDF page rendering, and direct text reading for text/markdown/code files. ' + + 'Pass assetId and the config matching the asset type (imageConfig, config3d, videoConfig, or docConfig); omit the others. ' + + 'Optionally pass annotationId to view visual markups drawn on the asset. ' + + 'Rendering frames or pages can take a while: if the result has status "pending", call get_asset_render with its taskId.', + input: z.object({ + assetId: z.string().min(1).describe('The ID of the asset to inspect.'), + annotationId: z + .string() + .optional() + .describe( + 'The ID of a comment or message (from ) whose visual markup should be overlaid.', + ), + config3d: readerOptionSchemas.model3d.schema.optional(), + imageConfig: readerOptionSchemas.image.schema.optional(), + videoConfig: readerOptionSchemas.video.schema.optional(), + docConfig: readerOptionSchemas.document.schema.optional(), + includeContent: z + .boolean() + .optional() + .describe('Set to false to skip reading text content and only describe the asset.'), + maxWaitSeconds, + }), + output: assetReadOutputSchema, + surfaces: ['agent'], + annotations: { readOnly: true, destructive: false }, +}) + +export const getAssetRender = defineOperation({ + name: 'get_asset_render', + title: 'Get Asset Render', + description: + 'Gets the result of a read_asset call that returned status "pending". ' + + 'Returns "pending" again while rendering is still running; wait retryAfterSeconds and try again.', + input: z.object({ + taskId: z.string().min(1).describe('The taskId returned by read_asset.'), + maxWaitSeconds, + }), + output: assetReadOutputSchema, + surfaces: ['agent'], + annotations: { readOnly: true, destructive: false, idempotent: true }, +}) diff --git a/packages/operations/src/index.ts b/packages/operations/src/index.ts index ad363dca..f23201cd 100644 --- a/packages/operations/src/index.ts +++ b/packages/operations/src/index.ts @@ -6,4 +6,5 @@ export { deleteAsset } from './asset/delete-asset' export { listAssets } from './asset/list-assets' export { moveAssets } from './asset/move-assets' export { renameAsset } from './asset/rename-asset' +export * from './asset/read-asset' export { listProjects } from './project/list-projects' diff --git a/packages/operations/src/registry.test.ts b/packages/operations/src/registry.test.ts index 47331b70..5152c7a7 100644 --- a/packages/operations/src/registry.test.ts +++ b/packages/operations/src/registry.test.ts @@ -15,13 +15,15 @@ describe('operation registry', () => { expect(new Set(names).size).toBe(names.length) }) - it('contains the initial asset and project operations', () => { + it('contains the asset and project operations', () => { expect(operationDefinitions.map((d) => d.name).sort()).toEqual([ 'create_folder', 'delete_asset', + 'get_asset_render', 'list_assets', 'list_projects', 'move_assets', + 'read_asset', 'rename_asset', ]) }) @@ -40,7 +42,12 @@ describe('operation registry', () => { const destructive = operationDefinitions.filter((d) => d.annotations.destructive) expect(destructive.map((d) => d.name)).toEqual(['delete_asset']) const readOnly = operationDefinitions.filter((d) => d.annotations.readOnly) - expect(readOnly.map((d) => d.name).sort()).toEqual(['list_assets', 'list_projects']) + expect(readOnly.map((d) => d.name).sort()).toEqual([ + 'get_asset_render', + 'list_assets', + 'list_projects', + 'read_asset', + ]) }) it.each(operationDefinitions.map((d) => [d.name, d] as const))( @@ -71,6 +78,9 @@ describe('operation registry', () => { ['rename_asset', ['rename']], ['move_assets', ['move', 'mv']], ['delete_asset', ['delete', 'rm']], + // CLI commands for reading assets arrive with the CLI surface + ['read_asset', []], + ['get_asset_render', []], ]) }) diff --git a/packages/operations/src/registry.ts b/packages/operations/src/registry.ts index 7e6bde8d..7cc87d07 100644 --- a/packages/operations/src/registry.ts +++ b/packages/operations/src/registry.ts @@ -2,6 +2,7 @@ import { createFolder } from './asset/create-folder' import { deleteAsset } from './asset/delete-asset' import { listAssets } from './asset/list-assets' import { moveAssets } from './asset/move-assets' +import { getAssetRender, readAsset } from './asset/read-asset' import { renameAsset } from './asset/rename-asset' import type { AnyOperationDefinition, Surface } from './define' import { listProjects } from './project/list-projects' @@ -17,6 +18,8 @@ export const operationDefinitions = [ renameAsset, moveAssets, deleteAsset, + readAsset, + getAssetRender, ] as const export type RegisteredOperationDefinition = (typeof operationDefinitions)[number] diff --git a/packages/webui/components/settings/AgentFormDialog.tsx b/packages/webui/components/settings/AgentFormDialog.tsx index d17a7ea3..b3b09c40 100644 --- a/packages/webui/components/settings/AgentFormDialog.tsx +++ b/packages/webui/components/settings/AgentFormDialog.tsx @@ -82,6 +82,11 @@ const BUILTIN_TOOLS = [ name: () => m.agent_tool_read_asset_name(), description: () => m.agent_tool_read_asset_desc(), }, + { + id: 'get_asset_render', + name: () => m.agent_tool_get_asset_render_name(), + description: () => m.agent_tool_get_asset_render_desc(), + }, { id: 'list_projects', name: () => m.agent_tool_list_projects_name(), diff --git a/packages/webui/messages/en.json b/packages/webui/messages/en.json index af1ceb7b..6bebf2c1 100644 --- a/packages/webui/messages/en.json +++ b/packages/webui/messages/en.json @@ -609,6 +609,8 @@ "agent_tool_read_skill_desc": "Allows the agent to load and read skill definitions to perform specialized actions.", "agent_tool_read_asset_name": "Read Asset", "agent_tool_read_asset_desc": "Allows the agent to view and inspect images, video frames, and document pages.", + "agent_tool_get_asset_render_name": "Get Asset Render", + "agent_tool_get_asset_render_desc": "Allows the agent to pick up video frames or document pages that took longer to render.", "agent_tool_list_projects_name": "List Projects", "agent_tool_list_projects_desc": "Allows the agent to list the projects you have access to and find their root folders.", "agent_tool_list_assets_name": "Browse Shumai", diff --git a/packages/webui/messages/zh.json b/packages/webui/messages/zh.json index fd889e43..f9c7d992 100644 --- a/packages/webui/messages/zh.json +++ b/packages/webui/messages/zh.json @@ -609,6 +609,8 @@ "agent_tool_read_skill_desc": "允许智能体加载和读取技能定义以执行特定操作。", "agent_tool_read_asset_name": "读取资源", "agent_tool_read_asset_desc": "允许智能体查看和分析图片、视频画面和文档页面。", + "agent_tool_get_asset_render_name": "获取资源渲染结果", + "agent_tool_get_asset_render_desc": "允许智能体获取渲染耗时较长的视频画面或文档页面。", "agent_tool_list_projects_name": "列出项目", "agent_tool_list_projects_desc": "允许智能体列出您有权访问的项目并找到其根文件夹。", "agent_tool_list_assets_name": "浏览 Shumai", diff --git a/packages/workflow-core/src/workflow.test.ts b/packages/workflow-core/src/workflow.test.ts index 71f958c2..323385c4 100644 --- a/packages/workflow-core/src/workflow.test.ts +++ b/packages/workflow-core/src/workflow.test.ts @@ -189,6 +189,61 @@ describe('WorkflowService', () => { submitSpy.mockRestore() }) + describe('waitFor', () => { + async function createTask(status: WorkflowTaskStatus, output?: Record) { + const submitSpy = vi.spyOn(workflowService, 'submit').mockImplementation(async () => 'id') + const task = await prisma.workflowTask.create({ + data: { assetId: 'wait-asset', type: WorkflowTaskType.transcode_screenshot, status }, + }) + if (output) { + await prisma.workflowTask.update({ where: { id: task.id }, data: { output } }) + } + submitSpy.mockRestore() + return task + } + + it('returns done: true with the completed task', async () => { + const task = await createTask(WorkflowTaskStatus.completed, { screenshots: [] }) + + const result = await workflowService.waitFor(task, 1000) + + expect(result.done).toBe(true) + expect(result.task.output).toEqual({ screenshots: [] }) + }) + + it('returns done: false instead of throwing when the budget runs out', async () => { + const task = await createTask(WorkflowTaskStatus.processing) + + const start = Date.now() + const result = await workflowService.waitFor(task, 300) + + expect(result.done).toBe(false) + expect(result.task.status).toBe(WorkflowTaskStatus.processing) + expect(Date.now() - start).toBeLessThan(2000) + }) + + it('checks once and returns immediately with a zero budget', async () => { + const task = await createTask(WorkflowTaskStatus.pending) + + const result = await workflowService.waitFor(task, 0) + + expect(result.done).toBe(false) + }) + + it('throws the task error when the task failed', async () => { + const task = await createTask(WorkflowTaskStatus.failed, { error: 'ffmpeg exploded' }) + + await expect(workflowService.waitFor(task, 1000)).rejects.toThrow('ffmpeg exploded') + }) + + it('keeps executeWait throwing on timeout', async () => { + const task = await createTask(WorkflowTaskStatus.processing) + + await expect(workflowService.executeWait(task, 200)).rejects.toThrow( + 'Workflow task timed out', + ) + }) + }) describe('Stale Task Recovery', () => { it('should resume stale pending workflow task in DBOS and increment recoveryAttempts', async () => { const task = await prisma.workflowTask.create({ diff --git a/packages/workflow-core/src/workflow.ts b/packages/workflow-core/src/workflow.ts index 5be41799..22d9e8b3 100644 --- a/packages/workflow-core/src/workflow.ts +++ b/packages/workflow-core/src/workflow.ts @@ -30,8 +30,24 @@ export class WorkflowService { } async executeWait(task: WorkflowTask, timeoutMs: number = 30000): Promise { + const { task: finished, done } = await this.waitFor(task, timeoutMs) + if (!done) { + throw new Error('Workflow task timed out') + } + return finished + } + + /** + * Wait up to `timeoutMs` for a task to finish. Unlike `executeWait`, running out of time is + * not an error: it resolves with the latest task state and `done: false`, so callers can hand + * the task ID back and complete the work later. Throws if the task failed or disappeared. + */ + async waitFor( + task: WorkflowTask, + timeoutMs: number, + ): Promise<{ task: WorkflowTask; done: boolean }> { const start = Date.now() - while (Date.now() - start < timeoutMs) { + for (;;) { const updatedTask = await prisma.workflowTask.findUnique({ where: { id: task.id }, }) @@ -41,7 +57,7 @@ export class WorkflowService { } if (updatedTask.status === 'completed') { - return updatedTask + return { task: updatedTask, done: true } } if (updatedTask.status === 'failed') { @@ -50,10 +66,12 @@ export class WorkflowService { throw new Error(error) } + if (Date.now() - start >= timeoutMs) { + return { task: updatedTask, done: false } + } + await new Promise((resolve) => setTimeout(resolve, 200)) } - - throw new Error('Workflow task timed out') } start(listenQueues?: string[]): void {