feat(agent): consolidate media tools into read_asset and add S3 key support to download_asset (#376)

* feat(agent): consolidate media tools into read_asset and add S3 key support to download_asset

* fix(agent): refine parameter descriptions and remove download URL from read_asset text content

* fix(agent): resolve owning asset and enforce RBAC permissions when downloading by S3 key
This commit is contained in:
Yiling-J
2026-09-02 21:51:07 +08:00
committed by GitHub
parent 35b57cc0ab
commit e58dfaa71a
38 changed files with 1732 additions and 1851 deletions
+12 -12
View File
@@ -30,36 +30,36 @@ The following tools can be individually toggled in the agent configuration dialo
- **Description:** Allows the agent to load and read skill definitions to perform specialized actions.
- **Why it matters:** Enables the agent to inspect the instructions/configurations of custom skills registered in your workspace before executing them.
### 3. Analyze Image (`analyze_image`)
### 3. Read Asset (`read_asset`)
- **Description:** Allows the agent to view and analyze the visual contents of an image file.
- **Why it matters:** Enables multimodal agents to "see" image assets uploaded to Shumai, allowing them to answer comments or write metadata about images.
- **Description:** Allows the agent to inspect images, extract still frames from video files, render PDF document pages, and read text/markdown documents.
- **Why it matters:** Enables multimodal agents to view visual media assets uploaded to Shumai, extract frame-by-frame video screenshots, inspect document layouts, and answer questions about uploaded creative assets.
### 4. Video Frame Extraction (`screenshot`)
- **Description:** Allows the agent to extract still image frames from video files for analysis.
- **Why it matters:** Enables the agent to perform frame-by-frame analysis on videos (e.g. for comments, annotation, or search).
### 5. Browse Shumai (`list_assets`)
### 4. Browse Shumai (`list_assets`)
- **Description:** Allows the agent to list folders and browse files in Shumai.
- **Why it matters:** Gives the agent visual mapping of the project structure so it can search for other files or locate related assets.
### 6. Create Folder (`create_folder`)
### 5. Create Folder (`create_folder`)
- **Description:** Allows the agent to create new folders in Shumai.
- **Why it matters:** Needed if the agent organizes outputs or places files in custom subdirectory folders.
### 7. Create File (`create_file`)
### 6. Create File (`create_file`)
- **Description:** Allows the agent to create and upload new files to Shumai.
- **Why it matters:** Allows the agent to generate new assets (e.g. text logs, transcoded clips, reports) and save them directly back into your project.
### 8. Create Version (`create_version`)
### 7. Create Version (`create_version`)
- **Description:** Allows the agent to upload and stack a new version of an existing file in Shumai.
- **Why it matters:** Allows the agent to update files in a clean, non-destructive way by stacking revisions rather than creating duplicate files.
### 8. Download Asset (`download_asset`)
- **Description:** Allows the agent to download a workspace asset or specific storage S3 key to the local `.pi` directory for processing with local scripts or bash commands.
- **Why it matters:** Enables agents executing custom skill scripts to fetch files or intermediate transcode artifacts to local disk.
---
## Restricting Tools
+1 -1
View File
@@ -53,7 +53,7 @@ Shumai supports quota enforcement across multiple layers of the AI and agent arc
| **AI Cost ($)** | Dollar expenditure accrued from AI model providers. | Team, Role, or Members |
| **MCP Server Calls** | Number of tool invocations directed to a connected Model Context Protocol (MCP) server. | Target MCP Server ID |
| **Sandboxed Bash Commands** | Shell commands executed by the agent inside the isolated sandbox. | Wildcard pattern (e.g. `npm *`, `*`) |
| **Agent Tool Calls** | Calls to specific native tools (e.g. `analyze_image`, `screenshot`, `create_file`). | Tool name |
| **Agent Tool Calls** | Calls to specific native tools (e.g. `read_asset`, `download_asset`, `create_file`). | Tool name |
### Bash Command Wildcard Matching
+1 -1
View File
@@ -118,7 +118,7 @@ If you need to create files in the local filesystem (for example, a temporary fi
When creating a file or version, you may attach metadata (for example, the AI model or prompt used to generate the asset). The allowed field keys and value types are declared directly in the 'metadata' parameter of the 'create_file' and 'create_version' tools.
# Message Context & Tools
User messages may contain a <context> block detailing the user, active asset location, playback position, and attachments. Use the asset IDs from <context> when invoking tools (e.g. 'analyze_image', 'screenshot', 'read_pdf_pages', 'download_asset', 'list_assets'). When an <annotation id="..." /> tag is present, it indicates the user has drawn visual markups on the asset at the specified position. When calling 'screenshot', 'analyze_image', or 'read_pdf_pages', pass the 'annotationId' parameter with that ID to view the image with the visual markups overlaid.`
User messages may contain a <context> block detailing the user, active asset location, playback position, and attachments. Use the asset IDs from <context> when invoking tools (e.g. 'read_asset', 'download_asset', 'list_assets'). When an <annotation id="..." /> tag is present, it indicates the user has drawn visual markups on the asset at the specified position. When calling 'read_asset', pass the 'annotationId' parameter with that ID to view the image or rendered pages with the visual markups overlaid.`
if (agent.soul) {
systemPrompt = `${systemPrompt}\n\nAgent Personality and Core Instructions:\n${agent.soul}`
@@ -7,8 +7,7 @@ import { DatabaseSessionStorage } from './database-session-storage'
import { agentService } from '@shumai/core/src/agent/agent'
import { createAgentSession, type DbProviderInfo } from './index'
import * as sandboxedBashModule from './tools/sandboxed-bash'
import * as analyzeImageModule from './tools/analyze-image'
import * as screenshotModule from './tools/screenshot'
import * as readAssetModule from './tools/read-asset'
describe('DatabaseSessionStorage', () => {
setupTestDbHooks()
@@ -451,9 +450,8 @@ describe('DatabaseSessionStorage', () => {
})
const sessionId = initialStorage.sessionId
// 2. Spy on media tools constructors
const createAnalyzeImageToolSpy = vi.spyOn(analyzeImageModule, 'createAnalyzeImageTool')
const createScreenshotToolSpy = vi.spyOn(screenshotModule, 'createScreenshotTool')
// 2. Spy on media tool constructor
const createReadAssetToolSpy = vi.spyOn(readAssetModule, 'createReadAssetTool')
// 3. Resume the session via createAgentSession with a NEW comment ID
await createAgentSession({
@@ -470,9 +468,8 @@ describe('DatabaseSessionStorage', () => {
providers: mockProviders,
})
// 4. Assert that the tools were instantiated using user ID
expect(createAnalyzeImageToolSpy).toHaveBeenCalledWith(user.id)
expect(createScreenshotToolSpy).toHaveBeenCalledWith(user.id)
// 4. Assert that the tool was instantiated using user ID
expect(createReadAssetToolSpy).toHaveBeenCalledWith(user.id)
})
it('should instantiate media tools with user ID in subsequent turns (video)', async () => {
@@ -505,9 +502,8 @@ describe('DatabaseSessionStorage', () => {
})
const sessionId = initialStorage.sessionId
// 2. Spy on media tools constructors
const createAnalyzeImageToolSpy = vi.spyOn(analyzeImageModule, 'createAnalyzeImageTool')
const createScreenshotToolSpy = vi.spyOn(screenshotModule, 'createScreenshotTool')
// 2. Spy on media tool constructor
const createReadAssetToolSpy = vi.spyOn(readAssetModule, 'createReadAssetTool')
// 3. Resume the session via createAgentSession with a NEW comment ID
await createAgentSession({
@@ -524,9 +520,8 @@ describe('DatabaseSessionStorage', () => {
providers: mockProviders,
})
// 4. Assert that the tools were instantiated using user ID
expect(createScreenshotToolSpy).toHaveBeenCalledWith(user.id)
expect(createAnalyzeImageToolSpy).toHaveBeenCalledWith(user.id)
// 4. Assert that the tool was instantiated using user ID
expect(createReadAssetToolSpy).toHaveBeenCalledWith(user.id)
})
describe('P1 Bug Reproductions', () => {
+2 -8
View File
@@ -19,17 +19,15 @@ import * as os from 'os'
import * as path from 'path'
import { DatabaseSessionStorage } from './database-session-storage'
import { metadataService } from '@shumai/core/src/metadata/metadata'
import { createAnalyzeImageTool } from './tools/analyze-image'
import { createCreateFileTool } from './tools/create-file'
import { createCreateFolderTool } from './tools/create-folder'
import { createCreateVersionTool } from './tools/create-version'
import { createDownloadAssetTool } from './tools/download-asset'
import { createListAssetsTool } from './tools/list-assets'
import { createReadPdfPagesTool } from './tools/read-pdf-pages'
import { createReadAssetTool } from './tools/read-asset'
import { createReadSkillTool } from './tools/read-skill'
import { createReadThreadTool } from './tools/read-thread'
import { createSandboxedBashTool } from './tools/sandboxed-bash'
import { createScreenshotTool } from './tools/screenshot'
import { serializeContextToXml } from './context/serialize-context'
import type { ShumaiMessageContext } from '@shumai/dtos'
@@ -314,11 +312,7 @@ export async function createAgentSession(params: CreateAgentSessionParams) {
const mediaTools: AgentTool[] = []
if (userId) {
mediaTools.push(
createAnalyzeImageTool(userId),
createScreenshotTool(userId),
createReadPdfPagesTool(userId),
)
mediaTools.push(createReadAssetTool(userId))
}
const agent = await prisma.agent.findUnique({
@@ -1,262 +0,0 @@
import { describe, it, expect, vi, beforeEach } from 'vitest'
import { createAnalyzeImageTool } from './analyze-image'
import {
prisma,
WorkflowTaskStatus,
type Asset,
type AssetComment,
type AgentSessionEntry,
type User,
type WorkflowTask,
} from '@shumai/db'
import { s3Service } from '@shumai/core/src/s3/s3'
import { workflowService } from '@shumai/workflow-core'
import { authzService } from '@shumai/core/src/authz/authz'
vi.mock('@shumai/db', async (importOriginal) => {
const actual = await importOriginal<typeof import('@shumai/db')>()
return {
...actual,
prisma: {
user: {
findUnique: vi.fn(),
},
asset: {
findUnique: vi.fn(),
},
assetComment: {
findUnique: vi.fn(),
},
agentSessionEntry: {
findUnique: vi.fn(),
},
workflowTask: {
create: vi.fn(),
},
},
}
})
vi.mock('@shumai/core/src/s3/s3', () => ({
s3Service: {
getObject: vi.fn(),
},
}))
vi.mock('@shumai/workflow-core', () => ({
workflowService: {
executeWait: vi.fn(),
},
}))
vi.mock('@shumai/core/src/authz/authz', () => ({
authzService: {
hasPermission: vi.fn(),
},
Permission: { Read: 'Read' },
ResourceType: { Asset: 'asset' },
}))
describe('analyzeImageTool', () => {
beforeEach(() => {
vi.clearAllMocks()
})
it('should check user permissions and fetch the image from S3 when annotationId is omitted', async () => {
vi.mocked(prisma.user.findUnique).mockResolvedValue({ id: 'user-1' } as User)
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'asset-1',
projectId: 'project-1',
storageKey: { key: 'raw-image.png' },
media: null,
} as unknown as Asset)
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('fake-image-bytes'),
contentType: 'image/png',
} as unknown as { buffer: Buffer; contentType: string })
const tool = createAnalyzeImageTool('user-1')
const result = await tool.execute('call-1', { assetId: 'asset-1' })
expect(authzService.hasPermission).toHaveBeenCalledWith({
user: { id: 'user-1' },
permission: 'Read',
type: 'asset',
id: 'asset-1',
})
expect(result.content[0].type).toBe('image')
expect((result.content[0] as unknown as { data: string }).data).toBe(
Buffer.from('fake-image-bytes').toString('base64'),
)
expect(result.details.sourceKeys).toEqual(['raw-image.png'])
expect(prisma.workflowTask.create).not.toHaveBeenCalled()
})
it('should trigger overlay transcode workflow when annotationId matches AssetComment with annotations', async () => {
vi.mocked(prisma.user.findUnique).mockResolvedValue({ id: 'user-1' } as User)
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'asset-1',
projectId: 'project-1',
storageKey: { key: 'raw-image.png' },
media: {
imageTranscodes: [{ key: 'transcoded-image.webp' }],
},
} as unknown as Asset)
vi.mocked(prisma.assetComment.findUnique).mockResolvedValue({
id: 'comment-1',
assetId: 'asset-1',
annotation: [
{
type: 'box',
color: '#ff0000',
points: [
[0, 0],
[1, 1],
],
},
],
} as unknown as AssetComment)
vi.mocked(prisma.workflowTask.create).mockResolvedValue({
id: 'task-1',
} as unknown as WorkflowTask)
vi.mocked(workflowService.executeWait).mockResolvedValue({
id: 'task-1',
status: WorkflowTaskStatus.completed,
output: { key: 'annotated-image.webp' },
} as unknown as WorkflowTask)
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('fake-annotated-image-bytes'),
contentType: 'image/webp',
} as unknown as { buffer: Buffer; contentType: string })
const tool = createAnalyzeImageTool('user-1')
const result = await tool.execute('call-1', { assetId: 'asset-1', annotationId: 'comment-1' })
expect(prisma.workflowTask.create).toHaveBeenCalledWith({
data: {
assetId: 'asset-1',
projectId: 'project-1',
type: 'transcode_image_annotation',
status: 'pending',
payload: {
projectId: 'project-1',
imageAnnotation: {
annotations: [
{
type: 'box',
color: '#ff0000',
points: [
[0, 0],
[1, 1],
],
},
],
},
},
},
})
expect(result.content[0].type).toBe('image')
expect((result.content[0] as unknown as { data: string }).data).toBe(
Buffer.from('fake-annotated-image-bytes').toString('base64'),
)
expect(result.details.sourceKeys).toEqual(['annotated-image.webp'])
})
it('should trigger overlay transcode workflow when annotationId matches AgentSessionEntry (1-on-1 chat mode)', async () => {
vi.mocked(prisma.user.findUnique).mockResolvedValue({ id: 'user-1' } as User)
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'asset-1',
projectId: 'project-1',
storageKey: { key: 'raw-image.png' },
media: null,
} as unknown as Asset)
vi.mocked(prisma.assetComment.findUnique).mockResolvedValue(null)
vi.mocked(prisma.agentSessionEntry.findUnique).mockResolvedValue({
id: 'entry-1',
assetId: 'asset-1',
data: {
details: {
annotations: [{ type: 'arrow', x1: 5, y1: 5, x2: 20, y2: 20 }],
},
},
} as unknown as AgentSessionEntry)
vi.mocked(prisma.workflowTask.create).mockResolvedValue({
id: 'task-1',
} as unknown as WorkflowTask)
vi.mocked(workflowService.executeWait).mockResolvedValue({
id: 'task-1',
status: WorkflowTaskStatus.completed,
output: { key: 'chat-annotated-image.webp' },
} as unknown as WorkflowTask)
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('fake-chat-annotated-bytes'),
contentType: 'image/webp',
} as unknown as { buffer: Buffer; contentType: string })
const tool = createAnalyzeImageTool('user-1')
const result = await tool.execute('call-2', { assetId: 'asset-1', annotationId: 'entry-1' })
expect(prisma.workflowTask.create).toHaveBeenCalledWith({
data: {
assetId: 'asset-1',
projectId: 'project-1',
type: 'transcode_image_annotation',
status: 'pending',
payload: {
projectId: 'project-1',
imageAnnotation: {
annotations: [{ type: 'arrow', x1: 5, y1: 5, x2: 20, y2: 20 }],
},
},
},
})
expect(result.details.sourceKeys).toEqual(['chat-annotated-image.webp'])
})
it('should throw an error when annotationId belongs to a different asset', async () => {
vi.mocked(prisma.user.findUnique).mockResolvedValue({ id: 'user-1' } as User)
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'asset-1',
projectId: 'project-1',
storageKey: { key: 'raw-image.png' },
media: null,
} as unknown as Asset)
vi.mocked(prisma.assetComment.findUnique).mockResolvedValue({
id: 'comment-on-other-asset',
assetId: 'asset-999',
annotation: [{ type: 'arrow' }],
} as unknown as AssetComment)
const tool = createAnalyzeImageTool('user-1')
await expect(
tool.execute('call-mismatch', {
assetId: 'asset-1',
annotationId: 'comment-on-other-asset',
}),
).rejects.toThrow(
'Annotation "comment-on-other-asset" belongs to asset "asset-999", not target asset "asset-1".',
)
})
})
-132
View File
@@ -1,132 +0,0 @@
import { Type } from 'typebox'
import { type AgentTool } from '@earendil-works/pi-agent-core'
import { type ImageContent } from '@earendil-works/pi-ai'
import { prisma, WorkflowTaskType, WorkflowTaskStatus, type User } from '@shumai/db'
import { s3Service } from '@shumai/core/src/s3/s3'
import { workflowService } from '@shumai/workflow-core'
import { authzService, Permission, ResourceType } from '@shumai/core/src/authz/authz'
import { resolveAnnotationsById } from './annotation-resolver'
const analyzeImageSchema = Type.Object({
assetId: Type.String({
description: 'The asset ID of the image to analyze. This parameter is required.',
}),
annotationId: Type.Optional(
Type.String({
description:
'The ID of the comment or message entry from <annotation id="..." /> whose visual markup should be overlaid on the image.',
}),
),
})
export function createAnalyzeImageTool(userId: string): AgentTool<typeof analyzeImageSchema> {
return {
name: 'analyze_image',
label: 'Analyze Image',
description:
'Retrieves and views the image content. Call this tool if you need to analyze the image.',
parameters: analyzeImageSchema,
execute: async (_toolCallId, params) => {
const assetId = params.assetId
if (!userId) {
throw new Error('User ID is required for authorization.')
}
await authzService.hasPermission({
user: { id: userId } as User,
permission: Permission.Read,
type: ResourceType.Asset,
id: assetId,
})
let asset = await prisma.asset.findUnique({
where: { id: assetId },
include: { storageKey: true },
})
if (!asset) {
throw new Error(`Asset with ID ${assetId} not found.`)
}
if (asset.type === 'version_stack') {
const latestVersion = await prisma.asset.findFirst({
where: { parentId: asset.id, isDeleted: false },
orderBy: { sortIndex: 'asc' },
include: { storageKey: true },
})
if (latestVersion) {
asset = latestVersion
}
}
// Get default media key
let mediaKey = asset.storageKey?.key
if (asset.media) {
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const mediaInfo = asset.media as any
if (mediaInfo.imageTranscodes && mediaInfo.imageTranscodes.length > 0) {
mediaKey = mediaInfo.imageTranscodes[0].key || mediaKey
}
}
if (!mediaKey) {
throw new Error('No media content found for this asset.')
}
// Resolve annotations dynamically by annotationId
const { annotations } = await resolveAnnotationsById(assetId, params.annotationId)
let keyToUse = mediaKey
if (annotations) {
// Trigger transcode workflow to draw annotation
const task = await prisma.workflowTask.create({
data: {
assetId,
projectId: asset.projectId || 'none',
type: WorkflowTaskType.transcode_image_annotation,
status: WorkflowTaskStatus.pending,
payload: {
projectId: asset.projectId || 'none',
imageAnnotation: {
annotations,
},
},
},
})
const completedTask = await workflowService.executeWait(task)
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const output = completedTask.output as any
if (output?.key) {
keyToUse = output.key
} else {
throw new Error('Image annotation transcode workflow failed to output S3 key.')
}
}
// Retrieve the image content from S3
const bucket = process.env.S3_BUCKET || 'shumai'
const { buffer, contentType } = await s3Service.getObject(bucket, keyToUse)
const mimeType =
contentType && contentType !== 'application/octet-stream' ? contentType : 'image/webp'
const content: ImageContent[] = [
{
type: 'image',
data: buffer.toString('base64'),
mimeType,
},
]
return {
content,
details: {
sourceKeys: [keyToUse],
},
}
},
}
}
@@ -3,7 +3,7 @@ import type { ShumaiMessageContext } from '@shumai/dtos'
export async function resolveAnnotationsById(
assetId: string,
annotationId?: string,
annotationId?: string | null,
): Promise<{ annotations: PrismaJson.AnnotationList | null; timestamp: number | null }> {
if (!annotationId) {
return { annotations: null, timestamp: null }
+272 -143
View File
@@ -1,6 +1,6 @@
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'
import { createDownloadAssetTool } from './download-asset'
import { prisma, type Asset } from '@shumai/db'
import { createDownloadAssetTool, resolveAssetIdFromKey } from './download-asset'
import { prisma, type Asset, type StorageKey } from '@shumai/db'
import { s3Service } from '@shumai/core/src/s3/s3'
import { authzService } from '@shumai/core/src/authz/authz'
import * as fs from 'fs'
@@ -15,6 +15,10 @@ vi.mock('@shumai/db', async (importOriginal) => {
findUnique: vi.fn(),
findFirst: vi.fn(),
},
storageKey: {
findFirst: vi.fn(),
findUnique: vi.fn(),
},
},
}
})
@@ -52,163 +56,288 @@ describe('downloadAssetTool', () => {
it('should throw authorization error if user ID is empty', async () => {
const tool = createDownloadAssetTool('')
await expect(tool.execute('call-1', { assetId: 'asset-1' })).rejects.toThrow(
await expect(tool.execute('call-1', { assetId: 'asset-1', key: null })).rejects.toThrow(
'User ID is required for authorization.',
)
})
it('should throw error if asset not found', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue(null)
it('should throw error if both assetId and key are null', async () => {
const tool = createDownloadAssetTool('user-1')
await expect(tool.execute('call-1', { assetId: 'asset-1' })).rejects.toThrow(
'Asset with ID asset-1 not found.',
await expect(tool.execute('call-1', { assetId: null, key: null })).rejects.toThrow(
'Provide exactly one of "assetId" (to download a workspace asset) or "key" (to download a specific S3 storage key). Set the unused parameter to null.',
)
})
it('should download proxy image to .pi directory when imageTranscodes is present', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'asset-img-1',
name: 'sample_photo.jpg',
storageKey: { key: 'raw/sample_photo.jpg' },
media: {
proxyType: 'image',
imageTranscodes: [{ key: 'proxy/sample_photo.webp' }],
},
} as unknown as Asset)
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('fake-proxy-image-bytes'),
contentType: 'image/webp',
} as unknown as { buffer: Buffer; contentType: string })
it('should throw error if both assetId and key are provided', async () => {
const tool = createDownloadAssetTool('user-1')
const result = await tool.execute('call-1', { assetId: 'asset-img-1' })
expect(authzService.hasPermission).toHaveBeenCalledWith({
user: { id: 'user-1' },
permission: 'Read',
type: 'asset',
id: 'asset-img-1',
})
expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'proxy/sample_photo.webp')
const expectedPath = path.join(piDir, 'asset-img-1_sample_photo.jpg')
createdFiles.push(expectedPath)
expect(fs.existsSync(expectedPath)).toBe(true)
expect(fs.readFileSync(expectedPath, 'utf-8')).toBe('fake-proxy-image-bytes')
expect(result.details.filePath).toBe(path.join('.pi', 'asset-img-1_sample_photo.jpg'))
expect(result.details.contentType).toBe('image/webp')
expect((result.content[0] as { type: 'text'; text: string }).text).toContain(
'Remember to delete this temporary file when finished.',
await expect(
tool.execute('call-1', { assetId: 'asset-1', key: 'files/path/file.png' }),
).rejects.toThrow(
'Provide exactly one of "assetId" (to download a workspace asset) or "key" (to download a specific S3 storage key). Set the unused parameter to null.',
)
})
it('should download proxy video to .pi directory when videoTranscodes is present', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'asset-vid-1',
name: 'video_test.mov',
storageKey: { key: 'raw/video_test.mov' },
media: {
proxyType: 'video',
videoTranscodes: [{ key: 'proxy/video_test.mp4' }],
},
} as unknown as Asset)
describe('Download by assetId', () => {
it('should throw error if asset not found', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue(null)
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('fake-video-bytes'),
contentType: 'video/mp4',
} as unknown as { buffer: Buffer; contentType: string })
const tool = createDownloadAssetTool('user-1')
const result = await tool.execute('call-1', { assetId: 'asset-vid-1' })
expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'proxy/video_test.mp4')
const expectedPath = path.join(piDir, 'asset-vid-1_video_test.mov')
createdFiles.push(expectedPath)
expect(fs.existsSync(expectedPath)).toBe(true)
expect(result.details.size).toBe(Buffer.from('fake-video-bytes').length)
})
it('should download original file to .pi directory even when pdfTranscode is present', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'asset-md-1',
name: 'doc.md',
storageKey: { key: 'raw/doc.md' },
media: {
proxyType: 'pdf',
pdfTranscode: { key: 'proxy/doc.pdf' },
original: { key: 'raw/doc.md' },
},
} as unknown as Asset)
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('# fake markdown bytes'),
contentType: 'text/markdown',
} as unknown as { buffer: Buffer; contentType: string })
const tool = createDownloadAssetTool('user-1')
const result = await tool.execute('call-1', { assetId: 'asset-md-1' })
expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'raw/doc.md')
const expectedPath = path.join(piDir, 'asset-md-1_doc.md')
createdFiles.push(expectedPath)
expect(fs.existsSync(expectedPath)).toBe(true)
expect(result.details.contentType).toBe('text/markdown')
})
it('should resolve version_stack asset to its latest version file', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'stack-1',
name: '',
type: 'version_stack',
storageKey: null,
media: null,
} as unknown as Asset)
vi.mocked(prisma.asset.findFirst).mockResolvedValue({
id: 'file-ver-2',
name: 'banana_pig.png',
type: 'file',
storageKey: { key: 'raw/banana_pig.png' },
media: {
proxyType: 'image',
imageTranscodes: [{ key: 'proxy/banana_pig.webp' }],
},
} as unknown as Asset)
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('fake-version-bytes'),
contentType: 'image/webp',
} as unknown as { buffer: Buffer; contentType: string })
const tool = createDownloadAssetTool('user-1')
const result = await tool.execute('call-1', { assetId: 'stack-1' })
expect(prisma.asset.findFirst).toHaveBeenCalledWith({
where: { parentId: 'stack-1', isDeleted: false },
orderBy: { sortIndex: 'asc' },
include: { storageKey: true },
const tool = createDownloadAssetTool('user-1')
await expect(tool.execute('call-1', { assetId: 'asset-1', key: null })).rejects.toThrow(
'Asset with ID asset-1 not found.',
)
})
expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'proxy/banana_pig.webp')
it('should download proxy image to .pi directory when downloading by assetId', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'asset-img-1',
name: 'sample_photo.jpg',
storageKey: { key: 'raw/sample_photo.jpg' },
media: {
proxyType: 'image',
imageTranscodes: [{ key: 'proxy/sample_photo.webp' }],
},
} as unknown as Asset)
const expectedPath = path.join(piDir, 'file-ver-2_banana_pig.png')
createdFiles.push(expectedPath)
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('fake-proxy-image-bytes'),
contentType: 'image/webp',
} as unknown as { buffer: Buffer; contentType: string })
expect(fs.existsSync(expectedPath)).toBe(true)
expect(result.details.name).toBe('banana_pig.png')
const tool = createDownloadAssetTool('user-1')
const result = await tool.execute('call-1', { assetId: 'asset-img-1', key: null })
expect(authzService.hasPermission).toHaveBeenCalledWith({
user: { id: 'user-1' },
permission: 'Read',
type: 'asset',
id: 'asset-img-1',
})
expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'proxy/sample_photo.webp')
const expectedPath = path.join(piDir, 'asset-img-1_sample_photo.jpg')
createdFiles.push(expectedPath)
expect(fs.existsSync(expectedPath)).toBe(true)
expect(fs.readFileSync(expectedPath, 'utf-8')).toBe('fake-proxy-image-bytes')
expect(result.details.filePath).toBe(path.join('.pi', 'asset-img-1_sample_photo.jpg'))
expect(result.details.contentType).toBe('image/webp')
expect((result.content[0] as { type: 'text'; text: string }).text).toContain(
'Remember to delete this temporary file when finished.',
)
})
it('should resolve version_stack asset to its latest version file', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'stack-1',
name: '',
type: 'version_stack',
storageKey: null,
media: null,
} as unknown as Asset)
vi.mocked(prisma.asset.findFirst).mockResolvedValue({
id: 'file-ver-2',
name: 'banana_pig.png',
type: 'file',
storageKey: { key: 'raw/banana_pig.png' },
media: {
proxyType: 'image',
imageTranscodes: [{ key: 'proxy/banana_pig.webp' }],
},
} as unknown as Asset)
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('fake-version-bytes'),
contentType: 'image/webp',
} as unknown as { buffer: Buffer; contentType: string })
const tool = createDownloadAssetTool('user-1')
const result = await tool.execute('call-1', { assetId: 'stack-1', key: null })
expect(prisma.asset.findFirst).toHaveBeenCalledWith({
where: { parentId: 'stack-1', isDeleted: false },
orderBy: { sortIndex: 'asc' },
include: { storageKey: true },
})
expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'proxy/banana_pig.webp')
const expectedPath = path.join(piDir, 'file-ver-2_banana_pig.png')
createdFiles.push(expectedPath)
expect(fs.existsSync(expectedPath)).toBe(true)
expect(result.details.name).toBe('banana_pig.png')
})
})
describe('Download by S3 key and Authorization', () => {
it('should reject keys that do not start with files/', async () => {
const tool = createDownloadAssetTool('user-1')
await expect(
tool.execute('call-1', { assetId: null, key: 'avatars/user-1.png' }),
).rejects.toThrow('Invalid storage key format: key must start with "files/"')
})
it('should reject keys with path traversal (..)', async () => {
const tool = createDownloadAssetTool('user-1')
await expect(
tool.execute('call-1', { assetId: null, key: 'files/../../etc/passwd' }),
).rejects.toThrow('cannot contain ".."')
})
it('should reject keys with insufficient path segments', async () => {
const tool = createDownloadAssetTool('user-1')
await expect(tool.execute('call-1', { assetId: null, key: 'files/invalid' })).rejects.toThrow(
'Invalid storage key format',
)
})
it('should reject keys that do not belong to any valid asset in database', async () => {
vi.mocked(prisma.asset.findUnique).mockResolvedValue(null)
vi.mocked(prisma.storageKey.findFirst).mockResolvedValue(null)
vi.mocked(prisma.storageKey.findUnique).mockResolvedValue(null)
const tool = createDownloadAssetTool('user-1')
await expect(
tool.execute('call-1', { assetId: null, key: 'files/nonexistent-id/shot.webp' }),
).rejects.toThrow('does not belong to any valid asset or the asset has been deleted')
})
it('should reject download when user lacks read permission on the owning asset (cross-project protection)', async () => {
// Key belongs to asset-in-other-project
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'asset-in-other-project',
} as unknown as Asset)
// Authz rejects because user is in a different project/team
vi.mocked(authzService.hasPermission).mockRejectedValue(
new Error('User does not have Read permission on asset asset-in-other-project'),
)
const tool = createDownloadAssetTool('user-1')
await expect(
tool.execute('call-1', {
assetId: null,
key: 'files/asset-in-other-project/screenshots/secret_frame.webp',
}),
).rejects.toThrow('User does not have Read permission on asset asset-in-other-project')
expect(authzService.hasPermission).toHaveBeenCalledWith({
user: { id: 'user-1' },
permission: 'Read',
type: 'asset',
id: 'asset-in-other-project',
})
expect(s3Service.getObject).not.toHaveBeenCalled()
})
it('should authorize and download derived artifact key (files/<assetId>/...)', async () => {
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'ast-123',
} as unknown as Asset)
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('fake-screenshot-bytes'),
contentType: 'image/webp',
} as unknown as { buffer: Buffer; contentType: string })
const tool = createDownloadAssetTool('user-1')
const result = await tool.execute('call-1', {
assetId: null,
key: 'files/ast-123/screenshots/shot_5.0s.webp',
})
expect(authzService.hasPermission).toHaveBeenCalledWith({
user: { id: 'user-1' },
permission: 'Read',
type: 'asset',
id: 'ast-123',
})
expect(s3Service.getObject).toHaveBeenCalledWith(
'shumai',
'files/ast-123/screenshots/shot_5.0s.webp',
)
const expectedPath = path.join(piDir, 'shot_5.0s.webp')
createdFiles.push(expectedPath)
expect(fs.existsSync(expectedPath)).toBe(true)
expect(fs.readFileSync(expectedPath, 'utf-8')).toBe('fake-screenshot-bytes')
expect(result.details.filePath).toBe(path.join('.pi', 'shot_5.0s.webp'))
expect(result.details.key).toBe('files/ast-123/screenshots/shot_5.0s.webp')
expect(result.details.assetId).toBe('ast-123')
expect(result.details.size).toBe(Buffer.from('fake-screenshot-bytes').length)
})
it('should authorize and download original upload key (files/<uploadUlid>/...) via StorageKey', async () => {
vi.mocked(prisma.asset.findUnique).mockResolvedValue(null)
vi.mocked(prisma.storageKey.findFirst).mockResolvedValue({
id: 'sk-1',
key: 'files/upload-ulid-456/photo.png',
assets: [{ id: 'asset-photo-1' }],
} as unknown as StorageKey & { assets: Asset[] })
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('fake-original-photo-bytes'),
contentType: 'image/png',
} as unknown as { buffer: Buffer; contentType: string })
const tool = createDownloadAssetTool('user-1')
const result = await tool.execute('call-1', {
assetId: null,
key: 'files/upload-ulid-456/photo.png',
})
expect(authzService.hasPermission).toHaveBeenCalledWith({
user: { id: 'user-1' },
permission: 'Read',
type: 'asset',
id: 'asset-photo-1',
})
expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'files/upload-ulid-456/photo.png')
const expectedPath = path.join(piDir, 'photo.png')
createdFiles.push(expectedPath)
expect(fs.existsSync(expectedPath)).toBe(true)
expect(result.details.assetId).toBe('asset-photo-1')
})
})
describe('resolveAssetIdFromKey helper unit tests', () => {
it('resolves directly by asset ID', async () => {
vi.mocked(prisma.asset.findUnique).mockResolvedValue({ id: 'ast-999' } as unknown as Asset)
const assetId = await resolveAssetIdFromKey('files/ast-999/pdf_pages/page_1.webp')
expect(assetId).toBe('ast-999')
})
it('resolves via StorageKey prefix', async () => {
vi.mocked(prisma.asset.findUnique).mockResolvedValue(null)
vi.mocked(prisma.storageKey.findFirst).mockResolvedValue({
assets: [{ id: 'ast-888' }],
} as unknown as StorageKey & { assets: Asset[] })
const assetId = await resolveAssetIdFromKey('files/ulid-777/document.pdf')
expect(assetId).toBe('ast-888')
})
it('resolves via exact StorageKey match', async () => {
vi.mocked(prisma.asset.findUnique).mockResolvedValue(null)
vi.mocked(prisma.storageKey.findFirst).mockResolvedValue(null)
vi.mocked(prisma.storageKey.findUnique).mockResolvedValue({
assets: [{ id: 'ast-777' }],
} as unknown as StorageKey & { assets: Asset[] })
const assetId = await resolveAssetIdFromKey('files/ulid-special/document.pdf')
expect(assetId).toBe('ast-777')
})
})
})
+208 -83
View File
@@ -7,82 +7,107 @@ import { sanitizeFilename } from '@shumai/core/src/utils/filename'
import * as fs from 'fs'
import * as path from 'path'
const downloadAssetSchema = Type.Object({
assetId: Type.String({
description:
'The asset ID of the workspace asset to download to local disk. This parameter is required.',
}),
})
export const downloadAssetSchema = Type.Object(
{
assetId: Type.Union([
Type.String({
description:
'The asset ID of the workspace asset to download to local disk, or null if providing "key".',
}),
Type.Null(),
]),
key: Type.Union([
Type.String({
description:
'The storage S3 key of a specific media artifact (e.g. a screenshot or transcode key from read_asset) to download to local disk, or null if providing "assetId".',
}),
Type.Null(),
]),
},
{ additionalProperties: false },
)
export async function resolveAssetIdFromKey(key: string): Promise<string> {
if (!key.startsWith('files/') || key.includes('..')) {
throw new Error(
'Invalid storage key format: key must start with "files/" and cannot contain "..".',
)
}
const segments = key.split('/')
if (segments.length < 3) {
throw new Error(
'Invalid storage key format: key must follow "files/<identifier>/<path>" structure.',
)
}
const identifier = segments[1]
// 1. Check if identifier is directly an Asset ID (Derived Artifacts: screenshots, pdf_pages, annotations, proxies)
const assetById = await prisma.asset.findUnique({
where: { id: identifier, isDeleted: false },
select: { id: true },
})
if (assetById) {
return assetById.id
}
// 2. Check if identifier is a StorageKey directory prefix (Original uploads / agent-created files)
const storageKeyByPrefix = await prisma.storageKey.findFirst({
where: { key: { startsWith: `files/${identifier}/` } },
include: {
assets: {
where: { isDeleted: false },
select: { id: true },
},
},
})
if (storageKeyByPrefix?.assets && storageKeyByPrefix.assets.length > 0) {
return storageKeyByPrefix.assets[0].id
}
// 3. Exact StorageKey fallback
const exactStorageKey = await prisma.storageKey.findUnique({
where: { key },
include: {
assets: {
where: { isDeleted: false },
select: { id: true },
},
},
})
if (exactStorageKey?.assets && exactStorageKey.assets.length > 0) {
return exactStorageKey.assets[0].id
}
throw new Error(
`Storage key "${key}" does not belong to any valid asset or the asset has been deleted.`,
)
}
export function createDownloadAssetTool(userId: string): AgentTool<typeof downloadAssetSchema> {
return {
name: 'download_asset',
label: 'Download Asset',
description:
'Downloads a workspace asset (proxy version if available) to the local .pi directory for processing with local scripts or bash commands. Remember to delete the temporary downloaded file when finished.',
'Downloads a workspace asset or specific storage S3 key to the local .pi directory for processing with local scripts or bash commands. ' +
'Provide exactly one of "assetId" (for full assets) or "key" (for intermediate artifacts like screenshots from read_asset). ' +
'Remember to delete the temporary downloaded file when finished.',
parameters: downloadAssetSchema,
execute: async (_toolCallId, params) => {
const assetId = params.assetId
const { assetId, key } = params
if (!userId) {
throw new Error('User ID is required for authorization.')
}
await authzService.hasPermission({
user: { id: userId } as User,
permission: Permission.Read,
type: ResourceType.Asset,
id: assetId,
})
const hasAssetId = assetId !== null
const hasKey = key !== null
let asset = await prisma.asset.findUnique({
where: { id: assetId },
include: { storageKey: true },
})
if (!asset) {
throw new Error(`Asset with ID ${assetId} not found.`)
}
if (asset.type === 'version_stack') {
const latestVersion = await prisma.asset.findFirst({
where: { parentId: asset.id, isDeleted: false },
orderBy: { sortIndex: 'asc' },
include: { storageKey: true },
})
if (latestVersion) {
asset = latestVersion
}
}
let mediaKey = asset.storageKey?.key
if (asset.media) {
const mediaInfo = asset.media as unknown as PrismaJson.MediaInfo
if (
mediaInfo.proxyType === 'image' ||
(mediaInfo.imageTranscodes && mediaInfo.imageTranscodes.length > 0)
) {
if (mediaInfo.imageTranscodes && mediaInfo.imageTranscodes.length > 0) {
mediaKey = mediaInfo.imageTranscodes[0].key || mediaKey
}
} else if (
mediaInfo.proxyType === 'video' ||
(mediaInfo.videoTranscodes && mediaInfo.videoTranscodes.length > 0) ||
mediaInfo.videoPreview?.key
) {
if (mediaInfo.videoTranscodes && mediaInfo.videoTranscodes.length > 0) {
mediaKey = mediaInfo.videoTranscodes[0].key || mediaKey
} else if (mediaInfo.videoPreview?.key) {
mediaKey = mediaInfo.videoPreview.key
}
} else if (mediaInfo.original?.key) {
mediaKey = mediaInfo.original.key || mediaKey
}
}
if (!mediaKey) {
throw new Error(`No media content found for asset ${assetId}.`)
if (hasAssetId === hasKey) {
throw new Error(
'Provide exactly one of "assetId" (to download a workspace asset) or "key" (to download a specific S3 storage key). Set the unused parameter to null.',
)
}
const piDir = path.join(process.cwd(), '.pi')
@@ -90,32 +115,132 @@ export function createDownloadAssetTool(userId: string): AgentTool<typeof downlo
fs.mkdirSync(piDir, { recursive: true })
}
const safeName = sanitizeFilename(asset.name || 'file')
const filename = `${asset.id}_${safeName}`
const targetFilePath = path.join(piDir, filename)
const relativePath = path.join('.pi', filename)
const bucket = process.env.S3_BUCKET || 'shumai'
const { buffer, contentType } = await s3Service.getObject(bucket, mediaKey)
fs.writeFileSync(targetFilePath, buffer)
// Branch A: Download by assetId
if (assetId) {
await authzService.hasPermission({
user: { id: userId } as User,
permission: Permission.Read,
type: ResourceType.Asset,
id: assetId,
})
return {
content: [
{
type: 'text',
text: `Downloaded asset "${asset.name}" (ID: ${assetId}) to "${relativePath}". Remember to delete this temporary file when finished.`,
let asset = await prisma.asset.findUnique({
where: { id: assetId },
include: { storageKey: true },
})
if (!asset) {
throw new Error(`Asset with ID ${assetId} not found.`)
}
if (asset.type === 'version_stack') {
const latestVersion = await prisma.asset.findFirst({
where: { parentId: asset.id, isDeleted: false },
orderBy: { sortIndex: 'asc' },
include: { storageKey: true },
})
if (latestVersion) {
asset = latestVersion
}
}
let mediaKey = asset.storageKey?.key
if (asset.media) {
const mediaInfo = asset.media as unknown as PrismaJson.MediaInfo
if (
mediaInfo.proxyType === 'image' ||
(mediaInfo.imageTranscodes && mediaInfo.imageTranscodes.length > 0)
) {
if (mediaInfo.imageTranscodes && mediaInfo.imageTranscodes.length > 0) {
mediaKey = mediaInfo.imageTranscodes[0].key || mediaKey
}
} else if (
mediaInfo.proxyType === 'video' ||
(mediaInfo.videoTranscodes && mediaInfo.videoTranscodes.length > 0) ||
mediaInfo.videoPreview?.key
) {
if (mediaInfo.videoTranscodes && mediaInfo.videoTranscodes.length > 0) {
mediaKey = mediaInfo.videoTranscodes[0].key || mediaKey
} else if (mediaInfo.videoPreview?.key) {
mediaKey = mediaInfo.videoPreview.key
}
} else if (mediaInfo.original?.key) {
mediaKey = mediaInfo.original.key || mediaKey
}
}
if (!mediaKey) {
throw new Error(`No media content found for asset ${assetId}.`)
}
const safeName = sanitizeFilename(asset.name || 'file')
const filename = `${asset.id}_${safeName}`
const targetFilePath = path.join(piDir, filename)
const relativePath = path.join('.pi', filename)
const { buffer, contentType } = await s3Service.getObject(bucket, mediaKey)
fs.writeFileSync(targetFilePath, buffer)
return {
content: [
{
type: 'text',
text: `Downloaded asset "${asset.name}" (ID: ${assetId}) to "${relativePath}". Remember to delete this temporary file when finished.`,
},
],
details: {
assetId,
name: asset.name,
filePath: relativePath,
absolutePath: targetFilePath,
contentType: contentType || 'application/octet-stream',
size: buffer.length,
},
],
details: {
assetId,
name: asset.name,
filePath: relativePath,
absolutePath: targetFilePath,
contentType: contentType || 'application/octet-stream',
size: buffer.length,
},
}
}
// Branch B: Download by S3 storage key
if (key) {
const owningAssetId = await resolveAssetIdFromKey(key)
await authzService.hasPermission({
user: { id: userId } as User,
permission: Permission.Read,
type: ResourceType.Asset,
id: owningAssetId,
})
const rawBasename = path.basename(key)
const safeBasename = sanitizeFilename(rawBasename) || 'downloaded_file'
const filename = safeBasename
const targetFilePath = path.join(piDir, filename)
const relativePath = path.join('.pi', filename)
const { buffer, contentType } = await s3Service.getObject(bucket, key)
fs.writeFileSync(targetFilePath, buffer)
return {
content: [
{
type: 'text',
text: `Downloaded storage key "${key}" to "${relativePath}". Remember to delete this temporary file when finished.`,
},
],
details: {
key,
assetId: owningAssetId,
filePath: relativePath,
absolutePath: targetFilePath,
contentType: contentType || 'application/octet-stream',
size: buffer.length,
},
}
}
throw new Error('Invalid download_asset parameters.')
},
}
}
+544
View File
@@ -0,0 +1,544 @@
import { describe, it, expect, vi, beforeEach } from 'vitest'
import { createReadAssetTool } from './read-asset'
import {
prisma,
WorkflowTaskType,
WorkflowTaskStatus,
type Asset,
type WorkflowTask,
} from '@shumai/db'
import { s3Service } from '@shumai/core/src/s3/s3'
import { workflowService } from '@shumai/workflow-core'
import { authzService } from '@shumai/core/src/authz/authz'
import * as annotationResolver from './annotation-resolver'
vi.mock('@shumai/db', async (importOriginal) => {
const actual = await importOriginal<typeof import('@shumai/db')>()
return {
...actual,
prisma: {
asset: {
findUnique: vi.fn(),
findFirst: vi.fn(),
},
workflowTask: {
create: vi.fn(),
},
},
}
})
vi.mock('@shumai/core/src/s3/s3', () => ({
s3Service: {
getObject: vi.fn(),
presign: vi.fn().mockResolvedValue('https://mock-presigned-url.com'),
},
}))
vi.mock('@shumai/workflow-core', () => ({
workflowService: {
executeWait: vi.fn(),
},
}))
vi.mock('@shumai/core/src/authz/authz', () => ({
authzService: {
hasPermission: vi.fn(),
},
Permission: { Read: 'Read' },
ResourceType: { Asset: 'asset' },
}))
describe('readAssetTool', () => {
beforeEach(() => {
vi.clearAllMocks()
vi.spyOn(annotationResolver, 'resolveAnnotationsById').mockResolvedValue({
annotations: null,
timestamp: null,
})
})
it('should throw authorization error if user ID is empty', async () => {
const tool = createReadAssetTool('')
await expect(
tool.execute('call-1', {
assetId: 'asset-1',
annotationId: null,
imageConfig: null,
videoConfig: null,
docConfig: null,
}),
).rejects.toThrow('User ID is required for authorization.')
})
it('should throw error if asset not found', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue(null)
const tool = createReadAssetTool('user-1')
await expect(
tool.execute('call-1', {
assetId: 'asset-1',
annotationId: null,
imageConfig: null,
videoConfig: null,
docConfig: null,
}),
).rejects.toThrow('Asset with ID asset-1 not found.')
})
describe('Image Assets', () => {
it('should read image and return ImageContent and S3 key', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'img-1',
name: 'photo.png',
mediaType: 'image/png',
storageKey: { key: 'raw/photo.png' },
media: {
proxyType: 'image',
imageTranscodes: [{ key: 'proxy/photo.webp' }],
},
} as unknown as Asset)
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('fake-image-bytes'),
contentType: 'image/webp',
} as unknown as { buffer: Buffer; contentType: string })
const tool = createReadAssetTool('user-1')
const result = await tool.execute('call-1', {
assetId: 'img-1',
annotationId: null,
imageConfig: null,
videoConfig: null,
docConfig: null,
})
expect(authzService.hasPermission).toHaveBeenCalledWith({
user: { id: 'user-1' },
permission: 'Read',
type: 'asset',
id: 'img-1',
})
expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'proxy/photo.webp')
const textItem = result.content[0] as { type: 'text'; text: string }
const imageItem = result.content[1] as { type: 'image'; data: string; mimeType: string }
expect(textItem.text).toContain('proxy/photo.webp')
expect(imageItem.type).toBe('image')
expect(imageItem.data).toBe(Buffer.from('fake-image-bytes').toString('base64'))
expect(result.details.key).toBe('proxy/photo.webp')
})
it('should trigger image annotation transcode workflow when annotationId is provided', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'img-1',
name: 'photo.png',
mediaType: 'image/png',
projectId: 'project-1',
storageKey: { key: 'raw/photo.png' },
media: {
proxyType: 'image',
imageTranscodes: [{ key: 'proxy/photo.webp' }],
},
} as unknown as Asset)
vi.spyOn(annotationResolver, 'resolveAnnotationsById').mockResolvedValue({
annotations: [
{
type: 'arrow',
color: '#ff0000',
points: [
[0, 0],
[10, 10],
],
},
],
timestamp: null,
})
vi.mocked(prisma.workflowTask.create).mockResolvedValue({
id: 'task-anno',
} as unknown as WorkflowTask)
vi.mocked(workflowService.executeWait).mockResolvedValue({
id: 'task-anno',
output: { key: 'annotated/photo.webp' },
} as unknown as WorkflowTask)
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('annotated-bytes'),
contentType: 'image/webp',
} as unknown as { buffer: Buffer; contentType: string })
const tool = createReadAssetTool('user-1')
const result = await tool.execute('call-1', {
assetId: 'img-1',
annotationId: 'comment-1',
imageConfig: null,
videoConfig: null,
docConfig: null,
})
expect(prisma.workflowTask.create).toHaveBeenCalledWith({
data: {
assetId: 'img-1',
projectId: 'project-1',
type: WorkflowTaskType.transcode_image_annotation,
status: WorkflowTaskStatus.pending,
payload: {
projectId: 'project-1',
imageAnnotation: {
annotations: [
{
type: 'arrow',
color: '#ff0000',
points: [
[0, 0],
[10, 10],
],
},
],
},
},
},
})
expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'annotated/photo.webp')
expect(result.details.key).toBe('annotated/photo.webp')
})
})
describe('Video Assets', () => {
it('should throw error if videoConfig is missing for video assets', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'vid-1',
name: 'video.mp4',
mediaType: 'video/mp4',
media: { proxyType: 'video' },
} as unknown as Asset)
const tool = createReadAssetTool('user-1')
await expect(
tool.execute('call-1', {
assetId: 'vid-1',
annotationId: null,
imageConfig: null,
videoConfig: null,
docConfig: null,
}),
).rejects.toThrow('videoConfig with start, end, and count is required')
})
it('should throw error on invalid video time range', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'vid-1',
name: 'video.mp4',
mediaType: 'video/mp4',
media: { proxyType: 'video' },
} as unknown as Asset)
const tool = createReadAssetTool('user-1')
await expect(
tool.execute('call-1', {
assetId: 'vid-1',
annotationId: null,
imageConfig: null,
videoConfig: { start: 10, end: 5, count: 1 },
docConfig: null,
}),
).rejects.toThrow(
'Invalid video time range: start (10) must be less than or equal to end (5).',
)
})
it('should extract frames and return ImageContent array and keys', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'vid-1',
name: 'video.mp4',
mediaType: 'video/mp4',
projectId: 'project-1',
media: { proxyType: 'video' },
} as unknown as Asset)
vi.mocked(prisma.workflowTask.create).mockResolvedValue({
id: 'task-shot',
} as unknown as WorkflowTask)
vi.mocked(workflowService.executeWait).mockResolvedValue({
id: 'task-shot',
output: {
screenshots: [
{ key: 'screenshots/shot1.webp', timestamp: 2.0 },
{ key: 'screenshots/shot2.webp', timestamp: 4.0 },
],
},
} as unknown as WorkflowTask)
vi.mocked(s3Service.getObject).mockImplementation(async (_bucket, key) => ({
buffer: Buffer.from(`bytes-for-${key}`),
contentType: 'image/webp',
}))
const tool = createReadAssetTool('user-1')
const result = await tool.execute('call-1', {
assetId: 'vid-1',
annotationId: null,
imageConfig: null,
videoConfig: { start: 2.0, end: 4.0, count: 2 },
docConfig: null,
})
expect(prisma.workflowTask.create).toHaveBeenCalledWith({
data: {
assetId: 'vid-1',
projectId: 'project-1',
type: WorkflowTaskType.transcode_screenshot,
status: WorkflowTaskStatus.pending,
payload: {
projectId: 'project-1',
screenshot: {
start: 2.0,
end: 4.0,
count: 2,
commentTimestamp: null,
annotations: null,
},
},
},
})
expect(result.content.length).toBe(3) // 1 text + 2 images
expect(result.details.keys).toEqual(['screenshots/shot1.webp', 'screenshots/shot2.webp'])
})
})
describe('Document Assets', () => {
it('should throw error if docConfig is missing for document assets', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'doc-1',
name: 'doc.pdf',
mediaType: 'application/pdf',
media: { proxyType: 'pdf' },
} as unknown as Asset)
const tool = createReadAssetTool('user-1')
await expect(
tool.execute('call-1', {
assetId: 'doc-1',
annotationId: null,
imageConfig: null,
videoConfig: null,
docConfig: null,
}),
).rejects.toThrow('docConfig with mode ("pages" or "text") is required')
})
it('should render PDF pages in mode: "pages"', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'doc-1',
name: 'doc.pdf',
mediaType: 'application/pdf',
projectId: 'project-1',
media: { proxyType: 'pdf' },
} as unknown as Asset)
vi.mocked(prisma.workflowTask.create).mockResolvedValue({
id: 'task-pdf',
} as unknown as WorkflowTask)
vi.mocked(workflowService.executeWait).mockResolvedValue({
id: 'task-pdf',
output: {
pages: [
{ key: 'pdf_pages/page1.webp', page: 1 },
{ key: 'pdf_pages/page2.webp', page: 2 },
],
},
} as unknown as WorkflowTask)
vi.mocked(s3Service.getObject).mockImplementation(async (_bucket, key) => ({
buffer: Buffer.from(`bytes-for-${key}`),
contentType: 'image/webp',
}))
const tool = createReadAssetTool('user-1')
const result = await tool.execute('call-1', {
assetId: 'doc-1',
annotationId: null,
imageConfig: null,
videoConfig: null,
docConfig: { mode: 'pages', startPage: 1, endPage: 2 },
})
expect(result.content.length).toBe(3) // 1 text + 2 images
expect(result.details.keys).toEqual(['pdf_pages/page1.webp', 'pdf_pages/page2.webp'])
})
it('should throw error in mode: "pages" when page range exceeds 20 pages', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'doc-1',
name: 'doc.pdf',
mediaType: 'application/pdf',
media: { proxyType: 'pdf' },
} as unknown as Asset)
const tool = createReadAssetTool('user-1')
await expect(
tool.execute('call-1', {
assetId: 'doc-1',
annotationId: null,
imageConfig: null,
videoConfig: null,
docConfig: { mode: 'pages', startPage: 1, endPage: 25 },
}),
).rejects.toThrow('exceeds the maximum limit of 20 pages')
})
it('should throw error in mode: "text" when called on binary PDF', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'doc-1',
name: 'manual.pdf',
mediaType: 'application/pdf',
media: { proxyType: 'pdf' },
} as unknown as Asset)
const tool = createReadAssetTool('user-1')
await expect(
tool.execute('call-1', {
assetId: 'doc-1',
annotationId: null,
imageConfig: null,
videoConfig: null,
docConfig: { mode: 'text', startPage: null, endPage: null },
}),
).rejects.toThrow('is a binary PDF/document and cannot be read as raw text')
})
it('should throw error in mode: "text" when annotationId is provided', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'doc-1',
name: 'notes.md',
mediaType: 'text/markdown',
media: { proxyType: 'pdf' },
} as unknown as Asset)
const tool = createReadAssetTool('user-1')
await expect(
tool.execute('call-1', {
assetId: 'doc-1',
annotationId: 'comment-123',
imageConfig: null,
videoConfig: null,
docConfig: { mode: 'text', startPage: null, endPage: null },
}),
).rejects.toThrow('annotationId cannot be used with docConfig mode "text"')
})
it('should throw error in mode: "text" when startPage or endPage is non-null', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'doc-1',
name: 'notes.md',
mediaType: 'text/markdown',
media: { proxyType: 'pdf' },
} as unknown as Asset)
const tool = createReadAssetTool('user-1')
await expect(
tool.execute('call-1', {
assetId: 'doc-1',
annotationId: null,
imageConfig: null,
videoConfig: null,
docConfig: { mode: 'text', startPage: 1, endPage: 2 },
}),
).rejects.toThrow('startPage and endPage must be null when docConfig mode is "text"')
})
it('should read raw text from S3 in mode: "text" for markdown files', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'doc-md',
name: 'README.md',
mediaType: 'text/markdown',
storageKey: { key: 'raw/README.md' },
media: { proxyType: 'pdf' },
} as unknown as Asset)
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('# Hello Shumai\nThis is markdown text.'),
contentType: 'text/markdown',
} as unknown as { buffer: Buffer; contentType: string })
const tool = createReadAssetTool('user-1')
const result = await tool.execute('call-1', {
assetId: 'doc-md',
annotationId: null,
imageConfig: null,
videoConfig: null,
docConfig: { mode: 'text', startPage: null, endPage: null },
})
expect(s3Service.getObject).toHaveBeenCalledWith('shumai', 'raw/README.md')
expect(result.content[0].type).toBe('text')
expect((result.content[0] as { type: 'text'; text: string }).text).toContain(
'# Hello Shumai\nThis is markdown text.',
)
expect(result.details.key).toBe('raw/README.md')
})
})
describe('Version Stack Resolution', () => {
it('should resolve version_stack to latest version asset', async () => {
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'stack-1',
name: '',
type: 'version_stack',
} as unknown as Asset)
vi.mocked(prisma.asset.findFirst).mockResolvedValue({
id: 'file-v2',
name: 'diagram_v2.png',
type: 'file',
mediaType: 'image/png',
storageKey: { key: 'raw/diagram_v2.png' },
media: {
proxyType: 'image',
imageTranscodes: [{ key: 'proxy/diagram_v2.webp' }],
},
} as unknown as Asset)
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('v2-image-bytes'),
contentType: 'image/webp',
} as unknown as { buffer: Buffer; contentType: string })
const tool = createReadAssetTool('user-1')
const result = await tool.execute('call-1', {
assetId: 'stack-1',
annotationId: null,
imageConfig: null,
videoConfig: null,
docConfig: null,
})
expect(prisma.asset.findFirst).toHaveBeenCalledWith({
where: { parentId: 'stack-1', isDeleted: false },
orderBy: { sortIndex: 'asc' },
include: { storageKey: true },
})
expect(result.details.name).toBe('diagram_v2.png')
expect(result.details.key).toBe('proxy/diagram_v2.webp')
})
})
})
+576
View File
@@ -0,0 +1,576 @@
import { Type } from 'typebox'
import { type AgentTool } from '@earendil-works/pi-agent-core'
import { type ImageContent, type TextContent } from '@earendil-works/pi-ai'
import { prisma, WorkflowTaskType, WorkflowTaskStatus, type User } from '@shumai/db'
import { s3Service } from '@shumai/core/src/s3/s3'
import { workflowService } from '@shumai/workflow-core'
import { authzService, Permission, ResourceType } from '@shumai/core/src/authz/authz'
import { resolveAnnotationsById } from './annotation-resolver'
export const readAssetSchema = Type.Object(
{
assetId: Type.String({
description: 'The asset ID of the workspace asset to inspect. Required.',
}),
annotationId: Type.Union([
Type.String({
description:
'The ID of the comment or message entry from <annotation id="..." /> whose visual markup should be overlaid, or null.',
}),
Type.Null(),
]),
imageConfig: Type.Union([
Type.Object(
{},
{
additionalProperties: false,
description: 'Optional configuration for image assets, or null.',
},
),
Type.Null(),
]),
videoConfig: Type.Union([
Type.Object(
{
start: Type.Number({ description: 'Start time in seconds.' }),
end: Type.Number({ description: 'End time in seconds.' }),
count: Type.Number({ description: 'Number of frames to extract (e.g. 1 to 10).' }),
},
{
additionalProperties: false,
description:
'Configuration for video assets. Required when inspecting videos, or null for other asset types.',
},
),
Type.Null(),
]),
docConfig: Type.Union([
Type.Object(
{
mode: Type.Union([
Type.Literal('pages', {
description:
'Render specific page range as images for visual layout, charts, or visual annotations (required for binary PDFs).',
}),
Type.Literal('text', {
description:
'Read raw text content directly (ideal for .md, .txt, .csv, code files).',
}),
]),
startPage: Type.Union([
Type.Number({
description:
'Start page number (1-based index, required when mode="pages"), or null when mode="text".',
}),
Type.Null(),
]),
endPage: Type.Union([
Type.Number({
description:
'End page number (1-based index, max 20 pages per call, required when mode="pages"), or null when mode="text".',
}),
Type.Null(),
]),
},
{
additionalProperties: false,
description:
'Configuration for document/PDF/text assets, or null if inspecting another asset type.',
},
),
Type.Null(),
]),
},
{ additionalProperties: false },
)
export function isPlainTextAsset(mediaType: string, filename: string): boolean {
if (
mediaType.startsWith('text/') ||
mediaType === 'application/json' ||
mediaType === 'application/javascript' ||
mediaType === 'application/typescript' ||
mediaType === 'application/xml' ||
mediaType === 'application/x-yaml'
) {
return true
}
const textExtensions = [
'.txt',
'.md',
'.markdown',
'.json',
'.csv',
'.tsv',
'.js',
'.ts',
'.tsx',
'.jsx',
'.html',
'.htm',
'.css',
'.scss',
'.yaml',
'.yml',
'.xml',
'.py',
'.rb',
'.go',
'.rs',
'.c',
'.cpp',
'.h',
'.hpp',
'.java',
'.sh',
'.bash',
'.zsh',
'.sql',
]
return textExtensions.some((ext) => filename.endsWith(ext))
}
const MAX_TEXT_BYTES = 50 * 1024
const MAX_TEXT_LINES = 1000
export function createReadAssetTool(userId: string): AgentTool<typeof readAssetSchema> {
return {
name: 'read_asset',
label: 'Read Asset',
description:
'Inspects and reads the content of an asset in the workspace. ' +
'Supports visual analysis for images, video frame extraction, PDF page rendering, and direct text reading for text/markdown files. ' +
'Pass assetId, and provide the specific config (imageConfig, videoConfig, or docConfig) while setting unused configs to null. ' +
'Optionally provide annotationId to view visual markups drawn on the asset.',
parameters: readAssetSchema,
execute: async (_toolCallId, params) => {
const { assetId, annotationId, videoConfig, docConfig } = params
if (!userId) {
throw new Error('User ID is required for authorization.')
}
await authzService.hasPermission({
user: { id: userId } as User,
permission: Permission.Read,
type: ResourceType.Asset,
id: assetId,
})
let asset = await prisma.asset.findUnique({
where: { id: assetId },
include: { storageKey: true },
})
if (!asset) {
throw new Error(`Asset with ID ${assetId} not found.`)
}
if (asset.type === 'version_stack') {
const latestVersion = await prisma.asset.findFirst({
where: { parentId: asset.id, isDeleted: false },
orderBy: { sortIndex: 'asc' },
include: { storageKey: true },
})
if (latestVersion) {
asset = latestVersion
}
}
const mediaInfo = asset.media as unknown as PrismaJson.MediaInfo | null
const proxyType = mediaInfo?.proxyType
const mediaType = asset.mediaType?.toLowerCase() || ''
const filename = asset.name?.toLowerCase() || ''
const bucket = process.env.S3_BUCKET || 'shumai'
// ----------------------------------------------------------------------
// Branch 1: Image Assets
// ----------------------------------------------------------------------
if (proxyType === 'image' || mediaType.startsWith('image/') || filename.endsWith('.psd')) {
let mediaKey = asset.storageKey?.key
if (mediaInfo?.imageTranscodes && mediaInfo.imageTranscodes.length > 0) {
mediaKey = mediaInfo.imageTranscodes[0].key || mediaKey
}
if (!mediaKey) {
throw new Error('No media content found for this asset.')
}
const { annotations } = await resolveAnnotationsById(asset.id, annotationId)
let keyToUse = mediaKey
if (annotations) {
const task = await prisma.workflowTask.create({
data: {
assetId: asset.id,
projectId: asset.projectId || 'none',
type: WorkflowTaskType.transcode_image_annotation,
status: WorkflowTaskStatus.pending,
payload: {
projectId: asset.projectId || 'none',
imageAnnotation: {
annotations,
},
},
},
})
const completedTask = await workflowService.executeWait(task)
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- transcode image annotation task output
const output = completedTask.output as any
if (output?.key) {
keyToUse = output.key
} else {
throw new Error('Image annotation transcode workflow failed to output S3 key.')
}
}
const { buffer, contentType } = await s3Service.getObject(bucket, keyToUse)
const mimeType =
contentType && contentType !== 'application/octet-stream' ? contentType : 'image/webp'
const downloadUrl = await s3Service.presign(
bucket,
keyToUse,
'GET',
true,
asset.name || undefined,
)
const content: Array<TextContent | ImageContent> = [
{
type: 'text',
text: `Image asset "${asset.name}" (ID: ${asset.id}):\n- S3 Key: "${keyToUse}"`,
},
{
type: 'image',
data: buffer.toString('base64'),
mimeType,
},
]
return {
content,
details: {
assetId: asset.id,
name: asset.name,
mediaType: asset.mediaType,
key: keyToUse,
downloadUrl,
sourceKeys: [keyToUse],
},
}
}
// ----------------------------------------------------------------------
// Branch 2: Video Assets
// ----------------------------------------------------------------------
if (proxyType === 'video' || mediaType.startsWith('video/')) {
if (!videoConfig) {
throw new Error(
'videoConfig with start, end, and count is required when inspecting video assets.',
)
}
const { start, end, count } = videoConfig
if (start < 0 || end < 0) {
throw new Error('Invalid video time range: start and end must be non-negative numbers.')
}
if (start > end) {
throw new Error(
`Invalid video time range: start (${start}) must be less than or equal to end (${end}).`,
)
}
if (count < 1) {
throw new Error(`Invalid frame count: count (${count}) must be at least 1.`)
}
const { annotations, timestamp: commentTimestamp } = await resolveAnnotationsById(
asset.id,
annotationId,
)
const task = await prisma.workflowTask.create({
data: {
assetId: asset.id,
projectId: asset.projectId || 'none',
type: WorkflowTaskType.transcode_screenshot,
status: WorkflowTaskStatus.pending,
payload: {
projectId: asset.projectId || 'none',
screenshot: {
start,
end,
count,
commentTimestamp,
annotations,
},
},
},
})
const completedTask = await workflowService.executeWait(task)
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- transcode screenshot output
const output = completedTask.output as any
const screenshots = output?.screenshots as
| Array<{ key: string; timestamp: number }>
| undefined
if (!screenshots || screenshots.length === 0) {
throw new Error('No screenshots were generated.')
}
const content: Array<TextContent | ImageContent> = []
const sourceKeys: string[] = []
const results: Array<{ key: string; timestamp: number; downloadUrl: string }> = []
const textLines: string[] = [
`Extracted ${screenshots.length} frame(s) from "${asset.name}" (${start}s - ${end}s):`,
]
for (const shot of screenshots) {
const { buffer, contentType } = await s3Service.getObject(bucket, shot.key)
const mimeType =
contentType && contentType !== 'application/octet-stream' ? contentType : 'image/webp'
const downloadUrl = await s3Service.presign(
bucket,
shot.key,
'GET',
true,
asset.name || undefined,
)
textLines.push(`- Frame at ${shot.timestamp}s: S3 Key: "${shot.key}"`)
content.push({
type: 'image',
data: buffer.toString('base64'),
mimeType,
})
sourceKeys.push(shot.key)
results.push({
key: shot.key,
timestamp: shot.timestamp,
downloadUrl,
})
}
content.unshift({
type: 'text',
text: textLines.join('\n'),
})
return {
content,
details: {
assetId: asset.id,
name: asset.name,
mediaType: asset.mediaType,
keys: sourceKeys,
results,
sourceKeys,
},
}
}
// ----------------------------------------------------------------------
// Branch 3: Documents & PDF Assets
// ----------------------------------------------------------------------
if (
proxyType === 'pdf' ||
mediaType === 'application/pdf' ||
isPlainTextAsset(mediaType, filename)
) {
if (!docConfig) {
throw new Error(
'docConfig with mode ("pages" or "text") is required when inspecting document/PDF assets.',
)
}
if (docConfig.mode === 'text') {
if (annotationId !== null) {
throw new Error(
'annotationId cannot be used with docConfig mode "text" because visual markups can only be rendered on visual pages. Use docConfig: { mode: "pages", startPage: ..., endPage: ... } instead.',
)
}
if (docConfig.startPage !== null || docConfig.endPage !== null) {
throw new Error(
'startPage and endPage must be null when docConfig mode is "text". To inspect specific page numbers, use mode: "pages".',
)
}
if (!isPlainTextAsset(mediaType, filename)) {
throw new Error(
`Asset "${asset.name}" is a binary PDF/document and cannot be read as raw text. Please call read_asset with docConfig: { mode: "pages", startPage: 1, endPage: ... } to view its visual pages.`,
)
}
const mediaKey = asset.storageKey?.key
if (!mediaKey) {
throw new Error(`No media content found for asset ${asset.id}.`)
}
const { buffer } = await s3Service.getObject(bucket, mediaKey)
const rawText = buffer.toString('utf-8')
let text = rawText
let isTruncated = false
const lines = text.split('\n')
if (lines.length > MAX_TEXT_LINES) {
text = lines.slice(0, MAX_TEXT_LINES).join('\n')
isTruncated = true
}
if (Buffer.byteLength(text, 'utf-8') > MAX_TEXT_BYTES) {
text = Buffer.from(text, 'utf-8').subarray(0, MAX_TEXT_BYTES).toString('utf-8')
isTruncated = true
}
if (isTruncated) {
text += `\n\n[Content truncated to ${MAX_TEXT_LINES} lines / ${MAX_TEXT_BYTES / 1024}KB limit. Use download_asset to process the full file.]`
}
const downloadUrl = await s3Service.presign(
bucket,
mediaKey,
'GET',
true,
asset.name || undefined,
)
return {
content: [
{
type: 'text',
text: `Document "${asset.name}" (ID: ${asset.id}, S3 Key: "${mediaKey}"):\n\n${text}`,
},
],
details: {
assetId: asset.id,
name: asset.name,
mediaType: asset.mediaType,
key: mediaKey,
downloadUrl,
sourceKeys: [mediaKey],
size: buffer.length,
isTruncated,
},
}
}
// docConfig.mode === 'pages'
if (docConfig.startPage === null || docConfig.endPage === null) {
throw new Error('startPage and endPage are required when docConfig mode is "pages".')
}
const start = docConfig.startPage
const end = docConfig.endPage
if (start < 1) {
throw new Error(`Invalid page range: start page (${start}) must be at least 1.`)
}
if (start > end) {
throw new Error(
`Invalid page range: start page (${start}) must be less than or equal to end page (${end}).`,
)
}
const MAX_PAGE_RANGE = 20
const requestedPages = end - start + 1
if (requestedPages > MAX_PAGE_RANGE) {
throw new Error(
`Page range (${requestedPages} pages requested) exceeds the maximum limit of ${MAX_PAGE_RANGE} pages per request.`,
)
}
const { annotations, timestamp: commentTimestamp } = await resolveAnnotationsById(
asset.id,
annotationId,
)
const task = await prisma.workflowTask.create({
data: {
assetId: asset.id,
projectId: asset.projectId || 'none',
type: WorkflowTaskType.transcode_pdf_pages,
status: WorkflowTaskStatus.pending,
payload: {
projectId: asset.projectId || 'none',
pdfPages: {
start,
end,
commentTimestamp,
annotations,
},
},
},
})
const completedTask = await workflowService.executeWait(task)
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- transcode pdf pages output
const output = completedTask.output as any
const pages = output?.pages as Array<{ key: string; page: number }> | undefined
if (!pages || pages.length === 0) {
throw new Error('No PDF page images were generated.')
}
const content: Array<TextContent | ImageContent> = []
const sourceKeys: string[] = []
const results: Array<{ key: string; page: number; downloadUrl: string }> = []
const textLines: string[] = [
`Rendered ${pages.length} page(s) of "${asset.name}" (Pages ${start} to ${end}):`,
]
for (const pageItem of pages) {
const { buffer, contentType } = await s3Service.getObject(bucket, pageItem.key)
const mimeType =
contentType && contentType !== 'application/octet-stream' ? contentType : 'image/webp'
const downloadUrl = await s3Service.presign(
bucket,
pageItem.key,
'GET',
true,
asset.name || undefined,
)
textLines.push(`- Page ${pageItem.page}: S3 Key: "${pageItem.key}"`)
content.push({
type: 'image',
data: buffer.toString('base64'),
mimeType,
})
sourceKeys.push(pageItem.key)
results.push({
key: pageItem.key,
page: pageItem.page,
downloadUrl,
})
}
content.unshift({
type: 'text',
text: textLines.join('\n'),
})
return {
content,
details: {
assetId: asset.id,
name: asset.name,
mediaType: asset.mediaType,
keys: sourceKeys,
results,
sourceKeys,
},
}
}
throw new Error(
`Unsupported asset type for read_asset. Asset ${asset.id} has media type "${mediaType || 'unknown'}". Use download_asset to download this file to disk.`,
)
},
}
}
@@ -1,301 +0,0 @@
import { describe, it, expect, vi, beforeEach } from 'vitest'
import { createReadPdfPagesTool } from './read-pdf-pages'
import {
prisma,
WorkflowTaskStatus,
type Asset,
type AssetComment,
type AgentSessionEntry,
type User,
type WorkflowTask,
} from '@shumai/db'
import { s3Service } from '@shumai/core/src/s3/s3'
import { workflowService } from '@shumai/workflow-core'
import { authzService } from '@shumai/core/src/authz/authz'
vi.mock('@shumai/db', async (importOriginal) => {
const actual = await importOriginal<typeof import('@shumai/db')>()
return {
...actual,
prisma: {
user: {
findUnique: vi.fn(),
},
asset: {
findUnique: vi.fn(),
},
assetComment: {
findUnique: vi.fn(),
},
agentSessionEntry: {
findUnique: vi.fn(),
},
workflowTask: {
create: vi.fn(),
},
},
}
})
vi.mock('@shumai/core/src/s3/s3', () => ({
s3Service: {
getObject: vi.fn(),
},
}))
vi.mock('@shumai/workflow-core', () => ({
workflowService: {
executeWait: vi.fn(),
},
}))
vi.mock('@shumai/core/src/authz/authz', () => ({
authzService: {
hasPermission: vi.fn(),
},
Permission: { Read: 'Read' },
ResourceType: { Asset: 'asset' },
}))
describe('readPdfPagesTool', () => {
beforeEach(() => {
vi.clearAllMocks()
vi.mocked(authzService.hasPermission).mockResolvedValue()
})
it('should trigger pdfPages transcode workflow and return image outputs with annotation from AssetComment', async () => {
vi.mocked(prisma.user.findUnique).mockResolvedValue({ id: 'user-1' } as User)
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'asset-1',
projectId: 'project-1',
media: { proxyType: 'pdf' },
} as unknown as Asset)
vi.mocked(prisma.assetComment.findUnique).mockResolvedValue({
id: 'comment-1',
assetId: 'asset-1',
second: 2,
annotation: [
{
type: 'box',
color: '#ff0000',
points: [
[0, 0],
[1, 1],
],
},
],
} as unknown as AssetComment)
vi.mocked(prisma.workflowTask.create).mockResolvedValue({
id: 'task-1',
} as unknown as WorkflowTask)
vi.mocked(workflowService.executeWait).mockResolvedValue({
id: 'task-1',
status: WorkflowTaskStatus.completed,
output: {
pages: [
{ key: 'pdf_pages/page_1.webp', page: 1 },
{ key: 'pdf_pages/page_2.webp', page: 2 },
],
},
} as unknown as WorkflowTask)
vi.mocked(s3Service.getObject).mockImplementation(async (_bucket: string, key: string) => {
return {
buffer: Buffer.from(`fake-bytes-for-${key}`),
contentType: 'image/webp',
} as unknown as { buffer: Buffer; contentType: string }
})
const tool = createReadPdfPagesTool('user-1')
const result = await tool.execute('call-1', {
assetId: 'asset-1',
start: 1,
end: 2,
annotationId: 'comment-1',
})
expect(authzService.hasPermission).toHaveBeenCalledWith({
user: { id: 'user-1' },
permission: 'Read',
type: 'asset',
id: 'asset-1',
})
expect(prisma.workflowTask.create).toHaveBeenCalledWith({
data: {
assetId: 'asset-1',
projectId: 'project-1',
type: 'transcode_pdf_pages',
status: 'pending',
payload: {
projectId: 'project-1',
pdfPages: {
start: 1,
end: 2,
commentTimestamp: 2,
annotations: [
{
type: 'box',
color: '#ff0000',
points: [
[0, 0],
[1, 1],
],
},
],
},
},
},
})
expect(result.content.length).toBe(2)
expect(result.content[0].type).toBe('image')
expect((result.content[0] as unknown as { data: string }).data).toBe(
Buffer.from('fake-bytes-for-pdf_pages/page_1.webp').toString('base64'),
)
expect(result.details.sourceKeys).toEqual(['pdf_pages/page_1.webp', 'pdf_pages/page_2.webp'])
})
it('should trigger pdfPages transcode workflow with annotations from AgentSessionEntry (1-on-1 chat mode)', async () => {
vi.mocked(prisma.user.findUnique).mockResolvedValue({ id: 'user-1' } as User)
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'asset-1',
projectId: 'project-1',
media: { proxyType: 'pdf' },
} as unknown as Asset)
vi.mocked(prisma.assetComment.findUnique).mockResolvedValue(null)
vi.mocked(prisma.agentSessionEntry.findUnique).mockResolvedValue({
id: 'entry-1',
assetId: 'asset-1',
data: {
details: {
position: { type: 'page', page: 3 },
annotations: [{ type: 'highlight', page: 3 }],
},
},
} as unknown as AgentSessionEntry)
vi.mocked(prisma.workflowTask.create).mockResolvedValue({
id: 'task-1',
} as unknown as WorkflowTask)
vi.mocked(workflowService.executeWait).mockResolvedValue({
id: 'task-1',
status: WorkflowTaskStatus.completed,
output: {
pages: [{ key: 'pdf_pages/page_3.webp', page: 3 }],
},
} as unknown as WorkflowTask)
vi.mocked(s3Service.getObject).mockImplementation(async (_bucket: string, key: string) => {
return {
buffer: Buffer.from(`fake-bytes-for-${key}`),
contentType: 'image/webp',
} as unknown as { buffer: Buffer; contentType: string }
})
const tool = createReadPdfPagesTool('user-1')
await tool.execute('call-2', {
assetId: 'asset-1',
start: 1,
end: 3,
annotationId: 'entry-1',
})
expect(prisma.workflowTask.create).toHaveBeenCalledWith({
data: {
assetId: 'asset-1',
projectId: 'project-1',
type: 'transcode_pdf_pages',
status: 'pending',
payload: {
projectId: 'project-1',
pdfPages: {
start: 1,
end: 3,
commentTimestamp: null,
annotations: [{ type: 'highlight', page: 3 }],
},
},
},
})
})
it('should throw an error when annotationId belongs to a different asset', async () => {
vi.mocked(prisma.user.findUnique).mockResolvedValue({ id: 'user-1' } as User)
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'asset-1',
projectId: 'project-1',
media: { proxyType: 'pdf' },
} as unknown as Asset)
vi.mocked(prisma.assetComment.findUnique).mockResolvedValue({
id: 'comment-on-other-asset',
assetId: 'asset-pdf-2',
annotation: [{ type: 'highlight' }],
} as unknown as AssetComment)
const tool = createReadPdfPagesTool('user-1')
await expect(
tool.execute('call-mismatch', {
assetId: 'asset-1',
start: 1,
end: 3,
annotationId: 'comment-on-other-asset',
}),
).rejects.toThrow(
'Annotation "comment-on-other-asset" belongs to asset "asset-pdf-2", not target asset "asset-1".',
)
})
it('should throw error if start page is less than 1', async () => {
const tool = createReadPdfPagesTool('user-1')
await expect(tool.execute('call-1', { assetId: 'asset-1', start: 0, end: 5 })).rejects.toThrow(
'Invalid page range: start page (0) must be at least 1.',
)
})
it('should throw error if start page is greater than end page', async () => {
const tool = createReadPdfPagesTool('user-1')
await expect(tool.execute('call-1', { assetId: 'asset-1', start: 6, end: 5 })).rejects.toThrow(
'Invalid page range: start page (6) must be less than or equal to end page (5).',
)
})
it('should throw error if requested page range exceeds maximum limit of 20', async () => {
const tool = createReadPdfPagesTool('user-1')
await expect(tool.execute('call-1', { assetId: 'asset-1', start: 1, end: 22 })).rejects.toThrow(
'Page range (22 pages requested) exceeds the maximum limit of 20 pages per request.',
)
})
it('should throw error if asset is not found', async () => {
vi.mocked(prisma.user.findUnique).mockResolvedValue({ id: 'user-1' } as User)
vi.mocked(prisma.asset.findUnique).mockResolvedValue(null)
const tool = createReadPdfPagesTool('user-1')
await expect(
tool.execute('call-1', { assetId: 'non-existent', start: 1, end: 2 }),
).rejects.toThrow('Asset with ID non-existent not found.')
})
it('should throw error if asset is not a PDF proxyType', async () => {
vi.mocked(prisma.user.findUnique).mockResolvedValue({ id: 'user-1' } as User)
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'asset-video',
media: { proxyType: 'video' },
} as unknown as Asset)
const tool = createReadPdfPagesTool('user-1')
await expect(
tool.execute('call-1', { assetId: 'asset-video', start: 1, end: 2 }),
).rejects.toThrow('Asset asset-video is not a PDF or document.')
})
})
-144
View File
@@ -1,144 +0,0 @@
import { Type } from 'typebox'
import { type AgentTool } from '@earendil-works/pi-agent-core'
import { type ImageContent } from '@earendil-works/pi-ai'
import { prisma, WorkflowTaskType, WorkflowTaskStatus, type User } from '@shumai/db'
import { s3Service } from '@shumai/core/src/s3/s3'
import { workflowService } from '@shumai/workflow-core'
import { authzService, Permission, ResourceType } from '@shumai/core/src/authz/authz'
import { resolveAnnotationsById } from './annotation-resolver'
const readPdfPagesSchema = Type.Object({
assetId: Type.String({
description: 'Asset ID of the PDF document. This parameter is required.',
}),
start: Type.Number({ description: 'Start page number (1-based index)' }),
end: Type.Number({ description: 'End page number (1-based index)' }),
annotationId: Type.Optional(
Type.String({
description:
'The ID of the comment or message entry from <annotation id="..." /> whose visual markup should be overlaid on the rendered PDF pages.',
}),
),
})
export function createReadPdfPagesTool(userId: string): AgentTool<typeof readPdfPagesSchema> {
return {
name: 'read_pdf_pages',
label: 'Read PDF Pages',
description:
'Converts specific page range of a PDF document into 1080p WebP images for visual analysis. Maximum 20 pages allowed per call.',
parameters: readPdfPagesSchema,
execute: async (_toolCallId, params) => {
if (params.start < 1) {
throw new Error(`Invalid page range: start page (${params.start}) must be at least 1.`)
}
if (params.start > params.end) {
throw new Error(
`Invalid page range: start page (${params.start}) must be less than or equal to end page (${params.end}).`,
)
}
const MAX_PAGE_RANGE = 20
const requestedPages = params.end - params.start + 1
if (requestedPages > MAX_PAGE_RANGE) {
throw new Error(
`Page range (${requestedPages} pages requested) exceeds the maximum limit of ${MAX_PAGE_RANGE} pages per request.`,
)
}
const targetAssetId = params.assetId
if (!userId) {
throw new Error('User ID is required for authorization.')
}
await authzService.hasPermission({
user: { id: userId } as User,
permission: Permission.Read,
type: ResourceType.Asset,
id: targetAssetId,
})
let asset = await prisma.asset.findUnique({
where: { id: targetAssetId },
})
if (!asset) {
throw new Error(`Asset with ID ${targetAssetId} not found.`)
}
if (asset.type === 'version_stack') {
const latestVersion = await prisma.asset.findFirst({
where: { parentId: asset.id, isDeleted: false },
orderBy: { sortIndex: 'asc' },
})
if (latestVersion) {
asset = latestVersion
}
}
const proxyType = (asset.media as PrismaJson.MediaInfo | null)?.proxyType
if (proxyType !== 'pdf') {
throw new Error(`Asset ${targetAssetId} is not a PDF or document.`)
}
// Resolve annotations dynamically by annotationId
const { annotations, timestamp: commentTimestamp } = await resolveAnnotationsById(
targetAssetId,
params.annotationId,
)
// Trigger transcode workflow to render PDF pages
const task = await prisma.workflowTask.create({
data: {
assetId: targetAssetId,
projectId: asset.projectId || 'none',
type: WorkflowTaskType.transcode_pdf_pages,
status: WorkflowTaskStatus.pending,
payload: {
projectId: asset.projectId || 'none',
pdfPages: {
start: params.start,
end: params.end,
commentTimestamp,
annotations,
},
},
},
})
const completedTask = await workflowService.executeWait(task)
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const output = completedTask.output as any
const pages = output?.pages as Array<{ key: string; page: number }> | undefined
if (!pages || pages.length === 0) {
throw new Error('No PDF page images were generated.')
}
const content: ImageContent[] = []
const sourceKeys: string[] = []
const bucket = process.env.S3_BUCKET || 'shumai'
for (const pageItem of pages) {
const { buffer, contentType } = await s3Service.getObject(bucket, pageItem.key)
const mimeType =
contentType && contentType !== 'application/octet-stream' ? contentType : 'image/webp'
content.push({
type: 'image',
data: buffer.toString('base64'),
mimeType,
})
sourceKeys.push(pageItem.key)
}
return {
content,
details: {
sourceKeys,
},
}
},
}
}
-320
View File
@@ -1,320 +0,0 @@
import { describe, it, expect, vi, beforeEach } from 'vitest'
import { createScreenshotTool } from './screenshot'
import {
prisma,
WorkflowTaskStatus,
type Asset,
type AssetComment,
type AgentSessionEntry,
type User,
type WorkflowTask,
} from '@shumai/db'
import { s3Service } from '@shumai/core/src/s3/s3'
import { workflowService } from '@shumai/workflow-core'
import { authzService } from '@shumai/core/src/authz/authz'
vi.mock('@shumai/db', async (importOriginal) => {
const actual = await importOriginal<typeof import('@shumai/db')>()
return {
...actual,
prisma: {
user: {
findUnique: vi.fn(),
},
asset: {
findUnique: vi.fn(),
},
assetComment: {
findUnique: vi.fn(),
},
agentSessionEntry: {
findUnique: vi.fn(),
},
workflowTask: {
create: vi.fn(),
},
},
}
})
vi.mock('@shumai/core/src/s3/s3', () => ({
s3Service: {
getObject: vi.fn(),
},
}))
vi.mock('@shumai/workflow-core', () => ({
workflowService: {
executeWait: vi.fn(),
},
}))
vi.mock('@shumai/core/src/authz/authz', () => ({
authzService: {
hasPermission: vi.fn(),
},
Permission: { Read: 'Read' },
ResourceType: { Asset: 'asset' },
}))
describe('screenshotTool', () => {
beforeEach(() => {
vi.clearAllMocks()
})
it('should trigger screenshot transcode workflow with annotations when annotationId matches AssetComment', async () => {
vi.mocked(prisma.user.findUnique).mockResolvedValue({ id: 'user-1' } as User)
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'asset-1',
projectId: 'project-1',
} as unknown as Asset)
vi.mocked(prisma.assetComment.findUnique).mockResolvedValue({
id: 'comment-1',
assetId: 'asset-1',
second: 5.0,
annotation: [
{
type: 'line',
color: '#00ff00',
points: [
[0, 0],
[1, 1],
],
},
],
} as unknown as AssetComment)
vi.mocked(prisma.workflowTask.create).mockResolvedValue({
id: 'task-1',
} as unknown as WorkflowTask)
vi.mocked(workflowService.executeWait).mockResolvedValue({
id: 'task-1',
status: WorkflowTaskStatus.completed,
output: {
screenshots: [
{ key: 'screenshots/shot1.webp', timestamp: 0.0 },
{ key: 'screenshots/shot2.webp', timestamp: 5.0 },
],
},
} as unknown as WorkflowTask)
vi.mocked(s3Service.getObject).mockImplementation(async (_bucket: string, key: string) => {
return {
buffer: Buffer.from(`fake-bytes-for-${key}`),
contentType: 'image/webp',
} as unknown as { buffer: Buffer; contentType: string }
})
const tool = createScreenshotTool('user-1')
const result = await tool.execute('call-1', {
assetId: 'asset-1',
start: 0,
end: 10,
count: 2,
annotationId: 'comment-1',
})
expect(authzService.hasPermission).toHaveBeenCalledWith({
user: { id: 'user-1' },
permission: 'Read',
type: 'asset',
id: 'asset-1',
})
expect(prisma.workflowTask.create).toHaveBeenCalledWith({
data: {
assetId: 'asset-1',
projectId: 'project-1',
type: 'transcode_screenshot',
status: 'pending',
payload: {
projectId: 'project-1',
screenshot: {
start: 0,
end: 10,
count: 2,
commentTimestamp: 5.0,
annotations: [
{
type: 'line',
color: '#00ff00',
points: [
[0, 0],
[1, 1],
],
},
],
},
},
},
})
expect(result.content.length).toBe(2)
expect(result.content[0].type).toBe('image')
expect((result.content[0] as unknown as { data: string }).data).toBe(
Buffer.from('fake-bytes-for-screenshots/shot1.webp').toString('base64'),
)
expect((result.content[1] as unknown as { data: string }).data).toBe(
Buffer.from('fake-bytes-for-screenshots/shot2.webp').toString('base64'),
)
expect(result.details.sourceKeys).toEqual(['screenshots/shot1.webp', 'screenshots/shot2.webp'])
})
it('should trigger screenshot transcode workflow with annotations when annotationId matches AgentSessionEntry (1-on-1 chat)', async () => {
vi.mocked(prisma.user.findUnique).mockResolvedValue({ id: 'user-1' } as User)
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'asset-1',
projectId: 'project-1',
} as unknown as Asset)
vi.mocked(prisma.assetComment.findUnique).mockResolvedValue(null)
vi.mocked(prisma.agentSessionEntry.findUnique).mockResolvedValue({
id: 'entry-1',
assetId: 'asset-1',
data: {
details: {
position: { type: 'time', seconds: 7.25 },
annotations: [{ type: 'rectangle', x: 10, y: 20, width: 30, height: 40 }],
},
},
} as unknown as AgentSessionEntry)
vi.mocked(prisma.workflowTask.create).mockResolvedValue({
id: 'task-1',
} as unknown as WorkflowTask)
vi.mocked(workflowService.executeWait).mockResolvedValue({
id: 'task-1',
status: WorkflowTaskStatus.completed,
output: {
screenshots: [{ key: 'screenshots/shot-chat.webp', timestamp: 7.25 }],
},
} as unknown as WorkflowTask)
vi.mocked(s3Service.getObject).mockImplementation(async (_bucket: string, key: string) => {
return {
buffer: Buffer.from(`fake-bytes-for-${key}`),
contentType: 'image/webp',
} as unknown as { buffer: Buffer; contentType: string }
})
const tool = createScreenshotTool('user-1')
await tool.execute('call-2', {
assetId: 'asset-1',
start: 5,
end: 10,
count: 1,
annotationId: 'entry-1',
})
expect(prisma.workflowTask.create).toHaveBeenCalledWith({
data: {
assetId: 'asset-1',
projectId: 'project-1',
type: 'transcode_screenshot',
status: 'pending',
payload: {
projectId: 'project-1',
screenshot: {
start: 5,
end: 10,
count: 1,
commentTimestamp: 7.25,
annotations: [{ type: 'rectangle', x: 10, y: 20, width: 30, height: 40 }],
},
},
},
})
})
it('should throw an error when annotationId belongs to a different asset', async () => {
vi.mocked(prisma.user.findUnique).mockResolvedValue({ id: 'user-1' } as User)
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'asset-1',
projectId: 'project-1',
} as unknown as Asset)
vi.mocked(prisma.assetComment.findUnique).mockResolvedValue({
id: 'comment-on-other-asset',
assetId: 'asset-2',
annotation: [{ type: 'arrow' }],
} as unknown as AssetComment)
const tool = createScreenshotTool('user-1')
await expect(
tool.execute('call-mismatch', {
assetId: 'asset-1',
start: 0,
end: 5,
count: 1,
annotationId: 'comment-on-other-asset',
}),
).rejects.toThrow(
'Annotation "comment-on-other-asset" belongs to asset "asset-2", not target asset "asset-1".',
)
})
it('should take unannotated screenshots when annotationId is omitted', async () => {
vi.mocked(prisma.user.findUnique).mockResolvedValue({ id: 'user-1' } as User)
vi.mocked(authzService.hasPermission).mockResolvedValue()
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
id: 'asset-1',
projectId: 'project-1',
} as unknown as Asset)
vi.mocked(prisma.workflowTask.create).mockResolvedValue({
id: 'task-1',
} as unknown as WorkflowTask)
vi.mocked(workflowService.executeWait).mockResolvedValue({
id: 'task-1',
status: WorkflowTaskStatus.completed,
output: {
screenshots: [{ key: 'screenshots/shot-clean.webp', timestamp: 0.0 }],
},
} as unknown as WorkflowTask)
vi.mocked(s3Service.getObject).mockImplementation(async (_bucket: string, key: string) => {
return {
buffer: Buffer.from(`fake-bytes-for-${key}`),
contentType: 'image/webp',
} as unknown as { buffer: Buffer; contentType: string }
})
const tool = createScreenshotTool('user-1')
await tool.execute('call-3', {
assetId: 'asset-1',
start: 0,
end: 5,
count: 1,
})
expect(prisma.workflowTask.create).toHaveBeenCalledWith({
data: {
assetId: 'asset-1',
projectId: 'project-1',
type: 'transcode_screenshot',
status: 'pending',
payload: {
projectId: 'project-1',
screenshot: {
start: 0,
end: 5,
count: 1,
commentTimestamp: null,
annotations: null,
},
},
},
})
})
})
-115
View File
@@ -1,115 +0,0 @@
import { Type } from 'typebox'
import { type AgentTool } from '@earendil-works/pi-agent-core'
import { type ImageContent } from '@earendil-works/pi-ai'
import { prisma, WorkflowTaskType, WorkflowTaskStatus, type User } from '@shumai/db'
import { s3Service } from '@shumai/core/src/s3/s3'
import { workflowService } from '@shumai/workflow-core'
import { authzService, Permission, ResourceType } from '@shumai/core/src/authz/authz'
import { resolveAnnotationsById } from './annotation-resolver'
const screenshotSchema = Type.Object({
assetId: Type.String({
description: 'The asset ID of the video asset. This parameter is required.',
}),
start: Type.Number({ description: 'Start time in seconds' }),
end: Type.Number({ description: 'End time in seconds' }),
count: Type.Number({ description: 'Number of screenshots to take' }),
annotationId: Type.Optional(
Type.String({
description:
'The ID of the comment or message entry from <annotation id="..." /> whose visual markup should be overlaid on the screenshot.',
}),
),
})
export function createScreenshotTool(userId: string): AgentTool<typeof screenshotSchema> {
return {
name: 'screenshot',
label: 'Take Video Screenshots',
description:
'Takes screenshots for a video at specified interval. It returns the images to the agent context.',
parameters: screenshotSchema,
execute: async (_toolCallId, params) => {
const assetId = params.assetId
if (!userId) {
throw new Error('User ID is required for authorization.')
}
await authzService.hasPermission({
user: { id: userId } as User,
permission: Permission.Read,
type: ResourceType.Asset,
id: assetId,
})
const asset = await prisma.asset.findUnique({
where: { id: assetId },
})
if (!asset) {
throw new Error(`Asset with ID ${assetId} not found.`)
}
// Resolve annotations dynamically by annotationId
const { annotations, timestamp: commentTimestamp } = await resolveAnnotationsById(
assetId,
params.annotationId,
)
// Trigger transcode workflow to take screenshots
const task = await prisma.workflowTask.create({
data: {
assetId,
projectId: asset.projectId || 'none',
type: WorkflowTaskType.transcode_screenshot,
status: WorkflowTaskStatus.pending,
payload: {
projectId: asset.projectId || 'none',
screenshot: {
start: params.start,
end: params.end,
count: params.count,
commentTimestamp,
annotations,
},
},
},
})
const completedTask = await workflowService.executeWait(task)
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const output = completedTask.output as any
const screenshots = output?.screenshots as
| Array<{ key: string; timestamp: number }>
| undefined
if (!screenshots || screenshots.length === 0) {
throw new Error('No screenshots were generated.')
}
const content: ImageContent[] = []
const sourceKeys: string[] = []
const bucket = process.env.S3_BUCKET || 'shumai'
for (const shot of screenshots) {
const { buffer, contentType } = await s3Service.getObject(bucket, shot.key)
const mimeType =
contentType && contentType !== 'application/octet-stream' ? contentType : 'image/webp'
content.push({
type: 'image',
data: buffer.toString('base64'),
mimeType,
})
sourceKeys.push(shot.key)
}
return {
content,
details: {
sourceKeys,
},
}
},
}
}
+10 -10
View File
@@ -21,12 +21,12 @@ describe('QuotaRule Wildcard Matchers', () => {
it('correctly matches agent tool name wildcards', () => {
const starTool = wildcardToRegex('*')
expect(starTool.test('analyze_image')).toBe(true)
expect(starTool.test('screenshot')).toBe(true)
expect(starTool.test('read_asset')).toBe(true)
expect(starTool.test('create_file')).toBe(true)
const exactTool = wildcardToRegex('analyze_image')
expect(exactTool.test('analyze_image')).toBe(true)
expect(exactTool.test('screenshot')).toBe(false)
const exactTool = wildcardToRegex('read_asset')
expect(exactTool.test('read_asset')).toBe(true)
expect(exactTool.test('create_file')).toBe(false)
})
})
@@ -405,29 +405,29 @@ describe('QuotaService', () => {
await quotaService.createRule(team.id, {
scopeMode: 'all_members',
resource: 'agent_tool_call_count',
resourceData: { name: 'analyze_image' },
resourceData: { name: 'read_asset' },
limit: 1,
period: '1hour',
enabled: true,
})
// First call to analyze_image
// First call to read_asset
await quotaService.consumeQuota(
{
teamId: team.id,
resource: 'agent_tool_call_count',
resourceData: { name: 'analyze_image' },
resourceData: { name: 'read_asset' },
},
1,
)
// 2nd call to analyze_image is blocked
// 2nd call to read_asset is blocked
await expect(
quotaService.checkQuota(
{
teamId: team.id,
resource: 'agent_tool_call_count',
resourceData: { name: 'analyze_image' },
resourceData: { name: 'read_asset' },
},
1,
),
@@ -70,19 +70,9 @@ const BUILTIN_TOOLS = [
description: () => m.agent_tool_read_skill_desc(),
},
{
id: 'analyze_image',
name: () => m.agent_tool_analyze_image_name(),
description: () => m.agent_tool_analyze_image_desc(),
},
{
id: 'screenshot',
name: () => m.agent_tool_screenshot_name(),
description: () => m.agent_tool_screenshot_desc(),
},
{
id: 'read_pdf_pages',
name: () => m.agent_tool_read_pdf_pages_name(),
description: () => m.agent_tool_read_pdf_pages_desc(),
id: 'read_asset',
name: () => m.agent_tool_read_asset_name(),
description: () => m.agent_tool_read_asset_desc(),
},
{
id: 'list_assets',
@@ -69,9 +69,7 @@ interface QuotaRuleDialogProps {
}
const STANDARD_TOOLS = [
{ value: 'analyze_image', label: () => m.agent_tool_analyze_image_name() },
{ value: 'screenshot', label: () => m.agent_tool_screenshot_name() },
{ value: 'read_pdf_pages', label: () => m.agent_tool_read_pdf_pages_name() },
{ value: 'read_asset', label: () => m.agent_tool_read_asset_name() },
{ value: 'list_assets', label: () => m.agent_tool_list_assets_name() },
{ value: 'create_folder', label: () => m.agent_tool_create_folder_name() },
{ value: 'create_file', label: () => m.agent_tool_create_file_name() },
@@ -129,7 +127,7 @@ export const QuotaRuleDialog: React.FC<QuotaRuleDialogProps> = ({
const [resource, setResource] = useState<QuotaResourceTypeEnum>('agent_total_tokens')
const [mcpServerId, setMcpServerId] = useState<string>('')
const [bashMatch, setBashMatch] = useState<string>('*')
const [toolName, setToolName] = useState<string>('analyze_image')
const [toolName, setToolName] = useState<string>('read_asset')
const [limit, setLimit] = useState<number>(100000)
const [period, setPeriod] = useState<QuotaPeriodEnum>('1day')
const [enabled, setEnabled] = useState<boolean>(true)
@@ -177,7 +175,7 @@ export const QuotaRuleDialog: React.FC<QuotaRuleDialogProps> = ({
? resData.name
: typeof resData.toolName === 'string'
? resData.toolName
: 'analyze_image',
: 'read_asset',
)
setLimit(rule.limit)
setPeriod(formatQuotaPeriod(rule.period) as QuotaPeriodEnum)
@@ -189,7 +187,7 @@ export const QuotaRuleDialog: React.FC<QuotaRuleDialogProps> = ({
setResource('agent_total_tokens')
setMcpServerId('')
setBashMatch('*')
setToolName('analyze_image')
setToolName('read_asset')
setLimit(RESOURCE_CONFIG.agent_total_tokens.defaultLimit)
setPeriod('1day')
setEnabled(true)
@@ -213,7 +211,7 @@ export const QuotaRuleDialog: React.FC<QuotaRuleDialogProps> = ({
} else if (resource === 'agent_bash_call_count') {
resourceData.match = bashMatch.trim() || '*'
} else if (resource === 'agent_tool_call_count') {
resourceData.name = toolName || 'analyze_image'
resourceData.name = toolName || 'read_asset'
}
const res = await client.api.teams[':teamId'].quotas.$post({
@@ -261,7 +259,7 @@ export const QuotaRuleDialog: React.FC<QuotaRuleDialogProps> = ({
} else if (resource === 'agent_bash_call_count') {
resourceData.match = bashMatch.trim() || '*'
} else if (resource === 'agent_tool_call_count') {
resourceData.name = toolName || 'analyze_image'
resourceData.name = toolName || 'read_asset'
}
const res = await client.api.teams[':teamId'].quotas[':id'].$put({
+2 -6
View File
@@ -477,12 +477,8 @@
"agent_tool_bash_desc": "Allows the agent to run terminal/shell commands in a highly isolated environment.",
"agent_tool_read_skill_name": "Read Skills",
"agent_tool_read_skill_desc": "Allows the agent to load and read skill definitions to perform specialized actions.",
"agent_tool_analyze_image_name": "Analyze Image",
"agent_tool_analyze_image_desc": "Allows the agent to view and analyze the visual contents of an image file.",
"agent_tool_screenshot_name": "Video Frame Extraction",
"agent_tool_screenshot_desc": "Allows the agent to extract still image frames from video files for analysis.",
"agent_tool_read_pdf_pages_name": "Read PDF Pages",
"agent_tool_read_pdf_pages_desc": "Allows the agent to convert and view specific pages of a PDF document as images.",
"agent_tool_read_asset_name": "Read Asset",
"agent_tool_read_asset_desc": "Allows the agent to view and inspect images, video frames, and document pages.",
"agent_tool_list_assets_name": "Browse Shumai",
"agent_tool_list_assets_desc": "Allows the agent to list folders and browse files in Shumai.",
"agent_tool_create_folder_name": "Create Folder",
+2 -6
View File
@@ -477,12 +477,8 @@
"agent_tool_bash_desc": "允许智能体在高度隔离的环境中运行终端/命令行命令。",
"agent_tool_read_skill_name": "读取技能",
"agent_tool_read_skill_desc": "允许智能体加载和读取技能定义以执行特定操作。",
"agent_tool_analyze_image_name": "分析图片",
"agent_tool_analyze_image_desc": "允许智能体查看和分析图片文件的视觉内容。",
"agent_tool_screenshot_name": "提取视频帧",
"agent_tool_screenshot_desc": "允许智能体从视频文件中提取静态画面进行分析。",
"agent_tool_read_pdf_pages_name": "读取 PDF 页面",
"agent_tool_read_pdf_pages_desc": "允许智能体将 PDF 文件的指定页面转换为图片并查看。",
"agent_tool_read_asset_name": "读取资源",
"agent_tool_read_asset_desc": "允许智能体查看和分析图片、视频画面和文档页面。",
"agent_tool_list_assets_name": "浏览 Shumai",
"agent_tool_list_assets_desc": "允许智能体列出文件夹并浏览 Shumai 中的文件。",
"agent_tool_create_folder_name": "创建文件夹",
+2 -6
View File
@@ -458,12 +458,8 @@ export * from "./agent_tool_bash_name.js";
export * from "./agent_tool_bash_desc.js";
export * from "./agent_tool_read_skill_name.js";
export * from "./agent_tool_read_skill_desc.js";
export * from "./agent_tool_analyze_image_name.js";
export * from "./agent_tool_analyze_image_desc.js";
export * from "./agent_tool_screenshot_name.js";
export * from "./agent_tool_screenshot_desc.js";
export * from "./agent_tool_read_pdf_pages_name.js";
export * from "./agent_tool_read_pdf_pages_desc.js";
export * from "./agent_tool_read_asset_name.js";
export * from "./agent_tool_read_asset_desc.js";
export * from "./agent_tool_list_assets_name.js";
export * from "./agent_tool_list_assets_desc.js";
export * from "./agent_tool_create_folder_name.js";
+2 -6
View File
@@ -460,12 +460,8 @@ export * from './agent_tool_bash_name.js'
export * from './agent_tool_bash_desc.js'
export * from './agent_tool_read_skill_name.js'
export * from './agent_tool_read_skill_desc.js'
export * from './agent_tool_analyze_image_name.js'
export * from './agent_tool_analyze_image_desc.js'
export * from './agent_tool_screenshot_name.js'
export * from './agent_tool_screenshot_desc.js'
export * from './agent_tool_read_pdf_pages_name.js'
export * from './agent_tool_read_pdf_pages_desc.js'
export * from './agent_tool_read_asset_name.js'
export * from './agent_tool_read_asset_desc.js'
export * from './agent_tool_list_assets_name.js'
export * from './agent_tool_list_assets_desc.js'
export * from './agent_tool_create_folder_name.js'
-16
View File
@@ -1,16 +0,0 @@
/**
* | output |
* | --- |
* | "Allows the agent to view and analyze the visual contents of an image file." |
*
* @param {Agent_Tool_Analyze_Image_DescInputs} inputs
* @param {{ locale?: "en" | "zh" }} options
* @returns {LocalizedString}
*/
export const agent_tool_analyze_image_desc: ((inputs?: Agent_Tool_Analyze_Image_DescInputs, options?: {
locale?: "en" | "zh";
}) => LocalizedString) & import("../runtime.js").MessageMetadata<Agent_Tool_Analyze_Image_DescInputs, {
locale?: "en" | "zh";
}, {}>;
export type LocalizedString = import("../runtime.js").LocalizedString;
export type Agent_Tool_Analyze_Image_DescInputs = {};
@@ -1,29 +0,0 @@
/* eslint-disable */
import { getLocale, experimentalStaticLocale } from '../runtime.js';
/** @typedef {import('../runtime.js').LocalizedString} LocalizedString */
/** @typedef {{}} Agent_Tool_Analyze_Image_DescInputs */
const en_agent_tool_analyze_image_desc = /** @type {(inputs: Agent_Tool_Analyze_Image_DescInputs) => LocalizedString} */ () => {
return /** @type {LocalizedString} */ (`Allows the agent to view and analyze the visual contents of an image file.`)
};
const zh_agent_tool_analyze_image_desc = /** @type {(inputs: Agent_Tool_Analyze_Image_DescInputs) => LocalizedString} */ () => {
return /** @type {LocalizedString} */ (`允许智能体查看和分析图片文件的视觉内容。`)
};
/**
* | output |
* | --- |
* | "Allows the agent to view and analyze the visual contents of an image file." |
*
* @param {Agent_Tool_Analyze_Image_DescInputs} inputs
* @param {{ locale?: "en" | "zh" }} options
* @returns {LocalizedString}
*/
export const agent_tool_analyze_image_desc = /** @type {((inputs?: Agent_Tool_Analyze_Image_DescInputs, options?: { locale?: "en" | "zh" }) => LocalizedString) & import('../runtime.js').MessageMetadata<Agent_Tool_Analyze_Image_DescInputs, { locale?: "en" | "zh" }, {}>} */ ((inputs = {}, options = {}) => {
const locale = experimentalStaticLocale ?? options.locale ?? getLocale()
if (locale === "en") return en_agent_tool_analyze_image_desc(inputs)
return zh_agent_tool_analyze_image_desc(inputs)
});
-16
View File
@@ -1,16 +0,0 @@
/**
* | output |
* | --- |
* | "Analyze Image" |
*
* @param {Agent_Tool_Analyze_Image_NameInputs} inputs
* @param {{ locale?: "en" | "zh" }} options
* @returns {LocalizedString}
*/
export const agent_tool_analyze_image_name: ((inputs?: Agent_Tool_Analyze_Image_NameInputs, options?: {
locale?: "en" | "zh";
}) => LocalizedString) & import("../runtime.js").MessageMetadata<Agent_Tool_Analyze_Image_NameInputs, {
locale?: "en" | "zh";
}, {}>;
export type LocalizedString = import("../runtime.js").LocalizedString;
export type Agent_Tool_Analyze_Image_NameInputs = {};
@@ -1,29 +0,0 @@
/* eslint-disable */
import { getLocale, experimentalStaticLocale } from '../runtime.js';
/** @typedef {import('../runtime.js').LocalizedString} LocalizedString */
/** @typedef {{}} Agent_Tool_Analyze_Image_NameInputs */
const en_agent_tool_analyze_image_name = /** @type {(inputs: Agent_Tool_Analyze_Image_NameInputs) => LocalizedString} */ () => {
return /** @type {LocalizedString} */ (`Analyze Image`)
};
const zh_agent_tool_analyze_image_name = /** @type {(inputs: Agent_Tool_Analyze_Image_NameInputs) => LocalizedString} */ () => {
return /** @type {LocalizedString} */ (`分析图片`)
};
/**
* | output |
* | --- |
* | "Analyze Image" |
*
* @param {Agent_Tool_Analyze_Image_NameInputs} inputs
* @param {{ locale?: "en" | "zh" }} options
* @returns {LocalizedString}
*/
export const agent_tool_analyze_image_name = /** @type {((inputs?: Agent_Tool_Analyze_Image_NameInputs, options?: { locale?: "en" | "zh" }) => LocalizedString) & import('../runtime.js').MessageMetadata<Agent_Tool_Analyze_Image_NameInputs, { locale?: "en" | "zh" }, {}>} */ ((inputs = {}, options = {}) => {
const locale = experimentalStaticLocale ?? options.locale ?? getLocale()
if (locale === "en") return en_agent_tool_analyze_image_name(inputs)
return zh_agent_tool_analyze_image_name(inputs)
});
+16
View File
@@ -0,0 +1,16 @@
/**
* | output |
* | --- |
* | "Allows the agent to view and inspect images, video frames, and document pages." |
*
* @param {Agent_Tool_Read_Asset_DescInputs} inputs
* @param {{ locale?: "en" | "zh" }} options
* @returns {LocalizedString}
*/
export const agent_tool_read_asset_desc: ((inputs?: Agent_Tool_Read_Asset_DescInputs, options?: {
locale?: "en" | "zh";
}) => LocalizedString) & import("../runtime.js").MessageMetadata<Agent_Tool_Read_Asset_DescInputs, {
locale?: "en" | "zh";
}, {}>;
export type LocalizedString = import("../runtime.js").LocalizedString;
export type Agent_Tool_Read_Asset_DescInputs = {};
@@ -0,0 +1,29 @@
/* eslint-disable */
import { getLocale, experimentalStaticLocale } from '../runtime.js';
/** @typedef {import('../runtime.js').LocalizedString} LocalizedString */
/** @typedef {{}} Agent_Tool_Read_Asset_DescInputs */
const en_agent_tool_read_asset_desc = /** @type {(inputs: Agent_Tool_Read_Asset_DescInputs) => LocalizedString} */ () => {
return /** @type {LocalizedString} */ (`Allows the agent to view and inspect images, video frames, and document pages.`)
};
const zh_agent_tool_read_asset_desc = /** @type {(inputs: Agent_Tool_Read_Asset_DescInputs) => LocalizedString} */ () => {
return /** @type {LocalizedString} */ (`允许智能体查看和分析图片、视频画面和文档页面。`)
};
/**
* | output |
* | --- |
* | "Allows the agent to view and inspect images, video frames, and document pages." |
*
* @param {Agent_Tool_Read_Asset_DescInputs} inputs
* @param {{ locale?: "en" | "zh" }} options
* @returns {LocalizedString}
*/
export const agent_tool_read_asset_desc = /** @type {((inputs?: Agent_Tool_Read_Asset_DescInputs, options?: { locale?: "en" | "zh" }) => LocalizedString) & import('../runtime.js').MessageMetadata<Agent_Tool_Read_Asset_DescInputs, { locale?: "en" | "zh" }, {}>} */ ((inputs = {}, options = {}) => {
const locale = experimentalStaticLocale ?? options.locale ?? getLocale()
if (locale === "en") return en_agent_tool_read_asset_desc(inputs)
return zh_agent_tool_read_asset_desc(inputs)
});
@@ -1,16 +1,16 @@
/**
* | output |
* | --- |
* | "Video Frame Extraction" |
* | "Read Asset" |
*
* @param {Agent_Tool_Screenshot_NameInputs} inputs
* @param {Agent_Tool_Read_Asset_NameInputs} inputs
* @param {{ locale?: "en" | "zh" }} options
* @returns {LocalizedString}
*/
export const agent_tool_screenshot_name: ((inputs?: Agent_Tool_Screenshot_NameInputs, options?: {
export const agent_tool_read_asset_name: ((inputs?: Agent_Tool_Read_Asset_NameInputs, options?: {
locale?: "en" | "zh";
}) => LocalizedString) & import("../runtime.js").MessageMetadata<Agent_Tool_Screenshot_NameInputs, {
}) => LocalizedString) & import("../runtime.js").MessageMetadata<Agent_Tool_Read_Asset_NameInputs, {
locale?: "en" | "zh";
}, {}>;
export type LocalizedString = import("../runtime.js").LocalizedString;
export type Agent_Tool_Screenshot_NameInputs = {};
export type Agent_Tool_Read_Asset_NameInputs = {};
@@ -0,0 +1,29 @@
/* eslint-disable */
import { getLocale, experimentalStaticLocale } from '../runtime.js';
/** @typedef {import('../runtime.js').LocalizedString} LocalizedString */
/** @typedef {{}} Agent_Tool_Read_Asset_NameInputs */
const en_agent_tool_read_asset_name = /** @type {(inputs: Agent_Tool_Read_Asset_NameInputs) => LocalizedString} */ () => {
return /** @type {LocalizedString} */ (`Read Asset`)
};
const zh_agent_tool_read_asset_name = /** @type {(inputs: Agent_Tool_Read_Asset_NameInputs) => LocalizedString} */ () => {
return /** @type {LocalizedString} */ (`读取资源`)
};
/**
* | output |
* | --- |
* | "Read Asset" |
*
* @param {Agent_Tool_Read_Asset_NameInputs} inputs
* @param {{ locale?: "en" | "zh" }} options
* @returns {LocalizedString}
*/
export const agent_tool_read_asset_name = /** @type {((inputs?: Agent_Tool_Read_Asset_NameInputs, options?: { locale?: "en" | "zh" }) => LocalizedString) & import('../runtime.js').MessageMetadata<Agent_Tool_Read_Asset_NameInputs, { locale?: "en" | "zh" }, {}>} */ ((inputs = {}, options = {}) => {
const locale = experimentalStaticLocale ?? options.locale ?? getLocale()
if (locale === "en") return en_agent_tool_read_asset_name(inputs)
return zh_agent_tool_read_asset_name(inputs)
});
-16
View File
@@ -1,16 +0,0 @@
/**
* | output |
* | --- |
* | "Allows the agent to convert and view specific pages of a PDF document as images." |
*
* @param {Agent_Tool_Read_Pdf_Pages_DescInputs} inputs
* @param {{ locale?: "en" | "zh" }} options
* @returns {LocalizedString}
*/
export const agent_tool_read_pdf_pages_desc: ((inputs?: Agent_Tool_Read_Pdf_Pages_DescInputs, options?: {
locale?: "en" | "zh";
}) => LocalizedString) & import("../runtime.js").MessageMetadata<Agent_Tool_Read_Pdf_Pages_DescInputs, {
locale?: "en" | "zh";
}, {}>;
export type LocalizedString = import("../runtime.js").LocalizedString;
export type Agent_Tool_Read_Pdf_Pages_DescInputs = {};
@@ -1,29 +0,0 @@
/* eslint-disable */
import { getLocale, experimentalStaticLocale } from '../runtime.js';
/** @typedef {import('../runtime.js').LocalizedString} LocalizedString */
/** @typedef {{}} Agent_Tool_Read_Pdf_Pages_DescInputs */
const en_agent_tool_read_pdf_pages_desc = /** @type {(inputs: Agent_Tool_Read_Pdf_Pages_DescInputs) => LocalizedString} */ () => {
return /** @type {LocalizedString} */ (`Allows the agent to convert and view specific pages of a PDF document as images.`)
};
const zh_agent_tool_read_pdf_pages_desc = /** @type {(inputs: Agent_Tool_Read_Pdf_Pages_DescInputs) => LocalizedString} */ () => {
return /** @type {LocalizedString} */ (`允许智能体将 PDF 文件的指定页面转换为图片并查看。`)
};
/**
* | output |
* | --- |
* | "Allows the agent to convert and view specific pages of a PDF document as images." |
*
* @param {Agent_Tool_Read_Pdf_Pages_DescInputs} inputs
* @param {{ locale?: "en" | "zh" }} options
* @returns {LocalizedString}
*/
export const agent_tool_read_pdf_pages_desc = /** @type {((inputs?: Agent_Tool_Read_Pdf_Pages_DescInputs, options?: { locale?: "en" | "zh" }) => LocalizedString) & import('../runtime.js').MessageMetadata<Agent_Tool_Read_Pdf_Pages_DescInputs, { locale?: "en" | "zh" }, {}>} */ ((inputs = {}, options = {}) => {
const locale = experimentalStaticLocale ?? options.locale ?? getLocale()
if (locale === "en") return en_agent_tool_read_pdf_pages_desc(inputs)
return zh_agent_tool_read_pdf_pages_desc(inputs)
});
-16
View File
@@ -1,16 +0,0 @@
/**
* | output |
* | --- |
* | "Read PDF Pages" |
*
* @param {Agent_Tool_Read_Pdf_Pages_NameInputs} inputs
* @param {{ locale?: "en" | "zh" }} options
* @returns {LocalizedString}
*/
export const agent_tool_read_pdf_pages_name: ((inputs?: Agent_Tool_Read_Pdf_Pages_NameInputs, options?: {
locale?: "en" | "zh";
}) => LocalizedString) & import("../runtime.js").MessageMetadata<Agent_Tool_Read_Pdf_Pages_NameInputs, {
locale?: "en" | "zh";
}, {}>;
export type LocalizedString = import("../runtime.js").LocalizedString;
export type Agent_Tool_Read_Pdf_Pages_NameInputs = {};
@@ -1,29 +0,0 @@
/* eslint-disable */
import { getLocale, experimentalStaticLocale } from '../runtime.js';
/** @typedef {import('../runtime.js').LocalizedString} LocalizedString */
/** @typedef {{}} Agent_Tool_Read_Pdf_Pages_NameInputs */
const en_agent_tool_read_pdf_pages_name = /** @type {(inputs: Agent_Tool_Read_Pdf_Pages_NameInputs) => LocalizedString} */ () => {
return /** @type {LocalizedString} */ (`Read PDF Pages`)
};
const zh_agent_tool_read_pdf_pages_name = /** @type {(inputs: Agent_Tool_Read_Pdf_Pages_NameInputs) => LocalizedString} */ () => {
return /** @type {LocalizedString} */ (`读取 PDF 页面`)
};
/**
* | output |
* | --- |
* | "Read PDF Pages" |
*
* @param {Agent_Tool_Read_Pdf_Pages_NameInputs} inputs
* @param {{ locale?: "en" | "zh" }} options
* @returns {LocalizedString}
*/
export const agent_tool_read_pdf_pages_name = /** @type {((inputs?: Agent_Tool_Read_Pdf_Pages_NameInputs, options?: { locale?: "en" | "zh" }) => LocalizedString) & import('../runtime.js').MessageMetadata<Agent_Tool_Read_Pdf_Pages_NameInputs, { locale?: "en" | "zh" }, {}>} */ ((inputs = {}, options = {}) => {
const locale = experimentalStaticLocale ?? options.locale ?? getLocale()
if (locale === "en") return en_agent_tool_read_pdf_pages_name(inputs)
return zh_agent_tool_read_pdf_pages_name(inputs)
});
-16
View File
@@ -1,16 +0,0 @@
/**
* | output |
* | --- |
* | "Allows the agent to extract still image frames from video files for analysis." |
*
* @param {Agent_Tool_Screenshot_DescInputs} inputs
* @param {{ locale?: "en" | "zh" }} options
* @returns {LocalizedString}
*/
export const agent_tool_screenshot_desc: ((inputs?: Agent_Tool_Screenshot_DescInputs, options?: {
locale?: "en" | "zh";
}) => LocalizedString) & import("../runtime.js").MessageMetadata<Agent_Tool_Screenshot_DescInputs, {
locale?: "en" | "zh";
}, {}>;
export type LocalizedString = import("../runtime.js").LocalizedString;
export type Agent_Tool_Screenshot_DescInputs = {};
@@ -1,29 +0,0 @@
/* eslint-disable */
import { getLocale, experimentalStaticLocale } from '../runtime.js';
/** @typedef {import('../runtime.js').LocalizedString} LocalizedString */
/** @typedef {{}} Agent_Tool_Screenshot_DescInputs */
const en_agent_tool_screenshot_desc = /** @type {(inputs: Agent_Tool_Screenshot_DescInputs) => LocalizedString} */ () => {
return /** @type {LocalizedString} */ (`Allows the agent to extract still image frames from video files for analysis.`)
};
const zh_agent_tool_screenshot_desc = /** @type {(inputs: Agent_Tool_Screenshot_DescInputs) => LocalizedString} */ () => {
return /** @type {LocalizedString} */ (`允许智能体从视频文件中提取静态画面进行分析。`)
};
/**
* | output |
* | --- |
* | "Allows the agent to extract still image frames from video files for analysis." |
*
* @param {Agent_Tool_Screenshot_DescInputs} inputs
* @param {{ locale?: "en" | "zh" }} options
* @returns {LocalizedString}
*/
export const agent_tool_screenshot_desc = /** @type {((inputs?: Agent_Tool_Screenshot_DescInputs, options?: { locale?: "en" | "zh" }) => LocalizedString) & import('../runtime.js').MessageMetadata<Agent_Tool_Screenshot_DescInputs, { locale?: "en" | "zh" }, {}>} */ ((inputs = {}, options = {}) => {
const locale = experimentalStaticLocale ?? options.locale ?? getLocale()
if (locale === "en") return en_agent_tool_screenshot_desc(inputs)
return zh_agent_tool_screenshot_desc(inputs)
});
@@ -1,29 +0,0 @@
/* eslint-disable */
import { getLocale, experimentalStaticLocale } from '../runtime.js';
/** @typedef {import('../runtime.js').LocalizedString} LocalizedString */
/** @typedef {{}} Agent_Tool_Screenshot_NameInputs */
const en_agent_tool_screenshot_name = /** @type {(inputs: Agent_Tool_Screenshot_NameInputs) => LocalizedString} */ () => {
return /** @type {LocalizedString} */ (`Video Frame Extraction`)
};
const zh_agent_tool_screenshot_name = /** @type {(inputs: Agent_Tool_Screenshot_NameInputs) => LocalizedString} */ () => {
return /** @type {LocalizedString} */ (`提取视频帧`)
};
/**
* | output |
* | --- |
* | "Video Frame Extraction" |
*
* @param {Agent_Tool_Screenshot_NameInputs} inputs
* @param {{ locale?: "en" | "zh" }} options
* @returns {LocalizedString}
*/
export const agent_tool_screenshot_name = /** @type {((inputs?: Agent_Tool_Screenshot_NameInputs, options?: { locale?: "en" | "zh" }) => LocalizedString) & import('../runtime.js').MessageMetadata<Agent_Tool_Screenshot_NameInputs, { locale?: "en" | "zh" }, {}>} */ ((inputs = {}, options = {}) => {
const locale = experimentalStaticLocale ?? options.locale ?? getLocale()
if (locale === "en") return en_agent_tool_screenshot_name(inputs)
return zh_agent_tool_screenshot_name(inputs)
});