feat(asset-read): rewrite read_asset as a reader registry and Operations (#498)

Replace the monolithic agent read_asset tool with a reader registry in
@shumai/core/src/asset-read (model3d, image, video, document) behind a
shared pipeline that handles access checks, version stacks, annotations,
render tasks, and waiting.

- Add read_asset and get_asset_render Operations (agent surface only).
  Renders that outlast the caller's wait budget return "pending" with the
  workflow task ID; the read request is stored on the task payload so
  get_asset_render works across restarts and instances (requester only).
- Add workflowService.waitFor and presign expiry support.
- Keep agent output compatible via an agent tool extension (s3KeyOnly,
  image blocks, session image details); autofill keeps its team-scoped tool.
- Cap video frames at 20 per call; add the get_asset_render agent toggle.
This commit is contained in:
Yiling-J
2026-10-27 08:09:10 +08:00
committed by GitHub
parent e44938cb8f
commit 155fb95a0e
40 changed files with 3787 additions and 2229 deletions
+13
View File
@@ -234,6 +234,19 @@ An **Operation** is a capability agents can use, defined once and exposed automa
- **Register** the definition in `packages/operations/src/registry.ts` and the implementation in `packages/core/src/operations/registry.ts`. A definition without an implementation fails typecheck.
- **Run** Operations only through `runOperation(op, { userId, via, agentId? }, input)`. `via` (`api`, `agent`, `mcp`, `cli`) is recorded in audit logs.
- **Tests**: real-database tests per Operation covering success, permission denial (no side effects), and audit entries.
- **Agent presentation**: by default an agent tool returns the Operation output as JSON. To add agent-only parameters or render output differently (e.g. image blocks), register an `AgentToolExtension` in `packages/agent/src/operation-extensions/`.
### Asset Readers
`read_asset` is backed by a registry of readers in `packages/core/src/asset-read/readers/`, one per asset family (`model3d`, `image`, `video`, `document`). The shared pipeline (`assetReadService`) handles access checks, version stacks, annotations, render tasks, and waiting; a reader only contains type-specific logic.
To add a reader:
1. Add its kind to `READER_KINDS` and its options to `readerOptionSchemas` in `packages/operations/src/asset/read-asset.ts` (one optional `<name>Config` field on `read_asset`).
2. Implement `AssetReader` (`matches`, synchronous `validate`, `plan`, and `collect` if `plan` can return `render`), and add it to `readers` in `packages/core/src/asset-read/registry.ts`. Order matters: the first matching reader wins. A declared kind without a reader fails typecheck.
3. `plan` returns either `ready` with the result, or `render` with a workflow task spec and optional `context`. The pipeline stores the request on the task payload (`assetRead`), so `collect` and `get_asset_render` work across restarts and instances.
4. Use `badRequest(...)` for invalid input. Return storage keys in `ReadResult`; the pipeline turns them into short-lived links (and only gives keys to Shumai's built-in agent).
5. Add reader unit tests (`readers.test.ts`), pipeline cases in `asset-read.test.ts`, and a workflow E2E case if it renders.
## Frontend Architecture
+1
View File
@@ -21,6 +21,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
* **cli**: `shumai-cli` has clearer help for every command (`shumai-cli <command> --help`), suggests the right command when you mistype one, and can print raw JSON with `--json`. `ls` can now filter by type and page through very large folders. Existing commands work as before; the npm package now needs Node.js 22.12 or later
* **agent**: When the agent lists a folder's contents, large folders are now paged with a "next page" marker instead of page numbers, and items appear in the same order as in the file browser
* **agent**: When the agent looks at a long video or a large document that takes more than about a minute to render, it no longer fails with a timeout. The agent is told the render is still in progress and picks up the frames or pages when they're ready, using the new "Get Asset Render" tool. Each call can now read at most 20 video frames, the same limit as PDF pages
* **api-token**: API tokens are now stored securely, so they can no longer be read back from the database. A new token is shown only once when you create it, so copy it right away; the token list now shows just the last few characters. Existing tokens keep working without any changes.
## [0.6.1] - 2026-10-24
+17 -11
View File
@@ -34,60 +34,66 @@ The following tools can be individually toggled in the agent configuration dialo
- **Description:** Allows the agent to inspect images, extract still frames from video files, render PDF document pages, and read text/markdown documents.
- **Why it matters:** Enables multimodal agents to view visual media assets uploaded to Shumai, extract frame-by-frame video screenshots, inspect document layouts, and answer questions about uploaded creative assets.
- **Limits:** Up to 20 video frames or PDF pages per call. If rendering takes longer than about a minute, the tool tells the agent the render is still in progress instead of failing.
### 4. Browse Shumai (`list_assets`)
### 4. Get Asset Render (`get_asset_render`)
- **Description:** Allows the agent to pick up video frames or document pages that were still rendering when `read_asset` returned.
- **Why it matters:** Long videos and large documents can take a while to render. With this tool the agent can come back for the result instead of starting over. It only returns renders that `read_asset` started for the same user.
### 5. Browse Shumai (`list_assets`)
- **Description:** Allows the agent to list folders and browse files in Shumai. Large folders are returned page by page.
- **Why it matters:** Gives the agent visual mapping of the project structure so it can search for other files or locate related assets.
### 5. List Projects (`list_projects`)
### 6. List Projects (`list_projects`)
- **Description:** Allows the agent to list the projects you have access to across your teams, including each project's root folder.
- **Why it matters:** Lets the agent find the right project and its root folder on its own before browsing or creating files.
### 6. Create Folder (`create_folder`)
### 7. Create Folder (`create_folder`)
- **Description:** Allows the agent to create new folders in Shumai.
- **Why it matters:** Needed if the agent organizes outputs or places files in custom subdirectory folders.
### 7. Create File (`create_file`)
### 8. Create File (`create_file`)
- **Description:** Allows the agent to create and upload new files to Shumai.
- **Why it matters:** Allows the agent to generate new assets (e.g. text logs, transcoded clips, reports) and save them directly back into your project.
### 8. Create Version (`create_version`)
### 9. Create Version (`create_version`)
- **Description:** Allows the agent to upload and stack a new version of an existing file in Shumai.
- **Why it matters:** Allows the agent to update files in a clean, non-destructive way by stacking revisions rather than creating duplicate files.
### 9. Generate Image (`generate_image`)
### 10. Generate Image (`generate_image`)
- **Description:** Allows the agent to generate still images from text prompts and reference images using configured AI image models.
- **Why it matters:** Enables agents to create concept art, mockups, textures, and illustrations directly inside Shumai.
- **Prerequisite:** This tool requires at least one image model to be configured and enabled in [Image / Video Generation](/using-shumai/image-video-generation). If no image models are enabled under a configured provider, this tool is omitted from the agent session even if enabled in the agent settings dialog.
### 10. Generate Video (`generate_video`)
### 11. Generate Video (`generate_video`)
- **Description:** Allows the agent to generate video clips from text prompts, image animation, or transition keyframes using configured AI video models.
- **Why it matters:** Enables agents to produce short animated clips, camera motion shots, and video transitions.
- **Prerequisite:** This tool requires at least one video model to be configured and enabled in [Image / Video Generation](/using-shumai/image-video-generation). If no video models are enabled under a configured provider, this tool is omitted from the agent session even if enabled in the agent settings dialog.
### 11. Download Asset (`download_asset`)
### 12. Download Asset (`download_asset`)
- **Description:** Allows the agent to download a workspace asset or specific storage S3 key to the local `.pi` directory for processing with local scripts or bash commands.
- **Why it matters:** Enables agents executing custom skill scripts to fetch files or intermediate transcode artifacts to local disk.
### 12. Rename Asset (`rename_asset`)
### 13. Rename Asset (`rename_asset`)
- **Description:** Allows the agent to rename an existing file or folder in Shumai.
- **Why it matters:** Enables agents to keep project files neatly named and organized according to workspace naming conventions. Enabled by default.
### 13. Move Assets (`move_assets`)
### 14. Move Assets (`move_assets`)
- **Description:** Allows the agent to move one or more assets to a destination parent folder.
- **Why it matters:** Enables agents to reorganize workspace structures, relocate generated outputs, or tidy up project folders. Enabled by default.
### 14. Delete Asset (`delete_asset`)
### 15. Delete Asset (`delete_asset`)
- **Description:** Allows the agent to delete a single file or folder by moving it to the Trash bin.
- **Why it matters:** Provides agents with the ability to clean up outdated or redundant assets. To prevent accidental data loss, this tool is **disabled by default** and strictly limited to deleting one asset at a time. Team Owners must explicitly enable this tool for an agent.
@@ -1,8 +1,8 @@
import { describe, it, expect, beforeEach } from 'vitest'
import { prisma } from '@shumai/db'
import { setupTestDbHooks } from '@shumai/db/test'
import { resolveAnnotationsById } from './annotation-resolver'
import { DatabaseSessionStorage } from '../database-session-storage'
import { resolveAnnotationsById } from '@shumai/core/src/asset-read/annotations'
import { DatabaseSessionStorage } from './database-session-storage'
import type { SessionTreeEntry } from '@earendil-works/pi-agent-core'
import { degreeToSecond } from '@shumai/core/src/utils/model3d'
@@ -7,7 +7,7 @@ import { DatabaseSessionStorage } from './database-session-storage'
import { agentService } from '@shumai/core/src/agent/agent'
import { createAgentSession, type DbProviderInfo } from './index'
import * as sandboxedBashModule from './tools/sandboxed-bash'
import * as readAssetModule from './tools/read-asset'
import * as operationToolsModule from './operation-tools'
describe('DatabaseSessionStorage', () => {
setupTestDbHooks()
@@ -450,8 +450,8 @@ describe('DatabaseSessionStorage', () => {
})
const sessionId = initialStorage.sessionId
// 2. Spy on media tool constructor
const createReadAssetToolSpy = vi.spyOn(readAssetModule, 'createReadAssetTool')
// 2. Spy on the Operation tools builder (read_asset comes from the Operation registry)
const buildOperationToolsSpy = vi.spyOn(operationToolsModule, 'buildOperationTools')
// 3. Resume the session via createAgentSession with a NEW comment ID
await createAgentSession({
@@ -468,8 +468,10 @@ describe('DatabaseSessionStorage', () => {
providers: mockProviders,
})
// 4. Assert that the tool was instantiated using user ID
expect(createReadAssetToolSpy).toHaveBeenCalledWith(user.id)
// 4. Assert that the tools were built for the user
expect(buildOperationToolsSpy).toHaveBeenCalledWith(
expect.objectContaining({ userId: user.id }),
)
})
it('should instantiate media tools with user ID in subsequent turns (video)', async () => {
@@ -502,8 +504,8 @@ describe('DatabaseSessionStorage', () => {
})
const sessionId = initialStorage.sessionId
// 2. Spy on media tool constructor
const createReadAssetToolSpy = vi.spyOn(readAssetModule, 'createReadAssetTool')
// 2. Spy on the Operation tools builder (read_asset comes from the Operation registry)
const buildOperationToolsSpy = vi.spyOn(operationToolsModule, 'buildOperationTools')
// 3. Resume the session via createAgentSession with a NEW comment ID
await createAgentSession({
@@ -520,8 +522,10 @@ describe('DatabaseSessionStorage', () => {
providers: mockProviders,
})
// 4. Assert that the tool was instantiated using user ID
expect(createReadAssetToolSpy).toHaveBeenCalledWith(user.id)
// 4. Assert that the tools were built for the user
expect(buildOperationToolsSpy).toHaveBeenCalledWith(
expect.objectContaining({ userId: user.id }),
)
})
describe('P1 Bug Reproductions', () => {
+7 -7
View File
@@ -21,13 +21,13 @@ import { metadataService } from '@shumai/core/src/metadata/metadata'
import { createCreateFileTool } from './tools/create-file'
import { createCreateVersionTool } from './tools/create-version'
import { createDownloadAssetTool } from './tools/download-asset'
import { createReadAssetTool } from './tools/read-asset'
import { createReadSkillTool } from './tools/read-skill'
import { createReadThreadTool } from './tools/read-thread'
import { createSandboxedBashTool } from './tools/sandboxed-bash'
import { createGenerateImageTool } from './tools/generate-image'
import { createGenerateVideoTool } from './tools/generate-video'
import { buildOperationTools } from './operation-tools'
import { createAutofillReadAssetTool } from './operation-extensions'
import { mediaGenerationService } from '@shumai/core/src/media-generation/media-generation'
import { serializeContextToXml } from './context/serialize-context'
import { DEFAULT_DENIED_TOOLS, type ShumaiMessageContext } from '@shumai/dtos'
@@ -267,13 +267,12 @@ export async function createAgentSession(params: CreateAgentSessionParams) {
const deniedTools = agentConfig?.deniedTools ?? [...DEFAULT_DENIED_TOOLS]
const mediaTools: AgentTool[] = []
if (userId) {
mediaTools.push(createReadAssetTool(userId))
} else if (agent?.type === 'autofill' && assetId) {
// Agents acting for a user get read_asset / get_asset_render from the Operation registry
// (see systemTools below); the autofill system agent has its own team-scoped read_asset.
if (!userId && agent?.type === 'autofill' && assetId) {
mediaTools.push(
createReadAssetTool({
createAutofillReadAssetTool({
teamId,
agentType: 'autofill',
targetAssetId: assetId,
}),
)
@@ -337,7 +336,8 @@ export async function createAgentSession(params: CreateAgentSessionParams) {
const metadataSchema = await buildProjectMetadataSchema(projectId)
const agentContext = { teamId, agentId }
systemTools.push(
// Shared Operations (also exposed via MCP and the CLI): list/create/rename/move/delete
// Shared Operations (also exposed via MCP and the CLI): list/create/rename/move/delete,
// read_asset and get_asset_render
...buildOperationTools({ userId, agentId, teamId, projectId }),
createCreateFileTool(userId, metadataSchema, agentContext),
createCreateVersionTool(userId, metadataSchema, agentContext),
@@ -0,0 +1,2 @@
// Registers agent-specific presentation for Operations. Import once before building tools.
export { createAutofillReadAssetTool, presentReadOutput } from './read-asset'
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,155 @@
import { z } from 'zod'
import { type AgentTool, type AgentToolResult } from '@earendil-works/pi-agent-core'
import { type ImageContent, type TextContent } from '@earendil-works/pi-ai'
import { readAsset, type AssetReadOutput } from '@shumai/operations'
import { assetReadService, toAssetReadOutput } from '@shumai/core/src/asset-read'
import { getOperation, READ_WAIT_BUDGET_MS } from '@shumai/core/src/operations'
import { s3Service } from '@shumai/core/src/s3/s3'
import { getFileMimeType } from '@shumai/core/src/utils/file-mime'
import { MAX_TEXT_BYTES, MAX_TEXT_LINES } from '@shumai/core/src/asset-read'
import {
registerAgentToolExtension,
splitAgentParams,
toToolParameters,
type AgentToolExtension,
} from '../operation-tools'
const extraParameters = z.object({
s3KeyOnly: z
.boolean()
.optional()
.describe(
'When true, returns only the storage S3 key(s) and MIME types of the asset/frames without attaching base64 image or text content to the agent context. Use this to obtain S3 keys for generate_image or generate_video references without consuming context tokens.',
),
})
/** Details kept compatible with the previous tool; `sourceKeys` drives session image storage. */
interface ReadAssetDetails {
assetId: string
name: string
mediaType: string | null
key?: string
keys: string[]
downloadUrl?: string
results: Array<{ key: string; label: string; downloadUrl?: string }>
sourceKeys: string[]
size?: number
isTruncated?: boolean
}
/** Render a read_asset / get_asset_render output as agent content (text plus image blocks). */
export async function presentReadOutput(
output: AssetReadOutput,
opts: { s3KeyOnly: boolean },
): Promise<AgentToolResult<ReadAssetDetails | { taskId?: string; status: 'pending' }>> {
if (output.status === 'pending') {
const retry = output.retryAfterSeconds ?? 5
return {
content: [
{
type: 'text',
text: `${output.summary} Call get_asset_render with taskId "${output.taskId}" in about ${retry} seconds to get the result.`,
},
],
details: { taskId: output.taskId, status: 'pending' },
}
}
const bucket = process.env.S3_BUCKET || 'shumai'
const imageBlocks: ImageContent[] = []
const imageLines: string[] = []
for (const image of output.images) {
const key = image.key ?? ''
let mimeType = image.mimeType
if (!opts.s3KeyOnly && key) {
const { buffer, contentType } = await s3Service.getObject(bucket, key)
mimeType =
contentType && contentType !== 'application/octet-stream'
? contentType
: getFileMimeType(buffer, key, image.mimeType)
imageBlocks.push({ type: 'image', data: buffer.toString('base64'), mimeType })
}
imageLines.push(`- ${image.label} (S3 Key: "${key}", MIME: ${mimeType})`)
}
const lines = [output.summary, ...output.details, ...imageLines]
if (output.text) {
lines.push(`- Source file (S3 Key: "${output.text.key ?? ''}", MIME: ${output.text.mimeType})`)
}
let text = lines.join('\n')
if (output.text && !opts.s3KeyOnly) {
text += `\n\n${output.text.content}`
if (output.text.truncated) {
text += `\n\n[Content truncated to ${MAX_TEXT_LINES} lines / ${MAX_TEXT_BYTES / 1024}KB limit. Use download_asset to process the full file.]`
}
}
const imageKeys = output.images.map((i) => i.key ?? '')
const content: Array<TextContent | ImageContent> = [{ type: 'text', text }, ...imageBlocks]
return {
content,
details: {
assetId: output.asset.id,
name: output.asset.name,
mediaType: output.asset.mediaType,
key: imageKeys[0] ?? output.text?.key,
keys: imageKeys,
downloadUrl: output.images[0]?.url ?? output.text?.url,
results: output.images.map((i) => ({ key: i.key ?? '', label: i.label, downloadUrl: i.url })),
sourceKeys: output.text?.key && imageKeys.length === 0 ? [output.text.key] : imageKeys,
...(output.text ? { size: output.text.size, isTruncated: output.text.truncated } : {}),
},
}
}
const readAssetExtension: AgentToolExtension = {
extraParameters,
// s3KeyOnly also means "don't read text content", as before.
toInput: (input, extra) => (extra.s3KeyOnly ? { ...input, includeContent: false } : input),
present: (output, extra) =>
presentReadOutput(output as AssetReadOutput, { s3KeyOnly: extra.s3KeyOnly === true }),
}
const getAssetRenderExtension: AgentToolExtension = {
present: (output) => presentReadOutput(output as AssetReadOutput, { s3KeyOnly: false }),
}
registerAgentToolExtension('read_asset', readAssetExtension)
registerAgentToolExtension('get_asset_render', getAssetRenderExtension)
/** Longer than the chat budget: autofill can't poll get_asset_render (it has no user). */
const AUTOFILL_WAIT_BUDGET_MS = 2 * READ_WAIT_BUDGET_MS.agent
/**
* `read_asset` for the autofill system agent, which runs without a user. It uses the same
* pipeline and presentation, but access is confined to the agent's team and target asset.
*/
export function createAutofillReadAssetTool(access: {
teamId?: string
targetAssetId?: string
}): AgentTool {
const op = getOperation(readAsset.name)!
return {
name: readAsset.name,
label: readAsset.title,
description: readAsset.description,
parameters: toToolParameters(op, readAssetExtension),
execute: async (_toolCallId, params) => {
const { input, extra } = splitAgentParams(op, readAssetExtension, params)
const parsed = readAsset.input.safeParse(input)
if (!parsed.success) {
throw new Error(`Invalid input for read_asset: ${z.prettifyError(parsed.error)}`)
}
const { maxWaitSeconds, ...request } = parsed.data
const budget =
maxWaitSeconds === undefined
? AUTOFILL_WAIT_BUDGET_MS
: Math.min(AUTOFILL_WAIT_BUDGET_MS, maxWaitSeconds * 1000)
const outcome = await assetReadService.read({ kind: 'autofill', ...access }, request, {
waitBudgetMs: budget,
})
const output = await toAssetReadOutput(outcome, { includeKeys: true })
return presentReadOutput(output, { s3KeyOnly: extra.s3KeyOnly === true })
},
}
}
+3 -1
View File
@@ -61,7 +61,7 @@ describe('operation-backed agent tools', () => {
})
describe('tool definitions', () => {
it('keeps the existing tool names and adds list_projects', () => {
it('keeps the existing tool names and adds list_projects and get_asset_render', () => {
expect(
buildOperationTools(ctx)
.map((t) => t.name)
@@ -69,9 +69,11 @@ describe('operation-backed agent tools', () => {
).toEqual([
'create_folder',
'delete_asset',
'get_asset_render',
'list_assets',
'list_projects',
'move_assets',
'read_asset',
'rename_asset',
])
})
+60 -12
View File
@@ -1,7 +1,7 @@
import { z } from 'zod'
import type { TSchema } from 'typebox'
import { Compile } from 'typebox/compile'
import { type AgentTool } from '@earendil-works/pi-agent-core'
import { type AgentTool, type AgentToolResult } from '@earendil-works/pi-agent-core'
import {
getOperationsForSurface,
runOperation,
@@ -12,13 +12,41 @@ import {
/** Context for agent-invoked Operations; `via` is always `agent`. */
export type AgentOperationContext = Omit<OperationContext, 'via'>
/**
* Agent-specific presentation for an Operation. By default a tool returns the Operation output
* as JSON text; an extension can add agent-only parameters, map them into the Operation input,
* and render the output (e.g. as image blocks).
*/
export interface AgentToolExtension {
/** Extra parameters only the agent sees; they are removed before the Operation runs. */
extraParameters?: z.ZodObject
/** Adjust the Operation input using the extra parameters. */
toInput?(input: Record<string, unknown>, extra: Record<string, unknown>): Record<string, unknown>
present(output: unknown, extra: Record<string, unknown>): Promise<AgentToolResult<unknown>>
}
const extensions = new Map<string, AgentToolExtension>()
/** Register agent presentation for an Operation (see `./operation-extensions`). */
export function registerAgentToolExtension(name: string, extension: AgentToolExtension): void {
extensions.set(name, extension)
}
function toolInputSchema(op: Operation, extension?: AgentToolExtension): z.ZodObject {
return extension?.extraParameters
? op.definition.input.extend(extension.extraParameters.shape)
: op.definition.input
}
/**
* Convert an Operation's Zod input schema into the JSON Schema the agent harness expects.
* TypeBox schemas are plain JSON Schema at runtime, so the converted schema is used as-is.
* Throws when the schema cannot be compiled, so a bad definition fails loudly in tests.
*/
export function toToolParameters(op: Operation): TSchema {
const schema: Record<string, unknown> = z.toJSONSchema(op.definition.input, { io: 'input' })
export function toToolParameters(op: Operation, extension?: AgentToolExtension): TSchema {
const schema: Record<string, unknown> = z.toJSONSchema(toolInputSchema(op, extension), {
io: 'input',
})
// `$schema` is not understood by the harness validator or every model provider.
delete schema.$schema
Compile(schema)
@@ -27,11 +55,11 @@ export function toToolParameters(op: Operation): TSchema {
/**
* Models sometimes send `null` for optional parameters they don't want to set. Drop those
* so the Operation's input schema treats them as omitted; nulls for required fields still fail.
* so the input schema treats them as omitted; nulls for required fields still fail.
*/
function dropNullOptionalFields(op: Operation, params: unknown): unknown {
export function dropNullOptionalFields(schema: z.ZodObject, params: unknown): unknown {
if (!params || typeof params !== 'object' || Array.isArray(params)) return params
const shape = op.definition.input.shape
const shape = schema.shape
return Object.fromEntries(
Object.entries(params).filter(
([key, value]) => !(value === null && shape[key]?.safeParse(undefined).success),
@@ -39,20 +67,40 @@ function dropNullOptionalFields(op: Operation, params: unknown): unknown {
)
}
/** Split agent-only parameters from the Operation input. */
export function splitAgentParams(
op: Operation,
extension: AgentToolExtension | undefined,
params: unknown,
): { input: unknown; extra: Record<string, unknown> } {
const cleaned = dropNullOptionalFields(toolInputSchema(op, extension), params)
if (!extension?.extraParameters || !cleaned || typeof cleaned !== 'object') {
return { input: cleaned, extra: {} }
}
const extraKeys = new Set(Object.keys(extension.extraParameters.shape))
const input: Record<string, unknown> = {}
const extra: Record<string, unknown> = {}
for (const [key, value] of Object.entries(cleaned)) {
;(extraKeys.has(key) ? extra : input)[key] = value
}
return { input: extension.toInput ? extension.toInput(input, extra) : input, extra }
}
/** Expose an Operation as a native agent tool that runs through `runOperation`. */
export function toAgentTool(op: Operation, ctx: AgentOperationContext): AgentTool {
const { definition } = op
const extension = extensions.get(definition.name)
return {
name: definition.name,
label: definition.title,
description: definition.description,
parameters: toToolParameters(op),
parameters: toToolParameters(op, extension),
execute: async (_toolCallId, params) => {
const output = await runOperation(
op,
{ ...ctx, via: 'agent' },
dropNullOptionalFields(op, params),
)
const { input, extra } = splitAgentParams(op, extension, params)
const output = await runOperation(op, { ...ctx, via: 'agent' }, input)
if (extension) {
return extension.present(output, extra)
}
return {
content: [{ type: 'text', text: JSON.stringify(output, null, 2) }],
details: {},
File diff suppressed because it is too large Load Diff
-922
View File
@@ -1,922 +0,0 @@
import { Type } from 'typebox'
import { type AgentTool } from '@earendil-works/pi-agent-core'
import { type ImageContent, type TextContent } from '@earendil-works/pi-ai'
import { prisma, WorkflowTaskType, WorkflowTaskStatus, type User } from '@shumai/db'
import { s3Service } from '@shumai/core/src/s3/s3'
import { workflowService } from '@shumai/workflow-core'
import { authzService, Permission, ResourceType } from '@shumai/core/src/authz/authz'
import { resolveAnnotationsById } from './annotation-resolver'
import { getFileMimeType } from '@shumai/core/src/utils/file-mime'
import { is3dModel } from '@shumai/core/src/utils/mime'
import {
isValidTurntableDegree,
secondToDegree,
degreeToSecond,
} from '@shumai/core/src/utils/model3d'
export const readAssetSchema = Type.Object(
{
assetId: Type.String({
description: 'The asset ID of the workspace asset to inspect. Required.',
}),
annotationId: Type.Union([
Type.String({
description:
'The ID of the comment or message entry from <annotation id="..." /> whose visual markup should be overlaid, or null.',
}),
Type.Null(),
]),
s3KeyOnly: Type.Optional(
Type.Boolean({
description:
'When true, returns only the storage S3 key(s) and MIME types of the asset/frames without attaching base64 image or text content to the agent context. Use this to obtain S3 keys for generate_image or generate_video references without consuming context tokens.',
default: false,
}),
),
imageConfig: Type.Union([
Type.Object(
{},
{
additionalProperties: false,
description: 'Optional configuration for image assets, or null.',
},
),
Type.Null(),
]),
config3d: Type.Optional(
Type.Union([
Type.Object(
{
degree: Type.Optional(
Type.Union([
Type.Integer({
minimum: 0,
maximum: 345,
description:
'Turntable rotation degree (integer: 0, 15, 30, ..., 345 in 15° increments) to inspect a specific angle of a 3D model. Leave null/omitted to view the 4-view composite overview.',
}),
Type.Null(),
]),
),
},
{
additionalProperties: false,
description:
'Configuration for 3D model assets. Pass null if inspecting another asset type. Optionally pass degree to inspect a specific turntable angle, or leave null/omitted to view the 4-view composite overview.',
},
),
Type.Null(),
]),
),
videoConfig: Type.Union([
Type.Object(
{
start: Type.Number({ description: 'Start time in seconds.' }),
end: Type.Number({ description: 'End time in seconds.' }),
count: Type.Number({ description: 'Number of frames to extract (e.g. 1 to 10).' }),
},
{
additionalProperties: false,
description:
'Configuration for video assets. Required when inspecting videos, or null for other asset types.',
},
),
Type.Null(),
]),
docConfig: Type.Union([
Type.Object(
{
mode: Type.Union([
Type.Literal('pages', {
description:
'Render specific page range as images for visual layout, charts, or visual annotations (required for binary PDFs).',
}),
Type.Literal('text', {
description:
'Read raw text content directly (ideal for .md, .txt, .csv, code files).',
}),
]),
startPage: Type.Union([
Type.Number({
description:
'Start page number (1-based index, required when mode="pages"), or null when mode="text".',
}),
Type.Null(),
]),
endPage: Type.Union([
Type.Number({
description:
'End page number (1-based index, max 20 pages per call, required when mode="pages"), or null when mode="text".',
}),
Type.Null(),
]),
},
{
additionalProperties: false,
description:
'Configuration for document/PDF/text assets, or null if inspecting another asset type.',
},
),
Type.Null(),
]),
},
{ additionalProperties: false },
)
export function isPlainTextAsset(mediaType: string, filename: string): boolean {
if (
mediaType.startsWith('text/') ||
mediaType === 'application/json' ||
mediaType === 'application/javascript' ||
mediaType === 'application/typescript' ||
mediaType === 'application/xml' ||
mediaType === 'application/x-yaml'
) {
return true
}
const textExtensions = [
'.txt',
'.md',
'.markdown',
'.json',
'.csv',
'.tsv',
'.js',
'.ts',
'.tsx',
'.jsx',
'.html',
'.htm',
'.css',
'.scss',
'.yaml',
'.yml',
'.xml',
'.py',
'.rb',
'.go',
'.rs',
'.c',
'.cpp',
'.h',
'.hpp',
'.java',
'.sh',
'.bash',
'.zsh',
'.sql',
]
return textExtensions.some((ext) => filename.endsWith(ext))
}
const MAX_TEXT_BYTES = 50 * 1024
const MAX_TEXT_LINES = 1000
export type ReadAssetAuthContext =
| {
userId?: string
teamId?: string
agentType?: 'chat' | 'embedding'
targetAssetId?: string
}
| {
userId?: undefined
teamId: string
agentType: 'autofill'
targetAssetId: string
}
export function createReadAssetTool(
auth: string | ReadAssetAuthContext,
): AgentTool<typeof readAssetSchema> {
const authContext: ReadAssetAuthContext =
typeof auth === 'string' ? { userId: auth, agentType: 'chat' } : auth
const { userId, teamId, agentType = 'chat', targetAssetId } = authContext
return {
name: 'read_asset',
label: 'Read Asset',
description:
'Inspects and reads the content of an asset in the workspace. ' +
'Supports visual analysis for images, 3D models (overview or specific degree angle), video frame extraction, PDF page rendering, and direct text reading for text/markdown files. ' +
'Pass assetId, and provide the specific config (imageConfig, config3d, videoConfig, or docConfig) while setting unused configs to null. ' +
'Optionally provide annotationId to view visual markups drawn on the asset.',
parameters: readAssetSchema,
execute: async (_toolCallId, params) => {
const { assetId, annotationId, s3KeyOnly, config3d, videoConfig, docConfig } = params
if (agentType === 'autofill' && !userId) {
if (!targetAssetId) {
throw new Error('Target asset ID is required for autofill agent authorization.')
}
if (!teamId) {
throw new Error('Team ID is required for autofill agent authorization.')
}
// System background execution for Autofill Agent
const assetRecord = await prisma.asset.findUnique({
where: { id: assetId },
include: { project: true, teamRootFolder: true },
})
if (!assetRecord) {
throw new Error(`Asset with ID ${assetId} not found.`)
}
// Strict team boundary check
const assetTeamId = assetRecord.project?.teamId || assetRecord.teamRootFolder?.id
if (!assetTeamId || assetTeamId !== teamId) {
throw new Error(`Access denied: asset ${assetId} does not belong to team ${teamId}.`)
}
// Strict target asset boundary check (allows the target asset or its version stack children)
if (assetRecord.id !== targetAssetId && assetRecord.parentId !== targetAssetId) {
throw new Error(
`Access denied: autofill agent can only read target asset ${targetAssetId}.`,
)
}
} else {
// Chat agents or user-driven executions strictly require a userId and full ACL validation
if (!userId) {
throw new Error('User ID is required for authorization.')
}
await authzService.hasPermission({
user: { id: userId } as User,
permission: Permission.Read,
type: ResourceType.Asset,
id: assetId,
})
}
let asset = await prisma.asset.findUnique({
where: { id: assetId },
include: { storageKey: true },
})
if (!asset) {
throw new Error(`Asset with ID ${assetId} not found.`)
}
if (asset.type === 'version_stack') {
const latestVersion = await prisma.asset.findFirst({
where: { parentId: asset.id, isDeleted: false },
orderBy: { sortIndex: 'asc' },
include: { storageKey: true },
})
if (latestVersion) {
asset = latestVersion
}
}
const mediaInfo = asset.media as unknown as PrismaJson.MediaInfo | null
const proxyType = mediaInfo?.proxyType
const mediaType = asset.mediaType?.toLowerCase() || ''
const filename = asset.name?.toLowerCase() || ''
const bucket = process.env.S3_BUCKET || 'shumai'
// ----------------------------------------------------------------------
// Branch 0: 3D Assets (Orthogonal 4-View Composite Image or Degree Angle)
// ----------------------------------------------------------------------
const is3d = proxyType === '3d' || is3dModel(mediaType, filename)
if (is3d) {
if (videoConfig) {
throw new Error(
`Asset ${asset.id} is a 3D model. To inspect specific angles pass 'config3d: { degree: ... }'; do not pass videoConfig.`,
)
}
if (config3d?.degree !== undefined && config3d?.degree !== null) {
if (!isValidTurntableDegree(config3d.degree)) {
throw new Error(
`Invalid turntable degree: ${config3d.degree}. Degree must be an integer multiple of 15 between 0 and 345 (e.g. 0, 15, 30, 45, ..., 345).`,
)
}
}
let annotations: PrismaJson.AnnotationList | null = null
let commentTimestamp: number | null = null
if (annotationId) {
const resolved = await resolveAnnotationsById(asset.id, annotationId)
annotations = resolved.annotations
commentTimestamp = resolved.timestamp
if (commentTimestamp !== null && commentTimestamp !== undefined) {
const commentDegree = secondToDegree(commentTimestamp)
if (
config3d?.degree !== undefined &&
config3d?.degree !== null &&
config3d.degree !== commentDegree
) {
throw new Error(
`Annotation "${annotationId}" was drawn at ${commentDegree}°, but degree ${config3d.degree}° was specified. Do not pass a conflicting degree with annotationId.`,
)
}
}
}
if ((config3d?.degree !== undefined && config3d?.degree !== null) || annotationId) {
if (!mediaInfo?.videoTranscodes || mediaInfo.videoTranscodes.length === 0) {
throw new Error(
`Asset ${asset.id} is a 3D model, but its turntable video (turntable.mp4) is missing.`,
)
}
let targetDegree: number
let targetSecond: number
if (config3d?.degree !== undefined && config3d?.degree !== null) {
targetDegree = config3d.degree
targetSecond = degreeToSecond(targetDegree)
} else {
targetSecond =
commentTimestamp !== null && commentTimestamp !== undefined ? commentTimestamp : 0
targetDegree = secondToDegree(targetSecond)
}
const task = await prisma.workflowTask.create({
data: {
assetId: asset.id,
projectId: asset.projectId || 'none',
type: WorkflowTaskType.transcode_screenshot,
status: WorkflowTaskStatus.pending,
payload: {
projectId: asset.projectId || 'none',
screenshot: {
start: targetSecond,
end: targetSecond,
count: 1,
commentTimestamp: targetSecond,
annotations: annotations ?? undefined,
},
},
},
})
const completedTask = await workflowService.executeWait(task)
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- transcode screenshot output
const output = completedTask.output as any
const screenshots = output?.screenshots as
Array<{ key: string; timestamp: number }> | undefined
if (!screenshots || screenshots.length === 0) {
throw new Error(
`Failed to extract 3D turntable frame for asset ${asset.id} at ${targetDegree}°.`,
)
}
const shotKey = screenshots[0].key
const downloadUrl = await s3Service.presign(
bucket,
shotKey,
'GET',
true,
asset.name || undefined,
)
const content: Array<TextContent | ImageContent> = []
const hasMarkup = Boolean(annotations && annotations.length > 0)
const textDescription = hasMarkup
? `3D asset "${asset.name}" (ID: ${asset.id}, S3 Key: "${shotKey}") view at ${targetDegree}° with visual markups overlaid:`
: `3D asset "${asset.name}" (ID: ${asset.id}, S3 Key: "${shotKey}") view at ${targetDegree}°:`
if (s3KeyOnly) {
content.push({
type: 'text',
text: textDescription,
})
} else {
const { buffer, contentType } = await s3Service.getObject(bucket, shotKey)
const actualMimeType =
contentType && contentType !== 'application/octet-stream' ? contentType : 'image/webp'
content.push({
type: 'text',
text: textDescription,
})
content.push({
type: 'image',
data: buffer.toString('base64'),
mimeType: actualMimeType,
})
}
return {
content,
details: {
assetId: asset.id,
name: asset.name,
mediaType: asset.mediaType,
key: shotKey,
downloadUrl,
sourceKeys: [shotKey],
},
}
}
if (!mediaInfo?.imageTranscodes || mediaInfo.imageTranscodes.length === 0) {
throw new Error(
`Asset ${asset.id} is a 3D model, but its 4-views image (views.webp) is missing.`,
)
}
const mediaKey = mediaInfo.imageTranscodes[0].key
if (!mediaKey) {
throw new Error(`Asset ${asset.id} is a 3D model, but its 4-views image key is missing.`)
}
const keyToUse = mediaKey
const mimeType = 'image/webp'
const downloadUrl = await s3Service.presign(
bucket,
keyToUse,
'GET',
true,
asset.name || undefined,
)
const content: Array<TextContent | ImageContent> = []
const textDescription = [
`3D asset "${asset.name}" (ID: ${asset.id}, MIME: ${mimeType}, S3 Key: "${keyToUse}"):`,
'Orthogonal 4-view composite image (2x2 grid across 360° rotation):',
'- Top-Left quadrant: Front view (0°)',
'- Top-Right quadrant: Right side view (90°)',
'- Bottom-Left quadrant: Back view (180°)',
'- Bottom-Right quadrant: Left side view (270°)',
].join('\n')
if (s3KeyOnly) {
content.push({
type: 'text',
text: textDescription,
})
} else {
const { buffer, contentType } = await s3Service.getObject(bucket, keyToUse)
const actualMimeType =
contentType && contentType !== 'application/octet-stream' ? contentType : mimeType
content.push({
type: 'text',
text: textDescription,
})
content.push({
type: 'image',
data: buffer.toString('base64'),
mimeType: actualMimeType,
})
}
return {
content,
details: {
assetId: asset.id,
name: asset.name,
mediaType: asset.mediaType,
key: keyToUse,
downloadUrl,
sourceKeys: [keyToUse],
},
}
}
// ----------------------------------------------------------------------
// Branch 1: Image Assets
// ----------------------------------------------------------------------
if (proxyType === 'image' || mediaType.startsWith('image/') || filename.endsWith('.psd')) {
let mediaKey = asset.storageKey?.key
if (mediaInfo?.imageTranscodes && mediaInfo.imageTranscodes.length > 0) {
mediaKey = mediaInfo.imageTranscodes[0].key || mediaKey
}
if (!mediaKey) {
throw new Error('No media content found for this asset.')
}
let keyToUse = mediaKey
let mimeType = getFileMimeType(null, keyToUse, asset.mediaType || 'image/png')
if (annotationId) {
const { annotations } = await resolveAnnotationsById(asset.id, annotationId)
if (annotations && annotations.length > 0) {
const task = await prisma.workflowTask.create({
data: {
assetId: asset.id,
projectId: asset.projectId || 'none',
type: WorkflowTaskType.transcode_image_annotation,
status: WorkflowTaskStatus.pending,
payload: {
projectId: asset.projectId || 'none',
imageAnnotation: {
annotations,
},
},
},
})
const completedTask = await workflowService.executeWait(task)
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- transcode image annotation task output
const output = completedTask.output as any
if (output?.key) {
keyToUse = output.key
mimeType = 'image/webp'
} else {
throw new Error('Image annotation transcode workflow failed to output S3 key.')
}
}
}
const downloadUrl = await s3Service.presign(
bucket,
keyToUse,
'GET',
true,
asset.name || undefined,
)
const content: Array<TextContent | ImageContent> = []
if (s3KeyOnly) {
content.push({
type: 'text',
text: `Image asset "${asset.name}" (ID: ${asset.id}, MIME: ${mimeType}, S3 Key: "${keyToUse}")`,
})
} else {
const { buffer, contentType } = await s3Service.getObject(bucket, keyToUse)
const actualMimeType =
contentType && contentType !== 'application/octet-stream' ? contentType : mimeType
content.push({
type: 'text',
text: `Image asset "${asset.name}" (ID: ${asset.id}, MIME: ${actualMimeType}, S3 Key: "${keyToUse}")`,
})
content.push({
type: 'image',
data: buffer.toString('base64'),
mimeType: actualMimeType,
})
}
return {
content,
details: {
assetId: asset.id,
name: asset.name,
mediaType: asset.mediaType,
key: keyToUse,
downloadUrl,
sourceKeys: [keyToUse],
},
}
}
// ----------------------------------------------------------------------
// Branch 2: Video Assets
// ----------------------------------------------------------------------
if (proxyType === 'video' || mediaType.startsWith('video/')) {
if (!videoConfig) {
throw new Error(
'videoConfig with start, end, and count is required when inspecting video assets.',
)
}
const { start, end, count } = videoConfig
if (start < 0 || end < 0) {
throw new Error('Invalid video time range: start and end must be non-negative numbers.')
}
if (start > end) {
throw new Error(
`Invalid video time range: start (${start}) must be less than or equal to end (${end}).`,
)
}
if (count < 1) {
throw new Error(`Invalid frame count: count (${count}) must be at least 1.`)
}
const { annotations, timestamp: commentTimestamp } = await resolveAnnotationsById(
asset.id,
annotationId,
)
const task = await prisma.workflowTask.create({
data: {
assetId: asset.id,
projectId: asset.projectId || 'none',
type: WorkflowTaskType.transcode_screenshot,
status: WorkflowTaskStatus.pending,
payload: {
projectId: asset.projectId || 'none',
screenshot: {
start,
end,
count,
commentTimestamp,
annotations,
},
},
},
})
const completedTask = await workflowService.executeWait(task)
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- transcode screenshot output
const output = completedTask.output as any
const screenshots = output?.screenshots as
Array<{ key: string; timestamp: number }> | undefined
if (!screenshots || screenshots.length === 0) {
throw new Error('No screenshots were generated.')
}
const content: Array<TextContent | ImageContent> = []
const sourceKeys: string[] = []
const results: Array<{ key: string; timestamp: number; downloadUrl: string }> = []
const textLines: string[] = [
`Extracted ${screenshots.length} frame(s) from "${asset.name}" (${start}s - ${end}s):`,
]
for (const shot of screenshots) {
const downloadUrl = await s3Service.presign(
bucket,
shot.key,
'GET',
true,
asset.name || undefined,
)
sourceKeys.push(shot.key)
results.push({
key: shot.key,
timestamp: shot.timestamp,
downloadUrl,
})
if (s3KeyOnly) {
const mimeType = getFileMimeType(null, shot.key, 'image/webp')
textLines.push(
`- Frame at ${shot.timestamp}s (S3 Key: "${shot.key}", MIME: ${mimeType})`,
)
} else {
const { buffer, contentType } = await s3Service.getObject(bucket, shot.key)
const mimeType =
contentType && contentType !== 'application/octet-stream'
? contentType
: getFileMimeType(buffer, shot.key, 'image/webp')
textLines.push(
`- Frame at ${shot.timestamp}s (S3 Key: "${shot.key}", MIME: ${mimeType})`,
)
content.push({
type: 'image',
data: buffer.toString('base64'),
mimeType,
})
}
}
content.unshift({
type: 'text',
text: textLines.join('\n'),
})
return {
content,
details: {
assetId: asset.id,
name: asset.name,
mediaType: asset.mediaType,
keys: sourceKeys,
results,
sourceKeys,
},
}
}
// ----------------------------------------------------------------------
// Branch 3: Documents & PDF Assets
// ----------------------------------------------------------------------
if (
proxyType === 'pdf' ||
mediaType === 'application/pdf' ||
isPlainTextAsset(mediaType, filename)
) {
if (!docConfig) {
throw new Error(
'docConfig with mode ("pages" or "text") is required when inspecting document/PDF assets.',
)
}
if (docConfig.mode === 'text') {
if (annotationId !== null) {
throw new Error(
'annotationId cannot be used with docConfig mode "text" because visual markups can only be rendered on visual pages. Use docConfig: { mode: "pages", startPage: ..., endPage: ... } instead.',
)
}
if (docConfig.startPage !== null || docConfig.endPage !== null) {
throw new Error(
'startPage and endPage must be null when docConfig mode is "text". To inspect specific page numbers, use mode: "pages".',
)
}
if (!isPlainTextAsset(mediaType, filename)) {
throw new Error(
`Asset "${asset.name}" is a binary PDF/document and cannot be read as raw text. Please call read_asset with docConfig: { mode: "pages", startPage: 1, endPage: ... } to view its visual pages.`,
)
}
const mediaKey = asset.storageKey?.key
if (!mediaKey) {
throw new Error(`No media content found for asset ${asset.id}.`)
}
const docMimeType =
asset.mediaType || getFileMimeType(null, asset.name || mediaKey, 'text/plain')
const downloadUrl = await s3Service.presign(
bucket,
mediaKey,
'GET',
true,
asset.name || undefined,
)
if (s3KeyOnly) {
return {
content: [
{
type: 'text',
text: `Document "${asset.name}" (ID: ${asset.id}, MIME: ${docMimeType}, S3 Key: "${mediaKey}")`,
},
],
details: {
assetId: asset.id,
name: asset.name,
mediaType: asset.mediaType,
key: mediaKey,
downloadUrl,
sourceKeys: [mediaKey],
size: 0,
isTruncated: false,
},
}
}
const { buffer } = await s3Service.getObject(bucket, mediaKey)
const rawText = buffer.toString('utf-8')
let text = rawText
let isTruncated = false
const lines = text.split('\n')
if (lines.length > MAX_TEXT_LINES) {
text = lines.slice(0, MAX_TEXT_LINES).join('\n')
isTruncated = true
}
if (Buffer.byteLength(text, 'utf-8') > MAX_TEXT_BYTES) {
text = Buffer.from(text, 'utf-8').subarray(0, MAX_TEXT_BYTES).toString('utf-8')
isTruncated = true
}
if (isTruncated) {
text += `\n\n[Content truncated to ${MAX_TEXT_LINES} lines / ${MAX_TEXT_BYTES / 1024}KB limit. Use download_asset to process the full file.]`
}
return {
content: [
{
type: 'text',
text: `Document "${asset.name}" (ID: ${asset.id}, MIME: ${docMimeType}, S3 Key: "${mediaKey}"):\n\n${text}`,
},
],
details: {
assetId: asset.id,
name: asset.name,
mediaType: asset.mediaType,
key: mediaKey,
downloadUrl,
sourceKeys: [mediaKey],
size: buffer.length,
isTruncated,
},
}
}
// docConfig.mode === 'pages'
if (docConfig.startPage === null || docConfig.endPage === null) {
throw new Error('startPage and endPage are required when docConfig mode is "pages".')
}
const start = docConfig.startPage
const end = docConfig.endPage
if (start < 1) {
throw new Error(`Invalid page range: start page (${start}) must be at least 1.`)
}
if (start > end) {
throw new Error(
`Invalid page range: start page (${start}) must be less than or equal to end page (${end}).`,
)
}
const MAX_PAGE_RANGE = 20
const requestedPages = end - start + 1
if (requestedPages > MAX_PAGE_RANGE) {
throw new Error(
`Page range (${requestedPages} pages requested) exceeds the maximum limit of ${MAX_PAGE_RANGE} pages per request.`,
)
}
const { annotations, timestamp: commentTimestamp } = await resolveAnnotationsById(
asset.id,
annotationId,
)
const task = await prisma.workflowTask.create({
data: {
assetId: asset.id,
projectId: asset.projectId || 'none',
type: WorkflowTaskType.transcode_pdf_pages,
status: WorkflowTaskStatus.pending,
payload: {
projectId: asset.projectId || 'none',
pdfPages: {
start,
end,
commentTimestamp,
annotations,
},
},
},
})
const completedTask = await workflowService.executeWait(task)
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- transcode pdf pages output
const output = completedTask.output as any
const pages = output?.pages as Array<{ key: string; page: number }> | undefined
if (!pages || pages.length === 0) {
throw new Error('No PDF page images were generated.')
}
const content: Array<TextContent | ImageContent> = []
const sourceKeys: string[] = []
const results: Array<{ key: string; page: number; downloadUrl: string }> = []
const textLines: string[] = [
`Rendered ${pages.length} page(s) of "${asset.name}" (Pages ${start} to ${end}):`,
]
for (const pageItem of pages) {
const downloadUrl = await s3Service.presign(
bucket,
pageItem.key,
'GET',
true,
asset.name || undefined,
)
sourceKeys.push(pageItem.key)
results.push({
key: pageItem.key,
page: pageItem.page,
downloadUrl,
})
if (s3KeyOnly) {
const mimeType = getFileMimeType(null, pageItem.key, 'image/webp')
textLines.push(`- Page ${pageItem.page} (S3 Key: "${pageItem.key}", MIME: ${mimeType})`)
} else {
const { buffer, contentType } = await s3Service.getObject(bucket, pageItem.key)
const mimeType =
contentType && contentType !== 'application/octet-stream'
? contentType
: getFileMimeType(buffer, pageItem.key, 'image/webp')
textLines.push(`- Page ${pageItem.page} (S3 Key: "${pageItem.key}", MIME: ${mimeType})`)
content.push({
type: 'image',
data: buffer.toString('base64'),
mimeType,
})
}
}
content.unshift({
type: 'text',
text: textLines.join('\n'),
})
return {
content,
details: {
assetId: asset.id,
name: asset.name,
mediaType: asset.mediaType,
keys: sourceKeys,
results,
sourceKeys,
},
}
}
throw new Error(
`Unsupported asset type for read_asset. Asset ${asset.id} has media type "${mediaType || 'unknown'}". Use download_asset to download this file to disk.`,
)
},
}
}
@@ -1,7 +1,12 @@
import { prisma } from '@shumai/db'
import type { ShumaiMessageContext } from '@shumai/dtos'
import { HTTPException } from 'hono/http-exception'
import { degreeToSecond } from '@shumai/core/src/utils/model3d'
/**
* Resolve the visual markup referenced by an annotation ID: a comment (comments mode) or an
* agent session entry (1-on-1 chat mode). The annotation must belong to `assetId`.
*/
export async function resolveAnnotationsById(
assetId: string,
annotationId?: string | null,
@@ -17,9 +22,9 @@ export async function resolveAnnotationsById(
})
if (comment) {
if (comment.assetId !== assetId) {
throw new Error(
`Annotation "${annotationId}" belongs to asset "${comment.assetId}", not target asset "${assetId}".`,
)
throw new HTTPException(400, {
message: `Annotation "${annotationId}" belongs to asset "${comment.assetId}", not target asset "${assetId}".`,
})
}
const list = comment.annotation
const annotations =
@@ -37,9 +42,9 @@ export async function resolveAnnotationsById(
ShumaiMessageContext | undefined
const entryAssetId = entry.assetId || details?.currentAsset?.id
if (entryAssetId && entryAssetId !== assetId) {
throw new Error(
`Annotation "${annotationId}" belongs to asset "${entryAssetId}", not target asset "${assetId}".`,
)
throw new HTTPException(400, {
message: `Annotation "${annotationId}" belongs to asset "${entryAssetId}", not target asset "${assetId}".`,
})
}
const list = details?.annotations
const annotations =
@@ -55,5 +60,5 @@ export async function resolveAnnotationsById(
return { annotations, timestamp }
}
throw new Error(`Annotation with ID "${annotationId}" not found.`)
throw new HTTPException(404, { message: `Annotation with ID "${annotationId}" not found.` })
}
@@ -0,0 +1,601 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import {
AssetType,
prisma,
WorkflowTaskStatus,
WorkflowTaskType,
type Prisma,
type User,
} from '@shumai/db'
import { setupTestDbHooks } from '@shumai/db/test'
import { workflowService } from '@shumai/workflow-core'
import { projectService } from '@shumai/core/src/project/project'
import { s3Service } from '@shumai/core/src/s3/s3'
import { getAssetRenderOp, readAssetOp, runOperation } from '@shumai/core/src/operations'
import type { OperationContext } from '@shumai/core/src/operations'
import { assetReadService, READ_URL_EXPIRES_SECONDS, toAssetReadOutput } from './asset-read'
describe('AssetReadService', () => {
setupTestDbHooks()
let owner: User
let reviewer: User
let outsider: User
let teamId: string
let projectId: string
let rootFolderId: string
const user = (u: User) => ({ kind: 'user' as const, userId: u.id })
const now = { waitBudgetMs: 0 }
beforeEach(async () => {
owner = await prisma.user.create({
data: { name: 'Owner', email: 'read-owner@test.com', password: 'pw' },
})
reviewer = await prisma.user.create({
data: { name: 'Reviewer', email: 'read-reviewer@test.com', password: 'pw' },
})
outsider = await prisma.user.create({
data: { name: 'Outsider', email: 'read-outsider@test.com', password: 'pw' },
})
const team = await prisma.team.create({ data: { name: 'Read Team' } })
teamId = team.id
await prisma.teamMember.create({ data: { teamId, userId: owner.id, role: 'owner' } })
await prisma.teamMember.create({ data: { teamId, userId: reviewer.id, role: 'reviewer' } })
const project = await projectService.createProject(owner, { teamId, name: 'Read Project' })
projectId = project.id
rootFolderId = project.rootFolder!
vi.spyOn(s3Service, 'presign').mockImplementation(
async (_bucket, key) => `https://signed.example/${key}`,
)
vi.spyOn(s3Service, 'getObject').mockImplementation(async (_bucket, key) => ({
buffer: Buffer.from(`content of ${key}`),
contentType: 'text/plain',
}))
})
afterEach(() => {
vi.restoreAllMocks()
})
async function createFile(
name: string,
data: Partial<Prisma.AssetUncheckedCreateInput> & { key?: string } = {},
) {
const { key, ...rest } = data
const storageKey = key ? await prisma.storageKey.create({ data: { key } }) : null
return prisma.asset.create({
data: {
name,
type: AssetType.file,
status: 'uploaded',
projectId,
parentId: rootFolderId,
storageKeyId: storageKey?.id,
...rest,
},
})
}
const imageMedia = {
proxyType: 'image',
imageTranscodes: [{ key: 'proxy/photo.webp' }],
} as unknown as PrismaJson.MediaInfo
async function createVideo() {
return createFile('clip.mp4', {
mediaType: 'video/mp4',
media: { proxyType: 'video' } as unknown as PrismaJson.MediaInfo,
key: 'raw/clip.mp4',
})
}
describe('access', () => {
it('lets a reviewer read an image without creating a render task', async () => {
const image = await createFile('photo.png', { mediaType: 'image/png', media: imageMedia })
const outcome = await assetReadService.read(user(reviewer), { assetId: image.id }, now)
expect(outcome).toMatchObject({
status: 'ready',
kind: 'image',
asset: { id: image.id, name: 'photo.png' },
result: {
summary: `Image asset "photo.png" (ID: ${image.id}):`,
images: [{ key: 'proxy/photo.webp', label: 'Image', mimeType: 'image/webp' }],
},
})
expect(await prisma.workflowTask.count({ where: { assetId: image.id } })).toBe(0)
})
it('rejects users without Read access before creating anything', async () => {
const video = await createVideo()
await expect(
assetReadService.read(
user(outsider),
{ assetId: video.id, videoConfig: { start: 0, end: 1, count: 1 } },
now,
),
).rejects.toMatchObject({ status: 403 })
expect(await prisma.workflowTask.count({ where: { assetId: video.id } })).toBe(0)
})
it('confines the autofill agent to its team and target asset', async () => {
const image = await createFile('photo.png', { mediaType: 'image/png', media: imageMedia })
const other = await createFile('other.png', { mediaType: 'image/png', media: imageMedia })
const ok = await assetReadService.read(
{ kind: 'autofill', teamId, targetAssetId: image.id },
{ assetId: image.id },
now,
)
expect(ok.status).toBe('ready')
await expect(
assetReadService.read(
{ kind: 'autofill', teamId, targetAssetId: image.id },
{ assetId: other.id },
now,
),
).rejects.toThrow(`autofill agent can only read target asset ${image.id}`)
await expect(
assetReadService.read(
{ kind: 'autofill', teamId: 'another-team', targetAssetId: image.id },
{ assetId: image.id },
now,
),
).rejects.toThrow(`does not belong to team another-team`)
})
})
it('reads the latest version of a version stack', async () => {
const stack = await prisma.asset.create({
data: {
name: 'diagram',
type: AssetType.version_stack,
status: 'uploaded',
projectId,
parentId: rootFolderId,
},
})
await createFile('diagram_v1.png', {
parentId: stack.id,
sortIndex: 'b',
mediaType: 'image/png',
key: 'raw/v1.png',
})
const v2 = await createFile('diagram_v2.png', {
parentId: stack.id,
sortIndex: 'a',
mediaType: 'image/png',
key: 'raw/v2.png',
})
const outcome = await assetReadService.read(user(owner), { assetId: stack.id }, now)
expect(outcome.asset.id).toBe(v2.id)
expect(outcome.status === 'ready' && outcome.result.images[0].key).toBe('raw/v2.png')
})
describe('reader selection and options', () => {
it.each([
['model.glb', 'model/gltf-binary', 'model3d'],
['photo.psd', 'image/vnd.adobe.photoshop', 'image'],
['clip.mov', 'video/quicktime', 'video'],
['doc.pdf', 'application/pdf', 'document'],
['notes.md', 'text/markdown', 'document'],
['script.ts', null, 'document'],
])('reads %s with the %s reader', async (name, mediaType, kind) => {
const asset = await createFile(name, { mediaType, key: `raw/${name}` })
// An empty-object config of the wrong type is ignored; the error names the right reader.
const error = await assetReadService
.read(user(owner), { assetId: asset.id, imageConfig: {}, config3d: { degree: 7 } }, now)
.catch((err: unknown) => err)
if (kind === 'model3d') {
expect(error).toMatchObject({
message: expect.stringContaining('Invalid turntable degree'),
})
} else {
expect(error).toMatchObject({
status: 400,
message: expect.stringContaining(`Asset ${asset.id} is a`),
})
expect((error as Error).message).toContain('do not pass config3d')
}
})
it('rejects unsupported files', async () => {
const zip = await createFile('bundle.zip', { mediaType: 'application/zip' })
await expect(assetReadService.read(user(owner), { assetId: zip.id }, now)).rejects.toThrow(
'Unsupported asset type for read_asset',
)
})
it('reads text content and reports truncation', async () => {
const doc = await createFile('notes.md', { mediaType: 'text/markdown', key: 'raw/notes.md' })
const outcome = await assetReadService.read(
user(owner),
{ assetId: doc.id, docConfig: { mode: 'text' } },
now,
)
expect(outcome).toMatchObject({
status: 'ready',
kind: 'document',
result: {
images: [],
text: {
content: 'content of raw/notes.md',
truncated: false,
size: 'content of raw/notes.md'.length,
key: 'raw/notes.md',
mimeType: 'text/markdown',
},
},
})
})
it('skips reading text when includeContent is false', async () => {
const doc = await createFile('notes.md', { mediaType: 'text/markdown', key: 'raw/notes.md' })
const outcome = await assetReadService.read(
user(owner),
{ assetId: doc.id, docConfig: { mode: 'text' }, includeContent: false },
now,
)
expect(s3Service.getObject).not.toHaveBeenCalled()
expect(outcome.status === 'ready' && outcome.result.text?.content).toBe('')
})
})
describe('render tasks', () => {
it('stores the read request on the task and returns pending when the budget runs out', async () => {
const video = await createVideo()
const outcome = await assetReadService.read(
user(reviewer),
{ assetId: video.id, videoConfig: { start: 0, end: 4, count: 2 } },
now,
)
expect(outcome).toMatchObject({ status: 'pending', kind: 'video', retryAfterSeconds: 5 })
const taskId = outcome.status === 'pending' ? outcome.taskId : ''
const task = await prisma.workflowTask.findUniqueOrThrow({ where: { id: taskId } })
expect(task).toMatchObject({
assetId: video.id,
projectId,
type: WorkflowTaskType.transcode_screenshot,
status: WorkflowTaskStatus.pending,
})
expect(task.payload).toEqual({
projectId,
screenshot: { start: 0, end: 4, count: 2, commentTimestamp: null, annotations: null },
assetRead: {
reader: 'video',
options: { start: 0, end: 4, count: 2 },
requestedBy: reviewer.id,
},
})
})
it('overlays annotations from a comment on the image', async () => {
const image = await createFile('photo.png', { mediaType: 'image/png', media: imageMedia })
const annotation = [
{
type: 'box',
color: '#ff0000',
points: [
[0, 0],
[10, 10],
],
},
] as unknown as PrismaJson.AnnotationList
const comment = await prisma.assetComment.create({
data: { assetId: image.id, creatorId: owner.id, message: 'here', annotation },
})
const outcome = await assetReadService.read(
user(owner),
{ assetId: image.id, annotationId: comment.id },
now,
)
expect(outcome).toMatchObject({ status: 'pending', kind: 'image', retryAfterSeconds: 3 })
const task = await prisma.workflowTask.findFirstOrThrow({ where: { assetId: image.id } })
expect(task.type).toBe(WorkflowTaskType.transcode_image_annotation)
expect(task.payload?.imageAnnotation).toEqual({ annotations: annotation })
expect(task.payload?.assetRead).toMatchObject({ reader: 'image', annotationId: comment.id })
})
it('rejects annotations from another asset', async () => {
const image = await createFile('photo.png', { mediaType: 'image/png', media: imageMedia })
const other = await createFile('other.png', { mediaType: 'image/png', media: imageMedia })
const comment = await prisma.assetComment.create({
data: { assetId: other.id, creatorId: owner.id, message: 'elsewhere' },
})
await expect(
assetReadService.read(user(owner), { assetId: image.id, annotationId: comment.id }, now),
).rejects.toMatchObject({ status: 400 })
})
it('stores reader context for 3D angles and collects it later', async () => {
const model = await createFile('robot.glb', {
mediaType: 'model/gltf-binary',
media: {
proxyType: '3d',
videoTranscodes: [{ key: 'proxy/turntable.mp4' }],
} as unknown as PrismaJson.MediaInfo,
})
const pending = await assetReadService.read(
user(owner),
{ assetId: model.id, config3d: { degree: 90 } },
now,
)
const taskId = pending.status === 'pending' ? pending.taskId : ''
const task = await prisma.workflowTask.findUniqueOrThrow({ where: { id: taskId } })
expect(task.payload?.assetRead?.context).toEqual({ degree: 90, hasMarkup: false })
await prisma.workflowTask.update({
where: { id: taskId },
data: {
status: WorkflowTaskStatus.completed,
output: { screenshots: [{ key: 'shots/90.webp', timestamp: 1 }] },
},
})
const ready = await assetReadService.getRender(owner.id, taskId, now)
expect(ready).toMatchObject({
status: 'ready',
kind: 'model3d',
result: {
summary: `3D asset "robot.glb" (ID: ${model.id}) view at 90°:`,
images: [{ key: 'shots/90.webp', label: 'View at 90°' }],
},
})
})
})
describe('getRender', () => {
async function pendingVideoRead(by: User) {
const video = await createVideo()
const outcome = await assetReadService.read(
user(by),
{ assetId: video.id, videoConfig: { start: 0, end: 4, count: 2 } },
now,
)
if (outcome.status !== 'pending') throw new Error('expected pending')
return { video, taskId: outcome.taskId }
}
it('stays pending until the task completes, then returns the frames', async () => {
const { taskId } = await pendingVideoRead(reviewer)
expect(await assetReadService.getRender(reviewer.id, taskId, now)).toMatchObject({
status: 'pending',
taskId,
})
await prisma.workflowTask.update({
where: { id: taskId },
data: {
status: WorkflowTaskStatus.completed,
output: {
screenshots: [
{ key: 'shots/0.webp', timestamp: 0 },
{ key: 'shots/4.webp', timestamp: 4 },
],
},
},
})
const ready = await assetReadService.getRender(reviewer.id, taskId, now)
expect(ready).toMatchObject({
status: 'ready',
result: {
summary: 'Extracted 2 frame(s) from "clip.mp4" (0s - 4s):',
images: [
{ key: 'shots/0.webp', label: 'Frame at 0s', mimeType: 'image/webp' },
{ key: 'shots/4.webp', label: 'Frame at 4s', mimeType: 'image/webp' },
],
},
})
})
it('surfaces a failed render', async () => {
const { taskId } = await pendingVideoRead(owner)
await prisma.workflowTask.update({
where: { id: taskId },
data: { status: WorkflowTaskStatus.failed, output: { error: 'ffmpeg exploded' } },
})
await expect(assetReadService.getRender(owner.id, taskId, now)).rejects.toThrow(
'ffmpeg exploded',
)
})
it('returns 404 to anyone but the requester', async () => {
const { taskId } = await pendingVideoRead(reviewer)
await expect(assetReadService.getRender(owner.id, taskId, now)).rejects.toMatchObject({
status: 404,
message: 'Render not found',
})
})
it('returns 404 once the requester loses access', async () => {
const { taskId } = await pendingVideoRead(reviewer)
await prisma.teamMember.deleteMany({ where: { userId: reviewer.id } })
await expect(assetReadService.getRender(reviewer.id, taskId, now)).rejects.toMatchObject({
status: 404,
})
})
it('returns 404 for unknown tasks and tasks that are not asset reads', async () => {
const video = await createVideo()
const other = await prisma.workflowTask.create({
data: {
assetId: video.id,
projectId,
type: WorkflowTaskType.transcode_video,
status: WorkflowTaskStatus.completed,
payload: { projectId },
},
})
await expect(assetReadService.getRender(owner.id, other.id, now)).rejects.toMatchObject({
status: 404,
})
await expect(assetReadService.getRender(owner.id, 'nope', now)).rejects.toMatchObject({
status: 404,
})
})
})
describe('toAssetReadOutput', () => {
it('returns short-lived links and only includes storage keys when asked', async () => {
const image = await createFile('photo.png', { mediaType: 'image/png', media: imageMedia })
const outcome = await assetReadService.read(user(owner), { assetId: image.id }, now)
const external = await toAssetReadOutput(outcome, { includeKeys: false })
expect(external.images).toEqual([
{ label: 'Image', mimeType: 'image/webp', url: 'https://signed.example/proxy/photo.webp' },
])
expect(s3Service.presign).toHaveBeenCalledWith(
expect.any(String),
'proxy/photo.webp',
'GET',
false,
undefined,
false,
READ_URL_EXPIRES_SECONDS,
)
const internal = await toAssetReadOutput(outcome, { includeKeys: true })
expect(internal.images[0].key).toBe('proxy/photo.webp')
})
it('hides the text storage key and link from external callers', async () => {
const doc = await createFile('notes.md', { mediaType: 'text/markdown', key: 'raw/notes.md' })
const outcome = await assetReadService.read(
user(owner),
{ assetId: doc.id, docConfig: { mode: 'text' } },
now,
)
const external = await toAssetReadOutput(outcome, { includeKeys: false })
expect(external.text).toEqual({
content: 'content of raw/notes.md',
truncated: false,
size: 23,
mimeType: 'text/markdown',
})
const internal = await toAssetReadOutput(outcome, { includeKeys: true })
expect(internal.text).toMatchObject({
key: 'raw/notes.md',
url: 'https://signed.example/raw/notes.md',
})
})
})
})
describe('read_asset and get_asset_render Operations', () => {
setupTestDbHooks()
let owner: User
let videoId: string
let imageId: string
const as = (via: OperationContext['via']): OperationContext => ({ userId: owner.id, via })
beforeEach(async () => {
owner = await prisma.user.create({
data: { name: 'Owner', email: 'read-op-owner@test.com', password: 'pw' },
})
const team = await prisma.team.create({ data: { name: 'Read Op Team' } })
await prisma.teamMember.create({ data: { teamId: team.id, userId: owner.id, role: 'owner' } })
const project = await projectService.createProject(owner, {
teamId: team.id,
name: 'Read Op Project',
})
const base = {
type: AssetType.file,
status: 'uploaded' as const,
projectId: project.id,
parentId: project.rootFolder!,
}
videoId = (
await prisma.asset.create({
data: {
...base,
name: 'clip.mp4',
mediaType: 'video/mp4',
media: { proxyType: 'video' } as unknown as PrismaJson.MediaInfo,
},
})
).id
imageId = (
await prisma.asset.create({
data: {
...base,
name: 'photo.png',
mediaType: 'image/png',
media: {
proxyType: 'image',
imageTranscodes: [{ key: 'proxy/photo.webp' }],
} as unknown as PrismaJson.MediaInfo,
},
})
).id
vi.spyOn(s3Service, 'presign').mockResolvedValue('https://signed.example/x')
})
afterEach(() => {
vi.restoreAllMocks()
})
it.each([
['agent', true],
['mcp', false],
['cli', false],
['api', false],
] as const)('via %s includes storage keys: %s', async (via, includesKey) => {
const output = await runOperation(readAssetOp, as(via), { assetId: imageId })
expect(output.status).toBe('ready')
expect(output.images[0].url).toBe('https://signed.example/x')
expect('key' in output.images[0]).toBe(includesKey)
})
it.each([
['agent', undefined, 60_000],
['mcp', undefined, 20_000],
['cli', undefined, 120_000],
['api', undefined, 20_000],
['cli', 5, 5_000],
['mcp', 600, 20_000],
['agent', 0, 0],
] as const)('via %s with maxWaitSeconds %s waits %sms', async (via, maxWaitSeconds, budget) => {
const waitFor = vi
.spyOn(workflowService, 'waitFor')
.mockImplementation(async (task) => ({ task, done: false }))
const output = await runOperation(readAssetOp, as(via), {
assetId: videoId,
videoConfig: { start: 0, end: 1, count: 1 },
maxWaitSeconds,
})
expect(waitFor).toHaveBeenCalledWith(expect.anything(), budget)
expect(output).toMatchObject({
status: 'pending',
summary: 'Rendering is still in progress for "clip.mp4".',
asset: { id: videoId, kind: 'video' },
images: [],
taskId: expect.any(String),
retryAfterSeconds: 5,
})
await runOperation(getAssetRenderOp, as(via), { taskId: output.taskId!, maxWaitSeconds })
expect(waitFor).toHaveBeenLastCalledWith(expect.anything(), budget)
})
it('rejects invalid input before touching the asset', async () => {
await expect(
runOperation(readAssetOp, as('mcp'), {
assetId: videoId,
videoConfig: { start: 0, end: 1 },
}),
).rejects.toMatchObject({ status: 400 })
expect(await prisma.workflowTask.count({ where: { assetId: videoId } })).toBe(0)
})
})
+342
View File
@@ -0,0 +1,342 @@
import {
readerOptionSchemas,
type AssetReadOutput,
type ReaderConfigKey,
type ReaderKind,
} from '@shumai/operations'
import { AssetType, prisma, WorkflowTaskStatus, type Prisma, type WorkflowTask } from '@shumai/db'
import { HTTPException } from 'hono/http-exception'
import { workflowService } from '@shumai/workflow-core'
import { authzService, Permission, ResourceType } from '@shumai/core/src/authz/authz'
import { s3Service } from '@shumai/core/src/s3/s3'
import { resolveAnnotationsById } from './annotations'
import { findReader, getReader, readers } from './registry'
import { badRequest, storageBucket, type ReadableAsset, type ReadResult } from './types'
/** Who is reading: a user (Read permission required) or the autofill system agent. */
export type ReadAccess =
{ kind: 'user'; userId: string } | { kind: 'autofill'; teamId?: string; targetAssetId?: string }
export type ReadAssetRequest = {
assetId: string
annotationId?: string
includeContent?: boolean
} & { [K in ReaderConfigKey]?: Record<string, unknown> }
export type ReadOutcome =
| { status: 'ready'; asset: ReadableAsset; kind: ReaderKind; result: ReadResult }
| {
status: 'pending'
asset: ReadableAsset
kind: ReaderKind
taskId: string
retryAfterSeconds: number
}
/** Lifetime of image links handed to callers outside the server. */
export const READ_URL_EXPIRES_SECONDS = 3600
const assetWithStorage = { storageKey: true } satisfies Prisma.AssetInclude
type AssetRecord = Prisma.AssetGetPayload<{ include: typeof assetWithStorage }>
function toReadable(asset: AssetRecord): ReadableAsset {
return {
id: asset.id,
name: asset.name,
projectId: asset.projectId,
mediaType: asset.mediaType,
mediaTypeLower: asset.mediaType?.toLowerCase() || '',
filenameLower: asset.name?.toLowerCase() || '',
storageKey: asset.storageKey?.key ?? null,
media: asset.media as PrismaJson.MediaInfo | null,
}
}
function hasContent(value: unknown): boolean {
return (
value !== null &&
value !== undefined &&
typeof value === 'object' &&
Object.values(value).some((v) => v !== null && v !== undefined)
)
}
function notFoundRender(): HTTPException {
return new HTTPException(404, { message: 'Render not found' })
}
/**
* Shared pipeline behind `read_asset` and `get_asset_render`: access checks, version stacks,
* reader selection, option validation, annotations, render tasks, and waiting. Readers in
* `./readers` only contain the type-specific logic.
*/
export class AssetReadService {
async read(
access: ReadAccess,
request: ReadAssetRequest,
opts: { waitBudgetMs: number },
): Promise<ReadOutcome> {
await this.checkAccess(access, request.assetId)
const asset = await this.loadLatestVersion(request.assetId)
const reader = findReader(asset)
if (!reader) {
throw badRequest(
`Unsupported asset type for read_asset. Asset ${asset.id} has media type "${asset.mediaTypeLower || 'unknown'}". Use download_asset to download this file to disk.`,
)
}
// Reject configs meant for another type (empty objects like `imageConfig: {}` are ignored).
const configKey = readerOptionSchemas[reader.kind].key
for (const other of readers) {
const otherKey = readerOptionSchemas[other.kind].key
if (otherKey !== configKey && hasContent(request[otherKey])) {
throw badRequest(
`Asset ${asset.id} is a ${reader.label}. ${reader.configHint}; do not pass ${otherKey}.`,
)
}
}
const options = request[configKey]
reader.validate(asset, options as never, { hasAnnotation: Boolean(request.annotationId) })
const annotation = request.annotationId
? {
annotationId: request.annotationId,
...(await resolveAnnotationsById(asset.id, request.annotationId)),
}
: null
const plan = await reader.plan(asset, options as never, {
annotation,
includeContent: request.includeContent !== false,
})
if (plan.status === 'ready') {
return { status: 'ready', asset, kind: reader.kind, result: plan.result }
}
const projectId = asset.projectId || 'none'
const task = await prisma.workflowTask.create({
data: {
assetId: asset.id,
projectId,
type: plan.task.type,
status: WorkflowTaskStatus.pending,
payload: {
projectId,
...plan.task.spec,
assetRead: {
reader: reader.kind,
options: options ?? undefined,
annotationId: request.annotationId,
context: plan.context,
requestedBy: access.kind === 'user' ? access.userId : undefined,
},
},
},
})
return this.finish(task, asset, opts.waitBudgetMs)
}
/** Completes a read that returned `pending`. Only the requesting user can fetch it. */
async getRender(
userId: string,
taskId: string,
opts: { waitBudgetMs: number },
): Promise<ReadOutcome> {
const task = await prisma.workflowTask.findUnique({ where: { id: taskId } })
const request = task?.payload?.assetRead
if (!task || !request || !task.assetId || !userId || request.requestedBy !== userId) {
throw notFoundRender()
}
try {
await authzService.hasPermission({
user: { id: userId },
permission: Permission.Read,
type: ResourceType.Asset,
id: task.assetId,
})
} catch {
// Don't reveal that the task exists to someone who can no longer see the asset.
throw notFoundRender()
}
const record = await prisma.asset.findUnique({
where: { id: task.assetId },
include: assetWithStorage,
})
if (!record) throw notFoundRender()
return this.finish(task, toReadable(record), opts.waitBudgetMs)
}
private async finish(
task: WorkflowTask,
asset: ReadableAsset,
waitBudgetMs: number,
): Promise<ReadOutcome> {
const request = task.payload!.assetRead!
const reader = getReader(request.reader)
const { task: latest, done } = await workflowService.waitFor(task, waitBudgetMs)
if (!done) {
return {
status: 'pending',
asset,
kind: reader.kind,
taskId: task.id,
retryAfterSeconds: reader.kind === 'video' || reader.kind === 'document' ? 5 : 3,
}
}
if (!reader.collect) {
throw new Error(`Reader "${reader.kind}" cannot collect render results`)
}
const result = reader.collect(
asset,
request.options as never,
latest.output ?? {},
request.context ?? {},
)
return { status: 'ready', asset, kind: reader.kind, result }
}
private async checkAccess(access: ReadAccess, assetId: string): Promise<void> {
if (access.kind === 'user') {
if (!access.userId) {
throw new HTTPException(401, { message: 'User ID is required for authorization.' })
}
await authzService.hasPermission({
user: { id: access.userId },
permission: Permission.Read,
type: ResourceType.Asset,
id: assetId,
})
return
}
// Autofill system agent: no user; confined to its team and target asset.
if (!access.targetAssetId) {
throw new HTTPException(403, {
message: 'Target asset ID is required for autofill agent authorization.',
})
}
if (!access.teamId) {
throw new HTTPException(403, {
message: 'Team ID is required for autofill agent authorization.',
})
}
const asset = await prisma.asset.findUnique({
where: { id: assetId },
include: { project: true, teamRootFolder: true },
})
if (!asset) {
throw new HTTPException(404, { message: `Asset with ID ${assetId} not found.` })
}
const assetTeamId = asset.project?.teamId || asset.teamRootFolder?.id
if (!assetTeamId || assetTeamId !== access.teamId) {
throw new HTTPException(403, {
message: `Access denied: asset ${assetId} does not belong to team ${access.teamId}.`,
})
}
if (asset.id !== access.targetAssetId && asset.parentId !== access.targetAssetId) {
throw new HTTPException(403, {
message: `Access denied: autofill agent can only read target asset ${access.targetAssetId}.`,
})
}
}
private async loadLatestVersion(assetId: string): Promise<ReadableAsset> {
let asset = await prisma.asset.findUnique({
where: { id: assetId },
include: assetWithStorage,
})
if (!asset) {
throw new HTTPException(404, { message: `Asset with ID ${assetId} not found.` })
}
if (asset.type === AssetType.version_stack) {
const latest = await prisma.asset.findFirst({
where: { parentId: asset.id, isDeleted: false },
orderBy: { sortIndex: 'asc' },
include: assetWithStorage,
})
if (latest) asset = latest
}
return toReadable(asset)
}
}
export const assetReadService = new AssetReadService()
/**
* Converts a read outcome into the Operation output. Storage keys are included only for
* Shumai's built-in agent; every image gets a short-lived link.
*/
export async function toAssetReadOutput(
outcome: ReadOutcome,
opts: { includeKeys: boolean },
): Promise<AssetReadOutput> {
const asset = {
id: outcome.asset.id,
name: outcome.asset.name,
kind: outcome.kind,
mediaType: outcome.asset.mediaType,
}
if (outcome.status === 'pending') {
return {
status: 'pending',
summary: `Rendering is still in progress for "${outcome.asset.name}".`,
asset,
details: [],
images: [],
taskId: outcome.taskId,
retryAfterSeconds: outcome.retryAfterSeconds,
}
}
const bucket = storageBucket()
const { result } = outcome
const images = await Promise.all(
result.images.map(async (image) => ({
label: image.label,
mimeType: image.mimeType,
url: await s3Service.presign(
bucket,
image.key,
'GET',
false,
undefined,
false,
READ_URL_EXPIRES_SECONDS,
),
...(opts.includeKeys ? { key: image.key } : {}),
})),
)
const text = result.text
? {
content: result.text.content,
truncated: result.text.truncated,
size: result.text.size,
mimeType: result.text.mimeType,
...(opts.includeKeys
? {
key: result.text.key,
url: await s3Service.presign(
bucket,
result.text.key,
'GET',
true,
outcome.asset.name || undefined,
false,
READ_URL_EXPIRES_SECONDS,
),
}
: {}),
}
: undefined
return {
status: 'ready',
summary: result.summary,
asset,
details: result.details,
images,
...(text ? { text } : {}),
}
}
+13
View File
@@ -0,0 +1,13 @@
export {
assetReadService,
AssetReadService,
READ_URL_EXPIRES_SECONDS,
toAssetReadOutput,
type ReadAccess,
type ReadAssetRequest,
type ReadOutcome,
} from './asset-read'
export { resolveAnnotationsById } from './annotations'
export { findReader, getReader, readers } from './registry'
export { isPlainTextAsset, MAX_TEXT_BYTES, MAX_TEXT_LINES } from './readers/document'
export type { AssetReader, ReadableAsset, ReadImage, ReadPlan, ReadResult } from './types'
@@ -0,0 +1,188 @@
import { MAX_RENDER_ITEMS } from '@shumai/operations'
import { WorkflowTaskType } from '@shumai/db'
import { getFileMimeType } from '@shumai/core/src/utils/file-mime'
import { s3Service } from '@shumai/core/src/s3/s3'
import { badRequest, storageBucket, type AssetReader } from '../types'
export const MAX_TEXT_BYTES = 50 * 1024
export const MAX_TEXT_LINES = 1000
const TEXT_MEDIA_TYPES = new Set([
'application/json',
'application/javascript',
'application/typescript',
'application/xml',
'application/x-yaml',
])
const TEXT_EXTENSIONS = [
'.txt',
'.md',
'.markdown',
'.json',
'.csv',
'.tsv',
'.js',
'.ts',
'.tsx',
'.jsx',
'.html',
'.htm',
'.css',
'.scss',
'.yaml',
'.yml',
'.xml',
'.py',
'.rb',
'.go',
'.rs',
'.c',
'.cpp',
'.h',
'.hpp',
'.java',
'.sh',
'.bash',
'.zsh',
'.sql',
]
export function isPlainTextAsset(mediaType: string, filename: string): boolean {
if (mediaType.startsWith('text/') || TEXT_MEDIA_TYPES.has(mediaType)) {
return true
}
return TEXT_EXTENSIONS.some((ext) => filename.endsWith(ext))
}
/** Truncate to the line and byte limits; returns the kept text and whether anything was cut. */
export function truncateText(raw: string): { content: string; truncated: boolean } {
let content = raw
let truncated = false
const lines = content.split('\n')
if (lines.length > MAX_TEXT_LINES) {
content = lines.slice(0, MAX_TEXT_LINES).join('\n')
truncated = true
}
if (Buffer.byteLength(content, 'utf-8') > MAX_TEXT_BYTES) {
content = Buffer.from(content, 'utf-8').subarray(0, MAX_TEXT_BYTES).toString('utf-8')
truncated = true
}
return { content, truncated }
}
/** PDFs and text files: rendered page images ("pages") or raw text content ("text"). */
export const documentReader: AssetReader<'document'> = {
kind: 'document',
label: 'document',
configHint: "Pass 'docConfig: { mode, startPage, endPage }'",
matches: (asset) =>
asset.media?.proxyType === 'pdf' ||
asset.mediaTypeLower === 'application/pdf' ||
isPlainTextAsset(asset.mediaTypeLower, asset.filenameLower),
validate(asset, options, { hasAnnotation }) {
if (!options) {
throw badRequest(
'docConfig with mode ("pages" or "text") is required when inspecting document/PDF assets.',
)
}
const startPage = options.startPage ?? null
const endPage = options.endPage ?? null
if (options.mode === 'text') {
if (hasAnnotation) {
throw badRequest(
'annotationId cannot be used with docConfig mode "text" because visual markups can only be rendered on visual pages. Use docConfig: { mode: "pages", startPage: ..., endPage: ... } instead.',
)
}
if (startPage !== null || endPage !== null) {
throw badRequest(
'startPage and endPage must be null when docConfig mode is "text". To inspect specific page numbers, use mode: "pages".',
)
}
if (!isPlainTextAsset(asset.mediaTypeLower, asset.filenameLower)) {
throw badRequest(
`Asset "${asset.name}" is a binary PDF/document and cannot be read as raw text. Please call read_asset with docConfig: { mode: "pages", startPage: 1, endPage: ... } to view its visual pages.`,
)
}
return
}
if (startPage === null || endPage === null) {
throw badRequest('startPage and endPage are required when docConfig mode is "pages".')
}
if (startPage < 1) {
throw badRequest(`Invalid page range: start page (${startPage}) must be at least 1.`)
}
if (startPage > endPage) {
throw badRequest(
`Invalid page range: start page (${startPage}) must be less than or equal to end page (${endPage}).`,
)
}
const requested = endPage - startPage + 1
if (requested > MAX_RENDER_ITEMS) {
throw badRequest(
`Page range (${requested} pages requested) exceeds the maximum limit of ${MAX_RENDER_ITEMS} pages per request.`,
)
}
},
async plan(asset, options, { annotation, includeContent }) {
// validate() guarantees options are present and consistent
const opts = options!
if (opts.mode === 'text') {
const key = asset.storageKey
if (!key) {
throw new Error(`No media content found for asset ${asset.id}.`)
}
const mimeType = asset.mediaType || getFileMimeType(null, asset.name || key, 'text/plain')
let text = { content: '', truncated: false, size: 0, key, mimeType }
if (includeContent) {
const { buffer } = await s3Service.getObject(storageBucket(), key)
text = { ...truncateText(buffer.toString('utf-8')), size: buffer.length, key, mimeType }
}
return {
status: 'ready',
result: {
summary: `Document "${asset.name}" (ID: ${asset.id}):`,
details: [],
images: [],
text,
},
}
}
return {
status: 'render',
task: {
type: WorkflowTaskType.transcode_pdf_pages,
spec: {
pdfPages: {
start: opts.startPage!,
end: opts.endPage!,
commentTimestamp: annotation?.timestamp ?? null,
annotations: annotation?.annotations ?? null,
},
},
},
}
},
collect(asset, options, output) {
const pages = output.pages as Array<{ key: string; page: number }> | undefined
if (!pages || pages.length === 0) {
throw new Error('No PDF page images were generated.')
}
return {
summary: `Rendered ${pages.length} page(s) of "${asset.name}" (Pages ${options!.startPage} to ${options!.endPage}):`,
details: [],
images: pages.map((page) => ({
key: page.key,
label: `Page ${page.page}`,
mimeType: getFileMimeType(null, page.key, 'image/webp'),
})),
}
},
}
@@ -0,0 +1,61 @@
import { WorkflowTaskType } from '@shumai/db'
import { getFileMimeType } from '@shumai/core/src/utils/file-mime'
import type { AssetReader } from '../types'
/** Images (including PSD): the web proxy, or a copy with the annotation drawn on top. */
export const imageReader: AssetReader<'image'> = {
kind: 'image',
label: 'image',
configHint: "Pass 'imageConfig: {}' or no config",
matches: (asset) =>
asset.media?.proxyType === 'image' ||
asset.mediaTypeLower.startsWith('image/') ||
asset.filenameLower.endsWith('.psd'),
validate() {},
async plan(asset, _options, { annotation }) {
const key = asset.media?.imageTranscodes?.[0]?.key || asset.storageKey
if (!key) {
throw new Error('No media content found for this asset.')
}
if (annotation?.annotations && annotation.annotations.length > 0) {
return {
status: 'render',
task: {
type: WorkflowTaskType.transcode_image_annotation,
spec: { imageAnnotation: { annotations: annotation.annotations } },
},
}
}
return {
status: 'ready',
result: {
summary: `Image asset "${asset.name}" (ID: ${asset.id}):`,
details: [],
images: [
{
key,
label: 'Image',
mimeType: getFileMimeType(null, key, asset.mediaType || 'image/png'),
},
],
},
}
},
collect(asset, _options, output) {
const key = typeof output.key === 'string' ? output.key : null
if (!key) {
throw new Error('Image annotation transcode workflow failed to output S3 key.')
}
return {
summary: `Image asset "${asset.name}" (ID: ${asset.id}) with visual markups overlaid:`,
details: [],
images: [{ key, label: 'Image with markups', mimeType: 'image/webp' }],
}
},
}
@@ -0,0 +1,112 @@
import { WorkflowTaskType } from '@shumai/db'
import { is3dModel } from '@shumai/core/src/utils/mime'
import {
degreeToSecond,
isValidTurntableDegree,
secondToDegree,
} from '@shumai/core/src/utils/model3d'
import { badRequest, type AssetReader, type ReadResult } from '../types'
const QUADRANT_LEGEND = [
'Orthogonal 4-view composite image (2x2 grid across 360° rotation):',
'- Top-Left quadrant: Front view (0°)',
'- Top-Right quadrant: Right side view (90°)',
'- Bottom-Left quadrant: Back view (180°)',
'- Bottom-Right quadrant: Left side view (270°)',
]
function angleSummary(name: string, id: string, degree: number, hasMarkup: boolean): string {
return hasMarkup
? `3D asset "${name}" (ID: ${id}) view at ${degree}° with visual markups overlaid:`
: `3D asset "${name}" (ID: ${id}) view at ${degree}°:`
}
/** 3D models: the 4-view composite, or one turntable angle rendered from the turntable video. */
export const model3dReader: AssetReader<'model3d'> = {
kind: 'model3d',
label: '3D model',
configHint: "To inspect specific angles pass 'config3d: { degree: ... }'",
matches: (asset) =>
asset.media?.proxyType === '3d' || is3dModel(asset.mediaTypeLower, asset.filenameLower),
validate(_asset, options) {
const degree = options?.degree
if (degree !== undefined && degree !== null && !isValidTurntableDegree(degree)) {
throw badRequest(
`Invalid turntable degree: ${degree}. Degree must be an integer multiple of 15 between 0 and 345 (e.g. 0, 15, 30, 45, ..., 345).`,
)
}
},
async plan(asset, options, { annotation }) {
const requestedDegree = options?.degree ?? null
const annotations = annotation?.annotations ?? null
const commentTimestamp = annotation?.timestamp ?? null
if (annotation && commentTimestamp !== null) {
const commentDegree = secondToDegree(commentTimestamp)
if (requestedDegree !== null && requestedDegree !== commentDegree) {
throw badRequest(
`Annotation "${annotation.annotationId}" was drawn at ${commentDegree}°, but degree ${requestedDegree}° was specified. Do not pass a conflicting degree with annotationId.`,
)
}
}
if (requestedDegree !== null || annotation) {
if (!asset.media?.videoTranscodes || asset.media.videoTranscodes.length === 0) {
throw new Error(
`Asset ${asset.id} is a 3D model, but its turntable video (turntable.mp4) is missing.`,
)
}
const second =
requestedDegree !== null ? degreeToSecond(requestedDegree) : (commentTimestamp ?? 0)
const degree = requestedDegree ?? secondToDegree(second)
return {
status: 'render',
task: {
type: WorkflowTaskType.transcode_screenshot,
spec: {
screenshot: {
start: second,
end: second,
count: 1,
commentTimestamp: second,
annotations: annotations ?? undefined,
},
},
},
context: { degree, hasMarkup: Boolean(annotations && annotations.length > 0) },
}
}
const key = asset.media?.imageTranscodes?.[0]?.key
if (!asset.media?.imageTranscodes || asset.media.imageTranscodes.length === 0) {
throw new Error(
`Asset ${asset.id} is a 3D model, but its 4-views image (views.webp) is missing.`,
)
}
if (!key) {
throw new Error(`Asset ${asset.id} is a 3D model, but its 4-views image key is missing.`)
}
const result: ReadResult = {
summary: `3D asset "${asset.name}" (ID: ${asset.id}):`,
details: QUADRANT_LEGEND,
images: [{ key, label: '4-view composite', mimeType: 'image/webp' }],
}
return { status: 'ready', result }
},
collect(asset, _options, output, context) {
const degree = Number(context.degree)
const screenshots = output.screenshots as Array<{ key: string }> | undefined
if (!screenshots || screenshots.length === 0) {
throw new Error(`Failed to extract 3D turntable frame for asset ${asset.id} at ${degree}°.`)
}
return {
summary: angleSummary(asset.name, asset.id, degree, context.hasMarkup === true),
details: [],
images: [{ key: screenshots[0].key, label: `View at ${degree}°`, mimeType: 'image/webp' }],
}
},
}
@@ -0,0 +1,223 @@
import { describe, expect, it } from 'vitest'
import { READER_KINDS, readerOptionSchemas } from '@shumai/operations'
import { findReader, getReader, readers } from '../registry'
import type { ReadableAsset } from '../types'
import { documentReader, isPlainTextAsset, MAX_TEXT_BYTES, truncateText } from './document'
import { imageReader } from './image'
import { model3dReader } from './model3d'
import { videoReader } from './video'
function asset(overrides: Partial<ReadableAsset> & { name: string }): ReadableAsset {
const mediaType = overrides.mediaType ?? null
return {
id: 'asset-1',
projectId: 'project-1',
storageKey: null,
media: null,
mediaType,
mediaTypeLower: mediaType?.toLowerCase() ?? '',
filenameLower: overrides.name.toLowerCase(),
...overrides,
}
}
const noAnnotation = { annotation: null, includeContent: true }
describe('reader registry', () => {
it('has exactly one reader per declared kind, each with a config field', () => {
expect(readers.map((r) => r.kind).sort()).toEqual([...READER_KINDS].sort())
for (const kind of READER_KINDS) {
expect(getReader(kind).kind).toBe(kind)
expect(readerOptionSchemas[kind].key).toMatch(/Config$|^config3d$/)
}
})
it.each([
[{ name: 'robot.glb', mediaType: 'model/gltf-binary' }, 'model3d'],
// 3D models also have image transcodes; the 3D reader must win.
[{ name: 'robot.png', mediaType: 'image/png', media: { proxyType: '3d' } }, 'model3d'],
[{ name: 'photo.JPG', mediaType: 'IMAGE/JPEG' }, 'image'],
[{ name: 'layers.psd', mediaType: 'application/octet-stream' }, 'image'],
[{ name: 'clip.webm', mediaType: 'video/webm' }, 'video'],
[{ name: 'slides.pdf', mediaType: 'application/pdf' }, 'document'],
[{ name: 'data.csv', mediaType: null }, 'document'],
[{ name: 'config.json', mediaType: 'application/json' }, 'document'],
] as const)('%o is read by %s', (overrides, kind) => {
expect(findReader(asset(overrides as Partial<ReadableAsset> & { name: string }))?.kind).toBe(
kind,
)
})
it('has no reader for archives', () => {
expect(findReader(asset({ name: 'bundle.zip', mediaType: 'application/zip' }))).toBeUndefined()
})
})
describe('imageReader', () => {
it('prefers the web proxy over the original', async () => {
const plan = await imageReader.plan(
asset({
name: 'photo.png',
mediaType: 'image/png',
storageKey: 'raw/photo.png',
media: { imageTranscodes: [{ key: 'proxy/photo.webp' }] } as PrismaJson.MediaInfo,
}),
undefined,
noAnnotation,
)
expect(plan).toMatchObject({
status: 'ready',
result: { images: [{ key: 'proxy/photo.webp', mimeType: 'image/webp' }] },
})
})
it('fails without any stored content', async () => {
await expect(
imageReader.plan(asset({ name: 'photo.png' }), undefined, noAnnotation),
).rejects.toThrow('No media content found')
})
it('requires the annotated key from the render', () => {
expect(() => imageReader.collect!(asset({ name: 'photo.png' }), undefined, {}, {})).toThrow(
'failed to output S3 key',
)
})
})
describe('videoReader', () => {
const video = asset({ name: 'clip.mp4', mediaType: 'video/mp4' })
it.each([
[{ start: -1, end: 2, count: 1 }, 'must be non-negative'],
[{ start: 3, end: 2, count: 1 }, 'must be less than or equal to end'],
[{ start: 0, end: 2, count: 0 }, 'must be at least 1'],
[{ start: 0, end: 2, count: 21 }, 'exceeds the maximum of 20 frames'],
])('rejects %o', (options, message) => {
expect(() => videoReader.validate(video, options, { hasAnnotation: false })).toThrow(message)
})
it('accepts exactly 20 frames', () => {
expect(() =>
videoReader.validate(video, { start: 0, end: 60, count: 20 }, { hasAnnotation: false }),
).not.toThrow()
})
it('renders frames through the screenshot workflow with the annotation', async () => {
const plan = await videoReader.plan(
video,
{ start: 1, end: 3, count: 2 },
{
annotation: { annotationId: 'c1', annotations: [], timestamp: 2 },
includeContent: true,
},
)
expect(plan).toEqual({
status: 'render',
task: {
type: 'transcode_screenshot',
spec: {
screenshot: { start: 1, end: 3, count: 2, commentTimestamp: 2, annotations: [] },
},
},
})
})
it('fails when the render produced no frames', () => {
expect(() =>
videoReader.collect!(video, { start: 0, end: 1, count: 1 }, { screenshots: [] }, {}),
).toThrow('No screenshots were generated.')
})
})
describe('model3dReader', () => {
const model = asset({
name: 'robot.glb',
mediaType: 'model/gltf-binary',
media: {
proxyType: '3d',
videoTranscodes: [{ key: 'proxy/turntable.mp4' }],
imageTranscodes: [{ key: 'proxy/views.webp' }],
} as unknown as PrismaJson.MediaInfo,
})
it('accepts null as "show the overview"', async () => {
expect(() =>
model3dReader.validate(model, { degree: null }, { hasAnnotation: false }),
).not.toThrow()
const plan = await model3dReader.plan(model, { degree: null }, noAnnotation)
expect(plan.status).toBe('ready')
})
it('carries the angle to collect through the task context', async () => {
const plan = await model3dReader.plan(model, { degree: 180 }, noAnnotation)
expect(plan).toMatchObject({
status: 'render',
context: { degree: 180, hasMarkup: false },
})
const result = model3dReader.collect!(
model,
{ degree: 180 },
{ screenshots: [{ key: 'shots/180.webp' }] },
{ degree: 180, hasMarkup: false },
)
expect(result.images).toEqual([
{ key: 'shots/180.webp', label: 'View at 180°', mimeType: 'image/webp' },
])
})
})
describe('documentReader', () => {
const pdf = asset({ name: 'slides.pdf', mediaType: 'application/pdf' })
it.each([
[undefined, 'docConfig with mode'],
[{ mode: 'pages' as const }, 'startPage and endPage are required'],
[{ mode: 'pages' as const, startPage: 0, endPage: 1 }, 'must be at least 1'],
[{ mode: 'pages' as const, startPage: 3, endPage: 2 }, 'less than or equal to end page'],
[{ mode: 'pages' as const, startPage: 1, endPage: 21 }, 'exceeds the maximum limit of 20'],
[{ mode: 'text' as const }, 'cannot be read as raw text'],
])('rejects %o for PDFs', (options, message) => {
expect(() => documentReader.validate(pdf, options, { hasAnnotation: false })).toThrow(message)
})
it('renders a page range with annotations', async () => {
const plan = await documentReader.plan(
pdf,
{ mode: 'pages', startPage: 2, endPage: 3 },
{ annotation: { annotationId: 'c1', annotations: null, timestamp: 2 }, includeContent: true },
)
expect(plan).toMatchObject({
status: 'render',
task: {
type: 'transcode_pdf_pages',
spec: { pdfPages: { start: 2, end: 3, commentTimestamp: 2, annotations: null } },
},
})
})
it('labels rendered pages by page number', () => {
const result = documentReader.collect!(
pdf,
{ mode: 'pages', startPage: 2, endPage: 3 },
{ pages: [{ key: 'p/2.webp', page: 2 }] },
{},
)
expect(result.summary).toBe('Rendered 1 page(s) of "slides.pdf" (Pages 2 to 3):')
expect(result.images).toEqual([{ key: 'p/2.webp', label: 'Page 2', mimeType: 'image/webp' }])
})
it('detects plain text by media type or extension', () => {
expect(isPlainTextAsset('text/plain', 'a.bin')).toBe(true)
expect(isPlainTextAsset('application/x-yaml', 'a')).toBe(true)
expect(isPlainTextAsset('', 'main.rs')).toBe(true)
expect(isPlainTextAsset('application/pdf', 'a.pdf')).toBe(false)
})
it('truncates by bytes as well as lines', () => {
const big = 'x'.repeat(MAX_TEXT_BYTES + 10)
const { content, truncated } = truncateText(big)
expect(truncated).toBe(true)
expect(Buffer.byteLength(content)).toBe(MAX_TEXT_BYTES)
expect(truncateText('short')).toEqual({ content: 'short', truncated: false })
})
})
@@ -0,0 +1,75 @@
import { MAX_RENDER_ITEMS } from '@shumai/operations'
import { WorkflowTaskType } from '@shumai/db'
import { getFileMimeType } from '@shumai/core/src/utils/file-mime'
import { badRequest, type AssetReader } from '../types'
/** Videos: evenly spaced frames between `start` and `end`, rendered by the screenshot workflow. */
export const videoReader: AssetReader<'video'> = {
kind: 'video',
label: 'video',
configHint: "Pass 'videoConfig: { start, end, count }'",
matches: (asset) =>
asset.media?.proxyType === 'video' || asset.mediaTypeLower.startsWith('video/'),
validate(_asset, options) {
if (!options) {
throw badRequest(
'videoConfig with start, end, and count is required when inspecting video assets.',
)
}
const { start, end, count } = options
if (start < 0 || end < 0) {
throw badRequest('Invalid video time range: start and end must be non-negative numbers.')
}
if (start > end) {
throw badRequest(
`Invalid video time range: start (${start}) must be less than or equal to end (${end}).`,
)
}
if (count < 1) {
throw badRequest(`Invalid frame count: count (${count}) must be at least 1.`)
}
if (count > MAX_RENDER_ITEMS) {
throw badRequest(
`Invalid frame count: count (${count}) exceeds the maximum of ${MAX_RENDER_ITEMS} frames per request. Read a shorter range or fewer frames per call.`,
)
}
},
async plan(_asset, options, { annotation }) {
// validate() guarantees options are present
const { start, end, count } = options!
return {
status: 'render',
task: {
type: WorkflowTaskType.transcode_screenshot,
spec: {
screenshot: {
start,
end,
count,
commentTimestamp: annotation?.timestamp ?? null,
annotations: annotation?.annotations ?? null,
},
},
},
}
},
collect(asset, options, output) {
const screenshots = output.screenshots as Array<{ key: string; timestamp: number }> | undefined
if (!screenshots || screenshots.length === 0) {
throw new Error('No screenshots were generated.')
}
return {
summary: `Extracted ${screenshots.length} frame(s) from "${asset.name}" (${options!.start}s - ${options!.end}s):`,
details: [],
images: screenshots.map((shot) => ({
key: shot.key,
label: `Frame at ${shot.timestamp}s`,
mimeType: getFileMimeType(null, shot.key, 'image/webp'),
})),
}
},
}
+42
View File
@@ -0,0 +1,42 @@
import type { ReaderKind } from '@shumai/operations'
import { documentReader } from './readers/document'
import { imageReader } from './readers/image'
import { model3dReader } from './readers/model3d'
import { videoReader } from './readers/video'
import type { AssetReader, ReadableAsset } from './types'
type MissingKinds<TReaders extends readonly AssetReader[]> = Exclude<
ReaderKind,
TReaders[number]['kind']
>
/** Compile-time guard: every reader kind declared in `@shumai/operations` needs a reader. */
function defineReaderRegistry<const TReaders extends readonly AssetReader[]>(
readers: TReaders &
([MissingKinds<TReaders>] extends [never]
? unknown
: { missingReaders: MissingKinds<TReaders> }),
): TReaders {
return readers
}
/**
* Readers in matching order: the first reader whose `matches` returns true handles the asset.
* 3D comes before image (3D models also have image transcodes); documents cover PDFs and text.
*/
export const readers = defineReaderRegistry([
model3dReader,
imageReader,
videoReader,
documentReader,
] as const)
export function findReader(asset: ReadableAsset): AssetReader | undefined {
return readers.find((reader) => reader.matches(asset)) as AssetReader | undefined
}
export function getReader(kind: ReaderKind): AssetReader {
const reader = readers.find((r) => r.kind === kind)
if (!reader) throw new Error(`No reader registered for "${kind}"`)
return reader as AssetReader
}
+100
View File
@@ -0,0 +1,100 @@
import type { ReaderKind, ReaderOptions } from '@shumai/operations'
import type { WorkflowTaskType } from '@shumai/db'
import { HTTPException } from 'hono/http-exception'
/** The asset being read, after version stacks are resolved to their latest version. */
export interface ReadableAsset {
id: string
name: string
projectId: string | null
/** Media type as stored (may be null). */
mediaType: string | null
/** Lowercased media type ('' when unknown), for matching. */
mediaTypeLower: string
/** Lowercased file name, for extension matching. */
filenameLower: string
storageKey: string | null
media: PrismaJson.MediaInfo | null
}
export interface ReadAnnotation {
annotationId: string
annotations: PrismaJson.AnnotationList | null
/** Seconds (video, 3D turntable) or page number (PDF) where the markup was drawn. */
timestamp: number | null
}
export interface ReadImage {
key: string
/** What the image shows, e.g. "Frame at 30s", "Page 3", "View at 90°". */
label: string
mimeType: string
}
export interface ReadResult {
/** One-line description, e.g. `Extracted 4 frame(s) from "clip.mp4" (0s - 60s):`. */
summary: string
/** Extra description lines (e.g. the 3D quadrant legend). */
details: string[]
images: ReadImage[]
text?: { content: string; truncated: boolean; size: number; key: string; mimeType: string }
}
export interface PlanContext {
annotation: ReadAnnotation | null
/** False when the caller only wants a description (skip reading text content). */
includeContent: boolean
}
export type RenderSpec = Pick<
PrismaJson.WorkflowTaskPayload,
'screenshot' | 'imageAnnotation' | 'pdfPages'
>
export type ReadPlan =
| { status: 'ready'; result: ReadResult }
| {
status: 'render'
task: { type: WorkflowTaskType; spec: RenderSpec }
/** Reader-private values stored on the task and passed back to `collect`. */
context?: Record<string, unknown>
}
/**
* Reads one family of assets. The shared pipeline (`assetReadService`) handles access checks,
* version stacks, annotations, workflow tasks, and waiting; a reader only decides what to show.
*/
export interface AssetReader<TKind extends ReaderKind = ReaderKind> {
kind: TKind
/** Shown in errors, e.g. "Asset X is a 3D model." */
label: string
/** How to pass this reader's options, shown when another reader's config is passed. */
configHint: string
matches(asset: ReadableAsset): boolean
/** Synchronous argument checks, run before annotations are resolved or anything is rendered. */
validate(
asset: ReadableAsset,
options: ReaderOptions<TKind> | undefined,
ctx: { hasAnnotation: boolean },
): void
plan(
asset: ReadableAsset,
options: ReaderOptions<TKind> | undefined,
ctx: PlanContext,
): Promise<ReadPlan>
/** Builds the result from a finished render. Required when `plan` can return `render`. */
collect?(
asset: ReadableAsset,
options: ReaderOptions<TKind> | undefined,
renderOutput: PrismaJson.WorkflowTaskOutput,
context: Record<string, unknown>,
): ReadResult
}
export function badRequest(message: string): HTTPException {
return new HTTPException(400, { message })
}
export function storageBucket(): string {
return process.env.S3_BUCKET || 'shumai'
}
@@ -0,0 +1,43 @@
import { getAssetRender, readAsset, type Via } from '@shumai/operations'
import { assetReadService, toAssetReadOutput } from '@shumai/core/src/asset-read'
import { implementOperation } from '../implement'
/**
* How long each surface waits for a render before returning `pending`. MCP clients commonly
* time out around 60s, so MCP and generic API callers get a short budget.
*/
export const READ_WAIT_BUDGET_MS: Record<Via, number> = {
agent: 60_000,
mcp: 20_000,
cli: 120_000,
api: 20_000,
}
/** `maxWaitSeconds` can only shorten the surface budget. */
export function waitBudgetMs(via: Via, maxWaitSeconds: number | undefined): number {
const budget = READ_WAIT_BUDGET_MS[via]
return maxWaitSeconds === undefined ? budget : Math.min(budget, maxWaitSeconds * 1000)
}
export const readAssetOp = implementOperation(readAsset, {
// assetReadService checks Read on the asset before planning or creating any render task.
authz: () => [],
execute: async (input, ctx) => {
const { maxWaitSeconds, ...request } = input
const outcome = await assetReadService.read({ kind: 'user', userId: ctx.userId }, request, {
waitBudgetMs: waitBudgetMs(ctx.via, maxWaitSeconds),
})
return toAssetReadOutput(outcome, { includeKeys: ctx.via === 'agent' })
},
})
export const getAssetRenderOp = implementOperation(getAssetRender, {
// Only the requesting user, with Read on the asset, can fetch a render (checked by the service).
authz: () => [],
execute: async (input, ctx) => {
const outcome = await assetReadService.getRender(ctx.userId, input.taskId, {
waitBudgetMs: waitBudgetMs(ctx.via, input.maxWaitSeconds),
})
return toAssetReadOutput(outcome, { includeKeys: ctx.via === 'agent' })
},
})
+6
View File
@@ -14,5 +14,11 @@ export { createFolderOp } from './asset/create-folder'
export { deleteAssetOp } from './asset/delete-asset'
export { listAssetsOp } from './asset/list-assets'
export { moveAssetsOp } from './asset/move-assets'
export {
getAssetRenderOp,
READ_WAIT_BUDGET_MS,
readAssetOp,
waitBudgetMs,
} from './asset/read-asset'
export { renameAssetOp } from './asset/rename-asset'
export { listProjectsOp } from './project/list-projects'
+3
View File
@@ -3,6 +3,7 @@ import { createFolderOp } from './asset/create-folder'
import { deleteAssetOp } from './asset/delete-asset'
import { listAssetsOp } from './asset/list-assets'
import { moveAssetsOp } from './asset/move-assets'
import { getAssetRenderOp, readAssetOp } from './asset/read-asset'
import { renameAssetOp } from './asset/rename-asset'
import type { Operation } from './implement'
import { listProjectsOp } from './project/list-projects'
@@ -31,6 +32,8 @@ export const operations = defineRegistry([
renameAssetOp,
moveAssetsOp,
deleteAssetOp,
readAssetOp,
getAssetRenderOp,
] as const)
const operationsByName = new Map<string, Operation>()
+37
View File
@@ -94,6 +94,43 @@ describe('S3Service implementations', () => {
expect(getSignedUrlSpy).toHaveBeenCalledTimes(1)
})
it('signs with an explicit lifetime and caches it separately from the default', async () => {
const s3 = new S3StorageService('http://localhost:9000', 'key', 'secret', 'test-bucket')
getSignedUrlSpy.mockClear()
getSignedUrlSpy.mockResolvedValueOnce('http://default-url')
getSignedUrlSpy.mockResolvedValueOnce('http://short-url')
const defaultUrl = await s3.presign('bucket', 'expiry-key', 'GET')
const shortUrl = await s3.presign(
'bucket',
'expiry-key',
'GET',
false,
undefined,
false,
3600,
)
const shortAgain = await s3.presign(
'bucket',
'expiry-key',
'GET',
false,
undefined,
false,
3600,
)
expect(defaultUrl).toBe('http://default-url')
expect(shortUrl).toBe('http://short-url')
expect(shortAgain).toBe('http://short-url')
expect(getSignedUrlSpy).toHaveBeenCalledTimes(2)
expect(getSignedUrlSpy).toHaveBeenLastCalledWith(
expect.anything(),
expect.any(GetObjectCommand),
expect.objectContaining({ expiresIn: 3600 }),
)
})
it('should NOT cache PUT presign URLs', async () => {
const s3 = new S3StorageService('http://localhost:9000', 'key', 'secret', 'test-bucket')
+11 -6
View File
@@ -112,6 +112,8 @@ export interface S3Service {
download?: boolean,
filename?: string,
internal?: boolean,
/** URL lifetime; defaults to PRESIGNED_URL_EXPIRES_IN hours. Ignored by local storage. */
expiresInSeconds?: number,
) => Promise<string>
presignMultipart: (
bucket: string,
@@ -360,12 +362,15 @@ export class S3StorageService implements S3Service {
download?: boolean,
filename?: string,
internal?: boolean,
expiresInSeconds?: number,
): Promise<string> {
const expireHours = parseInt(process.env.PRESIGNED_URL_EXPIRES_IN || '5', 10)
const expiresInSeconds = expireHours * 3600
const cacheTtlMs = expiresInSeconds * 1000
const lifetimeSeconds = expiresInSeconds ?? expireHours * 3600
// With an explicit (usually short) lifetime, only reuse a cached URL while it has at
// least half of its lifetime left, so callers never receive a nearly expired link.
const cacheTtlMs = expiresInSeconds ? (lifetimeSeconds * 1000) / 2 : lifetimeSeconds * 1000
const cacheKey = `${bucket}/${key}?download=${download || false}&filename=${filename || ''}&internal=${internal || false}`
const cacheKey = `${bucket}/${key}?download=${download || false}&filename=${filename || ''}&internal=${internal || false}&expires=${expiresInSeconds ?? 'default'}`
if (method === 'GET' && !download) {
const cached = this.presignCache.get(cacheKey)
if (cached) {
@@ -382,19 +387,19 @@ export class S3StorageService implements S3Service {
Key: key,
ResponseContentDisposition: download ? buildContentDisposition(filename) : undefined,
})
url = await getSignedUrl(signingClient, command, { expiresIn: expiresInSeconds })
url = await getSignedUrl(signingClient, command, { expiresIn: lifetimeSeconds })
} else if (method === 'PUT') {
const command = new PutObjectCommand({
Bucket: bucket,
Key: key,
})
url = await getSignedUrl(signingClient, command, { expiresIn: expiresInSeconds })
url = await getSignedUrl(signingClient, command, { expiresIn: lifetimeSeconds })
} else if (method === 'DELETE') {
const command = new DeleteObjectCommand({
Bucket: bucket,
Key: key,
})
url = await getSignedUrl(signingClient, command, { expiresIn: expiresInSeconds })
url = await getSignedUrl(signingClient, command, { expiresIn: lifetimeSeconds })
} else {
throw new Error(`Unsupported method for presign: ${method}`)
}
+17
View File
@@ -359,10 +359,27 @@ declare global {
screenshot?: ScreenshotSpec
imageAnnotation?: ImageAnnotationSpec
pdfPages?: PdfPagesSpec
/** Render that produced this result, if any (set by the read_asset pipeline). */
assetRead?: AssetReadRequest
watermark?: WatermarkTaskPayload
recoveryAttempts?: number
}
/**
* Records which reader started a render so the read can be completed later from the task
* alone (see `get_asset_render`), on any server instance.
*/
export interface AssetReadRequest {
reader: 'model3d' | 'image' | 'video' | 'document'
/** The reader's validated options. */
options?: Record<string, unknown>
annotationId?: string
/** Reader-private values computed while planning (e.g. the resolved 3D angle). */
context?: Record<string, unknown>
/** User who requested the read; undefined for system agents (autofill). */
requestedBy?: string
}
export interface WorkflowTaskOutput {
error?: string
recoveryAttempts?: number
+292
View File
@@ -0,0 +1,292 @@
import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest'
import { prisma, type User } from '@shumai/db'
import { setupTestDbHooks } from '@shumai/db/test'
import { workflowService, TaskQueueAgent, TaskQueueTranscode } from '@shumai/workflow-core'
import { initAgentWorkflows } from '@shumai/agent'
import { initTranscodeWorkflows } from '@shumai/transcode'
import { s3Service } from '@shumai/core/src/s3/s3'
import { projectService } from '@shumai/core/src/project/project'
import { getAssetRenderOp, readAssetOp, runOperation } from '@shumai/core/src/operations'
import { AgentHarness } from '@earendil-works/pi-agent-core'
import { fileURLToPath } from 'url'
import * as path from 'path'
import * as fs from 'fs'
const BUCKET = 'shumai-e2e-test-bucket-read-asset'
const fixturesDir = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../fixtures')
type ToolResult = { content: Array<{ type: string; text?: string; data?: string }> }
type ExecutableTool = {
execute: (id: string, params: Record<string, unknown>) => Promise<ToolResult>
}
/**
* read_asset end to end: real render workflows (frames, PDF pages) through the reader
* pipeline, the pending → get_asset_render path, and the agent tool in a chat workflow.
* Only the model call is mocked.
*/
describe('Workflow E2E - read_asset', () => {
setupTestDbHooks()
let videoAssetId = ''
beforeAll(async () => {
process.env.S3_BUCKET = BUCKET
process.env.GEMINI_API_KEY = 'dummy-key'
initTranscodeWorkflows()
initAgentWorkflows()
vi.spyOn(AgentHarness.prototype, 'prompt').mockImplementation(async function (
this: AgentHarness,
) {
const readAsset = this.getTools().find((t) => t.name === 'read_asset') as unknown as
ExecutableTool | undefined
const hasRenderTool = this.getTools().some((t) => t.name === 'get_asset_render')
if (!readAsset) {
return {
content: [{ type: 'text', text: 'MISSING_READ_ASSET' }],
usage: { input: 1, output: 1 },
stopReason: 'stop',
} as unknown as Awaited<ReturnType<typeof AgentHarness.prototype.prompt>>
}
const result = await readAsset.execute('call-1', {
assetId: videoAssetId,
videoConfig: { start: 0, end: 1, count: 2 },
})
const text = result.content.find((c) => c.type === 'text')?.text ?? ''
const images = result.content.filter((c) => c.type === 'image' && c.data)
const summary = [
`images:${images.length}`,
`frames:${(text.match(/- Frame at /g) ?? []).length}`,
`render_tool:${hasRenderTool ? 'present' : 'absent'}`,
].join('|')
return {
content: [{ type: 'text', text: summary }],
usage: { input: 10, output: 20 },
stopReason: 'stop',
} as unknown as Awaited<ReturnType<typeof AgentHarness.prototype.prompt>>
})
await workflowService.startWorkers([TaskQueueTranscode, TaskQueueAgent])
})
afterAll(async () => {
await workflowService.shutdownWorkers()
workflowService.close()
vi.restoreAllMocks()
try {
await s3Service.deletePrefix(BUCKET, '')
} catch (err) {
console.error('Failed to clean up E2E storage folder:', err)
}
})
async function seed(prefix: string) {
const team = await prisma.team.create({ data: { name: `E2E Read ${prefix} Team` } })
const user = await prisma.user.create({
data: { name: 'Reader', email: `read-${prefix}@example.com`, type: 'human' },
})
await prisma.teamMember.create({ data: { teamId: team.id, userId: user.id, role: 'owner' } })
const project = await projectService.createProject(user, {
teamId: team.id,
name: `E2E Read ${prefix} Project`,
})
return { team, user, project }
}
async function createFileAsset(
projectId: string,
parentId: string,
fixture: string,
name: string,
mediaType: string,
proxyType: string,
) {
const key = `projects/${projectId}/${name}`
const buffer = fs.readFileSync(path.join(fixturesDir, fixture))
await s3Service.putObject(BUCKET, key, buffer, buffer.length, mediaType)
const storageKey = await prisma.storageKey.create({ data: { key } })
return prisma.asset.create({
data: {
name,
type: 'file',
status: 'uploaded',
mediaType,
projectId,
parentId,
storageKeyId: storageKey.id,
media: { proxyType } as unknown as PrismaJson.MediaInfo,
},
})
}
const cli = (user: User) => ({ userId: user.id, via: 'cli' as const })
it('extracts real video frames and returns links without storage keys', async () => {
const { user, project } = await seed('frames')
const video = await createFileAsset(
project.id,
project.rootFolder!,
'small.mp4',
'clip.mp4',
'video/mp4',
'video',
)
const output = await runOperation(readAssetOp, cli(user), {
assetId: video.id,
videoConfig: { start: 0, end: 1, count: 2 },
})
expect(output.status).toBe('ready')
expect(output.summary).toBe('Extracted 2 frame(s) from "clip.mp4" (0s - 1s):')
expect(output.images).toHaveLength(2)
for (const image of output.images) {
expect(image.label).toMatch(/^Frame at [\d.]+s$/)
expect(image.url).toEqual(expect.any(String))
expect(image.key).toBeUndefined()
}
}, 60000)
it('returns pending and completes the read through get_asset_render', async () => {
const { user, project } = await seed('pending')
const pdf = await createFileAsset(
project.id,
project.rootFolder!,
'test.pdf',
'deck.pdf',
'application/pdf',
'pdf',
)
const pending = await runOperation(readAssetOp, cli(user), {
assetId: pdf.id,
docConfig: { mode: 'pages', startPage: 1, endPage: 1 },
maxWaitSeconds: 0,
})
expect(pending).toMatchObject({
status: 'pending',
asset: { id: pdf.id, kind: 'document' },
taskId: expect.any(String),
retryAfterSeconds: 5,
})
const ready = await runOperation(getAssetRenderOp, cli(user), { taskId: pending.taskId! })
expect(ready.status).toBe('ready')
expect(ready.summary).toBe('Rendered 1 page(s) of "deck.pdf" (Pages 1 to 1):')
expect(ready.images).toEqual([
{ label: 'Page 1', mimeType: 'image/webp', url: expect.any(String) },
])
// Only the requester can pick up the render.
const other = await prisma.user.create({
data: { name: 'Other', email: 'read-other@example.com', type: 'human' },
})
await expect(
runOperation(getAssetRenderOp, cli(other), { taskId: pending.taskId! }),
).rejects.toMatchObject({ status: 404 })
}, 60000)
it('gives the chat agent video frames as images', async () => {
const { team, user, project } = await seed('agent')
const video = await createFileAsset(
project.id,
project.rootFolder!,
'small.mp4',
'agent-clip.mp4',
'video/mp4',
'video',
)
videoAssetId = video.id
const agentUser = await prisma.user.create({
data: { name: 'E2E Read Agent', email: 'e2e-read-agent@shumai.ai', type: 'agent' },
})
await prisma.teamMember.create({
data: { teamId: team.id, userId: agentUser.id, role: 'editor' },
})
const provider = await prisma.provider.create({
data: {
name: 'google',
teamId: team.id,
config: { api: 'google-generative-ai', apiKey: 'dummy-google-api-key' },
},
})
const model = await prisma.model.create({
data: {
modelId: 'gemini',
name: 'Gemini',
providerId: provider.id,
config: {
input: ['text', 'image'],
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
contextWindow: 8192,
maxTokens: 2048,
reasoning: false,
},
},
})
await prisma.agent.create({
data: {
id: agentUser.id,
teamId: team.id,
type: 'chat',
enabled: true,
providerId: provider.id,
modelId: model.id,
config: { provider: 'google', model: 'gemini' },
},
})
await prisma.agentSession.create({
data: {
id: 'read-asset-session',
agentId: agentUser.id,
userId: user.id,
cwd: process.cwd(),
},
})
const userComment = await prisma.assetComment.create({
data: {
assetId: video.id,
message: 'What happens in the first second?',
creatorId: user.id,
sessionId: 'read-asset-session',
},
})
const task = await prisma.workflowTask.create({
data: {
type: 'chat',
status: 'pending',
assetId: video.id,
projectId: project.id,
teamId: team.id,
payload: {
projectId: project.id,
agent: {
agentId: agentUser.id,
sessionId: 'read-asset-session',
userCommentId: userComment.id,
userId: user.id,
},
},
},
})
const completed = await workflowService.executeWait(task, 90000)
expect(completed.status).toBe('completed')
const [reply] = await prisma.assetComment.findMany({
where: { assetId: video.id, replyToId: userComment.id },
})
expect(reply?.message).toContain('images:2')
expect(reply?.message).toContain('frames:2')
expect(reply?.message).toContain('render_tool:present')
const render = await prisma.workflowTask.findFirstOrThrow({
where: { assetId: video.id, type: 'transcode_screenshot' },
})
expect(render.payload?.assetRead).toMatchObject({ reader: 'video', requestedBy: user.id })
}, 100000)
})
+176
View File
@@ -0,0 +1,176 @@
import { z } from 'zod'
import { defineOperation } from '../define'
/**
* Asset readers, in matching order. Each reader handles one family of assets and contributes
* one optional config field to `read_asset`. Implementations live in
* `@shumai/core/src/asset-read/readers`.
*/
export const READER_KINDS = ['model3d', 'image', 'video', 'document'] as const
export type ReaderKind = (typeof READER_KINDS)[number]
/** Most frames or pages a single read may render. */
export const MAX_RENDER_ITEMS = 20
export const readerOptionSchemas = {
model3d: {
key: 'config3d',
schema: z
.object({
degree: z
.number()
.int()
.nullable()
.optional()
.describe(
'Turntable rotation degree (integer: 0, 15, 30, ..., 345 in 15° increments) to inspect a specific angle of a 3D model. Leave null/omitted to view the 4-view composite overview.',
),
})
.describe(
'Configuration for 3D model assets. Optionally pass degree to inspect a specific turntable angle, or leave it out to view the 4-view composite overview.',
),
},
image: {
key: 'imageConfig',
schema: z.object({}).describe('Optional configuration for image assets.'),
},
video: {
key: 'videoConfig',
schema: z
.object({
start: z.number().describe('Start time in seconds.'),
end: z.number().describe('End time in seconds.'),
count: z
.number()
.describe(`Number of frames to extract, evenly spaced (1 to ${MAX_RENDER_ITEMS}).`),
})
.describe('Configuration for video assets. Required when inspecting videos.'),
},
document: {
key: 'docConfig',
schema: z
.object({
mode: z
.enum(['pages', 'text'])
.describe(
'"pages" renders a page range as images (visual layout, charts, markups; required for binary PDFs). "text" reads raw text (ideal for .md, .txt, .csv, code files).',
),
startPage: z
.number()
.nullable()
.optional()
.describe('Start page number (1-based, required when mode="pages"; omit for "text").'),
endPage: z
.number()
.nullable()
.optional()
.describe(
`End page number (1-based, max ${MAX_RENDER_ITEMS} pages per call, required when mode="pages"; omit for "text").`,
),
})
.describe('Configuration for document, PDF, and text assets.'),
},
} as const satisfies Record<ReaderKind, { key: string; schema: z.ZodObject }>
export type ReaderOptions<TKind extends ReaderKind> = z.output<
(typeof readerOptionSchemas)[TKind]['schema']
>
/** The config field name each reader contributes, e.g. `videoConfig`. */
export type ReaderConfigKey = (typeof readerOptionSchemas)[ReaderKind]['key']
const readImageSchema = z.object({
label: z.string().describe('What the image shows, e.g. "Frame at 30s" or "Page 3".'),
mimeType: z.string(),
url: z
.string()
.optional()
.describe('Short-lived link to the image (about 1 hour). Download it to look at the image.'),
/** Storage key; only returned to Shumai's built-in agent. */
key: z.string().optional(),
})
/** Shared by `read_asset` and `get_asset_render`. */
export const assetReadOutputSchema = z.object({
status: z
.enum(['ready', 'pending'])
.describe('"pending" means a render is still running; call get_asset_render with taskId.'),
summary: z.string(),
asset: z.object({
id: z.string(),
name: z.string(),
kind: z.enum(READER_KINDS),
mediaType: z.string().nullable(),
}),
details: z.array(z.string()),
images: z.array(readImageSchema),
text: z
.object({
content: z.string(),
truncated: z.boolean(),
size: z.number(),
mimeType: z.string(),
url: z.string().optional(),
key: z.string().optional(),
})
.optional(),
taskId: z.string().optional(),
retryAfterSeconds: z.number().optional(),
})
export type AssetReadOutput = z.infer<typeof assetReadOutputSchema>
const maxWaitSeconds = z
.number()
.int()
.min(0)
.optional()
.describe(
'Longest time to wait for rendering, in seconds. If rendering takes longer, the result is "pending" with a taskId to pass to get_asset_render. 0 returns immediately.',
)
export const readAsset = defineOperation({
name: 'read_asset',
title: 'Read Asset',
description:
'Inspects and reads the content of an asset without downloading it. ' +
'Supports images, 3D models (4-view overview or a specific degree angle), video frame extraction, PDF page rendering, and direct text reading for text/markdown/code files. ' +
'Pass assetId and the config matching the asset type (imageConfig, config3d, videoConfig, or docConfig); omit the others. ' +
'Optionally pass annotationId to view visual markups drawn on the asset. ' +
'Rendering frames or pages can take a while: if the result has status "pending", call get_asset_render with its taskId.',
input: z.object({
assetId: z.string().min(1).describe('The ID of the asset to inspect.'),
annotationId: z
.string()
.optional()
.describe(
'The ID of a comment or message (from <annotation id="..." />) whose visual markup should be overlaid.',
),
config3d: readerOptionSchemas.model3d.schema.optional(),
imageConfig: readerOptionSchemas.image.schema.optional(),
videoConfig: readerOptionSchemas.video.schema.optional(),
docConfig: readerOptionSchemas.document.schema.optional(),
includeContent: z
.boolean()
.optional()
.describe('Set to false to skip reading text content and only describe the asset.'),
maxWaitSeconds,
}),
output: assetReadOutputSchema,
surfaces: ['agent'],
annotations: { readOnly: true, destructive: false },
})
export const getAssetRender = defineOperation({
name: 'get_asset_render',
title: 'Get Asset Render',
description:
'Gets the result of a read_asset call that returned status "pending". ' +
'Returns "pending" again while rendering is still running; wait retryAfterSeconds and try again.',
input: z.object({
taskId: z.string().min(1).describe('The taskId returned by read_asset.'),
maxWaitSeconds,
}),
output: assetReadOutputSchema,
surfaces: ['agent'],
annotations: { readOnly: true, destructive: false, idempotent: true },
})
+1
View File
@@ -6,4 +6,5 @@ export { deleteAsset } from './asset/delete-asset'
export { listAssets } from './asset/list-assets'
export { moveAssets } from './asset/move-assets'
export { renameAsset } from './asset/rename-asset'
export * from './asset/read-asset'
export { listProjects } from './project/list-projects'
+12 -2
View File
@@ -15,13 +15,15 @@ describe('operation registry', () => {
expect(new Set(names).size).toBe(names.length)
})
it('contains the initial asset and project operations', () => {
it('contains the asset and project operations', () => {
expect(operationDefinitions.map((d) => d.name).sort()).toEqual([
'create_folder',
'delete_asset',
'get_asset_render',
'list_assets',
'list_projects',
'move_assets',
'read_asset',
'rename_asset',
])
})
@@ -40,7 +42,12 @@ describe('operation registry', () => {
const destructive = operationDefinitions.filter((d) => d.annotations.destructive)
expect(destructive.map((d) => d.name)).toEqual(['delete_asset'])
const readOnly = operationDefinitions.filter((d) => d.annotations.readOnly)
expect(readOnly.map((d) => d.name).sort()).toEqual(['list_assets', 'list_projects'])
expect(readOnly.map((d) => d.name).sort()).toEqual([
'get_asset_render',
'list_assets',
'list_projects',
'read_asset',
])
})
it.each(operationDefinitions.map((d) => [d.name, d] as const))(
@@ -71,6 +78,9 @@ describe('operation registry', () => {
['rename_asset', ['rename']],
['move_assets', ['move', 'mv']],
['delete_asset', ['delete', 'rm']],
// CLI commands for reading assets arrive with the CLI surface
['read_asset', []],
['get_asset_render', []],
])
})
+3
View File
@@ -2,6 +2,7 @@ import { createFolder } from './asset/create-folder'
import { deleteAsset } from './asset/delete-asset'
import { listAssets } from './asset/list-assets'
import { moveAssets } from './asset/move-assets'
import { getAssetRender, readAsset } from './asset/read-asset'
import { renameAsset } from './asset/rename-asset'
import type { AnyOperationDefinition, Surface } from './define'
import { listProjects } from './project/list-projects'
@@ -17,6 +18,8 @@ export const operationDefinitions = [
renameAsset,
moveAssets,
deleteAsset,
readAsset,
getAssetRender,
] as const
export type RegisteredOperationDefinition = (typeof operationDefinitions)[number]
@@ -82,6 +82,11 @@ const BUILTIN_TOOLS = [
name: () => m.agent_tool_read_asset_name(),
description: () => m.agent_tool_read_asset_desc(),
},
{
id: 'get_asset_render',
name: () => m.agent_tool_get_asset_render_name(),
description: () => m.agent_tool_get_asset_render_desc(),
},
{
id: 'list_projects',
name: () => m.agent_tool_list_projects_name(),
+2
View File
@@ -609,6 +609,8 @@
"agent_tool_read_skill_desc": "Allows the agent to load and read skill definitions to perform specialized actions.",
"agent_tool_read_asset_name": "Read Asset",
"agent_tool_read_asset_desc": "Allows the agent to view and inspect images, video frames, and document pages.",
"agent_tool_get_asset_render_name": "Get Asset Render",
"agent_tool_get_asset_render_desc": "Allows the agent to pick up video frames or document pages that took longer to render.",
"agent_tool_list_projects_name": "List Projects",
"agent_tool_list_projects_desc": "Allows the agent to list the projects you have access to and find their root folders.",
"agent_tool_list_assets_name": "Browse Shumai",
+2
View File
@@ -609,6 +609,8 @@
"agent_tool_read_skill_desc": "允许智能体加载和读取技能定义以执行特定操作。",
"agent_tool_read_asset_name": "读取资源",
"agent_tool_read_asset_desc": "允许智能体查看和分析图片、视频画面和文档页面。",
"agent_tool_get_asset_render_name": "获取资源渲染结果",
"agent_tool_get_asset_render_desc": "允许智能体获取渲染耗时较长的视频画面或文档页面。",
"agent_tool_list_projects_name": "列出项目",
"agent_tool_list_projects_desc": "允许智能体列出您有权访问的项目并找到其根文件夹。",
"agent_tool_list_assets_name": "浏览 Shumai",
@@ -189,6 +189,61 @@ describe('WorkflowService', () => {
submitSpy.mockRestore()
})
describe('waitFor', () => {
async function createTask(status: WorkflowTaskStatus, output?: Record<string, unknown>) {
const submitSpy = vi.spyOn(workflowService, 'submit').mockImplementation(async () => 'id')
const task = await prisma.workflowTask.create({
data: { assetId: 'wait-asset', type: WorkflowTaskType.transcode_screenshot, status },
})
if (output) {
await prisma.workflowTask.update({ where: { id: task.id }, data: { output } })
}
submitSpy.mockRestore()
return task
}
it('returns done: true with the completed task', async () => {
const task = await createTask(WorkflowTaskStatus.completed, { screenshots: [] })
const result = await workflowService.waitFor(task, 1000)
expect(result.done).toBe(true)
expect(result.task.output).toEqual({ screenshots: [] })
})
it('returns done: false instead of throwing when the budget runs out', async () => {
const task = await createTask(WorkflowTaskStatus.processing)
const start = Date.now()
const result = await workflowService.waitFor(task, 300)
expect(result.done).toBe(false)
expect(result.task.status).toBe(WorkflowTaskStatus.processing)
expect(Date.now() - start).toBeLessThan(2000)
})
it('checks once and returns immediately with a zero budget', async () => {
const task = await createTask(WorkflowTaskStatus.pending)
const result = await workflowService.waitFor(task, 0)
expect(result.done).toBe(false)
})
it('throws the task error when the task failed', async () => {
const task = await createTask(WorkflowTaskStatus.failed, { error: 'ffmpeg exploded' })
await expect(workflowService.waitFor(task, 1000)).rejects.toThrow('ffmpeg exploded')
})
it('keeps executeWait throwing on timeout', async () => {
const task = await createTask(WorkflowTaskStatus.processing)
await expect(workflowService.executeWait(task, 200)).rejects.toThrow(
'Workflow task timed out',
)
})
})
describe('Stale Task Recovery', () => {
it('should resume stale pending workflow task in DBOS and increment recoveryAttempts', async () => {
const task = await prisma.workflowTask.create({
+22 -4
View File
@@ -30,8 +30,24 @@ export class WorkflowService {
}
async executeWait(task: WorkflowTask, timeoutMs: number = 30000): Promise<WorkflowTask> {
const { task: finished, done } = await this.waitFor(task, timeoutMs)
if (!done) {
throw new Error('Workflow task timed out')
}
return finished
}
/**
* Wait up to `timeoutMs` for a task to finish. Unlike `executeWait`, running out of time is
* not an error: it resolves with the latest task state and `done: false`, so callers can hand
* the task ID back and complete the work later. Throws if the task failed or disappeared.
*/
async waitFor(
task: WorkflowTask,
timeoutMs: number,
): Promise<{ task: WorkflowTask; done: boolean }> {
const start = Date.now()
while (Date.now() - start < timeoutMs) {
for (;;) {
const updatedTask = await prisma.workflowTask.findUnique({
where: { id: task.id },
})
@@ -41,7 +57,7 @@ export class WorkflowService {
}
if (updatedTask.status === 'completed') {
return updatedTask
return { task: updatedTask, done: true }
}
if (updatedTask.status === 'failed') {
@@ -50,10 +66,12 @@ export class WorkflowService {
throw new Error(error)
}
if (Date.now() - start >= timeoutMs) {
return { task: updatedTask, done: false }
}
await new Promise((resolve) => setTimeout(resolve, 200))
}
throw new Error('Workflow task timed out')
}
start(listenQueues?: string[]): void {