mirror of
https://github.com/shumaiOne/shumai.git
synced 2026-11-19 14:16:29 +00:00
feat: expose read_asset and download_asset on MCP and the CLI (#499)
* feat(asset-read): expose read_asset on MCP and the CLI Enable the mcp and cli surfaces for read_asset and get_asset_render. - MCP: add per-Operation tool extensions. The read presenter attaches up to 8 images (2 MB each) inline, lists image links, supports inlineImages: false, and tells the model to call get_asset_render when a render is pending. Storage keys are stripped from the result. - CLI: generated commands can flatten object inputs into prefixed flags (cli.flatten) and omit fields (cli.omit); add per-command extensions. New `read` and `render` commands print image links, download them with --out, and support --no-wait with a printed `render <taskId>` hint. - Round frame timestamps in labels to 3 decimals. * fix(mcp): make tool input schemas portable to single-type dialects Nullable inputs were emitted as `type: ["number", "null"]` (or a `null` anyOf branch), which clients that map tool schemas onto an OpenAPI-style subset (e.g. Gemini function declarations) may reject. - Make read_asset's startPage, endPage, and degree optional instead of nullable. - Move null handling into a shared dropNullOptionals helper in @shumai/operations that also strips nulls inside nested config objects. The agent uses it, and the MCP server applies it before the SDK validates arguments, so models that send null keep working. - Test that no Operation input schema (and no listed MCP tool) uses a nullable type. * feat(asset-download): share download_asset across agent, MCP and CLI Turn the agent-only download_asset tool into an Operation on all three surfaces. - download_asset takes up to 50 file or version stack IDs and returns presigned links. proxy (default true) picks the web preview when there is one, as the agent always did; proxy: false returns the original. Folders are rejected. Read is checked on every ID before anything runs. - Agent: an extension saves the files into the .pi sandbox and keeps the agent-only storage key mode (e.g. frames from read_asset). Agent tool extensions gain description, parameterOverrides and intercept. - CLI: `shumai-cli download <assetId...> [--out <dir>] [--original]` streams files to disk without overwriting existing names. - read_asset's unsupported-type error now names the right way to download on each surface (download_asset, or `shumai-cli download`). - Add the Download Asset toggle to agent settings.
This commit is contained in:
@@ -234,7 +234,11 @@ An **Operation** is a capability agents can use, defined once and exposed automa
|
||||
- **Register** the definition in `packages/operations/src/registry.ts` and the implementation in `packages/core/src/operations/registry.ts`. A definition without an implementation fails typecheck.
|
||||
- **Run** Operations only through `runOperation(op, { userId, via, agentId? }, input)`. `via` (`api`, `agent`, `mcp`, `cli`) is recorded in audit logs.
|
||||
- **Tests**: real-database tests per Operation covering success, permission denial (no side effects), and audit entries.
|
||||
- **Agent presentation**: by default an agent tool returns the Operation output as JSON. To add agent-only parameters or render output differently (e.g. image blocks), register an `AgentToolExtension` in `packages/agent/src/operation-extensions/`.
|
||||
- **CLI spec**: object input fields must be listed in `cli.flatten` (one flag per sub-field, e.g. `{ videoConfig: 'video' }` → `--video-start`) or in `cli.omit` (optional fields the CLI doesn't expose).
|
||||
- **Surface presentation**: by default each surface shows the Operation output as JSON. To add surface-only parameters or render output differently, add an extension for that surface:
|
||||
- agent: `AgentToolExtension` in `packages/agent/src/operation-extensions/` (e.g. image blocks, saving downloads into the sandbox). `parameterOverrides` and `intercept` support agent-only modes, such as `download_asset` by storage key;
|
||||
- MCP: `McpToolExtension` in `packages/api/src/mcp-extensions/` (`structuredContent` must still match the output schema);
|
||||
- CLI: `CliCommandExtension` in `apps/cli/src/operations/extensions/` (local I/O such as downloads).
|
||||
|
||||
### Asset Readers
|
||||
|
||||
@@ -245,7 +249,7 @@ To add a reader:
|
||||
1. Add its kind to `READER_KINDS` and its options to `readerOptionSchemas` in `packages/operations/src/asset/read-asset.ts` (one optional `<name>Config` field on `read_asset`).
|
||||
2. Implement `AssetReader` (`matches`, synchronous `validate`, `plan`, and `collect` if `plan` can return `render`), and add it to `readers` in `packages/core/src/asset-read/registry.ts`. Order matters: the first matching reader wins. A declared kind without a reader fails typecheck.
|
||||
3. `plan` returns either `ready` with the result, or `render` with a workflow task spec and optional `context`. The pipeline stores the request on the task payload (`assetRead`), so `collect` and `get_asset_render` work across restarts and instances.
|
||||
4. Use `badRequest(...)` for invalid input. Return storage keys in `ReadResult`; the pipeline turns them into short-lived links (and only gives keys to Shumai's built-in agent).
|
||||
4. Use `badRequest(...)` for invalid input. Return storage keys in `ReadResult`; the pipeline turns them into short-lived links. Keys only go to server-side presenters (agent, MCP), which must not pass them on to clients.
|
||||
5. Add reader unit tests (`readers.test.ts`), pipeline cases in `asset-read.test.ts`, and a workflow E2E case if it renders.
|
||||
|
||||
## Frontend Architecture
|
||||
|
||||
@@ -10,6 +10,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
|
||||
* **mcp**: Shumai now has a built-in MCP server, so external AI agents such as Claude Code, Cursor, or Kiro can browse projects and create, rename, move, or delete folders and files with your API token and permissions
|
||||
* **agent**: The built-in agent can now list the projects you have access to, so it can find a project's root folder on its own. Team owners can turn this off per agent like other tools
|
||||
* **mcp**: External AI agents connected to Shumai's MCP server can now look at your files: images, video frames, PDF pages, 3D model views, and text files. Images are attached so the AI can see them, along with links that stay valid for about an hour. They can also get download links for files, as the web preview or the original upload
|
||||
* **cli**: New `shumai-cli download` command saves files to your computer, up to 50 at a time: the smaller web preview by default, or the original upload with `--original`
|
||||
* **cli**: New `shumai-cli read` command shows an image, video frames, PDF pages, a 3D model view, or the text of a file, and `--out` downloads the images into a folder. If rendering takes a while, `shumai-cli render <taskId>` picks up the result later
|
||||
|
||||
### Fixed
|
||||
|
||||
@@ -22,6 +25,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
* **cli**: `shumai-cli` has clearer help for every command (`shumai-cli <command> --help`), suggests the right command when you mistype one, and can print raw JSON with `--json`. `ls` can now filter by type and page through very large folders. Existing commands work as before; the npm package now needs Node.js 22.12 or later
|
||||
* **agent**: When the agent lists a folder's contents, large folders are now paged with a "next page" marker instead of page numbers, and items appear in the same order as in the file browser
|
||||
* **agent**: When the agent looks at a long video or a large document that takes more than about a minute to render, it no longer fails with a timeout. The agent is told the render is still in progress and picks up the frames or pages when they're ready, using the new "Get Asset Render" tool. Each call can now read at most 20 video frames, the same limit as PDF pages
|
||||
* **agent**: The agent can now download several files at once (up to 50), and can get the original upload instead of the smaller web preview. Downloaded files keep the preview's file type in their name (e.g. `photo.webp`). Team owners can now turn the Download Asset tool on or off per agent in agent settings
|
||||
* **api-token**: API tokens are now stored securely, so they can no longer be read back from the database. A new token is shown only once when you create it, so copy it right away; the token list now shows just the last few characters. Existing tokens keep working without any changes.
|
||||
|
||||
## [0.6.1] - 2026-10-24
|
||||
|
||||
+47
-2
@@ -2,7 +2,7 @@
|
||||
|
||||
A Command Line Interface (CLI) tool for interacting with your Shumai instance.
|
||||
|
||||
The CLI allows you to view projects, list folder contents, create folders, upload files and directories, and create new versions of existing files directly from the terminal.
|
||||
The CLI allows you to view projects, list folder contents, create folders, upload files and directories, create new versions of existing files, and look at images, video frames, and document pages directly from the terminal.
|
||||
|
||||
---
|
||||
|
||||
@@ -135,6 +135,51 @@ shumai-cli delete <assetId>
|
||||
shumai-cli rm <assetId>
|
||||
```
|
||||
|
||||
#### 9. Read an Asset
|
||||
|
||||
Look at a file: an image, video frames, PDF pages, a 3D model view, or the text of a text file. Prints a summary and a link to each image (links are valid for about an hour), or the file's text.
|
||||
|
||||
```bash
|
||||
# An image, or the 4-view overview of a 3D model
|
||||
shumai-cli read <assetId>
|
||||
# 5 evenly spaced frames between 0s and 30s of a video, downloaded into ./frames
|
||||
shumai-cli read <assetId> --video-start 0 --video-end 30 --video-count 5 --out frames
|
||||
# Pages 1 to 3 of a PDF, or the text of a text file
|
||||
shumai-cli read <assetId> --doc-mode pages --doc-start-page 1 --doc-end-page 3
|
||||
shumai-cli read <assetId> --doc-mode text
|
||||
# A 3D model at 90°
|
||||
shumai-cli read <assetId> --degree 90
|
||||
```
|
||||
|
||||
- **Options:**
|
||||
- `--video-start <s>`, `--video-end <s>`, `--video-count <n>`: Video frames to extract (up to 20).
|
||||
- `--doc-mode <pages|text>`, `--doc-start-page <n>`, `--doc-end-page <n>`: PDF pages to render (up to 20), or `text` to read a text file.
|
||||
- `--degree <deg>`: 3D model angle (0, 15, 30, ..., 345).
|
||||
- `--annotation-id <id>`: Draw a comment's markup on the image, frame, or page.
|
||||
- `-o`, `--out <dir>`: Download the images into a folder and print their paths. With `--json`, each image also gets a `path`.
|
||||
- `--max-wait-seconds <s>`, `--no-wait`: Rendering frames or pages can take a while. `read` waits up to 2 minutes; use these to wait less or return right away. If the render isn't done, `read` prints the `render` command to run next.
|
||||
|
||||
#### 10. Get a Render
|
||||
|
||||
Get the result of a `read` that was still rendering. Takes the same `--out`, `--max-wait-seconds`, and `--no-wait` options as `read`.
|
||||
|
||||
```bash
|
||||
shumai-cli render <taskId> --out frames
|
||||
```
|
||||
|
||||
#### 11. Download Files
|
||||
|
||||
Download up to 50 files at once. By default it saves the web preview version when there is one (for example the transcoded video or the web image), which is smaller; use `--original` for the uploaded file. A version stack downloads its latest version. Folders aren't supported: download the files inside them. Files keep their names; if a name is taken, the new file gets a number, like `clip (2).mp4`.
|
||||
|
||||
```bash
|
||||
shumai-cli download <assetId...> --out ./downloads
|
||||
shumai-cli download <assetId> --original
|
||||
```
|
||||
|
||||
- **Options:**
|
||||
- `-o`, `--out <dir>`: Folder to save the files in (default: the current folder). With `--json`, each file also gets a `path`.
|
||||
- `--original`: Download the original uploaded files instead of the web previews.
|
||||
|
||||
---
|
||||
|
||||
## Development
|
||||
@@ -147,4 +192,4 @@ bun run apps/cli/src/index.ts project ls
|
||||
|
||||
### Adding Commands
|
||||
|
||||
Most commands are generated from the shared Operation definitions in `packages/operations` (see "Operations" in `AGENTS.md`). An Operation with the `cli` surface and a `cli` spec becomes a command automatically, with arguments, options, and help text derived from its input schema. It is also available to Shumai's built-in agent and MCP server. Only commands that stream local files (`upload`, `create-version`) are written by hand in `src/program.ts`.
|
||||
Most commands are generated from the shared Operation definitions in `packages/operations` (see "Operations" in `AGENTS.md`). An Operation with the `cli` surface and a `cli` spec becomes a command automatically, with arguments, options, and help text derived from its input schema. It is also available to Shumai's built-in agent and MCP server. Behavior that needs the local machine, such as downloading files, goes in a CLI extension in `src/operations/extensions/` (extra options, input mapping, and output). Only commands that stream local files (`upload`, `create-version`) are written by hand in `src/program.ts`.
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import { mkdtemp, readFile, readdir, rm, writeFile } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import * as path from 'node:path'
|
||||
import type { DownloadAssetOutput } from '@shumai/operations'
|
||||
import { CliError } from '../errors'
|
||||
import { downloadAssetExtension, safeFileName, uniquePath } from './download-asset'
|
||||
|
||||
const output: DownloadAssetOutput = {
|
||||
files: [
|
||||
{
|
||||
assetId: 'a-1',
|
||||
name: 'clip.mp4',
|
||||
kind: 'proxy',
|
||||
size: 2048,
|
||||
mimeType: 'video/mp4',
|
||||
url: 'https://files.example/a-1',
|
||||
},
|
||||
{
|
||||
assetId: 'a-2',
|
||||
name: 'clip.mp4',
|
||||
kind: 'original',
|
||||
size: 10,
|
||||
mimeType: 'video/mp4',
|
||||
url: 'https://files.example/a-2',
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
describe('download CLI extension', () => {
|
||||
let dir: string
|
||||
let written: string
|
||||
|
||||
beforeEach(async () => {
|
||||
dir = await mkdtemp(path.join(tmpdir(), 'shumai-cli-download-'))
|
||||
written = ''
|
||||
vi.stubGlobal(
|
||||
'fetch',
|
||||
vi.fn(async (url: string) => new Response(`bytes of ${url}`, { status: 200 })),
|
||||
)
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
vi.unstubAllGlobals()
|
||||
await rm(dir, { recursive: true, force: true })
|
||||
})
|
||||
|
||||
const o = () => ({ out: { write: (s: string) => void (written += s) }, isTty: true, columns: 80 })
|
||||
|
||||
it('saves every file, numbering names that are already taken', async () => {
|
||||
await writeFile(path.join(dir, 'clip.mp4'), 'existing')
|
||||
|
||||
await downloadAssetExtension.present(output, { opts: { out: dir }, json: false, output: o() })
|
||||
|
||||
expect((await readdir(dir)).sort()).toEqual(['clip (2).mp4', 'clip (3).mp4', 'clip.mp4'])
|
||||
expect(await readFile(path.join(dir, 'clip.mp4'), 'utf8')).toBe('existing')
|
||||
expect(await readFile(path.join(dir, 'clip (2).mp4'), 'utf8')).toBe(
|
||||
'bytes of https://files.example/a-1',
|
||||
)
|
||||
expect(written).toBe(
|
||||
`Saved ${path.join(dir, 'clip (2).mp4')} (2 KB, proxy)\nSaved ${path.join(dir, 'clip (3).mp4')} (10 B, original)\n`,
|
||||
)
|
||||
})
|
||||
|
||||
it('creates the folder and prints JSON with the saved paths', async () => {
|
||||
const out = path.join(dir, 'nested', 'files')
|
||||
await downloadAssetExtension.present(output, { opts: { out }, json: true, output: o() })
|
||||
|
||||
const printed = JSON.parse(written) as { files: Array<{ path: string; url: string }> }
|
||||
expect(printed.files.map((f) => f.path)).toEqual([
|
||||
path.join(out, 'clip.mp4'),
|
||||
path.join(out, 'clip (2).mp4'),
|
||||
])
|
||||
expect(printed.files[0].url).toBe('https://files.example/a-1')
|
||||
})
|
||||
|
||||
it('fails clearly when a download fails', async () => {
|
||||
vi.stubGlobal(
|
||||
'fetch',
|
||||
vi.fn(async () => new Response('gone', { status: 403 })),
|
||||
)
|
||||
await expect(
|
||||
downloadAssetExtension.present(output, { opts: { out: dir }, json: false, output: o() }),
|
||||
).rejects.toThrow(new CliError('Failed to download "clip.mp4" (status 403).'))
|
||||
})
|
||||
|
||||
it('keeps file names inside the target folder', () => {
|
||||
expect(safeFileName('../../etc/passwd')).toBe('_.._etc_passwd')
|
||||
expect(safeFileName('a\\b.txt')).toBe('a_b.txt')
|
||||
expect(safeFileName('...')).toBe('file')
|
||||
const taken = new Set<string>()
|
||||
expect(uniquePath(dir, 'notes', taken)).toBe(path.join(dir, 'notes'))
|
||||
expect(uniquePath(dir, 'notes', taken)).toBe(path.join(dir, 'notes (2)'))
|
||||
})
|
||||
|
||||
it('maps --original to proxy false', () => {
|
||||
expect(downloadAssetExtension.toInput!({ assetIds: ['a'] }, { original: true })).toEqual({
|
||||
assetIds: ['a'],
|
||||
proxy: false,
|
||||
})
|
||||
expect(downloadAssetExtension.toInput!({ assetIds: ['a'] }, {})).toEqual({ assetIds: ['a'] })
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,58 @@
|
||||
import { existsSync } from 'node:fs'
|
||||
import { mkdir } from 'node:fs/promises'
|
||||
import * as path from 'node:path'
|
||||
import { Option } from 'commander'
|
||||
import type { DownloadAssetOutput } from '@shumai/operations'
|
||||
import type { CliCommandExtension } from '../generate'
|
||||
import { formatSize } from '../render'
|
||||
import { saveUrlToFile } from './save-url'
|
||||
|
||||
/** A file name that is safe to create inside the target folder (no paths or control chars). */
|
||||
export function safeFileName(name: string): string {
|
||||
const safe = name
|
||||
// eslint-disable-next-line no-control-regex
|
||||
.replace(/[\x00-\x1f\x7f/\\]/g, '_')
|
||||
.replace(/^[\s.]+|[\s.]+$/g, '')
|
||||
return safe || 'file'
|
||||
}
|
||||
|
||||
/** `name.ext`, or `name (2).ext`, `name (3).ext`, ... if taken on disk or in this download. */
|
||||
export function uniquePath(dir: string, name: string, taken: Set<string>): string {
|
||||
const ext = path.extname(name)
|
||||
const stem = ext ? name.slice(0, -ext.length) : name
|
||||
for (let n = 1; ; n++) {
|
||||
const candidate = path.join(dir, n === 1 ? name : `${stem} (${n})${ext}`)
|
||||
if (!taken.has(candidate) && !existsSync(candidate)) {
|
||||
taken.add(candidate)
|
||||
return candidate
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** `download`: saves each file into `--out` (the current folder by default). */
|
||||
export const downloadAssetExtension: CliCommandExtension = {
|
||||
options: () => [
|
||||
new Option('-o, --out <dir>', 'Folder to save the files in.').default('.'),
|
||||
new Option('--original', 'Download the original uploaded files instead of the web previews.'),
|
||||
],
|
||||
toInput: (input, opts) => (opts.original === true ? { ...input, proxy: false } : input),
|
||||
async present(raw, { opts, json, output: o }) {
|
||||
const output = raw as DownloadAssetOutput
|
||||
const dir = typeof opts.out === 'string' ? opts.out : '.'
|
||||
await mkdir(dir, { recursive: true })
|
||||
const taken = new Set<string>()
|
||||
const saved: string[] = []
|
||||
for (const file of output.files) {
|
||||
const filePath = uniquePath(dir, safeFileName(file.name), taken)
|
||||
await saveUrlToFile(file.url, filePath, file.name)
|
||||
saved.push(filePath)
|
||||
if (!json) {
|
||||
o.out.write(`Saved ${filePath} (${formatSize(file.size)}, ${file.kind})\n`)
|
||||
}
|
||||
}
|
||||
if (json) {
|
||||
const files = output.files.map((file, i) => ({ ...file, path: saved[i] }))
|
||||
o.out.write(`${JSON.stringify({ files }, null, 2)}\n`)
|
||||
}
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
import type { CliCommandExtension } from '../generate'
|
||||
import { downloadAssetExtension } from './download-asset'
|
||||
import { getAssetRenderExtension, readAssetExtension } from './read-asset'
|
||||
|
||||
/** CLI-specific behavior for generated commands, by Operation name. */
|
||||
export const cliExtensions: ReadonlyMap<string, CliCommandExtension> = new Map([
|
||||
['read_asset', readAssetExtension],
|
||||
['get_asset_render', getAssetRenderExtension],
|
||||
['download_asset', downloadAssetExtension],
|
||||
])
|
||||
@@ -0,0 +1,98 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import { mkdtemp, readFile, rm } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import * as path from 'node:path'
|
||||
import type { AssetReadOutput } from '@shumai/operations'
|
||||
import { CliError } from '../errors'
|
||||
import { downloadImages, imageFileName, readAssetExtension } from './read-asset'
|
||||
|
||||
const output: AssetReadOutput = {
|
||||
status: 'ready',
|
||||
summary: 'Rendered 2 page(s) of "deck.pdf" (Pages 1 to 2):',
|
||||
asset: { id: 'doc-1', name: 'deck.pdf', kind: 'document', mediaType: 'application/pdf' },
|
||||
details: [],
|
||||
images: [
|
||||
{ label: 'Page 1', mimeType: 'image/webp', url: 'https://files.example/p1' },
|
||||
{ label: 'Page 2', mimeType: 'image/png', url: 'https://files.example/p2' },
|
||||
],
|
||||
}
|
||||
|
||||
describe('read / render CLI extension', () => {
|
||||
let dir: string
|
||||
let written: string
|
||||
|
||||
beforeEach(async () => {
|
||||
dir = await mkdtemp(path.join(tmpdir(), 'shumai-cli-read-'))
|
||||
written = ''
|
||||
vi.stubGlobal(
|
||||
'fetch',
|
||||
vi.fn(async (url: string) => new Response(`bytes of ${url}`, { status: 200 })),
|
||||
)
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
vi.unstubAllGlobals()
|
||||
await rm(dir, { recursive: true, force: true })
|
||||
})
|
||||
|
||||
const o = () => ({ out: { write: (s: string) => void (written += s) }, isTty: true, columns: 80 })
|
||||
|
||||
it('names files after the asset, position, and label', () => {
|
||||
expect(imageFileName('vid-1', 0, { label: 'Frame at 2.5s', mimeType: 'image/webp' })).toBe(
|
||||
'vid-1-1-frame-at-2.5s.webp',
|
||||
)
|
||||
expect(imageFileName('m-1', 3, { label: 'View at 90°', mimeType: 'image/jpeg' })).toBe(
|
||||
'm-1-4-view-at-90.jpg',
|
||||
)
|
||||
})
|
||||
|
||||
it('downloads every image with --out and prints the saved paths', async () => {
|
||||
const out = path.join(dir, 'pages')
|
||||
await readAssetExtension.present(output, { opts: { out }, json: false, output: o() })
|
||||
|
||||
const first = path.join(out, 'doc-1-1-page-1.webp')
|
||||
const second = path.join(out, 'doc-1-2-page-2.png')
|
||||
expect(await readFile(first, 'utf8')).toBe('bytes of https://files.example/p1')
|
||||
expect(await readFile(second, 'utf8')).toBe('bytes of https://files.example/p2')
|
||||
expect(written).toBe(`${output.summary}\n- Page 1: ${first}\n- Page 2: ${second}\n`)
|
||||
})
|
||||
|
||||
it('adds the saved path to each image in --json mode', async () => {
|
||||
await readAssetExtension.present(output, { opts: { out: dir }, json: true, output: o() })
|
||||
|
||||
const printed = JSON.parse(written) as { images: Array<{ path: string }> }
|
||||
expect(printed.images.map((i) => i.path)).toEqual([
|
||||
path.join(dir, 'doc-1-1-page-1.webp'),
|
||||
path.join(dir, 'doc-1-2-page-2.png'),
|
||||
])
|
||||
})
|
||||
|
||||
it('fails clearly when a download fails', async () => {
|
||||
vi.stubGlobal(
|
||||
'fetch',
|
||||
vi.fn(async () => new Response('gone', { status: 403 })),
|
||||
)
|
||||
await expect(downloadImages(output, dir)).rejects.toThrow(
|
||||
new CliError('Failed to download "Page 1" (status 403).'),
|
||||
)
|
||||
})
|
||||
|
||||
it('does not download anything for a pending render', async () => {
|
||||
await readAssetExtension.present(
|
||||
{ ...output, status: 'pending', images: [], taskId: 't-1', retryAfterSeconds: 5 },
|
||||
{ opts: { out: dir }, json: false, output: o() },
|
||||
)
|
||||
expect(fetch).not.toHaveBeenCalled()
|
||||
expect(written).toContain(`shumai-cli render t-1 --out ${JSON.stringify(dir)}`)
|
||||
})
|
||||
|
||||
it('maps --no-wait to maxWaitSeconds 0', () => {
|
||||
expect(readAssetExtension.toInput!({ assetId: 'a' }, { wait: false })).toEqual({
|
||||
assetId: 'a',
|
||||
maxWaitSeconds: 0,
|
||||
})
|
||||
expect(readAssetExtension.toInput!({ assetId: 'a' }, { wait: true })).toEqual({
|
||||
assetId: 'a',
|
||||
})
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,116 @@
|
||||
import { mkdir } from 'node:fs/promises'
|
||||
import * as path from 'node:path'
|
||||
import { Option } from 'commander'
|
||||
import type { AssetReadOutput } from '@shumai/operations'
|
||||
import { CliError } from '../errors'
|
||||
import { saveUrlToFile } from './save-url'
|
||||
import type { CliCommandExtension } from '../generate'
|
||||
import type { Output } from '../render'
|
||||
|
||||
const EXTENSIONS: Record<string, string> = {
|
||||
'image/webp': 'webp',
|
||||
'image/png': 'png',
|
||||
'image/jpeg': 'jpg',
|
||||
'image/gif': 'gif',
|
||||
'image/avif': 'avif',
|
||||
}
|
||||
|
||||
function slug(value: string): string {
|
||||
return (
|
||||
value
|
||||
.toLowerCase()
|
||||
.replace(/[^a-z0-9.]+/g, '-')
|
||||
.replace(/^-+|-+$/g, '') || 'image'
|
||||
)
|
||||
}
|
||||
|
||||
/** File name for the n-th image of a read, e.g. `01ABC-1-frame-at-2s.webp`. */
|
||||
export function imageFileName(
|
||||
assetId: string,
|
||||
index: number,
|
||||
image: AssetReadOutput['images'][number],
|
||||
): string {
|
||||
const ext = EXTENSIONS[image.mimeType] ?? image.mimeType.split('/')[1] ?? 'bin'
|
||||
return `${assetId}-${index + 1}-${slug(image.label).replace(/\.+$/, '')}.${ext}`
|
||||
}
|
||||
|
||||
/** Downloads every image of a ready read into `dir`; returns the saved paths in order. */
|
||||
export async function downloadImages(output: AssetReadOutput, dir: string): Promise<string[]> {
|
||||
await mkdir(dir, { recursive: true })
|
||||
const saved: string[] = []
|
||||
for (const [index, image] of output.images.entries()) {
|
||||
if (!image.url) {
|
||||
throw new CliError(`No download link was returned for "${image.label}".`)
|
||||
}
|
||||
const filePath = path.join(dir, imageFileName(output.asset.id, index, image))
|
||||
await saveUrlToFile(image.url, filePath, image.label)
|
||||
saved.push(filePath)
|
||||
}
|
||||
return saved
|
||||
}
|
||||
|
||||
function renderCommand(taskId: string, out: unknown): string {
|
||||
const outArg = typeof out === 'string' ? ` --out ${JSON.stringify(out)}` : ''
|
||||
return `shumai-cli render ${taskId}${outArg}`
|
||||
}
|
||||
|
||||
function writeReady(output: AssetReadOutput, paths: string[] | null, o: Output): void {
|
||||
const lines = [output.summary, ...output.details]
|
||||
output.images.forEach((image, i) => {
|
||||
lines.push(`- ${image.label}: ${paths ? paths[i] : (image.url ?? '')}`)
|
||||
})
|
||||
o.out.write(`${lines.join('\n')}\n`)
|
||||
if (output.text) {
|
||||
o.out.write(`\n${output.text.content}`)
|
||||
if (!output.text.content.endsWith('\n')) o.out.write('\n')
|
||||
if (output.text.truncated) {
|
||||
o.out.write(
|
||||
`\n[Content truncated: showing the first part of a ${output.text.size}-byte file. Run \`shumai-cli download ${output.asset.id}\` to get the whole file.]\n`,
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* `read` and `render`: images are printed as links (valid for about an hour) or downloaded
|
||||
* with `--out`; a render that takes longer than the wait prints the `render` command to run.
|
||||
*/
|
||||
function readExtension(noWaitHelp: string): CliCommandExtension {
|
||||
return {
|
||||
options: () => [
|
||||
new Option('-o, --out <dir>', 'Download the images into this folder and print their paths.'),
|
||||
new Option('--no-wait', noWaitHelp),
|
||||
],
|
||||
toInput: (input, opts) => (opts.wait === false ? { ...input, maxWaitSeconds: 0 } : input),
|
||||
async present(raw, { opts, json, output: o }) {
|
||||
const output = raw as AssetReadOutput
|
||||
if (output.status === 'pending') {
|
||||
if (json) {
|
||||
o.out.write(`${JSON.stringify(output, null, 2)}\n`)
|
||||
return
|
||||
}
|
||||
o.out.write(
|
||||
`${output.summary}\nRun this to get the result: ${renderCommand(output.taskId ?? '', opts.out)}\n`,
|
||||
)
|
||||
return
|
||||
}
|
||||
|
||||
const paths = typeof opts.out === 'string' ? await downloadImages(output, opts.out) : null
|
||||
if (json) {
|
||||
const images = output.images.map((image, i) =>
|
||||
paths ? { ...image, path: paths[i] } : image,
|
||||
)
|
||||
o.out.write(`${JSON.stringify({ ...output, images }, null, 2)}\n`)
|
||||
return
|
||||
}
|
||||
writeReady(output, paths, o)
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
export const readAssetExtension = readExtension(
|
||||
'Return right away with a task ID if the asset needs rendering.',
|
||||
)
|
||||
export const getAssetRenderExtension = readExtension(
|
||||
'Check once and return right away if the render is still running.',
|
||||
)
|
||||
@@ -0,0 +1,32 @@
|
||||
import { createWriteStream } from 'node:fs'
|
||||
import { rm } from 'node:fs/promises'
|
||||
import { Readable } from 'node:stream'
|
||||
import { pipeline } from 'node:stream/promises'
|
||||
import type { ReadableStream as WebReadableStream } from 'node:stream/web'
|
||||
import { CliError } from '../errors'
|
||||
|
||||
/** Streams a download link to `filePath` (large files never sit in memory). */
|
||||
export async function saveUrlToFile(url: string, filePath: string, label: string): Promise<void> {
|
||||
let res: Response
|
||||
try {
|
||||
res = await fetch(url)
|
||||
} catch (err) {
|
||||
throw new CliError(
|
||||
`Failed to download "${label}": ${err instanceof Error ? err.message : String(err)}`,
|
||||
)
|
||||
}
|
||||
if (!res.ok || !res.body) {
|
||||
throw new CliError(`Failed to download "${label}" (status ${res.status}).`)
|
||||
}
|
||||
try {
|
||||
await pipeline(
|
||||
Readable.fromWeb(res.body as unknown as WebReadableStream<Uint8Array>),
|
||||
createWriteStream(filePath),
|
||||
)
|
||||
} catch (err) {
|
||||
await rm(filePath, { force: true })
|
||||
throw new CliError(
|
||||
`Failed to save "${label}" to ${filePath}: ${err instanceof Error ? err.message : String(err)}`,
|
||||
)
|
||||
}
|
||||
}
|
||||
@@ -5,11 +5,29 @@ import type { CallOperation } from './call-operation'
|
||||
import { CliError } from './errors'
|
||||
import { renderOutput, type Output } from './render'
|
||||
|
||||
/**
|
||||
* CLI-specific behavior for one Operation's command, for things a definition can't describe
|
||||
* (it must stay free of I/O): extra options, input mapping, and output that downloads files.
|
||||
*/
|
||||
export interface CliCommandExtension {
|
||||
/** Extra options only the CLI has; their values are passed to `toInput` and `present`. */
|
||||
options?(): Option[]
|
||||
/** Adjust the Operation input using the command's options. */
|
||||
toInput?(input: Record<string, unknown>, opts: Record<string, unknown>): Record<string, unknown>
|
||||
/** Writes the output, including in `--json` mode. Replaces the default rendering. */
|
||||
present(
|
||||
output: unknown,
|
||||
ctx: { opts: Record<string, unknown>; json: boolean; output: Output },
|
||||
): Promise<void>
|
||||
}
|
||||
|
||||
export interface GenerateDeps {
|
||||
callOperation: CallOperation
|
||||
output: Output
|
||||
/** Whether destructive Operations are allowed (SHUMAI_ALLOW_DELETE). */
|
||||
allowDestructive: () => boolean
|
||||
/** CLI extensions by Operation name. */
|
||||
extensions?: ReadonlyMap<string, CliCommandExtension>
|
||||
}
|
||||
|
||||
type FieldKind =
|
||||
@@ -40,7 +58,14 @@ function unwrap(schema: z.ZodType): z.ZodType {
|
||||
}
|
||||
|
||||
function describeField(definition: AnyOperationDefinition, name: string): FieldInfo {
|
||||
const schema = definition.input.shape[name] as z.ZodType
|
||||
return describeSchema(definition, name, definition.input.shape[name] as z.ZodType)
|
||||
}
|
||||
|
||||
function describeSchema(
|
||||
definition: AnyOperationDefinition,
|
||||
name: string,
|
||||
schema: z.ZodType,
|
||||
): FieldInfo {
|
||||
const base = unwrap(schema)
|
||||
let kind: FieldKind
|
||||
if (base instanceof z.ZodNumber) kind = { type: 'number' }
|
||||
@@ -59,6 +84,28 @@ function describeField(definition: AnyOperationDefinition, name: string): FieldI
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Input errors name the input path (e.g. `videoConfig.end`). For flattened fields, name the
|
||||
* flag the user typed instead (e.g. `--video-end`).
|
||||
*/
|
||||
function formatInputError(
|
||||
error: z.ZodError,
|
||||
options: ReadonlyArray<{ path: [string] | [string, string]; option: Option }>,
|
||||
): string {
|
||||
const flagFor = (path: PropertyKey[]): string | undefined =>
|
||||
options.find((o) => o.path.length === 2 && o.path[0] === path[0] && o.path[1] === path[1])
|
||||
?.option.long
|
||||
if (!error.issues.some((issue) => flagFor(issue.path))) {
|
||||
return z.prettifyError(error)
|
||||
}
|
||||
return error.issues
|
||||
.map((issue) => {
|
||||
const flag = flagFor(issue.path)
|
||||
return flag ? `✖ ${flag}: ${issue.message}` : z.prettifyError(new z.ZodError([issue]))
|
||||
})
|
||||
.join('\n')
|
||||
}
|
||||
|
||||
function parseNumber(value: string): number {
|
||||
const parsed = Number(value)
|
||||
if (value.trim() === '' || Number.isNaN(parsed)) {
|
||||
@@ -105,27 +152,59 @@ export function registerOperationCommands(
|
||||
}
|
||||
|
||||
const positionalNames = new Set(positional.map((f) => f.name))
|
||||
const options: Array<{ field: FieldInfo; option: Option }> = []
|
||||
for (const name of Object.keys(definition.input.shape)) {
|
||||
if (positionalNames.has(name)) continue
|
||||
const field = describeField(definition, name)
|
||||
const spellings = cli.flags?.[name] ?? [`--${kebab(name)}`]
|
||||
const omitted = new Set<string>(cli.omit ?? [])
|
||||
const flatten = (cli.flatten ?? {}) as Record<string, string | undefined>
|
||||
// `path` is where the value goes in the input: `[field]` or `[field, subField]`.
|
||||
const options: Array<{ path: [string] | [string, string]; option: Option }> = []
|
||||
const addOption = (
|
||||
path: [string] | [string, string],
|
||||
field: FieldInfo,
|
||||
spellings: readonly string[],
|
||||
) => {
|
||||
const valueHint =
|
||||
field.kind.type === 'boolean'
|
||||
? ''
|
||||
: field.kind.type === 'array'
|
||||
? ` <${name}...>`
|
||||
: ` <${name}>`
|
||||
? ` <${field.name}...>`
|
||||
: ` <${field.name}>`
|
||||
const option = new Option(`${spellings.join(', ')}${valueHint}`, field.description)
|
||||
if (field.kind.type === 'number') option.argParser(parseNumber)
|
||||
if (field.kind.type === 'enum') option.choices(field.kind.values)
|
||||
if (field.required) option.makeOptionMandatory()
|
||||
cmd.addOption(option)
|
||||
options.push({ path, option })
|
||||
return option
|
||||
}
|
||||
|
||||
for (const name of Object.keys(definition.input.shape)) {
|
||||
if (positionalNames.has(name) || omitted.has(name)) continue
|
||||
const prefix = flatten[name]
|
||||
if (prefix !== undefined) {
|
||||
const schema = definition.input.shape[name] as z.ZodType
|
||||
const base = unwrap(schema)
|
||||
if (!(base instanceof z.ZodObject)) {
|
||||
throw new Error(`${definition.name}.${name}: only object fields can be flattened`)
|
||||
}
|
||||
const parentRequired = !schema.safeParse(undefined).success
|
||||
for (const [subName, subSchema] of Object.entries(base.shape)) {
|
||||
const field = describeSchema(definition, `${name}.${subName}`, subSchema as z.ZodType)
|
||||
field.name = subName
|
||||
// Sub-fields are only mandatory when the object itself is.
|
||||
field.required = parentRequired && field.required
|
||||
const flag = prefix ? `--${prefix}-${kebab(subName)}` : `--${kebab(subName)}`
|
||||
addOption([name, subName], field, [flag])
|
||||
}
|
||||
continue
|
||||
}
|
||||
const field = describeField(definition, name)
|
||||
const option = addOption([name], field, cli.flags?.[name] ?? [`--${kebab(name)}`])
|
||||
const cliDefault = (cli.defaults as Record<string, unknown> | undefined)?.[name]
|
||||
if (cliDefault !== undefined) option.default(cliDefault)
|
||||
cmd.addOption(option)
|
||||
options.push({ field, option })
|
||||
}
|
||||
|
||||
const extension = deps.extensions?.get(definition.name)
|
||||
for (const option of extension?.options?.() ?? []) cmd.addOption(option)
|
||||
|
||||
cmd.action(async (...args: unknown[]) => {
|
||||
const command = args[args.length - 1] as Command
|
||||
const input: Record<string, unknown> = { ...cli.defaults }
|
||||
@@ -133,9 +212,15 @@ export function registerOperationCommands(
|
||||
if (args[i] !== undefined) input[field.name] = args[i]
|
||||
})
|
||||
const opts = command.opts()
|
||||
for (const { field, option } of options) {
|
||||
for (const { path, option } of options) {
|
||||
const value = opts[option.attributeName()]
|
||||
if (value !== undefined) input[field.name] = value
|
||||
if (value === undefined) continue
|
||||
if (path.length === 1) {
|
||||
input[path[0]] = value
|
||||
} else {
|
||||
const parent = (input[path[0]] ?? {}) as Record<string, unknown>
|
||||
input[path[0]] = { ...parent, [path[1]]: value }
|
||||
}
|
||||
}
|
||||
|
||||
if (definition.annotations.destructive && !deps.allowDestructive()) {
|
||||
@@ -144,13 +229,18 @@ export function registerOperationCommands(
|
||||
)
|
||||
}
|
||||
|
||||
const parsed = definition.input.safeParse(input)
|
||||
const parsed = definition.input.safeParse(
|
||||
extension?.toInput ? extension.toInput(input, opts) : input,
|
||||
)
|
||||
if (!parsed.success) {
|
||||
throw new CliError(z.prettifyError(parsed.error))
|
||||
throw new CliError(formatInputError(parsed.error, options))
|
||||
}
|
||||
|
||||
const output = await deps.callOperation(definition, parsed.data)
|
||||
if (command.optsWithGlobals().json) {
|
||||
const json = Boolean(command.optsWithGlobals().json)
|
||||
if (extension) {
|
||||
await extension.present(output, { opts, json, output: deps.output })
|
||||
} else if (json) {
|
||||
deps.output.out.write(`${JSON.stringify(output, null, 2)}\n`)
|
||||
} else {
|
||||
renderOutput(definition, output, deps.output)
|
||||
|
||||
@@ -3,6 +3,9 @@ import type { Command } from 'commander'
|
||||
import {
|
||||
createFolder,
|
||||
deleteAsset,
|
||||
downloadAsset,
|
||||
getAssetRender,
|
||||
readAsset,
|
||||
listAssets,
|
||||
listProjects,
|
||||
moveAssets,
|
||||
@@ -39,6 +42,27 @@ function setup() {
|
||||
|
||||
const folder = { id: 'f-1', name: 'Renders', type: 'folder', size: 0, projectId: 'p-1' }
|
||||
|
||||
const frames = {
|
||||
status: 'ready',
|
||||
summary: 'Extracted 2 frame(s) from "clip.mp4" (0s - 4s):',
|
||||
asset: { id: 'vid-1', name: 'clip.mp4', kind: 'video', mediaType: 'video/mp4' },
|
||||
details: [],
|
||||
images: [
|
||||
{ label: 'Frame at 0s', mimeType: 'image/webp', url: 'https://files.example/0.webp' },
|
||||
{ label: 'Frame at 4s', mimeType: 'image/webp', url: 'https://files.example/4.webp' },
|
||||
],
|
||||
}
|
||||
|
||||
const pendingRead = {
|
||||
status: 'pending',
|
||||
summary: 'Rendering is still in progress for "clip.mp4".',
|
||||
asset: frames.asset,
|
||||
details: [],
|
||||
images: [],
|
||||
taskId: 'task-1',
|
||||
retryAfterSeconds: 5,
|
||||
}
|
||||
|
||||
describe('shumai-cli', () => {
|
||||
const originalAllowDelete = process.env.SHUMAI_ALLOW_DELETE
|
||||
|
||||
@@ -251,6 +275,181 @@ describe('shumai-cli', () => {
|
||||
})
|
||||
})
|
||||
|
||||
describe('read', () => {
|
||||
it('maps flattened reader flags onto the config objects and prints image links', async () => {
|
||||
const cli = setup()
|
||||
cli.callOperation.mockResolvedValue(frames)
|
||||
|
||||
await cli.run(
|
||||
'read',
|
||||
'vid-1',
|
||||
'--video-start',
|
||||
'0',
|
||||
'--video-end',
|
||||
'4',
|
||||
'--video-count',
|
||||
'2',
|
||||
'--annotation-id',
|
||||
'c-1',
|
||||
)
|
||||
|
||||
expect(cli.callOperation).toHaveBeenCalledWith(readAsset, {
|
||||
assetId: 'vid-1',
|
||||
annotationId: 'c-1',
|
||||
videoConfig: { start: 0, end: 4, count: 2 },
|
||||
})
|
||||
expect(cli.stdout()).toBe(
|
||||
[
|
||||
'Extracted 2 frame(s) from "clip.mp4" (0s - 4s):',
|
||||
'- Frame at 0s: https://files.example/0.webp',
|
||||
'- Frame at 4s: https://files.example/4.webp',
|
||||
'',
|
||||
].join('\n'),
|
||||
)
|
||||
})
|
||||
|
||||
it.each([
|
||||
[
|
||||
['--doc-mode', 'pages', '--doc-start-page', '1', '--doc-end-page', '3'],
|
||||
{ docConfig: { mode: 'pages', startPage: 1, endPage: 3 } },
|
||||
],
|
||||
[['--doc-mode', 'text'], { docConfig: { mode: 'text' } }],
|
||||
[['--degree', '90'], { config3d: { degree: 90 } }],
|
||||
[[], {}],
|
||||
])('reads with %j', async (flags, config) => {
|
||||
const cli = setup()
|
||||
cli.callOperation.mockResolvedValue({ ...frames, images: [] })
|
||||
|
||||
await cli.run('read', 'a-1', ...flags)
|
||||
|
||||
expect(cli.callOperation).toHaveBeenCalledWith(readAsset, { assetId: 'a-1', ...config })
|
||||
})
|
||||
|
||||
it('names the flag that is missing from a reader config', async () => {
|
||||
const cli = setup()
|
||||
|
||||
const error = await cli
|
||||
.run('read', 'vid-1', '--video-start', '0', '--video-count', '2')
|
||||
.catch((err: unknown) => err)
|
||||
|
||||
expect(formatCliError(error)).toContain('--video-end')
|
||||
expect(cli.callOperation).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('prints text content after the summary', async () => {
|
||||
const cli = setup()
|
||||
cli.callOperation.mockResolvedValue({
|
||||
...frames,
|
||||
summary: 'Document "notes.md" (ID: d-1):',
|
||||
images: [],
|
||||
text: {
|
||||
content: '# Notes\nhello',
|
||||
truncated: true,
|
||||
size: 90000,
|
||||
mimeType: 'text/markdown',
|
||||
},
|
||||
})
|
||||
|
||||
await cli.run('read', 'd-1', '--doc-mode', 'text')
|
||||
|
||||
expect(cli.stdout()).toBe(
|
||||
'Document "notes.md" (ID: d-1):\n\n# Notes\nhello\n\n[Content truncated: showing the first part of a 90000-byte file. Run `shumai-cli download vid-1` to get the whole file.]\n',
|
||||
)
|
||||
})
|
||||
|
||||
it('returns right away with --no-wait and prints the render command', async () => {
|
||||
const cli = setup()
|
||||
cli.callOperation.mockResolvedValue(pendingRead)
|
||||
|
||||
await cli.run(
|
||||
'read',
|
||||
'vid-1',
|
||||
'--video-start',
|
||||
'0',
|
||||
'--video-end',
|
||||
'4',
|
||||
'--video-count',
|
||||
'2',
|
||||
'--no-wait',
|
||||
'--out',
|
||||
'frames',
|
||||
)
|
||||
|
||||
expect(cli.callOperation).toHaveBeenCalledWith(readAsset, {
|
||||
assetId: 'vid-1',
|
||||
videoConfig: { start: 0, end: 4, count: 2 },
|
||||
maxWaitSeconds: 0,
|
||||
})
|
||||
expect(cli.stdout()).toBe(
|
||||
'Rendering is still in progress for "clip.mp4".\nRun this to get the result: shumai-cli render task-1 --out "frames"\n',
|
||||
)
|
||||
})
|
||||
|
||||
it('prints the raw output with --json', async () => {
|
||||
const cli = setup()
|
||||
cli.callOperation.mockResolvedValue(frames)
|
||||
|
||||
await cli.run('--json', 'read', 'vid-1')
|
||||
|
||||
expect(JSON.parse(cli.stdout())).toEqual(frames)
|
||||
})
|
||||
})
|
||||
|
||||
describe('render', () => {
|
||||
it('fetches a pending render by task ID', async () => {
|
||||
const cli = setup()
|
||||
cli.callOperation.mockResolvedValue(frames)
|
||||
|
||||
await cli.run('render', 'task-1')
|
||||
|
||||
expect(cli.callOperation).toHaveBeenCalledWith(getAssetRender, { taskId: 'task-1' })
|
||||
expect(cli.stdout()).toContain('- Frame at 4s: https://files.example/4.webp')
|
||||
})
|
||||
|
||||
it('checks once with --no-wait', async () => {
|
||||
const cli = setup()
|
||||
cli.callOperation.mockResolvedValue(pendingRead)
|
||||
|
||||
await cli.run('render', 'task-1', '--no-wait')
|
||||
|
||||
expect(cli.callOperation).toHaveBeenCalledWith(getAssetRender, {
|
||||
taskId: 'task-1',
|
||||
maxWaitSeconds: 0,
|
||||
})
|
||||
expect(cli.stdout()).toContain('Run this to get the result: shumai-cli render task-1\n')
|
||||
})
|
||||
})
|
||||
|
||||
describe('download', () => {
|
||||
it('downloads the web previews by default, or the originals with --original', async () => {
|
||||
const cli = setup()
|
||||
cli.callOperation.mockResolvedValue({ files: [] })
|
||||
|
||||
await cli.run('download', 'a-1', 'a-2')
|
||||
expect(cli.callOperation).toHaveBeenLastCalledWith(downloadAsset, {
|
||||
assetIds: ['a-1', 'a-2'],
|
||||
})
|
||||
|
||||
await cli.run('download', 'a-1', '--original')
|
||||
expect(cli.callOperation).toHaveBeenLastCalledWith(downloadAsset, {
|
||||
assetIds: ['a-1'],
|
||||
proxy: false,
|
||||
})
|
||||
})
|
||||
|
||||
it('requires at least one asset ID and documents its options', async () => {
|
||||
const cli = setup()
|
||||
|
||||
await expect(cli.run('download')).rejects.toThrow()
|
||||
expect(cli.stderr()).toContain("missing required argument 'assetIds'")
|
||||
|
||||
await expect(cli.run('download', '--help')).rejects.toThrow()
|
||||
expect(cli.stdout()).toContain('-o, --out <dir>')
|
||||
expect(cli.stdout()).toContain('--original')
|
||||
expect(cli.stdout()).not.toContain('--proxy')
|
||||
})
|
||||
})
|
||||
|
||||
describe('--json', () => {
|
||||
it('prints the raw Operation output', async () => {
|
||||
const cli = setup()
|
||||
@@ -312,6 +511,9 @@ describe('shumai-cli', () => {
|
||||
'rename',
|
||||
'move|mv',
|
||||
'delete|rm',
|
||||
'read',
|
||||
'render',
|
||||
'download',
|
||||
'upload',
|
||||
'create-version',
|
||||
]) {
|
||||
@@ -333,6 +535,30 @@ describe('shumai-cli', () => {
|
||||
expect(help).toContain('(default: 200)')
|
||||
expect(help).toContain('List the files and folders inside a parent folder.')
|
||||
})
|
||||
|
||||
it('documents flattened reader options for read', async () => {
|
||||
const cli = setup()
|
||||
|
||||
await expect(cli.run('read', '--help')).rejects.toThrow()
|
||||
|
||||
const help = cli.stdout()
|
||||
for (const flag of [
|
||||
'--video-start <start>',
|
||||
'--video-end <end>',
|
||||
'--video-count <count>',
|
||||
'--doc-mode <mode>',
|
||||
'--doc-start-page <startPage>',
|
||||
'--doc-end-page <endPage>',
|
||||
'--degree <degree>',
|
||||
'--annotation-id <annotationId>',
|
||||
'-o, --out <dir>',
|
||||
'--no-wait',
|
||||
]) {
|
||||
expect(help).toContain(flag)
|
||||
}
|
||||
expect(help).not.toContain('--include-content')
|
||||
expect(help).not.toContain('--image')
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
|
||||
@@ -3,6 +3,7 @@ import { operationDefinitions } from '@shumai/operations'
|
||||
import { createVersion } from './commands/create-version'
|
||||
import { upload } from './commands/upload'
|
||||
import { callOperation, type CallOperation } from './operations/call-operation'
|
||||
import { cliExtensions } from './operations/extensions'
|
||||
import { registerOperationCommands } from './operations/generate'
|
||||
import type { Output } from './operations/render'
|
||||
|
||||
@@ -18,7 +19,7 @@ function isDeleteAllowed(): boolean {
|
||||
|
||||
/**
|
||||
* Builds the `shumai-cli` program. Operation commands (ls, mkdir, rename, move, delete,
|
||||
* project ls, ...) are generated from `@shumai/operations`; upload and create-version stream
|
||||
* read, render, project ls, ...) are generated from `@shumai/operations`; upload and create-version stream
|
||||
* local files and are defined here.
|
||||
*/
|
||||
export function createProgram(options: ProgramOptions = {}): Command {
|
||||
@@ -37,6 +38,7 @@ export function createProgram(options: ProgramOptions = {}): Command {
|
||||
columns: process.stdout.columns || 80,
|
||||
},
|
||||
allowDestructive: isDeleteAllowed,
|
||||
extensions: cliExtensions,
|
||||
})
|
||||
|
||||
program
|
||||
|
||||
@@ -3,7 +3,7 @@ title: Shumai MCP Server
|
||||
description: Let external AI agents such as Claude Code, Cursor, or Kiro work with your Shumai projects.
|
||||
---
|
||||
|
||||
Shumai includes a built-in [Model Context Protocol](https://modelcontextprotocol.io) (MCP) server. Any MCP-compatible agent can connect to it and browse projects, create folders, and organize assets on your behalf, using the same tools as Shumai's built-in agent.
|
||||
Shumai includes a built-in [Model Context Protocol](https://modelcontextprotocol.io) (MCP) server. Any MCP-compatible agent can connect to it and browse projects, look at images, videos, and documents, create folders, and organize assets on your behalf, using the same tools as Shumai's built-in agent.
|
||||
|
||||
<Note>
|
||||
This page is about connecting **external agents to Shumai**. To connect Shumai's own agents to
|
||||
@@ -58,14 +58,23 @@ claude mcp add --transport http shumai https://<your-shumai-host>/api/mcp \
|
||||
|
||||
## Available Tools
|
||||
|
||||
| Tool | What it does |
|
||||
| --------------- | ----------------------------------------------------------------------------------------------------- |
|
||||
| `list_projects` | Lists the projects you can access, with each project's root folder. |
|
||||
| `list_assets` | Lists files and folders in a folder, page by page. |
|
||||
| `create_folder` | Creates a folder. |
|
||||
| `rename_asset` | Renames a file or folder. |
|
||||
| `move_assets` | Moves files or folders into another folder. |
|
||||
| `delete_asset` | Moves a file or folder to the project trash. Marked as destructive so clients can ask you to confirm. |
|
||||
| Tool | What it does |
|
||||
| ------------------ | ---------------------------------------------------------------------------------------------------------------- |
|
||||
| `list_projects` | Lists the projects you can access, with each project's root folder. |
|
||||
| `list_assets` | Lists files and folders in a folder, page by page. |
|
||||
| `create_folder` | Creates a folder. |
|
||||
| `rename_asset` | Renames a file or folder. |
|
||||
| `move_assets` | Moves files or folders into another folder. |
|
||||
| `delete_asset` | Moves a file or folder to the project trash. Marked as destructive so clients can ask you to confirm. |
|
||||
| `read_asset` | Looks at a file: an image, video frames, PDF pages, a 3D model view, or the text of a text file. |
|
||||
| `get_asset_render` | Gets the result of a `read_asset` call that was still rendering. |
|
||||
| `download_asset` | Gets download links for up to 50 files: the web preview version by default, or the original with `proxy: false`. |
|
||||
|
||||
### Reading Files
|
||||
|
||||
`read_asset` attaches the images to the result so the AI model can see them: up to 8 images per call, each up to 2 MB. Any other images are listed as links only. Every result also includes a link to each image, valid for about an hour. Pass `inlineImages: false` to get links only, for example when your client can't show images.
|
||||
|
||||
Rendering video frames or PDF pages can take a while. If it takes longer than about 20 seconds, `read_asset` returns `status: "pending"` with a `taskId`, and the model calls `get_asset_render` with that `taskId` to get the result. Only the user who started the read can get its result.
|
||||
|
||||
Every call runs with your permissions: for example, a **Reviewer** can list assets but cannot create or rename them. Actions are recorded in the team audit log and attributed to you, marked as coming from MCP.
|
||||
|
||||
|
||||
@@ -80,7 +80,7 @@ The following tools can be individually toggled in the agent configuration dialo
|
||||
|
||||
### 12. Download Asset (`download_asset`)
|
||||
|
||||
- **Description:** Allows the agent to download a workspace asset or specific storage S3 key to the local `.pi` directory for processing with local scripts or bash commands.
|
||||
- **Description:** Allows the agent to download up to 50 workspace files, or a specific storage S3 key (such as a frame from `read_asset`), to the local `.pi` directory for processing with local scripts or bash commands. By default it downloads the web preview version when there is one (for example the transcoded video or the web image); the agent can ask for the original uploaded file instead. Folders are not supported.
|
||||
- **Why it matters:** Enables agents executing custom skill scripts to fetch files or intermediate transcode artifacts to local disk.
|
||||
|
||||
### 13. Rename Asset (`rename_asset`)
|
||||
|
||||
@@ -20,7 +20,6 @@ import { DatabaseSessionStorage } from './database-session-storage'
|
||||
import { metadataService } from '@shumai/core/src/metadata/metadata'
|
||||
import { createCreateFileTool } from './tools/create-file'
|
||||
import { createCreateVersionTool } from './tools/create-version'
|
||||
import { createDownloadAssetTool } from './tools/download-asset'
|
||||
import { createReadSkillTool } from './tools/read-skill'
|
||||
import { createReadThreadTool } from './tools/read-thread'
|
||||
import { createSandboxedBashTool } from './tools/sandboxed-bash'
|
||||
@@ -337,11 +336,10 @@ export async function createAgentSession(params: CreateAgentSessionParams) {
|
||||
const agentContext = { teamId, agentId }
|
||||
systemTools.push(
|
||||
// Shared Operations (also exposed via MCP and the CLI): list/create/rename/move/delete,
|
||||
// read_asset and get_asset_render
|
||||
// read_asset, get_asset_render, and download_asset
|
||||
...buildOperationTools({ userId, agentId, teamId, projectId }),
|
||||
createCreateFileTool(userId, metadataSchema, agentContext),
|
||||
createCreateVersionTool(userId, metadataSchema, agentContext),
|
||||
createDownloadAssetTool(userId),
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,294 @@
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'
|
||||
import * as fs from 'fs'
|
||||
import * as path from 'path'
|
||||
import { prisma, type Asset, type StorageKey } from '@shumai/db'
|
||||
import { s3Service } from '@shumai/core/src/s3/s3'
|
||||
import { authzService } from '@shumai/core/src/authz/authz'
|
||||
import { getOperation } from '@shumai/core/src/operations'
|
||||
import { toAgentTool } from '../operation-tools'
|
||||
import './download-asset'
|
||||
|
||||
vi.mock('@shumai/db', async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import('@shumai/db')>()
|
||||
return {
|
||||
...actual,
|
||||
prisma: {
|
||||
asset: { findUnique: vi.fn(), findFirst: vi.fn() },
|
||||
storageKey: { findFirst: vi.fn(), findUnique: vi.fn() },
|
||||
},
|
||||
}
|
||||
})
|
||||
|
||||
vi.mock('@shumai/core/src/s3/s3', async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import('@shumai/core/src/s3/s3')>()
|
||||
return {
|
||||
...actual,
|
||||
s3Service: {
|
||||
headObject: vi.fn(),
|
||||
downloadToFile: vi.fn(),
|
||||
presign: vi.fn().mockResolvedValue('https://signed.example/file'),
|
||||
},
|
||||
}
|
||||
})
|
||||
|
||||
vi.mock('@shumai/core/src/authz/authz', async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import('@shumai/core/src/authz/authz')>()
|
||||
return { ...actual, authzService: { hasPermission: vi.fn() } }
|
||||
})
|
||||
|
||||
type Head = Awaited<ReturnType<typeof s3Service.headObject>>
|
||||
type Saved = { filePath: string; absolutePath: string; contentType: string; size: number }
|
||||
|
||||
const piDir = path.join(process.cwd(), '.pi')
|
||||
|
||||
function downloadTool(userId = 'user-1') {
|
||||
return toAgentTool(getOperation('download_asset')!, { userId })
|
||||
}
|
||||
|
||||
function textOf(result: { content: unknown[] }): string {
|
||||
return (result.content[0] as { text: string }).text
|
||||
}
|
||||
|
||||
function mockAssets(...assets: Array<Record<string, unknown>>) {
|
||||
vi.mocked(prisma.asset.findUnique).mockImplementation(
|
||||
(async (args: { where: { id: string } }) =>
|
||||
assets.find((a) => a.id === args.where.id) ?? null) as never,
|
||||
)
|
||||
}
|
||||
|
||||
function mockStorage(contentType: string, write = 'bytes') {
|
||||
vi.mocked(s3Service.headObject).mockResolvedValue({ size: 1234, contentType } as Head)
|
||||
vi.mocked(s3Service.downloadToFile).mockImplementation(async (_bucket, _key, dest) => {
|
||||
await fs.promises.writeFile(dest, write)
|
||||
})
|
||||
}
|
||||
|
||||
const photo = {
|
||||
id: 'img-1',
|
||||
name: 'sample_photo.jpg',
|
||||
type: 'file',
|
||||
isDeleted: false,
|
||||
storageKey: { key: 'files/up-1/sample_photo.jpg' },
|
||||
media: { proxyType: 'image', imageTranscodes: [{ key: 'files/img-1/proxy.webp' }] },
|
||||
}
|
||||
|
||||
describe('download_asset agent tool', () => {
|
||||
const created: string[] = []
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
vi.mocked(authzService.hasPermission).mockResolvedValue()
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
for (const file of created.splice(0)) fs.rmSync(file, { force: true })
|
||||
})
|
||||
|
||||
it('offers assetIds, proxy, and the agent-only key parameter', () => {
|
||||
const tool = downloadTool()
|
||||
const schema = tool.parameters as { properties: Record<string, unknown>; required?: string[] }
|
||||
expect(Object.keys(schema.properties)).toEqual(['assetIds', 'proxy', 'key'])
|
||||
expect(schema.required ?? []).not.toContain('assetIds')
|
||||
expect(tool.description).toContain('.pi directory')
|
||||
})
|
||||
|
||||
it('saves the web preview into .pi by default, named after the asset', async () => {
|
||||
mockAssets(photo)
|
||||
mockStorage('image/webp', 'proxy-bytes')
|
||||
|
||||
const result = await downloadTool().execute('call-1', { assetIds: ['img-1'] })
|
||||
|
||||
const expectedPath = path.join(piDir, 'img-1_sample_photo.webp')
|
||||
created.push(expectedPath)
|
||||
expect(authzService.hasPermission).toHaveBeenCalledWith({
|
||||
user: { id: 'user-1' },
|
||||
permission: 'Read',
|
||||
type: 'asset',
|
||||
id: 'img-1',
|
||||
})
|
||||
expect(s3Service.downloadToFile).toHaveBeenCalledWith(
|
||||
'shumai',
|
||||
'files/img-1/proxy.webp',
|
||||
expectedPath,
|
||||
)
|
||||
expect(fs.readFileSync(expectedPath, 'utf-8')).toBe('proxy-bytes')
|
||||
expect(textOf(result)).toBe(
|
||||
[
|
||||
`Downloaded "sample_photo.webp" (ID: img-1, proxy, image/webp, 11 B) to "${path.join('.pi', 'img-1_sample_photo.webp')}".`,
|
||||
'Remember to delete these temporary files when finished.',
|
||||
].join('\n'),
|
||||
)
|
||||
const details = result.details as { files: Saved[] }
|
||||
expect(details.files[0]).toMatchObject({
|
||||
filePath: path.join('.pi', 'img-1_sample_photo.webp'),
|
||||
contentType: 'image/webp',
|
||||
size: 11,
|
||||
})
|
||||
})
|
||||
|
||||
it('saves the original with proxy false', async () => {
|
||||
mockAssets(photo)
|
||||
mockStorage('image/jpeg')
|
||||
|
||||
await downloadTool().execute('call-1', { assetIds: ['img-1'], proxy: false })
|
||||
|
||||
const expectedPath = path.join(piDir, 'img-1_sample_photo.jpg')
|
||||
created.push(expectedPath)
|
||||
expect(s3Service.downloadToFile).toHaveBeenCalledWith(
|
||||
'shumai',
|
||||
'files/up-1/sample_photo.jpg',
|
||||
expectedPath,
|
||||
)
|
||||
})
|
||||
|
||||
it('downloads several files and the latest version of a version stack', async () => {
|
||||
mockAssets(photo, { id: 'stack-1', name: 'banana', type: 'version_stack', isDeleted: false })
|
||||
vi.mocked(prisma.asset.findFirst).mockResolvedValue({
|
||||
id: 'file-v2',
|
||||
name: 'banana_pig.png',
|
||||
type: 'file',
|
||||
isDeleted: false,
|
||||
storageKey: { key: 'files/up-2/banana_pig.png' },
|
||||
media: { proxyType: 'image', imageTranscodes: [{ key: 'files/file-v2/proxy.webp' }] },
|
||||
} as unknown as Asset)
|
||||
mockStorage('image/webp')
|
||||
|
||||
const result = await downloadTool().execute('call-1', {
|
||||
assetIds: ['img-1', 'stack-1'],
|
||||
key: null,
|
||||
})
|
||||
|
||||
created.push(
|
||||
path.join(piDir, 'img-1_sample_photo.webp'),
|
||||
path.join(piDir, 'file-v2_banana_pig.webp'),
|
||||
)
|
||||
expect(prisma.asset.findFirst).toHaveBeenCalledWith({
|
||||
where: { parentId: 'stack-1', isDeleted: false },
|
||||
orderBy: { sortIndex: 'asc' },
|
||||
include: { storageKey: true },
|
||||
})
|
||||
expect(s3Service.downloadToFile).toHaveBeenCalledTimes(2)
|
||||
expect(textOf(result)).toContain('Downloaded "banana_pig.webp" (ID: file-v2, proxy')
|
||||
})
|
||||
|
||||
it('rejects folders', async () => {
|
||||
mockAssets({ id: 'folder-1', name: 'Renders', type: 'folder', isDeleted: false })
|
||||
await expect(downloadTool().execute('call-1', { assetIds: ['folder-1'] })).rejects.toThrow(
|
||||
'Asset folder-1 ("Renders") is a folder. Download the files inside it instead.',
|
||||
)
|
||||
expect(s3Service.downloadToFile).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('checks Read on every asset before downloading anything', async () => {
|
||||
mockAssets(photo)
|
||||
vi.mocked(authzService.hasPermission).mockImplementation(async ({ id }) => {
|
||||
if (id === 'secret') throw new Error('Forbidden')
|
||||
})
|
||||
await expect(
|
||||
downloadTool().execute('call-1', { assetIds: ['img-1', 'secret'] }),
|
||||
).rejects.toThrow('Forbidden')
|
||||
expect(s3Service.downloadToFile).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('requires assetIds or key', async () => {
|
||||
await expect(downloadTool().execute('call-1', { assetIds: null, key: null })).rejects.toThrow(
|
||||
'Invalid input for download_asset',
|
||||
)
|
||||
})
|
||||
|
||||
it('rejects assetIds and key together', async () => {
|
||||
await expect(
|
||||
downloadTool().execute('call-1', { assetIds: ['img-1'], key: 'files/img-1/a.webp' }),
|
||||
).rejects.toThrow('Provide either "assetIds" or "key", not both.')
|
||||
})
|
||||
|
||||
describe('by storage key', () => {
|
||||
it('requires a user', async () => {
|
||||
await expect(
|
||||
downloadTool('').execute('call-1', { key: 'files/ast-1/shot.webp' }),
|
||||
).rejects.toThrow('User ID is required for authorization.')
|
||||
})
|
||||
|
||||
it('rejects keys outside files/ and path traversal', async () => {
|
||||
await expect(downloadTool().execute('call-1', { key: 'avatars/user-1.png' })).rejects.toThrow(
|
||||
'Invalid storage key format: key must start with "files/"',
|
||||
)
|
||||
await expect(
|
||||
downloadTool().execute('call-1', { key: 'files/../../etc/passwd' }),
|
||||
).rejects.toThrow('cannot contain ".."')
|
||||
})
|
||||
|
||||
it('rejects keys that belong to no asset', async () => {
|
||||
vi.mocked(prisma.asset.findUnique).mockResolvedValue(null)
|
||||
vi.mocked(prisma.storageKey.findFirst).mockResolvedValue(null)
|
||||
vi.mocked(prisma.storageKey.findUnique).mockResolvedValue(null)
|
||||
await expect(
|
||||
downloadTool().execute('call-1', { key: 'files/nope/shot.webp' }),
|
||||
).rejects.toThrow('does not belong to any valid asset or the asset has been deleted')
|
||||
})
|
||||
|
||||
it('checks Read on the owning asset', async () => {
|
||||
mockAssets({ id: 'other-project-asset' })
|
||||
vi.mocked(authzService.hasPermission).mockRejectedValue(new Error('Forbidden'))
|
||||
await expect(
|
||||
downloadTool().execute('call-1', { key: 'files/other-project-asset/shots/a.webp' }),
|
||||
).rejects.toThrow('Forbidden')
|
||||
expect(authzService.hasPermission).toHaveBeenCalledWith({
|
||||
user: { id: 'user-1' },
|
||||
permission: 'Read',
|
||||
type: 'asset',
|
||||
id: 'other-project-asset',
|
||||
})
|
||||
expect(s3Service.downloadToFile).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('downloads a read_asset artifact into .pi', async () => {
|
||||
mockAssets({ id: 'ast-123' })
|
||||
mockStorage('image/webp', 'frame-bytes')
|
||||
|
||||
const result = await downloadTool().execute('call-1', {
|
||||
key: 'files/ast-123/screenshots/shot_5.0s.webp',
|
||||
})
|
||||
|
||||
const expectedPath = path.join(piDir, 'shot_5.0s.webp')
|
||||
created.push(expectedPath)
|
||||
expect(fs.readFileSync(expectedPath, 'utf-8')).toBe('frame-bytes')
|
||||
expect(textOf(result)).toBe(
|
||||
`Downloaded storage key "files/ast-123/screenshots/shot_5.0s.webp" to "${path.join('.pi', 'shot_5.0s.webp')}". Remember to delete this temporary file when finished.`,
|
||||
)
|
||||
expect(result.details).toMatchObject({
|
||||
key: 'files/ast-123/screenshots/shot_5.0s.webp',
|
||||
assetId: 'ast-123',
|
||||
size: Buffer.from('frame-bytes').length,
|
||||
})
|
||||
})
|
||||
|
||||
it('resolves original upload keys through the storage key', async () => {
|
||||
vi.mocked(prisma.asset.findUnique).mockResolvedValue(null)
|
||||
vi.mocked(prisma.storageKey.findFirst).mockResolvedValue({
|
||||
id: 'sk-1',
|
||||
key: 'files/upload-ulid-456/photo.png',
|
||||
assets: [{ id: 'asset-photo-1' }],
|
||||
} as unknown as StorageKey & { assets: Asset[] })
|
||||
const pngHeader = Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a])
|
||||
vi.mocked(s3Service.headObject).mockResolvedValue({
|
||||
size: 8,
|
||||
contentType: 'application/octet-stream',
|
||||
} as Head)
|
||||
vi.mocked(s3Service.downloadToFile).mockImplementation(async (_b, _k, dest) => {
|
||||
await fs.promises.writeFile(dest, pngHeader)
|
||||
})
|
||||
|
||||
const result = await downloadTool().execute('call-1', {
|
||||
key: 'files/upload-ulid-456/photo.png',
|
||||
})
|
||||
|
||||
created.push(path.join(piDir, 'photo.png'))
|
||||
expect(authzService.hasPermission).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ id: 'asset-photo-1' }),
|
||||
)
|
||||
// Falls back to detecting the type from the bytes
|
||||
expect(result.details).toMatchObject({ assetId: 'asset-photo-1', contentType: 'image/png' })
|
||||
})
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,124 @@
|
||||
import * as fs from 'fs'
|
||||
import * as path from 'path'
|
||||
import { z } from 'zod'
|
||||
import type { AgentToolResult } from '@earendil-works/pi-agent-core'
|
||||
import { downloadAsset, MAX_DOWNLOAD_ASSETS, type DownloadAssetOutput } from '@shumai/operations'
|
||||
import type { User } from '@shumai/db'
|
||||
import { authzService, Permission, ResourceType } from '@shumai/core/src/authz/authz'
|
||||
import { storageBucket } from '@shumai/core/src/asset-read'
|
||||
import { s3Service } from '@shumai/core/src/s3/s3'
|
||||
import { sanitizeFilename } from '@shumai/core/src/utils/filename'
|
||||
import { readFileMimeType } from '@shumai/core/src/utils/file-mime'
|
||||
import { resolveAssetIdFromKey } from '../tools/download-asset'
|
||||
import { registerAgentToolExtension, type AgentToolExtension } from '../operation-tools'
|
||||
|
||||
const CLEANUP_REMINDER = 'Remember to delete these temporary files when finished.'
|
||||
|
||||
/** The agent's sandbox, where downloaded files are saved for local scripts and bash. */
|
||||
function sandboxDir(): string {
|
||||
const dir = path.join(process.cwd(), '.pi')
|
||||
fs.mkdirSync(dir, { recursive: true })
|
||||
return dir
|
||||
}
|
||||
|
||||
function formatSize(bytes: number): string {
|
||||
if (bytes < 1024) return `${bytes} B`
|
||||
if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(1)} KB`
|
||||
return `${(bytes / (1024 * 1024)).toFixed(1)} MB`
|
||||
}
|
||||
|
||||
/** Saves a storage object into the sandbox; returns the paths and detected content type. */
|
||||
async function saveToSandbox(key: string, filename: string) {
|
||||
const dir = sandboxDir()
|
||||
const absolutePath = path.join(dir, filename)
|
||||
const bucket = storageBucket()
|
||||
const info = await s3Service.headObject(bucket, key).catch(() => null)
|
||||
await s3Service.downloadToFile(bucket, key, absolutePath)
|
||||
const size = await fs.promises
|
||||
.stat(absolutePath)
|
||||
.then((s) => s.size)
|
||||
.catch(() => info?.size ?? 0)
|
||||
const contentType =
|
||||
info?.contentType && info.contentType !== 'application/octet-stream'
|
||||
? info.contentType
|
||||
: readFileMimeType(absolutePath)
|
||||
return { absolutePath, filePath: path.join('.pi', filename), size, contentType }
|
||||
}
|
||||
|
||||
/** Downloads one storage key (e.g. a frame from read_asset) after checking Read on its asset. */
|
||||
async function downloadKey(key: string, userId: string): Promise<AgentToolResult<unknown>> {
|
||||
const owningAssetId = await resolveAssetIdFromKey(key)
|
||||
await authzService.hasPermission({
|
||||
user: { id: userId } as User,
|
||||
permission: Permission.Read,
|
||||
type: ResourceType.Asset,
|
||||
id: owningAssetId,
|
||||
})
|
||||
const filename = sanitizeFilename(path.basename(key)) || 'downloaded_file'
|
||||
const saved = await saveToSandbox(key, filename)
|
||||
return {
|
||||
content: [
|
||||
{
|
||||
type: 'text',
|
||||
text: `Downloaded storage key "${key}" to "${saved.filePath}". Remember to delete this temporary file when finished.`,
|
||||
},
|
||||
],
|
||||
details: { key, assetId: owningAssetId, ...saved },
|
||||
}
|
||||
}
|
||||
|
||||
/** Saves each file of a download_asset result into the sandbox. */
|
||||
export async function presentDownloadOutput(
|
||||
output: DownloadAssetOutput,
|
||||
): Promise<AgentToolResult<unknown>> {
|
||||
const lines: string[] = []
|
||||
const files = []
|
||||
for (const file of output.files) {
|
||||
if (!file.key) throw new Error(`download_asset returned no storage key for ${file.assetId}`)
|
||||
const saved = await saveToSandbox(
|
||||
file.key,
|
||||
`${file.assetId}_${sanitizeFilename(file.name) || 'file'}`,
|
||||
)
|
||||
lines.push(
|
||||
`Downloaded "${file.name}" (ID: ${file.assetId}, ${file.kind}, ${saved.contentType}, ${formatSize(saved.size)}) to "${saved.filePath}".`,
|
||||
)
|
||||
files.push({ assetId: file.assetId, name: file.name, kind: file.kind, ...saved })
|
||||
}
|
||||
lines.push(CLEANUP_REMINDER)
|
||||
return { content: [{ type: 'text', text: lines.join('\n') }], details: { files } }
|
||||
}
|
||||
|
||||
const downloadAssetExtension: AgentToolExtension = {
|
||||
description:
|
||||
`Downloads up to ${MAX_DOWNLOAD_ASSETS} workspace files into the local .pi directory, for processing with local scripts or bash commands. ` +
|
||||
'By default it saves the web preview version (proxy) when there is one, e.g. the transcoded video or the web image; set proxy to false for the original uploaded file. ' +
|
||||
'A version stack downloads its latest version; folders are not supported. ' +
|
||||
'To download a specific artifact from read_asset (e.g. a frame or page), pass its storage key as "key" instead of assetIds. ' +
|
||||
'Remember to delete the downloaded files when finished.',
|
||||
parameterOverrides: z.object({
|
||||
assetIds: downloadAsset.input.shape.assetIds
|
||||
.optional()
|
||||
.describe(
|
||||
`IDs of the files or version stacks to download (1 to ${MAX_DOWNLOAD_ASSETS}). Omit when passing "key".`,
|
||||
),
|
||||
}),
|
||||
extraParameters: z.object({
|
||||
key: z
|
||||
.string()
|
||||
.optional()
|
||||
.describe(
|
||||
'Storage S3 key of a specific media artifact to download instead of assetIds, e.g. a screenshot or page key from read_asset.',
|
||||
),
|
||||
}),
|
||||
async intercept(input, extra, ctx) {
|
||||
if (typeof extra.key !== 'string') return undefined
|
||||
if (Array.isArray(input.assetIds) && input.assetIds.length > 0) {
|
||||
throw new Error('Provide either "assetIds" or "key", not both.')
|
||||
}
|
||||
if (!ctx.userId) throw new Error('User ID is required for authorization.')
|
||||
return downloadKey(extra.key, ctx.userId)
|
||||
},
|
||||
present: (output) => presentDownloadOutput(output as DownloadAssetOutput),
|
||||
}
|
||||
|
||||
registerAgentToolExtension('download_asset', downloadAssetExtension)
|
||||
@@ -1,2 +1,3 @@
|
||||
// Registers agent-specific presentation for Operations. Import once before building tools.
|
||||
export { createAutofillReadAssetTool, presentReadOutput } from './read-asset'
|
||||
export { presentDownloadOutput } from './download-asset'
|
||||
|
||||
@@ -752,7 +752,7 @@ describe('read_asset agent tool', () => {
|
||||
assetId: 'doc-md',
|
||||
docConfig: { mode: 'text', startPage: 1, endPage: 2 },
|
||||
}),
|
||||
).rejects.toThrow('startPage and endPage must be null when docConfig mode is "text"')
|
||||
).rejects.toThrow('startPage and endPage must be left out when docConfig mode is "text"')
|
||||
})
|
||||
|
||||
it('reads raw text in mode "text"', async () => {
|
||||
|
||||
@@ -61,7 +61,7 @@ describe('operation-backed agent tools', () => {
|
||||
})
|
||||
|
||||
describe('tool definitions', () => {
|
||||
it('keeps the existing tool names and adds list_projects and get_asset_render', () => {
|
||||
it('keeps the existing tool names and adds the newer Operations', () => {
|
||||
expect(
|
||||
buildOperationTools(ctx)
|
||||
.map((t) => t.name)
|
||||
@@ -69,6 +69,7 @@ describe('operation-backed agent tools', () => {
|
||||
).toEqual([
|
||||
'create_folder',
|
||||
'delete_asset',
|
||||
'download_asset',
|
||||
'get_asset_render',
|
||||
'list_assets',
|
||||
'list_projects',
|
||||
|
||||
@@ -2,6 +2,7 @@ import { z } from 'zod'
|
||||
import type { TSchema } from 'typebox'
|
||||
import { Compile } from 'typebox/compile'
|
||||
import { type AgentTool, type AgentToolResult } from '@earendil-works/pi-agent-core'
|
||||
import { dropNullOptionals } from '@shumai/operations'
|
||||
import {
|
||||
getOperationsForSurface,
|
||||
runOperation,
|
||||
@@ -18,10 +19,27 @@ export type AgentOperationContext = Omit<OperationContext, 'via'>
|
||||
* and render the output (e.g. as image blocks).
|
||||
*/
|
||||
export interface AgentToolExtension {
|
||||
/** Replaces the Operation description for the agent (e.g. where files end up). */
|
||||
description?: string
|
||||
/** Extra parameters only the agent sees; they are removed before the Operation runs. */
|
||||
extraParameters?: z.ZodObject
|
||||
/**
|
||||
* Agent-facing versions of Operation input fields, e.g. a required field made optional
|
||||
* because an extra parameter can be used instead. The Operation still validates its own
|
||||
* input when it runs.
|
||||
*/
|
||||
parameterOverrides?: z.ZodObject
|
||||
/** Adjust the Operation input using the extra parameters. */
|
||||
toInput?(input: Record<string, unknown>, extra: Record<string, unknown>): Record<string, unknown>
|
||||
/**
|
||||
* Handle an agent-only mode without running the Operation (e.g. a storage key instead of
|
||||
* an asset). Return undefined to run the Operation.
|
||||
*/
|
||||
intercept?(
|
||||
input: Record<string, unknown>,
|
||||
extra: Record<string, unknown>,
|
||||
ctx: AgentOperationContext,
|
||||
): Promise<AgentToolResult<unknown> | undefined>
|
||||
present(output: unknown, extra: Record<string, unknown>): Promise<AgentToolResult<unknown>>
|
||||
}
|
||||
|
||||
@@ -33,9 +51,10 @@ export function registerAgentToolExtension(name: string, extension: AgentToolExt
|
||||
}
|
||||
|
||||
function toolInputSchema(op: Operation, extension?: AgentToolExtension): z.ZodObject {
|
||||
return extension?.extraParameters
|
||||
? op.definition.input.extend(extension.extraParameters.shape)
|
||||
: op.definition.input
|
||||
let schema = op.definition.input
|
||||
if (extension?.parameterOverrides) schema = schema.extend(extension.parameterOverrides.shape)
|
||||
if (extension?.extraParameters) schema = schema.extend(extension.extraParameters.shape)
|
||||
return schema
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -53,27 +72,13 @@ export function toToolParameters(op: Operation, extension?: AgentToolExtension):
|
||||
return schema as unknown as TSchema
|
||||
}
|
||||
|
||||
/**
|
||||
* Models sometimes send `null` for optional parameters they don't want to set. Drop those
|
||||
* so the input schema treats them as omitted; nulls for required fields still fail.
|
||||
*/
|
||||
export function dropNullOptionalFields(schema: z.ZodObject, params: unknown): unknown {
|
||||
if (!params || typeof params !== 'object' || Array.isArray(params)) return params
|
||||
const shape = schema.shape
|
||||
return Object.fromEntries(
|
||||
Object.entries(params).filter(
|
||||
([key, value]) => !(value === null && shape[key]?.safeParse(undefined).success),
|
||||
),
|
||||
)
|
||||
}
|
||||
|
||||
/** Split agent-only parameters from the Operation input. */
|
||||
export function splitAgentParams(
|
||||
op: Operation,
|
||||
extension: AgentToolExtension | undefined,
|
||||
params: unknown,
|
||||
): { input: unknown; extra: Record<string, unknown> } {
|
||||
const cleaned = dropNullOptionalFields(toolInputSchema(op, extension), params)
|
||||
const cleaned = dropNullOptionals(toolInputSchema(op, extension), params)
|
||||
if (!extension?.extraParameters || !cleaned || typeof cleaned !== 'object') {
|
||||
return { input: cleaned, extra: {} }
|
||||
}
|
||||
@@ -93,10 +98,14 @@ export function toAgentTool(op: Operation, ctx: AgentOperationContext): AgentToo
|
||||
return {
|
||||
name: definition.name,
|
||||
label: definition.title,
|
||||
description: definition.description,
|
||||
description: extension?.description ?? definition.description,
|
||||
parameters: toToolParameters(op, extension),
|
||||
execute: async (_toolCallId, params) => {
|
||||
const { input, extra } = splitAgentParams(op, extension, params)
|
||||
if (extension?.intercept) {
|
||||
const handled = await extension.intercept(input as Record<string, unknown>, extra, ctx)
|
||||
if (handled) return handled
|
||||
}
|
||||
const output = await runOperation(op, { ...ctx, via: 'agent' }, input)
|
||||
if (extension) {
|
||||
return extension.present(output, extra)
|
||||
|
||||
@@ -1,14 +1,8 @@
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'
|
||||
import {
|
||||
createDownloadAssetTool,
|
||||
resolveAssetIdFromKey,
|
||||
resolveS3MediaBuffer,
|
||||
} from './download-asset'
|
||||
import { describe, it, expect, vi, beforeEach } from 'vitest'
|
||||
import { resolveAssetIdFromKey, resolveS3MediaBuffer } from './download-asset'
|
||||
import { prisma, type Asset, type StorageKey } from '@shumai/db'
|
||||
import { s3Service } from '@shumai/core/src/s3/s3'
|
||||
import { authzService } from '@shumai/core/src/authz/authz'
|
||||
import * as fs from 'fs'
|
||||
import * as path from 'path'
|
||||
|
||||
vi.mock('@shumai/db', async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import('@shumai/db')>()
|
||||
@@ -43,333 +37,11 @@ vi.mock('@shumai/core/src/authz/authz', () => ({
|
||||
ResourceType: { Asset: 'asset' },
|
||||
}))
|
||||
|
||||
describe('downloadAssetTool', () => {
|
||||
const piDir = path.join(process.cwd(), '.pi')
|
||||
const createdFiles: string[] = []
|
||||
|
||||
describe('download-asset storage key helpers', () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
for (const f of createdFiles) {
|
||||
if (fs.existsSync(f)) {
|
||||
fs.unlinkSync(f)
|
||||
}
|
||||
}
|
||||
createdFiles.length = 0
|
||||
})
|
||||
|
||||
it('should throw authorization error if user ID is empty', async () => {
|
||||
const tool = createDownloadAssetTool('')
|
||||
await expect(tool.execute('call-1', { assetId: 'asset-1', key: null })).rejects.toThrow(
|
||||
'User ID is required for authorization.',
|
||||
)
|
||||
})
|
||||
|
||||
it('should throw error if both assetId and key are null', async () => {
|
||||
const tool = createDownloadAssetTool('user-1')
|
||||
await expect(tool.execute('call-1', { assetId: null, key: null })).rejects.toThrow(
|
||||
'Provide exactly one of "assetId" (to download a workspace asset) or "key" (to download a specific S3 storage key). Set the unused parameter to null.',
|
||||
)
|
||||
})
|
||||
|
||||
it('should throw error if both assetId and key are provided', async () => {
|
||||
const tool = createDownloadAssetTool('user-1')
|
||||
await expect(
|
||||
tool.execute('call-1', { assetId: 'asset-1', key: 'files/path/file.png' }),
|
||||
).rejects.toThrow(
|
||||
'Provide exactly one of "assetId" (to download a workspace asset) or "key" (to download a specific S3 storage key). Set the unused parameter to null.',
|
||||
)
|
||||
})
|
||||
|
||||
describe('Download by assetId', () => {
|
||||
it('should throw error if asset not found', async () => {
|
||||
vi.mocked(authzService.hasPermission).mockResolvedValue()
|
||||
vi.mocked(prisma.asset.findUnique).mockResolvedValue(null)
|
||||
|
||||
const tool = createDownloadAssetTool('user-1')
|
||||
await expect(tool.execute('call-1', { assetId: 'asset-1', key: null })).rejects.toThrow(
|
||||
'Asset with ID asset-1 not found.',
|
||||
)
|
||||
})
|
||||
|
||||
it('should download proxy image to .pi directory when downloading by assetId', async () => {
|
||||
vi.mocked(authzService.hasPermission).mockResolvedValue()
|
||||
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
|
||||
id: 'asset-img-1',
|
||||
name: 'sample_photo.jpg',
|
||||
storageKey: { key: 'raw/sample_photo.jpg' },
|
||||
media: {
|
||||
proxyType: 'image',
|
||||
imageTranscodes: [{ key: 'proxy/sample_photo.webp' }],
|
||||
},
|
||||
} as unknown as Asset)
|
||||
|
||||
vi.mocked(s3Service.downloadToFile).mockImplementation(async (_b, _k, dest) => {
|
||||
await fs.promises.writeFile(dest, 'fake-proxy-image-bytes')
|
||||
})
|
||||
vi.mocked(s3Service.headObject).mockResolvedValue({
|
||||
contentType: 'image/webp',
|
||||
} as unknown as Awaited<ReturnType<typeof s3Service.headObject>>)
|
||||
|
||||
const tool = createDownloadAssetTool('user-1')
|
||||
const result = await tool.execute('call-1', { assetId: 'asset-img-1', key: null })
|
||||
|
||||
expect(authzService.hasPermission).toHaveBeenCalledWith({
|
||||
user: { id: 'user-1' },
|
||||
permission: 'Read',
|
||||
type: 'asset',
|
||||
id: 'asset-img-1',
|
||||
})
|
||||
|
||||
const expectedPath = path.join(piDir, 'asset-img-1_sample_photo.jpg')
|
||||
createdFiles.push(expectedPath)
|
||||
|
||||
expect(s3Service.downloadToFile).toHaveBeenCalledWith(
|
||||
'shumai',
|
||||
'proxy/sample_photo.webp',
|
||||
expectedPath,
|
||||
)
|
||||
|
||||
expect(fs.existsSync(expectedPath)).toBe(true)
|
||||
expect(fs.readFileSync(expectedPath, 'utf-8')).toBe('fake-proxy-image-bytes')
|
||||
|
||||
expect(result.details.filePath).toBe(path.join('.pi', 'asset-img-1_sample_photo.jpg'))
|
||||
expect(result.details.contentType).toBe('image/webp')
|
||||
expect((result.content[0] as { type: 'text'; text: string }).text).toContain(
|
||||
'Remember to delete this temporary file when finished.',
|
||||
)
|
||||
})
|
||||
|
||||
it('should resolve version_stack asset to its latest version file', async () => {
|
||||
vi.mocked(authzService.hasPermission).mockResolvedValue()
|
||||
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
|
||||
id: 'stack-1',
|
||||
name: '',
|
||||
type: 'version_stack',
|
||||
storageKey: null,
|
||||
media: null,
|
||||
} as unknown as Asset)
|
||||
|
||||
vi.mocked(prisma.asset.findFirst).mockResolvedValue({
|
||||
id: 'file-ver-2',
|
||||
name: 'banana_pig.png',
|
||||
type: 'file',
|
||||
storageKey: { key: 'raw/banana_pig.png' },
|
||||
media: {
|
||||
proxyType: 'image',
|
||||
imageTranscodes: [{ key: 'proxy/banana_pig.webp' }],
|
||||
},
|
||||
} as unknown as Asset)
|
||||
|
||||
vi.mocked(s3Service.downloadToFile).mockImplementation(async (_b, _k, dest) => {
|
||||
await fs.promises.writeFile(dest, 'fake-version-bytes')
|
||||
})
|
||||
vi.mocked(s3Service.headObject).mockResolvedValue({
|
||||
contentType: 'image/webp',
|
||||
} as unknown as Awaited<ReturnType<typeof s3Service.headObject>>)
|
||||
|
||||
const tool = createDownloadAssetTool('user-1')
|
||||
const result = await tool.execute('call-1', { assetId: 'stack-1', key: null })
|
||||
|
||||
expect(prisma.asset.findFirst).toHaveBeenCalledWith({
|
||||
where: { parentId: 'stack-1', isDeleted: false },
|
||||
orderBy: { sortIndex: 'asc' },
|
||||
include: { storageKey: true },
|
||||
})
|
||||
|
||||
const expectedPath = path.join(piDir, 'file-ver-2_banana_pig.png')
|
||||
createdFiles.push(expectedPath)
|
||||
|
||||
expect(s3Service.downloadToFile).toHaveBeenCalledWith(
|
||||
'shumai',
|
||||
'proxy/banana_pig.webp',
|
||||
expectedPath,
|
||||
)
|
||||
|
||||
expect(fs.existsSync(expectedPath)).toBe(true)
|
||||
expect(result.details.name).toBe('banana_pig.png')
|
||||
})
|
||||
})
|
||||
|
||||
describe('Download by S3 key and Authorization', () => {
|
||||
it('should reject keys that do not start with files/', async () => {
|
||||
const tool = createDownloadAssetTool('user-1')
|
||||
await expect(
|
||||
tool.execute('call-1', { assetId: null, key: 'avatars/user-1.png' }),
|
||||
).rejects.toThrow('Invalid storage key format: key must start with "files/"')
|
||||
})
|
||||
|
||||
it('should reject keys with path traversal (..)', async () => {
|
||||
const tool = createDownloadAssetTool('user-1')
|
||||
await expect(
|
||||
tool.execute('call-1', { assetId: null, key: 'files/../../etc/passwd' }),
|
||||
).rejects.toThrow('cannot contain ".."')
|
||||
})
|
||||
|
||||
it('should reject keys with insufficient path segments', async () => {
|
||||
const tool = createDownloadAssetTool('user-1')
|
||||
await expect(tool.execute('call-1', { assetId: null, key: 'files/invalid' })).rejects.toThrow(
|
||||
'Invalid storage key format',
|
||||
)
|
||||
})
|
||||
|
||||
it('should reject keys that do not belong to any valid asset in database', async () => {
|
||||
vi.mocked(prisma.asset.findUnique).mockResolvedValue(null)
|
||||
vi.mocked(prisma.storageKey.findFirst).mockResolvedValue(null)
|
||||
vi.mocked(prisma.storageKey.findUnique).mockResolvedValue(null)
|
||||
|
||||
const tool = createDownloadAssetTool('user-1')
|
||||
await expect(
|
||||
tool.execute('call-1', { assetId: null, key: 'files/nonexistent-id/shot.webp' }),
|
||||
).rejects.toThrow('does not belong to any valid asset or the asset has been deleted')
|
||||
})
|
||||
|
||||
it('should reject download when user lacks read permission on the owning asset (cross-project protection)', async () => {
|
||||
// Key belongs to asset-in-other-project
|
||||
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
|
||||
id: 'asset-in-other-project',
|
||||
} as unknown as Asset)
|
||||
|
||||
// Authz rejects because user is in a different project/team
|
||||
vi.mocked(authzService.hasPermission).mockRejectedValue(
|
||||
new Error('User does not have Read permission on asset asset-in-other-project'),
|
||||
)
|
||||
|
||||
const tool = createDownloadAssetTool('user-1')
|
||||
await expect(
|
||||
tool.execute('call-1', {
|
||||
assetId: null,
|
||||
key: 'files/asset-in-other-project/screenshots/secret_frame.webp',
|
||||
}),
|
||||
).rejects.toThrow('User does not have Read permission on asset asset-in-other-project')
|
||||
|
||||
expect(authzService.hasPermission).toHaveBeenCalledWith({
|
||||
user: { id: 'user-1' },
|
||||
permission: 'Read',
|
||||
type: 'asset',
|
||||
id: 'asset-in-other-project',
|
||||
})
|
||||
expect(s3Service.downloadToFile).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('should authorize and download derived artifact key (files/<assetId>/...)', async () => {
|
||||
vi.mocked(prisma.asset.findUnique).mockResolvedValue({
|
||||
id: 'ast-123',
|
||||
} as unknown as Asset)
|
||||
vi.mocked(authzService.hasPermission).mockResolvedValue()
|
||||
|
||||
vi.mocked(s3Service.downloadToFile).mockImplementation(async (_b, _k, dest) => {
|
||||
await fs.promises.writeFile(dest, 'fake-screenshot-bytes')
|
||||
})
|
||||
vi.mocked(s3Service.headObject).mockResolvedValue({
|
||||
contentType: 'image/webp',
|
||||
} as unknown as Awaited<ReturnType<typeof s3Service.headObject>>)
|
||||
|
||||
const tool = createDownloadAssetTool('user-1')
|
||||
const result = await tool.execute('call-1', {
|
||||
assetId: null,
|
||||
key: 'files/ast-123/screenshots/shot_5.0s.webp',
|
||||
})
|
||||
|
||||
expect(authzService.hasPermission).toHaveBeenCalledWith({
|
||||
user: { id: 'user-1' },
|
||||
permission: 'Read',
|
||||
type: 'asset',
|
||||
id: 'ast-123',
|
||||
})
|
||||
|
||||
const expectedPath = path.join(piDir, 'shot_5.0s.webp')
|
||||
createdFiles.push(expectedPath)
|
||||
|
||||
expect(s3Service.downloadToFile).toHaveBeenCalledWith(
|
||||
'shumai',
|
||||
'files/ast-123/screenshots/shot_5.0s.webp',
|
||||
expectedPath,
|
||||
)
|
||||
|
||||
expect(fs.existsSync(expectedPath)).toBe(true)
|
||||
expect(fs.readFileSync(expectedPath, 'utf-8')).toBe('fake-screenshot-bytes')
|
||||
|
||||
expect(result.details.filePath).toBe(path.join('.pi', 'shot_5.0s.webp'))
|
||||
expect(result.details.key).toBe('files/ast-123/screenshots/shot_5.0s.webp')
|
||||
expect(result.details.assetId).toBe('ast-123')
|
||||
expect(result.details.size).toBe(Buffer.from('fake-screenshot-bytes').length)
|
||||
})
|
||||
|
||||
it('should authorize and download original upload key (files/<uploadUlid>/...) via StorageKey', async () => {
|
||||
vi.mocked(prisma.asset.findUnique).mockResolvedValue(null)
|
||||
vi.mocked(prisma.storageKey.findFirst).mockResolvedValue({
|
||||
id: 'sk-1',
|
||||
key: 'files/upload-ulid-456/photo.png',
|
||||
assets: [{ id: 'asset-photo-1' }],
|
||||
} as unknown as StorageKey & { assets: Asset[] })
|
||||
vi.mocked(authzService.hasPermission).mockResolvedValue()
|
||||
|
||||
vi.mocked(s3Service.downloadToFile).mockImplementation(async (_b, _k, dest) => {
|
||||
await fs.promises.writeFile(dest, 'fake-original-photo-bytes')
|
||||
})
|
||||
vi.mocked(s3Service.headObject).mockResolvedValue({
|
||||
contentType: 'image/png',
|
||||
} as unknown as Awaited<ReturnType<typeof s3Service.headObject>>)
|
||||
|
||||
const tool = createDownloadAssetTool('user-1')
|
||||
const result = await tool.execute('call-1', {
|
||||
assetId: null,
|
||||
key: 'files/upload-ulid-456/photo.png',
|
||||
})
|
||||
|
||||
expect(authzService.hasPermission).toHaveBeenCalledWith({
|
||||
user: { id: 'user-1' },
|
||||
permission: 'Read',
|
||||
type: 'asset',
|
||||
id: 'asset-photo-1',
|
||||
})
|
||||
|
||||
const expectedPath = path.join(piDir, 'photo.png')
|
||||
createdFiles.push(expectedPath)
|
||||
|
||||
expect(s3Service.downloadToFile).toHaveBeenCalledWith(
|
||||
'shumai',
|
||||
'files/upload-ulid-456/photo.png',
|
||||
expectedPath,
|
||||
)
|
||||
|
||||
expect(fs.existsSync(expectedPath)).toBe(true)
|
||||
expect(result.details.assetId).toBe('asset-photo-1')
|
||||
})
|
||||
|
||||
it('should fall back to file MIME detection when headObject returns generic application/octet-stream', async () => {
|
||||
vi.mocked(prisma.asset.findUnique).mockResolvedValue(null)
|
||||
vi.mocked(prisma.storageKey.findFirst).mockResolvedValue({
|
||||
id: 'sk-2',
|
||||
key: 'files/upload-ulid-789/graphic.png',
|
||||
assets: [{ id: 'asset-graphic-1' }],
|
||||
} as unknown as StorageKey & { assets: Asset[] })
|
||||
vi.mocked(authzService.hasPermission).mockResolvedValue()
|
||||
|
||||
vi.mocked(s3Service.downloadToFile).mockImplementation(async (_b, _k, dest) => {
|
||||
const pngHeader = Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a])
|
||||
await fs.promises.writeFile(dest, pngHeader)
|
||||
})
|
||||
vi.mocked(s3Service.headObject).mockResolvedValue({
|
||||
contentType: 'application/octet-stream',
|
||||
} as unknown as Awaited<ReturnType<typeof s3Service.headObject>>)
|
||||
|
||||
const tool = createDownloadAssetTool('user-1')
|
||||
const result = await tool.execute('call-1', {
|
||||
assetId: null,
|
||||
key: 'files/upload-ulid-789/graphic.png',
|
||||
})
|
||||
|
||||
const expectedPath = path.join(piDir, 'graphic.png')
|
||||
createdFiles.push(expectedPath)
|
||||
|
||||
expect(result.details.contentType).toBe('image/png')
|
||||
})
|
||||
})
|
||||
|
||||
describe('resolveAssetIdFromKey helper unit tests', () => {
|
||||
it('resolves directly by asset ID', async () => {
|
||||
vi.mocked(prisma.asset.findUnique).mockResolvedValue({ id: 'ast-999' } as unknown as Asset)
|
||||
|
||||
@@ -1,32 +1,8 @@
|
||||
import { Type } from 'typebox'
|
||||
import { type AgentTool } from '@earendil-works/pi-agent-core'
|
||||
// Storage-key helpers shared by download_asset (operation-extensions) and the media generation tools.
|
||||
import { prisma, type User } from '@shumai/db'
|
||||
import { s3Service } from '@shumai/core/src/s3/s3'
|
||||
import { authzService, Permission, ResourceType } from '@shumai/core/src/authz/authz'
|
||||
import { sanitizeFilename } from '@shumai/core/src/utils/filename'
|
||||
import { getFileMimeType, readFileMimeType } from '@shumai/core/src/utils/file-mime'
|
||||
import * as fs from 'fs'
|
||||
import * as path from 'path'
|
||||
|
||||
export const downloadAssetSchema = Type.Object(
|
||||
{
|
||||
assetId: Type.Union([
|
||||
Type.String({
|
||||
description:
|
||||
'The asset ID of the workspace asset to download to local disk, or null if providing "key".',
|
||||
}),
|
||||
Type.Null(),
|
||||
]),
|
||||
key: Type.Union([
|
||||
Type.String({
|
||||
description:
|
||||
'The storage S3 key of a specific media artifact (e.g. a screenshot or transcode key from read_asset) to download to local disk, or null if providing "assetId".',
|
||||
}),
|
||||
Type.Null(),
|
||||
]),
|
||||
},
|
||||
{ additionalProperties: false },
|
||||
)
|
||||
import { getFileMimeType } from '@shumai/core/src/utils/file-mime'
|
||||
|
||||
export async function resolveAssetIdFromKey(key: string): Promise<string> {
|
||||
if (!key.startsWith('files/') || key.includes('..')) {
|
||||
@@ -118,177 +94,3 @@ export async function resolveS3MediaBuffer(
|
||||
|
||||
return { buffer, mimeType }
|
||||
}
|
||||
|
||||
export function createDownloadAssetTool(userId: string): AgentTool<typeof downloadAssetSchema> {
|
||||
return {
|
||||
name: 'download_asset',
|
||||
label: 'Download Asset',
|
||||
description:
|
||||
'Downloads a workspace asset or specific storage S3 key to the local .pi directory for processing with local scripts or bash commands. ' +
|
||||
'Provide exactly one of "assetId" (for full assets) or "key" (for intermediate artifacts like screenshots from read_asset). ' +
|
||||
'Remember to delete the temporary downloaded file when finished.',
|
||||
parameters: downloadAssetSchema,
|
||||
execute: async (_toolCallId, params) => {
|
||||
const { assetId, key } = params
|
||||
|
||||
if (!userId) {
|
||||
throw new Error('User ID is required for authorization.')
|
||||
}
|
||||
|
||||
const hasAssetId = assetId !== null
|
||||
const hasKey = key !== null
|
||||
|
||||
if (hasAssetId === hasKey) {
|
||||
throw new Error(
|
||||
'Provide exactly one of "assetId" (to download a workspace asset) or "key" (to download a specific S3 storage key). Set the unused parameter to null.',
|
||||
)
|
||||
}
|
||||
|
||||
const piDir = path.join(process.cwd(), '.pi')
|
||||
if (!fs.existsSync(piDir)) {
|
||||
fs.mkdirSync(piDir, { recursive: true })
|
||||
}
|
||||
|
||||
const bucket = process.env.S3_BUCKET || 'shumai'
|
||||
|
||||
// Branch A: Download by assetId
|
||||
if (assetId) {
|
||||
await authzService.hasPermission({
|
||||
user: { id: userId } as User,
|
||||
permission: Permission.Read,
|
||||
type: ResourceType.Asset,
|
||||
id: assetId,
|
||||
})
|
||||
|
||||
let asset = await prisma.asset.findUnique({
|
||||
where: { id: assetId },
|
||||
include: { storageKey: true },
|
||||
})
|
||||
|
||||
if (!asset) {
|
||||
throw new Error(`Asset with ID ${assetId} not found.`)
|
||||
}
|
||||
|
||||
if (asset.type === 'version_stack') {
|
||||
const latestVersion = await prisma.asset.findFirst({
|
||||
where: { parentId: asset.id, isDeleted: false },
|
||||
orderBy: { sortIndex: 'asc' },
|
||||
include: { storageKey: true },
|
||||
})
|
||||
if (latestVersion) {
|
||||
asset = latestVersion
|
||||
}
|
||||
}
|
||||
|
||||
let mediaKey = asset.storageKey?.key
|
||||
|
||||
if (asset.media) {
|
||||
const mediaInfo = asset.media as unknown as PrismaJson.MediaInfo
|
||||
if (
|
||||
mediaInfo.proxyType === 'image' ||
|
||||
(mediaInfo.imageTranscodes && mediaInfo.imageTranscodes.length > 0)
|
||||
) {
|
||||
if (mediaInfo.imageTranscodes && mediaInfo.imageTranscodes.length > 0) {
|
||||
mediaKey = mediaInfo.imageTranscodes[0].key || mediaKey
|
||||
}
|
||||
} else if (
|
||||
mediaInfo.proxyType === 'video' ||
|
||||
(mediaInfo.videoTranscodes && mediaInfo.videoTranscodes.length > 0) ||
|
||||
mediaInfo.videoPreview?.key
|
||||
) {
|
||||
if (mediaInfo.videoTranscodes && mediaInfo.videoTranscodes.length > 0) {
|
||||
mediaKey = mediaInfo.videoTranscodes[0].key || mediaKey
|
||||
} else if (mediaInfo.videoPreview?.key) {
|
||||
mediaKey = mediaInfo.videoPreview.key
|
||||
}
|
||||
} else if (mediaInfo.original?.key) {
|
||||
mediaKey = mediaInfo.original.key || mediaKey
|
||||
}
|
||||
}
|
||||
|
||||
if (!mediaKey) {
|
||||
throw new Error(`No media content found for asset ${assetId}.`)
|
||||
}
|
||||
|
||||
const safeName = sanitizeFilename(asset.name || 'file')
|
||||
const filename = `${asset.id}_${safeName}`
|
||||
const targetFilePath = path.join(piDir, filename)
|
||||
const relativePath = path.join('.pi', filename)
|
||||
|
||||
const objectInfo = await s3Service.headObject(bucket, mediaKey).catch(() => null)
|
||||
await s3Service.downloadToFile(bucket, mediaKey, targetFilePath)
|
||||
const stat = await fs.promises
|
||||
.stat(targetFilePath)
|
||||
.catch(() => ({ size: objectInfo?.size ?? 0 }))
|
||||
const contentType =
|
||||
objectInfo?.contentType && objectInfo.contentType !== 'application/octet-stream'
|
||||
? objectInfo.contentType
|
||||
: readFileMimeType(targetFilePath)
|
||||
|
||||
return {
|
||||
content: [
|
||||
{
|
||||
type: 'text',
|
||||
text: `Downloaded asset "${asset.name}" (ID: ${assetId}) to "${relativePath}". Remember to delete this temporary file when finished.`,
|
||||
},
|
||||
],
|
||||
details: {
|
||||
assetId,
|
||||
name: asset.name,
|
||||
filePath: relativePath,
|
||||
absolutePath: targetFilePath,
|
||||
contentType,
|
||||
size: stat.size,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// Branch B: Download by S3 storage key
|
||||
if (key) {
|
||||
const owningAssetId = await resolveAssetIdFromKey(key)
|
||||
|
||||
await authzService.hasPermission({
|
||||
user: { id: userId } as User,
|
||||
permission: Permission.Read,
|
||||
type: ResourceType.Asset,
|
||||
id: owningAssetId,
|
||||
})
|
||||
|
||||
const rawBasename = path.basename(key)
|
||||
const safeBasename = sanitizeFilename(rawBasename) || 'downloaded_file'
|
||||
const filename = safeBasename
|
||||
const targetFilePath = path.join(piDir, filename)
|
||||
const relativePath = path.join('.pi', filename)
|
||||
|
||||
const objectInfo = await s3Service.headObject(bucket, key).catch(() => null)
|
||||
await s3Service.downloadToFile(bucket, key, targetFilePath)
|
||||
const stat = await fs.promises
|
||||
.stat(targetFilePath)
|
||||
.catch(() => ({ size: objectInfo?.size ?? 0 }))
|
||||
const contentType =
|
||||
objectInfo?.contentType && objectInfo.contentType !== 'application/octet-stream'
|
||||
? objectInfo.contentType
|
||||
: readFileMimeType(targetFilePath)
|
||||
|
||||
return {
|
||||
content: [
|
||||
{
|
||||
type: 'text',
|
||||
text: `Downloaded storage key "${key}" to "${relativePath}". Remember to delete this temporary file when finished.`,
|
||||
},
|
||||
],
|
||||
details: {
|
||||
key,
|
||||
assetId: owningAssetId,
|
||||
filePath: relativePath,
|
||||
absolutePath: targetFilePath,
|
||||
contentType,
|
||||
size: stat.size,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
throw new Error('Invalid download_asset parameters.')
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
import { getAssetRenderMcpExtension, readAssetMcpExtension } from './read-asset'
|
||||
import type { McpToolExtension } from './types'
|
||||
|
||||
export type { McpToolExtension } from './types'
|
||||
|
||||
/** MCP-specific presentation for Operations, by Operation name. */
|
||||
export const mcpExtensions: ReadonlyMap<string, McpToolExtension> = new Map([
|
||||
['read_asset', readAssetMcpExtension],
|
||||
['get_asset_render', getAssetRenderMcpExtension],
|
||||
])
|
||||
@@ -0,0 +1,116 @@
|
||||
import { z } from 'zod'
|
||||
import type { CallToolResult, ImageContent } from '@modelcontextprotocol/server'
|
||||
import type { AssetReadOutput } from '@shumai/operations'
|
||||
import { storageBucket } from '@shumai/core/src/asset-read'
|
||||
import { logger } from '@shumai/core/src/logger'
|
||||
import { s3Service } from '@shumai/core/src/s3/s3'
|
||||
import { getFileMimeType } from '@shumai/core/src/utils/file-mime'
|
||||
import type { McpToolExtension } from './types'
|
||||
|
||||
/** Most images attached inline to one result; the rest are links only. */
|
||||
export const MAX_INLINE_IMAGES = 8
|
||||
/** Largest image attached inline; bigger ones are links only. */
|
||||
export const MAX_INLINE_IMAGE_BYTES = 2 * 1024 * 1024
|
||||
|
||||
const extraInput = z.object({
|
||||
inlineImages: z
|
||||
.boolean()
|
||||
.optional()
|
||||
.describe(
|
||||
`Attach images to the result so you can see them (default true, at most ${MAX_INLINE_IMAGES} images of up to 2 MB). Set to false to only get links.`,
|
||||
),
|
||||
})
|
||||
|
||||
/**
|
||||
* Removes storage keys and the source file link: the Operation returns keys to server-side
|
||||
* presenters so they can attach image bytes, but MCP clients only get image links.
|
||||
*/
|
||||
export function toClientOutput(output: AssetReadOutput): AssetReadOutput {
|
||||
const images = output.images.map(({ label, mimeType, url }) => ({ label, mimeType, url }))
|
||||
if (!output.text) return { ...output, images }
|
||||
const { content, truncated, size, mimeType } = output.text
|
||||
return { ...output, images, text: { content, truncated, size, mimeType } }
|
||||
}
|
||||
|
||||
async function loadImage(key: string, fallbackMime: string): Promise<ImageContent | 'too_large'> {
|
||||
const bucket = storageBucket()
|
||||
const size = await s3Service.getObjectSize(bucket, key)
|
||||
if (size > MAX_INLINE_IMAGE_BYTES) return 'too_large'
|
||||
const { buffer, contentType } = await s3Service.getObject(bucket, key)
|
||||
const mimeType =
|
||||
contentType && contentType !== 'application/octet-stream'
|
||||
? contentType
|
||||
: getFileMimeType(buffer, key, fallbackMime)
|
||||
return { type: 'image', data: buffer.toString('base64'), mimeType }
|
||||
}
|
||||
|
||||
/** Renders a `read_asset` / `get_asset_render` output as MCP content: text, then images. */
|
||||
export async function presentReadForMcp(
|
||||
raw: unknown,
|
||||
extra: Record<string, unknown>,
|
||||
): Promise<CallToolResult> {
|
||||
const output = raw as AssetReadOutput
|
||||
const client = toClientOutput(output)
|
||||
const structuredContent = client as unknown as Record<string, unknown>
|
||||
|
||||
if (output.status === 'pending') {
|
||||
const retry = output.retryAfterSeconds ?? 5
|
||||
return {
|
||||
content: [
|
||||
{
|
||||
type: 'text',
|
||||
text: `${output.summary} Call get_asset_render with taskId "${output.taskId}" in about ${retry} seconds to get the result.`,
|
||||
},
|
||||
],
|
||||
structuredContent,
|
||||
}
|
||||
}
|
||||
|
||||
const inline = extra.inlineImages !== false
|
||||
const images: ImageContent[] = []
|
||||
const lines = [output.summary, ...output.details]
|
||||
for (const [index, image] of output.images.entries()) {
|
||||
let note = ''
|
||||
if (inline) {
|
||||
if (index >= MAX_INLINE_IMAGES) {
|
||||
note = ` (not attached: only the first ${MAX_INLINE_IMAGES} images are attached)`
|
||||
} else if (image.key) {
|
||||
try {
|
||||
const loaded = await loadImage(image.key, image.mimeType)
|
||||
if (loaded === 'too_large') {
|
||||
note = ' (not attached: larger than 2 MB)'
|
||||
} else {
|
||||
images.push(loaded)
|
||||
note = ` (attached as image ${images.length})`
|
||||
}
|
||||
} catch (err) {
|
||||
logger.warn({ err, key: image.key }, 'Failed to attach image to MCP read result')
|
||||
note = ' (not attached: could not load the image)'
|
||||
}
|
||||
}
|
||||
}
|
||||
lines.push(`- ${image.label}${note}: ${image.url ?? ''}`)
|
||||
}
|
||||
if (output.images.length > 0) {
|
||||
lines.push('Image links expire in about an hour.')
|
||||
}
|
||||
|
||||
let text = lines.join('\n')
|
||||
if (output.text) {
|
||||
text += `\n\n${output.text.content}`
|
||||
if (output.text.truncated) {
|
||||
text += `\n\n[Content truncated: showing the first part of a ${output.text.size}-byte file. Use download_asset to get the whole file.]`
|
||||
}
|
||||
}
|
||||
return { content: [{ type: 'text', text }, ...images], structuredContent }
|
||||
}
|
||||
|
||||
export const readAssetMcpExtension: McpToolExtension = {
|
||||
extraInput,
|
||||
present: presentReadForMcp,
|
||||
}
|
||||
|
||||
export const getAssetRenderMcpExtension: McpToolExtension = {
|
||||
extraInput,
|
||||
present: presentReadForMcp,
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
import type { z } from 'zod'
|
||||
import type { CallToolResult } from '@modelcontextprotocol/server'
|
||||
|
||||
/**
|
||||
* MCP-specific presentation for an Operation. By default an MCP tool returns the output as
|
||||
* JSON text plus `structuredContent`; an extension can add MCP-only arguments and build the
|
||||
* result content itself (e.g. image blocks). `structuredContent` must still match the
|
||||
* Operation's output schema.
|
||||
*/
|
||||
export interface McpToolExtension {
|
||||
/** Extra arguments only MCP clients see; they are removed before the Operation runs. */
|
||||
extraInput?: z.ZodObject
|
||||
present(output: unknown, extra: Record<string, unknown>): Promise<CallToolResult>
|
||||
}
|
||||
@@ -1,8 +1,12 @@
|
||||
import { afterEach, beforeEach, describe, expect, it } from 'vitest'
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it } from 'vitest'
|
||||
import * as fs from 'fs'
|
||||
import * as path from 'path'
|
||||
import { fileURLToPath } from 'url'
|
||||
import { Client, StreamableHTTPClientTransport } from '@modelcontextprotocol/client'
|
||||
import { prisma, type User } from '@shumai/db'
|
||||
import { setupTestDbHooks } from '@shumai/db/test'
|
||||
import { projectService } from '@shumai/core/src/project/project'
|
||||
import { s3Service } from '@shumai/core/src/s3/s3'
|
||||
import { apiTokenService } from '@shumai/core/src/user/api-token'
|
||||
import app from './index'
|
||||
|
||||
@@ -158,4 +162,146 @@ describe('MCP server and /api/ops (integration)', () => {
|
||||
|
||||
expect(res.status).toBe(401)
|
||||
})
|
||||
|
||||
describe('read_asset', () => {
|
||||
const bucket = 'shumai-test-mcp-read'
|
||||
const originalBucket = process.env.S3_BUCKET
|
||||
const png = fs.readFileSync(
|
||||
path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../e2e/fixtures/small.png'),
|
||||
)
|
||||
let imageId: string
|
||||
|
||||
beforeAll(() => {
|
||||
process.env.S3_BUCKET = bucket
|
||||
})
|
||||
|
||||
afterAll(async () => {
|
||||
if (originalBucket === undefined) delete process.env.S3_BUCKET
|
||||
else process.env.S3_BUCKET = originalBucket
|
||||
await s3Service.deletePrefix(bucket, '').catch(() => undefined)
|
||||
})
|
||||
|
||||
beforeEach(async () => {
|
||||
const key = `projects/${projectId}/photo.png`
|
||||
await s3Service.putObject(bucket, key, png, png.length, 'image/png')
|
||||
const storageKey = await prisma.storageKey.create({ data: { key } })
|
||||
imageId = (
|
||||
await prisma.asset.create({
|
||||
data: {
|
||||
name: 'photo.png',
|
||||
type: 'file',
|
||||
status: 'uploaded',
|
||||
mediaType: 'image/png',
|
||||
projectId,
|
||||
parentId: rootFolderId,
|
||||
storageKeyId: storageKey.id,
|
||||
media: { proxyType: 'image' } as unknown as PrismaJson.MediaInfo,
|
||||
},
|
||||
})
|
||||
).id
|
||||
})
|
||||
|
||||
it('shows a reviewer the image inline, with links but no storage keys', async () => {
|
||||
const client = await connect(reviewerToken)
|
||||
|
||||
const result = await client.callTool({ name: 'read_asset', arguments: { assetId: imageId } })
|
||||
|
||||
expect(result.isError).toBeFalsy()
|
||||
const content = result.content as Array<{ type: string; data?: string; text?: string }>
|
||||
expect(content[0].text).toContain(`Image asset "photo.png" (ID: ${imageId}):`)
|
||||
expect(content[1]).toEqual({
|
||||
type: 'image',
|
||||
data: png.toString('base64'),
|
||||
mimeType: 'image/png',
|
||||
})
|
||||
const output = result.structuredContent as { images: Array<Record<string, unknown>> }
|
||||
expect(output.images).toEqual([
|
||||
{ label: 'Image', mimeType: 'image/png', url: expect.any(String) },
|
||||
])
|
||||
expect(JSON.stringify(result)).not.toContain('"key"')
|
||||
})
|
||||
|
||||
it('serves read and render to the CLI with links only', async () => {
|
||||
const headers = {
|
||||
Authorization: `Bearer ${reviewerToken}`,
|
||||
'Content-Type': 'application/json',
|
||||
'x-shumai-client': 'cli',
|
||||
}
|
||||
|
||||
const read = await app.request('/api/ops/read_asset', {
|
||||
method: 'POST',
|
||||
headers,
|
||||
body: JSON.stringify({ assetId: imageId }),
|
||||
})
|
||||
expect(read.status).toBe(200)
|
||||
const output = (await read.json()) as { status: string; images: Array<{ url: string }> }
|
||||
expect(output.status).toBe('ready')
|
||||
expect(output.images).toEqual([
|
||||
{ label: 'Image', mimeType: 'image/png', url: expect.any(String) },
|
||||
])
|
||||
|
||||
// The link is served by the API itself with local storage
|
||||
const link = new URL(output.images[0].url)
|
||||
const image = await app.request(link.pathname + link.search)
|
||||
expect(image.status).toBe(200)
|
||||
expect(Buffer.from(await image.arrayBuffer()).equals(png)).toBe(true)
|
||||
|
||||
const render = await app.request('/api/ops/get_asset_render', {
|
||||
method: 'POST',
|
||||
headers,
|
||||
body: JSON.stringify({ taskId: 'not-a-task' }),
|
||||
})
|
||||
expect(render.status).toBe(404)
|
||||
expect(await render.json()).toEqual({ error: 'Render not found' })
|
||||
})
|
||||
|
||||
it('gives MCP and CLI callers a download link to the file', async () => {
|
||||
const client = await connect(reviewerToken)
|
||||
const result = await client.callTool({
|
||||
name: 'download_asset',
|
||||
arguments: { assetIds: [imageId] },
|
||||
})
|
||||
expect(result.isError).toBeFalsy()
|
||||
const viaMcp = result.structuredContent as { files: Array<Record<string, unknown>> }
|
||||
expect(viaMcp.files).toEqual([
|
||||
{
|
||||
assetId: imageId,
|
||||
name: 'photo.png',
|
||||
kind: 'original',
|
||||
size: png.length,
|
||||
mimeType: 'image/png',
|
||||
url: expect.any(String),
|
||||
},
|
||||
])
|
||||
|
||||
const res = await app.request('/api/ops/download_asset', {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
Authorization: `Bearer ${reviewerToken}`,
|
||||
'Content-Type': 'application/json',
|
||||
'x-shumai-client': 'cli',
|
||||
},
|
||||
body: JSON.stringify({ assetIds: [imageId], proxy: false }),
|
||||
})
|
||||
expect(res.status).toBe(200)
|
||||
const viaCli = (await res.json()) as { files: Array<{ url: string; key?: string }> }
|
||||
expect(viaCli.files[0].key).toBeUndefined()
|
||||
const link = new URL(viaCli.files[0].url)
|
||||
const file = await app.request(link.pathname + link.search)
|
||||
expect(file.status).toBe(200)
|
||||
expect(Buffer.from(await file.arrayBuffer()).equals(png)).toBe(true)
|
||||
})
|
||||
|
||||
it('does not let users outside the project read', async () => {
|
||||
const outsider = await prisma.user.create({
|
||||
data: { name: 'Outsider', email: 'mcp-outsider@test.com', password: 'pw' },
|
||||
})
|
||||
const token = (await apiTokenService.createToken(outsider.id, 'mcp')).token
|
||||
const client = await connect(token)
|
||||
|
||||
const result = await client.callTool({ name: 'read_asset', arguments: { assetId: imageId } })
|
||||
|
||||
expect(result.isError).toBe(true)
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
@@ -3,7 +3,9 @@ import { Hono } from 'hono'
|
||||
import { HTTPException } from 'hono/http-exception'
|
||||
import { Client, StreamableHTTPClientTransport } from '@modelcontextprotocol/client'
|
||||
import { teamService } from '@shumai/core/src/team/team'
|
||||
import { createFolderOp, runOperation } from '@shumai/core/src/operations'
|
||||
import { s3Service } from '@shumai/core/src/s3/s3'
|
||||
import { createFolderOp, readAssetOp, runOperation } from '@shumai/core/src/operations'
|
||||
import { MAX_INLINE_IMAGE_BYTES, MAX_INLINE_IMAGES } from './mcp-extensions/read-asset'
|
||||
import mcpServerRoute, { MCP_SERVER_INSTRUCTIONS } from './mcp-server'
|
||||
|
||||
vi.mock('@shumai/core/src/operations', async (importOriginal) => ({
|
||||
@@ -53,9 +55,12 @@ describe('MCP server', () => {
|
||||
expect(tools.map((t) => t.name).sort()).toEqual([
|
||||
'create_folder',
|
||||
'delete_asset',
|
||||
'download_asset',
|
||||
'get_asset_render',
|
||||
'list_assets',
|
||||
'list_projects',
|
||||
'move_assets',
|
||||
'read_asset',
|
||||
'rename_asset',
|
||||
])
|
||||
|
||||
@@ -81,9 +86,58 @@ describe('MCP server', () => {
|
||||
expect(annotations('list_assets')).toMatchObject({ readOnlyHint: true, idempotentHint: true })
|
||||
})
|
||||
|
||||
it('advertises input schemas without nullable types', async () => {
|
||||
// Nullable types don't translate to every client's tool format (e.g. Gemini's)
|
||||
for (const tool of (await client.listTools()).tools) {
|
||||
const schema = JSON.stringify(tool.inputSchema)
|
||||
expect(schema, tool.name).not.toMatch(/"type":\[/)
|
||||
expect(schema, tool.name).not.toContain('"type":"null"')
|
||||
}
|
||||
})
|
||||
|
||||
it('treats null optional arguments as omitted, including nested ones', async () => {
|
||||
vi.mocked(runOperation).mockResolvedValue({
|
||||
status: 'pending',
|
||||
summary: 'Rendering is still in progress for "deck.pdf".',
|
||||
asset: { id: 'doc-1', name: 'deck.pdf', kind: 'document', mediaType: 'application/pdf' },
|
||||
details: [],
|
||||
images: [],
|
||||
taskId: 'task-1',
|
||||
retryAfterSeconds: 5,
|
||||
})
|
||||
|
||||
const result = await client.callTool({
|
||||
name: 'read_asset',
|
||||
arguments: {
|
||||
assetId: 'doc-1',
|
||||
annotationId: null,
|
||||
videoConfig: null,
|
||||
docConfig: { mode: 'pages', startPage: 1, endPage: null },
|
||||
inlineImages: null,
|
||||
},
|
||||
})
|
||||
|
||||
expect(result.isError).toBeFalsy()
|
||||
expect(runOperation).toHaveBeenCalledWith(
|
||||
readAssetOp,
|
||||
{ userId: 'user1', via: 'mcp' },
|
||||
{ assetId: 'doc-1', docConfig: { mode: 'pages', startPage: 1 } },
|
||||
)
|
||||
})
|
||||
|
||||
it('still rejects null for required arguments', async () => {
|
||||
const result = await client.callTool({
|
||||
name: 'create_folder',
|
||||
arguments: { parent: null, name: 'Renders' },
|
||||
})
|
||||
|
||||
expect(result.isError).toBe(true)
|
||||
expect(runOperation).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('does not expose agent-only capabilities', async () => {
|
||||
const names = (await client.listTools()).tools.map((t) => t.name)
|
||||
for (const agentOnly of ['bash', 'read_skill', 'read_asset', 'download_asset', 'mcp']) {
|
||||
for (const agentOnly of ['bash', 'read_skill', 'mcp']) {
|
||||
expect(names).not.toContain(agentOnly)
|
||||
}
|
||||
})
|
||||
@@ -141,4 +195,195 @@ describe('MCP server', () => {
|
||||
expect(result.content).toEqual([{ type: 'text', text: 'System is in read-only mode' }])
|
||||
expect(runOperation).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
describe('read_asset', () => {
|
||||
const frame = (n: number) => ({
|
||||
label: `Frame at ${n}s`,
|
||||
mimeType: 'image/webp',
|
||||
url: `https://files.example/${n}.webp`,
|
||||
key: `shots/${n}.webp`,
|
||||
})
|
||||
const ready = (count: number) => ({
|
||||
status: 'ready',
|
||||
summary: `Extracted ${count} frame(s) from "clip.mp4" (0s - 10s):`,
|
||||
asset: { id: 'vid-1', name: 'clip.mp4', kind: 'video', mediaType: 'video/mp4' },
|
||||
details: [],
|
||||
images: Array.from({ length: count }, (_, i) => frame(i)),
|
||||
})
|
||||
|
||||
beforeEach(() => {
|
||||
vi.spyOn(s3Service, 'getObjectSize').mockResolvedValue(1000)
|
||||
vi.spyOn(s3Service, 'getObject').mockImplementation(async (_bucket, key) => ({
|
||||
buffer: Buffer.from(`bytes of ${key}`),
|
||||
contentType: 'image/webp',
|
||||
}))
|
||||
})
|
||||
|
||||
it('advertises inlineImages and keeps it out of the Operation input', async () => {
|
||||
const { tools } = await client.listTools()
|
||||
const read = tools.find((t) => t.name === 'read_asset')!
|
||||
expect(read.inputSchema.properties).toHaveProperty('inlineImages')
|
||||
expect(read.inputSchema.properties).toHaveProperty('videoConfig')
|
||||
expect(read.annotations).toMatchObject({ readOnlyHint: true, destructiveHint: false })
|
||||
expect(
|
||||
tools.find((t) => t.name === 'get_asset_render')!.inputSchema.properties,
|
||||
).toHaveProperty('inlineImages')
|
||||
|
||||
vi.mocked(runOperation).mockResolvedValue(ready(1))
|
||||
await client.callTool({
|
||||
name: 'read_asset',
|
||||
arguments: {
|
||||
assetId: 'vid-1',
|
||||
videoConfig: { start: 0, end: 10, count: 1 },
|
||||
inlineImages: true,
|
||||
},
|
||||
})
|
||||
expect(runOperation).toHaveBeenCalledWith(
|
||||
readAssetOp,
|
||||
{ userId: 'user1', via: 'mcp' },
|
||||
{ assetId: 'vid-1', videoConfig: { start: 0, end: 10, count: 1 } },
|
||||
)
|
||||
})
|
||||
|
||||
it('attaches images inline and returns links without storage keys', async () => {
|
||||
vi.mocked(runOperation).mockResolvedValue(ready(2))
|
||||
|
||||
const result = await client.callTool({ name: 'read_asset', arguments: { assetId: 'vid-1' } })
|
||||
|
||||
expect(result.isError).toBeFalsy()
|
||||
expect(result.content).toEqual([
|
||||
{
|
||||
type: 'text',
|
||||
text: [
|
||||
'Extracted 2 frame(s) from "clip.mp4" (0s - 10s):',
|
||||
'- Frame at 0s (attached as image 1): https://files.example/0.webp',
|
||||
'- Frame at 1s (attached as image 2): https://files.example/1.webp',
|
||||
'Image links expire in about an hour.',
|
||||
].join('\n'),
|
||||
},
|
||||
{
|
||||
type: 'image',
|
||||
data: Buffer.from('bytes of shots/0.webp').toString('base64'),
|
||||
mimeType: 'image/webp',
|
||||
},
|
||||
{
|
||||
type: 'image',
|
||||
data: Buffer.from('bytes of shots/1.webp').toString('base64'),
|
||||
mimeType: 'image/webp',
|
||||
},
|
||||
])
|
||||
expect(result.structuredContent).toEqual({
|
||||
...ready(2),
|
||||
images: [0, 1].map((n) => ({
|
||||
label: `Frame at ${n}s`,
|
||||
mimeType: 'image/webp',
|
||||
url: `https://files.example/${n}.webp`,
|
||||
})),
|
||||
})
|
||||
expect(JSON.stringify(result)).not.toContain('shots/0.webp"')
|
||||
})
|
||||
|
||||
it(`attaches at most ${MAX_INLINE_IMAGES} images and skips large ones`, async () => {
|
||||
vi.mocked(runOperation).mockResolvedValue(ready(10))
|
||||
vi.mocked(s3Service.getObjectSize).mockImplementation(async (_bucket, key) =>
|
||||
key === 'shots/2.webp' ? MAX_INLINE_IMAGE_BYTES + 1 : 1000,
|
||||
)
|
||||
|
||||
const result = await client.callTool({ name: 'read_asset', arguments: { assetId: 'vid-1' } })
|
||||
|
||||
const content = result.content as Array<{ type: string; text?: string }>
|
||||
expect(content.filter((c) => c.type === 'image')).toHaveLength(MAX_INLINE_IMAGES - 1)
|
||||
const text = content[0].text!
|
||||
expect(text).toContain(
|
||||
'- Frame at 2s (not attached: larger than 2 MB): https://files.example/2.webp',
|
||||
)
|
||||
expect(text).toContain(
|
||||
`- Frame at 8s (not attached: only the first ${MAX_INLINE_IMAGES} images are attached)`,
|
||||
)
|
||||
expect(s3Service.getObject).not.toHaveBeenCalledWith(expect.anything(), 'shots/2.webp')
|
||||
expect(s3Service.getObject).not.toHaveBeenCalledWith(expect.anything(), 'shots/8.webp')
|
||||
})
|
||||
|
||||
it('returns links only with inlineImages false', async () => {
|
||||
vi.mocked(runOperation).mockResolvedValue(ready(2))
|
||||
|
||||
const result = await client.callTool({
|
||||
name: 'read_asset',
|
||||
arguments: { assetId: 'vid-1', inlineImages: false },
|
||||
})
|
||||
|
||||
expect(result.content).toEqual([
|
||||
{
|
||||
type: 'text',
|
||||
text: [
|
||||
'Extracted 2 frame(s) from "clip.mp4" (0s - 10s):',
|
||||
'- Frame at 0s: https://files.example/0.webp',
|
||||
'- Frame at 1s: https://files.example/1.webp',
|
||||
'Image links expire in about an hour.',
|
||||
].join('\n'),
|
||||
},
|
||||
])
|
||||
expect(s3Service.getObject).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('includes text content without the source storage key', async () => {
|
||||
vi.mocked(runOperation).mockResolvedValue({
|
||||
...ready(0),
|
||||
summary: 'Document "notes.md" (ID: d-1):',
|
||||
text: {
|
||||
content: '# Notes',
|
||||
truncated: false,
|
||||
size: 7,
|
||||
mimeType: 'text/markdown',
|
||||
key: 'raw/notes.md',
|
||||
url: 'https://files.example/notes.md?download',
|
||||
},
|
||||
})
|
||||
|
||||
const result = await client.callTool({ name: 'read_asset', arguments: { assetId: 'd-1' } })
|
||||
|
||||
expect(result.content).toEqual([
|
||||
{ type: 'text', text: 'Document "notes.md" (ID: d-1):\n\n# Notes' },
|
||||
])
|
||||
expect(result.structuredContent).toMatchObject({
|
||||
text: { content: '# Notes', truncated: false, size: 7, mimeType: 'text/markdown' },
|
||||
})
|
||||
expect((result.structuredContent as { text: object }).text).not.toHaveProperty('key')
|
||||
expect((result.structuredContent as { text: object }).text).not.toHaveProperty('url')
|
||||
})
|
||||
|
||||
it('tells the model to call get_asset_render when rendering is pending', async () => {
|
||||
vi.mocked(runOperation).mockResolvedValue({
|
||||
...ready(0),
|
||||
status: 'pending',
|
||||
summary: 'Rendering is still in progress for "clip.mp4".',
|
||||
taskId: 'task-1',
|
||||
retryAfterSeconds: 5,
|
||||
})
|
||||
|
||||
const result = await client.callTool({
|
||||
name: 'read_asset',
|
||||
arguments: { assetId: 'vid-1', videoConfig: { start: 0, end: 10, count: 5 } },
|
||||
})
|
||||
|
||||
expect(result.content).toEqual([
|
||||
{
|
||||
type: 'text',
|
||||
text: 'Rendering is still in progress for "clip.mp4". Call get_asset_render with taskId "task-1" in about 5 seconds to get the result.',
|
||||
},
|
||||
])
|
||||
expect(result.structuredContent).toMatchObject({ status: 'pending', taskId: 'task-1' })
|
||||
})
|
||||
|
||||
it('reports read errors as tool errors', async () => {
|
||||
vi.mocked(runOperation).mockRejectedValue(
|
||||
new HTTPException(400, { message: 'videoConfig with start, end, and count is required' }),
|
||||
)
|
||||
const result = await client.callTool({ name: 'read_asset', arguments: { assetId: 'vid-1' } })
|
||||
expect(result.isError).toBe(true)
|
||||
expect(result.content).toEqual([
|
||||
{ type: 'text', text: 'videoConfig with start, end, and count is required' },
|
||||
])
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
@@ -1,7 +1,10 @@
|
||||
import { Hono } from 'hono'
|
||||
import { createMcpHandler, McpServer, type CallToolResult } from '@modelcontextprotocol/server'
|
||||
import type { z } from 'zod'
|
||||
import type { Prisma } from '@shumai/db'
|
||||
import { dropNullOptionals } from '@shumai/operations'
|
||||
import { getOperationsForSurface, runOperation, type Operation } from '@shumai/core/src/operations'
|
||||
import { mcpExtensions, type McpToolExtension } from './mcp-extensions'
|
||||
import { assertWritableInDemoMode, describeOperationError } from './operation-support'
|
||||
|
||||
type User = Prisma.UserGetPayload<Record<string, never>>
|
||||
@@ -14,16 +17,58 @@ export const MCP_SERVER_INSTRUCTIONS = [
|
||||
'Start with list_projects to find a project and its rootFolderId, then pass folder IDs as',
|
||||
'`parent` to list_assets and create_folder. list_assets is paginated: when',
|
||||
'pageInfo.hasNextPage is true, call it again with after = pageInfo.endCursor.',
|
||||
'Use read_asset to look at a file (image, video frames, PDF pages, 3D views, or text);',
|
||||
'if it returns status "pending", call get_asset_render with its taskId.',
|
||||
].join(' ')
|
||||
|
||||
/**
|
||||
* The tool's input schema, with `null` for optional arguments treated as omitted. The SDK
|
||||
* validates arguments before calling the tool, so the nulls must be dropped here; the JSON
|
||||
* Schema shown to clients is unchanged (optional, not nullable, which more clients support).
|
||||
*/
|
||||
function toolInputSchema(schema: z.ZodObject): Pick<z.ZodObject, '~standard'> {
|
||||
const standard = schema['~standard']
|
||||
return {
|
||||
'~standard': {
|
||||
...standard,
|
||||
validate: (value) => standard.validate(dropNullOptionals(schema, value)),
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
/** Separate MCP-only arguments (declared by an extension) from the Operation input. */
|
||||
function splitArgs(
|
||||
extension: McpToolExtension | undefined,
|
||||
args: unknown,
|
||||
): { input: unknown; extra: Record<string, unknown> } {
|
||||
if (!extension?.extraInput || !args || typeof args !== 'object') {
|
||||
return { input: args, extra: {} }
|
||||
}
|
||||
const extraKeys = new Set(Object.keys(extension.extraInput.shape))
|
||||
const input: Record<string, unknown> = {}
|
||||
const extra: Record<string, unknown> = {}
|
||||
for (const [key, value] of Object.entries(args)) {
|
||||
;(extraKeys.has(key) ? extra : input)[key] = value
|
||||
}
|
||||
return { input, extra }
|
||||
}
|
||||
|
||||
async function callOperation(
|
||||
op: Operation,
|
||||
userId: string,
|
||||
args: unknown,
|
||||
): Promise<CallToolResult> {
|
||||
const extension = mcpExtensions.get(op.definition.name)
|
||||
try {
|
||||
await assertWritableInDemoMode(op, userId)
|
||||
const output = (await runOperation(op, { userId, via: 'mcp' }, args)) as Record<string, unknown>
|
||||
const { input, extra } = splitArgs(extension, args)
|
||||
const output = (await runOperation(op, { userId, via: 'mcp' }, input)) as Record<
|
||||
string,
|
||||
unknown
|
||||
>
|
||||
if (extension) {
|
||||
return await extension.present(output, extra)
|
||||
}
|
||||
return {
|
||||
content: [{ type: 'text', text: JSON.stringify(output) }],
|
||||
structuredContent: output,
|
||||
@@ -44,12 +89,17 @@ export function buildMcpServer(userId: string): McpServer {
|
||||
)
|
||||
for (const op of getOperationsForSurface('mcp')) {
|
||||
const { definition } = op
|
||||
const extension = mcpExtensions.get(definition.name)
|
||||
server.registerTool(
|
||||
definition.name,
|
||||
{
|
||||
title: definition.title,
|
||||
description: definition.description,
|
||||
inputSchema: definition.input,
|
||||
inputSchema: toolInputSchema(
|
||||
extension?.extraInput
|
||||
? definition.input.extend(extension.extraInput.shape)
|
||||
: definition.input,
|
||||
),
|
||||
outputSchema: definition.output,
|
||||
annotations: {
|
||||
title: definition.title,
|
||||
|
||||
@@ -0,0 +1,245 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import { AssetType, prisma, type Prisma, type User } from '@shumai/db'
|
||||
import { setupTestDbHooks } from '@shumai/db/test'
|
||||
import { projectService } from '@shumai/core/src/project/project'
|
||||
import { s3Service } from '@shumai/core/src/s3/s3'
|
||||
import { downloadAssetOp, runOperation, type OperationContext } from '@shumai/core/src/operations'
|
||||
import { READ_URL_EXPIRES_SECONDS } from '@shumai/core/src/asset-read/asset-read'
|
||||
import { proxyKeyOf } from './asset-download'
|
||||
|
||||
type Head = Awaited<ReturnType<typeof s3Service.headObject>>
|
||||
|
||||
describe('download_asset Operation', () => {
|
||||
setupTestDbHooks()
|
||||
|
||||
let owner: User
|
||||
let reviewer: User
|
||||
let outsider: User
|
||||
let projectId: string
|
||||
let rootFolderId: string
|
||||
|
||||
const as = (user: User, via: OperationContext['via'] = 'cli'): OperationContext => ({
|
||||
userId: user.id,
|
||||
via,
|
||||
})
|
||||
|
||||
beforeEach(async () => {
|
||||
owner = await prisma.user.create({
|
||||
data: { name: 'Owner', email: 'dl-owner@test.com', password: 'pw' },
|
||||
})
|
||||
reviewer = await prisma.user.create({
|
||||
data: { name: 'Reviewer', email: 'dl-reviewer@test.com', password: 'pw' },
|
||||
})
|
||||
outsider = await prisma.user.create({
|
||||
data: { name: 'Outsider', email: 'dl-outsider@test.com', password: 'pw' },
|
||||
})
|
||||
const team = await prisma.team.create({ data: { name: 'Download Team' } })
|
||||
await prisma.teamMember.create({ data: { teamId: team.id, userId: owner.id, role: 'owner' } })
|
||||
await prisma.teamMember.create({
|
||||
data: { teamId: team.id, userId: reviewer.id, role: 'reviewer' },
|
||||
})
|
||||
const project = await projectService.createProject(owner, {
|
||||
teamId: team.id,
|
||||
name: 'Download Project',
|
||||
})
|
||||
projectId = project.id
|
||||
rootFolderId = project.rootFolder!
|
||||
|
||||
vi.spyOn(s3Service, 'presign').mockImplementation(
|
||||
async (_bucket, key) => `https://signed.example/${key}`,
|
||||
)
|
||||
vi.spyOn(s3Service, 'headObject').mockImplementation(
|
||||
async (_bucket, key) => ({ key, size: 100, contentType: 'application/octet-stream' }) as Head,
|
||||
)
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks()
|
||||
})
|
||||
|
||||
async function createFile(
|
||||
name: string,
|
||||
data: Partial<Prisma.AssetUncheckedCreateInput> & { key?: string } = {},
|
||||
) {
|
||||
const { key, ...rest } = data
|
||||
const storageKey = key ? await prisma.storageKey.create({ data: { key } }) : null
|
||||
return prisma.asset.create({
|
||||
data: {
|
||||
name,
|
||||
type: AssetType.file,
|
||||
status: 'uploaded',
|
||||
projectId,
|
||||
parentId: rootFolderId,
|
||||
storageKeyId: storageKey?.id,
|
||||
...rest,
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
const media = (value: Record<string, unknown>) => value as unknown as PrismaJson.MediaInfo
|
||||
|
||||
it('returns the web preview by default and the original with proxy false', async () => {
|
||||
const photo = await createFile('photo.jpg', {
|
||||
key: 'files/up-1/photo.jpg',
|
||||
media: media({ proxyType: 'image', imageTranscodes: [{ key: 'files/p-1/web.webp' }] }),
|
||||
})
|
||||
|
||||
const preview = await runOperation(downloadAssetOp, as(reviewer), { assetIds: [photo.id] })
|
||||
expect(preview).toEqual({
|
||||
files: [
|
||||
{
|
||||
assetId: photo.id,
|
||||
name: 'photo.webp',
|
||||
kind: 'proxy',
|
||||
size: 100,
|
||||
mimeType: 'image/webp',
|
||||
url: 'https://signed.example/files/p-1/web.webp',
|
||||
},
|
||||
],
|
||||
})
|
||||
expect(s3Service.presign).toHaveBeenCalledWith(
|
||||
expect.any(String),
|
||||
'files/p-1/web.webp',
|
||||
'GET',
|
||||
true,
|
||||
'photo.webp',
|
||||
false,
|
||||
READ_URL_EXPIRES_SECONDS,
|
||||
)
|
||||
|
||||
const original = await runOperation(downloadAssetOp, as(reviewer), {
|
||||
assetIds: [photo.id],
|
||||
proxy: false,
|
||||
})
|
||||
expect(original.files[0]).toMatchObject({
|
||||
name: 'photo.jpg',
|
||||
kind: 'original',
|
||||
mimeType: 'image/jpeg',
|
||||
url: 'https://signed.example/files/up-1/photo.jpg',
|
||||
})
|
||||
})
|
||||
|
||||
it('renames a video proxy to its extension and falls back to the original without one', async () => {
|
||||
const clip = await createFile('clip.mov', {
|
||||
key: 'files/up-2/clip.mov',
|
||||
media: media({ proxyType: 'video', videoTranscodes: [{ key: 'files/c-1/720p.mp4' }] }),
|
||||
})
|
||||
const song = await createFile('song.wav', {
|
||||
key: 'files/up-3/song.wav',
|
||||
media: media({ proxyType: 'audio', original: { key: 'files/up-3/song.wav' } }),
|
||||
})
|
||||
const archive = await createFile('bundle.zip', { key: 'files/up-4/bundle.zip' })
|
||||
|
||||
const { files } = await runOperation(downloadAssetOp, as(owner), {
|
||||
assetIds: [clip.id, song.id, archive.id, clip.id],
|
||||
})
|
||||
|
||||
expect(files.map((f) => [f.name, f.kind])).toEqual([
|
||||
['clip.mp4', 'proxy'],
|
||||
['song.wav', 'original'],
|
||||
['bundle.zip', 'original'],
|
||||
])
|
||||
})
|
||||
|
||||
it('downloads the latest version of a version stack', async () => {
|
||||
const stack = await prisma.asset.create({
|
||||
data: {
|
||||
name: 'diagram',
|
||||
type: AssetType.version_stack,
|
||||
status: 'uploaded',
|
||||
projectId,
|
||||
parentId: rootFolderId,
|
||||
},
|
||||
})
|
||||
await createFile('diagram_v1.png', {
|
||||
parentId: stack.id,
|
||||
sortIndex: 'b',
|
||||
key: 'files/v1/diagram_v1.png',
|
||||
})
|
||||
const v2 = await createFile('diagram_v2.png', {
|
||||
parentId: stack.id,
|
||||
sortIndex: 'a',
|
||||
key: 'files/v2/diagram_v2.png',
|
||||
})
|
||||
|
||||
const { files } = await runOperation(downloadAssetOp, as(owner), { assetIds: [stack.id] })
|
||||
|
||||
expect(files).toEqual([expect.objectContaining({ assetId: v2.id, name: 'diagram_v2.png' })])
|
||||
})
|
||||
|
||||
it('only gives storage keys to the built-in agent', async () => {
|
||||
const doc = await createFile('notes.md', { key: 'files/up-5/notes.md' })
|
||||
for (const via of ['mcp', 'cli', 'api'] as const) {
|
||||
const { files } = await runOperation(downloadAssetOp, as(owner, via), { assetIds: [doc.id] })
|
||||
expect(files[0]).not.toHaveProperty('key')
|
||||
}
|
||||
const { files } = await runOperation(downloadAssetOp, as(owner, 'agent'), {
|
||||
assetIds: [doc.id],
|
||||
})
|
||||
expect(files[0].key).toBe('files/up-5/notes.md')
|
||||
})
|
||||
|
||||
it('rejects folders', async () => {
|
||||
const folder = await prisma.asset.create({
|
||||
data: {
|
||||
name: 'Renders',
|
||||
type: AssetType.folder,
|
||||
status: 'uploaded',
|
||||
projectId,
|
||||
parentId: rootFolderId,
|
||||
},
|
||||
})
|
||||
await expect(
|
||||
runOperation(downloadAssetOp, as(owner), { assetIds: [folder.id] }),
|
||||
).rejects.toMatchObject({
|
||||
status: 400,
|
||||
message: `Asset ${folder.id} ("Renders") is a folder. Download the files inside it instead.`,
|
||||
})
|
||||
})
|
||||
|
||||
it('rejects users without Read access before resolving anything', async () => {
|
||||
const doc = await createFile('notes.md', { key: 'files/up-6/notes.md' })
|
||||
await expect(
|
||||
runOperation(downloadAssetOp, as(outsider), { assetIds: [doc.id] }),
|
||||
).rejects.toMatchObject({ status: 403 })
|
||||
expect(s3Service.headObject).not.toHaveBeenCalled()
|
||||
expect(s3Service.presign).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('reports files missing from storage', async () => {
|
||||
const doc = await createFile('notes.md', { key: 'files/up-7/notes.md' })
|
||||
vi.mocked(s3Service.headObject).mockRejectedValue(new Error('NotFound'))
|
||||
await expect(
|
||||
runOperation(downloadAssetOp, as(owner), { assetIds: [doc.id] }),
|
||||
).rejects.toMatchObject({
|
||||
status: 404,
|
||||
message: `The file for asset ${doc.id} ("notes.md") is missing from storage.`,
|
||||
})
|
||||
})
|
||||
|
||||
it.each([[[]], [Array.from({ length: 51 }, (_, i) => `a-${i}`)]])(
|
||||
'rejects %# asset ID lists outside 1 to 50',
|
||||
async (assetIds) => {
|
||||
await expect(runOperation(downloadAssetOp, as(owner), { assetIds })).rejects.toMatchObject({
|
||||
status: 400,
|
||||
})
|
||||
},
|
||||
)
|
||||
})
|
||||
|
||||
describe('proxyKeyOf', () => {
|
||||
it.each([
|
||||
[{ proxyType: 'image', imageTranscodes: [{ key: 'i.webp' }] }, 'i.webp'],
|
||||
[{ proxyType: '3d', imageTranscodes: [{ key: 'views.webp' }] }, 'views.webp'],
|
||||
[{ proxyType: 'video', videoTranscodes: [{ key: 'v.mp4' }] }, 'v.mp4'],
|
||||
[{ proxyType: 'video', videoTranscodes: [], videoPreview: { key: 'p.mp4' } }, 'p.mp4'],
|
||||
[{ proxyType: 'pdf', original: { key: 'doc.pdf' } }, 'doc.pdf'],
|
||||
[{ proxyType: 'image', imageTranscodes: [] }, undefined],
|
||||
])('%o → %s', (value, expected) => {
|
||||
expect(proxyKeyOf(value as unknown as PrismaJson.MediaInfo)).toBe(expected)
|
||||
})
|
||||
|
||||
it('has no proxy without media', () => {
|
||||
expect(proxyKeyOf(null)).toBeUndefined()
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,140 @@
|
||||
import * as path from 'path'
|
||||
import type { DownloadAssetOutput } from '@shumai/operations'
|
||||
import { AssetType, prisma, type Prisma } from '@shumai/db'
|
||||
import { HTTPException } from 'hono/http-exception'
|
||||
import { s3Service } from '@shumai/core/src/s3/s3'
|
||||
import { getFileMimeType } from '@shumai/core/src/utils/file-mime'
|
||||
import { READ_URL_EXPIRES_SECONDS } from '@shumai/core/src/asset-read/asset-read'
|
||||
import { storageBucket } from '@shumai/core/src/asset-read/types'
|
||||
|
||||
const assetWithStorage = { storageKey: true } satisfies Prisma.AssetInclude
|
||||
type AssetRecord = Prisma.AssetGetPayload<{ include: typeof assetWithStorage }>
|
||||
|
||||
const FOLDER_TYPES: ReadonlySet<string> = new Set([
|
||||
AssetType.folder,
|
||||
AssetType.root,
|
||||
AssetType.share_root,
|
||||
AssetType.share,
|
||||
])
|
||||
|
||||
/**
|
||||
* The web preview version of a file: the web image for images (and the 4-view image for 3D
|
||||
* models), the transcoded video for videos, otherwise none. Matches what the built-in agent
|
||||
* has always downloaded by default.
|
||||
*/
|
||||
export function proxyKeyOf(media: PrismaJson.MediaInfo | null): string | undefined {
|
||||
if (!media) return undefined
|
||||
if (media.proxyType === 'image' || media.imageTranscodes?.length) {
|
||||
return media.imageTranscodes?.[0]?.key || undefined
|
||||
}
|
||||
if (media.proxyType === 'video' || media.videoTranscodes?.length || media.videoPreview?.key) {
|
||||
return media.videoTranscodes?.[0]?.key || media.videoPreview?.key || undefined
|
||||
}
|
||||
return media.original?.key || undefined
|
||||
}
|
||||
|
||||
/** `clip.mov` downloaded as its `.mp4` proxy is saved as `clip.mp4`. */
|
||||
function nameForKey(assetName: string, key: string, isOriginal: boolean): string {
|
||||
if (isOriginal) return assetName
|
||||
const ext = path.extname(key)
|
||||
if (!ext) return assetName
|
||||
const current = path.extname(assetName)
|
||||
return `${current ? assetName.slice(0, -current.length) : assetName}${ext}`
|
||||
}
|
||||
|
||||
export class AssetDownloadService {
|
||||
/**
|
||||
* Download links for files and version stacks. The caller checks Read on every ID first.
|
||||
* Folders are rejected; version stacks resolve to their latest version and symlinks to
|
||||
* their target.
|
||||
*/
|
||||
async getDownloads(
|
||||
assetIds: string[],
|
||||
opts: { proxy: boolean; includeKeys: boolean },
|
||||
): Promise<DownloadAssetOutput> {
|
||||
const files: DownloadAssetOutput['files'] = []
|
||||
for (const assetId of [...new Set(assetIds)]) {
|
||||
const asset = await this.resolveFile(assetId)
|
||||
const originalKey = asset.storageKey?.key
|
||||
const proxyKey = opts.proxy
|
||||
? proxyKeyOf(asset.media as PrismaJson.MediaInfo | null)
|
||||
: undefined
|
||||
const key = proxyKey || originalKey
|
||||
if (!key) {
|
||||
throw new HTTPException(400, {
|
||||
message: `Asset ${assetId} ("${asset.name}") has no file to download.`,
|
||||
})
|
||||
}
|
||||
const isOriginal = key === originalKey
|
||||
const name = nameForKey(asset.name || 'file', key, isOriginal)
|
||||
|
||||
const bucket = storageBucket()
|
||||
const info = await s3Service.headObject(bucket, key).catch(() => null)
|
||||
if (!info) {
|
||||
throw new HTTPException(404, {
|
||||
message: `The file for asset ${assetId} ("${asset.name}") is missing from storage.`,
|
||||
})
|
||||
}
|
||||
const mimeType =
|
||||
info.contentType && info.contentType !== 'application/octet-stream'
|
||||
? info.contentType
|
||||
: getFileMimeType(null, key, 'application/octet-stream')
|
||||
|
||||
files.push({
|
||||
assetId: asset.id,
|
||||
name,
|
||||
kind: isOriginal ? 'original' : 'proxy',
|
||||
size: info.size,
|
||||
mimeType,
|
||||
url: await s3Service.presign(
|
||||
bucket,
|
||||
key,
|
||||
'GET',
|
||||
true,
|
||||
name,
|
||||
false,
|
||||
READ_URL_EXPIRES_SECONDS,
|
||||
),
|
||||
...(opts.includeKeys ? { key } : {}),
|
||||
})
|
||||
}
|
||||
return { files }
|
||||
}
|
||||
|
||||
private async resolveFile(assetId: string): Promise<AssetRecord> {
|
||||
let asset = await prisma.asset.findUnique({
|
||||
where: { id: assetId },
|
||||
include: assetWithStorage,
|
||||
})
|
||||
if (asset?.type === AssetType.symlink && asset.targetId) {
|
||||
asset = await prisma.asset.findUnique({
|
||||
where: { id: asset.targetId },
|
||||
include: assetWithStorage,
|
||||
})
|
||||
}
|
||||
if (!asset || asset.isDeleted) {
|
||||
throw new HTTPException(404, { message: `Asset with ID ${assetId} not found.` })
|
||||
}
|
||||
if (FOLDER_TYPES.has(asset.type)) {
|
||||
throw new HTTPException(400, {
|
||||
message: `Asset ${assetId} ("${asset.name}") is a folder. Download the files inside it instead.`,
|
||||
})
|
||||
}
|
||||
if (asset.type === AssetType.version_stack) {
|
||||
const latest = await prisma.asset.findFirst({
|
||||
where: { parentId: asset.id, isDeleted: false },
|
||||
orderBy: { sortIndex: 'asc' },
|
||||
include: assetWithStorage,
|
||||
})
|
||||
if (!latest) {
|
||||
throw new HTTPException(400, {
|
||||
message: `Version stack ${assetId} ("${asset.name}") has no versions to download.`,
|
||||
})
|
||||
}
|
||||
asset = latest
|
||||
}
|
||||
return asset
|
||||
}
|
||||
}
|
||||
|
||||
export const assetDownloadService = new AssetDownloadService()
|
||||
@@ -547,7 +547,8 @@ describe('read_asset and get_asset_render Operations', () => {
|
||||
|
||||
it.each([
|
||||
['agent', true],
|
||||
['mcp', false],
|
||||
// The MCP presenter attaches image bytes, then strips the keys (packages/api)
|
||||
['mcp', true],
|
||||
['cli', false],
|
||||
['api', false],
|
||||
] as const)('via %s includes storage keys: %s', async (via, includesKey) => {
|
||||
@@ -589,6 +590,33 @@ describe('read_asset and get_asset_render Operations', () => {
|
||||
expect(waitFor).toHaveBeenLastCalledWith(expect.anything(), budget)
|
||||
})
|
||||
|
||||
it('points unsupported files to the download command of each surface', async () => {
|
||||
const video = await prisma.asset.findUniqueOrThrow({ where: { id: videoId } })
|
||||
const zip = await prisma.asset.create({
|
||||
data: {
|
||||
name: 'bundle.zip',
|
||||
type: AssetType.file,
|
||||
status: 'uploaded',
|
||||
mediaType: 'application/zip',
|
||||
projectId: video.projectId,
|
||||
parentId: video.parentId,
|
||||
},
|
||||
})
|
||||
const prefix = `Unsupported asset type for read_asset. Asset ${zip.id} has media type "application/zip".`
|
||||
|
||||
for (const via of ['agent', 'mcp', 'api'] as const) {
|
||||
await expect(runOperation(readAssetOp, as(via), { assetId: zip.id })).rejects.toMatchObject({
|
||||
status: 400,
|
||||
message: `${prefix} Use download_asset to download this file.`,
|
||||
})
|
||||
}
|
||||
// The CLI has no tools; it names the command instead.
|
||||
await expect(runOperation(readAssetOp, as('cli'), { assetId: zip.id })).rejects.toMatchObject({
|
||||
status: 400,
|
||||
message: `${prefix} Run \`shumai-cli download ${zip.id}\` to download it.`,
|
||||
})
|
||||
})
|
||||
|
||||
it('rejects invalid input before touching the asset', async () => {
|
||||
await expect(
|
||||
runOperation(readAssetOp, as('mcp'), {
|
||||
|
||||
@@ -11,7 +11,13 @@ import { authzService, Permission, ResourceType } from '@shumai/core/src/authz/a
|
||||
import { s3Service } from '@shumai/core/src/s3/s3'
|
||||
import { resolveAnnotationsById } from './annotations'
|
||||
import { findReader, getReader, readers } from './registry'
|
||||
import { badRequest, storageBucket, type ReadableAsset, type ReadResult } from './types'
|
||||
import {
|
||||
badRequest,
|
||||
storageBucket,
|
||||
UnsupportedAssetTypeError,
|
||||
type ReadableAsset,
|
||||
type ReadResult,
|
||||
} from './types'
|
||||
|
||||
/** Who is reading: a user (Read permission required) or the autofill system agent. */
|
||||
export type ReadAccess =
|
||||
@@ -81,9 +87,7 @@ export class AssetReadService {
|
||||
|
||||
const reader = findReader(asset)
|
||||
if (!reader) {
|
||||
throw badRequest(
|
||||
`Unsupported asset type for read_asset. Asset ${asset.id} has media type "${asset.mediaTypeLower || 'unknown'}". Use download_asset to download this file to disk.`,
|
||||
)
|
||||
throw new UnsupportedAssetTypeError(asset.id, asset.mediaTypeLower)
|
||||
}
|
||||
|
||||
// Reject configs meant for another type (empty objects like `imageConfig: {}` are ignored).
|
||||
@@ -265,8 +269,8 @@ export class AssetReadService {
|
||||
export const assetReadService = new AssetReadService()
|
||||
|
||||
/**
|
||||
* Converts a read outcome into the Operation output. Storage keys are included only for
|
||||
* Shumai's built-in agent; every image gets a short-lived link.
|
||||
* Converts a read outcome into the Operation output. Every image gets a short-lived link;
|
||||
* storage keys (and the source file link) only when the result is presented server-side.
|
||||
*/
|
||||
export async function toAssetReadOutput(
|
||||
outcome: ReadOutcome,
|
||||
|
||||
@@ -10,4 +10,5 @@ export {
|
||||
export { resolveAnnotationsById } from './annotations'
|
||||
export { findReader, getReader, readers } from './registry'
|
||||
export { isPlainTextAsset, MAX_TEXT_BYTES, MAX_TEXT_LINES } from './readers/document'
|
||||
export { storageBucket, UnsupportedAssetTypeError } from './types'
|
||||
export type { AssetReader, ReadableAsset, ReadImage, ReadPlan, ReadResult } from './types'
|
||||
|
||||
@@ -99,7 +99,7 @@ export const documentReader: AssetReader<'document'> = {
|
||||
}
|
||||
if (startPage !== null || endPage !== null) {
|
||||
throw badRequest(
|
||||
'startPage and endPage must be null when docConfig mode is "text". To inspect specific page numbers, use mode: "pages".',
|
||||
'startPage and endPage must be left out when docConfig mode is "text". To inspect specific page numbers, use mode: "pages".',
|
||||
)
|
||||
}
|
||||
if (!isPlainTextAsset(asset.mediaTypeLower, asset.filenameLower)) {
|
||||
|
||||
@@ -122,6 +122,16 @@ describe('videoReader', () => {
|
||||
})
|
||||
})
|
||||
|
||||
it('rounds frame timestamps in labels', () => {
|
||||
const result = videoReader.collect!(
|
||||
video,
|
||||
{ start: 0, end: 8, count: 3 },
|
||||
{ screenshots: [{ key: 's/1.webp', timestamp: 2.6666666666666665 }] },
|
||||
{},
|
||||
)
|
||||
expect(result.images[0].label).toBe('Frame at 2.667s')
|
||||
})
|
||||
|
||||
it('fails when the render produced no frames', () => {
|
||||
expect(() =>
|
||||
videoReader.collect!(video, { start: 0, end: 1, count: 1 }, { screenshots: [] }, {}),
|
||||
@@ -140,11 +150,9 @@ describe('model3dReader', () => {
|
||||
} as unknown as PrismaJson.MediaInfo,
|
||||
})
|
||||
|
||||
it('accepts null as "show the overview"', async () => {
|
||||
expect(() =>
|
||||
model3dReader.validate(model, { degree: null }, { hasAnnotation: false }),
|
||||
).not.toThrow()
|
||||
const plan = await model3dReader.plan(model, { degree: null }, noAnnotation)
|
||||
it('shows the overview when no degree is given', async () => {
|
||||
expect(() => model3dReader.validate(model, {}, { hasAnnotation: false })).not.toThrow()
|
||||
const plan = await model3dReader.plan(model, {}, noAnnotation)
|
||||
expect(plan.status).toBe('ready')
|
||||
})
|
||||
|
||||
|
||||
@@ -67,7 +67,8 @@ export const videoReader: AssetReader<'video'> = {
|
||||
details: [],
|
||||
images: screenshots.map((shot) => ({
|
||||
key: shot.key,
|
||||
label: `Frame at ${shot.timestamp}s`,
|
||||
// Evenly spaced timestamps can be long fractions (2.6666666666666665)
|
||||
label: `Frame at ${Number(shot.timestamp.toFixed(3))}s`,
|
||||
mimeType: getFileMimeType(null, shot.key, 'image/webp'),
|
||||
})),
|
||||
}
|
||||
|
||||
@@ -91,6 +91,19 @@ export interface AssetReader<TKind extends ReaderKind = ReaderKind> {
|
||||
): ReadResult
|
||||
}
|
||||
|
||||
/** No reader handles this asset; the message points to download_asset. */
|
||||
export class UnsupportedAssetTypeError extends HTTPException {
|
||||
constructor(
|
||||
readonly assetId: string,
|
||||
readonly mediaType: string,
|
||||
hint = 'Use download_asset to download this file.',
|
||||
) {
|
||||
super(400, {
|
||||
message: `Unsupported asset type for read_asset. Asset ${assetId} has media type "${mediaType || 'unknown'}". ${hint}`,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
export function badRequest(message: string): HTTPException {
|
||||
return new HTTPException(400, { message })
|
||||
}
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
import { downloadAsset } from '@shumai/operations'
|
||||
import { assetDownloadService } from '@shumai/core/src/asset-download/asset-download'
|
||||
import { Permission, ResourceType } from '@shumai/core/src/authz/authz'
|
||||
import { implementOperation } from '../implement'
|
||||
|
||||
export const downloadAssetOp = implementOperation(downloadAsset, {
|
||||
authz: (input) =>
|
||||
[...new Set(input.assetIds)].map((id) => ({
|
||||
permission: Permission.Read,
|
||||
type: ResourceType.Asset,
|
||||
id,
|
||||
})),
|
||||
execute: (input, ctx) =>
|
||||
assetDownloadService.getDownloads(input.assetIds, {
|
||||
proxy: input.proxy !== false,
|
||||
// The built-in agent saves files into its sandbox straight from storage.
|
||||
includeKeys: ctx.via === 'agent',
|
||||
}),
|
||||
})
|
||||
@@ -1,5 +1,9 @@
|
||||
import { getAssetRender, readAsset, type Via } from '@shumai/operations'
|
||||
import { assetReadService, toAssetReadOutput } from '@shumai/core/src/asset-read'
|
||||
import {
|
||||
assetReadService,
|
||||
toAssetReadOutput,
|
||||
UnsupportedAssetTypeError,
|
||||
} from '@shumai/core/src/asset-read'
|
||||
import { implementOperation } from '../implement'
|
||||
|
||||
/**
|
||||
@@ -13,6 +17,13 @@ export const READ_WAIT_BUDGET_MS: Record<Via, number> = {
|
||||
api: 20_000,
|
||||
}
|
||||
|
||||
/**
|
||||
* Surfaces whose results are presented on the server (agent tools, MCP tool results). They get
|
||||
* storage keys so the presenter can attach image bytes; the MCP presenter strips the keys
|
||||
* before anything leaves the server. CLI and REST callers only get links.
|
||||
*/
|
||||
const SERVER_PRESENTED: ReadonlySet<Via> = new Set(['agent', 'mcp'])
|
||||
|
||||
/** `maxWaitSeconds` can only shorten the surface budget. */
|
||||
export function waitBudgetMs(via: Via, maxWaitSeconds: number | undefined): number {
|
||||
const budget = READ_WAIT_BUDGET_MS[via]
|
||||
@@ -24,10 +35,22 @@ export const readAssetOp = implementOperation(readAsset, {
|
||||
authz: () => [],
|
||||
execute: async (input, ctx) => {
|
||||
const { maxWaitSeconds, ...request } = input
|
||||
const outcome = await assetReadService.read({ kind: 'user', userId: ctx.userId }, request, {
|
||||
waitBudgetMs: waitBudgetMs(ctx.via, maxWaitSeconds),
|
||||
})
|
||||
return toAssetReadOutput(outcome, { includeKeys: ctx.via === 'agent' })
|
||||
const outcome = await assetReadService
|
||||
.read({ kind: 'user', userId: ctx.userId }, request, {
|
||||
waitBudgetMs: waitBudgetMs(ctx.via, maxWaitSeconds),
|
||||
})
|
||||
.catch((err: unknown) => {
|
||||
// CLI users download with a command, not a tool.
|
||||
if (err instanceof UnsupportedAssetTypeError && ctx.via === 'cli') {
|
||||
throw new UnsupportedAssetTypeError(
|
||||
err.assetId,
|
||||
err.mediaType,
|
||||
`Run \`shumai-cli download ${err.assetId}\` to download it.`,
|
||||
)
|
||||
}
|
||||
throw err
|
||||
})
|
||||
return toAssetReadOutput(outcome, { includeKeys: SERVER_PRESENTED.has(ctx.via) })
|
||||
},
|
||||
})
|
||||
|
||||
@@ -38,6 +61,6 @@ export const getAssetRenderOp = implementOperation(getAssetRender, {
|
||||
const outcome = await assetReadService.getRender(ctx.userId, input.taskId, {
|
||||
waitBudgetMs: waitBudgetMs(ctx.via, input.maxWaitSeconds),
|
||||
})
|
||||
return toAssetReadOutput(outcome, { includeKeys: ctx.via === 'agent' })
|
||||
return toAssetReadOutput(outcome, { includeKeys: SERVER_PRESENTED.has(ctx.via) })
|
||||
},
|
||||
})
|
||||
|
||||
@@ -12,6 +12,7 @@ export { runOperation } from './run'
|
||||
export { getOperation, getOperationsForSurface, operations } from './registry'
|
||||
export { createFolderOp } from './asset/create-folder'
|
||||
export { deleteAssetOp } from './asset/delete-asset'
|
||||
export { downloadAssetOp } from './asset/download-asset'
|
||||
export { listAssetsOp } from './asset/list-assets'
|
||||
export { moveAssetsOp } from './asset/move-assets'
|
||||
export {
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import type { OperationName, Surface } from '@shumai/operations'
|
||||
import { createFolderOp } from './asset/create-folder'
|
||||
import { deleteAssetOp } from './asset/delete-asset'
|
||||
import { downloadAssetOp } from './asset/download-asset'
|
||||
import { listAssetsOp } from './asset/list-assets'
|
||||
import { moveAssetsOp } from './asset/move-assets'
|
||||
import { getAssetRenderOp, readAssetOp } from './asset/read-asset'
|
||||
@@ -34,6 +35,7 @@ export const operations = defineRegistry([
|
||||
deleteAssetOp,
|
||||
readAssetOp,
|
||||
getAssetRenderOp,
|
||||
downloadAssetOp,
|
||||
] as const)
|
||||
|
||||
const operationsByName = new Map<string, Operation>()
|
||||
|
||||
@@ -22,7 +22,8 @@ type ExecutableTool = {
|
||||
|
||||
/**
|
||||
* read_asset end to end: real render workflows (frames, PDF pages) through the reader
|
||||
* pipeline, the pending → get_asset_render path, and the agent tool in a chat workflow.
|
||||
* pipeline, the pending → get_asset_render path, and the agent's read_asset and
|
||||
* download_asset tools in a chat workflow.
|
||||
* Only the model call is mocked.
|
||||
*/
|
||||
describe('Workflow E2E - read_asset', () => {
|
||||
@@ -56,10 +57,20 @@ describe('Workflow E2E - read_asset', () => {
|
||||
})
|
||||
const text = result.content.find((c) => c.type === 'text')?.text ?? ''
|
||||
const images = result.content.filter((c) => c.type === 'image' && c.data)
|
||||
// download_asset saves the original into the agent's sandbox (.pi)
|
||||
const download = this.getTools().find((t) => t.name === 'download_asset') as unknown as
|
||||
ExecutableTool | undefined
|
||||
const downloaded = download
|
||||
? await download.execute('call-2', { assetIds: [videoAssetId], proxy: false })
|
||||
: undefined
|
||||
const savedPath = (
|
||||
downloaded as unknown as { details?: { files?: Array<{ absolutePath: string }> } }
|
||||
)?.details?.files?.[0]?.absolutePath
|
||||
const summary = [
|
||||
`images:${images.length}`,
|
||||
`frames:${(text.match(/- Frame at /g) ?? []).length}`,
|
||||
`render_tool:${hasRenderTool ? 'present' : 'absent'}`,
|
||||
`downloaded:${savedPath ?? 'none'}`,
|
||||
].join('|')
|
||||
return {
|
||||
content: [{ type: 'text', text: summary }],
|
||||
@@ -283,6 +294,12 @@ describe('Workflow E2E - read_asset', () => {
|
||||
expect(reply?.message).toContain('images:2')
|
||||
expect(reply?.message).toContain('frames:2')
|
||||
expect(reply?.message).toContain('render_tool:present')
|
||||
const savedPath = /downloaded:([^|]+)/.exec(reply?.message ?? '')?.[1] ?? 'none'
|
||||
expect(savedPath).not.toBe('none')
|
||||
expect(
|
||||
fs.readFileSync(savedPath).equals(fs.readFileSync(path.join(fixturesDir, 'small.mp4'))),
|
||||
).toBe(true)
|
||||
fs.rmSync(savedPath, { force: true })
|
||||
|
||||
const render = await prisma.workflowTask.findFirstOrThrow({
|
||||
where: { assetId: video.id, type: 'transcode_screenshot' },
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
import { z } from 'zod'
|
||||
import { defineOperation } from '../define'
|
||||
|
||||
/** Most assets one download call accepts. */
|
||||
export const MAX_DOWNLOAD_ASSETS = 50
|
||||
|
||||
const downloadFileSchema = z.object({
|
||||
assetId: z
|
||||
.string()
|
||||
.describe('The asset this file belongs to (the latest version for a version stack).'),
|
||||
name: z.string().describe('File name to save it as.'),
|
||||
kind: z
|
||||
.enum(['proxy', 'original'])
|
||||
.describe('"proxy" is the web preview version; "original" is the uploaded file.'),
|
||||
size: z.number().describe('Size in bytes.'),
|
||||
mimeType: z.string(),
|
||||
url: z.string().describe('Short-lived download link (about 1 hour).'),
|
||||
/** Storage key; only returned to Shumai's built-in agent. */
|
||||
key: z.string().optional(),
|
||||
})
|
||||
|
||||
export const downloadAssetOutputSchema = z.object({
|
||||
files: z.array(downloadFileSchema),
|
||||
})
|
||||
export type DownloadAssetOutput = z.infer<typeof downloadAssetOutputSchema>
|
||||
|
||||
export const downloadAsset = defineOperation({
|
||||
name: 'download_asset',
|
||||
title: 'Download Asset',
|
||||
description:
|
||||
`Gets download links for up to ${MAX_DOWNLOAD_ASSETS} files. ` +
|
||||
'By default it returns the web preview version (proxy) when there is one, e.g. the transcoded video or the web image, which is smaller and easier to process; set proxy to false for the original uploaded file. ' +
|
||||
'Files without a preview (e.g. audio, archives, text) always return the original. ' +
|
||||
'A version stack returns its latest version. Folders are not supported: download the files inside them instead. ' +
|
||||
'Links expire after about an hour.',
|
||||
input: z.object({
|
||||
assetIds: z
|
||||
.array(z.string().min(1))
|
||||
.min(1)
|
||||
.max(MAX_DOWNLOAD_ASSETS)
|
||||
.describe(`IDs of the files or version stacks to download (1 to ${MAX_DOWNLOAD_ASSETS}).`),
|
||||
proxy: z
|
||||
.boolean()
|
||||
.optional()
|
||||
.describe(
|
||||
'Download the web preview version when there is one (default true). Set to false for the original uploaded file.',
|
||||
),
|
||||
}),
|
||||
output: downloadAssetOutputSchema,
|
||||
surfaces: ['agent', 'mcp', 'cli'],
|
||||
annotations: { readOnly: true, destructive: false, idempotent: true },
|
||||
cli: {
|
||||
command: ['download'],
|
||||
description:
|
||||
'Download files into a folder (the current folder by default). Saves the web preview version when there is one; use --original for the uploaded file.',
|
||||
positional: ['assetIds'],
|
||||
// The CLI exposes `--original` instead of a `--proxy` flag that is on by default.
|
||||
omit: ['proxy'],
|
||||
},
|
||||
})
|
||||
@@ -20,10 +20,9 @@ export const readerOptionSchemas = {
|
||||
degree: z
|
||||
.number()
|
||||
.int()
|
||||
.nullable()
|
||||
.optional()
|
||||
.describe(
|
||||
'Turntable rotation degree (integer: 0, 15, 30, ..., 345 in 15° increments) to inspect a specific angle of a 3D model. Leave null/omitted to view the 4-view composite overview.',
|
||||
'Turntable rotation degree (integer: 0, 15, 30, ..., 345 in 15° increments) to inspect a specific angle of a 3D model. Leave it out to view the 4-view composite overview.',
|
||||
),
|
||||
})
|
||||
.describe(
|
||||
@@ -57,12 +56,10 @@ export const readerOptionSchemas = {
|
||||
),
|
||||
startPage: z
|
||||
.number()
|
||||
.nullable()
|
||||
.optional()
|
||||
.describe('Start page number (1-based, required when mode="pages"; omit for "text").'),
|
||||
endPage: z
|
||||
.number()
|
||||
.nullable()
|
||||
.optional()
|
||||
.describe(
|
||||
`End page number (1-based, max ${MAX_RENDER_ITEMS} pages per call, required when mode="pages"; omit for "text").`,
|
||||
@@ -156,8 +153,18 @@ export const readAsset = defineOperation({
|
||||
maxWaitSeconds,
|
||||
}),
|
||||
output: assetReadOutputSchema,
|
||||
surfaces: ['agent'],
|
||||
surfaces: ['agent', 'mcp', 'cli'],
|
||||
annotations: { readOnly: true, destructive: false },
|
||||
cli: {
|
||||
command: ['read'],
|
||||
description:
|
||||
'Read an asset: show an image, video frames, PDF pages, a 3D view, or the text of a text file. ' +
|
||||
'Prints links to the images; use --out to download them.',
|
||||
positional: ['assetId'],
|
||||
flatten: { videoConfig: 'video', docConfig: 'doc', config3d: '', imageConfig: 'image' },
|
||||
// Text content is cheap to print; describing without content is only useful to agents.
|
||||
omit: ['includeContent'],
|
||||
},
|
||||
})
|
||||
|
||||
export const getAssetRender = defineOperation({
|
||||
@@ -171,6 +178,11 @@ export const getAssetRender = defineOperation({
|
||||
maxWaitSeconds,
|
||||
}),
|
||||
output: assetReadOutputSchema,
|
||||
surfaces: ['agent'],
|
||||
surfaces: ['agent', 'mcp', 'cli'],
|
||||
annotations: { readOnly: true, destructive: false, idempotent: true },
|
||||
cli: {
|
||||
command: ['render'],
|
||||
description: 'Get the result of a `read` that is still rendering, by its task ID.',
|
||||
positional: ['taskId'],
|
||||
},
|
||||
})
|
||||
|
||||
@@ -81,4 +81,24 @@ describe('defineOperation', () => {
|
||||
}),
|
||||
).toThrow('unknown input "extra"')
|
||||
})
|
||||
|
||||
it('validates flattened and omitted cli fields', () => {
|
||||
const input = z.object({
|
||||
id: z.string(),
|
||||
config: z.object({ size: z.number() }).optional(),
|
||||
verbose: z.boolean().optional(),
|
||||
})
|
||||
const withCli = (cli: Record<string, unknown>) =>
|
||||
defineOperation({
|
||||
...baseDefinition(),
|
||||
input,
|
||||
surfaces: ['cli'],
|
||||
cli: { command: ['get'], ...cli },
|
||||
} as OperationDefinition<string, typeof input>)
|
||||
|
||||
expect(() => withCli({ flatten: { config: 'config' }, omit: ['verbose'] })).not.toThrow()
|
||||
expect(() => withCli({ flatten: { nope: 'n' } })).toThrow('unknown input "nope"')
|
||||
expect(() => withCli({ omit: ['gone'] })).toThrow('unknown input "gone"')
|
||||
expect(() => withCli({ omit: ['id'] })).toThrow('omits required input "id"')
|
||||
})
|
||||
})
|
||||
|
||||
@@ -37,6 +37,14 @@ export interface CliSpec<TInput, TOutput> {
|
||||
positional?: readonly (keyof TInput & string)[]
|
||||
/** Flag spellings for input fields, e.g. `{ parent: ['-p', '--parent'] }`. */
|
||||
flags?: { readonly [K in keyof TInput & string]?: readonly string[] }
|
||||
/**
|
||||
* Object input fields exposed as one flag per sub-field, keyed by flag prefix: with
|
||||
* `{ videoConfig: 'video' }`, `videoConfig.start` becomes `--video-start`. An empty prefix
|
||||
* uses the sub-field name alone (`--degree`). Object fields must be flattened or omitted.
|
||||
*/
|
||||
flatten?: { readonly [K in keyof TInput & string]?: string }
|
||||
/** Input fields the CLI does not expose. They must be optional. */
|
||||
omit?: readonly (keyof TInput & string)[]
|
||||
/** Input values the CLI uses when the user does not pass them (e.g. a larger page size). */
|
||||
defaults?: { readonly [K in keyof TInput & string]?: TInput[K] }
|
||||
/** Human-friendly rendering of the output; the CLI falls back to a generic renderer. */
|
||||
@@ -101,12 +109,19 @@ export function defineOperation<
|
||||
...(def.cli.positional ?? []),
|
||||
...Object.keys(def.cli.flags ?? {}),
|
||||
...Object.keys(def.cli.defaults ?? {}),
|
||||
...Object.keys(def.cli.flatten ?? {}),
|
||||
...(def.cli.omit ?? []),
|
||||
]
|
||||
for (const field of referenced) {
|
||||
if (!fields.has(field)) {
|
||||
throw new Error(`Operation "${def.name}" cli spec references unknown input "${field}"`)
|
||||
}
|
||||
}
|
||||
for (const field of def.cli.omit ?? []) {
|
||||
if (!(def.input.shape[field] as z.ZodType).safeParse(undefined).success) {
|
||||
throw new Error(`Operation "${def.name}" cli spec omits required input "${field}"`)
|
||||
}
|
||||
}
|
||||
}
|
||||
return Object.freeze(def)
|
||||
}
|
||||
|
||||
@@ -1,10 +1,12 @@
|
||||
export * from './define'
|
||||
export * from './schemas'
|
||||
export * from './registry'
|
||||
export { dropNullOptionals } from './null-optionals'
|
||||
export { createFolder } from './asset/create-folder'
|
||||
export { deleteAsset } from './asset/delete-asset'
|
||||
export { listAssets } from './asset/list-assets'
|
||||
export { moveAssets } from './asset/move-assets'
|
||||
export { renameAsset } from './asset/rename-asset'
|
||||
export * from './asset/read-asset'
|
||||
export * from './asset/download-asset'
|
||||
export { listProjects } from './project/list-projects'
|
||||
|
||||
@@ -0,0 +1,44 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { z } from 'zod'
|
||||
import { dropNullOptionals } from './null-optionals'
|
||||
|
||||
const schema = z.object({
|
||||
id: z.string(),
|
||||
note: z.string().optional(),
|
||||
config: z
|
||||
.object({
|
||||
mode: z.enum(['a', 'b']),
|
||||
start: z.number().optional(),
|
||||
})
|
||||
.optional(),
|
||||
tags: z.array(z.string()).optional(),
|
||||
})
|
||||
|
||||
describe('dropNullOptionals', () => {
|
||||
it('drops nulls for optional fields, including inside nested objects', () => {
|
||||
expect(
|
||||
dropNullOptionals(schema, {
|
||||
id: 'x',
|
||||
note: null,
|
||||
config: { mode: 'a', start: null },
|
||||
tags: null,
|
||||
}),
|
||||
).toEqual({ id: 'x', config: { mode: 'a' } })
|
||||
})
|
||||
|
||||
it('drops a null nested object when the object is optional', () => {
|
||||
expect(dropNullOptionals(schema, { id: 'x', config: null })).toEqual({ id: 'x' })
|
||||
})
|
||||
|
||||
it('keeps nulls for required fields so validation still fails', () => {
|
||||
const cleaned = dropNullOptionals(schema, { id: null, config: { mode: null } })
|
||||
expect(cleaned).toEqual({ id: null, config: { mode: null } })
|
||||
expect(schema.safeParse(cleaned).success).toBe(false)
|
||||
})
|
||||
|
||||
it('leaves unknown keys and non-object values alone', () => {
|
||||
expect(dropNullOptionals(schema, { id: 'x', other: null })).toEqual({ id: 'x', other: null })
|
||||
expect(dropNullOptionals(schema, null)).toBeNull()
|
||||
expect(dropNullOptionals(schema, ['a'])).toEqual(['a'])
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,29 @@
|
||||
import { z } from 'zod'
|
||||
|
||||
/** Strip optional/default wrappers to reach an object schema, if the field is one. */
|
||||
function objectSchemaOf(schema: z.ZodType): z.ZodObject | undefined {
|
||||
let current: z.ZodType = schema
|
||||
while (current instanceof z.ZodOptional || current instanceof z.ZodDefault) {
|
||||
current = current.unwrap() as z.ZodType
|
||||
}
|
||||
return current instanceof z.ZodObject ? current : undefined
|
||||
}
|
||||
|
||||
/**
|
||||
* Models often send `null` for optional parameters they don't want to set, including inside
|
||||
* nested config objects. Operation inputs are optional rather than nullable (nullable types
|
||||
* don't translate to every model's tool format), so drop those nulls before validating:
|
||||
* they are treated as omitted. Nulls for required fields are kept and still fail validation.
|
||||
*/
|
||||
export function dropNullOptionals(schema: z.ZodObject, value: unknown): unknown {
|
||||
if (!value || typeof value !== 'object' || Array.isArray(value)) return value
|
||||
const shape = schema.shape as Record<string, z.ZodType>
|
||||
const result: Record<string, unknown> = {}
|
||||
for (const [key, field] of Object.entries(value)) {
|
||||
const fieldSchema = shape[key]
|
||||
if (field === null && fieldSchema?.safeParse(undefined).success) continue
|
||||
const nested = fieldSchema && objectSchemaOf(fieldSchema)
|
||||
result[key] = nested ? dropNullOptionals(nested, field) : field
|
||||
}
|
||||
return result
|
||||
}
|
||||
@@ -19,6 +19,7 @@ describe('operation registry', () => {
|
||||
expect(operationDefinitions.map((d) => d.name).sort()).toEqual([
|
||||
'create_folder',
|
||||
'delete_asset',
|
||||
'download_asset',
|
||||
'get_asset_render',
|
||||
'list_assets',
|
||||
'list_projects',
|
||||
@@ -36,6 +37,10 @@ describe('operation registry', () => {
|
||||
it('filters definitions by surface', () => {
|
||||
const cli = getOperationDefinitionsForSurface('cli').map((d) => d.name)
|
||||
expect(cli).toContain('list_assets')
|
||||
expect(cli).toContain('read_asset')
|
||||
expect(getOperationDefinitionsForSurface('mcp').map((d) => d.name)).toContain(
|
||||
'get_asset_render',
|
||||
)
|
||||
})
|
||||
|
||||
it('annotates only delete_asset as destructive', () => {
|
||||
@@ -43,6 +48,7 @@ describe('operation registry', () => {
|
||||
expect(destructive.map((d) => d.name)).toEqual(['delete_asset'])
|
||||
const readOnly = operationDefinitions.filter((d) => d.annotations.readOnly)
|
||||
expect(readOnly.map((d) => d.name).sort()).toEqual([
|
||||
'download_asset',
|
||||
'get_asset_render',
|
||||
'list_assets',
|
||||
'list_projects',
|
||||
@@ -66,6 +72,18 @@ describe('operation registry', () => {
|
||||
},
|
||||
)
|
||||
|
||||
it.each(operationDefinitions.map((d) => [d.name, d] as const))(
|
||||
'%s input schema has no nullable types',
|
||||
(_name, definition) => {
|
||||
// Tool formats like Gemini's function declarations (an OpenAPI subset) can't express
|
||||
// `type: [..., "null"]` or a `null` branch, so inputs use optional fields instead.
|
||||
// Surfaces drop nulls that models send for optional fields (dropNullOptionals).
|
||||
const schema = JSON.stringify(z.toJSONSchema(definition.input, { io: 'input' }))
|
||||
expect(schema).not.toMatch(/"type":\[/)
|
||||
expect(schema).not.toContain('"type":"null"')
|
||||
},
|
||||
)
|
||||
|
||||
it('keeps legacy CLI invocations mapped', () => {
|
||||
const commands = operationDefinitions.map((d) => [
|
||||
d.name,
|
||||
@@ -78,9 +96,9 @@ describe('operation registry', () => {
|
||||
['rename_asset', ['rename']],
|
||||
['move_assets', ['move', 'mv']],
|
||||
['delete_asset', ['delete', 'rm']],
|
||||
// CLI commands for reading assets arrive with the CLI surface
|
||||
['read_asset', []],
|
||||
['get_asset_render', []],
|
||||
['read_asset', ['read']],
|
||||
['get_asset_render', ['render']],
|
||||
['download_asset', ['download']],
|
||||
])
|
||||
})
|
||||
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { createFolder } from './asset/create-folder'
|
||||
import { deleteAsset } from './asset/delete-asset'
|
||||
import { downloadAsset } from './asset/download-asset'
|
||||
import { listAssets } from './asset/list-assets'
|
||||
import { moveAssets } from './asset/move-assets'
|
||||
import { getAssetRender, readAsset } from './asset/read-asset'
|
||||
@@ -20,6 +21,7 @@ export const operationDefinitions = [
|
||||
deleteAsset,
|
||||
readAsset,
|
||||
getAssetRender,
|
||||
downloadAsset,
|
||||
] as const
|
||||
|
||||
export type RegisteredOperationDefinition = (typeof operationDefinitions)[number]
|
||||
|
||||
@@ -112,6 +112,11 @@ const BUILTIN_TOOLS = [
|
||||
name: () => m.agent_tool_create_version_name(),
|
||||
description: () => m.agent_tool_create_version_desc(),
|
||||
},
|
||||
{
|
||||
id: 'download_asset',
|
||||
name: () => m.agent_tool_download_asset_name(),
|
||||
description: () => m.agent_tool_download_asset_desc(),
|
||||
},
|
||||
{
|
||||
id: 'generate_image',
|
||||
name: () => m.agent_tool_generate_image_name(),
|
||||
|
||||
@@ -609,6 +609,8 @@
|
||||
"agent_tool_read_skill_desc": "Allows the agent to load and read skill definitions to perform specialized actions.",
|
||||
"agent_tool_read_asset_name": "Read Asset",
|
||||
"agent_tool_read_asset_desc": "Allows the agent to view and inspect images, video frames, and document pages.",
|
||||
"agent_tool_download_asset_name": "Download Asset",
|
||||
"agent_tool_download_asset_desc": "Allows the agent to download files to its workspace to process them with scripts or terminal commands.",
|
||||
"agent_tool_get_asset_render_name": "Get Asset Render",
|
||||
"agent_tool_get_asset_render_desc": "Allows the agent to pick up video frames or document pages that took longer to render.",
|
||||
"agent_tool_list_projects_name": "List Projects",
|
||||
|
||||
@@ -609,6 +609,8 @@
|
||||
"agent_tool_read_skill_desc": "允许智能体加载和读取技能定义以执行特定操作。",
|
||||
"agent_tool_read_asset_name": "读取资源",
|
||||
"agent_tool_read_asset_desc": "允许智能体查看和分析图片、视频画面和文档页面。",
|
||||
"agent_tool_download_asset_name": "下载资源",
|
||||
"agent_tool_download_asset_desc": "允许智能体将文件下载到其工作区,以便用脚本或终端命令处理。",
|
||||
"agent_tool_get_asset_render_name": "获取资源渲染结果",
|
||||
"agent_tool_get_asset_render_desc": "允许智能体获取渲染耗时较长的视频画面或文档页面。",
|
||||
"agent_tool_list_projects_name": "列出项目",
|
||||
|
||||
Reference in New Issue
Block a user