Compare commits

..
7 Commits
Author SHA1 Message Date
yiling f7ed3bafdf Release v0.5.3 2026-10-16 22:50:47 +08:00
Yiling-J 9f76e61e23 Add documentation for LiteRT-LM Embedding Server setup
Updated documentation for LiteRT-LM Embedding Server setup and other features.
2026-10-16 22:42:07 +08:00
Yiling-J 7c297895e5 feat(embedding): add compose tiers, docs, and regenerate-embeddings CLI (#480)
* feat(embedding): update compose tiers, docs, and add regenerate-embeddings CLI

- Add Lite, Embedding, and Full Docker Compose configurations (and Aliyun mirrors)
- Set CONCURRENCY_AGENT to 1 for compose files containing embedding-server
- Update README and README_CN with compose options, recommended RAM, and semantic search links
- Add LiteRT-LM Embedding Server setup guide in docs/using-shumai/embedding-server.mdx
- Add shumai regenerate-embeddings CLI command and EmbeddingService to regenerate embeddings across teams
- Add comprehensive test coverage for EmbeddingService

* fix(docker): remove basic auth for turntable-renderer in full compose

* fix(agent): lock and disable embedding agent when ENABLE_EMBEDDING_AGENT is false

Ensure that when ENABLE_EMBEDDING_AGENT is set to 'false', existing embedding agents are automatically disabled on startup, marked as read-only, and blocked from manual API modifications or creations. Additionally guard background task creation and semantic search when the environment variable disables embedding agents.

* test(upload): remove flaky status assertion on workflow tasks
2026-10-16 21:20:25 +08:00
Yiling-J 0f318b4050 feat(search): render results in responsive card grid with sprite preview and new-tab navigation (#479)
* feat(search): render results in responsive card grid with sprite preview and new-tab navigation

* fix(search): align semantic matched duration badge style with standard duration badge

* fix(webui): use pointer cursor instead of crosshair cursor for sprite scrubber on hover
2026-10-16 10:42:32 +08:00
Yiling-J 1a46f63160 fix(agent): adaptively set video embedding frame count based on aspect ratio (#477)
* fix(agent): adaptively set video embedding frame count based on aspect ratio

Adjust sampled video frame counts between 15 and 14 based on the video's aspect ratio. Standard aspect ratios (1:1, 4:3, 3:4, 16:9, 9:16, 21:9) stay within the 2048 token signature bucket when using 15 frames, while non-standard or custom aspect ratios use 14 frames to guarantee that sequence length never exceeds 2048 tokens, preventing out-of-memory errors on the embedding server.

* feat(agent): persist video chunk embeddings incrementally and resume on retry (#478)
2026-10-16 08:36:51 +08:00
Yiling-J 6cc90bf973 feat(search): generate query embeddings directly for semantic search (#476)
* feat(search): generate query embeddings directly for semantic search

* fix(search): resolve embedding provider from env, redact query logs, and preserve case in cache
2026-10-15 20:49:45 +08:00
Yiling-J b7271777e0 feat(agent): support gemma-embedding-2-openai provider (#475)
* feat(agent): support gemma-embedding-2-openai provider

Add support for local LiteRT-LM Gemma Embedding 2 provider with 768-dimensional vector embeddings, environment-configured read-only embedding agents, 16-frame uniform sampling for video chunks, and automatic WebP-to-JPEG conversion.

* refactor(agent): remove unused legacy generateEmbeddingActivity

* chore(agent): align sharp version to 0.35.4

* refactor(agent): rename embedding server env to GEMMA_EMBEDDING_OPENAI_BASE_URL

* refactor(agent,transcode): use native Bun.Archive for video chunk frames

* fix(transcode,agent): populate buffer entries in Bun.Archive and validate frame sizes
2026-10-15 15:49:24 +08:00
77 changed files with 4305 additions and 1356 deletions
+16
View File
@@ -12,6 +12,22 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
### Changed
## [0.5.3] - 2026-10-16
### Added
* **cli**: Add administrative `shumai regenerate-embeddings` command to regenerate multimodal vector embeddings for all existing media files across active teams
* **docker**: Provide three Docker Compose deployment tiers (Lite, with Embedding Server, and Full) with recommended memory specifications
* **agent**: Save video chunk embeddings incrementally to the database as each chunk finishes, preserving completed progress on failure and resuming from where it left off on retry
* **agent**: Add support for local LiteRT-LM Gemma Embedding 2 (`gemma-embedding-2-openai`) provider and environment-enforced read-only embedding agents with automatic 768-dimensional vector search indexing
### Fixed
### Changed
* **search**: Display search results in a responsive card grid with interactive hover video sprite scrubbing, semantic match chunk time range previews, and new-tab opening on click
* **search**: Significantly improve semantic search response times by generating query text embeddings directly with in-memory caching instead of dispatching background tasks
## [0.5.2] - 2026-10-13
### Added
+39 -192
View File
@@ -35,6 +35,7 @@ Shumai is an open-source Frame.io alternative built for creative teams to collab
* **Wide File Format Support**: Preview common creative file formats, including images (with PSD support), videos, audio, and documents, while allowing any file type to be uploaded and stored.
* **3D Asset Turntable Previews**: Generate interactive 3D model turntable previews and videos for formats like GLB, FBX, OBJ, STL, USD, and DAE using [turntable-renderer](https://docs.shumai.one/using-shumai/3d-asset-support).
* **Office & Document PDF Proxying via Gotenberg**: Convert Word, PowerPoint, Excel, Markdown, HTML, and CSV documents to high-quality PDF proxies for in-browser previewing using [Gotenberg](https://docs.shumai.one/using-shumai/gotenberg).
* **Semantic Search**: Find assets by visual or conceptual meaning using vector embeddings. See [documentation](https://docs.shumai.one/using-shumai/semantic-search).
* **NLE Export & Premiere Pro Plugin (Experimental)**: Export timestamped review comments to your favorite non-linear editor (Premiere Pro XML, DaVinci Resolve EDL, Final Cut Pro, Avid Media Composer) or use the experimental Premiere Pro UXP plugin to import media assets and auto-sync comments directly to sequence timeline markers. See [documentation](https://docs.shumai.one/using-shumai/nle-and-premiere).
But Shumai goes beyond being a Frame.io alternative. With modern models like GPT-Image-2 and Seedance 2.5, AI can already handle a surprising amount of creative work. That means you can create or update videos and images from a simple prompt without having to jump back into Photoshop, Premiere or Houdini. That is why Shumai includes a powerful agent system:
@@ -47,7 +48,6 @@ But Shumai goes beyond being a Frame.io alternative. With modern models like GPT
* **Model Context Protocol (MCP)**: Connect agents to external tools and APIs through remote MCP servers, with OAuth authentication and per-agent MCP access.
* **Isolated Sandbox Execution**: Run agent-submitted scripts safely in a sandboxed environment.
* **AI Metadata Autofill**: Automatically fill in custom metadata for new assets.
* **Semantic Search**: Find assets by visual or conceptual meaning using vector embeddings (Gemini Embedding 2).
* **Resource Quotas**: Set usage limits on AI tokens, costs, and tool execution per member, role, or team-wide, with real-time tracking.
Here is a short demo video showing what the Shumai agent can do:
@@ -83,13 +83,25 @@ Docker Compose is the fastest way to get Shumai running. You do not need to clon
```bash
mkdir shumai && cd shumai
```
2. Download the `docker-compose.yaml` file:
```bash
curl -o docker-compose.yaml https://raw.githubusercontent.com/shumaiOne/shumai/main/docker-compose/local/docker-compose.yaml
```
2. Download your preferred `docker-compose.yaml` configuration based on your needs and available RAM:
* **Lite (Suggested RAM: 2 GB+)**: Minimal deployment containing Shumai and PostgreSQL database.
```bash
curl -o docker-compose.yaml https://raw.githubusercontent.com/shumaiOne/shumai/main/docker-compose/local/docker-compose.yaml
```
* **With Embedding Server (Suggested RAM: 4 GB+)**: Adds local multimodal semantic search powered by `shumaiOne/embedding-server` (`EmbeddingGemma 2 740M`).
```bash
curl -o docker-compose.yaml https://raw.githubusercontent.com/shumaiOne/shumai/main/docker-compose/local/docker-compose.embedding.yaml
```
* **Full (Suggested RAM: 8 GB+)**: All-in-one setup including `embedding-server`, `turntable-renderer` (3D previews), and `Gotenberg` (document PDF proxying).
```bash
curl -o docker-compose.yaml https://raw.githubusercontent.com/shumaiOne/shumai/main/docker-compose/local/docker-compose.full.yaml
```
> [!NOTE]
> By default, the standard `docker-compose.yaml` uses CPU software transcoding. If your server has a GPU (NVIDIA NVENC, Intel QuickSync, AMD/Intel VA-API, or Rockchip RKMPP) and you want to enable GPU hardware acceleration, check the [Hardware Transcoding Guide](https://docs.shumai.one/configuration/hardware-transcoding).
> By default, the Docker Compose setups use CPU software transcoding. If your server has a GPU (NVIDIA NVENC, Intel QuickSync, AMD/Intel VA-API, or Rockchip RKMPP) and you want to enable GPU hardware acceleration, check the [Hardware Transcoding Guide](https://docs.shumai.one/configuration/hardware-transcoding).
3. Configure environment variables (optional):
* `SHUMAI_SERVER_PORT` controls the port the Shumai server listens on. The default is `3000`.
@@ -129,190 +141,7 @@ Docker Compose is the fastest way to get Shumai running. You do not need to clon
```
6. Open your browser and access Shumai at `http://localhost:3000` (or `http://<your-server-ip>:3000` for remote deployments).
### Option 2: Install via NPM / Package Manager
Shumai is published as `@shumai-one/shumai` on NPM. This option allows you to run Shumai globally or locally.
#### Step 1: Start PostgreSQL with pgvector
Shumai requires PostgreSQL with the `pgvector` extension. Start a pre-configured database container using Docker:
```bash
docker run --name shumai_postgres \
-e POSTGRES_USER=shumai \
-e POSTGRES_PASSWORD=shumai_password \
-e POSTGRES_DB=shumai_db \
-p 5432:5432 \
-d pgvector/pgvector:pg18
```
#### Step 2: Create a workspace folder
Create a dedicated directory to store your environment configuration and media files (which are saved in a `./data` directory by default):
```bash
mkdir shumai && cd shumai
```
#### Step 3: Install Platform-Specific Dependencies
#### Linux
Shumai requires the following system packages on Linux:
- **`ffmpeg`** – Used for media transcoding and metadata extraction.
- **`poppler`** (`poppler-utils`) – Used for PDF page image extraction and PDF sprite preview generation (`pdftoppm`).
- **`imagemagick`** – Used for PSD format transcoding and color profile conversion to sRGB.
- **`dcraw_emu`** (`libraw` / `libraw-bin`) – Used for camera RAW format image decoding and preview generation.
- **`bubblewrap`**, **`socat`**, and **`ripgrep`** – Required by the AI agent sandbox (`anthropic-experimental/sandbox-runtime`) for process isolation, networking, and workspace search.
Install all required packages with one command:
> Ubuntu/Debian
```bash
sudo apt install -y ffmpeg poppler-utils imagemagick bubblewrap socat ripgrep libraw-bin
```
> Fedora
```bash
sudo dnf install -y ffmpeg poppler-utils ImageMagick bubblewrap socat ripgrep LibRaw
```
> Arch Linux
```bash
sudo pacman -S --noconfirm ffmpeg poppler imagemagick bubblewrap socat ripgrep libraw
```
> [!NOTE]
> **Ubuntu 24.04+** restricts unprivileged user namespaces by default. This prevents `anthropic-experimental/sandbox-runtime` from using `bubblewrap` for sandbox isolation.
>
> To temporarily allow user namespaces:
>
> ```bash
> sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
> ```
>
> Alternatively, configure an AppArmor profile that grants the required `userns` permission to the relevant binaries.
---
#### macOS
Shumai requires:
- **`ffmpeg`** – Used for media transcoding and metadata extraction.
- **`poppler`** – Used for PDF page image extraction and PDF sprite preview generation (`pdftoppm`).
- **`imagemagick`** – Used for PSD format transcoding and color profile conversion to sRGB.
- **`dcraw_emu`** (`libraw`) – Used for camera RAW format image decoding and preview generation.
- **`ripgrep`** – Required by the AI agent sandbox (`anthropic-experimental/sandbox-runtime`).
Install the required packages with Homebrew:
```bash
brew install ffmpeg poppler imagemagick ripgrep libraw
```
---
#### Windows (alpha)
> [!WARNING]
> Windows support in anthropic-experimental/sandbox-runtime is **alpha and the design is in flux**.
>
> The current implementation provides:
> - Network egress filtering via the **Windows Filtering Platform (WFP)**.
> - File read/write denial via **ACL stamping**.
>
> However, it is **not a security boundary against a deliberately adversarial sandboxed process**.
>
> The sandbox implementation will be substantially revised in a future release, where sandboxed processes will run under a dedicated sandbox user account.
Shumai requires:
- **`ffmpeg`** – Used for media transcoding and metadata extraction.
- **`imagemagick`** – Used for PSD format transcoding and color profile conversion to sRGB.
Install required packages using your preferred package manager:
**winget**
```powershell
winget install Gyan.FFmpeg ImageMagick.ImageMagick
```
**Chocolatey**
```powershell
choco install ffmpeg imagemagick
```
#### Step 4: Install Shumai globally
Install Shumai globally using your preferred package manager:
```bash
# NPM
npm install -g @shumai-one/shumai
# PNPM
pnpm add -g @shumai-one/shumai
# Bun
bun add -g @shumai-one/shumai
```
#### Step 5: Configure Environment Variables
Create a `.env` file in your workspace folder (`shumai/`) and add the following configuration:
```env
DATABASE_URL=postgresql://shumai:shumai_password@localhost:5432/shumai_db?schema=public
BETTER_AUTH_SECRET=ySxs7DxzHDZBbeeHNPEwBuspYwipBqz5Gk5XdBjNhWw=
STORAGE_BACKEND=local
SHUMAI_SERVER_PORT=3000
AWS_ENDPOINT_URL_S3=http://localhost:3000
# Optional: base URL for MCP OAuth redirects (defaults to BETTER_AUTH_URL)
MCP_OAUTH_REDIRECT_BASE_URL=http://localhost:3000
```
> [!NOTE]
> `SHUMAI_SERVER_PORT` sets the port the server starts on, while `AWS_ENDPOINT_URL_S3` is used by the browser to build file upload URLs.
> If deploying on a remote server (e.g. `http://123.456.7.8`) with a mapped port (e.g. `12345:3000` in docker-compose), set `AWS_ENDPOINT_URL_S3` to `http://123.456.7.8:12345`.
> `MCP_OAUTH_REDIRECT_BASE_URL` is only needed when MCP servers use OAuth: the authorization redirect URI defaults to `{BETTER_AUTH_URL}/api/mcp/oauth/callback`; set it to an externally reachable URL if the app is served behind a proxy.
#### Step 6: Run Shumai
> [!IMPORTANT]
> **For Bun Users:** Since the published package binaries contain a `#!/usr/bin/env node` shebang, running the global command directly (e.g., `shumai`) will execute under Node.js. If you installed with Bun and want to run under the Bun runtime, you must prefix all commands with `bun run --bun` (e.g., `bun run --bun shumai`, `bun run --bun shumai -d`, `bun run --bun shumai stop`).
Start the application from your workspace folder:
```bash
shumai
```
Alternatively, you can run and manage Shumai in daemon mode:
- **Start in daemon mode**:
```bash
shumai -d
```
- **Stop Shumai**:
```bash
shumai stop
```
- **Restart Shumai**:
```bash
shumai restart
```
- **Show/tail logs**:
```bash
shumai logs
```
On startup, Shumai will automatically run database migrations and start the web server at `http://localhost:3000`.
---
### Option 3: Run from Source (Development)
### Option 2: Run from Source (Development)
To set up Shumai locally for development:
@@ -322,8 +151,23 @@ To set up Shumai locally for development:
cd shumai
bun install
```
2. Start the `pgvector` database container (as described in **Option 2, Step 1**).
3. Create a `.env` file at the root of the workspace using the configuration from **Option 2, Step 5**.
2. Start the `pgvector` database container:
```bash
docker run --name shumai_postgres \
-e POSTGRES_USER=shumai \
-e POSTGRES_PASSWORD=shumai_password \
-e POSTGRES_DB=shumai_db \
-p 5432:5432 \
-d pgvector/pgvector:pg18
```
3. Create a `.env` file at the root of the workspace:
```env
DATABASE_URL=postgresql://shumai:shumai_password@localhost:5432/shumai_db?schema=public
BETTER_AUTH_SECRET=ySxs7DxzHDZBbeeHNPEwBuspYwipBqz5Gk5XdBjNhWw=
STORAGE_BACKEND=local
SHUMAI_SERVER_PORT=3000
AWS_ENDPOINT_URL_S3=http://localhost:3000
```
4. Generate i18n translation files:
```bash
bun run i18n:compile
@@ -339,6 +183,9 @@ To set up Shumai locally for development:
bun run dev
```
> [!NOTE]
> Looking to install Shumai globally via NPM or your package manager? See our [NPM Installation Guide](https://docs.shumai.one/getting-started/quickstart).
---
## Command Line Interface (CLI)
+55 -221
View File
@@ -35,6 +35,7 @@ Shumai 是一个开源的 Frame.io 替代方案,专为创意团队打造,让
* **丰富的文件格式支持**:支持预览常见创意文件格式,包括图片(支持 PSD)、视频、音频和文档,同时可上传和存储任意类型的文件。
* **基于转盘渲染服务的 3D 资产预览**:通过集成 [turntable-renderer](https://docs.shumai.one/using-shumai/3d-asset-support) 支持为 GLB、FBX、OBJ、STL、USD 和 DAE 等 3D 模型生成高质量的交互式转盘预览与视频。
* **基于 Gotenberg 的文档 PDF 转码代理**:通过集成 [Gotenberg](https://docs.shumai.one/using-shumai/gotenberg) 支持将 Word、PowerPoint、Excel、Markdown、HTML 和 CSV 文档高质量转换为 PDF 代理文件进行在线预览。
* **语义搜索**:支持按画面内容或概念含义搜索素材,底层基于向量嵌入技术。详见[相关文档](https://docs.shumai.one/using-shumai/semantic-search)。
* **非编剪辑导出与 Premiere 插件(实验性)**:支持将时间戳审阅评论导出至常用非编软件(Premiere Pro XML、DaVinci Resolve EDL、Final Cut Pro、Avid Media Composer),或使用实验性的 Premiere Pro 原生 UXP 插件导入媒体素材并将评论自动同步为时间轴标记。详见[相关文档](https://docs.shumai.one/using-shumai/nle-and-premiere)。
但 Shumai 不仅仅只满足于 Frame.io 的开源替代。随着 GPT-Image-2、Seedance 2.5 这类强大模型的问世,AI已经生成产品级的图片/视频。很多时候,你只要给一句提示词,就能直接生成或修改图片和视频,而不用再打开 Photoshop、Premiere 或者 Houdini。也正因为这样,Shumai 才把 AI Agent 做成了核心能力:
@@ -45,7 +46,6 @@ Shumai 是一个开源的 Frame.io 替代方案,专为创意团队打造,让
* **Model Context Protocol (MCP) 支持**:通过远程 MCP 服务端为 Agent 接入外部工具与 API,支持 OAuth 鉴权与按 Agent 独立分配访问权限。
* **沙箱隔离执行**:Agent 生成的脚本会在隔离环境里运行,更安全。
* **AI 自动补全元数据**:新素材上传后,系统可以自动帮你补一些元数据。
* **语义搜索**:支持按画面内容或语义找素材,底层用的是向量 embedding(Gemini Embedding 2)。
* **资源配额**:支持按成员、角色或全团队对 AI Token、成本和工具调用设置使用上限与实时用量监控。
下面是一个简短的 demo 视频,展示了 Shumai Agent 能做什么:
@@ -99,10 +99,22 @@ https://gh-proxy.com/https://raw.githubusercontent.com/shumaiOne/shumai/main/REA
```
2. 下载 `docker-compose.yaml` 配置文件:
```bash
curl -o docker-compose.yaml https://raw.githubusercontent.com/shumaiOne/shumai/main/docker-compose/local/docker-compose-aliyun.yaml
```
2. 根据您的需求和机器可用内存,下载对应的 `docker-compose.yaml` 配置文件:
* **精简版(建议内存:2 GB+)**:包含 Shumai 核心应用与 PostgreSQL 数据库。
```bash
curl -o docker-compose.yaml https://raw.githubusercontent.com/shumaiOne/shumai/main/docker-compose/local/docker-compose-aliyun.yaml
```
* **含嵌入向量服务版(建议内存:4 GB+)**:新增基于 `shumaiOne/embedding-server` (`EmbeddingGemma 2 740M`) 的本地多模态语义搜索能力。
```bash
curl -o docker-compose.yaml https://raw.githubusercontent.com/shumaiOne/shumai/main/docker-compose/local/docker-compose-aliyun.embedding.yaml
```
* **完整版(建议内存:8 GB+)**:一站式全功能套件,包含 `embedding-server`、`turntable-renderer`(3D 模型交互转盘预览)与 `Gotenberg`(文档与表格 PDF 代理转码)。
```bash
curl -o docker-compose.yaml https://raw.githubusercontent.com/shumaiOne/shumai/main/docker-compose/local/docker-compose-aliyun.full.yaml
```
> [!NOTE]
> 默认的 Docker Compose 配置使用基于 CPU 的软件转码。如果您的服务器配备了支持的显卡(NVIDIA NVENC、Intel QuickSync、AMD/Intel VA-API 或 Rockchip RKMPP)并需要开启 GPU 硬件加速转码,请参考 [硬件加速转码配置文档](https://docs.shumai.one/configuration/hardware-transcoding)。
@@ -152,233 +164,55 @@ https://gh-proxy.com/https://raw.githubusercontent.com/shumaiOne/shumai/main/REA
5. 打开浏览器,访问 `http://localhost:3000` 即可开始使用(若是远程部署,请访问 `http://<您的服务器IP>:3000`)。
### 方案二:通过 NPM / 包管理器安装
通过 NPM 上的 `@shumai-one/shumai` 进行安装。
#### 第一步:启动带 pgvector 支持的 PostgreSQL
Shumai 依赖支持 `pgvector` 扩展的 PostgreSQL 数据库。您可以使用 Docker 快速启动一个预配置好的数据库容器:
```bash
docker run --name shumai_postgres \
-e POSTGRES_USER=shumai \
-e POSTGRES_PASSWORD=shumai_password \
-e POSTGRES_DB=shumai_db \
-p 5432:5432 \
-d docker.m.daocloud.io/pgvector/pgvector:pg18
```
#### 第二步:创建工作区目录
创建一个专属目录用来存放环境配置和媒体文件(媒体文件默认会保存在 `./data` 目录下):
```bash
mkdir shumai && cd shumai
```
#### 步骤 3:安装平台依赖
#### Linux
在 Linux 上运行 Shumai 需要安装以下系统依赖:
- **`ffmpeg`** —— 用于媒体转码和元数据提取。
- **`poppler`** (`poppler-utils`) —— 用于 PDF 页面图像提取和 PDF 雪碧图预览生成 (`pdftoppm`)。
- **`imagemagick`** —— 用于 PSD 格式图像转码和 sRGB 色彩空间转换。
- **`dcraw_emu`** (`libraw` / `libraw-bin`) —— 用于相机 RAW 格式图像解码和预览图生成。
- **`bubblewrap`**、**`socat`** 和 **`ripgrep`** —— AI Agent 沙箱(`anthropic-experimental/sandbox-runtime`)所需,用于进程隔离、网络通信和工作区搜索。
可以使用以下命令一次性安装所有依赖:
> Ubuntu/Debian
```bash
sudo apt install -y ffmpeg poppler-utils imagemagick bubblewrap socat ripgrep libraw-bin
```
> Fedora
```bash
sudo dnf install -y ffmpeg poppler-utils ImageMagick bubblewrap socat ripgrep LibRaw
```
> Arch Linux
```bash
sudo pacman -S --noconfirm ffmpeg poppler imagemagick bubblewrap socat ripgrep libraw
```
> [!NOTE]
> **Ubuntu 24.04+** 默认限制非特权用户创建 User Namespace,这会导致 `anthropic-experimental/sandbox-runtime` 无法使用 `bubblewrap` 提供沙箱隔离。
>
> 如需临时启用 User Namespace,请执行:
>
> ```bash
> sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
> ```
>
> 或者,配置 AppArmor Profile,为相关可执行文件授予所需的 `userns`权限。
---
#### macOS
在 macOS 上运行 Shumai 需要:
- **`ffmpeg`** —— 用于媒体转码和元数据提取。
- **`poppler`** —— 用于 PDF 页面图像提取和 PDF 雪碧图预览生成 (`pdftoppm`)。
- **`imagemagick`** —— 用于 PSD 格式图像转码和 sRGB 色彩空间转换。
- **`dcraw_emu`** (`libraw`) —— 用于相机 RAW 格式图像解码和预览图生成。
- **`ripgrep`** —— AI Agent 沙箱(`anthropic-experimental/sandbox-runtime`)所需。
使用 Homebrew 安装所需依赖:
```bash
brew install ffmpeg poppler imagemagick ripgrep libraw
```
---
#### Windows(Alpha)
> [!WARNING]
> anthropic-experimental/sandbox-runtime 的 Windows 支持目前仍处于 **Alpha** 阶段,整体设计仍在持续调整中。
>
> 当前实现提供了以下能力:
> - 通过 **Windows Filtering Platform (WFP)** 实现网络出口(egress)过滤。
> - 通过 **ACL Stamping** 实现文件读写权限限制。
>
> **但这并不能作为针对恶意沙箱进程的安全边界(security boundary)**。
>
> 未来版本将对沙箱机制进行重大调整,届时沙箱进程将运行在独立的沙箱用户账户下。
在 Windows 上运行 Shumai 需要:
- **`ffmpeg`** —— 用于媒体转码和元数据提取。
- **`imagemagick`** —— 用于 PSD 格式图像转码和 sRGB 色彩空间转换。
使用你偏好的包管理器安装所需依赖:
**winget**
```powershell
winget install Gyan.FFmpeg ImageMagick.ImageMagick
```
**Chocolatey**
```powershell
choco install ffmpeg imagemagick
```
#### 第四步:全局安装 Shumai
使用您常用的包管理器进行全局安装:
```bash
# 使用 NPM
npm install -g @shumai-one/shumai
# 使用 PNPM
pnpm add -g @shumai-one/shumai
# 使用 Bun
bun add -g @shumai-one/shumai
```
#### 第五步:配置环境配置
在刚才创建的工作区目录(`shumai/`)下新建一个 `.env` 文件,并填入以下配置:
```env
DATABASE_URL=postgresql://shumai:shumai_password@localhost:5432/shumai_db?schema=public
BETTER_AUTH_SECRET=ySxs7DxzHDZBbeeHNPEwBuspYwipBqz5Gk5XdBjNhWw=
STORAGE_BACKEND=local
SHUMAI_SERVER_PORT=3000
AWS_ENDPOINT_URL_S3=http://localhost:3000
```
> [!NOTE]
> `SHUMAI_SERVER_PORT` 用于设置 Web 服务的启动端口,而 `AWS_ENDPOINT_URL_S3` 供浏览器端构建文件上传链接。
> 如果您部署在远程服务器(如 `http://123.456.7.8`)并且做了端口映射(如 Docker 中的 `12345:3000`),请将 `AWS_ENDPOINT_URL_S3` 设置为 `http://123.456.7.8:12345`。
#### 第六步:启动 Shumai
> [!IMPORTANT]
> **Bun 用户特别提示:** 由于发布的包二进制文件包含 `#!/usr/bin/env node` 头部声明(Shebang),直接在终端执行全局命令(例如 `shumai`)默认会通过 Node.js 运行。如果您是用 Bun 安装的并且希望调用 Bun 运行时,请在所有命令前加上 `bun run --bun` 前缀(例如:`bun run --bun shumai`、`bun run --bun shumai -d`、`bun run --bun shumai stop`)。
在您的工作区目录下执行以下命令启动应用:
```bash
shumai
```
您也可以让 Shumai 在后台以守护进程(Daemon)模式运行和管理:
* **后台模式启动**:
```bash
shumai -d
```
* **停止服务**:
```bash
shumai stop
```
* **重启服务**:
```bash
shumai restart
```
* **查看并跟踪实时日志**:
```bash
shumai logs
```
启动时,Shumai 会自动执行数据库迁移(Migration)并在 `http://localhost:3000` 拉起 Web 服务。
---
### 方案三:源码编译运行(开发环境)
### 方案二:源码编译运行(开发环境)
如果您希望在本地参与 Shumai 的开发,请按以下步骤配置开发环境:
1. 克隆代码仓库并安装依赖项:
```bash
git clone [https://github.com/shumaiOne/shumai.git](https://github.com/shumaiOne/shumai.git)
cd shumai
bun install
```bash
git clone https://github.com/shumaiOne/shumai.git
cd shumai
bun install
```
```
2. 启动 `pgvector` 数据库容器:
```bash
docker run --name shumai_postgres \
-e POSTGRES_USER=shumai \
-e POSTGRES_PASSWORD=shumai_password \
-e POSTGRES_DB=shumai_db \
-p 5432:5432 \
-d docker.m.daocloud.io/pgvector/pgvector:pg18
```
3. 在项目根目录下创建一个 `.env` 文件,填入以下基础配置:
```env
DATABASE_URL=postgresql://shumai:shumai_password@localhost:5432/shumai_db?schema=public
BETTER_AUTH_SECRET=ySxs7DxzHDZBbeeHNPEwBuspYwipBqz5Gk5XdBjNhWw=
STORAGE_BACKEND=local
SHUMAI_SERVER_PORT=3000
AWS_ENDPOINT_URL_S3=http://localhost:3000
```
2. 启动 `pgvector` 数据库容器(参考**方案二中的第一步**)。
3. 参照**方案二中的第五步**,在项目根目录下创建一个 `.env` 文件。
4. 生成多语言国际化(i18n)代码:
```bash
bun run i18n:compile
```
> [!TIP]
> 在前端开发过程中,可以在单独的终端窗口中运行 `bun run i18n:watch`,在修改 `packages/webui/messages/` 下的翻译文案时自动重新编译。
```bash
bun run i18n:compile
```
> [!TIP]
> 在前端开发过程中,可以在单独的终端窗口中运行 `bun run i18n:watch`,在修改 `packages/webui/messages/` 下的翻译文案时自动重新编译。
5. 执行数据库结构迁移:
```bash
bun run db:migrate
```
```bash
bun run db:migrate
```
6. 启动本地开发服务器:
```bash
bun run dev
```
```bash
bun run dev
```
> [!NOTE]
> 如需通过 NPM 或包管理器全局安装运行 Shumai,请参考[官方 NPM 安装指南](https://docs.shumai.one/getting-started/quickstart)。
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/agent-app",
"version": "0.5.2",
"version": "0.5.3",
"private": true,
"type": "module",
"bin": {
-1
View File
@@ -1,4 +1,3 @@
export * from '@shumai/agent/src/workflows/agent-autofill'
export * from '@shumai/agent/src/workflows/agent-chat'
export * from '@shumai/agent/src/workflows/agent-embedding'
export * from '@shumai/agent/src/workflows/query-embedding-for-search'
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/cli",
"version": "0.5.2",
"version": "0.5.3",
"private": true,
"type": "module",
"bin": {
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/premiere-uxp",
"version": "0.5.2",
"version": "0.5.3",
"private": true,
"type": "commonjs",
"scripts": {
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/transcode-app",
"version": "0.5.2",
"version": "0.5.3",
"private": true,
"type": "module",
"bin": {
+1
View File
@@ -19,6 +19,7 @@ async function start() {
process.env.BETTER_AUTH_URL = 'http://localhost:5200'
process.env.NODE_ENV = 'development'
process.env.SHUMAI_E2E = 'true'
process.env.ENABLE_EMBEDDING_AGENT = 'false'
// Write variables to .env.e2e file so the teardown can load them
const envContent = [`DATABASE_URL=${process.env.DATABASE_URL}`, `S3_BUCKET=${bucketName}`].join(
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/web-app",
"version": "0.5.2",
"version": "0.5.3",
"private": true,
"type": "module",
"dependencies": {
+26
View File
@@ -12,9 +12,11 @@ import { metadataService } from '@shumai/core/src/metadata/metadata'
import { initTranscodeWorkflows } from '@shumai/transcode'
import { workflowService } from '@shumai/workflow-core'
import { migrateLegacyAgentAvatars } from '@shumai/core/src/agent/migration'
import { agentService } from '@shumai/core/src/agent/agent'
import { handleDaemonCommands } from '@shumai/core/src/utils/daemon'
import { authService } from '@shumai/core/src/auth/auth'
import { embeddingService } from '@shumai/core/src/search/embedding-service'
import { sandboxService } from '@shumai/core'
import { notificationJobService } from '@shumai/core/src/notification/notification-job'
@@ -47,6 +49,29 @@ if (resetCmdIndex !== -1) {
}
}
const regenCmdIndex = cliArgs.findIndex((arg) => arg === 'regenerate-embeddings')
if (regenCmdIndex !== -1) {
try {
console.log('🔄 Regenerating embeddings...')
const result = await embeddingService.regenerateEmbeddings()
console.log(
`✅ Successfully queued ${result.tasksCreated} embedding tasks for ${result.totalAssets} assets.`,
)
if (result.skippedTeams.length > 0) {
console.warn(
`⚠️ Skipped ${result.skippedTeams.length} team(s) without an active embedding agent.`,
)
}
console.log('The background workers will process the embedding tasks.')
process.exit(0)
} catch (err: unknown) {
const message = err instanceof Error ? err.message : 'Unknown error'
console.error(`Error: ${message}`)
process.exit(1)
}
}
async function run() {
// Initialize workflows and activities for local executor mode
initAgentWorkflows()
@@ -58,6 +83,7 @@ async function run() {
// Start services
await metadataService.syncSystemFields().catch(console.error)
await migrateLegacyAgentAvatars().catch(console.error)
await agentService.ensureSystemEmbeddingAgents().catch(console.error)
assetService.startCleanupJob()
notificationJobService.start()
workflowService.start()
+20 -18
View File
@@ -33,7 +33,7 @@
},
"apps/agent": {
"name": "@shumai/agent-app",
"version": "0.5.2",
"version": "0.5.3",
"bin": {
"shumai-agent": "./src/index.ts",
},
@@ -45,7 +45,7 @@
},
"apps/cli": {
"name": "@shumai/cli",
"version": "0.5.2",
"version": "0.5.3",
"bin": {
"shumai-cli": "./src/index.ts",
},
@@ -60,7 +60,7 @@
},
"apps/premiere-uxp": {
"name": "@shumai/premiere-uxp",
"version": "0.5.2",
"version": "0.5.3",
"dependencies": {
"@shumai/api": "workspace:*",
"@shumai/dtos": "workspace:*",
@@ -114,7 +114,7 @@
},
"apps/transcode": {
"name": "@shumai/transcode-app",
"version": "0.5.2",
"version": "0.5.3",
"bin": {
"shumai-transcode": "./src/index.ts",
},
@@ -126,7 +126,7 @@
},
"apps/web": {
"name": "@shumai/web-app",
"version": "0.5.2",
"version": "0.5.3",
"dependencies": {
"@shumai/agent": "workspace:*",
"@shumai/api": "workspace:*",
@@ -146,12 +146,12 @@
},
"packages/agent": {
"name": "@shumai/agent",
"version": "0.5.2",
"version": "0.5.3",
"dependencies": {
"@anthropic-ai/sandbox-runtime": "0.0.74",
"@earendil-works/pi-agent-core": "0.82.1",
"@earendil-works/pi-ai": "0.82.1",
"@google/genai": "2.19.0",
"@google/genai": "2.28.0",
"@shumai/core": "workspace:*",
"@shumai/db": "workspace:*",
"@shumai/dtos": "workspace:*",
@@ -161,6 +161,7 @@
"@temporalio/workflow": "1.23.0",
"adm-zip": "0.6.0",
"jittered-fractional-indexing": "1.0.1",
"sharp": "0.35.4",
"typebox": "1.3.22",
"ulid": "3.0.2",
"zod": "4.5.4",
@@ -171,7 +172,7 @@
},
"packages/api": {
"name": "@shumai/api",
"version": "0.5.2",
"version": "0.5.3",
"dependencies": {
"@hono/zod-validator": "0.7.6",
"@shumai/core": "workspace:*",
@@ -185,7 +186,7 @@
},
"packages/core": {
"name": "@shumai/core",
"version": "0.5.2",
"version": "0.5.3",
"dependencies": {
"@ai-sdk/alibaba": "2.0.38",
"@ai-sdk/amazon-bedrock": "5.0.67",
@@ -210,6 +211,7 @@
"@aws-sdk/s3-request-presigner": "3.1121.0",
"@earendil-works/pi-agent-core": "0.82.1",
"@earendil-works/pi-ai": "0.82.1",
"@google/genai": "2.28.0",
"@modelcontextprotocol/client": "2.0.0",
"@shumai/db": "workspace:*",
"@shumai/dtos": "workspace:*",
@@ -247,7 +249,7 @@
},
"packages/db": {
"name": "@shumai/db",
"version": "0.5.2",
"version": "0.5.3",
"dependencies": {
"@earendil-works/pi-agent-core": "0.82.1",
"@prisma/adapter-pg": "7.10.0",
@@ -263,7 +265,7 @@
},
"packages/dtos": {
"name": "@shumai/dtos",
"version": "0.5.2",
"version": "0.5.3",
"dependencies": {
"@earendil-works/pi-agent-core": "0.82.1",
"zod": "4.5.4",
@@ -271,10 +273,10 @@
},
"packages/e2e": {
"name": "@shumai/e2e",
"version": "0.5.2",
"version": "0.5.3",
"dependencies": {
"@earendil-works/pi-agent-core": "0.82.1",
"@google/genai": "2.19.0",
"@google/genai": "2.28.0",
"@shumai/agent": "workspace:*",
"@shumai/core": "workspace:*",
"@shumai/db": "workspace:*",
@@ -286,11 +288,11 @@
},
"packages/tableprinter": {
"name": "@shumai/tableprinter",
"version": "0.5.2",
"version": "0.5.3",
},
"packages/transcode": {
"name": "@shumai/transcode",
"version": "0.5.2",
"version": "0.5.3",
"dependencies": {
"@shumai/core": "workspace:*",
"@shumai/db": "workspace:*",
@@ -304,7 +306,7 @@
},
"packages/webui": {
"name": "@shumai/webui",
"version": "0.5.2",
"version": "0.5.3",
"dependencies": {
"@base-ui/react": "1.7.0",
"@dnd-kit/abstract": "0.5.0",
@@ -406,7 +408,7 @@
},
"packages/workflow-core": {
"name": "@shumai/workflow-core",
"version": "0.5.2",
"version": "0.5.3",
"dependencies": {
"@shumai/core": "workspace:*",
"@shumai/db": "workspace:*",
@@ -652,7 +654,7 @@
"@floating-ui/utils": ["@floating-ui/utils@0.2.12", "", {}, "sha512-HpCo8tmWzLVad5s2d19EhAz5zqrrQ6s69qd6moPMQvkOuSwDT1YgRfWSVuc4ennqrgv3OHppiOGMQ7oC13yIww=="],
"@google/genai": ["@google/genai@2.19.0", "", { "dependencies": { "google-auth-library": "^10.3.0", "p-retry": "^4.6.2", "protobufjs": "^7.5.4", "ws": "^8.18.0" }, "peerDependencies": { "@modelcontextprotocol/sdk": "^1.25.2" }, "optionalPeers": ["@modelcontextprotocol/sdk"] }, "sha512-IM3u1FjpOLyOPsPh0smGA6Ntj2olUWCPbTs/WQj7cA/2AKCGw/652W7IaySx0XtwuscFnMYBoraebQusSZ0zSQ=="],
"@google/genai": ["@google/genai@2.28.0", "", { "dependencies": { "google-auth-library": "^10.3.0", "p-retry": "^4.6.2", "protobufjs": "^7.5.4", "ws": "^8.18.0" }, "peerDependencies": { "@modelcontextprotocol/sdk": "^1.25.2" }, "optionalPeers": ["@modelcontextprotocol/sdk"] }, "sha512-hyNAJj+ywu3L46qlZod6KO/JvcT3QONz6VCiPevC/FWOjAD+LQb7DMLUoV8vzoLpt61Q0ruQX99iqbxEjPJWmw=="],
"@grpc/grpc-js": ["@grpc/grpc-js@1.14.4", "", { "dependencies": { "@grpc/proto-loader": "^0.8.0", "@js-sdsl/ordered-map": "^4.4.2" } }, "sha512-k9Dj3DV/itK9D06Y8f190Qgop7/Ui+D0njFV3LHMPwPT75DpXLQohE9Wmz0QElrJnzsjB7KPWiKJbOl7IPDArQ=="],
@@ -0,0 +1,69 @@
services:
postgres:
image: docker.m.daocloud.io/pgvector/pgvector:pg18
container_name: shumai_postgres
environment:
POSTGRES_USER: shumai
POSTGRES_PASSWORD: shumai_password
POSTGRES_DB: shumai_db
volumes:
- postgres_data:/var/lib/postgresql/18/docker
healthcheck:
test: ['CMD-SHELL', 'pg_isready -U shumai -d shumai_db']
interval: 5s
timeout: 5s
retries: 5
embedding-server:
image: shumaione/embedding-server:latest
container_name: shumai_embedding
restart: unless-stopped
shumai:
image: crpi-000nkafzu6xvpwsc.cn-hongkong.personal.cr.aliyuncs.com/shumai-one/shumai:latest
cap_add:
- SYS_ADMIN
security_opt:
- apparmor=unconfined
- seccomp=unconfined
container_name: shumai_app
ports:
- '3000:3000'
volumes:
- ./data:/app/data
# Hardware Acceleration (NVENC / QuickSync / VA-API / RKMPP)
# Uncomment the extends section below for hardware acceleration:
# extends:
# file: hwaccel.transcoding.yaml
# service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
environment:
# Database Configuration
DATABASE_URL: postgres://shumai:shumai_password@shumai_postgres:5432/shumai_db?search_path=public&sslmode=disable
# Server Configuration
SHUMAI_SERVER_PORT: 3000
# S3 Configuration
STORAGE_BACKEND: local
AWS_ENDPOINT_URL_S3: http://localhost:3000
# Better Auth
BETTER_AUTH_SECRET: qSxs9DxzHDZBbeeTNPEwBuspYwipBqz5Gk5XdBjNhWw=
# Sandbox Configuration
ENABLE_WEAKER_NESTED_SANDBOX: 'true'
# Embedding Configuration
EMBEDDING_PROVIDER: gemma-embedding-2-openai
GEMMA_EMBEDDING_OPENAI_BASE_URL: http://embedding-server:9379/v1
ENABLE_EMBEDDING_AGENT: 'true'
CONCURRENCY_AGENT: 1
depends_on:
postgres:
condition: service_healthy
embedding-server:
condition: service_started
volumes:
postgres_data:
@@ -0,0 +1,89 @@
services:
postgres:
image: docker.m.daocloud.io/pgvector/pgvector:pg18
container_name: shumai_postgres
environment:
POSTGRES_USER: shumai
POSTGRES_PASSWORD: shumai_password
POSTGRES_DB: shumai_db
volumes:
- postgres_data:/var/lib/postgresql/18/docker
healthcheck:
test: ['CMD-SHELL', 'pg_isready -U shumai -d shumai_db']
interval: 5s
timeout: 5s
retries: 5
embedding-server:
image: shumaione/embedding-server:latest
container_name: shumai_embedding
restart: unless-stopped
turntable-renderer:
image: shumaione/turntable-renderer:latest
container_name: shumai_turntable
restart: unless-stopped
gotenberg:
image: gotenberg/gotenberg:8
container_name: shumai_gotenberg
restart: unless-stopped
shumai:
image: crpi-000nkafzu6xvpwsc.cn-hongkong.personal.cr.aliyuncs.com/shumai-one/shumai:latest
cap_add:
- SYS_ADMIN
security_opt:
- apparmor=unconfined
- seccomp=unconfined
container_name: shumai_app
ports:
- '3000:3000'
volumes:
- ./data:/app/data
# Hardware Acceleration (NVENC / QuickSync / VA-API / RKMPP)
# Uncomment the extends section below for hardware acceleration:
# extends:
# file: hwaccel.transcoding.yaml
# service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
environment:
# Database Configuration
DATABASE_URL: postgres://shumai:shumai_password@shumai_postgres:5432/shumai_db?search_path=public&sslmode=disable
# Server Configuration
SHUMAI_SERVER_PORT: 3000
# S3 Configuration
STORAGE_BACKEND: local
AWS_ENDPOINT_URL_S3: http://localhost:3000
# Better Auth
BETTER_AUTH_SECRET: qSxs9DxzHDZBbeeTNPEwBuspYwipBqz5Gk5XdBjNhWw=
# Sandbox Configuration
ENABLE_WEAKER_NESTED_SANDBOX: 'true'
# Embedding Configuration
EMBEDDING_PROVIDER: gemma-embedding-2-openai
GEMMA_EMBEDDING_OPENAI_BASE_URL: http://embedding-server:9379/v1
ENABLE_EMBEDDING_AGENT: 'true'
CONCURRENCY_AGENT: 1
# 3D Turntable Renderer Configuration
TURNTABLE_RENDERER_URL: http://turntable-renderer:3000
# Gotenberg Document Converter Configuration
GOTENBERG_URL: http://gotenberg:3000
depends_on:
postgres:
condition: service_healthy
embedding-server:
condition: service_started
turntable-renderer:
condition: service_started
gotenberg:
condition: service_started
volumes:
postgres_data:
@@ -0,0 +1,69 @@
services:
postgres:
image: pgvector/pgvector:pg18
container_name: shumai_postgres
environment:
POSTGRES_USER: shumai
POSTGRES_PASSWORD: shumai_password
POSTGRES_DB: shumai_db
volumes:
- postgres_data:/var/lib/postgresql/18/docker
healthcheck:
test: ['CMD-SHELL', 'pg_isready -U shumai -d shumai_db']
interval: 5s
timeout: 5s
retries: 5
embedding-server:
image: shumaione/embedding-server:latest
container_name: shumai_embedding
restart: unless-stopped
shumai:
image: shumaione/shumai:latest
cap_add:
- SYS_ADMIN
security_opt:
- apparmor=unconfined
- seccomp=unconfined
container_name: shumai_app
ports:
- '3000:3000'
volumes:
- ./data:/app/data
# Hardware Acceleration (NVENC / QuickSync / VA-API / RKMPP)
# Uncomment the extends section below for hardware acceleration:
# extends:
# file: hwaccel.transcoding.yaml
# service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
environment:
# Database Configuration
DATABASE_URL: postgres://shumai:shumai_password@shumai_postgres:5432/shumai_db?search_path=public&sslmode=disable
# Server Configuration
SHUMAI_SERVER_PORT: 3000
# S3 Configuration
STORAGE_BACKEND: local
AWS_ENDPOINT_URL_S3: http://localhost:3000
# Better Auth
BETTER_AUTH_SECRET: qSxs9DxzHDZBbeeTNPEwBuspYwipBqz5Gk5XdBjNhWw=
# Sandbox Configuration
ENABLE_WEAKER_NESTED_SANDBOX: 'true'
# Embedding Configuration
EMBEDDING_PROVIDER: gemma-embedding-2-openai
GEMMA_EMBEDDING_OPENAI_BASE_URL: http://embedding-server:9379/v1
ENABLE_EMBEDDING_AGENT: 'true'
CONCURRENCY_AGENT: 1
depends_on:
postgres:
condition: service_healthy
embedding-server:
condition: service_started
volumes:
postgres_data:
@@ -0,0 +1,89 @@
services:
postgres:
image: pgvector/pgvector:pg18
container_name: shumai_postgres
environment:
POSTGRES_USER: shumai
POSTGRES_PASSWORD: shumai_password
POSTGRES_DB: shumai_db
volumes:
- postgres_data:/var/lib/postgresql/18/docker
healthcheck:
test: ['CMD-SHELL', 'pg_isready -U shumai -d shumai_db']
interval: 5s
timeout: 5s
retries: 5
embedding-server:
image: shumaione/embedding-server:latest
container_name: shumai_embedding
restart: unless-stopped
turntable-renderer:
image: shumaione/turntable-renderer:latest
container_name: shumai_turntable
restart: unless-stopped
gotenberg:
image: gotenberg/gotenberg:8
container_name: shumai_gotenberg
restart: unless-stopped
shumai:
image: shumaione/shumai:latest
cap_add:
- SYS_ADMIN
security_opt:
- apparmor=unconfined
- seccomp=unconfined
container_name: shumai_app
ports:
- '3000:3000'
volumes:
- ./data:/app/data
# Hardware Acceleration (NVENC / QuickSync / VA-API / RKMPP)
# Uncomment the extends section below for hardware acceleration:
# extends:
# file: hwaccel.transcoding.yaml
# service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
environment:
# Database Configuration
DATABASE_URL: postgres://shumai:shumai_password@shumai_postgres:5432/shumai_db?search_path=public&sslmode=disable
# Server Configuration
SHUMAI_SERVER_PORT: 3000
# S3 Configuration
STORAGE_BACKEND: local
AWS_ENDPOINT_URL_S3: http://localhost:3000
# Better Auth
BETTER_AUTH_SECRET: qSxs9DxzHDZBbeeTNPEwBuspYwipBqz5Gk5XdBjNhWw=
# Sandbox Configuration
ENABLE_WEAKER_NESTED_SANDBOX: 'true'
# Embedding Configuration
EMBEDDING_PROVIDER: gemma-embedding-2-openai
GEMMA_EMBEDDING_OPENAI_BASE_URL: http://embedding-server:9379/v1
ENABLE_EMBEDDING_AGENT: 'true'
CONCURRENCY_AGENT: 1
# 3D Turntable Renderer Configuration
TURNTABLE_RENDERER_URL: http://turntable-renderer:3000
# Gotenberg Document Converter Configuration
GOTENBERG_URL: http://gotenberg:3000
depends_on:
postgres:
condition: service_healthy
embedding-server:
condition: service_started
turntable-renderer:
condition: service_started
gotenberg:
condition: service_started
volumes:
postgres_data:
+18
View File
@@ -3,6 +3,24 @@ title: "Changelog"
description: "Recent updates, new features, and improvements in Shumai"
---
<Update label="v0.5.3" description="October 16, 2026">
### Added
* **cli**: Add administrative `shumai regenerate-embeddings` command to regenerate multimodal vector embeddings for all existing media files across active teams
* **docker**: Provide three Docker Compose deployment tiers (Lite, with Embedding Server, and Full) with recommended memory specifications
* **agent**: Save video chunk embeddings incrementally to the database as each chunk finishes, preserving completed progress on failure and resuming from where it left off on retry
* **agent**: Add support for local LiteRT-LM Gemma Embedding 2 (`gemma-embedding-2-openai`) provider and environment-enforced read-only embedding agents with automatic 768-dimensional vector search indexing
### Fixed
### Changed
* **search**: Display search results in a responsive card grid with interactive hover video sprite scrubbing, semantic match chunk time range previews, and new-tab opening on click
* **search**: Significantly improve semantic search response times by generating query text embeddings directly with in-memory caching instead of dispatching background tasks
</Update>
<Update label="v0.5.2" description="October 13, 2026">
### Added
+18 -6
View File
@@ -66,12 +66,24 @@ For example, if you deploy Shumai on a remote server at `http://123.456.7.8` and
Configures whether background transcoding and AI agents are executed locally or offloaded to a Temporal cluster.
| Variable | Description | Example / Default |
| :------------------------- | :------------------------------------------------------------------------------------------------- | :------------------------------ |
| `WORKFLOW_EXECUTOR` | Engine choice. Use `local` for in-process database polling, or `temporal` for distributed workers. | `local` (default) or `temporal` |
| `TEMPORAL_ADDRESS` | Host and port of the Temporal cluster gRPC service. | `localhost:7233` (default) |
| `EMBEDDING_CHUNK_DURATION` | The duration in seconds for each video chunk when generating video embeddings. | `60.0` (default) |
| `EMBEDDING_CHUNK_OVERLAP` | The duration in seconds to overlap video chunks when generating video embeddings. | `5.0` (default) |
| Variable | Description | Example / Default |
| :---------------------- | :------------------------------------------------------------------------------------------------- | :------------------------------ |
| `WORKFLOW_EXECUTOR` | Engine choice. Use `local` for in-process database polling, or `temporal` for distributed workers. | `local` (default) or `temporal` |
| `TEMPORAL_ADDRESS` | Host and port of the Temporal cluster gRPC service. | `localhost:7233` (default) |
| `CONCURRENCY_AGENT` | Maximum concurrent AI tasks executed by workers. Set to `1` when running local `embedding-server`. | `5` (default) |
| `CONCURRENCY_TRANSCODE` | Maximum concurrent media transcoding tasks executed by workers. | `1` (default) |
---
## Embedding Config
| Variable | Description | Example / Default |
| :-------------------------------- | :------------------------------------------------------------------------------------------------------------ | :-------------------------------- |
| `EMBEDDING_PROVIDER` | Multimodal embedding provider. Options: `gemini-embedding-2`, `gemma-embedding-2-openai`. | `gemini-embedding-2` (default) |
| `GEMMA_EMBEDDING_OPENAI_BASE_URL` | Base URL of the OpenAI-compatible LiteRT-LM embedding server. | `http://embedding-server:9379/v1` |
| `ENABLE_EMBEDDING_AGENT` | When set to `true`, automatically provisions and enables the system embedding agent for all teams on startup. | `false` (default) |
| `EMBEDDING_CHUNK_DURATION` | The duration in seconds for each video chunk when generating video embeddings. | `60.0` (default) |
| `EMBEDDING_CHUNK_OVERLAP` | The duration in seconds to overlap video chunks when generating video embeddings. | `5.0` (default) |
## Sandbox Config
+2 -1
View File
@@ -45,7 +45,8 @@
"using-shumai/share-links",
"using-shumai/nle-and-premiere",
"using-shumai/3d-asset-support",
"using-shumai/gotenberg"
"using-shumai/gotenberg",
"using-shumai/embedding-server"
]
},
{
+131
View File
@@ -0,0 +1,131 @@
---
title: Embedding Server Setup (EmbeddingGemma 2)
description: Run local, offline multimodal vector embeddings for semantic search using shumaiOne/embedding-server.
---
Shumai supports local, self-hosted multimodal vector embeddings for semantic search through [shumaiOne/embedding-server](https://github.com/shumaiOne/embedding-server) — a standalone Docker service powered by Google AI Edge's **LiteRT-LM** and **EmbeddingGemma 2 740M**.
The embedding server provides a unified 768-dimensional vector space across text, images, and video frames with an OpenAI-compatible HTTP API (`/v1/embeddings`). Because the model weights are pre-baked directly into the Docker image, the server operates completely offline and air-gapped without requiring external network access or third-party cloud API keys.
---
## Model Architecture & Specs
| Property | Value |
| :------------------------ | :------------------------------------------------------- |
| **Model** | `EmbeddingGemma 2 740M` (`google/embeddinggemma-2-740m`) |
| **Runtime** | Google AI Edge LiteRT-LM (Python 3.14 Debian slim) |
| **Vector Dimensionality** | 768 dimensions (multimodal: text, image, and video) |
| **Context Window** | 2,048 tokens |
| **Network Mode** | Completely offline capable (`--network none`) |
| **API Standard** | OpenAI-compatible `/v1/embeddings` |
---
## Video Processing & Sampling
For video assets, Shumai breaks videos into standardized segments and extracts keyframes:
- **Chunking**: Videos are divided into **60-second chunks** with a **5-second overlap** at segment boundaries to preserve narrative and visual context.
- **Frame Extraction Rate**: Shumai extracts **15 frames per minute** from each chunk.
- **Adaptive Frame Bounding**:
- For standard aspect ratios (1:1, 4:3, 3:4, 16:9, 9:16, 21:9), Shumai extracts 15 frames, which produces ~1,982 prompt tokens and stays safely within the 2,048-token context window.
- For non-standard or custom aspect ratios (e.g. 5:3, 1.4:1), Shumai automatically samples 14 frames (~1,990 tokens) to mathematically guarantee that prompt length never exceeds 2,048 tokens, preventing out-of-memory errors on the embedding server.
---
## Important: Changing Embedding Providers Breaks Existing Embeddings
> [!WARNING]
> **Vector spaces from different embedding models are completely incompatible.**
>
> If you change your embedding provider (for example, switching from Google Gemini's 1536-dimensional embeddings to EmbeddingGemma's 768-dimensional embeddings, or vice versa), **all existing vector embeddings in your database become invalid**.
>
> To protect production databases from accidental corruption and search failures, **Shumai intentionally does not provide a WebUI toggle to change the embedding provider**. The provider can only be configured by an administrator via environment variables (`EMBEDDING_PROVIDER` and `GEMMA_EMBEDDING_OPENAI_BASE_URL`).
---
## Regenerating Embeddings via CLI
When you do switch embedding providers or configure the embedding server on an existing Shumai deployment, you can regenerate embeddings for all existing media files using the administrative CLI command:
### Inside Docker
```bash
docker exec -it <shumai-container-name> shumai regenerate-embeddings
```
### Local / Bare Metal
```bash
shumai regenerate-embeddings
```
_(Alternatively, `bun apps/web/src/index.ts regenerate-embeddings`)_
### What the Command Does
1. **Scans Teams**: Checks all teams in your workspace. If a team does not have an active (enabled) embedding agent, it logs a warning and safely skips that team.
2. **Clears Existing Embeddings**: Purges all previous embeddings and cancels any pending/in-progress embedding tasks for eligible assets to ensure a clean transition to the new provider.
3. **Enqueues Fresh Tasks**: Queues new background `ai_embedding` tasks for all eligible assets (images, videos, and 3D assets with rendered previews) across active teams.
4. **Background Execution**: Background workers process the queued tasks according to your configured concurrency limits.
---
## Deployment & Configuration
### Docker Quickstart
To run the standalone embedding server container:
```bash
docker run -d \
--name shumai_embedding \
-p 9379:9379 \
shumaione/embedding-server:latest
```
### Docker Compose Setup
In Docker Compose environments, add `embedding-server` to your `docker-compose.yaml` and configure Shumai to connect to it over Docker's internal network:
```yaml
services:
embedding-server:
image: shumaione/embedding-server:latest
container_name: shumai_embedding
restart: unless-stopped
shumai:
image: shumaione/shumai:latest
environment:
# Embedding Server Configuration
EMBEDDING_PROVIDER: gemma-embedding-2-openai
GEMMA_EMBEDDING_OPENAI_BASE_URL: http://embedding-server:9379/v1
ENABLE_EMBEDDING_AGENT: 'true'
CONCURRENCY_AGENT: 1
depends_on:
- embedding-server
```
> [!NOTE]
> Since Shumai communicates with `embedding-server` over Docker's internal network, `embedding-server` does not need to expose any ports to the host machine.
### Environment Variables
| Variable | Required | Description | Default |
| :-------------------------------- | :---------- | :---------------------------------------------------------------------------------------------------------------------------- | :------------------------- |
| `EMBEDDING_PROVIDER` | Yes | Set to `gemma-embedding-2-openai` to use the local LiteRT-LM server. | `gemini-embedding-2` |
| `GEMMA_EMBEDDING_OPENAI_BASE_URL` | Yes | HTTP base URL of the embedding server (e.g. `http://embedding-server:9379/v1`). | `http://localhost:9379/v1` |
| `ENABLE_EMBEDDING_AGENT` | No | Automatically provisions and enables the system embedding agent for all teams on startup. | `false` |
| `CONCURRENCY_AGENT` | Recommended | Maximum concurrent AI tasks executed by workers. **Set to `1`** when running `embedding-server` to prevent high memory usage. | `5` |
---
## Hardware & Resource Requirements
- **Memory (RAM)**:
- We recommend at least **4 GB+ of dedicated system RAM** for the `embedding-server` container.
- Overall host system memory should be **4 GB+** when running the Lite+Embedding compose stack, and **8 GB+** for the full stack.
- **Concurrency**:
- Always set `CONCURRENCY_AGENT=1` in your Shumai configuration when using `embedding-server`. Parallel multimodal embedding generations consume significant memory and can cause Docker containers to be terminated by the OS Out-Of-Memory (OOM) killer on low-resource machines.
+4 -5
View File
@@ -12,11 +12,10 @@ Traditional search relies on matching exact characters (e.g., searching "dog" wi
Semantic search translates text queries and file contents into conceptual profiles. Similar ideas and visuals are mapped near each other.
1. **Index Generation**: When you upload an asset, Shumai's background agent scans the file's visual elements, text contents, or descriptions, translating them into a concept profile.
- **Video Processing**: Shumai uses **gemini-embedding-2** for video indexing.
- **Chunking**: For long videos, Shumai breaks them into chunks to ensure high-quality indexing. By default, chunks are **60 seconds** long with a **5-second overlap** to ensure no context is lost at the boundaries.
- **Sampling**: The model processes a maximum of 32 frames per chunk: short chunks (≤32s) are sampled at 1 fps, while longer chunks are uniformly sampled to 32 frames.
- **Optimization**: To optimize processing, Shumai automatically transcodes each chunk to **1 FPS** and **removes the audio track** before sending it to the embedding agent.
- **Configuration**: Administrators can customize these settings using the `EMBEDDING_CHUNK_DURATION` and `EMBEDDING_CHUNK_OVERLAP` environment variables.
- **Multimodal Embedding Providers**: Shumai supports two embedding providers:
- **EmbeddingGemma 2 (Local)**: Powered by `shumaiOne/embedding-server` with 768-dimensional multimodal vectors running completely offline. Videos are sampled at **15 frames per minute** (14 frames for non-standard aspect ratios) per 60-second chunk. See the [Embedding Server Setup Guide](./embedding-server) for full details.
- **Gemini Embedding 2 (Cloud)**: Powered by Google's `gemini-embedding-2` API with 1536-dimensional vectors. Videos are chunked into 60-second segments (sampled up to 32 frames per chunk at 1 fps with audio stripped).
- **Chunking Configuration**: Administrators can customize chunk duration and overlap using the `EMBEDDING_CHUNK_DURATION` (default: 60s) and `EMBEDDING_CHUNK_OVERLAP` (default: 5s) environment variables.
2. **Concept Matching**: When you search with the "Semantic" toggle enabled, Shumai matches the conceptual meaning of your query against your assets.
3. **Result**: Searching "sunset at the beach" will successfully find a video file of ocean waves at dusk, even if the filename is `IMG_4029.mp4`.
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "shumai",
"version": "0.5.2",
"version": "0.5.3",
"private": true,
"type": "module",
"repository": {
+3 -2
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/agent",
"version": "0.5.2",
"version": "0.5.3",
"private": true,
"type": "module",
"main": "./src/index.ts",
@@ -9,7 +9,7 @@
"@anthropic-ai/sandbox-runtime": "0.0.74",
"@earendil-works/pi-agent-core": "0.82.1",
"@earendil-works/pi-ai": "0.82.1",
"@google/genai": "2.19.0",
"@google/genai": "2.28.0",
"@shumai/core": "workspace:*",
"@shumai/db": "workspace:*",
"@shumai/dtos": "workspace:*",
@@ -19,6 +19,7 @@
"@temporalio/workflow": "1.23.0",
"adm-zip": "0.6.0",
"jittered-fractional-indexing": "1.0.1",
"sharp": "0.35.4",
"typebox": "1.3.22",
"ulid": "3.0.2",
"zod": "4.5.4"
+493 -30
View File
@@ -1,17 +1,19 @@
import { describe, it, expect, vi, beforeEach } from 'vitest'
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'
import {
generateTextEmbeddingActivity,
extractAiMetadataActivity,
getEmbeddingContextActivity,
saveAssetEmbeddingsActivity,
deleteAssetEmbeddingsActivity,
generateImageEmbeddingActivity,
generateVideoChunkEmbeddingActivity,
generateEmbeddingActivity,
getGemmaEmbeddingTargetFrameCount,
} from './ai'
import { s3Service } from '@shumai/core/src/s3/s3'
import { transcodeService } from '@shumai/core'
import { prisma, AssetType, AssetStatus } from '@shumai/db'
import { setupTestDbHooks } from '@shumai/db/test'
import sharp from 'sharp'
vi.mock('fs', async (importOriginal) => {
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- importOriginal returns the raw fs module, typed dynamically as any here
@@ -54,7 +56,6 @@ vi.mock('child_process', () => {
const execFileMock: any = vi.fn((file: any, args: any, cb: any) => {
cb(null, '120.0\n', '')
})
// Setup custom promisify behavior to return { stdout, stderr }
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- mock functions dynamically return varying types based on test command calls
execFileMock[Symbol.for('nodejs.util.promisify.custom')] = vi.fn(async (file: any) => {
if (file === 'ffprobe') {
@@ -90,6 +91,14 @@ describe('AI Activities Unit Tests', () => {
beforeEach(() => {
vi.clearAllMocks()
process.env.GEMINI_API_KEY = 'test-key'
delete process.env.EMBEDDING_PROVIDER
delete process.env.GEMMA_EMBEDDING_OPENAI_BASE_URL
})
afterEach(() => {
vi.unstubAllGlobals()
delete process.env.EMBEDDING_PROVIDER
delete process.env.GEMMA_EMBEDDING_OPENAI_BASE_URL
})
it('should generate image embedding', async () => {
@@ -126,33 +135,6 @@ describe('AI Activities Unit Tests', () => {
)
})
it('should resolve video input via resolveInput and avoid downloadToFile in generateEmbeddingActivity', async () => {
vi.mocked(s3Service.downloadToFile).mockClear()
vi.mocked(s3Service.resolveInput).mockClear()
vi.mocked(s3Service.resolveInput).mockResolvedValue(
'https://mock-storage/files/asset-123/video.mp4',
)
const res = await generateEmbeddingActivity({
teamId: 't1',
assetId: 'asset-123',
context: {
agent: {},
asset: {
mediaType: 'video/mp4',
name: 'sample.mp4',
storageKey: { key: 'files/asset-123/video.mp4' },
media: { proxyType: 'video' },
},
},
})
expect(s3Service.resolveInput).toHaveBeenCalledWith('shumai', 'files/asset-123/video.mp4')
expect(s3Service.downloadToFile).not.toHaveBeenCalled()
expect(s3Service.getObject).not.toHaveBeenCalledWith('shumai', 'files/asset-123/video.mp4')
expect(res.embeddings.length).toBeGreaterThan(0)
})
it('should generate text embedding successfully', async () => {
const res = await generateTextEmbeddingActivity({
text: 'Search query',
@@ -172,6 +154,356 @@ describe('AI Activities Unit Tests', () => {
)
})
it('should generate text embedding successfully with gemma-embedding-2-openai', async () => {
process.env.EMBEDDING_PROVIDER = 'gemma-embedding-2-openai'
const mockFetch = vi.fn().mockResolvedValue({
ok: true,
json: async () => ({
model: 'embeddinggemma-2',
data: [{ embedding: Array(768).fill(0.1) }],
// eslint-disable-next-line @typescript-eslint/naming-convention
usage: { prompt_tokens: 15 },
}),
})
vi.stubGlobal('fetch', mockFetch)
const res = await generateTextEmbeddingActivity({
text: 'Search query',
teamId: 't1',
})
expect(res.embedding).toHaveLength(768)
expect(res.usage.model).toBe('embeddinggemma-2')
expect(res.usage.inputTokens).toBe(15)
expect(mockFetch).toHaveBeenCalledWith(
'http://localhost:9379/v1/embeddings',
expect.objectContaining({
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
model: 'embeddinggemma-2',
input: 'task: search query | text: Search query',
}),
}),
)
})
it('should generate image embedding successfully with gemma-embedding-2-openai', async () => {
process.env.EMBEDDING_PROVIDER = 'gemma-embedding-2-openai'
const mockFetch = vi.fn().mockResolvedValue({
ok: true,
json: async () => ({
model: 'embeddinggemma-2',
data: [{ embedding: Array(768).fill(0.2) }],
// eslint-disable-next-line @typescript-eslint/naming-convention
usage: { prompt_tokens: 140 },
}),
})
vi.stubGlobal('fetch', mockFetch)
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('test-image'),
contentType: 'image/png',
} as unknown as { buffer: Buffer; contentType: string })
const res = await generateImageEmbeddingActivity({
teamId: 't1',
assetKey: 'img.png',
mediaType: 'image/png',
})
expect(res.embedding).toHaveLength(768)
expect(res.usage.model).toBe('embeddinggemma-2')
expect(res.usage.inputTokens).toBe(140)
expect(mockFetch).toHaveBeenCalledWith(
'http://localhost:9379/v1/embeddings',
expect.objectContaining({
method: 'POST',
body: JSON.stringify({
model: 'embeddinggemma-2',
input: [
{
type: 'image_url',
// eslint-disable-next-line @typescript-eslint/naming-convention
image_url: {
url: `data:image/png;base64,${Buffer.from('test-image').toString('base64')}`,
},
},
],
}),
}),
)
})
it('should respect custom GEMMA_EMBEDDING_OPENAI_BASE_URL with or without /v1 suffix', async () => {
process.env.EMBEDDING_PROVIDER = 'gemma-embedding-2-openai'
const mockFetch = vi.fn().mockResolvedValue({
ok: true,
json: async () => ({
model: 'embeddinggemma-2',
data: [{ embedding: Array(768).fill(0.2) }],
// eslint-disable-next-line @typescript-eslint/naming-convention
usage: { prompt_tokens: 10 },
}),
})
vi.stubGlobal('fetch', mockFetch)
// Test with /v1
process.env.GEMMA_EMBEDDING_OPENAI_BASE_URL = 'http://custom-host:9379/v1'
await generateTextEmbeddingActivity({ text: 'test query', teamId: 't1' })
expect(mockFetch).toHaveBeenLastCalledWith(
'http://custom-host:9379/v1/embeddings',
expect.anything(),
)
// Test without /v1
process.env.GEMMA_EMBEDDING_OPENAI_BASE_URL = 'http://custom-host:9379'
await generateTextEmbeddingActivity({ text: 'test query', teamId: 't1' })
expect(mockFetch).toHaveBeenLastCalledWith(
'http://custom-host:9379/v1/embeddings',
expect.anything(),
)
})
it('should convert webp image to jpeg with gemma-embedding-2-openai', async () => {
process.env.EMBEDDING_PROVIDER = 'gemma-embedding-2-openai'
const mockFetch = vi.fn().mockResolvedValue({
ok: true,
json: async () => ({
model: 'embeddinggemma-2',
data: [{ embedding: Array(768).fill(0.2) }],
// eslint-disable-next-line @typescript-eslint/naming-convention
usage: { prompt_tokens: 140 },
}),
})
vi.stubGlobal('fetch', mockFetch)
const webpBuffer = await sharp({
create: { width: 10, height: 10, channels: 3, background: { r: 255, g: 0, b: 0 } },
})
.webp()
.toBuffer()
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: webpBuffer,
contentType: 'image/webp',
} as unknown as { buffer: Buffer; contentType: string })
const res = await generateImageEmbeddingActivity({
teamId: 't1',
assetKey: 'img.webp',
mediaType: 'image/webp',
})
expect(res.embedding).toHaveLength(768)
expect(mockFetch).toHaveBeenCalledWith(
'http://localhost:9379/v1/embeddings',
expect.objectContaining({
method: 'POST',
body: expect.stringContaining('data:image/jpeg;base64,'),
}),
)
})
it('should generate video chunk embedding from tar archive with gemma-embedding-2-openai', async () => {
process.env.EMBEDDING_PROVIDER = 'gemma-embedding-2-openai'
const mockFetch = vi.fn().mockResolvedValue({
ok: true,
json: async () => ({
model: 'embeddinggemma-2',
data: [{ embedding: Array(768).fill(0.3) }],
// eslint-disable-next-line @typescript-eslint/naming-convention
usage: { prompt_tokens: 140 },
}),
})
vi.stubGlobal('fetch', mockFetch)
const sampleArchive = new Bun.Archive({
'frame_0001.jpg': Buffer.from('frame-1'),
})
const archiveBytes = await sampleArchive.bytes()
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from(archiveBytes.buffer, archiveBytes.byteOffset, archiveBytes.byteLength),
contentType: 'application/x-tar',
} as unknown as { buffer: Buffer; contentType: string })
const res = await generateVideoChunkEmbeddingActivity({
teamId: 't1',
chunkKey: 'files/asset-1/tmp-embedding-chunks/frames-0-60.tar',
})
expect(res.embedding).toHaveLength(768)
expect(res.usage.model).toBe('embeddinggemma-2')
expect(mockFetch).toHaveBeenCalled()
})
describe('getGemmaEmbeddingTargetFrameCount', () => {
it('should return 15 for standard aspect ratios', () => {
// 16:9
expect(getGemmaEmbeddingTargetFrameCount(1920, 1080)).toBe(15)
expect(getGemmaEmbeddingTargetFrameCount(1280, 720)).toBe(15)
expect(getGemmaEmbeddingTargetFrameCount(854, 480)).toBe(15)
// 9:16
expect(getGemmaEmbeddingTargetFrameCount(1080, 1920)).toBe(15)
// 1:1
expect(getGemmaEmbeddingTargetFrameCount(1080, 1080)).toBe(15)
expect(getGemmaEmbeddingTargetFrameCount(480, 480)).toBe(15)
// 4:3
expect(getGemmaEmbeddingTargetFrameCount(1440, 1080)).toBe(15)
expect(getGemmaEmbeddingTargetFrameCount(640, 480)).toBe(15)
// 3:4
expect(getGemmaEmbeddingTargetFrameCount(1080, 1440)).toBe(15)
// 21:9
expect(getGemmaEmbeddingTargetFrameCount(2560, 1080)).toBe(15)
expect(getGemmaEmbeddingTargetFrameCount(3440, 1440)).toBe(15)
})
it('should return 14 for non-standard aspect ratios and missing/invalid dimensions', () => {
expect(getGemmaEmbeddingTargetFrameCount(1000, 600)).toBe(14) // 5:3
expect(getGemmaEmbeddingTargetFrameCount(700, 500)).toBe(14) // 1.4:1
expect(getGemmaEmbeddingTargetFrameCount(854, 672)).toBe(14) // ~1.27:1
expect(getGemmaEmbeddingTargetFrameCount(undefined, undefined)).toBe(14)
expect(getGemmaEmbeddingTargetFrameCount(0, 0)).toBe(14)
expect(getGemmaEmbeddingTargetFrameCount(-100, 100)).toBe(14)
})
})
it('should sample uniform 15 frames when chunk archive contains standard 16:9 frames', async () => {
process.env.EMBEDDING_PROVIDER = 'gemma-embedding-2-openai'
const mockFetch = vi.fn().mockResolvedValue({
ok: true,
json: async () => ({
model: 'embeddinggemma-2',
data: [{ embedding: Array(768).fill(0.3) }],
// eslint-disable-next-line @typescript-eslint/naming-convention
usage: { prompt_tokens: 140 },
}),
})
vi.stubGlobal('fetch', mockFetch)
const sampleJpg = await sharp({
create: {
width: 160,
height: 90,
channels: 3,
background: { r: 0, g: 0, b: 0 },
},
})
.jpeg()
.toBuffer()
const frames: Record<string, Buffer> = {}
for (let i = 1; i <= 30; i++) {
frames[`frame_${String(i).padStart(4, '0')}.jpg`] = sampleJpg
}
const sampleArchive = new Bun.Archive(frames)
const archiveBytes = await sampleArchive.bytes()
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from(archiveBytes.buffer, archiveBytes.byteOffset, archiveBytes.byteLength),
contentType: 'application/x-tar',
} as unknown as { buffer: Buffer; contentType: string })
const res = await generateVideoChunkEmbeddingActivity({
teamId: 't1',
chunkKey: 'files/asset-1/tmp-embedding-chunks/frames-0-60.tar',
})
expect(res.embedding).toHaveLength(768)
const requestBody = JSON.parse(mockFetch.mock.calls[0][1].body)
expect(requestBody.input).toHaveLength(15)
})
it('should fallback to 14 frames when chunk archive contains non-standard or unparseable frames', async () => {
process.env.EMBEDDING_PROVIDER = 'gemma-embedding-2-openai'
const mockFetch = vi.fn().mockResolvedValue({
ok: true,
json: async () => ({
model: 'embeddinggemma-2',
data: [{ embedding: Array(768).fill(0.3) }],
// eslint-disable-next-line @typescript-eslint/naming-convention
usage: { prompt_tokens: 140 },
}),
})
vi.stubGlobal('fetch', mockFetch)
const frames: Record<string, Buffer> = {}
for (let i = 1; i <= 30; i++) {
frames[`frame_${String(i).padStart(4, '0')}.jpg`] = Buffer.from(`frame-${i}`)
}
const sampleArchive = new Bun.Archive(frames)
const archiveBytes = await sampleArchive.bytes()
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from(archiveBytes.buffer, archiveBytes.byteOffset, archiveBytes.byteLength),
contentType: 'application/x-tar',
} as unknown as { buffer: Buffer; contentType: string })
const res = await generateVideoChunkEmbeddingActivity({
teamId: 't1',
chunkKey: 'files/asset-1/tmp-embedding-chunks/frames-0-60.tar',
})
expect(res.embedding).toHaveLength(768)
const requestBody = JSON.parse(mockFetch.mock.calls[0][1].body)
expect(requestBody.input).toHaveLength(14)
})
it('should throw non-retryable ApplicationFailure when chunk archive contains no valid frames', async () => {
process.env.EMBEDDING_PROVIDER = 'gemma-embedding-2-openai'
const sampleArchive = new Bun.Archive({
'info.txt': Buffer.from('not an image'),
})
const archiveBytes = await sampleArchive.bytes()
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from(archiveBytes.buffer, archiveBytes.byteOffset, archiveBytes.byteLength),
contentType: 'application/x-tar',
} as unknown as { buffer: Buffer; contentType: string })
await expect(
generateVideoChunkEmbeddingActivity({
teamId: 't1',
chunkKey: 'files/asset-1/tmp-embedding-chunks/frames-0-60.tar',
}),
).rejects.toThrow('No extracted frames found in chunk archive')
})
it('should throw non-retryable ApplicationFailure when chunk archive data is corrupted', async () => {
process.env.EMBEDDING_PROVIDER = 'gemma-embedding-2-openai'
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from('corrupted non-tar bytes'),
contentType: 'application/x-tar',
} as unknown as { buffer: Buffer; contentType: string })
await expect(
generateVideoChunkEmbeddingActivity({
teamId: 't1',
chunkKey: 'files/asset-1/tmp-embedding-chunks/frames-0-60.tar',
}),
).rejects.toThrow('Failed to read chunk archive')
})
it('should throw non-retryable ApplicationFailure when an extracted frame is empty (0 bytes)', async () => {
process.env.EMBEDDING_PROVIDER = 'gemma-embedding-2-openai'
const sampleArchive = new Bun.Archive({
'frame_0001.jpg': Buffer.alloc(0),
})
const archiveBytes = await sampleArchive.bytes()
vi.mocked(s3Service.getObject).mockResolvedValue({
buffer: Buffer.from(archiveBytes.buffer, archiveBytes.byteOffset, archiveBytes.byteLength),
contentType: 'application/x-tar',
} as unknown as { buffer: Buffer; contentType: string })
await expect(
generateVideoChunkEmbeddingActivity({
teamId: 't1',
chunkKey: 'files/asset-1/tmp-embedding-chunks/frames-0-60.tar',
}),
).rejects.toThrow('is empty (0 bytes)')
})
describe('extractAiMetadataActivity', () => {
it('should return existing images if they are already in S3 storage', async () => {
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- mock returned object cast as any due to partial implementation
@@ -331,6 +663,18 @@ describe('AI Database Activities Integration', () => {
expect(ctx.agent.id).toBe(user.id)
expect(ctx.asset.id).toBe(asset.id)
expect(ctx.asset.mediaType).toBe('video/mp4')
process.env.ENABLE_EMBEDDING_AGENT = 'false'
try {
await expect(
getEmbeddingContextActivity({
teamId: team.id,
assetId: asset.id,
}),
).rejects.toThrow('embedding feature is disabled or agent not found')
} finally {
delete process.env.ENABLE_EMBEDDING_AGENT
}
})
it('should parse valid positive chunk duration and fallback on invalid or negative ones', async () => {
@@ -389,6 +733,45 @@ describe('AI Database Activities Integration', () => {
delete process.env.EMBEDDING_CHUNK_DURATION
delete process.env.EMBEDDING_CHUNK_OVERLAP
})
it('should return existing embeddings for the asset', async () => {
// Ensure agent exists
await prisma.agent.upsert({
where: { id: user.id },
update: {},
create: {
id: user.id,
teamId: team.id,
type: 'embedding',
enabled: true,
config: { provider: 'openai', model: 'gpt-4' },
},
})
await saveAssetEmbeddingsActivity({
assetId: asset.id,
embeddings: [
{ embedding: Array(1536).fill(0.01), startTime: 0.0, endTime: 60.0 },
{ embedding: Array(768).fill(0.02), startTime: 55.0, endTime: 115.0 },
],
})
const ctx = await getEmbeddingContextActivity({
teamId: team.id,
assetId: asset.id,
})
expect(ctx.existingEmbeddings).toHaveLength(2)
const chunk1 = ctx.existingEmbeddings.find((e) => e.startTime === 0)
expect(chunk1).toBeDefined()
expect(chunk1?.has1536).toBe(true)
expect(chunk1?.has768).toBe(false)
const chunk2 = ctx.existingEmbeddings.find((e) => e.startTime === 55)
expect(chunk2).toBeDefined()
expect(chunk2?.has1536).toBe(false)
expect(chunk2?.has768).toBe(true)
})
})
describe('saveAssetEmbeddingsActivity', () => {
@@ -423,5 +806,85 @@ describe('AI Database Activities Integration', () => {
expect(dbEmbeddings[0].endTime).toBe(60.0)
expect(dbEmbeddings[1].startTime).toBeNull()
})
it('should save 768-dim embeddings to embedding_768 column in database', async () => {
const embeddings = [
{
embedding: Array(768).fill(0.01),
startTime: 0.0,
endTime: 60.0,
},
{
embedding: Array(768).fill(0.02),
},
]
await saveAssetEmbeddingsActivity({
assetId: asset.id,
embeddings,
})
// Query raw embeddings from DB
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- db query output is a raw postgres result array typed as any
const dbEmbeddings: any[] = await prisma.$queryRaw`
SELECT id, asset_id as "assetId", (embedding_768 IS NOT NULL) as "has768", (embedding IS NOT NULL) as "has1536"
FROM asset_embeddings
WHERE asset_id = ${asset.id}
`
expect(dbEmbeddings.length).toBe(2)
expect(dbEmbeddings[0].has768).toBe(true)
expect(dbEmbeddings[0].has1536).toBe(false)
expect(dbEmbeddings[1].has768).toBe(true)
expect(dbEmbeddings[1].has1536).toBe(false)
})
})
describe('deleteAssetEmbeddingsActivity', () => {
it('should delete specific embeddings by id', async () => {
await saveAssetEmbeddingsActivity({
assetId: asset.id,
embeddings: [
{ embedding: Array(1536).fill(0.01), startTime: 0, endTime: 60 },
{ embedding: Array(1536).fill(0.02), startTime: 55, endTime: 115 },
],
})
const before = await prisma.assetEmbedding.findMany({
where: { assetId: asset.id },
})
expect(before).toHaveLength(2)
const targetId = before[0].id
await deleteAssetEmbeddingsActivity({
assetId: asset.id,
ids: [targetId],
})
const after = await prisma.assetEmbedding.findMany({
where: { assetId: asset.id },
})
expect(after).toHaveLength(1)
expect(after[0].id).not.toBe(targetId)
})
it('should delete all embeddings for an asset when ids is not provided', async () => {
await saveAssetEmbeddingsActivity({
assetId: asset.id,
embeddings: [
{ embedding: Array(1536).fill(0.01), startTime: 0, endTime: 60 },
{ embedding: Array(1536).fill(0.02), startTime: 55, endTime: 115 },
],
})
await deleteAssetEmbeddingsActivity({
assetId: asset.id,
})
const after = await prisma.assetEmbedding.findMany({
where: { assetId: asset.id },
})
expect(after).toHaveLength(0)
})
})
})
+264 -171
View File
@@ -1,17 +1,11 @@
import { GoogleGenAI } from '@google/genai'
import { s3Service } from '@shumai/core/src/s3/s3'
import { transcodeService } from '@shumai/core'
import { execFile } from 'child_process'
import * as fs from 'fs'
import * as os from 'os'
import * as path from 'path'
import { promisify } from 'util'
import { ApplicationFailure } from '@temporalio/activity'
import { prisma } from '@shumai/db'
import { getProxyType } from '@shumai/core/src/utils/mime'
import { ulid } from 'ulid'
const execFileAsync = promisify(execFile)
import sharp from 'sharp'
export interface Usage {
inputTokens: number
@@ -25,15 +19,6 @@ export interface GeneratedEmbedding {
endTime?: number
}
export interface GenerateEmbeddingParams {
teamId: string
assetId: string
context: {
agent: unknown
asset: unknown
}
}
async function generateMultimodalEmbedding(
ai: GoogleGenAI,
buffer: Buffer,
@@ -62,134 +47,66 @@ async function generateMultimodalEmbedding(
return values
}
export async function generateEmbeddingActivity(params: GenerateEmbeddingParams): Promise<{
embeddings: GeneratedEmbedding[]
interface OpenAiEmbeddingItem {
type?: string
// eslint-disable-next-line @typescript-eslint/naming-convention
image_url?: {
url: string
}
}
async function callGemmaEmbeddingServer(input: string | OpenAiEmbeddingItem[]): Promise<{
embedding: number[]
usage: Usage
}> {
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- context properties are fetched as JSON from the DB activity
const { asset } = params.context as any
const rawBaseUrl = process.env.GEMMA_EMBEDDING_OPENAI_BASE_URL || 'http://localhost:9379/v1'
if (!asset.mediaType) {
throw ApplicationFailure.create({ message: 'asset has no media type', nonRetryable: true })
}
const normalized = rawBaseUrl.replace(/\/+$/, '')
const endpoint = normalized.endsWith('/embeddings')
? normalized
: normalized.endsWith('/v1')
? `${normalized}/embeddings`
: `${normalized}/v1/embeddings`
const apiKey = process.env.GEMINI_API_KEY
if (!apiKey) {
const response = await fetch(endpoint, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
model: 'embeddinggemma-2',
input,
}),
})
if (!response.ok) {
const errorText = await response.text()
throw ApplicationFailure.create({
message: 'GEMINI_API_KEY environment variable is not configured',
nonRetryable: true,
message: `Embedding server returned status ${response.status}: ${errorText}`,
nonRetryable: false,
})
}
const resolvedApiKey = process.env[apiKey] || apiKey
const ai = new GoogleGenAI({ apiKey: resolvedApiKey })
const usage: Usage = {
model: 'gemini-embedding-2',
inputTokens: 0,
outputTokens: 0,
}
const proxyType =
(asset.media as PrismaJson.MediaInfo | null)?.proxyType ||
getProxyType(asset.mediaType, asset.name)
const isImage = proxyType === 'image'
const isVideo = proxyType === 'video'
if (!isImage && !isVideo) {
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- response json shape is dynamic from external endpoint
const json: any = await response.json()
const embedding = json.data?.[0]?.embedding
if (!embedding || !Array.isArray(embedding)) {
throw ApplicationFailure.create({
message: `unsupported media type for embeddings: ${asset.mediaType}`,
nonRetryable: true,
message: 'Invalid embedding returned from embedding server',
nonRetryable: false,
})
}
const key = asset.storageKey?.key
if (!key) {
throw ApplicationFailure.create({ message: 'asset has no key', nonRetryable: true })
}
const bucket = process.env.S3_BUCKET || 'shumai'
const results: GeneratedEmbedding[] = []
if (isImage) {
const { buffer: data } = await s3Service.getObject(bucket, key)
const embVec = await generateMultimodalEmbedding(ai, data, asset.mediaType)
results.push({ embedding: embVec })
} else if (isVideo) {
const inputSource = await s3Service.resolveInput(bucket, key)
const isRemote = inputSource.startsWith('http://') || inputSource.startsWith('https://')
const { stdout } = await execFileAsync('ffprobe', [
'-v',
'error',
'-show_entries',
'format=duration',
'-of',
'default=noprint_wrappers=1:nokey=1',
inputSource,
])
const duration = parseFloat(stdout.trim())
if (isNaN(duration)) {
throw ApplicationFailure.create({
message: 'failed to parse video duration',
nonRetryable: true,
})
}
const chunkSize = 60.0
for (let start = 0.0; start < duration; start += chunkSize) {
let end = start + chunkSize
if (end > duration) end = duration
const chunkTmp = path.join(os.tmpdir(), `video-chunk-${Date.now()}-${ulid()}.mp4`)
try {
await execFileAsync('ffmpeg', [
'-y',
'-loglevel',
'warning',
...(isRemote
? ['-reconnect', '1', '-reconnect_streamed', '1', '-reconnect_delay_max', '5']
: []),
'-ss',
start.toString(),
'-i',
inputSource,
'-map',
'0:V:0',
'-map',
'0:a:0?',
'-t',
(end - start).toString(),
'-c',
'copy',
chunkTmp,
])
const chunkData = fs.readFileSync(chunkTmp)
try {
const embVec = await generateMultimodalEmbedding(ai, chunkData, 'video/mp4')
results.push({ embedding: embVec, startTime: start, endTime: end })
// eslint-disable-next-line @typescript-eslint/no-explicit-any
} catch (e: any) {
console.error(
`failed to generate video embedding for chunk ${start}-${end}: ${e.message}`,
)
}
} finally {
if (fs.existsSync(chunkTmp)) {
fs.unlinkSync(chunkTmp)
}
}
}
}
return {
embeddings: results,
usage,
embedding,
usage: {
model: json.model || 'gemma-embedding-2-openai',
inputTokens: json.usage?.prompt_tokens ?? 0,
outputTokens: 0,
},
}
}
import { textEmbeddingService } from '@shumai/core/src/search/text-embedding'
export interface GenerateTextEmbeddingParams {
text: string
teamId: string
@@ -199,44 +116,36 @@ export async function generateTextEmbeddingActivity(params: GenerateTextEmbeddin
embedding: number[]
usage: Usage
}> {
const apiKey = process.env.GEMINI_API_KEY
if (!apiKey) {
const provider = process.env.EMBEDDING_PROVIDER || 'gemini-embedding-2'
if (provider !== 'gemma-embedding-2-openai' && !process.env.GEMINI_API_KEY) {
throw ApplicationFailure.create({
message: 'GEMINI_API_KEY environment variable is not configured',
nonRetryable: true,
})
}
const resolvedApiKey = process.env[apiKey] || apiKey
const ai = new GoogleGenAI({ apiKey: resolvedApiKey })
try {
const result = await textEmbeddingService.generateEmbedding(params.text, {
teamId: params.teamId,
skipCache: true,
})
const usage: Usage = {
model: 'gemini-embedding-2',
inputTokens: 0,
outputTokens: 0,
}
const response = await ai.models.embedContent({
model: 'gemini-embedding-2',
contents: [
{
parts: [{ text: params.text }],
role: 'user',
return {
embedding: result.embedding,
usage: {
model: result.usage?.model || provider,
inputTokens: result.usage?.inputTokens ?? 0,
outputTokens: result.usage?.outputTokens ?? 0,
},
],
config: {
outputDimensionality: 1536,
},
})
const values = response.embeddings?.[0]?.values
if (!values) {
throw new Error('No embedding values returned from gemini-embedding-2')
}
return {
embedding: values,
usage,
}
} catch (err) {
if (err instanceof ApplicationFailure) {
throw err
}
throw ApplicationFailure.create({
message: err instanceof Error ? err.message : String(err),
nonRetryable: false,
})
}
}
@@ -315,7 +224,7 @@ export async function getEmbeddingContextActivity(params: { teamId: string; asse
user: { teamMembers: { some: { teamId: params.teamId } } },
},
})
if (!agent) {
if (!agent || process.env.ENABLE_EMBEDDING_AGENT === 'false') {
throw ApplicationFailure.create({
message: 'embedding feature is disabled or agent not found',
nonRetryable: true,
@@ -339,12 +248,51 @@ export async function getEmbeddingContextActivity(params: { teamId: string; asse
const parsedOverlap = parseFloat(process.env.EMBEDDING_CHUNK_OVERLAP || '')
const chunkOverlap = Number.isFinite(parsedOverlap) && parsedOverlap >= 0 ? parsedOverlap : 5.0
const embeddingProvider = process.env.EMBEDDING_PROVIDER || 'gemini-embedding-2'
const existingEmbeddings = await prisma.$queryRaw<ExistingAssetEmbeddingInfo[]>`
SELECT
id,
start_time as "startTime",
end_time as "endTime",
(embedding IS NOT NULL) as "has1536",
(embedding_768 IS NOT NULL) as "has768"
FROM asset_embeddings
WHERE asset_id = ${params.assetId}
`
return {
agent,
asset,
chunkDuration,
chunkOverlap,
embeddingProvider,
existingEmbeddings,
}
}
export interface ExistingAssetEmbeddingInfo {
id: string
startTime: number | null
endTime: number | null
has1536: boolean
has768: boolean
}
export async function deleteAssetEmbeddingsActivity(params: { assetId: string; ids?: string[] }) {
if (params.ids && params.ids.length > 0) {
await prisma.assetEmbedding.deleteMany({
where: {
assetId: params.assetId,
id: { in: params.ids },
},
})
} else {
await prisma.assetEmbedding.deleteMany({
where: {
assetId: params.assetId,
},
})
}
}
@@ -358,16 +306,32 @@ export async function saveAssetEmbeddingsActivity(params: {
}) {
for (const item of params.embeddings) {
const embVec = JSON.stringify(item.embedding)
const is768 = item.embedding.length === 768
if (item.startTime !== undefined && item.endTime !== undefined) {
await prisma.$executeRaw`
INSERT INTO asset_embeddings (id, asset_id, embedding, start_time, end_time, updated_at)
VALUES (gen_random_uuid()::text, ${params.assetId}, ${embVec}::vector, ${item.startTime}, ${item.endTime}, NOW())
`
if (is768) {
await prisma.$executeRaw`
INSERT INTO asset_embeddings (id, asset_id, embedding_768, start_time, end_time, updated_at)
VALUES (gen_random_uuid()::text, ${params.assetId}, ${embVec}::vector, ${item.startTime}, ${item.endTime}, NOW())
`
} else {
await prisma.$executeRaw`
INSERT INTO asset_embeddings (id, asset_id, embedding, start_time, end_time, updated_at)
VALUES (gen_random_uuid()::text, ${params.assetId}, ${embVec}::vector, ${item.startTime}, ${item.endTime}, NOW())
`
}
} else {
await prisma.$executeRaw`
INSERT INTO asset_embeddings (id, asset_id, embedding, updated_at)
VALUES (gen_random_uuid()::text, ${params.assetId}, ${embVec}::vector, NOW())
`
if (is768) {
await prisma.$executeRaw`
INSERT INTO asset_embeddings (id, asset_id, embedding_768, updated_at)
VALUES (gen_random_uuid()::text, ${params.assetId}, ${embVec}::vector, NOW())
`
} else {
await prisma.$executeRaw`
INSERT INTO asset_embeddings (id, asset_id, embedding, updated_at)
VALUES (gen_random_uuid()::text, ${params.assetId}, ${embVec}::vector, NOW())
`
}
}
}
}
@@ -384,6 +348,30 @@ export async function generateImageEmbeddingActivity(
embedding: number[]
usage: Usage
}> {
const provider = process.env.EMBEDDING_PROVIDER || 'gemini-embedding-2'
const { buffer: data } = await s3Service.getObject(
process.env.S3_BUCKET || 'shumai',
params.assetKey,
)
if (provider === 'gemma-embedding-2-openai') {
let imgData = data
let mimeType = params.mediaType || 'image/jpeg'
if (mimeType !== 'image/jpeg' && mimeType !== 'image/png') {
imgData = await sharp(data).jpeg().toBuffer()
mimeType = 'image/jpeg'
}
return callGemmaEmbeddingServer([
{
type: 'image_url',
// eslint-disable-next-line @typescript-eslint/naming-convention
image_url: {
url: `data:${mimeType};base64,${imgData.toString('base64')}`,
},
},
])
}
const apiKey = process.env.GEMINI_API_KEY
if (!apiKey) {
throw ApplicationFailure.create({
@@ -401,10 +389,6 @@ export async function generateImageEmbeddingActivity(
outputTokens: 0,
}
const { buffer: data } = await s3Service.getObject(
process.env.S3_BUCKET || 'shumai',
params.assetKey,
)
const embVec = await generateMultimodalEmbedding(ai, data, params.mediaType)
return {
@@ -413,6 +397,38 @@ export async function generateImageEmbeddingActivity(
}
}
/**
* Standard aspect ratios for video embedding:
* 1:1, 4:3, 3:4, 16:9, 9:16, 21:9 (including ultrawide 64:27), and 9:21.
*
* For these standard ratios, 15 frames guarantees that total input tokens stay strictly
* within the 2048 token signature bucket (<= 1982 tokens). For non-standard or custom
* aspect ratios (e.g. 5:3, 1.4:1), 14 frames is used to mathematically guarantee
* that sequence length never exceeds 2048 tokens (<= 1990 tokens), preventing OOM errors.
*/
export function getGemmaEmbeddingTargetFrameCount(width?: number, height?: number): number {
if (!width || !height || width <= 0 || height <= 0) {
return 14
}
const ratio = width / height
const standardRatios = [
1 / 1, // 1:1
4 / 3, // 4:3
3 / 4, // 3:4
16 / 9, // 16:9
9 / 16, // 9:16
21 / 9, // 21:9
64 / 27, // 21:9 ultrawide (2560x1080, 3440x1440)
9 / 21, // 9:21
27 / 64, // 9:21 ultrawide portrait
]
// Allow up to 3% relative tolerance for minor encoder rounding (e.g. 854x480 vs 16:9)
const isStandard = standardRatios.some((std) => Math.abs(ratio - std) / std <= 0.03)
return isStandard ? 15 : 14
}
export interface GenerateVideoChunkEmbeddingParams {
teamId: string
chunkKey: string
@@ -424,6 +440,83 @@ export async function generateVideoChunkEmbeddingActivity(
embedding: number[]
usage: Usage
}> {
const provider = process.env.EMBEDDING_PROVIDER || 'gemini-embedding-2'
const isGemma = provider === 'gemma-embedding-2-openai' || params.chunkKey.endsWith('.tar')
if (isGemma) {
const { buffer: data } = await s3Service.getObject(
process.env.S3_BUCKET || 'shumai',
params.chunkKey,
)
let filesMap: Map<string, File>
try {
const archive = new Bun.Archive(data)
filesMap = await archive.files()
} catch (err) {
throw ApplicationFailure.create({
message: `Failed to read chunk archive ${params.chunkKey}: ${err instanceof Error ? err.message : String(err)}`,
nonRetryable: true,
})
}
const frameFiles = Array.from(filesMap.keys())
.filter((f) => /\.(jpe?g|png|webp)$/i.test(f))
.sort()
if (frameFiles.length === 0) {
throw ApplicationFailure.create({
message: `No extracted frames found in chunk archive ${params.chunkKey}`,
nonRetryable: true,
})
}
let targetFrameCount = 14
try {
const firstFile = filesMap.get(frameFiles[0])
if (firstFile) {
const firstFileBytes = await firstFile.bytes()
if (firstFileBytes.byteLength > 0) {
const meta = await sharp(Buffer.from(firstFileBytes)).metadata()
targetFrameCount = getGemmaEmbeddingTargetFrameCount(meta.width, meta.height)
}
}
} catch {
targetFrameCount = 14
}
let selectedFrames = frameFiles
if (selectedFrames.length > targetFrameCount) {
const step = (selectedFrames.length - 1) / (targetFrameCount - 1)
selectedFrames = Array.from(
{ length: targetFrameCount },
(_, i) => selectedFrames[Math.round(i * step)],
)
}
const input: OpenAiEmbeddingItem[] = await Promise.all(
selectedFrames.map(async (f) => {
const file = filesMap.get(f)!
const fileBytes = await file.bytes()
if (fileBytes.byteLength === 0) {
throw ApplicationFailure.create({
message: `Extracted frame ${f} in chunk archive ${params.chunkKey} is empty (0 bytes)`,
nonRetryable: true,
})
}
return {
type: 'image_url',
// eslint-disable-next-line @typescript-eslint/naming-convention
image_url: {
url: `data:image/jpeg;base64,${Buffer.from(fileBytes).toString('base64')}`,
},
}
}),
)
return await callGemmaEmbeddingServer(input)
}
const apiKey = process.env.GEMINI_API_KEY
if (!apiKey) {
throw ApplicationFailure.create({
+3 -5
View File
@@ -610,11 +610,9 @@ import * as aiActivities from './activities/ai'
import { agentAutofillMedia } from './workflows/agent-autofill'
import { agentChat } from './workflows/agent-chat'
import { agentEmbeddingMedia } from './workflows/agent-embedding'
import { queryEmbeddingForSearch } from './workflows/query-embedding-for-search'
export function initAgentWorkflows() {
registerWorkflow(WorkflowTaskType.ai_embedding, agentEmbeddingMedia)
registerWorkflow(WorkflowTaskType.query_embedding_for_search, queryEmbeddingForSearch)
registerWorkflow(WorkflowTaskType.ai_metadata_autofill, agentAutofillMedia)
registerWorkflow(WorkflowTaskType.chat, agentChat)
@@ -625,21 +623,21 @@ export function initAgentWorkflows() {
export * from './activities/agent'
export {
extractAiMetadataActivity,
generateEmbeddingActivity,
generateImageEmbeddingActivity,
generateTextEmbeddingActivity,
generateVideoChunkEmbeddingActivity,
saveAssetEmbeddingsActivity,
deleteAssetEmbeddingsActivity,
type ExtractAiMetadataParams,
type GeneratedEmbedding,
type GenerateEmbeddingParams,
type GenerateTextEmbeddingParams,
type ExistingAssetEmbeddingInfo,
} from './activities/ai'
export * from './database-session-storage'
export * from './context/serialize-context'
export * from './workflows/agent-autofill'
export * from './workflows/agent-chat'
export * from './workflows/agent-embedding'
export * from './workflows/query-embedding-for-search'
export * from './tools/generate-image'
export * from './tools/generate-video'
export * from './tools/rename-asset'
@@ -38,6 +38,9 @@ describe('Agent Embedding Workflow', () => {
saveAssetEmbeddingsActivity: Object.assign(vi.fn(), {
_activityName: 'saveAssetEmbeddingsActivity',
}),
deleteAssetEmbeddingsActivity: Object.assign(vi.fn(), {
_activityName: 'deleteAssetEmbeddingsActivity',
}),
updateTaskUsageActivity: Object.assign(vi.fn(), { _activityName: 'updateTaskUsageActivity' }),
createCommentActivity: Object.assign(vi.fn(), { _activityName: 'createCommentActivity' }),
updateCommentActivity: Object.assign(vi.fn(), { _activityName: 'updateCommentActivity' }),
@@ -314,18 +317,21 @@ describe('Agent Embedding Workflow', () => {
assetKey: 'test-transcoded.mp4',
startTime: 0,
endTime: 60,
format: 'video',
})
expect(mockActivities.transcodeVideoChunkActivity).toHaveBeenNthCalledWith(2, {
assetId: 'a1',
assetKey: 'test-transcoded.mp4',
startTime: 55,
endTime: 115,
format: 'video',
})
expect(mockActivities.transcodeVideoChunkActivity).toHaveBeenNthCalledWith(3, {
assetId: 'a1',
assetKey: 'test-transcoded.mp4',
startTime: 110,
endTime: 150,
format: 'video',
})
// Verify video chunk embedding generation (3 chunks) on agent queue
@@ -357,14 +363,19 @@ describe('Agent Embedding Workflow', () => {
model: 'gemini-embedding-2',
})
// Verify embeddings saved
expect(mockActivities.saveAssetEmbeddingsActivity).toHaveBeenCalledWith({
// Verify embeddings saved per chunk immediately
expect(mockActivities.saveAssetEmbeddingsActivity).toHaveBeenCalledTimes(3)
expect(mockActivities.saveAssetEmbeddingsActivity).toHaveBeenNthCalledWith(1, {
assetId: 'a1',
embeddings: [
{ embedding: [0.3, 0.4], startTime: 0, endTime: 60 },
{ embedding: [0.3, 0.4], startTime: 55, endTime: 115 },
{ embedding: [0.3, 0.4], startTime: 110, endTime: 150 },
],
embeddings: [{ embedding: [0.3, 0.4], startTime: 0, endTime: 60 }],
})
expect(mockActivities.saveAssetEmbeddingsActivity).toHaveBeenNthCalledWith(2, {
assetId: 'a1',
embeddings: [{ embedding: [0.3, 0.4], startTime: 55, endTime: 115 }],
})
expect(mockActivities.saveAssetEmbeddingsActivity).toHaveBeenNthCalledWith(3, {
assetId: 'a1',
embeddings: [{ embedding: [0.3, 0.4], startTime: 110, endTime: 150 }],
})
})
@@ -450,6 +461,49 @@ describe('Agent Embedding Workflow', () => {
})
})
it('should save completed chunks to DB immediately before subsequent chunk failure', async () => {
mockActivities.getEmbeddingContextActivity.mockResolvedValue({
agent: { id: 'b1' },
asset: {
id: 'a1',
mediaType: 'video/mp4',
storageKey: { key: 'test.mp4' },
media: {
duration: 150.0,
videoTranscodes: [{ key: 'test-transcoded.mp4' }],
},
},
chunkDuration: 60.0,
chunkOverlap: 5.0,
})
// Chunk 1 succeeds, Chunk 2 fails
mockActivities.generateVideoChunkEmbeddingActivity
.mockResolvedValueOnce({
embedding: [0.1, 0.2],
usage: { inputTokens: 3, outputTokens: 3, model: 'gemini-embedding-2' },
})
.mockRejectedValueOnce(new Error('Rate limit exceeded at chunk 2'))
const task = await prisma.workflowTask.create({
data: {
type: 'ai_embedding',
status: 'pending',
assetId: 'a1',
teamId: 't1',
},
})
await expect(agentEmbeddingMedia(task)).rejects.toThrow('Rate limit exceeded at chunk 2')
// Chunk 1 MUST have been saved to the database before chunk 2 failed
expect(mockActivities.saveAssetEmbeddingsActivity).toHaveBeenCalledTimes(1)
expect(mockActivities.saveAssetEmbeddingsActivity).toHaveBeenCalledWith({
assetId: 'a1',
embeddings: [{ embedding: [0.1, 0.2], startTime: 0, endTime: 60 }],
})
})
it('should not create redundant chunks when video exactly matches chunk duration', async () => {
mockActivities.getEmbeddingContextActivity.mockResolvedValue({
agent: { id: 'b1' },
@@ -485,4 +539,137 @@ describe('Agent Embedding Workflow', () => {
}),
)
})
it('should run agent embedding workflow for video with gemma-embedding-2-openai requesting frames format', async () => {
mockActivities.getEmbeddingContextActivity.mockResolvedValue({
agent: { id: 'b1' },
asset: {
id: 'a1',
mediaType: 'video/mp4',
storageKey: { key: 'test.mp4' },
media: {
duration: 30.0,
},
},
chunkDuration: 60.0,
chunkOverlap: 5.0,
embeddingProvider: 'gemma-embedding-2-openai',
})
const task = await prisma.workflowTask.create({
data: {
type: 'ai_embedding',
status: 'pending',
assetId: 'a1',
teamId: 't1',
},
})
await agentEmbeddingMedia(task)
expect(mockActivities.transcodeVideoChunkActivity).toHaveBeenCalledWith(
expect.objectContaining({
startTime: 0,
endTime: 30,
format: 'frames',
}),
)
expect(mockActivities.updateTaskUsageActivity).toHaveBeenCalledWith(
expect.objectContaining({
model: 'gemma-embedding-2-openai',
}),
)
})
it('should skip already completed chunks when resuming a video embedding task', async () => {
mockActivities.getEmbeddingContextActivity.mockResolvedValue({
agent: { id: 'b1' },
asset: {
id: 'a1',
mediaType: 'video/mp4',
storageKey: { key: 'test.mp4' },
media: {
duration: 150.0,
videoTranscodes: [{ key: 'test-transcoded.mp4' }],
},
},
chunkDuration: 60.0,
chunkOverlap: 5.0,
existingEmbeddings: [
{
id: 'emb-1',
startTime: 0,
endTime: 60,
has1536: true,
has768: false,
},
],
})
const task = await prisma.workflowTask.create({
data: {
type: 'ai_embedding',
status: 'pending',
assetId: 'a1',
teamId: 't1',
},
})
await agentEmbeddingMedia(task)
// Chunk 1 (0-60) should be skipped; only chunk 2 (55-115) and chunk 3 (110-150) should be transcoded and embedded
expect(mockActivities.transcodeVideoChunkActivity).toHaveBeenCalledTimes(2)
expect(mockActivities.transcodeVideoChunkActivity).toHaveBeenNthCalledWith(
1,
expect.objectContaining({ startTime: 55, endTime: 115 }),
)
expect(mockActivities.transcodeVideoChunkActivity).toHaveBeenNthCalledWith(
2,
expect.objectContaining({ startTime: 110, endTime: 150 }),
)
expect(mockActivities.saveAssetEmbeddingsActivity).toHaveBeenCalledTimes(2)
})
it('should purge incompatible embeddings before processing', async () => {
mockActivities.getEmbeddingContextActivity.mockResolvedValue({
agent: { id: 'b1' },
asset: {
id: 'a1',
mediaType: 'image/png',
storageKey: { key: 'test.png' },
},
embeddingProvider: 'gemma-embedding-2-openai',
existingEmbeddings: [
{
id: 'old-gemini-emb',
startTime: null,
endTime: null,
has1536: true,
has768: false,
},
],
})
const task = await prisma.workflowTask.create({
data: {
type: 'ai_embedding',
status: 'pending',
assetId: 'a1',
teamId: 't1',
},
})
await agentEmbeddingMedia(task)
// Should purge incompatible old-gemini-emb
expect(mockActivities.deleteAssetEmbeddingsActivity).toHaveBeenCalledWith({
assetId: 'a1',
ids: ['old-gemini-emb'],
})
// Should generate and save new embedding
expect(mockActivities.generateImageEmbeddingActivity).toHaveBeenCalled()
expect(mockActivities.saveAssetEmbeddingsActivity).toHaveBeenCalled()
})
})
+87 -28
View File
@@ -13,6 +13,14 @@ export interface GeneratedEmbedding {
endTime?: number
}
export interface ExistingAssetEmbeddingInfo {
id: string
startTime: number | null
endTime: number | null
has1536: boolean
has768: boolean
}
export async function agentEmbeddingMedia(task: WorkflowTask): Promise<void> {
const {
updateTaskStatusActivity,
@@ -20,6 +28,7 @@ export async function agentEmbeddingMedia(task: WorkflowTask): Promise<void> {
generateImageEmbeddingActivity,
generateVideoChunkEmbeddingActivity,
saveAssetEmbeddingsActivity,
deleteAssetEmbeddingsActivity,
updateTaskUsageActivity,
createCommentActivity,
updateCommentActivity,
@@ -62,13 +71,40 @@ export async function agentEmbeddingMedia(task: WorkflowTask): Promise<void> {
}
// 1. Fetch Agent Context
const context = await executeActivity(agentWorkerQueue, getEmbeddingContextActivity, {
const context = (await executeActivity(agentWorkerQueue, getEmbeddingContextActivity, {
teamId: task.teamId,
assetId: task.assetId,
})
})) as unknown as {
asset: {
id: string
mediaType: string | null
storageKey?: { key: string } | null
media?: unknown
}
chunkDuration: number
chunkOverlap: number
embeddingProvider: string
existingEmbeddings?: ExistingAssetEmbeddingInfo[]
}
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const { asset, chunkDuration, chunkOverlap } = context as any
const { asset, chunkDuration, chunkOverlap, embeddingProvider, existingEmbeddings } = context
const isGemmaProvider = embeddingProvider === 'gemma-embedding-2-openai'
const existing = existingEmbeddings || []
// Purge incompatible embeddings (e.g. 1536d when Gemma is used, or 768d when Gemini is used)
const incompatibleIds = existing
.filter((e) => (isGemmaProvider ? e.has1536 || !e.has768 : e.has768 || !e.has1536))
.map((e) => e.id)
if (incompatibleIds.length > 0) {
await executeActivity(agentWorkerQueue, deleteAssetEmbeddingsActivity, {
assetId: task.assetId,
ids: incompatibleIds,
})
}
const validExistingChunks = existing.filter((e) => !incompatibleIds.includes(e.id))
const mediaInfo = asset.media as PrismaJson.MediaInfo | null
const is3dWithViews =
@@ -113,23 +149,30 @@ export async function agentEmbeddingMedia(task: WorkflowTask): Promise<void> {
throw ApplicationFailure.create({ message: 'asset has no key', nonRetryable: true })
}
const results: GeneratedEmbedding[] = []
const usage = {
model: 'gemini-embedding-2',
model: isGemmaProvider ? 'gemma-embedding-2-openai' : 'gemini-embedding-2',
inputTokens: 0,
outputTokens: 0,
}
if (isImage) {
const result = await executeActivity(agentWorkerQueue, generateImageEmbeddingActivity, {
teamId: task.teamId,
assetKey: embeddingKey,
mediaType: embeddingMediaType,
})
results.push({ embedding: result.embedding })
if (result.usage) {
usage.inputTokens += result.usage.inputTokens || 0
usage.outputTokens += result.usage.outputTokens || 0
const alreadyEmbedded = validExistingChunks.some(
(e) => e.startTime === null && e.endTime === null,
)
if (!alreadyEmbedded) {
const result = await executeActivity(agentWorkerQueue, generateImageEmbeddingActivity, {
teamId: task.teamId,
assetKey: embeddingKey,
mediaType: embeddingMediaType,
})
await executeActivity(agentWorkerQueue, saveAssetEmbeddingsActivity, {
assetId: task.assetId,
embeddings: [{ embedding: result.embedding }],
})
if (result.usage) {
usage.inputTokens += result.usage.inputTokens || 0
usage.outputTokens += result.usage.outputTokens || 0
}
}
} else if (isVideo) {
// Get video duration
@@ -145,12 +188,30 @@ export async function agentEmbeddingMedia(task: WorkflowTask): Promise<void> {
end = duration
}
// Check if this chunk is already completed
const chunkExists = validExistingChunks.some(
(c) =>
c.startTime !== null &&
c.endTime !== null &&
Math.abs(c.startTime - start) < 0.1 &&
Math.abs(c.endTime - end) < 0.1,
)
if (chunkExists) {
// If this chunk reached the end of the video, stop
if (end >= duration) {
break
}
continue
}
// Step A: Slice chunk on transcode worker and upload to S3
const chunkRes = await executeActivity(transcodeWorkerQueue, transcodeVideoChunkActivity, {
assetId: asset.id,
assetKey: embeddingKey,
startTime: start,
endTime: end,
format: isGemmaProvider ? 'frames' : 'video',
})
const { chunkKey } = chunkRes
@@ -165,10 +226,16 @@ export async function agentEmbeddingMedia(task: WorkflowTask): Promise<void> {
},
)
results.push({
embedding: chunkResult.embedding,
startTime: start,
endTime: end,
// Step C: Save chunk embedding to database immediately
await executeActivity(agentWorkerQueue, saveAssetEmbeddingsActivity, {
assetId: task.assetId,
embeddings: [
{
embedding: chunkResult.embedding,
startTime: start,
endTime: end,
},
],
})
if (chunkResult.usage) {
@@ -176,7 +243,7 @@ export async function agentEmbeddingMedia(task: WorkflowTask): Promise<void> {
usage.outputTokens += chunkResult.usage.outputTokens || 0
}
} finally {
// Step C: Delete temporary chunk from S3
// Step D: Delete temporary chunk from S3
await executeActivity(transcodeWorkerQueue, deleteS3ObjectActivity, {
key: chunkKey,
})
@@ -189,14 +256,6 @@ export async function agentEmbeddingMedia(task: WorkflowTask): Promise<void> {
}
}
// Save computed embeddings
if (results.length > 0) {
await executeActivity(agentWorkerQueue, saveAssetEmbeddingsActivity, {
assetId: task.assetId,
embeddings: results,
})
}
// Update Usage
await executeActivity(agentWorkerQueue, updateTaskUsageActivity, {
taskId: task.id,
@@ -1,145 +0,0 @@
import { describe, it, expect, vi, beforeEach } from 'vitest'
import { queryEmbeddingForSearch } from './query-embedding-for-search'
import { prisma } from '@shumai/db'
import { setupTestDbHooks } from '@shumai/db/test'
import * as workflowUtils from '@shumai/workflow-core'
vi.mock('@shumai/workflow-core', async () => {
const actual = await vi.importActual('@shumai/workflow-core')
return {
...actual,
getActivities: vi.fn(),
executeActivity: vi.fn(),
}
})
describe('Query Embedding For Search Workflow', () => {
setupTestDbHooks()
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- mockActivities holds mock functions cast to expected types
let mockActivities: any
beforeEach(() => {
vi.clearAllMocks()
mockActivities = {
updateWorkflowTaskActivity: Object.assign(vi.fn(), {
_activityName: 'updateWorkflowTaskActivity',
}),
generateTextEmbeddingActivity: Object.assign(vi.fn(), {
_activityName: 'generateTextEmbeddingActivity',
}),
getAgentWorkerQueueActivity: Object.assign(vi.fn(), {
_activityName: 'getAgentWorkerQueueActivity',
}),
}
mockActivities.getAgentWorkerQueueActivity.mockResolvedValue('agent_queue')
mockActivities.updateWorkflowTaskActivity.mockResolvedValue({})
mockActivities.generateTextEmbeddingActivity.mockResolvedValue({
embedding: [0.1, 0.2, 0.3],
usage: { inputTokens: 4, outputTokens: 0, model: 'gemini-embedding-2' },
})
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- mockActivities contains vi.fn mock functions which are cast to expected activity proxy types
vi.mocked(workflowUtils.getActivities).mockReturnValue(mockActivities as any)
vi.mocked(workflowUtils.executeActivity).mockImplementation(async (_queue, act, ...args) => {
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- act is one of the mocked activities
return (act as any)(...args)
})
})
it('should run query embedding workflow successfully', async () => {
const task = await prisma.workflowTask.create({
data: {
type: 'query_embedding_for_search',
status: 'pending',
teamId: 't1',
assetId: 'a1',
payload: {
projectId: 'p1',
queryEmbeddingForSearch: {
text: 'search query',
},
},
},
})
await queryEmbeddingForSearch(task)
// Verify queue discovery
expect(mockActivities.getAgentWorkerQueueActivity).toHaveBeenCalled()
// Verify task processing status
expect(mockActivities.updateWorkflowTaskActivity).toHaveBeenCalledWith({
taskId: task.id,
status: 'processing',
heartbeat: true,
})
// Verify embedding generation
expect(mockActivities.generateTextEmbeddingActivity).toHaveBeenCalledWith({
text: 'search query',
teamId: 't1',
})
// Verify completed status and details
expect(mockActivities.updateWorkflowTaskActivity).toHaveBeenCalledWith({
taskId: task.id,
status: 'completed',
output: { embedding: [0.1, 0.2, 0.3] },
inputTokens: 4,
outputTokens: 0,
model: 'gemini-embedding-2',
})
})
it('should throw immediately if text is missing', async () => {
const task = await prisma.workflowTask.create({
data: {
type: 'query_embedding_for_search',
status: 'pending',
teamId: 't1',
assetId: 'a1',
},
})
await expect(queryEmbeddingForSearch(task)).rejects.toThrow(
'Missing text in query_embedding_for_search task payload',
)
// Should not call any activities because it throws before queue discovery
expect(mockActivities.getAgentWorkerQueueActivity).not.toHaveBeenCalled()
expect(mockActivities.updateWorkflowTaskActivity).not.toHaveBeenCalled()
})
it('should handle activity failure and update task status to failed', async () => {
mockActivities.generateTextEmbeddingActivity.mockRejectedValue(
new Error('Embedding API failed'),
)
const task = await prisma.workflowTask.create({
data: {
type: 'query_embedding_for_search',
status: 'pending',
teamId: 't1',
assetId: 'a1',
payload: {
projectId: 'p1',
queryEmbeddingForSearch: {
text: 'search query',
},
},
},
})
await expect(queryEmbeddingForSearch(task)).rejects.toThrow('Embedding API failed')
// Verify it updates task to failed
expect(mockActivities.updateWorkflowTaskActivity).toHaveBeenCalledWith({
taskId: task.id,
status: 'failed',
output: { error: 'Embedding API failed' },
})
})
})
@@ -1,56 +0,0 @@
import type { WorkflowTask } from '@shumai/db'
import { getActivities, executeActivity, TaskQueueAgent } from '@shumai/workflow-core'
export async function queryEmbeddingForSearch(task: WorkflowTask): Promise<void> {
const { generateTextEmbeddingActivity, updateWorkflowTaskActivity, getAgentWorkerQueueActivity } =
getActivities()
let agentWorkerQueue = ''
const payload = task.payload as PrismaJson.WorkflowTaskPayload | null
const text = payload?.queryEmbeddingForSearch?.text
if (!text) {
throw new Error('Missing text in query_embedding_for_search task payload')
}
try {
// 0. Discover queue
agentWorkerQueue = await executeActivity(TaskQueueAgent, getAgentWorkerQueueActivity)
// 1. Mark as processing
await executeActivity(agentWorkerQueue, updateWorkflowTaskActivity, {
taskId: task.id,
status: 'processing',
heartbeat: true,
})
// 2. Generate embedding
const result = await executeActivity(agentWorkerQueue, generateTextEmbeddingActivity, {
text,
teamId: task.teamId!,
})
// 3. Save output and complete
await executeActivity(agentWorkerQueue, updateWorkflowTaskActivity, {
taskId: task.id,
status: 'completed',
output: { embedding: result.embedding },
inputTokens: result.usage.inputTokens,
outputTokens: result.usage.outputTokens,
model: result.usage.model,
})
} catch (err) {
console.error(`queryEmbeddingForSearch failed for task ${task.id}:`, err)
if (agentWorkerQueue) {
await executeActivity(agentWorkerQueue, updateWorkflowTaskActivity, {
taskId: task.id,
status: 'failed',
output: {
error: err instanceof Error ? err.message : String(err),
},
})
}
throw err
}
}
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/api",
"version": "0.5.2",
"version": "0.5.3",
"private": true,
"type": "module",
"main": "./src/index.ts",
+199 -1
View File
@@ -1,4 +1,4 @@
import { describe, it, expect, vi, beforeEach } from 'vitest'
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'
import { HTTPException } from 'hono/http-exception'
import { agentService } from '@shumai/core/src/agent/agent'
import { authzService } from '@shumai/core/src/authz/authz'
@@ -210,6 +210,30 @@ describe('Agent API', () => {
itemId: 'agent1',
})
})
it('returns 403 when updating embedding agent and ENABLE_EMBEDDING_AGENT is true', async () => {
process.env.ENABLE_EMBEDDING_AGENT = 'true'
try {
vi.mocked(agentService.getAgent).mockResolvedValueOnce({
id: 'agent1',
teamId: 'team1',
type: 'embedding',
} as unknown as Awaited<ReturnType<typeof agentService.getAgent>>)
const res = await app.request('/agents/agent1', {
method: 'PUT',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
name: 'Updated Embedding Agent',
type: 'embedding',
}),
})
expect(res.status).toBe(403)
} finally {
delete process.env.ENABLE_EMBEDDING_AGENT
}
})
})
describe('DELETE /agents/:agentId', () => {
@@ -227,6 +251,25 @@ describe('Agent API', () => {
itemId: 'agent1',
})
})
it('returns 403 when deleting embedding agent and ENABLE_EMBEDDING_AGENT is true', async () => {
process.env.ENABLE_EMBEDDING_AGENT = 'true'
try {
vi.mocked(agentService.getAgent).mockResolvedValueOnce({
id: 'agent1',
teamId: 'team1',
type: 'embedding',
} as unknown as Awaited<ReturnType<typeof agentService.getAgent>>)
const res = await app.request('/agents/agent1', {
method: 'DELETE',
})
expect(res.status).toBe(403)
} finally {
delete process.env.ENABLE_EMBEDDING_AGENT
}
})
})
describe('PATCH /agents/:agentId/permission', () => {
@@ -271,6 +314,27 @@ describe('Agent API', () => {
})
})
it('returns 403 when modifying embedding agent permission and ENABLE_EMBEDDING_AGENT is true', async () => {
process.env.ENABLE_EMBEDDING_AGENT = 'true'
try {
vi.mocked(agentService.getAgent).mockResolvedValueOnce({
id: 'agent1',
teamId: 'team1',
type: 'embedding',
} as unknown as Awaited<ReturnType<typeof agentService.getAgent>>)
const res = await app.request('/agents/agent1/permission', {
method: 'PATCH',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ permission: 'owner' }),
})
expect(res.status).toBe(403)
} finally {
delete process.env.ENABLE_EMBEDDING_AGENT
}
})
it('denies access when user is not admin', async () => {
vi.mocked(authzService.hasPermission).mockRejectedValue(
new HTTPException(403, { message: 'Forbidden' }),
@@ -410,4 +474,138 @@ describe('Agent API', () => {
expect(agentService.listProjectChatAgents).not.toHaveBeenCalled()
})
})
describe('read-only embedding agent when ENABLE_EMBEDDING_AGENT is true', () => {
beforeEach(() => {
process.env.ENABLE_EMBEDDING_AGENT = 'true'
vi.mocked(agentService.getAgent).mockResolvedValue({
id: 'emb1',
teamId: 'team1',
type: 'embedding',
} as unknown as Awaited<ReturnType<typeof agentService.getAgent>>)
})
afterEach(() => {
delete process.env.ENABLE_EMBEDDING_AGENT
})
it('rejects PUT /agents/:agentId with 403', async () => {
const res = await app.request('/agents/emb1', {
method: 'PUT',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
name: 'Updated Embedding',
type: 'embedding',
enabled: false,
}),
})
expect(res.status).toBe(403)
expect(agentService.updateAgent).not.toHaveBeenCalled()
})
it('rejects DELETE /agents/:agentId with 403', async () => {
const res = await app.request('/agents/emb1', {
method: 'DELETE',
headers: { 'Content-Type': 'application/json' },
})
expect(res.status).toBe(403)
expect(agentService.deleteAgent).not.toHaveBeenCalled()
})
it('rejects PATCH /agents/:agentId/permission with 403', async () => {
const res = await app.request('/agents/emb1/permission', {
method: 'PATCH',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
permission: 'owner',
}),
})
expect(res.status).toBe(403)
expect(agentService.updateAgentPermission).not.toHaveBeenCalled()
})
it('rejects POST /teams/:teamId/agents with 403 for embedding agent', async () => {
const res = await app.request('/teams/team1/agents', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
name: 'Manual Embedding',
type: 'embedding',
}),
})
expect(res.status).toBe(403)
expect(agentService.createAgent).not.toHaveBeenCalled()
})
})
describe('read-only embedding agent when ENABLE_EMBEDDING_AGENT is false', () => {
beforeEach(() => {
process.env.ENABLE_EMBEDDING_AGENT = 'false'
vi.mocked(agentService.getAgent).mockResolvedValue({
id: 'emb1',
teamId: 'team1',
type: 'embedding',
} as unknown as Awaited<ReturnType<typeof agentService.getAgent>>)
})
afterEach(() => {
delete process.env.ENABLE_EMBEDDING_AGENT
})
it('rejects PUT /agents/:agentId with 403', async () => {
const res = await app.request('/agents/emb1', {
method: 'PUT',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
name: 'Updated Embedding',
type: 'embedding',
enabled: false,
}),
})
expect(res.status).toBe(403)
expect(agentService.updateAgent).not.toHaveBeenCalled()
})
it('rejects DELETE /agents/:agentId with 403', async () => {
const res = await app.request('/agents/emb1', {
method: 'DELETE',
headers: { 'Content-Type': 'application/json' },
})
expect(res.status).toBe(403)
expect(agentService.deleteAgent).not.toHaveBeenCalled()
})
it('rejects PATCH /agents/:agentId/permission with 403', async () => {
const res = await app.request('/agents/emb1/permission', {
method: 'PATCH',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
permission: 'owner',
}),
})
expect(res.status).toBe(403)
expect(agentService.updateAgentPermission).not.toHaveBeenCalled()
})
it('rejects POST /teams/:teamId/agents with 403 for embedding agent', async () => {
const res = await app.request('/teams/team1/agents', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
name: 'Manual Embedding',
type: 'embedding',
}),
})
expect(res.status).toBe(403)
expect(agentService.createAgent).not.toHaveBeenCalled()
})
})
})
+42
View File
@@ -1,4 +1,5 @@
import { Hono } from 'hono'
import { HTTPException } from 'hono/http-exception'
import { zValidator } from '@hono/zod-validator'
import { agentService } from '@shumai/core/src/agent/agent'
import { authzService, Permission, ResourceType } from '@shumai/core/src/authz/authz'
@@ -62,6 +63,16 @@ const route = new Hono<{ Variables: { user: User } }>()
id: teamId,
})
const isEmbeddingManagedByEnv =
process.env.ENABLE_EMBEDDING_AGENT === 'true' ||
process.env.ENABLE_EMBEDDING_AGENT === 'false'
if (req.type === 'embedding' && isEmbeddingManagedByEnv) {
throw new HTTPException(403, {
message: 'Embedding agent is managed by environment configuration and cannot be created',
})
}
const agent = await agentService.createAgent({
teamId,
...req,
@@ -92,6 +103,17 @@ const route = new Hono<{ Variables: { user: User } }>()
id: agentId,
})
const isEmbeddingManagedByEnv =
process.env.ENABLE_EMBEDDING_AGENT === 'true' ||
process.env.ENABLE_EMBEDDING_AGENT === 'false'
const existingAgent = await agentService.getAgent({ agentId })
if (existingAgent?.type === 'embedding' && isEmbeddingManagedByEnv) {
throw new HTTPException(403, {
message: 'Embedding agent is managed by environment configuration and cannot be modified',
})
}
const agent = await agentService.updateAgent({
agentId,
...req,
@@ -123,6 +145,17 @@ const route = new Hono<{ Variables: { user: User } }>()
id: agentId,
})
const isEmbeddingManagedByEnv =
process.env.ENABLE_EMBEDDING_AGENT === 'true' ||
process.env.ENABLE_EMBEDDING_AGENT === 'false'
const existingAgent = await agentService.getAgent({ agentId })
if (existingAgent?.type === 'embedding' && isEmbeddingManagedByEnv) {
throw new HTTPException(403, {
message: 'Embedding agent is managed by environment configuration and cannot be modified',
})
}
const agent = await agentService.updateAgentPermission(agentId, req.permission)
await auditLogService.logAction({
@@ -148,7 +181,16 @@ const route = new Hono<{ Variables: { user: User } }>()
id: agentId,
})
const isEmbeddingManagedByEnv =
process.env.ENABLE_EMBEDDING_AGENT === 'true' ||
process.env.ENABLE_EMBEDDING_AGENT === 'false'
const existingAgent = await agentService.getAgent({ agentId })
if (existingAgent?.type === 'embedding' && isEmbeddingManagedByEnv) {
throw new HTTPException(403, {
message: 'Embedding agent is managed by environment configuration and cannot be deleted',
})
}
await agentService.deleteAgent({
agentId,
+2 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/core",
"version": "0.5.2",
"version": "0.5.3",
"private": true,
"type": "module",
"main": "./src/index.ts",
@@ -29,6 +29,7 @@
"@aws-sdk/s3-request-presigner": "3.1121.0",
"@earendil-works/pi-agent-core": "0.82.1",
"@earendil-works/pi-ai": "0.82.1",
"@google/genai": "2.28.0",
"@modelcontextprotocol/client": "2.0.0",
"@shumai/db": "workspace:*",
"@shumai/dtos": "workspace:*",
+167
View File
@@ -976,4 +976,171 @@ describe('AgentService', () => {
expect(ids).toHaveLength(0)
})
})
describe('ensureEmbeddingAgent', () => {
test('creates embedding agent when ENABLE_EMBEDDING_AGENT is true', async () => {
const db = prisma
const svc = new AgentService()
const { team } = await setupTestData(db)
process.env.ENABLE_EMBEDDING_AGENT = 'true'
try {
await svc.ensureEmbeddingAgentForTeam(team.id)
const agent = await db.agent.findFirst({
where: { teamId: team.id, type: 'embedding' },
include: { user: true, skills: { include: { skill: true } }, mcpServers: true },
})
expect(agent).toBeDefined()
const info = await svc.toAgentInfo(agent!)
expect(info.name).toBe('embedding-agent')
expect(info.type).toBe('embedding')
expect(info.enabled).toBe(true)
expect(info.avatarPreset).toBe('avatar-1')
expect(info.avatar).toContain('avatar-1.webp')
expect(info.readOnly).toBe(true)
// Ensure idempotency
await svc.ensureEmbeddingAgentForTeam(team.id)
const count = await db.agent.count({
where: { teamId: team.id, type: 'embedding' },
})
expect(count).toBe(1)
} finally {
delete process.env.ENABLE_EMBEDDING_AGENT
}
})
test('enables existing embedding agent when ENABLE_EMBEDDING_AGENT is true', async () => {
const db = prisma
const svc = new AgentService()
const { team } = await setupTestData(db)
const bot = await db.user.create({
data: { name: 'Existing Embedding', email: 'existing-emb@shumai.ai', type: 'agent' },
})
await db.agent.create({
data: {
id: bot.id,
teamId: team.id,
type: 'embedding',
enabled: false,
config: { provider: '', model: '' },
},
})
process.env.ENABLE_EMBEDDING_AGENT = 'true'
try {
await svc.ensureEmbeddingAgentForTeam(team.id)
const agent = await db.agent.findFirst({
where: { teamId: team.id, type: 'embedding' },
include: { user: true, skills: { include: { skill: true } }, mcpServers: true },
})
expect(agent?.id).toBe(bot.id)
expect(agent?.enabled).toBe(true)
const info = await svc.toAgentInfo(agent!)
expect(info.readOnly).toBe(true)
} finally {
delete process.env.ENABLE_EMBEDDING_AGENT
}
})
test('does nothing when ENABLE_EMBEDDING_AGENT is not set', async () => {
const db = prisma
const svc = new AgentService()
const team = await db.team.create({ data: { name: 'Team No Agent' } })
delete process.env.ENABLE_EMBEDDING_AGENT
await svc.ensureEmbeddingAgentForTeam(team.id)
const agent = await db.agent.findFirst({
where: { teamId: team.id, type: 'embedding' },
})
expect(agent).toBeNull()
})
test('does not create embedding agent when ENABLE_EMBEDDING_AGENT is false and agent does not exist', async () => {
const db = prisma
const svc = new AgentService()
const { team } = await setupTestData(db)
process.env.ENABLE_EMBEDDING_AGENT = 'false'
try {
await svc.ensureEmbeddingAgentForTeam(team.id)
const agent = await db.agent.findFirst({
where: { teamId: team.id, type: 'embedding' },
})
expect(agent).toBeNull()
} finally {
delete process.env.ENABLE_EMBEDDING_AGENT
}
})
test('disables existing embedding agent and marks readOnly: true when ENABLE_EMBEDDING_AGENT is false', async () => {
const db = prisma
const svc = new AgentService()
const { team } = await setupTestData(db)
const bot = await db.user.create({
data: { name: 'Existing Embedding', email: 'existing-emb-false@shumai.ai', type: 'agent' },
})
await db.agent.create({
data: {
id: bot.id,
teamId: team.id,
type: 'embedding',
enabled: true,
config: { provider: '', model: '' },
},
})
process.env.ENABLE_EMBEDDING_AGENT = 'false'
try {
await svc.ensureEmbeddingAgentForTeam(team.id)
const agent = await db.agent.findFirst({
where: { teamId: team.id, type: 'embedding' },
include: { user: true, skills: { include: { skill: true } }, mcpServers: true },
})
expect(agent?.id).toBe(bot.id)
expect(agent?.enabled).toBe(false)
const info = await svc.toAgentInfo(agent!)
expect(info.readOnly).toBe(true)
expect(info.enabled).toBe(false)
} finally {
delete process.env.ENABLE_EMBEDDING_AGENT
}
})
test('ensureSystemEmbeddingAgents updates all teams when ENABLE_EMBEDDING_AGENT is false', async () => {
const db = prisma
const svc = new AgentService()
const { team: team1 } = await setupTestData(db)
const team2 = await db.team.create({ data: { name: 'Team 2' } })
// Create enabled embedding agent on team1
const bot1 = await db.user.create({
data: { name: 'Bot 1', email: 'b1-false@shumai.ai', type: 'agent' },
})
await db.agent.create({
data: {
id: bot1.id,
teamId: team1.id,
type: 'embedding',
enabled: true,
config: { provider: '', model: '' },
},
})
process.env.ENABLE_EMBEDDING_AGENT = 'false'
try {
await svc.ensureSystemEmbeddingAgents()
const a1 = await db.agent.findFirst({ where: { teamId: team1.id, type: 'embedding' } })
const a2 = await db.agent.findFirst({ where: { teamId: team2.id, type: 'embedding' } })
expect(a1?.enabled).toBe(false)
expect(a2).toBeNull()
} finally {
delete process.env.ENABLE_EMBEDDING_AGENT
}
})
})
})
+63 -3
View File
@@ -43,8 +43,8 @@ interface AgentInfoSource {
config: unknown
soul: string | null
user: { name: string; image: string | null }
skills: Array<{ id: string; skillId: string; skill: { id: string; name: string } }>
mcpServers: Array<{ mcpServerId: string }>
skills?: Array<{ id: string; skillId: string; skill: { id: string; name: string } }>
mcpServers?: Array<{ mcpServerId: string }>
}
function extractS3Key(urlOrKey?: string | null): string | null | undefined {
@@ -462,6 +462,11 @@ export class AgentService {
*/
async toAgentInfo(agent: AgentInfoSource): Promise<AgentInfo> {
const config = agent.config as unknown as PrismaJson.AgentConfig
const isManagedByEnv =
process.env.ENABLE_EMBEDDING_AGENT === 'true' ||
process.env.ENABLE_EMBEDDING_AGENT === 'false'
const isReadOnly = agent.type === 'embedding' && isManagedByEnv
return {
id: agent.id,
name: agent.user.name,
@@ -475,13 +480,14 @@ export class AgentService {
thinkingLevel: config.thinkingLevel || 'off',
systemPrompt: config.systemPrompt,
soul: agent.soul || undefined,
skills: agent.skills.map((s) => ({
skills: (agent.skills || []).map((s) => ({
id: s.id,
skillId: s.skillId,
skill: s.skill,
})),
mcpServerIds: (agent.mcpServers || []).map((m) => m.mcpServerId),
deniedTools: config.deniedTools || [],
readOnly: isReadOnly,
}
}
@@ -718,6 +724,60 @@ export class AgentService {
params,
)
}
async ensureSystemEmbeddingAgents(): Promise<void> {
const envVal = process.env.ENABLE_EMBEDDING_AGENT
if (envVal !== 'true' && envVal !== 'false') {
return
}
const teams = await this.prismaClient.team.findMany({
select: { id: true },
})
for (const team of teams) {
await this.ensureEmbeddingAgentForTeam(team.id)
}
}
async ensureEmbeddingAgentForTeam(teamId: string): Promise<void> {
const envVal = process.env.ENABLE_EMBEDDING_AGENT
if (envVal !== 'true' && envVal !== 'false') {
return
}
const existing = await this.prismaClient.agent.findFirst({
where: {
teamId,
type: 'embedding',
},
})
if (envVal === 'true') {
if (!existing) {
await this.createAgent({
teamId,
name: 'embedding-agent',
type: 'embedding',
avatar: 'avatar-1',
thinkingLevel: 'off',
enabled: true,
})
} else if (!existing.enabled) {
await this.prismaClient.agent.update({
where: { id: existing.id },
data: { enabled: true },
})
}
} else if (envVal === 'false') {
if (existing && existing.enabled) {
await this.prismaClient.agent.update({
where: { id: existing.id },
data: { enabled: false },
})
}
}
}
}
export const agentService = new AgentService()
+2
View File
@@ -10,6 +10,8 @@ export * from './project/project'
export * from './provider/provider'
export * from './s3/s3'
export * from './search/search'
export * from './search/text-embedding'
export * from './search/embedding-service'
export * from './share/share'
export * from './skill/skill'
export * from './team/team'
@@ -0,0 +1,256 @@
import { describe, expect, it } from 'vitest'
import { prisma, AssetType, WorkflowTaskStatus, WorkflowTaskType } from '@shumai/db'
import { setupTestDbHooks } from '@shumai/db/test'
import { agentService } from '@shumai/core/src/agent/agent'
import { embeddingService } from './embedding-service'
describe('EmbeddingService', () => {
setupTestDbHooks()
it('should skip teams without an enabled embedding agent', async () => {
const team = await prisma.team.create({
data: { name: 'Team Without Agent' },
})
const project = await prisma.project.create({
data: { name: 'Project 1', teamId: team.id },
})
await prisma.asset.create({
data: {
name: 'video.mp4',
type: AssetType.file,
status: 'processed',
mediaType: 'video/mp4',
projectId: project.id,
},
})
const result = await embeddingService.regenerateEmbeddings()
expect(result.tasksCreated).toBe(0)
expect(result.totalAssets).toBe(0)
expect(result.skippedTeams).toContain('Team Without Agent')
})
it('should skip teams where embedding agent exists but is disabled', async () => {
const team = await prisma.team.create({
data: { name: 'Team Disabled Agent' },
})
await agentService.createAgent({
teamId: team.id,
name: 'embedding-agent',
type: 'embedding',
avatar: 'avatar-1',
thinkingLevel: 'off',
enabled: false,
})
const project = await prisma.project.create({
data: { name: 'Project Disabled', teamId: team.id },
})
await prisma.asset.create({
data: {
name: 'photo.jpg',
type: AssetType.file,
status: 'processed',
mediaType: 'image/jpeg',
projectId: project.id,
},
})
const result = await embeddingService.regenerateEmbeddings()
expect(result.tasksCreated).toBe(0)
expect(result.totalAssets).toBe(0)
expect(result.skippedTeams).toContain('Team Disabled Agent')
})
it('should regenerate embeddings for eligible media assets and clean up old data', async () => {
const team = await prisma.team.create({
data: { name: 'Active Embedding Team' },
})
const agent = await agentService.createAgent({
teamId: team.id,
name: 'embedding-agent',
type: 'embedding',
avatar: 'avatar-1',
thinkingLevel: 'off',
enabled: true,
})
const project = await prisma.project.create({
data: { name: 'Active Project', teamId: team.id },
})
// 1. Eligible video asset
const videoAsset = await prisma.asset.create({
data: {
name: 'sample.mp4',
type: AssetType.file,
status: 'processed',
mediaType: 'video/mp4',
projectId: project.id,
},
})
// 2. Eligible image asset
const imageAsset = await prisma.asset.create({
data: {
name: 'sample.png',
type: AssetType.file,
status: 'processed',
mediaType: 'image/png',
projectId: project.id,
},
})
// 3. Eligible 3D asset with rendered views
const asset3d = await prisma.asset.create({
data: {
name: 'model.glb',
type: AssetType.file,
status: 'processed',
mediaType: 'model/gltf-binary',
media: {
proxyType: '3d',
imageTranscodes: [
{ key: 'files/3d/view.webp', format: 'webp', width: 100, height: 100, quality: 80 },
],
} as unknown as PrismaJson.MediaInfo,
projectId: project.id,
},
})
// 4. Ineligible asset (non-media document)
await prisma.asset.create({
data: {
name: 'document.pdf',
type: AssetType.file,
status: 'processed',
mediaType: 'application/pdf',
projectId: project.id,
},
})
// 5. Ineligible asset (trashed / deleted)
await prisma.asset.create({
data: {
name: 'deleted-video.mp4',
type: AssetType.file,
status: 'processed',
isDeleted: true,
mediaType: 'video/mp4',
projectId: project.id,
},
})
// 6. Ineligible asset (status not processed)
await prisma.asset.create({
data: {
name: 'uploading.mp4',
type: AssetType.file,
status: 'uploading',
mediaType: 'video/mp4',
projectId: project.id,
},
})
// Create preexisting embeddings and workflow task for videoAsset to verify cleanup
const dummyVec768 = `[${new Array(768).fill(0.1).join(',')}]`
await prisma.$executeRaw`
INSERT INTO asset_embeddings (id, asset_id, embedding_768, updated_at)
VALUES (gen_random_uuid()::text, ${videoAsset.id}, ${dummyVec768}::vector, NOW())
`
await prisma.workflowTask.create({
data: {
assetId: videoAsset.id,
type: WorkflowTaskType.ai_embedding,
status: WorkflowTaskStatus.processing,
teamId: team.id,
projectId: project.id,
},
})
const initialEmbeddings = await prisma.assetEmbedding.count({
where: { assetId: videoAsset.id },
})
expect(initialEmbeddings).toBe(1)
// Run regeneration
const result = await embeddingService.regenerateEmbeddings()
expect(result.totalAssets).toBe(3)
expect(result.tasksCreated).toBe(3)
expect(result.skippedTeams).not.toContain('Active Embedding Team')
// Verify old embeddings were wiped upfront
const remainingEmbeddings = await prisma.assetEmbedding.count({
where: { assetId: videoAsset.id },
})
expect(remainingEmbeddings).toBe(0)
// Verify tasks created
const createdTasks = await prisma.workflowTask.findMany({
where: {
type: WorkflowTaskType.ai_embedding,
assetId: { in: [videoAsset.id, imageAsset.id, asset3d.id] },
},
})
expect(createdTasks).toHaveLength(3)
for (const task of createdTasks) {
expect(task.status).toBe(WorkflowTaskStatus.pending)
expect(task.teamId).toBe(team.id)
expect(task.projectId).toBe(project.id)
const payload = task.payload as { agent?: { agentId?: string } }
expect(payload?.agent?.agentId).toBe(agent!.id)
}
})
it('should handle multiple teams where only some have active embedding agents', async () => {
// Team 1: has active agent
const team1 = await prisma.team.create({ data: { name: 'Team One' } })
await agentService.createAgent({
teamId: team1.id,
name: 'embedding-agent',
type: 'embedding',
avatar: 'avatar-1',
thinkingLevel: 'off',
enabled: true,
})
const project1 = await prisma.project.create({ data: { name: 'P1', teamId: team1.id } })
await prisma.asset.create({
data: {
name: 'v1.mp4',
type: AssetType.file,
status: 'processed',
mediaType: 'video/mp4',
projectId: project1.id,
},
})
// Team 2: no active agent
const team2 = await prisma.team.create({ data: { name: 'Team Two' } })
const project2 = await prisma.project.create({ data: { name: 'P2', teamId: team2.id } })
await prisma.asset.create({
data: {
name: 'v2.mp4',
type: AssetType.file,
status: 'processed',
mediaType: 'video/mp4',
projectId: project2.id,
},
})
const result = await embeddingService.regenerateEmbeddings()
expect(result.totalAssets).toBe(1)
expect(result.tasksCreated).toBe(1)
expect(result.skippedTeams).toContain('Team Two')
expect(result.skippedTeams).not.toContain('Team One')
const tasks = await prisma.workflowTask.findMany({
where: {
type: WorkflowTaskType.ai_embedding,
teamId: { in: [team1.id, team2.id] },
},
})
expect(tasks).toHaveLength(1)
expect(tasks[0].teamId).toBe(team1.id)
})
})
@@ -0,0 +1,135 @@
import { prisma, WorkflowTaskStatus, WorkflowTaskType } from '@shumai/db'
import { logger } from '@shumai/core/src/logger'
export interface RegenerateEmbeddingsResult {
tasksCreated: number
totalAssets: number
skippedTeams: string[]
}
export class EmbeddingService {
/**
* Regenerates vector embeddings across all teams.
*
* Skips teams that do not have an active (enabled) embedding agent and logs a warning.
* For teams with an active embedding agent, purges existing embeddings and pending/processing
* embedding tasks upfront, then creates fresh ai_embedding workflow tasks for all eligible assets.
*/
async regenerateEmbeddings(): Promise<RegenerateEmbeddingsResult> {
const teams = await prisma.team.findMany({
select: { id: true, name: true },
})
const activeAgents = await prisma.agent.findMany({
where: {
type: 'embedding',
enabled: true,
},
select: { id: true, teamId: true },
})
const activeAgentByTeamId = new Map(
activeAgents
.filter((a): a is typeof a & { teamId: string } => typeof a.teamId === 'string')
.map((a) => [a.teamId, a]),
)
const activeTeamIds: string[] = []
const skippedTeams: string[] = []
for (const team of teams) {
if (activeAgentByTeamId.has(team.id)) {
activeTeamIds.push(team.id)
} else {
skippedTeams.push(team.name || team.id)
logger.warn(
{ teamId: team.id, teamName: team.name },
'Skipping embedding regeneration for team: no active embedding agent found',
)
}
}
if (activeTeamIds.length === 0) {
logger.info('No teams found with an active embedding agent. 0 tasks created.')
return { tasksCreated: 0, totalAssets: 0, skippedTeams }
}
const candidateAssets = await prisma.asset.findMany({
where: {
isDeleted: false,
status: 'processed',
type: { in: ['file', 'version_stack'] },
project: { teamId: { in: activeTeamIds } },
},
include: { project: true },
})
const eligibleAssets = candidateAssets.filter((asset) => {
const isVideo = asset.mediaType?.startsWith('video/')
const mediaInfo = asset.media as PrismaJson.MediaInfo | null
const isImage =
asset.mediaType?.startsWith('image/') ||
(mediaInfo?.proxyType === '3d' && (mediaInfo.imageTranscodes?.length ?? 0) > 0)
return isVideo || isImage
})
const eligibleAssetIds = eligibleAssets.map((a) => a.id)
if (eligibleAssetIds.length > 0) {
// Clean up previous embedding tasks to avoid duplicate execution
await prisma.workflowTask.deleteMany({
where: {
assetId: { in: eligibleAssetIds },
type: WorkflowTaskType.ai_embedding,
},
})
// Purge existing embeddings so everything regenerates cleanly with the configured provider
await prisma.assetEmbedding.deleteMany({
where: {
assetId: { in: eligibleAssetIds },
},
})
}
let tasksCreated = 0
for (const asset of eligibleAssets) {
const teamId = asset.project?.teamId
if (!teamId) continue
const agent = activeAgentByTeamId.get(teamId)
if (!agent) continue
await prisma.workflowTask.create({
data: {
assetId: asset.id,
type: WorkflowTaskType.ai_embedding,
status: WorkflowTaskStatus.pending,
teamId,
projectId: asset.projectId,
payload: {
projectId: asset.projectId ?? '',
agent: { agentId: agent.id },
},
},
})
tasksCreated++
}
logger.info(
{
totalAssets: eligibleAssets.length,
tasksCreated,
skippedTeamsCount: skippedTeams.length,
},
'Successfully queued embedding regeneration tasks',
)
return {
tasksCreated,
totalAssets: eligibleAssets.length,
skippedTeams,
}
}
}
export const embeddingService = new EmbeddingService()
+248 -13
View File
@@ -4,7 +4,7 @@ import { setupTestDbHooks } from '@shumai/db/test'
import { SearchService } from './search'
import { SearchConditionOperator } from '@shumai/dtos'
import { metadataService } from '@shumai/core/src/metadata/metadata'
import { workflowService } from '@shumai/workflow-core'
import { textEmbeddingService } from './text-embedding'
import { AssetType } from '@shumai/db'
import { assetService } from '@shumai/core/src/asset/asset'
@@ -805,20 +805,14 @@ describe('SearchService', () => {
)
`
// Mock the embedding generation workflow task.
// Mock the embedding generation
// Our query vector points along the first dimension [1, 0, 0...]
const queryVector = Array(1536).fill(0)
queryVector[0] = 1.0
const mockExecuteWait = vi
.spyOn(workflowService, 'executeWait')
.mockImplementation(async (task) => {
return {
...task,
status: 'completed',
output: {
embedding: queryVector,
},
} as unknown as typeof task
const mockGenerateEmbedding = vi
.spyOn(textEmbeddingService, 'generateEmbedding')
.mockResolvedValue({
embedding: queryVector,
})
// Search using semantic mode
@@ -855,7 +849,248 @@ describe('SearchService', () => {
expect(result.data[2].startTime).toBe(10)
expect(result.data[2].endTime).toBe(20)
mockExecuteWait.mockRestore()
mockGenerateEmbedding.mockRestore()
})
it('matches and sorts 768-dimensional embeddings using embedding_768 column', async () => {
const user = await prisma.user.create({
data: { name: 'semantic-user-768', email: 'semantic-user-768@example.com', type: 'human' },
})
const team = await prisma.team.create({
data: { name: 'semantic-team-768' },
})
const project = await prisma.project.create({
data: { name: 'semantic-project-768', teamId: team.id },
})
const botUser = await prisma.user.create({
data: { name: 'Embedding Bot 768', email: 'emb-768@example.com', type: 'agent' },
})
const provider = await prisma.provider.create({
data: {
name: 'gemma',
teamId: team.id,
config: { api: 'openai', apiKey: 'key' } as unknown as PrismaJson.ProviderConfig,
},
})
const model = await prisma.model.create({
data: {
modelId: 'embeddinggemma-2',
name: 'EmbeddingGemma 2',
providerId: provider.id,
config: {
reasoning: false,
input: ['text'],
contextWindow: 1000,
maxTokens: 1000,
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
} as unknown as PrismaJson.ModelConfig,
},
})
await prisma.agent.create({
data: {
id: botUser.id,
teamId: team.id,
type: 'embedding',
enabled: true,
providerId: provider.id,
modelId: model.id,
config: { provider: 'gemma-embedding-2-openai', model: 'embeddinggemma-2' },
},
})
const rootFolder = await prisma.asset.create({
data: {
name: 'root-768',
type: AssetType.folder,
projectId: project.id,
creatorId: user.id,
status: 'uploaded',
},
})
const videoAssetA = await prisma.asset.create({
data: {
name: 'VideoA_768.mp4',
type: AssetType.file,
mediaType: 'video/mp4',
projectId: project.id,
parentId: rootFolder.id,
creatorId: user.id,
status: 'uploaded',
},
})
const videoAssetB = await prisma.asset.create({
data: {
name: 'VideoB_768.mp4',
type: AssetType.file,
mediaType: 'video/mp4',
projectId: project.id,
parentId: rootFolder.id,
creatorId: user.id,
status: 'uploaded',
},
})
const arrA1 = Array(768).fill(0)
arrA1[0] = 1.0
const vectorA1 = JSON.stringify(arrA1)
const arrB1 = Array(768).fill(0)
arrB1[0] = 1.0
arrB1[1] = 1.0
const vectorB1 = JSON.stringify(arrB1)
await prisma.$executeRaw`
INSERT INTO asset_embeddings (id, asset_id, embedding_768, start_time, end_time, created_at, updated_at)
VALUES (
'chunk-768-a1',
${videoAssetA.id},
${vectorA1}::vector,
0.0,
10.0,
NOW(),
NOW()
)
`
await prisma.$executeRaw`
INSERT INTO asset_embeddings (id, asset_id, embedding_768, start_time, end_time, created_at, updated_at)
VALUES (
'chunk-768-b1',
${videoAssetB.id},
${vectorB1}::vector,
5.0,
15.0,
NOW(),
NOW()
)
`
// Query vector with 768 dims pointing along dimension 0
const queryVector = Array(768).fill(0)
queryVector[0] = 1.0
const mockGenerateEmbedding = vi
.spyOn(textEmbeddingService, 'generateEmbedding')
.mockResolvedValue({
embedding: queryVector,
})
const result = await searchService.search(rootFolder.id, {
recursively: true,
assetType: 'file',
operator: 'AND',
conditions: [],
query: 'find 768 dim match',
isSemantic: true,
})
expect(result.data.length).toBeGreaterThanOrEqual(2)
expect(result.data[0].id).toBe(videoAssetA.id)
expect(result.data[1].id).toBe(videoAssetB.id)
mockGenerateEmbedding.mockRestore()
})
it('throws 502 if embedding generation fails during semantic search', async () => {
const user = await prisma.user.create({
data: { name: 'semantic-user-fail', email: 'semantic-fail@example.com', type: 'human' },
})
const team = await prisma.team.create({
data: { name: 'semantic-team-fail' },
})
const project = await prisma.project.create({
data: { name: 'semantic-project-fail', teamId: team.id },
})
const botUser = await prisma.user.create({
data: { name: 'Embedding Bot Fail', email: 'emb-fail@example.com', type: 'agent' },
})
await prisma.agent.create({
data: {
id: botUser.id,
teamId: team.id,
type: 'embedding',
enabled: true,
config: { provider: 'gemini-embedding-2', model: 'gemini-embedding-2' },
},
})
const rootFolder = await prisma.asset.create({
data: {
name: 'root-fail',
type: AssetType.folder,
projectId: project.id,
creatorId: user.id,
status: 'uploaded',
},
})
const mockGenerateEmbedding = vi
.spyOn(textEmbeddingService, 'generateEmbedding')
.mockRejectedValue(new Error('Rate limit exceeded'))
await expect(
searchService.search(rootFolder.id, {
recursively: true,
assetType: 'file',
operator: 'AND',
conditions: [],
query: 'fail query',
isSemantic: true,
}),
).rejects.toThrow('Failed to generate query embedding: Rate limit exceeded')
mockGenerateEmbedding.mockRestore()
})
it('rejects semantic search when ENABLE_EMBEDDING_AGENT is false', async () => {
const user = await prisma.user.create({
data: { name: 'u-emb-false', email: 'u-emb-false@example.com', type: 'human' },
})
const team = await prisma.team.create({ data: { name: 't-emb-false' } })
const project = await prisma.project.create({
data: { name: 'p-emb-false', teamId: team.id },
})
const rootFolder = await prisma.asset.create({
data: {
name: 'root-false',
type: AssetType.folder,
projectId: project.id,
creatorId: user.id,
status: 'uploaded',
},
})
// Even if an embedding agent is enabled in db
const bot = await prisma.user.create({
data: { name: 'Bot', email: 'b-emb-false@example.com', password: 'pw', type: 'agent' },
})
await prisma.agent.create({
data: {
id: bot.id,
teamId: team.id,
type: 'embedding',
enabled: true,
config: { provider: '', model: '' },
},
})
process.env.ENABLE_EMBEDDING_AGENT = 'false'
try {
await expect(
searchService.search(rootFolder.id, {
recursively: true,
assetType: 'file',
operator: 'AND',
conditions: [],
query: 'test query',
isSemantic: true,
}),
).rejects.toThrow('Embedding agent not configured or disabled for this team.')
} finally {
delete process.env.ENABLE_EMBEDDING_AGENT
}
})
})
+36 -21
View File
@@ -1,18 +1,19 @@
import { prisma } from '@shumai/db'
import { Prisma, AssetType, WorkflowTaskType } from '@shumai/db'
import { Prisma, AssetType } from '@shumai/db'
import { AssetService, assetService } from '@shumai/core/src/asset/asset'
import { AssetInfo } from '@shumai/dtos'
import { SearchRequest } from '@shumai/dtos'
import { PaginatedData, decodeCursor, encodeCursor, PageInfo } from '@shumai/core/src/pagination'
import { generateSearchNgrams } from '@shumai/core/src/utils/ngram'
import { workflowService } from '@shumai/workflow-core'
import { HTTPException } from 'hono/http-exception'
import { SqlQueryBuilder } from './sql-query-builder'
import { TextEmbeddingService, textEmbeddingService } from './text-embedding'
export class SearchService {
constructor(
private readonly prismaClient: typeof prisma = prisma,
private readonly assetSvc: AssetService = assetService,
private readonly embeddingSvc: TextEmbeddingService = textEmbeddingService,
) {}
async search(folderId: string, req: SearchRequest): Promise<PaginatedData<AssetInfo[]>> {
@@ -34,6 +35,12 @@ export class SearchService {
const teamId = team?.project?.teamId
if (!teamId) throw new Error('Team ID not found for folder')
if (process.env.ENABLE_EMBEDDING_AGENT === 'false') {
throw new HTTPException(422, {
message: 'Embedding agent not configured or disabled for this team.',
})
}
const embeddingAgent = await this.prismaClient.agent.findFirst({
where: { teamId, type: 'embedding', enabled: true },
})
@@ -44,36 +51,43 @@ export class SearchService {
})
}
// 2. Generate query embedding via workflow
const task = await this.prismaClient.workflowTask.create({
data: {
type: WorkflowTaskType.query_embedding_for_search,
// 2. Generate query embedding directly using configured environment provider
let queryVector: number[]
try {
const result = await this.embeddingSvc.generateEmbedding(req.query, {
teamId,
assetId: folderId,
payload: {
projectId: '',
queryEmbeddingForSearch: { text: req.query },
} as PrismaJson.WorkflowTaskPayload,
status: 'pending',
},
})
})
queryVector = result.embedding
} catch (err) {
const msg = err instanceof Error ? err.message : String(err)
throw new HTTPException(502, {
message: `Failed to generate query embedding: ${msg}`,
})
}
const completedTask = await workflowService.executeWait(task)
const output = completedTask.output as Record<string, unknown> | null
const queryVector = output?.embedding as number[] | undefined
if (!queryVector) {
throw new Error('Failed to generate query embedding')
if (!queryVector || queryVector.length === 0) {
throw new HTTPException(502, {
message: 'Failed to generate query embedding: empty vector returned',
})
}
// 3. Construct raw SQL using SqlQueryBuilder
const vectorJson = JSON.stringify(queryVector)
const is768 = queryVector.length === 768
const distanceExpr = is768
? Prisma.sql`(ae.embedding_768 <=> ${vectorJson}::vector)`
: Prisma.sql`(ae.embedding <=> ${vectorJson}::vector)`
const notNullCond = is768
? Prisma.sql`ae.embedding_768 IS NOT NULL`
: Prisma.sql`ae.embedding IS NOT NULL`
const builder = new SqlQueryBuilder()
.select(
Prisma.sql`a.id as "assetId", ae.start_time as "startTime", ae.end_time as "endTime", (ae.embedding <=> ${vectorJson}::vector) as "distance"`,
Prisma.sql`a.id as "assetId", ae.start_time as "startTime", ae.end_time as "endTime", ${distanceExpr} as "distance"`,
)
.from(Prisma.sql`assets a JOIN asset_embeddings ae ON a.id = ae.asset_id`)
.addWhere(Prisma.sql`a.is_deleted = false`)
.addWhere(notNullCond)
if (targetFolderIds.length > 0) {
builder.addWhere(Prisma.sql`a.parent_id = ANY(${targetFolderIds})`)
@@ -158,6 +172,7 @@ export class SearchService {
.select(Prisma.sql`COUNT(*)`)
.from(Prisma.sql`assets a JOIN asset_embeddings ae ON a.id = ae.asset_id`)
.addWhere(Prisma.sql`a.is_deleted = false`)
.addWhere(notNullCond)
if (targetFolderIds.length > 0) {
countBuilder.addWhere(Prisma.sql`a.parent_id = ANY(${targetFolderIds})`)
@@ -0,0 +1,236 @@
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'
import { TextEmbeddingService } from './text-embedding'
import { logger } from '@shumai/core/src/logger'
const mockEmbedContent = vi.fn()
vi.mock('@google/genai', () => {
return {
// eslint-disable-next-line @typescript-eslint/naming-convention
GoogleGenAI: class {
models = {
embedContent: mockEmbedContent,
}
},
}
})
describe('TextEmbeddingService', () => {
const originalEnv = { ...process.env }
let service: TextEmbeddingService
beforeEach(() => {
process.env = { ...originalEnv }
process.env.GEMINI_API_KEY = 'test-gemini-key'
delete process.env.EMBEDDING_PROVIDER
delete process.env.GEMMA_EMBEDDING_OPENAI_BASE_URL
mockEmbedContent.mockReset()
service = new TextEmbeddingService(500, 3600 * 1000)
})
afterEach(() => {
process.env = originalEnv
vi.restoreAllMocks()
})
it('throws an error if text is empty or whitespace', async () => {
await expect(service.generateEmbedding('')).rejects.toThrow('Text to embed must not be empty')
await expect(service.generateEmbedding(' ')).rejects.toThrow(
'Text to embed must not be empty',
)
})
it('generates Gemini embedding (1536d) by default', async () => {
const dummyEmbedding = Array(1536).fill(0.05)
mockEmbedContent.mockResolvedValueOnce({
embeddings: [{ values: dummyEmbedding }],
})
const result = await service.generateEmbedding('beautiful sunset')
expect(result.embedding).toEqual(dummyEmbedding)
expect(result.usage?.model).toBe('gemini-embedding-2')
expect(result.cached).toBeUndefined()
expect(mockEmbedContent).toHaveBeenCalledWith(
expect.objectContaining({
model: 'gemini-embedding-2',
contents: [
expect.objectContaining({
parts: [{ text: 'beautiful sunset' }],
}),
],
config: {
outputDimensionality: 1536,
},
}),
)
})
it('returns cached result on repeated call without re-invoking the provider', async () => {
const dummyEmbedding = Array(1536).fill(0.05)
mockEmbedContent.mockResolvedValueOnce({
embeddings: [{ values: dummyEmbedding }],
})
const first = await service.generateEmbedding('ocean breeze')
expect(first.embedding).toEqual(dummyEmbedding)
expect(mockEmbedContent).toHaveBeenCalledTimes(1)
// Second call with same text and extra whitespace
const second = await service.generateEmbedding(' ocean breeze ')
expect(second.embedding).toEqual(dummyEmbedding)
expect(second.cached).toBe(true)
expect(mockEmbedContent).toHaveBeenCalledTimes(1)
})
it('preserves case distinctions in cache keys', async () => {
const dummyApple = Array(1536).fill(0.1)
const dummyLowerApple = Array(1536).fill(0.2)
mockEmbedContent
.mockResolvedValueOnce({
embeddings: [{ values: dummyApple }],
})
.mockResolvedValueOnce({
embeddings: [{ values: dummyLowerApple }],
})
const upper = await service.generateEmbedding('Apple')
expect(upper.embedding).toEqual(dummyApple)
expect(mockEmbedContent).toHaveBeenCalledTimes(1)
const lower = await service.generateEmbedding('apple')
expect(lower.embedding).toEqual(dummyLowerApple)
expect(lower.cached).toBeUndefined()
expect(mockEmbedContent).toHaveBeenCalledTimes(2)
})
it('does not log raw query text on cache hits', async () => {
const dummyEmbedding = Array(1536).fill(0.05)
mockEmbedContent.mockResolvedValue({
embeddings: [{ values: dummyEmbedding }],
})
const debugSpy = vi.spyOn(logger, 'debug')
await service.generateEmbedding('confidential search query')
await service.generateEmbedding('confidential search query')
expect(debugSpy).toHaveBeenCalledWith(
expect.not.objectContaining({
key: expect.anything(),
text: expect.anything(),
}),
'Cache hit for text embedding',
)
})
it('clears cache when clearCache is called', async () => {
const dummyEmbedding = Array(1536).fill(0.05)
mockEmbedContent.mockResolvedValue({
embeddings: [{ values: dummyEmbedding }],
})
await service.generateEmbedding('query')
expect(mockEmbedContent).toHaveBeenCalledTimes(1)
service.clearCache()
await service.generateEmbedding('query')
expect(mockEmbedContent).toHaveBeenCalledTimes(2)
})
it('throws if GEMINI_API_KEY is not configured', async () => {
delete process.env.GEMINI_API_KEY
await expect(service.generateEmbedding('test')).rejects.toThrow(
'GEMINI_API_KEY environment variable is not configured',
)
})
it('throws if Gemini returns empty embedding values', async () => {
mockEmbedContent.mockResolvedValueOnce({
embeddings: [{ values: null }],
})
await expect(service.generateEmbedding('test')).rejects.toThrow(
'No embedding values returned from gemini-embedding-2',
)
})
it('generates Gemma embedding (768d) when provider is gemma-embedding-2-openai', async () => {
const dummyEmbedding = Array(768).fill(0.12)
const mockFetch = vi.fn().mockResolvedValue({
ok: true,
json: async () => ({
model: 'embeddinggemma-2',
data: [{ embedding: dummyEmbedding }],
// eslint-disable-next-line @typescript-eslint/naming-convention
usage: { prompt_tokens: 12 },
}),
})
vi.stubGlobal('fetch', mockFetch)
const result = await service.generateEmbedding('drone footage', {
provider: 'gemma-embedding-2-openai',
})
expect(result.embedding).toEqual(dummyEmbedding)
expect(result.usage?.model).toBe('embeddinggemma-2')
expect(result.usage?.inputTokens).toBe(12)
expect(mockFetch).toHaveBeenCalledWith(
'http://localhost:9379/v1/embeddings',
expect.objectContaining({
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
model: 'embeddinggemma-2',
input: 'task: search query | text: drone footage',
}),
}),
)
})
it('respects GEMMA_EMBEDDING_OPENAI_BASE_URL without /v1 suffix', async () => {
process.env.GEMMA_EMBEDDING_OPENAI_BASE_URL = 'http://custom-host:9379'
const dummyEmbedding = Array(768).fill(0.12)
const mockFetch = vi.fn().mockResolvedValue({
ok: true,
json: async () => ({
data: [{ embedding: dummyEmbedding }],
}),
})
vi.stubGlobal('fetch', mockFetch)
await service.generateEmbedding('test query', {
provider: 'gemma-embedding-2-openai',
})
expect(mockFetch).toHaveBeenCalledWith(
'http://custom-host:9379/v1/embeddings',
expect.anything(),
)
})
it('throws descriptive error when Gemma server returns non-ok status', async () => {
const mockFetch = vi.fn().mockResolvedValue({
ok: false,
status: 503,
text: async () => 'Service Unavailable',
})
vi.stubGlobal('fetch', mockFetch)
await expect(
service.generateEmbedding('test query', {
provider: 'gemma-embedding-2-openai',
}),
).rejects.toThrow('Embedding server returned status 503: Service Unavailable')
})
it('throws error when abort signal is triggered', async () => {
const controller = new AbortController()
controller.abort(new Error('Operation cancelled by user'))
await expect(
service.generateEmbedding('test query', {
signal: controller.signal,
}),
).rejects.toThrow('Operation cancelled by user')
})
})
+196
View File
@@ -0,0 +1,196 @@
import { GoogleGenAI } from '@google/genai'
import { LruTtlCache } from '@shumai/core/src/cache/lru-ttl-cache'
import { logger } from '@shumai/core/src/logger'
export interface TextEmbeddingUsage {
model: string
inputTokens: number
outputTokens: number
}
export interface TextEmbeddingResult {
embedding: number[]
usage?: TextEmbeddingUsage
cached?: boolean
}
export interface GenerateTextEmbeddingOptions {
provider?: string
model?: string
signal?: AbortSignal
teamId?: string
skipCache?: boolean
}
export class TextEmbeddingService {
private cache: LruTtlCache<string, number[]>
constructor(
cacheCapacity: number = 500,
private defaultTtlMs: number = 3600 * 1000,
) {
this.cache = new LruTtlCache<string, number[]>(cacheCapacity, 50)
}
/**
* Generates a text embedding vector directly using the configured provider.
* If a cached embedding exists for the text, provider, and model, it is returned immediately.
*/
async generateEmbedding(
text: string,
options?: GenerateTextEmbeddingOptions,
): Promise<TextEmbeddingResult> {
const trimmedText = text.trim()
if (!trimmedText) {
throw new Error('Text to embed must not be empty')
}
const provider = options?.provider || process.env.EMBEDDING_PROVIDER || 'gemini-embedding-2'
const model =
options?.model ||
(provider === 'gemma-embedding-2-openai' ? 'embeddinggemma-2' : 'gemini-embedding-2')
const cacheKey = `${provider}:${model}:${trimmedText}`
if (!options?.skipCache) {
const cachedVector = this.cache.get(cacheKey)
if (cachedVector) {
logger.debug({ provider, model }, 'Cache hit for text embedding')
return {
embedding: cachedVector,
usage: { model, inputTokens: 0, outputTokens: 0 },
cached: true,
}
}
}
const signal = options?.signal ?? AbortSignal.timeout(15000)
let result: TextEmbeddingResult
if (provider === 'gemma-embedding-2-openai') {
result = await this.callGemmaEmbeddingServer(trimmedText, model, signal)
} else {
result = await this.callGeminiEmbedding(trimmedText, model, signal)
}
if (!options?.skipCache) {
this.cache.set(cacheKey, result.embedding, this.defaultTtlMs)
}
return result
}
private async callGeminiEmbedding(
text: string,
model: string,
signal: AbortSignal,
): Promise<TextEmbeddingResult> {
const apiKey = process.env.GEMINI_API_KEY
if (!apiKey) {
throw new Error('GEMINI_API_KEY environment variable is not configured')
}
if (signal.aborted) {
throw signal.reason ?? new Error('Embedding generation aborted or timed out')
}
const resolvedApiKey = process.env[apiKey] || apiKey
const ai = new GoogleGenAI({ apiKey: resolvedApiKey })
const embedPromise = ai.models.embedContent({
model,
contents: [
{
parts: [{ text }],
role: 'user',
},
],
config: {
outputDimensionality: 1536,
},
})
const abortPromise = new Promise<never>((_, reject) => {
if (signal.aborted) {
reject(signal.reason ?? new Error('Embedding generation aborted or timed out'))
return
}
signal.addEventListener(
'abort',
() => {
reject(signal.reason ?? new Error('Embedding generation aborted or timed out'))
},
{ once: true },
)
})
const response = await Promise.race([embedPromise, abortPromise])
const values = response.embeddings?.[0]?.values
if (!values || !Array.isArray(values)) {
throw new Error(`No embedding values returned from ${model}`)
}
return {
embedding: values,
usage: {
model,
inputTokens: 0,
outputTokens: 0,
},
}
}
private async callGemmaEmbeddingServer(
text: string,
model: string,
signal: AbortSignal,
): Promise<TextEmbeddingResult> {
const rawBaseUrl = process.env.GEMMA_EMBEDDING_OPENAI_BASE_URL || 'http://localhost:9379/v1'
const normalized = rawBaseUrl.replace(/\/+$/, '')
const endpoint = normalized.endsWith('/embeddings')
? normalized
: normalized.endsWith('/v1')
? `${normalized}/embeddings`
: `${normalized}/v1/embeddings`
const response = await fetch(endpoint, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
model,
input: `task: search query | text: ${text}`,
}),
signal,
})
if (!response.ok) {
const errorText = await response.text()
throw new Error(`Embedding server returned status ${response.status}: ${errorText}`)
}
const json = (await response.json()) as {
model?: string
data?: Array<{ embedding?: number[] }>
// eslint-disable-next-line @typescript-eslint/naming-convention
usage?: { prompt_tokens?: number }
}
const embedding = json.data?.[0]?.embedding
if (!embedding || !Array.isArray(embedding)) {
throw new Error('Invalid embedding returned from embedding server')
}
return {
embedding,
usage: {
model: json.model || model,
inputTokens: json.usage?.prompt_tokens ?? 0,
outputTokens: 0,
},
}
}
clearCache(): void {
this.cache.clear()
}
}
export const textEmbeddingService = new TextEmbeddingService()
+28
View File
@@ -605,4 +605,32 @@ describe('TeamService', () => {
expect(settings.appearance?.hideAgent).toBe(true)
expect((settings as Record<string, unknown>).emailNotification).toBeUndefined()
})
it('should report semanticSearchEnabled false when ENABLE_EMBEDDING_AGENT is false even if agent is enabled in db', async () => {
const owner = await prisma.user.create({
data: { name: 'Owner', email: 'owner-emb-false@example.com', password: 'pw' },
})
const team = await teamService.createTeam(owner, { name: 'Emb False Team' })
const bot = await prisma.user.create({
data: { name: 'Bot', email: 'bot-emb-false@example.com', password: 'pw', type: 'agent' },
})
await prisma.agent.create({
data: {
id: bot.id,
teamId: team.id,
type: 'embedding',
enabled: true,
config: { provider: '', model: '' },
},
})
process.env.ENABLE_EMBEDDING_AGENT = 'false'
try {
const settings = await teamService.getSettings(team.id)
expect(settings.semanticSearchEnabled).toBe(false)
} finally {
delete process.env.ENABLE_EMBEDDING_AGENT
}
})
})
+11 -4
View File
@@ -6,6 +6,7 @@ import { notificationService } from '@shumai/core/src/notification/notification'
import { getAvatarUrl } from '@shumai/core/src/user/avatar'
import { HTTPException } from 'hono/http-exception'
import { getAllowedAgentRoles } from '@shumai/core/src/agent/permissions'
import { AgentService, agentService } from '@shumai/core/src/agent/agent'
import { sandboxService } from '@shumai/core/src/sandbox/sandbox-service'
import { turntableService } from '@shumai/core/src/turntable/turntable'
import { gotenbergService } from '@shumai/core/src/gotenberg/gotenberg'
@@ -46,6 +47,7 @@ export class TeamService {
},
})
await providerService.initBuiltinProviders(team.id)
await agentService.ensureEmbeddingAgentForTeam(team.id)
}
return team
@@ -86,10 +88,12 @@ export class TeamService {
},
})
// We use 'any' here because ProviderService expects the full PrismaClient,
// We use 'any' here because ProviderService/AgentService expect the full PrismaClient,
// but the TransactionClient is compatible with the methods we use.
// eslint-disable-next-line @typescript-eslint/no-explicit-any
await new ProviderService(tx as any).initBuiltinProviders(team.id)
// eslint-disable-next-line @typescript-eslint/no-explicit-any
await new AgentService(tx as any).ensureEmbeddingAgentForTeam(team.id)
return this.toTeamInfo(updatedTeam)
})
@@ -307,9 +311,12 @@ export class TeamService {
delete rawSettings.emailNotification
const settings = rawSettings as TeamSettingsResponse
const embeddingAgent = await prisma.agent.findFirst({
where: { teamId, type: 'embedding', enabled: true },
})
const embeddingAgent =
process.env.ENABLE_EMBEDDING_AGENT === 'false'
? null
: await prisma.agent.findFirst({
where: { teamId, type: 'embedding', enabled: true },
})
settings.semanticSearchEnabled = !!embeddingAgent
if (!settings.transcode) {
-2
View File
@@ -501,7 +501,6 @@ describe('UploadService', () => {
where: { assetId: asset.id, type: WorkflowTaskType.transcode_pdf },
})
expect(workflowTask).not.toBeNull()
expect(workflowTask?.status).toBe(TaskStatus.pending)
// Test HTML document as well
const htmlTask = await prisma.task.create({
@@ -530,7 +529,6 @@ describe('UploadService', () => {
where: { assetId: htmlAsset.id, type: WorkflowTaskType.transcode_pdf },
})
expect(htmlWorkflowTask).not.toBeNull()
expect(htmlWorkflowTask?.status).toBe(TaskStatus.pending)
} finally {
if (originalEnv !== undefined) {
process.env.GOTENBERG_URL = originalEnv
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/db",
"version": "0.5.2",
"version": "0.5.3",
"private": true,
"type": "module",
"main": "./src/index.ts",
@@ -0,0 +1,8 @@
-- AlterTable
ALTER TABLE "asset_embeddings" ADD COLUMN "embedding_768" vector(768);
-- CreateIndex: HNSW index on asset_embeddings.embedding_768
CREATE INDEX "AssetEmbedding_embedding_768_idx"
ON "asset_embeddings"
USING hnsw (embedding_768 vector_cosine_ops)
WITH (m = 16, ef_construction = 64);
+10 -8
View File
@@ -706,17 +706,19 @@ model AssetCommentReaction {
}
model AssetEmbedding {
id String @id @default(ulid())
embedding Unsupported("vector(1536)")?
startTime Float? @map("start_time")
endTime Float? @map("end_time")
createdAt DateTime @default(now()) @map("created_at")
updatedAt DateTime @updatedAt @map("updated_at")
id String @id @default(ulid())
embedding Unsupported("vector(1536)")?
embedding768 Unsupported("vector(768)")? @map("embedding_768")
startTime Float? @map("start_time")
endTime Float? @map("end_time")
createdAt DateTime @default(now()) @map("created_at")
updatedAt DateTime @updatedAt @map("updated_at")
assetId String @map("asset_id")
asset Asset @relation("AssetToEmbedding", fields: [assetId], references: [id], onDelete: Cascade)
assetId String @map("asset_id")
asset Asset @relation("AssetToEmbedding", fields: [assetId], references: [id], onDelete: Cascade)
@@index([embedding], name: "AssetEmbedding_embedding_idx")
@@index([embedding768], name: "AssetEmbedding_embedding_768_idx")
@@map("asset_embeddings")
}
File diff suppressed because one or more lines are too long
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/dtos",
"version": "0.5.2",
"version": "0.5.3",
"private": true,
"type": "module",
"main": "./src/index.ts",
+1
View File
@@ -42,6 +42,7 @@ export const agentInfoSchema = z.object({
skills: z.array(agentSkillSchema).optional(),
mcpServerIds: z.array(z.string()).optional(),
deniedTools: z.array(z.string()).optional(),
readOnly: z.boolean().optional(),
})
export type AgentInfo = z.infer<typeof agentInfoSchema>
+2 -2
View File
@@ -1,11 +1,11 @@
{
"name": "@shumai/e2e",
"version": "0.5.2",
"version": "0.5.3",
"private": true,
"type": "module",
"dependencies": {
"@earendil-works/pi-agent-core": "0.82.1",
"@google/genai": "2.19.0",
"@google/genai": "2.28.0",
"@shumai/agent": "workspace:*",
"@shumai/core": "workspace:*",
"@shumai/db": "workspace:*",
@@ -1,127 +0,0 @@
import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest'
import { prisma } from '@shumai/db'
import { setupTestDbHooks } from '@shumai/db/test'
import { workflowService, TaskQueueAgent } from '@shumai/workflow-core'
import { initAgentWorkflows } from '@shumai/agent'
import { s3Service } from '@shumai/core/src/s3/s3'
const mockEmbedContent = vi.fn().mockResolvedValue({
embeddings: [
{
values: Array(1536).fill(0.35),
},
],
})
vi.mock('@google/genai', () => {
return {
// eslint-disable-next-line @typescript-eslint/naming-convention
GoogleGenAI: class {
models = {
embedContent: mockEmbedContent,
}
},
}
})
import { fileURLToPath } from 'url'
import * as path from 'path'
const currentDir = path.dirname(fileURLToPath(import.meta.url))
const agentWorkflowsPath = path.resolve(currentDir, '../../../apps/agent/src/workflows.ts')
describe.each(['local', 'temporal'] as const)(
'Workflow E2E - queryEmbeddingForSearch (executor: %s)',
(mode) => {
setupTestDbHooks()
let agentWorkerPromise: Promise<void> | null = null
beforeAll(async () => {
process.env.S3_BUCKET = 'shumai-e2e-test-bucket-query'
process.env.GEMINI_API_KEY = 'dummy-key'
workflowService.setExecutorType(mode)
initAgentWorkflows()
if (mode === 'temporal') {
console.log('Starting background worker for query embedding Temporal E2E tests...')
agentWorkerPromise = workflowService.startWorkers(TaskQueueAgent, {
workflowsPath: agentWorkflowsPath,
})
await new Promise((resolve) => setTimeout(resolve, 2000))
} else {
console.log('Starting local workflow service polling...')
workflowService.start()
}
})
afterAll(async () => {
if (mode === 'temporal') {
console.log('Shutting down Temporal workers...')
await workflowService.shutdownWorkers()
await Promise.all([agentWorkerPromise].filter(Boolean))
}
workflowService.close()
vi.restoreAllMocks()
try {
console.log('Cleaning up local E2E storage files...')
await s3Service.deletePrefix('shumai-e2e-test-bucket-query', '')
} catch (err) {
console.error('Failed to clean up E2E storage folder:', err)
}
})
it('should run queryEmbeddingForSearch workflow and return embedding successfully', async () => {
// 1. Seed Database
const team = await prisma.team.create({
data: { name: 'E2E Query Embedding Team' },
})
const project = await prisma.project.create({
data: { name: 'E2E Query Embedding Project', teamId: team.id },
})
const asset = await prisma.asset.create({
data: {
name: 'dummy-search-asset',
type: 'file',
status: 'uploaded',
mediaType: 'text/plain',
projectId: project.id,
},
})
// 2. Create Workflow Task
const task = await prisma.workflowTask.create({
data: {
type: 'query_embedding_for_search',
status: 'pending',
teamId: team.id,
projectId: project.id,
assetId: asset.id,
payload: {
projectId: project.id,
queryEmbeddingForSearch: {
text: 'test search query',
},
},
},
})
// 3. Wait for workflow to complete
console.log(
`Submitted E2E Query Embedding Workflow Task. ID: ${task.id}. Awaiting completion...`,
)
const completedTask = await workflowService.executeWait(task, 45000)
// 4. Verification
expect(completedTask.status).toBe('completed')
// Verify output payload contains correct embedding values and usage metadata
const output = completedTask.output as unknown as { embedding: number[] }
expect(output).toBeDefined()
expect(output.embedding).toEqual(Array(1536).fill(0.35))
expect(completedTask.model).toBe('gemini-embedding-2')
}, 50000)
},
)
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/tableprinter",
"version": "0.5.2",
"version": "0.5.3",
"private": true,
"type": "module",
"main": "./src/index.ts",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/transcode",
"version": "0.5.2",
"version": "0.5.3",
"private": true,
"type": "module",
"main": "./src/index.ts",
@@ -1230,6 +1230,100 @@ describe('Transcode Activities', () => {
expect(res.chunkKey).toMatch(/^files\/asset-abc\/tmp-embedding-chunks\/chunk-10-25-.*\.mp4$/)
})
it('should extract frames archive when format is frames', async () => {
// eslint-disable-next-line @typescript-eslint/no-explicit-any
;(child_process.execFile as any).mockImplementation(
(
_file: string,
args: string[],
cb: (err: Error | null, result: { stdout: string; stderr: string }) => void,
) => {
if (_file === 'ffmpeg' && args) {
const outPattern = args[args.length - 1]
if (outPattern && outPattern.includes('frame_%04d.jpg')) {
// eslint-disable-next-line @typescript-eslint/no-require-imports
const fsActual = require('fs')
// eslint-disable-next-line @typescript-eslint/no-require-imports
const pathActual = require('path')
const dir = pathActual.dirname(outPattern)
fsActual.writeFileSync(
pathActual.join(dir, 'frame_0001.jpg'),
Buffer.from('mock-frame'),
)
}
}
cb(null, { stdout: '', stderr: '' })
},
)
const res = await transcodeVideoChunkActivity({
assetId: 'asset-abc',
filePath: '/tmp/full-video.mp4',
startTime: 10,
endTime: 25,
format: 'frames',
})
expect(child_process.execFile).toHaveBeenCalledWith(
'ffmpeg',
expect.arrayContaining(['-vf', 'fps=1', '-q:v', '2']),
expect.any(Function),
)
expect(child_process.execFile).not.toHaveBeenCalledWith(
'tar',
expect.anything(),
expect.anything(),
)
expect(s3Service.putObject).toHaveBeenCalledWith(
'shumai',
expect.stringMatching(/^files\/asset-abc\/tmp-embedding-chunks\/frames-10-25-.*\.tar$/),
expect.any(Buffer),
expect.any(Number),
'application/x-tar',
)
expect(res.chunkKey).toMatch(/^files\/asset-abc\/tmp-embedding-chunks\/frames-10-25-.*\.tar$/)
const putCall = vi.mocked(s3Service.putObject).mock.calls[0]
const uploadedBuffer = putCall[2] as Buffer
const uploadedArchive = new Bun.Archive(uploadedBuffer)
const archiveFiles = await uploadedArchive.files()
expect(archiveFiles.has('frame_0001.jpg')).toBe(true)
const uploadedFile = archiveFiles.get('frame_0001.jpg')!
expect(uploadedFile.size).toBeGreaterThan(0)
const uploadedFileBytes = await uploadedFile.bytes()
expect(uploadedFileBytes.byteLength).toBeGreaterThan(0)
})
it('should sample uniform 16 frames when chunk duration > 16s', async () => {
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- child_process.execFile mockImplementation returns void rather than ChildProcess in unit tests
;(child_process.execFile as any).mockImplementation(
(
_cmd: unknown,
_args: unknown,
cb: (err: null, res: { stdout: string; stderr: string }) => void,
) => {
cb(null, {
stdout: '',
stderr: '',
})
},
)
await transcodeVideoChunkActivity({
assetId: 'asset-abc',
filePath: '/tmp/full-video.mp4',
startTime: 0,
endTime: 60,
format: 'frames',
})
expect(child_process.execFile).toHaveBeenCalledWith(
'ffmpeg',
expect.arrayContaining(['-vf', `fps=${16 / 60}`, '-vframes', '16', '-q:v', '2']),
expect.any(Function),
)
})
it('should throw non-retryable ApplicationFailure when neither filePath nor assetKey is provided', async () => {
expect.assertions(2)
try {
@@ -1828,5 +1922,67 @@ describe('Transcode Activities', () => {
})
expect(task).toBeNull()
})
it('should NOT create ai_embedding task when ENABLE_EMBEDDING_AGENT is false', async () => {
const user = await prisma.user.create({
data: { name: 'Test User Embedding 3', email: 'test-embedding-agent-3@test.com' },
})
const team = await prisma.team.create({
data: { name: 'Test Team Embedding 3' },
})
await prisma.teamMember.create({
data: { teamId: team.id, userId: user.id, role: 'owner' },
})
const project = await prisma.project.create({
data: { name: 'Test Project Embedding 3', teamId: team.id },
})
await prisma.agent.create({
data: {
id: user.id,
type: 'embedding',
enabled: true,
teamId: team.id,
// eslint-disable-next-line @typescript-eslint/no-explicit-any
config: { provider: 'test', model: 'test' } as any,
},
})
const asset = await prisma.asset.create({
data: {
name: 'robot.glb',
mediaType: 'model/gltf-binary',
type: 'file',
status: 'processed',
projectId: project.id,
media: {
proxyType: '3d',
imageTranscodes: [
{
key: 'files/asset-1/views.webp',
width: 2880,
height: 2880,
quality: 85,
format: 'webp',
},
],
} as unknown as PrismaJson.MediaInfo,
},
})
process.env.ENABLE_EMBEDDING_AGENT = 'false'
try {
await createEmbeddingTaskIfEnabledActivity({
assetId: asset.id,
teamId: team.id,
projectId: project.id,
})
const task = await prisma.workflowTask.findFirst({
where: { assetId: asset.id, type: 'ai_embedding' },
})
expect(task).toBeNull()
} finally {
delete process.env.ENABLE_EMBEDDING_AGENT
}
})
})
})
+78 -1
View File
@@ -1372,7 +1372,7 @@ export async function createEmbeddingTaskIfEnabledActivity(
const teamId = params.teamId || asset.project?.teamId || null
const projectId = params.projectId || asset.projectId
if (!teamId) {
if (!teamId || process.env.ENABLE_EMBEDDING_AGENT === 'false') {
return
}
@@ -1490,6 +1490,7 @@ export interface TranscodeVideoChunkParams {
filePath?: string
startTime: number
endTime: number
format?: 'video' | 'frames'
}
export async function transcodeVideoChunkActivity(
@@ -1502,6 +1503,82 @@ export async function transcodeVideoChunkActivity(
})
}
if (params.format === 'frames') {
const tmpDir = transcodeService.createTempDir('video-chunk-frames-')
try {
const bucket = process.env.S3_BUCKET || 'shumai'
const inputSource =
params.filePath ||
(params.assetKey ? await s3Service.resolveInput(bucket, params.assetKey) : '')
if (!inputSource) {
throw ApplicationFailure.create({
message: 'Neither filePath nor assetKey was provided for video chunk transcoding',
nonRetryable: true,
})
}
const isRemote = inputSource.startsWith('http://') || inputSource.startsWith('https://')
const duration = Math.max(0.1, params.endTime - params.startTime)
const fpsFilter = duration > 16 ? `fps=${16 / duration}` : 'fps=1'
const vframesArgs = duration > 16 ? ['-vframes', '16'] : []
await execFileAsync('ffmpeg', [
'-y',
'-loglevel',
'warning',
...(isRemote
? ['-reconnect', '1', '-reconnect_streamed', '1', '-reconnect_delay_max', '5']
: []),
'-ss',
params.startTime.toString(),
'-i',
inputSource,
'-map',
'0:V:0',
'-t',
duration.toString(),
'-vf',
fpsFilter,
...vframesArgs,
'-q:v',
'2',
path.join(tmpDir, 'frame_%04d.jpg'),
])
const frameFiles = fs
.readdirSync(tmpDir)
.filter((f) => /\.(jpe?g|png|webp)$/i.test(f))
.sort()
const archiveEntries: Record<string, Buffer> = {}
for (const file of frameFiles) {
archiveEntries[file] = fs.readFileSync(path.join(tmpDir, file))
}
const archive = new Bun.Archive(archiveEntries)
const archiveBytes = await archive.bytes()
const archiveData = Buffer.from(
archiveBytes.buffer,
archiveBytes.byteOffset,
archiveBytes.byteLength,
)
const key = `files/${params.assetId}/tmp-embedding-chunks/frames-${params.startTime}-${params.endTime}-${ulid()}.tar`
await s3Service.putObject(bucket, key, archiveData, archiveData.length, 'application/x-tar')
return { chunkKey: key }
} catch (err) {
if (err instanceof ApplicationFailure) {
throw err
}
throw ApplicationFailure.create({
message: `Failed to extract video chunk frames for ${params.startTime}-${params.endTime}: ${err instanceof Error ? err.message : String(err)}`,
nonRetryable: false,
})
} finally {
transcodeService.removeDir(tmpDir)
}
}
const chunkTmp = path.join(os.tmpdir(), `video-chunk-${Date.now()}-${ulid()}.mp4`)
try {
const bucket = process.env.S3_BUCKET || 'shumai'
@@ -441,4 +441,39 @@ describe('FileCard', () => {
expect(tooltip.textContent).toContain('FFmpeg transcode failed: exit code 1')
vi.useRealTimers()
})
it('does not render selection checkbox when hideCheckbox is true', () => {
renderComponent({ item: fileItem, hideCheckbox: true })
expect(screen.queryByRole('checkbox')).toBeNull()
})
it('renders selection checkbox by default when hideCheckbox is false or omitted', () => {
renderComponent({ item: fileItem })
expect(screen.getByRole('checkbox')).toBeTruthy()
})
it('does not render 3-dot dropdown menu when hideActions is true', () => {
renderComponent({ item: fileItem, hideActions: true })
// No dropdown trigger button rendered
expect(screen.queryByRole('button')).toBeNull()
})
it('renders matched duration chunk in preview when startTime and endTime are present', () => {
const videoWithMatch: AssetInfo = {
...fileItem,
startTime: 65,
endTime: 125,
preview: {
proxyType: 'video',
thumbnailUrl: 'https://example.com/poster.webp',
duration: 300,
},
} as AssetInfo
renderComponent({ item: videoWithMatch })
const matchedDuration = screen.getByTestId('file-preview-matched-duration')
expect(matchedDuration).toBeTruthy()
expect(matchedDuration.textContent).toBe('01:05 - 02:05')
})
})
@@ -48,18 +48,18 @@ import { FIELD_TYPE_ICONS } from '../fields-manager'
interface FileCardProps {
teamId?: string
item: AssetInfo
isSelected: boolean
isChecked: boolean
isEditing: boolean
onSelect: (item: AssetInfo, event: React.MouseEvent) => void
onDoubleClick: (item: AssetInfo) => void
onContextMenu: (e: React.MouseEvent, item: AssetInfo) => void
isSelected?: boolean
isChecked?: boolean
isEditing?: boolean
onSelect?: (item: AssetInfo, event: React.MouseEvent) => void
onDoubleClick?: (item: AssetInfo) => void
onContextMenu?: (e: React.MouseEvent, item: AssetInfo) => void
// Deprecated manual drag handlers replaced by dnd-kit
onDragStart: (item: AssetInfo) => void
onDrop: (target: AssetInfo) => void
onRename: (newName: string) => void
onFinishEditing: () => void
onSaveField: (fieldId: string, value: unknown) => void
onDragStart?: (item: AssetInfo) => void
onDrop?: (target: AssetInfo) => void
onRename?: (newName: string) => void
onFinishEditing?: () => void
onSaveField?: (fieldId: string, value: unknown) => void
dragState?: DragState
disabled?: boolean
onAction?: (
@@ -69,39 +69,43 @@ interface FileCardProps {
isRecentlyDeleted?: boolean
isRecents?: boolean
selectedCount?: number
fields: MetadataFieldInfo[]
fields?: MetadataFieldInfo[]
isShareView?: boolean
canEdit?: boolean
/** When false, hides the download action. Defaults to true. */
allowDownload?: boolean
hideCheckbox?: boolean
hideActions?: boolean
}
export function FileCard({
teamId,
item,
isSelected,
isChecked,
isEditing,
onSelect,
onDoubleClick,
onContextMenu,
isSelected = false,
isChecked = false,
isEditing = false,
onSelect = () => {},
onDoubleClick = () => {},
onContextMenu = () => {},
// eslint-disable-next-line @typescript-eslint/no-unused-vars
onDragStart: _onDragStart,
// eslint-disable-next-line @typescript-eslint/no-unused-vars
onDrop: _onDrop,
onRename,
onFinishEditing,
onSaveField,
onRename = () => {},
onFinishEditing = () => {},
onSaveField = () => {},
dragState,
disabled,
onAction,
isRecentlyDeleted,
isRecents,
selectedCount,
fields,
fields = [],
isShareView,
canEdit = true,
allowDownload = true,
hideCheckbox = false,
hideActions = false,
}: FileCardProps) {
const [name, setName] = useState(item.name)
const inputRef = useRef<HTMLInputElement>(null)
@@ -170,10 +174,12 @@ export function FileCard({
id: displayItem.id,
item: displayItem,
},
disabled: disabled,
})
// Combine refs
const setNodeRef = (node: HTMLElement | null) => {
if (disabled) return
setDraggableRef(node)
setDroppableRef(node)
}
@@ -332,17 +338,19 @@ export function FileCard({
showDropFeedback && 'border-primary outline-1 outline-primary',
)}
>
<div className="absolute left-2 top-2 z-10 flex items-center gap-1.5">
<Checkbox
checked={isChecked}
onCheckedChange={() => {}}
onClick={(e) => {
e.stopPropagation()
onSelect(displayItem, e)
}}
className="h-4 w-4 bg-white/20 dark:bg-white/20 border-2 data-[state=checked]:bg-primary data-[state=checked]:border-primary border-foreground/15"
/>
</div>
{!hideCheckbox && (
<div className="absolute left-2 top-2 z-10 flex items-center gap-1.5">
<Checkbox
checked={isChecked}
onCheckedChange={() => {}}
onClick={(e) => {
e.stopPropagation()
onSelect(displayItem, e)
}}
className="h-4 w-4 bg-white/20 dark:bg-white/20 border-2 data-[state=checked]:bg-primary data-[state=checked]:border-primary border-foreground/15"
/>
</div>
)}
<div className="absolute right-2 top-2 z-10 flex items-center gap-1.5">
{displayItem.agent && (
@@ -395,7 +403,15 @@ export function FileCard({
data-testid="file-card-preview-media"
className={cn('h-full w-full', isProcessing && 'animate-pulse')}
>
<FilePreview item={displayItem} showDuration={!isProcessing && !isFailed} />
<FilePreview
item={displayItem}
showDuration={!isProcessing && !isFailed}
matchDuration={
displayItem.startTime != null && displayItem.endTime != null
? { startTime: displayItem.startTime, endTime: displayItem.endTime }
: null
}
/>
</div>
{previewBadges}
</>
@@ -435,7 +451,15 @@ export function FileCard({
</div>
) : (
<>
<FilePreview item={displayItem} showDuration />
<FilePreview
item={displayItem}
showDuration
matchDuration={
displayItem.startTime != null && displayItem.endTime != null
? { startTime: displayItem.startTime, endTime: displayItem.endTime }
: null
}
/>
{previewBadges}
</>
)}
@@ -487,7 +511,7 @@ export function FileCard({
</Tooltip>
</TooltipProvider>
</div>
{!isRecents && (
{!hideActions && !isRecents && (
<DropdownMenu modal={false}>
<DropdownMenuTrigger asChild>
<Button
@@ -20,9 +20,13 @@ type FilePreviewItem = {
interface FilePreviewProps {
item: FilePreviewItem
showDuration?: boolean
matchDuration?: {
startTime?: number | null
endTime?: number | null
} | null
}
export const FilePreview = ({ item, showDuration = false }: FilePreviewProps) => {
export const FilePreview = ({ item, showDuration = false, matchDuration }: FilePreviewProps) => {
const proxyType = item.preview?.proxyType || item.proxyType
const isVideo = proxyType === 'video'
const isAudio = proxyType === 'audio'
@@ -32,16 +36,25 @@ export const FilePreview = ({ item, showDuration = false }: FilePreviewProps) =>
const duration = item.preview?.duration
const pageCount = item.preview?.pageCount
const overlay =
showDuration && hasDuration && typeof duration === 'number' && duration > 0 ? (
<span className="pointer-events-none absolute bottom-1 right-1 rounded bg-black/60 px-1 py-0.5 text-xs font-medium tabular-nums text-white">
{formatTime(duration)}
</span>
) : showDuration && isPdf && typeof pageCount === 'number' && pageCount > 0 ? (
<span className="pointer-events-none absolute bottom-1 right-1 rounded bg-black/60 px-1 py-0.5 text-xs font-medium tabular-nums text-white">
{pageCount} P
</span>
) : null
const hasMatchedChunk =
matchDuration != null && matchDuration.startTime != null && matchDuration.endTime != null
const overlay = hasMatchedChunk ? (
<span
data-testid="file-preview-matched-duration"
className="pointer-events-none absolute bottom-1 right-1 rounded bg-black/60 px-1 py-0.5 text-xs font-medium tabular-nums text-white"
>
{formatTime(matchDuration.startTime!)} - {formatTime(matchDuration.endTime!)}
</span>
) : showDuration && hasDuration && typeof duration === 'number' && duration > 0 ? (
<span className="pointer-events-none absolute bottom-1 right-1 rounded bg-black/60 px-1 py-0.5 text-xs font-medium tabular-nums text-white">
{formatTime(duration)}
</span>
) : showDuration && isPdf && typeof pageCount === 'number' && pageCount > 0 ? (
<span className="pointer-events-none absolute bottom-1 right-1 rounded bg-black/60 px-1 py-0.5 text-xs font-medium tabular-nums text-white">
{pageCount} P
</span>
) : null
const isMediaWithSprite =
(isVideo || isPdf || is3D) &&
@@ -96,4 +96,19 @@ describe('FolderCard', () => {
renderComponent({ item: folderItem, isRecentlyDeleted: true })
expect(screen.queryByTestId('folder-card-days-left')).toBeNull()
})
it('does not render selection checkbox when hideCheckbox is true', () => {
renderComponent({ item: folderItem, hideCheckbox: true })
expect(screen.queryByRole('checkbox')).toBeNull()
})
it('renders selection checkbox by default when hideCheckbox is false or omitted', () => {
renderComponent({ item: folderItem })
expect(screen.getByRole('checkbox')).toBeTruthy()
})
it('does not render 3-dot dropdown menu when hideActions is true', () => {
renderComponent({ item: folderItem, hideActions: true })
expect(screen.queryByRole('button')).toBeNull()
})
})
@@ -20,17 +20,17 @@ import { getAllFilesFromEntries } from '@/ui/lib/dnd-utils'
interface FolderCardProps {
item: AssetInfo
isSelected: boolean
isChecked: boolean
isEditing: boolean
onSelect: (item: AssetInfo, event: React.MouseEvent) => void
onDoubleClick: (item: AssetInfo) => void
onContextMenu: (e: React.MouseEvent, item: AssetInfo) => void
isSelected?: boolean
isChecked?: boolean
isEditing?: boolean
onSelect?: (item: AssetInfo, event: React.MouseEvent) => void
onDoubleClick?: (item: AssetInfo) => void
onContextMenu?: (e: React.MouseEvent, item: AssetInfo) => void
// Deprecated manual drag props replaced by dnd-kit
onDragStart: (item: AssetInfo) => void
onDrop: (target: AssetInfo) => void
onRename: (newName: string) => void
onFinishEditing: () => void
onDragStart?: (item: AssetInfo) => void
onDrop?: (target: AssetInfo) => void
onRename?: (newName: string) => void
onFinishEditing?: () => void
dragState?: DragState
disabled?: boolean
onAction?: (
@@ -49,6 +49,8 @@ interface FolderCardProps {
onExternalDrop?: (files: File[], folderId: string) => void
/** When false, hides the download action. Defaults to true. */
allowDownload?: boolean
hideCheckbox?: boolean
hideActions?: boolean
}
const FolderPreviewGrid = ({ items }: { items: ChildPreview[] }) => {
@@ -106,18 +108,18 @@ const FolderPreviewGrid = ({ items }: { items: ChildPreview[] }) => {
export function FolderCard({
item,
isSelected,
isChecked,
isEditing,
onSelect,
onDoubleClick,
onContextMenu,
isSelected = false,
isChecked = false,
isEditing = false,
onSelect = () => {},
onDoubleClick = () => {},
onContextMenu = () => {},
// eslint-disable-next-line @typescript-eslint/no-unused-vars
onDragStart: _onDragStart,
// eslint-disable-next-line @typescript-eslint/no-unused-vars
onDrop: _onDrop,
onRename,
onFinishEditing,
onRename = () => {},
onFinishEditing = () => {},
dragState,
disabled,
onAction,
@@ -132,6 +134,8 @@ export function FolderCard({
resetExternalDragState,
onExternalDrop,
allowDownload = true,
hideCheckbox = false,
hideActions = false,
}: FolderCardProps) {
const [name, setName] = useState(item.name || '')
const inputRef = useRef<HTMLInputElement>(null)
@@ -154,9 +158,11 @@ export function FolderCard({
id: item.id,
item: item,
},
disabled: disabled,
})
const setNodeRef = (node: HTMLElement | null) => {
if (disabled) return
setDraggableRef(node)
setDroppableRef(node)
}
@@ -333,13 +339,15 @@ export function FolderCard({
<div className="shadow-[0_-4px_6px_rgba(0,0,0,0.08)] relative w-full flex flex-1 flex-col p-2 mt-5 bg-muted dark:bg-muted/50 rounded-sm">
<div className="relative w-full aspect-video shrink-0 overflow-hidden">
<div className="absolute top-1 left-1 z-30 transition-all duration-200">
<Checkbox
checked={isChecked}
onClick={(e) => e.stopPropagation()} // Let parent handler deal with it
className="w-5 h-5 border-1 border-foreground/20 bg-background/50 data-[state=checked]:bg-primary data-[state=checked]:border-primary"
/>
</div>
{!hideCheckbox && (
<div className="absolute top-1 left-1 z-30 transition-all duration-200">
<Checkbox
checked={isChecked}
onClick={(e) => e.stopPropagation()} // Let parent handler deal with it
className="w-5 h-5 border-1 border-foreground/20 bg-background/50 data-[state=checked]:bg-primary data-[state=checked]:border-primary"
/>
</div>
)}
{item.latestChildren && item.latestChildren.length > 0 && (
<div data-testid="folder-preview-grid" className="h-full w-full">
@@ -376,7 +384,7 @@ export function FolderCard({
? m.n_items_singular({ count: item.fileCount || 0 })
: m.n_items_plural({ count: item.fileCount || 0 })}
</span>
{!isRecents && (
{!hideActions && !isRecents && (
<DropdownMenu modal={false}>
<DropdownMenuTrigger asChild>
<button
@@ -0,0 +1,328 @@
// @vitest-environment happy-dom
import { cleanup, fireEvent, render, screen, waitFor } from '@testing-library/react'
import React from 'react'
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import { QueryClient, QueryClientProvider } from '@tanstack/react-query'
import type { AssetInfo } from '@shumai/dtos'
import { SearchFilterDialog } from './search-filter-dialog'
vi.mock('@dnd-kit/react', () => ({
useDraggable: () => ({
ref: vi.fn(),
isDragging: false,
}),
useDroppable: () => ({
ref: vi.fn(),
isDropTarget: false,
}),
}))
const mockSearchPost = vi.fn()
const mockSettingsGet = vi.fn().mockResolvedValue({
ok: true,
json: async () => ({ semanticSearchEnabled: true }),
})
vi.mock('@/ui/api/client', () => ({
client: {
api: {
teams: {
':teamId': {
settings: {
$get: (...args: unknown[]) => mockSettingsGet(...args),
},
},
},
folders: {
':folderId': {
search: {
$post: (...args: unknown[]) => mockSearchPost(...args),
},
},
},
files: {
':fileId': {
$get: vi.fn().mockResolvedValue({
ok: true,
json: async () => ({}),
}),
},
},
projects: {
':projectId': {
collections: {
$post: vi.fn().mockResolvedValue({
ok: true,
json: async () => ({ id: 'col-1' }),
}),
},
},
},
},
},
}))
vi.mock('@tanstack/react-router', () => ({
useNavigate: () => vi.fn(),
useRouterState: () => '/projects/proj-1',
}))
describe('SearchFilterDialog', () => {
let queryClient: QueryClient
let windowOpenSpy: ReturnType<typeof vi.spyOn>
beforeEach(() => {
queryClient = new QueryClient({
defaultOptions: { queries: { retry: false } },
})
windowOpenSpy = vi.spyOn(window, 'open').mockImplementation(() => null)
})
afterEach(() => {
cleanup()
vi.clearAllMocks()
windowOpenSpy.mockRestore()
})
const defaultProps = {
open: true,
onOpenChange: vi.fn(),
teamId: 'team-1',
projectId: 'proj-1',
assetId: 'folder-root',
fields: [],
initialConditions: [],
onApply: vi.fn(),
}
it('renders initial ready to search state when open and no query', () => {
render(
<QueryClientProvider client={queryClient}>
<SearchFilterDialog {...defaultProps} />
</QueryClientProvider>,
)
expect(screen.getByText(/Ready to search|准备搜索/i)).toBeTruthy()
})
it('renders search results in a responsive grid using FileCard and FolderCard without checkboxes or action menus', async () => {
const mockFile: AssetInfo = {
id: 'file-101',
name: 'video_clip.mp4',
type: 'file',
sizeByte: 1048576,
status: 'processed',
createdAt: '2026-01-01T00:00:00Z',
updatedAt: '2026-01-01T00:00:00Z',
preview: {
proxyType: 'video',
thumbnailUrl: 'https://example.com/clip.jpg',
duration: 120,
},
} as AssetInfo
const mockFolder: AssetInfo = {
id: 'folder-202',
name: 'B-Roll Folder',
type: 'folder',
fileCount: 4,
status: 'processed',
createdAt: '2026-01-01T00:00:00Z',
updatedAt: '2026-01-01T00:00:00Z',
} as AssetInfo
mockSearchPost.mockResolvedValueOnce({
ok: true,
json: async () => ({
data: [mockFile, mockFolder],
pageInfo: { total: 2 },
}),
})
render(
<QueryClientProvider client={queryClient}>
<SearchFilterDialog
{...defaultProps}
initialConditions={[{ field: 'name', operator: 'contains', value: 'video' }]}
/>
</QueryClientProvider>,
)
await waitFor(() => {
expect(screen.getByText('video_clip.mp4')).toBeTruthy()
expect(screen.getByText('B-Roll Folder')).toBeTruthy()
})
// Neither card renders checkboxes or 3-dots action menus in search dialog
expect(screen.queryByRole('checkbox')).toBeNull()
expect(screen.queryByRole('button', { name: /more/i })).toBeNull()
// Grid container is present
const fileCard = screen.getByText('video_clip.mp4').closest('[data-testid="file-card"]')
expect(fileCard).toBeTruthy()
const gridContainer = fileCard?.parentElement
expect(gridContainer?.className).toContain('grid')
})
it('single-clicking a file opens detail page in a new tab seeking to startTime without closing dialog', async () => {
const onOpenChange = vi.fn()
const mockSemanticFile: AssetInfo = {
id: 'file-sem-1',
name: 'interview.mp4',
type: 'file',
status: 'processed',
startTime: 45,
endTime: 90,
createdAt: '2026-01-01T00:00:00Z',
updatedAt: '2026-01-01T00:00:00Z',
preview: {
proxyType: 'video',
thumbnailUrl: 'https://example.com/interview.jpg',
duration: 300,
},
} as AssetInfo
mockSearchPost.mockResolvedValueOnce({
ok: true,
json: async () => ({
data: [mockSemanticFile],
pageInfo: { total: 1 },
}),
})
render(
<QueryClientProvider client={queryClient}>
<SearchFilterDialog
{...defaultProps}
onOpenChange={onOpenChange}
initialConditions={[{ field: 'name', operator: 'contains', value: 'interview' }]}
/>
</QueryClientProvider>,
)
await waitFor(() => {
expect(screen.getByText('interview.mp4')).toBeTruthy()
})
// Matched chunk duration replaces total duration in preview
const matchedBadge = screen.getByTestId('file-preview-matched-duration')
expect(matchedBadge.textContent).toBe('00:45 - 01:30')
// Single click on file card
const card = screen.getByText('interview.mp4').closest('[data-testid="file-card"]')!
fireEvent.click(card)
expect(windowOpenSpy).toHaveBeenCalledWith(
'/projects/proj-1/files/file-sem-1?start=45',
'_blank',
'noopener,noreferrer',
)
// Dialog should NOT be closed
expect(onOpenChange).not.toHaveBeenCalled()
})
it('single-clicking a folder opens folder page in a new tab without closing dialog', async () => {
const onOpenChange = vi.fn()
const mockFolder: AssetInfo = {
id: 'folder-sub-1',
name: 'Subfolder Alpha',
type: 'folder',
status: 'processed',
createdAt: '2026-01-01T00:00:00Z',
updatedAt: '2026-01-01T00:00:00Z',
} as AssetInfo
mockSearchPost.mockResolvedValueOnce({
ok: true,
json: async () => ({
data: [mockFolder],
pageInfo: { total: 1 },
}),
})
render(
<QueryClientProvider client={queryClient}>
<SearchFilterDialog
{...defaultProps}
onOpenChange={onOpenChange}
initialConditions={[{ field: 'name', operator: 'contains', value: 'Subfolder' }]}
/>
</QueryClientProvider>,
)
await waitFor(() => {
expect(screen.getByText('Subfolder Alpha')).toBeTruthy()
})
const card = screen.getByText('Subfolder Alpha').closest('[data-testid="file-card"]')!
fireEvent.click(card)
expect(windowOpenSpy).toHaveBeenCalledWith(
'/projects/proj-1/folders/folder-sub-1',
'_blank',
'noopener,noreferrer',
)
expect(onOpenChange).not.toHaveBeenCalled()
})
it('renders multiple distinct chunk matches for the same video file with unique timestamps', async () => {
const chunk1: AssetInfo = {
id: 'video-same',
name: 'drone_shots.mp4',
type: 'file',
status: 'processed',
startTime: 10,
endTime: 30,
createdAt: '2026-01-01T00:00:00Z',
updatedAt: '2026-01-01T00:00:00Z',
preview: {
proxyType: 'video',
thumbnailUrl: 'https://example.com/drone.jpg',
duration: 200,
},
} as AssetInfo
const chunk2: AssetInfo = {
id: 'video-same',
name: 'drone_shots.mp4',
type: 'file',
status: 'processed',
startTime: 120,
endTime: 150,
createdAt: '2026-01-01T00:00:00Z',
updatedAt: '2026-01-01T00:00:00Z',
preview: {
proxyType: 'video',
thumbnailUrl: 'https://example.com/drone.jpg',
duration: 200,
},
} as AssetInfo
mockSearchPost.mockResolvedValueOnce({
ok: true,
json: async () => ({
data: [chunk1, chunk2],
pageInfo: { total: 2 },
}),
})
render(
<QueryClientProvider client={queryClient}>
<SearchFilterDialog
{...defaultProps}
initialConditions={[{ field: 'name', operator: 'contains', value: 'drone' }]}
/>
</QueryClientProvider>,
)
await waitFor(() => {
const titles = screen.getAllByText('drone_shots.mp4')
expect(titles).toHaveLength(2)
})
const badges = screen.getAllByTestId('file-preview-matched-duration')
expect(badges).toHaveLength(2)
expect(badges[0].textContent).toBe('00:10 - 00:30')
expect(badges[1].textContent).toBe('02:00 - 02:30')
})
})
@@ -8,23 +8,15 @@ import { ScrollArea } from '@/ui/components/ui/scroll-area'
import { Input } from '@/ui/components/ui/input'
import { Label } from '@/ui/components/ui/label'
import { Switch } from '@/ui/components/ui/switch'
import { formatSize } from '@/ui/lib/format'
import { cn } from '@/ui/lib/utils'
import { useInfiniteQuery, useMutation, useQuery, useQueryClient } from '@tanstack/react-query'
import { useNavigate } from '@tanstack/react-router'
import {
AlertCircle,
ChevronDown,
ChevronUp,
FileIcon,
FolderIcon,
Loader2,
Search,
X,
} from 'lucide-react'
import { AlertCircle, ChevronDown, ChevronUp, Loader2, Search, X } from 'lucide-react'
import { useEffect, useMemo, useState } from 'react'
import { useInView } from 'react-intersection-observer'
import { toast } from 'sonner'
import { FileCard } from '@/ui/components/file-browser/file-card'
import { FolderCard } from '@/ui/components/file-browser/folder-card'
import { FilterPanel } from './filter-panel'
import { m } from '@/ui/paraglide/messages.js'
@@ -225,36 +217,25 @@ export function SearchFilterDialog({
setConditions([])
}
const formatTimestamp = (seconds: number) => {
const m = Math.floor(seconds / 60)
const s = Math.floor(seconds % 60)
return `${m.toString().padStart(2, '0')}:${s.toString().padStart(2, '0')}`
}
const handleResultClick = (item: AssetInfo) => {
if (item.type === 'folder') {
navigate({
to: '/projects/$projectId/folders/$folderId',
params: { projectId, folderId: item.id! },
})
window.open(`/projects/${projectId}/folders/${item.id}`, '_blank', 'noopener,noreferrer')
} else {
navigate({
to: '/projects/$projectId/files/$fileId',
params: { projectId, fileId: item.versionStack ? item.versionStack.id : item.id! },
search: {
version: undefined,
start: item.startTime && item.startTime > 0 ? item.startTime : undefined,
},
})
const fileId = item.versionStack ? item.versionStack.id : item.id!
const searchParams = new URLSearchParams()
if (item.startTime !== undefined && item.startTime !== null && item.startTime > 0) {
searchParams.set('start', String(item.startTime))
}
const query = searchParams.toString() ? `?${searchParams.toString()}` : ''
window.open(`/projects/${projectId}/files/${fileId}${query}`, '_blank', 'noopener,noreferrer')
}
onOpenChange(false)
}
const activeFiltersCount = conditions.filter((c) => c.value !== '').length
return (
<Dialog open={open} onOpenChange={onOpenChange}>
<DialogContent className="sm:max-w-[900px] p-0 gap-0 overflow-hidden bg-background border-border rounded-2xl shadow-2xl flex flex-col h-[85vh]">
<DialogContent className="sm:max-w-4xl lg:max-w-5xl p-0 gap-0 overflow-hidden bg-background border-border rounded-2xl shadow-2xl flex flex-col h-[85vh]">
<DialogHeader className="px-6 pt-3 pb-0 text-left">
<DialogTitle className="text-lg font-bold">{m.search_dialog_title()}</DialogTitle>
</DialogHeader>
@@ -394,121 +375,94 @@ export function SearchFilterDialog({
)}
</div>
<div className="flex flex-col border border-border rounded-xl divide-y divide-border overflow-hidden min-w-0">
{!hasActiveCriteria ? (
<div className="py-20 text-center flex flex-col items-center justify-center bg-muted/20 px-4">
<Search className="w-10 h-10 text-muted-foreground/30 mb-2" />
<h3 className="text-sm font-semibold text-foreground/80">
{m.ready_to_search()}
</h3>
<p className="text-xs text-muted-foreground max-w-sm mt-1 mx-auto">
{m.type_query_instruction()}
{!hasActiveCriteria ? (
<div className="border border-border rounded-xl py-20 text-center flex flex-col items-center justify-center bg-muted/20 px-4">
<Search className="w-10 h-10 text-muted-foreground/30 mb-2" />
<h3 className="text-sm font-semibold text-foreground/80">{m.ready_to_search()}</h3>
<p className="text-xs text-muted-foreground max-w-sm mt-1 mx-auto">
{m.type_query_instruction()}
</p>
{isSemantic && (
<p className="text-xs text-muted-foreground/80 max-w-md mt-4 mx-auto border-t border-border/60 pt-3">
{m.semantic_search_note()}
</p>
{isSemantic && (
<p className="text-xs text-muted-foreground/80 max-w-md mt-4 mx-auto border-t border-border/60 pt-3">
{m.semantic_search_note()}
</p>
)}
</div>
) : searchError ? (
<div className="py-12 px-4 text-center flex flex-col items-center justify-center bg-muted/20">
<AlertCircle className="w-10 h-10 text-destructive/50 mb-2" />
<h3 className="text-sm font-semibold text-foreground/80">{m.search_failed()}</h3>
<p className="text-xs text-muted-foreground max-w-sm mt-1 mx-auto">
{searchError.message.includes('Embedding agent not configured')
? m.enable_embedding_agent_hint()
: searchError.message}
</p>
</div>
) : isLoading ? (
Array.from({ length: 3 }).map((_, idx) => (
)}
</div>
) : searchError ? (
<div className="border border-border rounded-xl py-12 px-4 text-center flex flex-col items-center justify-center bg-muted/20">
<AlertCircle className="w-10 h-10 text-destructive/50 mb-2" />
<h3 className="text-sm font-semibold text-foreground/80">{m.search_failed()}</h3>
<p className="text-xs text-muted-foreground max-w-sm mt-1 mx-auto">
{searchError.message.includes('Embedding agent not configured')
? m.enable_embedding_agent_hint()
: searchError.message}
</p>
</div>
) : isLoading ? (
<div className="grid grid-cols-2 sm:grid-cols-3 lg:grid-cols-4 gap-4">
{Array.from({ length: 8 }).map((_, idx) => (
<div
key={idx}
className="p-3.5 flex items-center gap-3 animate-pulse bg-muted/10"
className="flex flex-col rounded-xl border border-border bg-card p-2 space-y-2 animate-pulse"
>
<div className="w-11 h-11 rounded-xl bg-muted shrink-0" />
<div className="flex-1 space-y-2">
<div className="h-4 bg-muted rounded w-1/3" />
<div className="h-3 bg-muted/50 rounded w-1/4" />
<div className="aspect-square w-full rounded-lg bg-muted/60" />
<div className="p-1 space-y-1.5">
<div className="h-4 bg-muted/60 rounded w-3/4" />
<div className="h-3 bg-muted/40 rounded w-1/2" />
</div>
</div>
))
) : flattenedResults.length === 0 ? (
<div className="py-12 px-4 text-center flex flex-col items-center justify-center bg-muted/20">
<AlertCircle className="w-10 h-10 text-muted-foreground/30 mb-2" />
<h3 className="text-sm font-semibold text-foreground/80">
{m.no_records_matched()}
</h3>
<p className="text-xs text-muted-foreground max-w-sm mt-1 mx-auto">
{m.try_adjusting_spelling()}
</p>
))}
</div>
) : flattenedResults.length === 0 ? (
<div className="border border-border rounded-xl py-12 px-4 text-center flex flex-col items-center justify-center bg-muted/20">
<AlertCircle className="w-10 h-10 text-muted-foreground/30 mb-2" />
<h3 className="text-sm font-semibold text-foreground/80">
{m.no_records_matched()}
</h3>
<p className="text-xs text-muted-foreground max-w-sm mt-1 mx-auto">
{m.try_adjusting_spelling()}
</p>
</div>
) : (
<>
<div className="grid grid-cols-2 sm:grid-cols-3 lg:grid-cols-4 gap-4">
{flattenedResults.map((record) => {
const itemKey =
record.startTime != null && record.endTime != null
? `${record.id}-${record.startTime}-${record.endTime}`
: record.id
if (record.type === 'folder') {
return (
<FolderCard
key={itemKey}
item={record}
hideCheckbox
hideActions
disabled
onSelect={() => handleResultClick(record)}
/>
)
}
return (
<FileCard
key={itemKey}
teamId={teamId}
item={record}
hideCheckbox
hideActions
disabled
onSelect={() => handleResultClick(record)}
/>
)
})}
</div>
) : (
<>
{flattenedResults.map((record) => (
<div
key={record.id}
onClick={() => handleResultClick(record)}
className="p-3.5 flex items-center gap-3 hover:bg-muted/40 transition-colors group/record cursor-pointer min-w-0"
>
<div
className={cn(
'shrink-0 w-11 h-11 rounded-xl shadow-xs flex items-center justify-center text-white font-bold relative group-hover/record:scale-105 transition-transform bg-muted overflow-hidden',
record.type === 'folder' ? 'bg-primary/5' : '',
)}
>
{record.preview?.thumbnailUrl ? (
<img
src={record.preview.thumbnailUrl}
alt=""
className="w-full h-full object-cover rounded-xl"
/>
) : record.type === 'folder' ? (
<FolderIcon className="h-6 w-6 text-primary/70" />
) : (
<FileIcon className="h-6 w-6 text-muted-foreground" />
)}
</div>
<div className="flex-1 min-w-0">
<div className="flex items-center justify-between gap-3 min-w-0">
<div className="flex-1 min-w-0">
<span className="font-medium text-sm text-foreground group-hover/record:text-primary transition-colors block truncate">
{record.name}
</span>
<span className="text-xs text-muted-foreground block mt-0.5 truncate">
{record.type === 'folder'
? m.folder()
: `${m.size_prefix()} ${formatSize(record.sizeByte || 0)}`}
</span>
{record.startTime !== undefined &&
record.startTime !== null &&
record.endTime !== undefined &&
record.endTime !== null && (
<span className="text-[10px] font-semibold text-primary bg-primary/10 border border-primary/20 px-1.5 py-0.5 rounded mt-1 inline-block max-w-full truncate">
{m.match_found_at()} {formatTimestamp(record.startTime)} -{' '}
{formatTimestamp(record.endTime)}
</span>
)}
</div>
<span className="text-xs text-muted-foreground font-mono shrink-0">
{new Date(record.createdAt).toLocaleDateString(undefined, {
year: 'numeric',
month: 'short',
})}
</span>
</div>
</div>
</div>
))}
{(hasNextPage || isFetchingNextPage) && (
<div ref={ref} className="p-4 flex justify-center items-center">
<Loader2 className="h-6 w-6 animate-spin text-muted-foreground" />
</div>
)}
</>
)}
</div>
{(hasNextPage || isFetchingNextPage) && (
<div ref={ref} className="p-4 flex justify-center items-center">
<Loader2 className="h-6 w-6 animate-spin text-muted-foreground" />
</div>
)}
</>
)}
</div>
</ScrollArea>
@@ -92,4 +92,39 @@ describe('AgentFormDialog', () => {
expect(screen.getByText('Move Assets')).toBeDefined()
expect(screen.getByText('Delete Asset')).toBeDefined()
})
it('renders read-only state with system managed hint and only Close button', () => {
render(
<QueryClientProvider client={queryClient}>
<AgentFormDialog
isOpen={true}
readOnly={true}
onClose={vi.fn()}
teamId="team-123"
type="embedding"
title="Edit Embedding Agent"
initialValues={{
id: 'agent-1',
name: 'embedding-agent',
type: 'embedding',
enabled: true,
permission: 'reviewer',
thinkingLevel: 'off',
skills: [],
mcpServerIds: [],
deniedTools: [],
readOnly: true,
}}
/>
</QueryClientProvider>,
)
expect(
screen.getByText(
'This agent is managed by the server environment configuration and cannot be modified or deleted.',
),
).toBeDefined()
expect(screen.getAllByRole('button', { name: 'Close' }).length).toBe(2)
expect(screen.queryByRole('button', { name: 'Save Changes' })).toBeNull()
})
})
@@ -48,6 +48,7 @@ interface AgentFormDialogProps {
type?: AgentType
initialValues?: AgentInfo
title: string
readOnly?: boolean
}
const agentFormSchema = z.object({
@@ -137,6 +138,7 @@ export function AgentFormDialog({
type,
initialValues,
title,
readOnly = false,
}: AgentFormDialogProps) {
const queryClient = useQueryClient()
const { canAdmin } = usePermissions()
@@ -368,6 +370,11 @@ export function AgentFormDialog({
>
<ScrollArea className="flex-1 min-h-0">
<div className="p-6 pt-2 space-y-6">
{readOnly && (
<div className="p-3 bg-muted text-muted-foreground text-sm rounded-lg border border-border">
{m.agent_managed_by_system_hint()}
</div>
)}
<div className="grid grid-cols-1 md:grid-cols-2 gap-6">
<form.Field
name="name"
@@ -382,6 +389,7 @@ export function AgentFormDialog({
onChange={(e) => field.handleChange(e.target.value)}
placeholder={m.agent_name_placeholder()}
aria-invalid={isInvalid}
disabled={readOnly}
/>
{isInvalid && <FieldError errors={mapErrors(field.state.meta.errors)} />}
</Field>
@@ -423,7 +431,7 @@ export function AgentFormDialog({
onValueChange={(val) =>
field.handleChange(val as 'owner' | 'editor' | 'reviewer')
}
disabled={!canAdmin}
disabled={readOnly || !canAdmin}
>
<SelectTrigger>
<SelectValue />
@@ -453,12 +461,15 @@ export function AgentFormDialog({
<button
key={preset.id}
type="button"
onClick={() => field.handleChange(preset.id)}
disabled={readOnly}
onClick={() => !readOnly && field.handleChange(preset.id)}
className={cn(
'relative w-14 h-14 rounded-full overflow-hidden border-2 transition-all hover:scale-105',
'relative w-14 h-14 rounded-full overflow-hidden border-2 transition-all',
!readOnly && 'hover:scale-105',
isSelected
? 'border-primary ring-2 ring-primary ring-offset-2'
: 'border-border hover:border-muted-foreground',
readOnly && 'cursor-default opacity-80',
)}
>
<img
@@ -473,7 +484,8 @@ export function AgentFormDialog({
!AVAILABLE_AVATARS.some((p) => p.id === field.state.value) && (
<button
type="button"
className="relative w-14 h-14 rounded-full overflow-hidden border-2 border-primary ring-2 ring-primary ring-offset-2 scale-105"
disabled={true}
className="relative w-14 h-14 rounded-full overflow-hidden border-2 border-primary ring-2 ring-primary ring-offset-2 scale-105 cursor-default"
>
<img
src={field.state.value}
@@ -1024,29 +1036,37 @@ export function AgentFormDialog({
</ScrollArea>
<DialogFooter className="p-6 pt-4 border-t border-border flex-shrink-0">
<Button variant="outline" onClick={onClose} type="button">
{m.cancel()}
</Button>
<form.Subscribe
selector={(state) => [state.canSubmit, state.isSubmitting]}
children={([canSubmit, isSubmitting]) => (
<Button
type="submit"
className="bg-primary hover:bg-primary/90 text-primary-foreground min-w-[120px]"
disabled={
!canSubmit ||
isSubmitting ||
createMutation.isPending ||
updateMutation.isPending
}
>
{(isSubmitting || createMutation.isPending || updateMutation.isPending) && (
<Loader2 className="w-4 h-4 animate-spin mr-2" />
)}
{initialValues ? m.save_changes() : m.create_agent()}
{readOnly ? (
<Button variant="outline" onClick={onClose} type="button">
{m.close()}
</Button>
) : (
<>
<Button variant="outline" onClick={onClose} type="button">
{m.cancel()}
</Button>
)}
/>
<form.Subscribe
selector={(state) => [state.canSubmit, state.isSubmitting]}
children={([canSubmit, isSubmitting]) => (
<Button
type="submit"
className="bg-primary hover:bg-primary/90 text-primary-foreground min-w-[120px]"
disabled={
!canSubmit ||
isSubmitting ||
createMutation.isPending ||
updateMutation.isPending
}
>
{(isSubmitting || createMutation.isPending || updateMutation.isPending) && (
<Loader2 className="w-4 h-4 animate-spin mr-2" />
)}
{initialValues ? m.save_changes() : m.create_agent()}
</Button>
)}
/>
</>
)}
</DialogFooter>
</form>
</DialogContent>
@@ -189,6 +189,7 @@ export function AgentsSettings({ teamId }: AgentsSettingsProps) {
{(isCreateDialogOpen || editingAgent) && (
<AgentFormDialog
isOpen={true}
readOnly={editingAgent?.readOnly}
title={
editingAgent
? m.edit_item({ name: editingAgent.name })
@@ -340,29 +341,38 @@ export function AgentsSettings({ teamId }: AgentsSettingsProps) {
</div>
</div>
</div>
<DropdownMenu modal={false}>
<DropdownMenuTrigger asChild onClick={(e) => e.stopPropagation()}>
<Button
variant="ghost"
size="icon"
className="h-8 w-8 text-muted-foreground hover:text-foreground"
>
<MoreVertical className="w-4 h-4" />
</Button>
</DropdownMenuTrigger>
<DropdownMenuContent align="end" className="w-32">
<DropdownMenuItem
onClick={(e) => {
e.stopPropagation()
setAgentToDelete(agent)
setIsDeleteDialogOpen(true)
}}
className="text-red-500 focus:text-red-500 gap-2 cursor-pointer"
>
<Trash2 className="w-4 h-4" /> {m.delete()}
</DropdownMenuItem>
</DropdownMenuContent>
</DropdownMenu>
{agent.readOnly ? (
<Badge
variant="outline"
className="text-xs text-muted-foreground border-border"
>
{m.agent_readonly()}
</Badge>
) : (
<DropdownMenu modal={false}>
<DropdownMenuTrigger asChild onClick={(e) => e.stopPropagation()}>
<Button
variant="ghost"
size="icon"
className="h-8 w-8 text-muted-foreground hover:text-foreground"
>
<MoreVertical className="w-4 h-4" />
</Button>
</DropdownMenuTrigger>
<DropdownMenuContent align="end" className="w-32">
<DropdownMenuItem
onClick={(e) => {
e.stopPropagation()
setAgentToDelete(agent)
setIsDeleteDialogOpen(true)
}}
className="text-red-500 focus:text-red-500 gap-2 cursor-pointer"
>
<Trash2 className="w-4 h-4" /> {m.delete()}
</DropdownMenuItem>
</DropdownMenuContent>
</DropdownMenu>
)}
</div>
<div className="flex items-center justify-between gap-2 pt-2 border-t border-border/50">
@@ -408,6 +418,7 @@ export function AgentsSettings({ teamId }: AgentsSettingsProps) {
<div onClick={(e) => e.stopPropagation()}>
<Switch
checked={agent.enabled}
disabled={agent.readOnly}
onCheckedChange={(enabled) => {
updateAgentMutation.mutate({
teamId,
@@ -155,4 +155,19 @@ describe('SpriteScrubber', () => {
fireEvent.mouseMove(scrubberVideo, { clientX: 25 })
expect(scrubberVideo.getAttribute('data-frame')).toBe('25')
})
it('renders with cursor-pointer instead of cursor-crosshair for cards', () => {
const { getByTestId } = render(
<SpriteScrubber
spriteUrl="https://example.com/sprite.webp"
thumbnailUrl="https://example.com/thumb.webp"
videoWidth={1920}
videoHeight={1080}
/>,
)
const scrubber = getByTestId('sprite-scrubber')
expect(scrubber.className).toContain('cursor-pointer')
expect(scrubber.className).not.toContain('cursor-crosshair')
})
})
@@ -165,7 +165,7 @@ export const SpriteScrubber: React.FC<SpriteScrubberProps> = ({
ref={containerRef}
data-testid="sprite-scrubber"
data-frame={currentFrame}
className={`relative w-full h-full overflow-hidden cursor-crosshair group bg-black select-none ${className}`}
className={`relative w-full h-full overflow-hidden cursor-pointer group bg-black select-none ${className}`}
onMouseEnter={handleMouseEnter}
onMouseLeave={handleMouseLeave}
onMouseMove={handleMouseMove}
+2
View File
@@ -633,6 +633,8 @@
"chat_agent": "Chat Agent",
"chat_agents": "Chat Agents",
"embedding_agent": "Embedding Agent",
"agent_readonly": "Read-only",
"agent_managed_by_system_hint": "This agent is managed by the server environment configuration and cannot be modified or deleted.",
"skills_count_value": "{count} Skills",
"autofill_agent": "Autofill Agent",
"ai_provider": "AI Provider",
+2
View File
@@ -633,6 +633,8 @@
"chat_agent": "对话智能体",
"chat_agents": "对话智能体",
"embedding_agent": "嵌入智能体",
"agent_readonly": "只读",
"agent_managed_by_system_hint": "此智能体由服务器环境配置管理,无法进行修改或删除。",
"skills_count_value": "{count} 个技能",
"autofill_agent": "自动填充智能体",
"ai_provider": "AI 提供商",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/webui",
"version": "0.5.2",
"version": "0.5.3",
"private": true,
"type": "module",
"main": "./index.html",
+1 -1
View File
@@ -1 +1 @@
export const APP_VERSION = '0.5.2'
export const APP_VERSION = '0.5.3'
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/workflow-core",
"version": "0.5.2",
"version": "0.5.3",
"private": true,
"type": "module",
"main": "./src/index.ts",
@@ -8,7 +8,6 @@ import { WorkflowTaskStatus, WorkflowTaskType } from '@shumai/db'
// Mock the individual workflows
const mocks = {
agentEmbeddingMedia: vi.fn(),
queryEmbeddingForSearch: vi.fn(),
agentAutofillMedia: vi.fn(),
agentChat: vi.fn(),
transcodeMedia: vi.fn(),
@@ -26,9 +25,6 @@ describe('LocalExecutor Integration Tests', () => {
// Register workflows dynamically for testing
registerWorkflow(WorkflowTaskType.ai_embedding, async (task) => mocks.agentEmbeddingMedia(task))
registerWorkflow(WorkflowTaskType.query_embedding_for_search, async (task) =>
mocks.queryEmbeddingForSearch(task),
)
registerWorkflow(WorkflowTaskType.ai_metadata_autofill, async (task) =>
mocks.agentAutofillMedia(task),
)
@@ -22,10 +22,6 @@ export class TemporalExecutor implements Executor {
workflowName = 'agentEmbeddingWorkflow'
taskQueue = TaskQueueAgent
break
case WorkflowTaskType.query_embedding_for_search:
workflowName = 'queryEmbeddingForSearch'
taskQueue = TaskQueueAgent
break
case WorkflowTaskType.ai_metadata_autofill:
workflowName = 'agentAutofillWorkflow'
taskQueue = TaskQueueAgent