mirror of
https://github.com/shumaiOne/shumai.git
synced 2026-11-20 05:29:48 +00:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f7ed3bafdf | ||
|
|
9f76e61e23 | ||
|
|
7c297895e5 | ||
|
|
0f318b4050 | ||
|
|
1a46f63160 | ||
|
|
6cc90bf973 | ||
|
|
b7271777e0 |
@@ -12,6 +12,22 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
|
||||
### Changed
|
||||
|
||||
## [0.5.3] - 2026-10-16
|
||||
|
||||
### Added
|
||||
|
||||
* **cli**: Add administrative `shumai regenerate-embeddings` command to regenerate multimodal vector embeddings for all existing media files across active teams
|
||||
* **docker**: Provide three Docker Compose deployment tiers (Lite, with Embedding Server, and Full) with recommended memory specifications
|
||||
* **agent**: Save video chunk embeddings incrementally to the database as each chunk finishes, preserving completed progress on failure and resuming from where it left off on retry
|
||||
* **agent**: Add support for local LiteRT-LM Gemma Embedding 2 (`gemma-embedding-2-openai`) provider and environment-enforced read-only embedding agents with automatic 768-dimensional vector search indexing
|
||||
|
||||
### Fixed
|
||||
|
||||
### Changed
|
||||
|
||||
* **search**: Display search results in a responsive card grid with interactive hover video sprite scrubbing, semantic match chunk time range previews, and new-tab opening on click
|
||||
* **search**: Significantly improve semantic search response times by generating query text embeddings directly with in-memory caching instead of dispatching background tasks
|
||||
|
||||
## [0.5.2] - 2026-10-13
|
||||
|
||||
### Added
|
||||
|
||||
@@ -35,6 +35,7 @@ Shumai is an open-source Frame.io alternative built for creative teams to collab
|
||||
* **Wide File Format Support**: Preview common creative file formats, including images (with PSD support), videos, audio, and documents, while allowing any file type to be uploaded and stored.
|
||||
* **3D Asset Turntable Previews**: Generate interactive 3D model turntable previews and videos for formats like GLB, FBX, OBJ, STL, USD, and DAE using [turntable-renderer](https://docs.shumai.one/using-shumai/3d-asset-support).
|
||||
* **Office & Document PDF Proxying via Gotenberg**: Convert Word, PowerPoint, Excel, Markdown, HTML, and CSV documents to high-quality PDF proxies for in-browser previewing using [Gotenberg](https://docs.shumai.one/using-shumai/gotenberg).
|
||||
* **Semantic Search**: Find assets by visual or conceptual meaning using vector embeddings. See [documentation](https://docs.shumai.one/using-shumai/semantic-search).
|
||||
* **NLE Export & Premiere Pro Plugin (Experimental)**: Export timestamped review comments to your favorite non-linear editor (Premiere Pro XML, DaVinci Resolve EDL, Final Cut Pro, Avid Media Composer) or use the experimental Premiere Pro UXP plugin to import media assets and auto-sync comments directly to sequence timeline markers. See [documentation](https://docs.shumai.one/using-shumai/nle-and-premiere).
|
||||
|
||||
But Shumai goes beyond being a Frame.io alternative. With modern models like GPT-Image-2 and Seedance 2.5, AI can already handle a surprising amount of creative work. That means you can create or update videos and images from a simple prompt without having to jump back into Photoshop, Premiere or Houdini. That is why Shumai includes a powerful agent system:
|
||||
@@ -47,7 +48,6 @@ But Shumai goes beyond being a Frame.io alternative. With modern models like GPT
|
||||
* **Model Context Protocol (MCP)**: Connect agents to external tools and APIs through remote MCP servers, with OAuth authentication and per-agent MCP access.
|
||||
* **Isolated Sandbox Execution**: Run agent-submitted scripts safely in a sandboxed environment.
|
||||
* **AI Metadata Autofill**: Automatically fill in custom metadata for new assets.
|
||||
* **Semantic Search**: Find assets by visual or conceptual meaning using vector embeddings (Gemini Embedding 2).
|
||||
* **Resource Quotas**: Set usage limits on AI tokens, costs, and tool execution per member, role, or team-wide, with real-time tracking.
|
||||
|
||||
Here is a short demo video showing what the Shumai agent can do:
|
||||
@@ -83,13 +83,25 @@ Docker Compose is the fastest way to get Shumai running. You do not need to clon
|
||||
```bash
|
||||
mkdir shumai && cd shumai
|
||||
```
|
||||
2. Download the `docker-compose.yaml` file:
|
||||
```bash
|
||||
curl -o docker-compose.yaml https://raw.githubusercontent.com/shumaiOne/shumai/main/docker-compose/local/docker-compose.yaml
|
||||
```
|
||||
2. Download your preferred `docker-compose.yaml` configuration based on your needs and available RAM:
|
||||
|
||||
* **Lite (Suggested RAM: 2 GB+)**: Minimal deployment containing Shumai and PostgreSQL database.
|
||||
```bash
|
||||
curl -o docker-compose.yaml https://raw.githubusercontent.com/shumaiOne/shumai/main/docker-compose/local/docker-compose.yaml
|
||||
```
|
||||
|
||||
* **With Embedding Server (Suggested RAM: 4 GB+)**: Adds local multimodal semantic search powered by `shumaiOne/embedding-server` (`EmbeddingGemma 2 740M`).
|
||||
```bash
|
||||
curl -o docker-compose.yaml https://raw.githubusercontent.com/shumaiOne/shumai/main/docker-compose/local/docker-compose.embedding.yaml
|
||||
```
|
||||
|
||||
* **Full (Suggested RAM: 8 GB+)**: All-in-one setup including `embedding-server`, `turntable-renderer` (3D previews), and `Gotenberg` (document PDF proxying).
|
||||
```bash
|
||||
curl -o docker-compose.yaml https://raw.githubusercontent.com/shumaiOne/shumai/main/docker-compose/local/docker-compose.full.yaml
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> By default, the standard `docker-compose.yaml` uses CPU software transcoding. If your server has a GPU (NVIDIA NVENC, Intel QuickSync, AMD/Intel VA-API, or Rockchip RKMPP) and you want to enable GPU hardware acceleration, check the [Hardware Transcoding Guide](https://docs.shumai.one/configuration/hardware-transcoding).
|
||||
> By default, the Docker Compose setups use CPU software transcoding. If your server has a GPU (NVIDIA NVENC, Intel QuickSync, AMD/Intel VA-API, or Rockchip RKMPP) and you want to enable GPU hardware acceleration, check the [Hardware Transcoding Guide](https://docs.shumai.one/configuration/hardware-transcoding).
|
||||
3. Configure environment variables (optional):
|
||||
|
||||
* `SHUMAI_SERVER_PORT` controls the port the Shumai server listens on. The default is `3000`.
|
||||
@@ -129,190 +141,7 @@ Docker Compose is the fastest way to get Shumai running. You do not need to clon
|
||||
```
|
||||
6. Open your browser and access Shumai at `http://localhost:3000` (or `http://<your-server-ip>:3000` for remote deployments).
|
||||
|
||||
### Option 2: Install via NPM / Package Manager
|
||||
|
||||
Shumai is published as `@shumai-one/shumai` on NPM. This option allows you to run Shumai globally or locally.
|
||||
|
||||
#### Step 1: Start PostgreSQL with pgvector
|
||||
|
||||
Shumai requires PostgreSQL with the `pgvector` extension. Start a pre-configured database container using Docker:
|
||||
|
||||
```bash
|
||||
docker run --name shumai_postgres \
|
||||
-e POSTGRES_USER=shumai \
|
||||
-e POSTGRES_PASSWORD=shumai_password \
|
||||
-e POSTGRES_DB=shumai_db \
|
||||
-p 5432:5432 \
|
||||
-d pgvector/pgvector:pg18
|
||||
```
|
||||
|
||||
#### Step 2: Create a workspace folder
|
||||
|
||||
Create a dedicated directory to store your environment configuration and media files (which are saved in a `./data` directory by default):
|
||||
|
||||
```bash
|
||||
mkdir shumai && cd shumai
|
||||
```
|
||||
|
||||
#### Step 3: Install Platform-Specific Dependencies
|
||||
|
||||
#### Linux
|
||||
|
||||
Shumai requires the following system packages on Linux:
|
||||
|
||||
- **`ffmpeg`** – Used for media transcoding and metadata extraction.
|
||||
- **`poppler`** (`poppler-utils`) – Used for PDF page image extraction and PDF sprite preview generation (`pdftoppm`).
|
||||
- **`imagemagick`** – Used for PSD format transcoding and color profile conversion to sRGB.
|
||||
- **`dcraw_emu`** (`libraw` / `libraw-bin`) – Used for camera RAW format image decoding and preview generation.
|
||||
- **`bubblewrap`**, **`socat`**, and **`ripgrep`** – Required by the AI agent sandbox (`anthropic-experimental/sandbox-runtime`) for process isolation, networking, and workspace search.
|
||||
|
||||
Install all required packages with one command:
|
||||
|
||||
> Ubuntu/Debian
|
||||
```bash
|
||||
sudo apt install -y ffmpeg poppler-utils imagemagick bubblewrap socat ripgrep libraw-bin
|
||||
```
|
||||
|
||||
> Fedora
|
||||
```bash
|
||||
sudo dnf install -y ffmpeg poppler-utils ImageMagick bubblewrap socat ripgrep LibRaw
|
||||
```
|
||||
|
||||
> Arch Linux
|
||||
```bash
|
||||
sudo pacman -S --noconfirm ffmpeg poppler imagemagick bubblewrap socat ripgrep libraw
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> **Ubuntu 24.04+** restricts unprivileged user namespaces by default. This prevents `anthropic-experimental/sandbox-runtime` from using `bubblewrap` for sandbox isolation.
|
||||
>
|
||||
> To temporarily allow user namespaces:
|
||||
>
|
||||
> ```bash
|
||||
> sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
|
||||
> ```
|
||||
>
|
||||
> Alternatively, configure an AppArmor profile that grants the required `userns` permission to the relevant binaries.
|
||||
|
||||
---
|
||||
|
||||
#### macOS
|
||||
|
||||
Shumai requires:
|
||||
|
||||
- **`ffmpeg`** – Used for media transcoding and metadata extraction.
|
||||
- **`poppler`** – Used for PDF page image extraction and PDF sprite preview generation (`pdftoppm`).
|
||||
- **`imagemagick`** – Used for PSD format transcoding and color profile conversion to sRGB.
|
||||
- **`dcraw_emu`** (`libraw`) – Used for camera RAW format image decoding and preview generation.
|
||||
- **`ripgrep`** – Required by the AI agent sandbox (`anthropic-experimental/sandbox-runtime`).
|
||||
|
||||
Install the required packages with Homebrew:
|
||||
|
||||
```bash
|
||||
brew install ffmpeg poppler imagemagick ripgrep libraw
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### Windows (alpha)
|
||||
|
||||
> [!WARNING]
|
||||
> Windows support in anthropic-experimental/sandbox-runtime is **alpha and the design is in flux**.
|
||||
>
|
||||
> The current implementation provides:
|
||||
> - Network egress filtering via the **Windows Filtering Platform (WFP)**.
|
||||
> - File read/write denial via **ACL stamping**.
|
||||
>
|
||||
> However, it is **not a security boundary against a deliberately adversarial sandboxed process**.
|
||||
>
|
||||
> The sandbox implementation will be substantially revised in a future release, where sandboxed processes will run under a dedicated sandbox user account.
|
||||
|
||||
Shumai requires:
|
||||
|
||||
- **`ffmpeg`** – Used for media transcoding and metadata extraction.
|
||||
- **`imagemagick`** – Used for PSD format transcoding and color profile conversion to sRGB.
|
||||
|
||||
Install required packages using your preferred package manager:
|
||||
|
||||
**winget**
|
||||
```powershell
|
||||
winget install Gyan.FFmpeg ImageMagick.ImageMagick
|
||||
```
|
||||
|
||||
**Chocolatey**
|
||||
```powershell
|
||||
choco install ffmpeg imagemagick
|
||||
```
|
||||
|
||||
#### Step 4: Install Shumai globally
|
||||
|
||||
Install Shumai globally using your preferred package manager:
|
||||
|
||||
```bash
|
||||
# NPM
|
||||
npm install -g @shumai-one/shumai
|
||||
|
||||
# PNPM
|
||||
pnpm add -g @shumai-one/shumai
|
||||
|
||||
# Bun
|
||||
bun add -g @shumai-one/shumai
|
||||
```
|
||||
|
||||
#### Step 5: Configure Environment Variables
|
||||
|
||||
Create a `.env` file in your workspace folder (`shumai/`) and add the following configuration:
|
||||
|
||||
```env
|
||||
DATABASE_URL=postgresql://shumai:shumai_password@localhost:5432/shumai_db?schema=public
|
||||
BETTER_AUTH_SECRET=ySxs7DxzHDZBbeeHNPEwBuspYwipBqz5Gk5XdBjNhWw=
|
||||
STORAGE_BACKEND=local
|
||||
SHUMAI_SERVER_PORT=3000
|
||||
AWS_ENDPOINT_URL_S3=http://localhost:3000
|
||||
# Optional: base URL for MCP OAuth redirects (defaults to BETTER_AUTH_URL)
|
||||
MCP_OAUTH_REDIRECT_BASE_URL=http://localhost:3000
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> `SHUMAI_SERVER_PORT` sets the port the server starts on, while `AWS_ENDPOINT_URL_S3` is used by the browser to build file upload URLs.
|
||||
> If deploying on a remote server (e.g. `http://123.456.7.8`) with a mapped port (e.g. `12345:3000` in docker-compose), set `AWS_ENDPOINT_URL_S3` to `http://123.456.7.8:12345`.
|
||||
> `MCP_OAUTH_REDIRECT_BASE_URL` is only needed when MCP servers use OAuth: the authorization redirect URI defaults to `{BETTER_AUTH_URL}/api/mcp/oauth/callback`; set it to an externally reachable URL if the app is served behind a proxy.
|
||||
|
||||
#### Step 6: Run Shumai
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **For Bun Users:** Since the published package binaries contain a `#!/usr/bin/env node` shebang, running the global command directly (e.g., `shumai`) will execute under Node.js. If you installed with Bun and want to run under the Bun runtime, you must prefix all commands with `bun run --bun` (e.g., `bun run --bun shumai`, `bun run --bun shumai -d`, `bun run --bun shumai stop`).
|
||||
|
||||
Start the application from your workspace folder:
|
||||
|
||||
```bash
|
||||
shumai
|
||||
```
|
||||
|
||||
Alternatively, you can run and manage Shumai in daemon mode:
|
||||
|
||||
- **Start in daemon mode**:
|
||||
```bash
|
||||
shumai -d
|
||||
```
|
||||
- **Stop Shumai**:
|
||||
```bash
|
||||
shumai stop
|
||||
```
|
||||
- **Restart Shumai**:
|
||||
```bash
|
||||
shumai restart
|
||||
```
|
||||
- **Show/tail logs**:
|
||||
```bash
|
||||
shumai logs
|
||||
```
|
||||
|
||||
On startup, Shumai will automatically run database migrations and start the web server at `http://localhost:3000`.
|
||||
|
||||
---
|
||||
|
||||
### Option 3: Run from Source (Development)
|
||||
### Option 2: Run from Source (Development)
|
||||
|
||||
To set up Shumai locally for development:
|
||||
|
||||
@@ -322,8 +151,23 @@ To set up Shumai locally for development:
|
||||
cd shumai
|
||||
bun install
|
||||
```
|
||||
2. Start the `pgvector` database container (as described in **Option 2, Step 1**).
|
||||
3. Create a `.env` file at the root of the workspace using the configuration from **Option 2, Step 5**.
|
||||
2. Start the `pgvector` database container:
|
||||
```bash
|
||||
docker run --name shumai_postgres \
|
||||
-e POSTGRES_USER=shumai \
|
||||
-e POSTGRES_PASSWORD=shumai_password \
|
||||
-e POSTGRES_DB=shumai_db \
|
||||
-p 5432:5432 \
|
||||
-d pgvector/pgvector:pg18
|
||||
```
|
||||
3. Create a `.env` file at the root of the workspace:
|
||||
```env
|
||||
DATABASE_URL=postgresql://shumai:shumai_password@localhost:5432/shumai_db?schema=public
|
||||
BETTER_AUTH_SECRET=ySxs7DxzHDZBbeeHNPEwBuspYwipBqz5Gk5XdBjNhWw=
|
||||
STORAGE_BACKEND=local
|
||||
SHUMAI_SERVER_PORT=3000
|
||||
AWS_ENDPOINT_URL_S3=http://localhost:3000
|
||||
```
|
||||
4. Generate i18n translation files:
|
||||
```bash
|
||||
bun run i18n:compile
|
||||
@@ -339,6 +183,9 @@ To set up Shumai locally for development:
|
||||
bun run dev
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> Looking to install Shumai globally via NPM or your package manager? See our [NPM Installation Guide](https://docs.shumai.one/getting-started/quickstart).
|
||||
|
||||
---
|
||||
|
||||
## Command Line Interface (CLI)
|
||||
|
||||
+55
-221
@@ -35,6 +35,7 @@ Shumai 是一个开源的 Frame.io 替代方案,专为创意团队打造,让
|
||||
* **丰富的文件格式支持**:支持预览常见创意文件格式,包括图片(支持 PSD)、视频、音频和文档,同时可上传和存储任意类型的文件。
|
||||
* **基于转盘渲染服务的 3D 资产预览**:通过集成 [turntable-renderer](https://docs.shumai.one/using-shumai/3d-asset-support) 支持为 GLB、FBX、OBJ、STL、USD 和 DAE 等 3D 模型生成高质量的交互式转盘预览与视频。
|
||||
* **基于 Gotenberg 的文档 PDF 转码代理**:通过集成 [Gotenberg](https://docs.shumai.one/using-shumai/gotenberg) 支持将 Word、PowerPoint、Excel、Markdown、HTML 和 CSV 文档高质量转换为 PDF 代理文件进行在线预览。
|
||||
* **语义搜索**:支持按画面内容或概念含义搜索素材,底层基于向量嵌入技术。详见[相关文档](https://docs.shumai.one/using-shumai/semantic-search)。
|
||||
* **非编剪辑导出与 Premiere 插件(实验性)**:支持将时间戳审阅评论导出至常用非编软件(Premiere Pro XML、DaVinci Resolve EDL、Final Cut Pro、Avid Media Composer),或使用实验性的 Premiere Pro 原生 UXP 插件导入媒体素材并将评论自动同步为时间轴标记。详见[相关文档](https://docs.shumai.one/using-shumai/nle-and-premiere)。
|
||||
|
||||
但 Shumai 不仅仅只满足于 Frame.io 的开源替代。随着 GPT-Image-2、Seedance 2.5 这类强大模型的问世,AI已经生成产品级的图片/视频。很多时候,你只要给一句提示词,就能直接生成或修改图片和视频,而不用再打开 Photoshop、Premiere 或者 Houdini。也正因为这样,Shumai 才把 AI Agent 做成了核心能力:
|
||||
@@ -45,7 +46,6 @@ Shumai 是一个开源的 Frame.io 替代方案,专为创意团队打造,让
|
||||
* **Model Context Protocol (MCP) 支持**:通过远程 MCP 服务端为 Agent 接入外部工具与 API,支持 OAuth 鉴权与按 Agent 独立分配访问权限。
|
||||
* **沙箱隔离执行**:Agent 生成的脚本会在隔离环境里运行,更安全。
|
||||
* **AI 自动补全元数据**:新素材上传后,系统可以自动帮你补一些元数据。
|
||||
* **语义搜索**:支持按画面内容或语义找素材,底层用的是向量 embedding(Gemini Embedding 2)。
|
||||
* **资源配额**:支持按成员、角色或全团队对 AI Token、成本和工具调用设置使用上限与实时用量监控。
|
||||
|
||||
下面是一个简短的 demo 视频,展示了 Shumai Agent 能做什么:
|
||||
@@ -99,10 +99,22 @@ https://gh-proxy.com/https://raw.githubusercontent.com/shumaiOne/shumai/main/REA
|
||||
|
||||
```
|
||||
|
||||
2. 下载 `docker-compose.yaml` 配置文件:
|
||||
```bash
|
||||
curl -o docker-compose.yaml https://raw.githubusercontent.com/shumaiOne/shumai/main/docker-compose/local/docker-compose-aliyun.yaml
|
||||
```
|
||||
2. 根据您的需求和机器可用内存,下载对应的 `docker-compose.yaml` 配置文件:
|
||||
|
||||
* **精简版(建议内存:2 GB+)**:包含 Shumai 核心应用与 PostgreSQL 数据库。
|
||||
```bash
|
||||
curl -o docker-compose.yaml https://raw.githubusercontent.com/shumaiOne/shumai/main/docker-compose/local/docker-compose-aliyun.yaml
|
||||
```
|
||||
|
||||
* **含嵌入向量服务版(建议内存:4 GB+)**:新增基于 `shumaiOne/embedding-server` (`EmbeddingGemma 2 740M`) 的本地多模态语义搜索能力。
|
||||
```bash
|
||||
curl -o docker-compose.yaml https://raw.githubusercontent.com/shumaiOne/shumai/main/docker-compose/local/docker-compose-aliyun.embedding.yaml
|
||||
```
|
||||
|
||||
* **完整版(建议内存:8 GB+)**:一站式全功能套件,包含 `embedding-server`、`turntable-renderer`(3D 模型交互转盘预览)与 `Gotenberg`(文档与表格 PDF 代理转码)。
|
||||
```bash
|
||||
curl -o docker-compose.yaml https://raw.githubusercontent.com/shumaiOne/shumai/main/docker-compose/local/docker-compose-aliyun.full.yaml
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> 默认的 Docker Compose 配置使用基于 CPU 的软件转码。如果您的服务器配备了支持的显卡(NVIDIA NVENC、Intel QuickSync、AMD/Intel VA-API 或 Rockchip RKMPP)并需要开启 GPU 硬件加速转码,请参考 [硬件加速转码配置文档](https://docs.shumai.one/configuration/hardware-transcoding)。
|
||||
@@ -152,233 +164,55 @@ https://gh-proxy.com/https://raw.githubusercontent.com/shumaiOne/shumai/main/REA
|
||||
|
||||
5. 打开浏览器,访问 `http://localhost:3000` 即可开始使用(若是远程部署,请访问 `http://<您的服务器IP>:3000`)。
|
||||
|
||||
### 方案二:通过 NPM / 包管理器安装
|
||||
|
||||
通过 NPM 上的 `@shumai-one/shumai` 进行安装。
|
||||
|
||||
#### 第一步:启动带 pgvector 支持的 PostgreSQL
|
||||
|
||||
Shumai 依赖支持 `pgvector` 扩展的 PostgreSQL 数据库。您可以使用 Docker 快速启动一个预配置好的数据库容器:
|
||||
|
||||
```bash
|
||||
docker run --name shumai_postgres \
|
||||
-e POSTGRES_USER=shumai \
|
||||
-e POSTGRES_PASSWORD=shumai_password \
|
||||
-e POSTGRES_DB=shumai_db \
|
||||
-p 5432:5432 \
|
||||
-d docker.m.daocloud.io/pgvector/pgvector:pg18
|
||||
|
||||
```
|
||||
|
||||
#### 第二步:创建工作区目录
|
||||
|
||||
创建一个专属目录用来存放环境配置和媒体文件(媒体文件默认会保存在 `./data` 目录下):
|
||||
|
||||
```bash
|
||||
mkdir shumai && cd shumai
|
||||
|
||||
```
|
||||
|
||||
#### 步骤 3:安装平台依赖
|
||||
|
||||
#### Linux
|
||||
|
||||
在 Linux 上运行 Shumai 需要安装以下系统依赖:
|
||||
|
||||
- **`ffmpeg`** —— 用于媒体转码和元数据提取。
|
||||
- **`poppler`** (`poppler-utils`) —— 用于 PDF 页面图像提取和 PDF 雪碧图预览生成 (`pdftoppm`)。
|
||||
- **`imagemagick`** —— 用于 PSD 格式图像转码和 sRGB 色彩空间转换。
|
||||
- **`dcraw_emu`** (`libraw` / `libraw-bin`) —— 用于相机 RAW 格式图像解码和预览图生成。
|
||||
- **`bubblewrap`**、**`socat`** 和 **`ripgrep`** —— AI Agent 沙箱(`anthropic-experimental/sandbox-runtime`)所需,用于进程隔离、网络通信和工作区搜索。
|
||||
|
||||
可以使用以下命令一次性安装所有依赖:
|
||||
|
||||
> Ubuntu/Debian
|
||||
```bash
|
||||
sudo apt install -y ffmpeg poppler-utils imagemagick bubblewrap socat ripgrep libraw-bin
|
||||
```
|
||||
|
||||
> Fedora
|
||||
```bash
|
||||
sudo dnf install -y ffmpeg poppler-utils ImageMagick bubblewrap socat ripgrep LibRaw
|
||||
```
|
||||
|
||||
> Arch Linux
|
||||
```bash
|
||||
sudo pacman -S --noconfirm ffmpeg poppler imagemagick bubblewrap socat ripgrep libraw
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> **Ubuntu 24.04+** 默认限制非特权用户创建 User Namespace,这会导致 `anthropic-experimental/sandbox-runtime` 无法使用 `bubblewrap` 提供沙箱隔离。
|
||||
>
|
||||
> 如需临时启用 User Namespace,请执行:
|
||||
>
|
||||
> ```bash
|
||||
> sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
|
||||
> ```
|
||||
>
|
||||
> 或者,配置 AppArmor Profile,为相关可执行文件授予所需的 `userns`权限。
|
||||
|
||||
---
|
||||
|
||||
#### macOS
|
||||
|
||||
在 macOS 上运行 Shumai 需要:
|
||||
|
||||
- **`ffmpeg`** —— 用于媒体转码和元数据提取。
|
||||
- **`poppler`** —— 用于 PDF 页面图像提取和 PDF 雪碧图预览生成 (`pdftoppm`)。
|
||||
- **`imagemagick`** —— 用于 PSD 格式图像转码和 sRGB 色彩空间转换。
|
||||
- **`dcraw_emu`** (`libraw`) —— 用于相机 RAW 格式图像解码和预览图生成。
|
||||
- **`ripgrep`** —— AI Agent 沙箱(`anthropic-experimental/sandbox-runtime`)所需。
|
||||
|
||||
使用 Homebrew 安装所需依赖:
|
||||
|
||||
```bash
|
||||
brew install ffmpeg poppler imagemagick ripgrep libraw
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### Windows(Alpha)
|
||||
|
||||
> [!WARNING]
|
||||
> anthropic-experimental/sandbox-runtime 的 Windows 支持目前仍处于 **Alpha** 阶段,整体设计仍在持续调整中。
|
||||
>
|
||||
> 当前实现提供了以下能力:
|
||||
> - 通过 **Windows Filtering Platform (WFP)** 实现网络出口(egress)过滤。
|
||||
> - 通过 **ACL Stamping** 实现文件读写权限限制。
|
||||
>
|
||||
> **但这并不能作为针对恶意沙箱进程的安全边界(security boundary)**。
|
||||
>
|
||||
> 未来版本将对沙箱机制进行重大调整,届时沙箱进程将运行在独立的沙箱用户账户下。
|
||||
|
||||
在 Windows 上运行 Shumai 需要:
|
||||
|
||||
- **`ffmpeg`** —— 用于媒体转码和元数据提取。
|
||||
- **`imagemagick`** —— 用于 PSD 格式图像转码和 sRGB 色彩空间转换。
|
||||
|
||||
使用你偏好的包管理器安装所需依赖:
|
||||
|
||||
**winget**
|
||||
```powershell
|
||||
winget install Gyan.FFmpeg ImageMagick.ImageMagick
|
||||
```
|
||||
|
||||
**Chocolatey**
|
||||
```powershell
|
||||
choco install ffmpeg imagemagick
|
||||
```
|
||||
|
||||
#### 第四步:全局安装 Shumai
|
||||
|
||||
使用您常用的包管理器进行全局安装:
|
||||
|
||||
```bash
|
||||
# 使用 NPM
|
||||
npm install -g @shumai-one/shumai
|
||||
|
||||
# 使用 PNPM
|
||||
pnpm add -g @shumai-one/shumai
|
||||
|
||||
# 使用 Bun
|
||||
bun add -g @shumai-one/shumai
|
||||
|
||||
```
|
||||
|
||||
#### 第五步:配置环境配置
|
||||
|
||||
在刚才创建的工作区目录(`shumai/`)下新建一个 `.env` 文件,并填入以下配置:
|
||||
|
||||
```env
|
||||
DATABASE_URL=postgresql://shumai:shumai_password@localhost:5432/shumai_db?schema=public
|
||||
BETTER_AUTH_SECRET=ySxs7DxzHDZBbeeHNPEwBuspYwipBqz5Gk5XdBjNhWw=
|
||||
STORAGE_BACKEND=local
|
||||
SHUMAI_SERVER_PORT=3000
|
||||
AWS_ENDPOINT_URL_S3=http://localhost:3000
|
||||
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> `SHUMAI_SERVER_PORT` 用于设置 Web 服务的启动端口,而 `AWS_ENDPOINT_URL_S3` 供浏览器端构建文件上传链接。
|
||||
> 如果您部署在远程服务器(如 `http://123.456.7.8`)并且做了端口映射(如 Docker 中的 `12345:3000`),请将 `AWS_ENDPOINT_URL_S3` 设置为 `http://123.456.7.8:12345`。
|
||||
|
||||
#### 第六步:启动 Shumai
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **Bun 用户特别提示:** 由于发布的包二进制文件包含 `#!/usr/bin/env node` 头部声明(Shebang),直接在终端执行全局命令(例如 `shumai`)默认会通过 Node.js 运行。如果您是用 Bun 安装的并且希望调用 Bun 运行时,请在所有命令前加上 `bun run --bun` 前缀(例如:`bun run --bun shumai`、`bun run --bun shumai -d`、`bun run --bun shumai stop`)。
|
||||
|
||||
在您的工作区目录下执行以下命令启动应用:
|
||||
|
||||
```bash
|
||||
shumai
|
||||
|
||||
```
|
||||
|
||||
您也可以让 Shumai 在后台以守护进程(Daemon)模式运行和管理:
|
||||
|
||||
* **后台模式启动**:
|
||||
```bash
|
||||
shumai -d
|
||||
|
||||
```
|
||||
|
||||
|
||||
* **停止服务**:
|
||||
```bash
|
||||
shumai stop
|
||||
|
||||
```
|
||||
|
||||
|
||||
* **重启服务**:
|
||||
```bash
|
||||
shumai restart
|
||||
|
||||
```
|
||||
|
||||
|
||||
* **查看并跟踪实时日志**:
|
||||
```bash
|
||||
shumai logs
|
||||
|
||||
```
|
||||
|
||||
|
||||
|
||||
启动时,Shumai 会自动执行数据库迁移(Migration)并在 `http://localhost:3000` 拉起 Web 服务。
|
||||
|
||||
---
|
||||
|
||||
### 方案三:源码编译运行(开发环境)
|
||||
### 方案二:源码编译运行(开发环境)
|
||||
|
||||
如果您希望在本地参与 Shumai 的开发,请按以下步骤配置开发环境:
|
||||
|
||||
1. 克隆代码仓库并安装依赖项:
|
||||
```bash
|
||||
git clone [https://github.com/shumaiOne/shumai.git](https://github.com/shumaiOne/shumai.git)
|
||||
cd shumai
|
||||
bun install
|
||||
```bash
|
||||
git clone https://github.com/shumaiOne/shumai.git
|
||||
cd shumai
|
||||
bun install
|
||||
```
|
||||
|
||||
```
|
||||
2. 启动 `pgvector` 数据库容器:
|
||||
```bash
|
||||
docker run --name shumai_postgres \
|
||||
-e POSTGRES_USER=shumai \
|
||||
-e POSTGRES_PASSWORD=shumai_password \
|
||||
-e POSTGRES_DB=shumai_db \
|
||||
-p 5432:5432 \
|
||||
-d docker.m.daocloud.io/pgvector/pgvector:pg18
|
||||
```
|
||||
|
||||
3. 在项目根目录下创建一个 `.env` 文件,填入以下基础配置:
|
||||
```env
|
||||
DATABASE_URL=postgresql://shumai:shumai_password@localhost:5432/shumai_db?schema=public
|
||||
BETTER_AUTH_SECRET=ySxs7DxzHDZBbeeHNPEwBuspYwipBqz5Gk5XdBjNhWw=
|
||||
STORAGE_BACKEND=local
|
||||
SHUMAI_SERVER_PORT=3000
|
||||
AWS_ENDPOINT_URL_S3=http://localhost:3000
|
||||
```
|
||||
|
||||
2. 启动 `pgvector` 数据库容器(参考**方案二中的第一步**)。
|
||||
3. 参照**方案二中的第五步**,在项目根目录下创建一个 `.env` 文件。
|
||||
4. 生成多语言国际化(i18n)代码:
|
||||
```bash
|
||||
bun run i18n:compile
|
||||
```
|
||||
> [!TIP]
|
||||
> 在前端开发过程中,可以在单独的终端窗口中运行 `bun run i18n:watch`,在修改 `packages/webui/messages/` 下的翻译文案时自动重新编译。
|
||||
```bash
|
||||
bun run i18n:compile
|
||||
```
|
||||
> [!TIP]
|
||||
> 在前端开发过程中,可以在单独的终端窗口中运行 `bun run i18n:watch`,在修改 `packages/webui/messages/` 下的翻译文案时自动重新编译。
|
||||
|
||||
5. 执行数据库结构迁移:
|
||||
```bash
|
||||
bun run db:migrate
|
||||
```
|
||||
```bash
|
||||
bun run db:migrate
|
||||
```
|
||||
|
||||
6. 启动本地开发服务器:
|
||||
```bash
|
||||
bun run dev
|
||||
```
|
||||
```bash
|
||||
bun run dev
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> 如需通过 NPM 或包管理器全局安装运行 Shumai,请参考[官方 NPM 安装指南](https://docs.shumai.one/getting-started/quickstart)。
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@shumai/agent-app",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"bin": {
|
||||
|
||||
@@ -1,4 +1,3 @@
|
||||
export * from '@shumai/agent/src/workflows/agent-autofill'
|
||||
export * from '@shumai/agent/src/workflows/agent-chat'
|
||||
export * from '@shumai/agent/src/workflows/agent-embedding'
|
||||
export * from '@shumai/agent/src/workflows/query-embedding-for-search'
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@shumai/cli",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"bin": {
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@shumai/premiere-uxp",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"private": true,
|
||||
"type": "commonjs",
|
||||
"scripts": {
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@shumai/transcode-app",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"bin": {
|
||||
|
||||
@@ -19,6 +19,7 @@ async function start() {
|
||||
process.env.BETTER_AUTH_URL = 'http://localhost:5200'
|
||||
process.env.NODE_ENV = 'development'
|
||||
process.env.SHUMAI_E2E = 'true'
|
||||
process.env.ENABLE_EMBEDDING_AGENT = 'false'
|
||||
|
||||
// Write variables to .env.e2e file so the teardown can load them
|
||||
const envContent = [`DATABASE_URL=${process.env.DATABASE_URL}`, `S3_BUCKET=${bucketName}`].join(
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@shumai/web-app",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"dependencies": {
|
||||
|
||||
@@ -12,9 +12,11 @@ import { metadataService } from '@shumai/core/src/metadata/metadata'
|
||||
import { initTranscodeWorkflows } from '@shumai/transcode'
|
||||
import { workflowService } from '@shumai/workflow-core'
|
||||
import { migrateLegacyAgentAvatars } from '@shumai/core/src/agent/migration'
|
||||
import { agentService } from '@shumai/core/src/agent/agent'
|
||||
|
||||
import { handleDaemonCommands } from '@shumai/core/src/utils/daemon'
|
||||
import { authService } from '@shumai/core/src/auth/auth'
|
||||
import { embeddingService } from '@shumai/core/src/search/embedding-service'
|
||||
import { sandboxService } from '@shumai/core'
|
||||
import { notificationJobService } from '@shumai/core/src/notification/notification-job'
|
||||
|
||||
@@ -47,6 +49,29 @@ if (resetCmdIndex !== -1) {
|
||||
}
|
||||
}
|
||||
|
||||
const regenCmdIndex = cliArgs.findIndex((arg) => arg === 'regenerate-embeddings')
|
||||
|
||||
if (regenCmdIndex !== -1) {
|
||||
try {
|
||||
console.log('🔄 Regenerating embeddings...')
|
||||
const result = await embeddingService.regenerateEmbeddings()
|
||||
console.log(
|
||||
`✅ Successfully queued ${result.tasksCreated} embedding tasks for ${result.totalAssets} assets.`,
|
||||
)
|
||||
if (result.skippedTeams.length > 0) {
|
||||
console.warn(
|
||||
`⚠️ Skipped ${result.skippedTeams.length} team(s) without an active embedding agent.`,
|
||||
)
|
||||
}
|
||||
console.log('The background workers will process the embedding tasks.')
|
||||
process.exit(0)
|
||||
} catch (err: unknown) {
|
||||
const message = err instanceof Error ? err.message : 'Unknown error'
|
||||
console.error(`Error: ${message}`)
|
||||
process.exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
async function run() {
|
||||
// Initialize workflows and activities for local executor mode
|
||||
initAgentWorkflows()
|
||||
@@ -58,6 +83,7 @@ async function run() {
|
||||
// Start services
|
||||
await metadataService.syncSystemFields().catch(console.error)
|
||||
await migrateLegacyAgentAvatars().catch(console.error)
|
||||
await agentService.ensureSystemEmbeddingAgents().catch(console.error)
|
||||
assetService.startCleanupJob()
|
||||
notificationJobService.start()
|
||||
workflowService.start()
|
||||
|
||||
@@ -33,7 +33,7 @@
|
||||
},
|
||||
"apps/agent": {
|
||||
"name": "@shumai/agent-app",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"bin": {
|
||||
"shumai-agent": "./src/index.ts",
|
||||
},
|
||||
@@ -45,7 +45,7 @@
|
||||
},
|
||||
"apps/cli": {
|
||||
"name": "@shumai/cli",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"bin": {
|
||||
"shumai-cli": "./src/index.ts",
|
||||
},
|
||||
@@ -60,7 +60,7 @@
|
||||
},
|
||||
"apps/premiere-uxp": {
|
||||
"name": "@shumai/premiere-uxp",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"dependencies": {
|
||||
"@shumai/api": "workspace:*",
|
||||
"@shumai/dtos": "workspace:*",
|
||||
@@ -114,7 +114,7 @@
|
||||
},
|
||||
"apps/transcode": {
|
||||
"name": "@shumai/transcode-app",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"bin": {
|
||||
"shumai-transcode": "./src/index.ts",
|
||||
},
|
||||
@@ -126,7 +126,7 @@
|
||||
},
|
||||
"apps/web": {
|
||||
"name": "@shumai/web-app",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"dependencies": {
|
||||
"@shumai/agent": "workspace:*",
|
||||
"@shumai/api": "workspace:*",
|
||||
@@ -146,12 +146,12 @@
|
||||
},
|
||||
"packages/agent": {
|
||||
"name": "@shumai/agent",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"dependencies": {
|
||||
"@anthropic-ai/sandbox-runtime": "0.0.74",
|
||||
"@earendil-works/pi-agent-core": "0.82.1",
|
||||
"@earendil-works/pi-ai": "0.82.1",
|
||||
"@google/genai": "2.19.0",
|
||||
"@google/genai": "2.28.0",
|
||||
"@shumai/core": "workspace:*",
|
||||
"@shumai/db": "workspace:*",
|
||||
"@shumai/dtos": "workspace:*",
|
||||
@@ -161,6 +161,7 @@
|
||||
"@temporalio/workflow": "1.23.0",
|
||||
"adm-zip": "0.6.0",
|
||||
"jittered-fractional-indexing": "1.0.1",
|
||||
"sharp": "0.35.4",
|
||||
"typebox": "1.3.22",
|
||||
"ulid": "3.0.2",
|
||||
"zod": "4.5.4",
|
||||
@@ -171,7 +172,7 @@
|
||||
},
|
||||
"packages/api": {
|
||||
"name": "@shumai/api",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"dependencies": {
|
||||
"@hono/zod-validator": "0.7.6",
|
||||
"@shumai/core": "workspace:*",
|
||||
@@ -185,7 +186,7 @@
|
||||
},
|
||||
"packages/core": {
|
||||
"name": "@shumai/core",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"dependencies": {
|
||||
"@ai-sdk/alibaba": "2.0.38",
|
||||
"@ai-sdk/amazon-bedrock": "5.0.67",
|
||||
@@ -210,6 +211,7 @@
|
||||
"@aws-sdk/s3-request-presigner": "3.1121.0",
|
||||
"@earendil-works/pi-agent-core": "0.82.1",
|
||||
"@earendil-works/pi-ai": "0.82.1",
|
||||
"@google/genai": "2.28.0",
|
||||
"@modelcontextprotocol/client": "2.0.0",
|
||||
"@shumai/db": "workspace:*",
|
||||
"@shumai/dtos": "workspace:*",
|
||||
@@ -247,7 +249,7 @@
|
||||
},
|
||||
"packages/db": {
|
||||
"name": "@shumai/db",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"dependencies": {
|
||||
"@earendil-works/pi-agent-core": "0.82.1",
|
||||
"@prisma/adapter-pg": "7.10.0",
|
||||
@@ -263,7 +265,7 @@
|
||||
},
|
||||
"packages/dtos": {
|
||||
"name": "@shumai/dtos",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"dependencies": {
|
||||
"@earendil-works/pi-agent-core": "0.82.1",
|
||||
"zod": "4.5.4",
|
||||
@@ -271,10 +273,10 @@
|
||||
},
|
||||
"packages/e2e": {
|
||||
"name": "@shumai/e2e",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"dependencies": {
|
||||
"@earendil-works/pi-agent-core": "0.82.1",
|
||||
"@google/genai": "2.19.0",
|
||||
"@google/genai": "2.28.0",
|
||||
"@shumai/agent": "workspace:*",
|
||||
"@shumai/core": "workspace:*",
|
||||
"@shumai/db": "workspace:*",
|
||||
@@ -286,11 +288,11 @@
|
||||
},
|
||||
"packages/tableprinter": {
|
||||
"name": "@shumai/tableprinter",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
},
|
||||
"packages/transcode": {
|
||||
"name": "@shumai/transcode",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"dependencies": {
|
||||
"@shumai/core": "workspace:*",
|
||||
"@shumai/db": "workspace:*",
|
||||
@@ -304,7 +306,7 @@
|
||||
},
|
||||
"packages/webui": {
|
||||
"name": "@shumai/webui",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"dependencies": {
|
||||
"@base-ui/react": "1.7.0",
|
||||
"@dnd-kit/abstract": "0.5.0",
|
||||
@@ -406,7 +408,7 @@
|
||||
},
|
||||
"packages/workflow-core": {
|
||||
"name": "@shumai/workflow-core",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"dependencies": {
|
||||
"@shumai/core": "workspace:*",
|
||||
"@shumai/db": "workspace:*",
|
||||
@@ -652,7 +654,7 @@
|
||||
|
||||
"@floating-ui/utils": ["@floating-ui/utils@0.2.12", "", {}, "sha512-HpCo8tmWzLVad5s2d19EhAz5zqrrQ6s69qd6moPMQvkOuSwDT1YgRfWSVuc4ennqrgv3OHppiOGMQ7oC13yIww=="],
|
||||
|
||||
"@google/genai": ["@google/genai@2.19.0", "", { "dependencies": { "google-auth-library": "^10.3.0", "p-retry": "^4.6.2", "protobufjs": "^7.5.4", "ws": "^8.18.0" }, "peerDependencies": { "@modelcontextprotocol/sdk": "^1.25.2" }, "optionalPeers": ["@modelcontextprotocol/sdk"] }, "sha512-IM3u1FjpOLyOPsPh0smGA6Ntj2olUWCPbTs/WQj7cA/2AKCGw/652W7IaySx0XtwuscFnMYBoraebQusSZ0zSQ=="],
|
||||
"@google/genai": ["@google/genai@2.28.0", "", { "dependencies": { "google-auth-library": "^10.3.0", "p-retry": "^4.6.2", "protobufjs": "^7.5.4", "ws": "^8.18.0" }, "peerDependencies": { "@modelcontextprotocol/sdk": "^1.25.2" }, "optionalPeers": ["@modelcontextprotocol/sdk"] }, "sha512-hyNAJj+ywu3L46qlZod6KO/JvcT3QONz6VCiPevC/FWOjAD+LQb7DMLUoV8vzoLpt61Q0ruQX99iqbxEjPJWmw=="],
|
||||
|
||||
"@grpc/grpc-js": ["@grpc/grpc-js@1.14.4", "", { "dependencies": { "@grpc/proto-loader": "^0.8.0", "@js-sdsl/ordered-map": "^4.4.2" } }, "sha512-k9Dj3DV/itK9D06Y8f190Qgop7/Ui+D0njFV3LHMPwPT75DpXLQohE9Wmz0QElrJnzsjB7KPWiKJbOl7IPDArQ=="],
|
||||
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
services:
|
||||
postgres:
|
||||
image: docker.m.daocloud.io/pgvector/pgvector:pg18
|
||||
container_name: shumai_postgres
|
||||
environment:
|
||||
POSTGRES_USER: shumai
|
||||
POSTGRES_PASSWORD: shumai_password
|
||||
POSTGRES_DB: shumai_db
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/18/docker
|
||||
healthcheck:
|
||||
test: ['CMD-SHELL', 'pg_isready -U shumai -d shumai_db']
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
|
||||
embedding-server:
|
||||
image: shumaione/embedding-server:latest
|
||||
container_name: shumai_embedding
|
||||
restart: unless-stopped
|
||||
|
||||
shumai:
|
||||
image: crpi-000nkafzu6xvpwsc.cn-hongkong.personal.cr.aliyuncs.com/shumai-one/shumai:latest
|
||||
cap_add:
|
||||
- SYS_ADMIN
|
||||
security_opt:
|
||||
- apparmor=unconfined
|
||||
- seccomp=unconfined
|
||||
container_name: shumai_app
|
||||
ports:
|
||||
- '3000:3000'
|
||||
volumes:
|
||||
- ./data:/app/data
|
||||
# Hardware Acceleration (NVENC / QuickSync / VA-API / RKMPP)
|
||||
# Uncomment the extends section below for hardware acceleration:
|
||||
# extends:
|
||||
# file: hwaccel.transcoding.yaml
|
||||
# service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
|
||||
environment:
|
||||
# Database Configuration
|
||||
DATABASE_URL: postgres://shumai:shumai_password@shumai_postgres:5432/shumai_db?search_path=public&sslmode=disable
|
||||
|
||||
# Server Configuration
|
||||
SHUMAI_SERVER_PORT: 3000
|
||||
|
||||
# S3 Configuration
|
||||
STORAGE_BACKEND: local
|
||||
AWS_ENDPOINT_URL_S3: http://localhost:3000
|
||||
|
||||
# Better Auth
|
||||
BETTER_AUTH_SECRET: qSxs9DxzHDZBbeeTNPEwBuspYwipBqz5Gk5XdBjNhWw=
|
||||
|
||||
# Sandbox Configuration
|
||||
ENABLE_WEAKER_NESTED_SANDBOX: 'true'
|
||||
|
||||
# Embedding Configuration
|
||||
EMBEDDING_PROVIDER: gemma-embedding-2-openai
|
||||
GEMMA_EMBEDDING_OPENAI_BASE_URL: http://embedding-server:9379/v1
|
||||
ENABLE_EMBEDDING_AGENT: 'true'
|
||||
CONCURRENCY_AGENT: 1
|
||||
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
embedding-server:
|
||||
condition: service_started
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
@@ -0,0 +1,89 @@
|
||||
services:
|
||||
postgres:
|
||||
image: docker.m.daocloud.io/pgvector/pgvector:pg18
|
||||
container_name: shumai_postgres
|
||||
environment:
|
||||
POSTGRES_USER: shumai
|
||||
POSTGRES_PASSWORD: shumai_password
|
||||
POSTGRES_DB: shumai_db
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/18/docker
|
||||
healthcheck:
|
||||
test: ['CMD-SHELL', 'pg_isready -U shumai -d shumai_db']
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
|
||||
embedding-server:
|
||||
image: shumaione/embedding-server:latest
|
||||
container_name: shumai_embedding
|
||||
restart: unless-stopped
|
||||
|
||||
turntable-renderer:
|
||||
image: shumaione/turntable-renderer:latest
|
||||
container_name: shumai_turntable
|
||||
restart: unless-stopped
|
||||
|
||||
gotenberg:
|
||||
image: gotenberg/gotenberg:8
|
||||
container_name: shumai_gotenberg
|
||||
restart: unless-stopped
|
||||
|
||||
shumai:
|
||||
image: crpi-000nkafzu6xvpwsc.cn-hongkong.personal.cr.aliyuncs.com/shumai-one/shumai:latest
|
||||
cap_add:
|
||||
- SYS_ADMIN
|
||||
security_opt:
|
||||
- apparmor=unconfined
|
||||
- seccomp=unconfined
|
||||
container_name: shumai_app
|
||||
ports:
|
||||
- '3000:3000'
|
||||
volumes:
|
||||
- ./data:/app/data
|
||||
# Hardware Acceleration (NVENC / QuickSync / VA-API / RKMPP)
|
||||
# Uncomment the extends section below for hardware acceleration:
|
||||
# extends:
|
||||
# file: hwaccel.transcoding.yaml
|
||||
# service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
|
||||
environment:
|
||||
# Database Configuration
|
||||
DATABASE_URL: postgres://shumai:shumai_password@shumai_postgres:5432/shumai_db?search_path=public&sslmode=disable
|
||||
|
||||
# Server Configuration
|
||||
SHUMAI_SERVER_PORT: 3000
|
||||
|
||||
# S3 Configuration
|
||||
STORAGE_BACKEND: local
|
||||
AWS_ENDPOINT_URL_S3: http://localhost:3000
|
||||
|
||||
# Better Auth
|
||||
BETTER_AUTH_SECRET: qSxs9DxzHDZBbeeTNPEwBuspYwipBqz5Gk5XdBjNhWw=
|
||||
|
||||
# Sandbox Configuration
|
||||
ENABLE_WEAKER_NESTED_SANDBOX: 'true'
|
||||
|
||||
# Embedding Configuration
|
||||
EMBEDDING_PROVIDER: gemma-embedding-2-openai
|
||||
GEMMA_EMBEDDING_OPENAI_BASE_URL: http://embedding-server:9379/v1
|
||||
ENABLE_EMBEDDING_AGENT: 'true'
|
||||
CONCURRENCY_AGENT: 1
|
||||
|
||||
# 3D Turntable Renderer Configuration
|
||||
TURNTABLE_RENDERER_URL: http://turntable-renderer:3000
|
||||
|
||||
# Gotenberg Document Converter Configuration
|
||||
GOTENBERG_URL: http://gotenberg:3000
|
||||
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
embedding-server:
|
||||
condition: service_started
|
||||
turntable-renderer:
|
||||
condition: service_started
|
||||
gotenberg:
|
||||
condition: service_started
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
@@ -0,0 +1,69 @@
|
||||
services:
|
||||
postgres:
|
||||
image: pgvector/pgvector:pg18
|
||||
container_name: shumai_postgres
|
||||
environment:
|
||||
POSTGRES_USER: shumai
|
||||
POSTGRES_PASSWORD: shumai_password
|
||||
POSTGRES_DB: shumai_db
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/18/docker
|
||||
healthcheck:
|
||||
test: ['CMD-SHELL', 'pg_isready -U shumai -d shumai_db']
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
|
||||
embedding-server:
|
||||
image: shumaione/embedding-server:latest
|
||||
container_name: shumai_embedding
|
||||
restart: unless-stopped
|
||||
|
||||
shumai:
|
||||
image: shumaione/shumai:latest
|
||||
cap_add:
|
||||
- SYS_ADMIN
|
||||
security_opt:
|
||||
- apparmor=unconfined
|
||||
- seccomp=unconfined
|
||||
container_name: shumai_app
|
||||
ports:
|
||||
- '3000:3000'
|
||||
volumes:
|
||||
- ./data:/app/data
|
||||
# Hardware Acceleration (NVENC / QuickSync / VA-API / RKMPP)
|
||||
# Uncomment the extends section below for hardware acceleration:
|
||||
# extends:
|
||||
# file: hwaccel.transcoding.yaml
|
||||
# service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
|
||||
environment:
|
||||
# Database Configuration
|
||||
DATABASE_URL: postgres://shumai:shumai_password@shumai_postgres:5432/shumai_db?search_path=public&sslmode=disable
|
||||
|
||||
# Server Configuration
|
||||
SHUMAI_SERVER_PORT: 3000
|
||||
|
||||
# S3 Configuration
|
||||
STORAGE_BACKEND: local
|
||||
AWS_ENDPOINT_URL_S3: http://localhost:3000
|
||||
|
||||
# Better Auth
|
||||
BETTER_AUTH_SECRET: qSxs9DxzHDZBbeeTNPEwBuspYwipBqz5Gk5XdBjNhWw=
|
||||
|
||||
# Sandbox Configuration
|
||||
ENABLE_WEAKER_NESTED_SANDBOX: 'true'
|
||||
|
||||
# Embedding Configuration
|
||||
EMBEDDING_PROVIDER: gemma-embedding-2-openai
|
||||
GEMMA_EMBEDDING_OPENAI_BASE_URL: http://embedding-server:9379/v1
|
||||
ENABLE_EMBEDDING_AGENT: 'true'
|
||||
CONCURRENCY_AGENT: 1
|
||||
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
embedding-server:
|
||||
condition: service_started
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
@@ -0,0 +1,89 @@
|
||||
services:
|
||||
postgres:
|
||||
image: pgvector/pgvector:pg18
|
||||
container_name: shumai_postgres
|
||||
environment:
|
||||
POSTGRES_USER: shumai
|
||||
POSTGRES_PASSWORD: shumai_password
|
||||
POSTGRES_DB: shumai_db
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/18/docker
|
||||
healthcheck:
|
||||
test: ['CMD-SHELL', 'pg_isready -U shumai -d shumai_db']
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
|
||||
embedding-server:
|
||||
image: shumaione/embedding-server:latest
|
||||
container_name: shumai_embedding
|
||||
restart: unless-stopped
|
||||
|
||||
turntable-renderer:
|
||||
image: shumaione/turntable-renderer:latest
|
||||
container_name: shumai_turntable
|
||||
restart: unless-stopped
|
||||
|
||||
gotenberg:
|
||||
image: gotenberg/gotenberg:8
|
||||
container_name: shumai_gotenberg
|
||||
restart: unless-stopped
|
||||
|
||||
shumai:
|
||||
image: shumaione/shumai:latest
|
||||
cap_add:
|
||||
- SYS_ADMIN
|
||||
security_opt:
|
||||
- apparmor=unconfined
|
||||
- seccomp=unconfined
|
||||
container_name: shumai_app
|
||||
ports:
|
||||
- '3000:3000'
|
||||
volumes:
|
||||
- ./data:/app/data
|
||||
# Hardware Acceleration (NVENC / QuickSync / VA-API / RKMPP)
|
||||
# Uncomment the extends section below for hardware acceleration:
|
||||
# extends:
|
||||
# file: hwaccel.transcoding.yaml
|
||||
# service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
|
||||
environment:
|
||||
# Database Configuration
|
||||
DATABASE_URL: postgres://shumai:shumai_password@shumai_postgres:5432/shumai_db?search_path=public&sslmode=disable
|
||||
|
||||
# Server Configuration
|
||||
SHUMAI_SERVER_PORT: 3000
|
||||
|
||||
# S3 Configuration
|
||||
STORAGE_BACKEND: local
|
||||
AWS_ENDPOINT_URL_S3: http://localhost:3000
|
||||
|
||||
# Better Auth
|
||||
BETTER_AUTH_SECRET: qSxs9DxzHDZBbeeTNPEwBuspYwipBqz5Gk5XdBjNhWw=
|
||||
|
||||
# Sandbox Configuration
|
||||
ENABLE_WEAKER_NESTED_SANDBOX: 'true'
|
||||
|
||||
# Embedding Configuration
|
||||
EMBEDDING_PROVIDER: gemma-embedding-2-openai
|
||||
GEMMA_EMBEDDING_OPENAI_BASE_URL: http://embedding-server:9379/v1
|
||||
ENABLE_EMBEDDING_AGENT: 'true'
|
||||
CONCURRENCY_AGENT: 1
|
||||
|
||||
# 3D Turntable Renderer Configuration
|
||||
TURNTABLE_RENDERER_URL: http://turntable-renderer:3000
|
||||
|
||||
# Gotenberg Document Converter Configuration
|
||||
GOTENBERG_URL: http://gotenberg:3000
|
||||
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
embedding-server:
|
||||
condition: service_started
|
||||
turntable-renderer:
|
||||
condition: service_started
|
||||
gotenberg:
|
||||
condition: service_started
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
@@ -3,6 +3,24 @@ title: "Changelog"
|
||||
description: "Recent updates, new features, and improvements in Shumai"
|
||||
---
|
||||
|
||||
<Update label="v0.5.3" description="October 16, 2026">
|
||||
|
||||
### Added
|
||||
|
||||
* **cli**: Add administrative `shumai regenerate-embeddings` command to regenerate multimodal vector embeddings for all existing media files across active teams
|
||||
* **docker**: Provide three Docker Compose deployment tiers (Lite, with Embedding Server, and Full) with recommended memory specifications
|
||||
* **agent**: Save video chunk embeddings incrementally to the database as each chunk finishes, preserving completed progress on failure and resuming from where it left off on retry
|
||||
* **agent**: Add support for local LiteRT-LM Gemma Embedding 2 (`gemma-embedding-2-openai`) provider and environment-enforced read-only embedding agents with automatic 768-dimensional vector search indexing
|
||||
|
||||
### Fixed
|
||||
|
||||
### Changed
|
||||
|
||||
* **search**: Display search results in a responsive card grid with interactive hover video sprite scrubbing, semantic match chunk time range previews, and new-tab opening on click
|
||||
* **search**: Significantly improve semantic search response times by generating query text embeddings directly with in-memory caching instead of dispatching background tasks
|
||||
|
||||
</Update>
|
||||
|
||||
<Update label="v0.5.2" description="October 13, 2026">
|
||||
|
||||
### Added
|
||||
|
||||
@@ -66,12 +66,24 @@ For example, if you deploy Shumai on a remote server at `http://123.456.7.8` and
|
||||
|
||||
Configures whether background transcoding and AI agents are executed locally or offloaded to a Temporal cluster.
|
||||
|
||||
| Variable | Description | Example / Default |
|
||||
| :------------------------- | :------------------------------------------------------------------------------------------------- | :------------------------------ |
|
||||
| `WORKFLOW_EXECUTOR` | Engine choice. Use `local` for in-process database polling, or `temporal` for distributed workers. | `local` (default) or `temporal` |
|
||||
| `TEMPORAL_ADDRESS` | Host and port of the Temporal cluster gRPC service. | `localhost:7233` (default) |
|
||||
| `EMBEDDING_CHUNK_DURATION` | The duration in seconds for each video chunk when generating video embeddings. | `60.0` (default) |
|
||||
| `EMBEDDING_CHUNK_OVERLAP` | The duration in seconds to overlap video chunks when generating video embeddings. | `5.0` (default) |
|
||||
| Variable | Description | Example / Default |
|
||||
| :---------------------- | :------------------------------------------------------------------------------------------------- | :------------------------------ |
|
||||
| `WORKFLOW_EXECUTOR` | Engine choice. Use `local` for in-process database polling, or `temporal` for distributed workers. | `local` (default) or `temporal` |
|
||||
| `TEMPORAL_ADDRESS` | Host and port of the Temporal cluster gRPC service. | `localhost:7233` (default) |
|
||||
| `CONCURRENCY_AGENT` | Maximum concurrent AI tasks executed by workers. Set to `1` when running local `embedding-server`. | `5` (default) |
|
||||
| `CONCURRENCY_TRANSCODE` | Maximum concurrent media transcoding tasks executed by workers. | `1` (default) |
|
||||
|
||||
---
|
||||
|
||||
## Embedding Config
|
||||
|
||||
| Variable | Description | Example / Default |
|
||||
| :-------------------------------- | :------------------------------------------------------------------------------------------------------------ | :-------------------------------- |
|
||||
| `EMBEDDING_PROVIDER` | Multimodal embedding provider. Options: `gemini-embedding-2`, `gemma-embedding-2-openai`. | `gemini-embedding-2` (default) |
|
||||
| `GEMMA_EMBEDDING_OPENAI_BASE_URL` | Base URL of the OpenAI-compatible LiteRT-LM embedding server. | `http://embedding-server:9379/v1` |
|
||||
| `ENABLE_EMBEDDING_AGENT` | When set to `true`, automatically provisions and enables the system embedding agent for all teams on startup. | `false` (default) |
|
||||
| `EMBEDDING_CHUNK_DURATION` | The duration in seconds for each video chunk when generating video embeddings. | `60.0` (default) |
|
||||
| `EMBEDDING_CHUNK_OVERLAP` | The duration in seconds to overlap video chunks when generating video embeddings. | `5.0` (default) |
|
||||
|
||||
## Sandbox Config
|
||||
|
||||
|
||||
+2
-1
@@ -45,7 +45,8 @@
|
||||
"using-shumai/share-links",
|
||||
"using-shumai/nle-and-premiere",
|
||||
"using-shumai/3d-asset-support",
|
||||
"using-shumai/gotenberg"
|
||||
"using-shumai/gotenberg",
|
||||
"using-shumai/embedding-server"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
@@ -0,0 +1,131 @@
|
||||
---
|
||||
title: Embedding Server Setup (EmbeddingGemma 2)
|
||||
description: Run local, offline multimodal vector embeddings for semantic search using shumaiOne/embedding-server.
|
||||
---
|
||||
|
||||
Shumai supports local, self-hosted multimodal vector embeddings for semantic search through [shumaiOne/embedding-server](https://github.com/shumaiOne/embedding-server) — a standalone Docker service powered by Google AI Edge's **LiteRT-LM** and **EmbeddingGemma 2 740M**.
|
||||
|
||||
The embedding server provides a unified 768-dimensional vector space across text, images, and video frames with an OpenAI-compatible HTTP API (`/v1/embeddings`). Because the model weights are pre-baked directly into the Docker image, the server operates completely offline and air-gapped without requiring external network access or third-party cloud API keys.
|
||||
|
||||
---
|
||||
|
||||
## Model Architecture & Specs
|
||||
|
||||
| Property | Value |
|
||||
| :------------------------ | :------------------------------------------------------- |
|
||||
| **Model** | `EmbeddingGemma 2 740M` (`google/embeddinggemma-2-740m`) |
|
||||
| **Runtime** | Google AI Edge LiteRT-LM (Python 3.14 Debian slim) |
|
||||
| **Vector Dimensionality** | 768 dimensions (multimodal: text, image, and video) |
|
||||
| **Context Window** | 2,048 tokens |
|
||||
| **Network Mode** | Completely offline capable (`--network none`) |
|
||||
| **API Standard** | OpenAI-compatible `/v1/embeddings` |
|
||||
|
||||
---
|
||||
|
||||
## Video Processing & Sampling
|
||||
|
||||
For video assets, Shumai breaks videos into standardized segments and extracts keyframes:
|
||||
|
||||
- **Chunking**: Videos are divided into **60-second chunks** with a **5-second overlap** at segment boundaries to preserve narrative and visual context.
|
||||
- **Frame Extraction Rate**: Shumai extracts **15 frames per minute** from each chunk.
|
||||
- **Adaptive Frame Bounding**:
|
||||
- For standard aspect ratios (1:1, 4:3, 3:4, 16:9, 9:16, 21:9), Shumai extracts 15 frames, which produces ~1,982 prompt tokens and stays safely within the 2,048-token context window.
|
||||
- For non-standard or custom aspect ratios (e.g. 5:3, 1.4:1), Shumai automatically samples 14 frames (~1,990 tokens) to mathematically guarantee that prompt length never exceeds 2,048 tokens, preventing out-of-memory errors on the embedding server.
|
||||
|
||||
---
|
||||
|
||||
## Important: Changing Embedding Providers Breaks Existing Embeddings
|
||||
|
||||
> [!WARNING]
|
||||
> **Vector spaces from different embedding models are completely incompatible.**
|
||||
>
|
||||
> If you change your embedding provider (for example, switching from Google Gemini's 1536-dimensional embeddings to EmbeddingGemma's 768-dimensional embeddings, or vice versa), **all existing vector embeddings in your database become invalid**.
|
||||
>
|
||||
> To protect production databases from accidental corruption and search failures, **Shumai intentionally does not provide a WebUI toggle to change the embedding provider**. The provider can only be configured by an administrator via environment variables (`EMBEDDING_PROVIDER` and `GEMMA_EMBEDDING_OPENAI_BASE_URL`).
|
||||
|
||||
---
|
||||
|
||||
## Regenerating Embeddings via CLI
|
||||
|
||||
When you do switch embedding providers or configure the embedding server on an existing Shumai deployment, you can regenerate embeddings for all existing media files using the administrative CLI command:
|
||||
|
||||
### Inside Docker
|
||||
|
||||
```bash
|
||||
docker exec -it <shumai-container-name> shumai regenerate-embeddings
|
||||
```
|
||||
|
||||
### Local / Bare Metal
|
||||
|
||||
```bash
|
||||
shumai regenerate-embeddings
|
||||
```
|
||||
|
||||
_(Alternatively, `bun apps/web/src/index.ts regenerate-embeddings`)_
|
||||
|
||||
### What the Command Does
|
||||
|
||||
1. **Scans Teams**: Checks all teams in your workspace. If a team does not have an active (enabled) embedding agent, it logs a warning and safely skips that team.
|
||||
2. **Clears Existing Embeddings**: Purges all previous embeddings and cancels any pending/in-progress embedding tasks for eligible assets to ensure a clean transition to the new provider.
|
||||
3. **Enqueues Fresh Tasks**: Queues new background `ai_embedding` tasks for all eligible assets (images, videos, and 3D assets with rendered previews) across active teams.
|
||||
4. **Background Execution**: Background workers process the queued tasks according to your configured concurrency limits.
|
||||
|
||||
---
|
||||
|
||||
## Deployment & Configuration
|
||||
|
||||
### Docker Quickstart
|
||||
|
||||
To run the standalone embedding server container:
|
||||
|
||||
```bash
|
||||
docker run -d \
|
||||
--name shumai_embedding \
|
||||
-p 9379:9379 \
|
||||
shumaione/embedding-server:latest
|
||||
```
|
||||
|
||||
### Docker Compose Setup
|
||||
|
||||
In Docker Compose environments, add `embedding-server` to your `docker-compose.yaml` and configure Shumai to connect to it over Docker's internal network:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
embedding-server:
|
||||
image: shumaione/embedding-server:latest
|
||||
container_name: shumai_embedding
|
||||
restart: unless-stopped
|
||||
|
||||
shumai:
|
||||
image: shumaione/shumai:latest
|
||||
environment:
|
||||
# Embedding Server Configuration
|
||||
EMBEDDING_PROVIDER: gemma-embedding-2-openai
|
||||
GEMMA_EMBEDDING_OPENAI_BASE_URL: http://embedding-server:9379/v1
|
||||
ENABLE_EMBEDDING_AGENT: 'true'
|
||||
CONCURRENCY_AGENT: 1
|
||||
depends_on:
|
||||
- embedding-server
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> Since Shumai communicates with `embedding-server` over Docker's internal network, `embedding-server` does not need to expose any ports to the host machine.
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Required | Description | Default |
|
||||
| :-------------------------------- | :---------- | :---------------------------------------------------------------------------------------------------------------------------- | :------------------------- |
|
||||
| `EMBEDDING_PROVIDER` | Yes | Set to `gemma-embedding-2-openai` to use the local LiteRT-LM server. | `gemini-embedding-2` |
|
||||
| `GEMMA_EMBEDDING_OPENAI_BASE_URL` | Yes | HTTP base URL of the embedding server (e.g. `http://embedding-server:9379/v1`). | `http://localhost:9379/v1` |
|
||||
| `ENABLE_EMBEDDING_AGENT` | No | Automatically provisions and enables the system embedding agent for all teams on startup. | `false` |
|
||||
| `CONCURRENCY_AGENT` | Recommended | Maximum concurrent AI tasks executed by workers. **Set to `1`** when running `embedding-server` to prevent high memory usage. | `5` |
|
||||
|
||||
---
|
||||
|
||||
## Hardware & Resource Requirements
|
||||
|
||||
- **Memory (RAM)**:
|
||||
- We recommend at least **4 GB+ of dedicated system RAM** for the `embedding-server` container.
|
||||
- Overall host system memory should be **4 GB+** when running the Lite+Embedding compose stack, and **8 GB+** for the full stack.
|
||||
- **Concurrency**:
|
||||
- Always set `CONCURRENCY_AGENT=1` in your Shumai configuration when using `embedding-server`. Parallel multimodal embedding generations consume significant memory and can cause Docker containers to be terminated by the OS Out-Of-Memory (OOM) killer on low-resource machines.
|
||||
@@ -12,11 +12,10 @@ Traditional search relies on matching exact characters (e.g., searching "dog" wi
|
||||
Semantic search translates text queries and file contents into conceptual profiles. Similar ideas and visuals are mapped near each other.
|
||||
|
||||
1. **Index Generation**: When you upload an asset, Shumai's background agent scans the file's visual elements, text contents, or descriptions, translating them into a concept profile.
|
||||
- **Video Processing**: Shumai uses **gemini-embedding-2** for video indexing.
|
||||
- **Chunking**: For long videos, Shumai breaks them into chunks to ensure high-quality indexing. By default, chunks are **60 seconds** long with a **5-second overlap** to ensure no context is lost at the boundaries.
|
||||
- **Sampling**: The model processes a maximum of 32 frames per chunk: short chunks (≤32s) are sampled at 1 fps, while longer chunks are uniformly sampled to 32 frames.
|
||||
- **Optimization**: To optimize processing, Shumai automatically transcodes each chunk to **1 FPS** and **removes the audio track** before sending it to the embedding agent.
|
||||
- **Configuration**: Administrators can customize these settings using the `EMBEDDING_CHUNK_DURATION` and `EMBEDDING_CHUNK_OVERLAP` environment variables.
|
||||
- **Multimodal Embedding Providers**: Shumai supports two embedding providers:
|
||||
- **EmbeddingGemma 2 (Local)**: Powered by `shumaiOne/embedding-server` with 768-dimensional multimodal vectors running completely offline. Videos are sampled at **15 frames per minute** (14 frames for non-standard aspect ratios) per 60-second chunk. See the [Embedding Server Setup Guide](./embedding-server) for full details.
|
||||
- **Gemini Embedding 2 (Cloud)**: Powered by Google's `gemini-embedding-2` API with 1536-dimensional vectors. Videos are chunked into 60-second segments (sampled up to 32 frames per chunk at 1 fps with audio stripped).
|
||||
- **Chunking Configuration**: Administrators can customize chunk duration and overlap using the `EMBEDDING_CHUNK_DURATION` (default: 60s) and `EMBEDDING_CHUNK_OVERLAP` (default: 5s) environment variables.
|
||||
2. **Concept Matching**: When you search with the "Semantic" toggle enabled, Shumai matches the conceptual meaning of your query against your assets.
|
||||
3. **Result**: Searching "sunset at the beach" will successfully find a video file of ocean waves at dusk, even if the filename is `IMG_4029.mp4`.
|
||||
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "shumai",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"repository": {
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@shumai/agent",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "./src/index.ts",
|
||||
@@ -9,7 +9,7 @@
|
||||
"@anthropic-ai/sandbox-runtime": "0.0.74",
|
||||
"@earendil-works/pi-agent-core": "0.82.1",
|
||||
"@earendil-works/pi-ai": "0.82.1",
|
||||
"@google/genai": "2.19.0",
|
||||
"@google/genai": "2.28.0",
|
||||
"@shumai/core": "workspace:*",
|
||||
"@shumai/db": "workspace:*",
|
||||
"@shumai/dtos": "workspace:*",
|
||||
@@ -19,6 +19,7 @@
|
||||
"@temporalio/workflow": "1.23.0",
|
||||
"adm-zip": "0.6.0",
|
||||
"jittered-fractional-indexing": "1.0.1",
|
||||
"sharp": "0.35.4",
|
||||
"typebox": "1.3.22",
|
||||
"ulid": "3.0.2",
|
||||
"zod": "4.5.4"
|
||||
|
||||
@@ -1,17 +1,19 @@
|
||||
import { describe, it, expect, vi, beforeEach } from 'vitest'
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'
|
||||
import {
|
||||
generateTextEmbeddingActivity,
|
||||
extractAiMetadataActivity,
|
||||
getEmbeddingContextActivity,
|
||||
saveAssetEmbeddingsActivity,
|
||||
deleteAssetEmbeddingsActivity,
|
||||
generateImageEmbeddingActivity,
|
||||
generateVideoChunkEmbeddingActivity,
|
||||
generateEmbeddingActivity,
|
||||
getGemmaEmbeddingTargetFrameCount,
|
||||
} from './ai'
|
||||
import { s3Service } from '@shumai/core/src/s3/s3'
|
||||
import { transcodeService } from '@shumai/core'
|
||||
import { prisma, AssetType, AssetStatus } from '@shumai/db'
|
||||
import { setupTestDbHooks } from '@shumai/db/test'
|
||||
import sharp from 'sharp'
|
||||
|
||||
vi.mock('fs', async (importOriginal) => {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- importOriginal returns the raw fs module, typed dynamically as any here
|
||||
@@ -54,7 +56,6 @@ vi.mock('child_process', () => {
|
||||
const execFileMock: any = vi.fn((file: any, args: any, cb: any) => {
|
||||
cb(null, '120.0\n', '')
|
||||
})
|
||||
// Setup custom promisify behavior to return { stdout, stderr }
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- mock functions dynamically return varying types based on test command calls
|
||||
execFileMock[Symbol.for('nodejs.util.promisify.custom')] = vi.fn(async (file: any) => {
|
||||
if (file === 'ffprobe') {
|
||||
@@ -90,6 +91,14 @@ describe('AI Activities Unit Tests', () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
process.env.GEMINI_API_KEY = 'test-key'
|
||||
delete process.env.EMBEDDING_PROVIDER
|
||||
delete process.env.GEMMA_EMBEDDING_OPENAI_BASE_URL
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
vi.unstubAllGlobals()
|
||||
delete process.env.EMBEDDING_PROVIDER
|
||||
delete process.env.GEMMA_EMBEDDING_OPENAI_BASE_URL
|
||||
})
|
||||
|
||||
it('should generate image embedding', async () => {
|
||||
@@ -126,33 +135,6 @@ describe('AI Activities Unit Tests', () => {
|
||||
)
|
||||
})
|
||||
|
||||
it('should resolve video input via resolveInput and avoid downloadToFile in generateEmbeddingActivity', async () => {
|
||||
vi.mocked(s3Service.downloadToFile).mockClear()
|
||||
vi.mocked(s3Service.resolveInput).mockClear()
|
||||
vi.mocked(s3Service.resolveInput).mockResolvedValue(
|
||||
'https://mock-storage/files/asset-123/video.mp4',
|
||||
)
|
||||
|
||||
const res = await generateEmbeddingActivity({
|
||||
teamId: 't1',
|
||||
assetId: 'asset-123',
|
||||
context: {
|
||||
agent: {},
|
||||
asset: {
|
||||
mediaType: 'video/mp4',
|
||||
name: 'sample.mp4',
|
||||
storageKey: { key: 'files/asset-123/video.mp4' },
|
||||
media: { proxyType: 'video' },
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
expect(s3Service.resolveInput).toHaveBeenCalledWith('shumai', 'files/asset-123/video.mp4')
|
||||
expect(s3Service.downloadToFile).not.toHaveBeenCalled()
|
||||
expect(s3Service.getObject).not.toHaveBeenCalledWith('shumai', 'files/asset-123/video.mp4')
|
||||
expect(res.embeddings.length).toBeGreaterThan(0)
|
||||
})
|
||||
|
||||
it('should generate text embedding successfully', async () => {
|
||||
const res = await generateTextEmbeddingActivity({
|
||||
text: 'Search query',
|
||||
@@ -172,6 +154,356 @@ describe('AI Activities Unit Tests', () => {
|
||||
)
|
||||
})
|
||||
|
||||
it('should generate text embedding successfully with gemma-embedding-2-openai', async () => {
|
||||
process.env.EMBEDDING_PROVIDER = 'gemma-embedding-2-openai'
|
||||
const mockFetch = vi.fn().mockResolvedValue({
|
||||
ok: true,
|
||||
json: async () => ({
|
||||
model: 'embeddinggemma-2',
|
||||
data: [{ embedding: Array(768).fill(0.1) }],
|
||||
// eslint-disable-next-line @typescript-eslint/naming-convention
|
||||
usage: { prompt_tokens: 15 },
|
||||
}),
|
||||
})
|
||||
vi.stubGlobal('fetch', mockFetch)
|
||||
|
||||
const res = await generateTextEmbeddingActivity({
|
||||
text: 'Search query',
|
||||
teamId: 't1',
|
||||
})
|
||||
|
||||
expect(res.embedding).toHaveLength(768)
|
||||
expect(res.usage.model).toBe('embeddinggemma-2')
|
||||
expect(res.usage.inputTokens).toBe(15)
|
||||
expect(mockFetch).toHaveBeenCalledWith(
|
||||
'http://localhost:9379/v1/embeddings',
|
||||
expect.objectContaining({
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
model: 'embeddinggemma-2',
|
||||
input: 'task: search query | text: Search query',
|
||||
}),
|
||||
}),
|
||||
)
|
||||
})
|
||||
|
||||
it('should generate image embedding successfully with gemma-embedding-2-openai', async () => {
|
||||
process.env.EMBEDDING_PROVIDER = 'gemma-embedding-2-openai'
|
||||
const mockFetch = vi.fn().mockResolvedValue({
|
||||
ok: true,
|
||||
json: async () => ({
|
||||
model: 'embeddinggemma-2',
|
||||
data: [{ embedding: Array(768).fill(0.2) }],
|
||||
// eslint-disable-next-line @typescript-eslint/naming-convention
|
||||
usage: { prompt_tokens: 140 },
|
||||
}),
|
||||
})
|
||||
vi.stubGlobal('fetch', mockFetch)
|
||||
|
||||
vi.mocked(s3Service.getObject).mockResolvedValue({
|
||||
buffer: Buffer.from('test-image'),
|
||||
contentType: 'image/png',
|
||||
} as unknown as { buffer: Buffer; contentType: string })
|
||||
|
||||
const res = await generateImageEmbeddingActivity({
|
||||
teamId: 't1',
|
||||
assetKey: 'img.png',
|
||||
mediaType: 'image/png',
|
||||
})
|
||||
|
||||
expect(res.embedding).toHaveLength(768)
|
||||
expect(res.usage.model).toBe('embeddinggemma-2')
|
||||
expect(res.usage.inputTokens).toBe(140)
|
||||
expect(mockFetch).toHaveBeenCalledWith(
|
||||
'http://localhost:9379/v1/embeddings',
|
||||
expect.objectContaining({
|
||||
method: 'POST',
|
||||
body: JSON.stringify({
|
||||
model: 'embeddinggemma-2',
|
||||
input: [
|
||||
{
|
||||
type: 'image_url',
|
||||
// eslint-disable-next-line @typescript-eslint/naming-convention
|
||||
image_url: {
|
||||
url: `data:image/png;base64,${Buffer.from('test-image').toString('base64')}`,
|
||||
},
|
||||
},
|
||||
],
|
||||
}),
|
||||
}),
|
||||
)
|
||||
})
|
||||
|
||||
it('should respect custom GEMMA_EMBEDDING_OPENAI_BASE_URL with or without /v1 suffix', async () => {
|
||||
process.env.EMBEDDING_PROVIDER = 'gemma-embedding-2-openai'
|
||||
const mockFetch = vi.fn().mockResolvedValue({
|
||||
ok: true,
|
||||
json: async () => ({
|
||||
model: 'embeddinggemma-2',
|
||||
data: [{ embedding: Array(768).fill(0.2) }],
|
||||
// eslint-disable-next-line @typescript-eslint/naming-convention
|
||||
usage: { prompt_tokens: 10 },
|
||||
}),
|
||||
})
|
||||
vi.stubGlobal('fetch', mockFetch)
|
||||
|
||||
// Test with /v1
|
||||
process.env.GEMMA_EMBEDDING_OPENAI_BASE_URL = 'http://custom-host:9379/v1'
|
||||
await generateTextEmbeddingActivity({ text: 'test query', teamId: 't1' })
|
||||
expect(mockFetch).toHaveBeenLastCalledWith(
|
||||
'http://custom-host:9379/v1/embeddings',
|
||||
expect.anything(),
|
||||
)
|
||||
|
||||
// Test without /v1
|
||||
process.env.GEMMA_EMBEDDING_OPENAI_BASE_URL = 'http://custom-host:9379'
|
||||
await generateTextEmbeddingActivity({ text: 'test query', teamId: 't1' })
|
||||
expect(mockFetch).toHaveBeenLastCalledWith(
|
||||
'http://custom-host:9379/v1/embeddings',
|
||||
expect.anything(),
|
||||
)
|
||||
})
|
||||
|
||||
it('should convert webp image to jpeg with gemma-embedding-2-openai', async () => {
|
||||
process.env.EMBEDDING_PROVIDER = 'gemma-embedding-2-openai'
|
||||
const mockFetch = vi.fn().mockResolvedValue({
|
||||
ok: true,
|
||||
json: async () => ({
|
||||
model: 'embeddinggemma-2',
|
||||
data: [{ embedding: Array(768).fill(0.2) }],
|
||||
// eslint-disable-next-line @typescript-eslint/naming-convention
|
||||
usage: { prompt_tokens: 140 },
|
||||
}),
|
||||
})
|
||||
vi.stubGlobal('fetch', mockFetch)
|
||||
|
||||
const webpBuffer = await sharp({
|
||||
create: { width: 10, height: 10, channels: 3, background: { r: 255, g: 0, b: 0 } },
|
||||
})
|
||||
.webp()
|
||||
.toBuffer()
|
||||
vi.mocked(s3Service.getObject).mockResolvedValue({
|
||||
buffer: webpBuffer,
|
||||
contentType: 'image/webp',
|
||||
} as unknown as { buffer: Buffer; contentType: string })
|
||||
|
||||
const res = await generateImageEmbeddingActivity({
|
||||
teamId: 't1',
|
||||
assetKey: 'img.webp',
|
||||
mediaType: 'image/webp',
|
||||
})
|
||||
|
||||
expect(res.embedding).toHaveLength(768)
|
||||
expect(mockFetch).toHaveBeenCalledWith(
|
||||
'http://localhost:9379/v1/embeddings',
|
||||
expect.objectContaining({
|
||||
method: 'POST',
|
||||
body: expect.stringContaining('data:image/jpeg;base64,'),
|
||||
}),
|
||||
)
|
||||
})
|
||||
|
||||
it('should generate video chunk embedding from tar archive with gemma-embedding-2-openai', async () => {
|
||||
process.env.EMBEDDING_PROVIDER = 'gemma-embedding-2-openai'
|
||||
const mockFetch = vi.fn().mockResolvedValue({
|
||||
ok: true,
|
||||
json: async () => ({
|
||||
model: 'embeddinggemma-2',
|
||||
data: [{ embedding: Array(768).fill(0.3) }],
|
||||
// eslint-disable-next-line @typescript-eslint/naming-convention
|
||||
usage: { prompt_tokens: 140 },
|
||||
}),
|
||||
})
|
||||
vi.stubGlobal('fetch', mockFetch)
|
||||
|
||||
const sampleArchive = new Bun.Archive({
|
||||
'frame_0001.jpg': Buffer.from('frame-1'),
|
||||
})
|
||||
const archiveBytes = await sampleArchive.bytes()
|
||||
|
||||
vi.mocked(s3Service.getObject).mockResolvedValue({
|
||||
buffer: Buffer.from(archiveBytes.buffer, archiveBytes.byteOffset, archiveBytes.byteLength),
|
||||
contentType: 'application/x-tar',
|
||||
} as unknown as { buffer: Buffer; contentType: string })
|
||||
|
||||
const res = await generateVideoChunkEmbeddingActivity({
|
||||
teamId: 't1',
|
||||
chunkKey: 'files/asset-1/tmp-embedding-chunks/frames-0-60.tar',
|
||||
})
|
||||
|
||||
expect(res.embedding).toHaveLength(768)
|
||||
expect(res.usage.model).toBe('embeddinggemma-2')
|
||||
expect(mockFetch).toHaveBeenCalled()
|
||||
})
|
||||
|
||||
describe('getGemmaEmbeddingTargetFrameCount', () => {
|
||||
it('should return 15 for standard aspect ratios', () => {
|
||||
// 16:9
|
||||
expect(getGemmaEmbeddingTargetFrameCount(1920, 1080)).toBe(15)
|
||||
expect(getGemmaEmbeddingTargetFrameCount(1280, 720)).toBe(15)
|
||||
expect(getGemmaEmbeddingTargetFrameCount(854, 480)).toBe(15)
|
||||
// 9:16
|
||||
expect(getGemmaEmbeddingTargetFrameCount(1080, 1920)).toBe(15)
|
||||
// 1:1
|
||||
expect(getGemmaEmbeddingTargetFrameCount(1080, 1080)).toBe(15)
|
||||
expect(getGemmaEmbeddingTargetFrameCount(480, 480)).toBe(15)
|
||||
// 4:3
|
||||
expect(getGemmaEmbeddingTargetFrameCount(1440, 1080)).toBe(15)
|
||||
expect(getGemmaEmbeddingTargetFrameCount(640, 480)).toBe(15)
|
||||
// 3:4
|
||||
expect(getGemmaEmbeddingTargetFrameCount(1080, 1440)).toBe(15)
|
||||
// 21:9
|
||||
expect(getGemmaEmbeddingTargetFrameCount(2560, 1080)).toBe(15)
|
||||
expect(getGemmaEmbeddingTargetFrameCount(3440, 1440)).toBe(15)
|
||||
})
|
||||
|
||||
it('should return 14 for non-standard aspect ratios and missing/invalid dimensions', () => {
|
||||
expect(getGemmaEmbeddingTargetFrameCount(1000, 600)).toBe(14) // 5:3
|
||||
expect(getGemmaEmbeddingTargetFrameCount(700, 500)).toBe(14) // 1.4:1
|
||||
expect(getGemmaEmbeddingTargetFrameCount(854, 672)).toBe(14) // ~1.27:1
|
||||
expect(getGemmaEmbeddingTargetFrameCount(undefined, undefined)).toBe(14)
|
||||
expect(getGemmaEmbeddingTargetFrameCount(0, 0)).toBe(14)
|
||||
expect(getGemmaEmbeddingTargetFrameCount(-100, 100)).toBe(14)
|
||||
})
|
||||
})
|
||||
|
||||
it('should sample uniform 15 frames when chunk archive contains standard 16:9 frames', async () => {
|
||||
process.env.EMBEDDING_PROVIDER = 'gemma-embedding-2-openai'
|
||||
const mockFetch = vi.fn().mockResolvedValue({
|
||||
ok: true,
|
||||
json: async () => ({
|
||||
model: 'embeddinggemma-2',
|
||||
data: [{ embedding: Array(768).fill(0.3) }],
|
||||
// eslint-disable-next-line @typescript-eslint/naming-convention
|
||||
usage: { prompt_tokens: 140 },
|
||||
}),
|
||||
})
|
||||
vi.stubGlobal('fetch', mockFetch)
|
||||
|
||||
const sampleJpg = await sharp({
|
||||
create: {
|
||||
width: 160,
|
||||
height: 90,
|
||||
channels: 3,
|
||||
background: { r: 0, g: 0, b: 0 },
|
||||
},
|
||||
})
|
||||
.jpeg()
|
||||
.toBuffer()
|
||||
|
||||
const frames: Record<string, Buffer> = {}
|
||||
for (let i = 1; i <= 30; i++) {
|
||||
frames[`frame_${String(i).padStart(4, '0')}.jpg`] = sampleJpg
|
||||
}
|
||||
const sampleArchive = new Bun.Archive(frames)
|
||||
const archiveBytes = await sampleArchive.bytes()
|
||||
|
||||
vi.mocked(s3Service.getObject).mockResolvedValue({
|
||||
buffer: Buffer.from(archiveBytes.buffer, archiveBytes.byteOffset, archiveBytes.byteLength),
|
||||
contentType: 'application/x-tar',
|
||||
} as unknown as { buffer: Buffer; contentType: string })
|
||||
|
||||
const res = await generateVideoChunkEmbeddingActivity({
|
||||
teamId: 't1',
|
||||
chunkKey: 'files/asset-1/tmp-embedding-chunks/frames-0-60.tar',
|
||||
})
|
||||
|
||||
expect(res.embedding).toHaveLength(768)
|
||||
const requestBody = JSON.parse(mockFetch.mock.calls[0][1].body)
|
||||
expect(requestBody.input).toHaveLength(15)
|
||||
})
|
||||
|
||||
it('should fallback to 14 frames when chunk archive contains non-standard or unparseable frames', async () => {
|
||||
process.env.EMBEDDING_PROVIDER = 'gemma-embedding-2-openai'
|
||||
const mockFetch = vi.fn().mockResolvedValue({
|
||||
ok: true,
|
||||
json: async () => ({
|
||||
model: 'embeddinggemma-2',
|
||||
data: [{ embedding: Array(768).fill(0.3) }],
|
||||
// eslint-disable-next-line @typescript-eslint/naming-convention
|
||||
usage: { prompt_tokens: 140 },
|
||||
}),
|
||||
})
|
||||
vi.stubGlobal('fetch', mockFetch)
|
||||
|
||||
const frames: Record<string, Buffer> = {}
|
||||
for (let i = 1; i <= 30; i++) {
|
||||
frames[`frame_${String(i).padStart(4, '0')}.jpg`] = Buffer.from(`frame-${i}`)
|
||||
}
|
||||
const sampleArchive = new Bun.Archive(frames)
|
||||
const archiveBytes = await sampleArchive.bytes()
|
||||
|
||||
vi.mocked(s3Service.getObject).mockResolvedValue({
|
||||
buffer: Buffer.from(archiveBytes.buffer, archiveBytes.byteOffset, archiveBytes.byteLength),
|
||||
contentType: 'application/x-tar',
|
||||
} as unknown as { buffer: Buffer; contentType: string })
|
||||
|
||||
const res = await generateVideoChunkEmbeddingActivity({
|
||||
teamId: 't1',
|
||||
chunkKey: 'files/asset-1/tmp-embedding-chunks/frames-0-60.tar',
|
||||
})
|
||||
|
||||
expect(res.embedding).toHaveLength(768)
|
||||
const requestBody = JSON.parse(mockFetch.mock.calls[0][1].body)
|
||||
expect(requestBody.input).toHaveLength(14)
|
||||
})
|
||||
|
||||
it('should throw non-retryable ApplicationFailure when chunk archive contains no valid frames', async () => {
|
||||
process.env.EMBEDDING_PROVIDER = 'gemma-embedding-2-openai'
|
||||
const sampleArchive = new Bun.Archive({
|
||||
'info.txt': Buffer.from('not an image'),
|
||||
})
|
||||
const archiveBytes = await sampleArchive.bytes()
|
||||
|
||||
vi.mocked(s3Service.getObject).mockResolvedValue({
|
||||
buffer: Buffer.from(archiveBytes.buffer, archiveBytes.byteOffset, archiveBytes.byteLength),
|
||||
contentType: 'application/x-tar',
|
||||
} as unknown as { buffer: Buffer; contentType: string })
|
||||
|
||||
await expect(
|
||||
generateVideoChunkEmbeddingActivity({
|
||||
teamId: 't1',
|
||||
chunkKey: 'files/asset-1/tmp-embedding-chunks/frames-0-60.tar',
|
||||
}),
|
||||
).rejects.toThrow('No extracted frames found in chunk archive')
|
||||
})
|
||||
|
||||
it('should throw non-retryable ApplicationFailure when chunk archive data is corrupted', async () => {
|
||||
process.env.EMBEDDING_PROVIDER = 'gemma-embedding-2-openai'
|
||||
vi.mocked(s3Service.getObject).mockResolvedValue({
|
||||
buffer: Buffer.from('corrupted non-tar bytes'),
|
||||
contentType: 'application/x-tar',
|
||||
} as unknown as { buffer: Buffer; contentType: string })
|
||||
|
||||
await expect(
|
||||
generateVideoChunkEmbeddingActivity({
|
||||
teamId: 't1',
|
||||
chunkKey: 'files/asset-1/tmp-embedding-chunks/frames-0-60.tar',
|
||||
}),
|
||||
).rejects.toThrow('Failed to read chunk archive')
|
||||
})
|
||||
|
||||
it('should throw non-retryable ApplicationFailure when an extracted frame is empty (0 bytes)', async () => {
|
||||
process.env.EMBEDDING_PROVIDER = 'gemma-embedding-2-openai'
|
||||
const sampleArchive = new Bun.Archive({
|
||||
'frame_0001.jpg': Buffer.alloc(0),
|
||||
})
|
||||
const archiveBytes = await sampleArchive.bytes()
|
||||
|
||||
vi.mocked(s3Service.getObject).mockResolvedValue({
|
||||
buffer: Buffer.from(archiveBytes.buffer, archiveBytes.byteOffset, archiveBytes.byteLength),
|
||||
contentType: 'application/x-tar',
|
||||
} as unknown as { buffer: Buffer; contentType: string })
|
||||
|
||||
await expect(
|
||||
generateVideoChunkEmbeddingActivity({
|
||||
teamId: 't1',
|
||||
chunkKey: 'files/asset-1/tmp-embedding-chunks/frames-0-60.tar',
|
||||
}),
|
||||
).rejects.toThrow('is empty (0 bytes)')
|
||||
})
|
||||
|
||||
describe('extractAiMetadataActivity', () => {
|
||||
it('should return existing images if they are already in S3 storage', async () => {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- mock returned object cast as any due to partial implementation
|
||||
@@ -331,6 +663,18 @@ describe('AI Database Activities Integration', () => {
|
||||
expect(ctx.agent.id).toBe(user.id)
|
||||
expect(ctx.asset.id).toBe(asset.id)
|
||||
expect(ctx.asset.mediaType).toBe('video/mp4')
|
||||
|
||||
process.env.ENABLE_EMBEDDING_AGENT = 'false'
|
||||
try {
|
||||
await expect(
|
||||
getEmbeddingContextActivity({
|
||||
teamId: team.id,
|
||||
assetId: asset.id,
|
||||
}),
|
||||
).rejects.toThrow('embedding feature is disabled or agent not found')
|
||||
} finally {
|
||||
delete process.env.ENABLE_EMBEDDING_AGENT
|
||||
}
|
||||
})
|
||||
|
||||
it('should parse valid positive chunk duration and fallback on invalid or negative ones', async () => {
|
||||
@@ -389,6 +733,45 @@ describe('AI Database Activities Integration', () => {
|
||||
delete process.env.EMBEDDING_CHUNK_DURATION
|
||||
delete process.env.EMBEDDING_CHUNK_OVERLAP
|
||||
})
|
||||
|
||||
it('should return existing embeddings for the asset', async () => {
|
||||
// Ensure agent exists
|
||||
await prisma.agent.upsert({
|
||||
where: { id: user.id },
|
||||
update: {},
|
||||
create: {
|
||||
id: user.id,
|
||||
teamId: team.id,
|
||||
type: 'embedding',
|
||||
enabled: true,
|
||||
config: { provider: 'openai', model: 'gpt-4' },
|
||||
},
|
||||
})
|
||||
|
||||
await saveAssetEmbeddingsActivity({
|
||||
assetId: asset.id,
|
||||
embeddings: [
|
||||
{ embedding: Array(1536).fill(0.01), startTime: 0.0, endTime: 60.0 },
|
||||
{ embedding: Array(768).fill(0.02), startTime: 55.0, endTime: 115.0 },
|
||||
],
|
||||
})
|
||||
|
||||
const ctx = await getEmbeddingContextActivity({
|
||||
teamId: team.id,
|
||||
assetId: asset.id,
|
||||
})
|
||||
|
||||
expect(ctx.existingEmbeddings).toHaveLength(2)
|
||||
const chunk1 = ctx.existingEmbeddings.find((e) => e.startTime === 0)
|
||||
expect(chunk1).toBeDefined()
|
||||
expect(chunk1?.has1536).toBe(true)
|
||||
expect(chunk1?.has768).toBe(false)
|
||||
|
||||
const chunk2 = ctx.existingEmbeddings.find((e) => e.startTime === 55)
|
||||
expect(chunk2).toBeDefined()
|
||||
expect(chunk2?.has1536).toBe(false)
|
||||
expect(chunk2?.has768).toBe(true)
|
||||
})
|
||||
})
|
||||
|
||||
describe('saveAssetEmbeddingsActivity', () => {
|
||||
@@ -423,5 +806,85 @@ describe('AI Database Activities Integration', () => {
|
||||
expect(dbEmbeddings[0].endTime).toBe(60.0)
|
||||
expect(dbEmbeddings[1].startTime).toBeNull()
|
||||
})
|
||||
|
||||
it('should save 768-dim embeddings to embedding_768 column in database', async () => {
|
||||
const embeddings = [
|
||||
{
|
||||
embedding: Array(768).fill(0.01),
|
||||
startTime: 0.0,
|
||||
endTime: 60.0,
|
||||
},
|
||||
{
|
||||
embedding: Array(768).fill(0.02),
|
||||
},
|
||||
]
|
||||
|
||||
await saveAssetEmbeddingsActivity({
|
||||
assetId: asset.id,
|
||||
embeddings,
|
||||
})
|
||||
|
||||
// Query raw embeddings from DB
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- db query output is a raw postgres result array typed as any
|
||||
const dbEmbeddings: any[] = await prisma.$queryRaw`
|
||||
SELECT id, asset_id as "assetId", (embedding_768 IS NOT NULL) as "has768", (embedding IS NOT NULL) as "has1536"
|
||||
FROM asset_embeddings
|
||||
WHERE asset_id = ${asset.id}
|
||||
`
|
||||
|
||||
expect(dbEmbeddings.length).toBe(2)
|
||||
expect(dbEmbeddings[0].has768).toBe(true)
|
||||
expect(dbEmbeddings[0].has1536).toBe(false)
|
||||
expect(dbEmbeddings[1].has768).toBe(true)
|
||||
expect(dbEmbeddings[1].has1536).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
describe('deleteAssetEmbeddingsActivity', () => {
|
||||
it('should delete specific embeddings by id', async () => {
|
||||
await saveAssetEmbeddingsActivity({
|
||||
assetId: asset.id,
|
||||
embeddings: [
|
||||
{ embedding: Array(1536).fill(0.01), startTime: 0, endTime: 60 },
|
||||
{ embedding: Array(1536).fill(0.02), startTime: 55, endTime: 115 },
|
||||
],
|
||||
})
|
||||
|
||||
const before = await prisma.assetEmbedding.findMany({
|
||||
where: { assetId: asset.id },
|
||||
})
|
||||
expect(before).toHaveLength(2)
|
||||
|
||||
const targetId = before[0].id
|
||||
await deleteAssetEmbeddingsActivity({
|
||||
assetId: asset.id,
|
||||
ids: [targetId],
|
||||
})
|
||||
|
||||
const after = await prisma.assetEmbedding.findMany({
|
||||
where: { assetId: asset.id },
|
||||
})
|
||||
expect(after).toHaveLength(1)
|
||||
expect(after[0].id).not.toBe(targetId)
|
||||
})
|
||||
|
||||
it('should delete all embeddings for an asset when ids is not provided', async () => {
|
||||
await saveAssetEmbeddingsActivity({
|
||||
assetId: asset.id,
|
||||
embeddings: [
|
||||
{ embedding: Array(1536).fill(0.01), startTime: 0, endTime: 60 },
|
||||
{ embedding: Array(1536).fill(0.02), startTime: 55, endTime: 115 },
|
||||
],
|
||||
})
|
||||
|
||||
await deleteAssetEmbeddingsActivity({
|
||||
assetId: asset.id,
|
||||
})
|
||||
|
||||
const after = await prisma.assetEmbedding.findMany({
|
||||
where: { assetId: asset.id },
|
||||
})
|
||||
expect(after).toHaveLength(0)
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
+264
-171
@@ -1,17 +1,11 @@
|
||||
import { GoogleGenAI } from '@google/genai'
|
||||
import { s3Service } from '@shumai/core/src/s3/s3'
|
||||
import { transcodeService } from '@shumai/core'
|
||||
import { execFile } from 'child_process'
|
||||
import * as fs from 'fs'
|
||||
import * as os from 'os'
|
||||
import * as path from 'path'
|
||||
import { promisify } from 'util'
|
||||
import { ApplicationFailure } from '@temporalio/activity'
|
||||
import { prisma } from '@shumai/db'
|
||||
import { getProxyType } from '@shumai/core/src/utils/mime'
|
||||
import { ulid } from 'ulid'
|
||||
|
||||
const execFileAsync = promisify(execFile)
|
||||
import sharp from 'sharp'
|
||||
|
||||
export interface Usage {
|
||||
inputTokens: number
|
||||
@@ -25,15 +19,6 @@ export interface GeneratedEmbedding {
|
||||
endTime?: number
|
||||
}
|
||||
|
||||
export interface GenerateEmbeddingParams {
|
||||
teamId: string
|
||||
assetId: string
|
||||
context: {
|
||||
agent: unknown
|
||||
asset: unknown
|
||||
}
|
||||
}
|
||||
|
||||
async function generateMultimodalEmbedding(
|
||||
ai: GoogleGenAI,
|
||||
buffer: Buffer,
|
||||
@@ -62,134 +47,66 @@ async function generateMultimodalEmbedding(
|
||||
return values
|
||||
}
|
||||
|
||||
export async function generateEmbeddingActivity(params: GenerateEmbeddingParams): Promise<{
|
||||
embeddings: GeneratedEmbedding[]
|
||||
interface OpenAiEmbeddingItem {
|
||||
type?: string
|
||||
// eslint-disable-next-line @typescript-eslint/naming-convention
|
||||
image_url?: {
|
||||
url: string
|
||||
}
|
||||
}
|
||||
|
||||
async function callGemmaEmbeddingServer(input: string | OpenAiEmbeddingItem[]): Promise<{
|
||||
embedding: number[]
|
||||
usage: Usage
|
||||
}> {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- context properties are fetched as JSON from the DB activity
|
||||
const { asset } = params.context as any
|
||||
const rawBaseUrl = process.env.GEMMA_EMBEDDING_OPENAI_BASE_URL || 'http://localhost:9379/v1'
|
||||
|
||||
if (!asset.mediaType) {
|
||||
throw ApplicationFailure.create({ message: 'asset has no media type', nonRetryable: true })
|
||||
}
|
||||
const normalized = rawBaseUrl.replace(/\/+$/, '')
|
||||
const endpoint = normalized.endsWith('/embeddings')
|
||||
? normalized
|
||||
: normalized.endsWith('/v1')
|
||||
? `${normalized}/embeddings`
|
||||
: `${normalized}/v1/embeddings`
|
||||
|
||||
const apiKey = process.env.GEMINI_API_KEY
|
||||
if (!apiKey) {
|
||||
const response = await fetch(endpoint, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
model: 'embeddinggemma-2',
|
||||
input,
|
||||
}),
|
||||
})
|
||||
|
||||
if (!response.ok) {
|
||||
const errorText = await response.text()
|
||||
throw ApplicationFailure.create({
|
||||
message: 'GEMINI_API_KEY environment variable is not configured',
|
||||
nonRetryable: true,
|
||||
message: `Embedding server returned status ${response.status}: ${errorText}`,
|
||||
nonRetryable: false,
|
||||
})
|
||||
}
|
||||
|
||||
const resolvedApiKey = process.env[apiKey] || apiKey
|
||||
const ai = new GoogleGenAI({ apiKey: resolvedApiKey })
|
||||
|
||||
const usage: Usage = {
|
||||
model: 'gemini-embedding-2',
|
||||
inputTokens: 0,
|
||||
outputTokens: 0,
|
||||
}
|
||||
|
||||
const proxyType =
|
||||
(asset.media as PrismaJson.MediaInfo | null)?.proxyType ||
|
||||
getProxyType(asset.mediaType, asset.name)
|
||||
const isImage = proxyType === 'image'
|
||||
const isVideo = proxyType === 'video'
|
||||
|
||||
if (!isImage && !isVideo) {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- response json shape is dynamic from external endpoint
|
||||
const json: any = await response.json()
|
||||
const embedding = json.data?.[0]?.embedding
|
||||
if (!embedding || !Array.isArray(embedding)) {
|
||||
throw ApplicationFailure.create({
|
||||
message: `unsupported media type for embeddings: ${asset.mediaType}`,
|
||||
nonRetryable: true,
|
||||
message: 'Invalid embedding returned from embedding server',
|
||||
nonRetryable: false,
|
||||
})
|
||||
}
|
||||
|
||||
const key = asset.storageKey?.key
|
||||
if (!key) {
|
||||
throw ApplicationFailure.create({ message: 'asset has no key', nonRetryable: true })
|
||||
}
|
||||
|
||||
const bucket = process.env.S3_BUCKET || 'shumai'
|
||||
const results: GeneratedEmbedding[] = []
|
||||
|
||||
if (isImage) {
|
||||
const { buffer: data } = await s3Service.getObject(bucket, key)
|
||||
const embVec = await generateMultimodalEmbedding(ai, data, asset.mediaType)
|
||||
results.push({ embedding: embVec })
|
||||
} else if (isVideo) {
|
||||
const inputSource = await s3Service.resolveInput(bucket, key)
|
||||
const isRemote = inputSource.startsWith('http://') || inputSource.startsWith('https://')
|
||||
|
||||
const { stdout } = await execFileAsync('ffprobe', [
|
||||
'-v',
|
||||
'error',
|
||||
'-show_entries',
|
||||
'format=duration',
|
||||
'-of',
|
||||
'default=noprint_wrappers=1:nokey=1',
|
||||
inputSource,
|
||||
])
|
||||
const duration = parseFloat(stdout.trim())
|
||||
if (isNaN(duration)) {
|
||||
throw ApplicationFailure.create({
|
||||
message: 'failed to parse video duration',
|
||||
nonRetryable: true,
|
||||
})
|
||||
}
|
||||
|
||||
const chunkSize = 60.0
|
||||
for (let start = 0.0; start < duration; start += chunkSize) {
|
||||
let end = start + chunkSize
|
||||
if (end > duration) end = duration
|
||||
|
||||
const chunkTmp = path.join(os.tmpdir(), `video-chunk-${Date.now()}-${ulid()}.mp4`)
|
||||
try {
|
||||
await execFileAsync('ffmpeg', [
|
||||
'-y',
|
||||
'-loglevel',
|
||||
'warning',
|
||||
...(isRemote
|
||||
? ['-reconnect', '1', '-reconnect_streamed', '1', '-reconnect_delay_max', '5']
|
||||
: []),
|
||||
'-ss',
|
||||
start.toString(),
|
||||
'-i',
|
||||
inputSource,
|
||||
'-map',
|
||||
'0:V:0',
|
||||
'-map',
|
||||
'0:a:0?',
|
||||
'-t',
|
||||
(end - start).toString(),
|
||||
'-c',
|
||||
'copy',
|
||||
chunkTmp,
|
||||
])
|
||||
|
||||
const chunkData = fs.readFileSync(chunkTmp)
|
||||
|
||||
try {
|
||||
const embVec = await generateMultimodalEmbedding(ai, chunkData, 'video/mp4')
|
||||
results.push({ embedding: embVec, startTime: start, endTime: end })
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
} catch (e: any) {
|
||||
console.error(
|
||||
`failed to generate video embedding for chunk ${start}-${end}: ${e.message}`,
|
||||
)
|
||||
}
|
||||
} finally {
|
||||
if (fs.existsSync(chunkTmp)) {
|
||||
fs.unlinkSync(chunkTmp)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
embeddings: results,
|
||||
usage,
|
||||
embedding,
|
||||
usage: {
|
||||
model: json.model || 'gemma-embedding-2-openai',
|
||||
inputTokens: json.usage?.prompt_tokens ?? 0,
|
||||
outputTokens: 0,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
import { textEmbeddingService } from '@shumai/core/src/search/text-embedding'
|
||||
|
||||
export interface GenerateTextEmbeddingParams {
|
||||
text: string
|
||||
teamId: string
|
||||
@@ -199,44 +116,36 @@ export async function generateTextEmbeddingActivity(params: GenerateTextEmbeddin
|
||||
embedding: number[]
|
||||
usage: Usage
|
||||
}> {
|
||||
const apiKey = process.env.GEMINI_API_KEY
|
||||
if (!apiKey) {
|
||||
const provider = process.env.EMBEDDING_PROVIDER || 'gemini-embedding-2'
|
||||
if (provider !== 'gemma-embedding-2-openai' && !process.env.GEMINI_API_KEY) {
|
||||
throw ApplicationFailure.create({
|
||||
message: 'GEMINI_API_KEY environment variable is not configured',
|
||||
nonRetryable: true,
|
||||
})
|
||||
}
|
||||
|
||||
const resolvedApiKey = process.env[apiKey] || apiKey
|
||||
const ai = new GoogleGenAI({ apiKey: resolvedApiKey })
|
||||
try {
|
||||
const result = await textEmbeddingService.generateEmbedding(params.text, {
|
||||
teamId: params.teamId,
|
||||
skipCache: true,
|
||||
})
|
||||
|
||||
const usage: Usage = {
|
||||
model: 'gemini-embedding-2',
|
||||
inputTokens: 0,
|
||||
outputTokens: 0,
|
||||
}
|
||||
|
||||
const response = await ai.models.embedContent({
|
||||
model: 'gemini-embedding-2',
|
||||
contents: [
|
||||
{
|
||||
parts: [{ text: params.text }],
|
||||
role: 'user',
|
||||
return {
|
||||
embedding: result.embedding,
|
||||
usage: {
|
||||
model: result.usage?.model || provider,
|
||||
inputTokens: result.usage?.inputTokens ?? 0,
|
||||
outputTokens: result.usage?.outputTokens ?? 0,
|
||||
},
|
||||
],
|
||||
config: {
|
||||
outputDimensionality: 1536,
|
||||
},
|
||||
})
|
||||
|
||||
const values = response.embeddings?.[0]?.values
|
||||
if (!values) {
|
||||
throw new Error('No embedding values returned from gemini-embedding-2')
|
||||
}
|
||||
|
||||
return {
|
||||
embedding: values,
|
||||
usage,
|
||||
}
|
||||
} catch (err) {
|
||||
if (err instanceof ApplicationFailure) {
|
||||
throw err
|
||||
}
|
||||
throw ApplicationFailure.create({
|
||||
message: err instanceof Error ? err.message : String(err),
|
||||
nonRetryable: false,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -315,7 +224,7 @@ export async function getEmbeddingContextActivity(params: { teamId: string; asse
|
||||
user: { teamMembers: { some: { teamId: params.teamId } } },
|
||||
},
|
||||
})
|
||||
if (!agent) {
|
||||
if (!agent || process.env.ENABLE_EMBEDDING_AGENT === 'false') {
|
||||
throw ApplicationFailure.create({
|
||||
message: 'embedding feature is disabled or agent not found',
|
||||
nonRetryable: true,
|
||||
@@ -339,12 +248,51 @@ export async function getEmbeddingContextActivity(params: { teamId: string; asse
|
||||
|
||||
const parsedOverlap = parseFloat(process.env.EMBEDDING_CHUNK_OVERLAP || '')
|
||||
const chunkOverlap = Number.isFinite(parsedOverlap) && parsedOverlap >= 0 ? parsedOverlap : 5.0
|
||||
const embeddingProvider = process.env.EMBEDDING_PROVIDER || 'gemini-embedding-2'
|
||||
|
||||
const existingEmbeddings = await prisma.$queryRaw<ExistingAssetEmbeddingInfo[]>`
|
||||
SELECT
|
||||
id,
|
||||
start_time as "startTime",
|
||||
end_time as "endTime",
|
||||
(embedding IS NOT NULL) as "has1536",
|
||||
(embedding_768 IS NOT NULL) as "has768"
|
||||
FROM asset_embeddings
|
||||
WHERE asset_id = ${params.assetId}
|
||||
`
|
||||
|
||||
return {
|
||||
agent,
|
||||
asset,
|
||||
chunkDuration,
|
||||
chunkOverlap,
|
||||
embeddingProvider,
|
||||
existingEmbeddings,
|
||||
}
|
||||
}
|
||||
|
||||
export interface ExistingAssetEmbeddingInfo {
|
||||
id: string
|
||||
startTime: number | null
|
||||
endTime: number | null
|
||||
has1536: boolean
|
||||
has768: boolean
|
||||
}
|
||||
|
||||
export async function deleteAssetEmbeddingsActivity(params: { assetId: string; ids?: string[] }) {
|
||||
if (params.ids && params.ids.length > 0) {
|
||||
await prisma.assetEmbedding.deleteMany({
|
||||
where: {
|
||||
assetId: params.assetId,
|
||||
id: { in: params.ids },
|
||||
},
|
||||
})
|
||||
} else {
|
||||
await prisma.assetEmbedding.deleteMany({
|
||||
where: {
|
||||
assetId: params.assetId,
|
||||
},
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -358,16 +306,32 @@ export async function saveAssetEmbeddingsActivity(params: {
|
||||
}) {
|
||||
for (const item of params.embeddings) {
|
||||
const embVec = JSON.stringify(item.embedding)
|
||||
const is768 = item.embedding.length === 768
|
||||
|
||||
if (item.startTime !== undefined && item.endTime !== undefined) {
|
||||
await prisma.$executeRaw`
|
||||
INSERT INTO asset_embeddings (id, asset_id, embedding, start_time, end_time, updated_at)
|
||||
VALUES (gen_random_uuid()::text, ${params.assetId}, ${embVec}::vector, ${item.startTime}, ${item.endTime}, NOW())
|
||||
`
|
||||
if (is768) {
|
||||
await prisma.$executeRaw`
|
||||
INSERT INTO asset_embeddings (id, asset_id, embedding_768, start_time, end_time, updated_at)
|
||||
VALUES (gen_random_uuid()::text, ${params.assetId}, ${embVec}::vector, ${item.startTime}, ${item.endTime}, NOW())
|
||||
`
|
||||
} else {
|
||||
await prisma.$executeRaw`
|
||||
INSERT INTO asset_embeddings (id, asset_id, embedding, start_time, end_time, updated_at)
|
||||
VALUES (gen_random_uuid()::text, ${params.assetId}, ${embVec}::vector, ${item.startTime}, ${item.endTime}, NOW())
|
||||
`
|
||||
}
|
||||
} else {
|
||||
await prisma.$executeRaw`
|
||||
INSERT INTO asset_embeddings (id, asset_id, embedding, updated_at)
|
||||
VALUES (gen_random_uuid()::text, ${params.assetId}, ${embVec}::vector, NOW())
|
||||
`
|
||||
if (is768) {
|
||||
await prisma.$executeRaw`
|
||||
INSERT INTO asset_embeddings (id, asset_id, embedding_768, updated_at)
|
||||
VALUES (gen_random_uuid()::text, ${params.assetId}, ${embVec}::vector, NOW())
|
||||
`
|
||||
} else {
|
||||
await prisma.$executeRaw`
|
||||
INSERT INTO asset_embeddings (id, asset_id, embedding, updated_at)
|
||||
VALUES (gen_random_uuid()::text, ${params.assetId}, ${embVec}::vector, NOW())
|
||||
`
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -384,6 +348,30 @@ export async function generateImageEmbeddingActivity(
|
||||
embedding: number[]
|
||||
usage: Usage
|
||||
}> {
|
||||
const provider = process.env.EMBEDDING_PROVIDER || 'gemini-embedding-2'
|
||||
const { buffer: data } = await s3Service.getObject(
|
||||
process.env.S3_BUCKET || 'shumai',
|
||||
params.assetKey,
|
||||
)
|
||||
|
||||
if (provider === 'gemma-embedding-2-openai') {
|
||||
let imgData = data
|
||||
let mimeType = params.mediaType || 'image/jpeg'
|
||||
if (mimeType !== 'image/jpeg' && mimeType !== 'image/png') {
|
||||
imgData = await sharp(data).jpeg().toBuffer()
|
||||
mimeType = 'image/jpeg'
|
||||
}
|
||||
return callGemmaEmbeddingServer([
|
||||
{
|
||||
type: 'image_url',
|
||||
// eslint-disable-next-line @typescript-eslint/naming-convention
|
||||
image_url: {
|
||||
url: `data:${mimeType};base64,${imgData.toString('base64')}`,
|
||||
},
|
||||
},
|
||||
])
|
||||
}
|
||||
|
||||
const apiKey = process.env.GEMINI_API_KEY
|
||||
if (!apiKey) {
|
||||
throw ApplicationFailure.create({
|
||||
@@ -401,10 +389,6 @@ export async function generateImageEmbeddingActivity(
|
||||
outputTokens: 0,
|
||||
}
|
||||
|
||||
const { buffer: data } = await s3Service.getObject(
|
||||
process.env.S3_BUCKET || 'shumai',
|
||||
params.assetKey,
|
||||
)
|
||||
const embVec = await generateMultimodalEmbedding(ai, data, params.mediaType)
|
||||
|
||||
return {
|
||||
@@ -413,6 +397,38 @@ export async function generateImageEmbeddingActivity(
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Standard aspect ratios for video embedding:
|
||||
* 1:1, 4:3, 3:4, 16:9, 9:16, 21:9 (including ultrawide 64:27), and 9:21.
|
||||
*
|
||||
* For these standard ratios, 15 frames guarantees that total input tokens stay strictly
|
||||
* within the 2048 token signature bucket (<= 1982 tokens). For non-standard or custom
|
||||
* aspect ratios (e.g. 5:3, 1.4:1), 14 frames is used to mathematically guarantee
|
||||
* that sequence length never exceeds 2048 tokens (<= 1990 tokens), preventing OOM errors.
|
||||
*/
|
||||
export function getGemmaEmbeddingTargetFrameCount(width?: number, height?: number): number {
|
||||
if (!width || !height || width <= 0 || height <= 0) {
|
||||
return 14
|
||||
}
|
||||
|
||||
const ratio = width / height
|
||||
const standardRatios = [
|
||||
1 / 1, // 1:1
|
||||
4 / 3, // 4:3
|
||||
3 / 4, // 3:4
|
||||
16 / 9, // 16:9
|
||||
9 / 16, // 9:16
|
||||
21 / 9, // 21:9
|
||||
64 / 27, // 21:9 ultrawide (2560x1080, 3440x1440)
|
||||
9 / 21, // 9:21
|
||||
27 / 64, // 9:21 ultrawide portrait
|
||||
]
|
||||
|
||||
// Allow up to 3% relative tolerance for minor encoder rounding (e.g. 854x480 vs 16:9)
|
||||
const isStandard = standardRatios.some((std) => Math.abs(ratio - std) / std <= 0.03)
|
||||
return isStandard ? 15 : 14
|
||||
}
|
||||
|
||||
export interface GenerateVideoChunkEmbeddingParams {
|
||||
teamId: string
|
||||
chunkKey: string
|
||||
@@ -424,6 +440,83 @@ export async function generateVideoChunkEmbeddingActivity(
|
||||
embedding: number[]
|
||||
usage: Usage
|
||||
}> {
|
||||
const provider = process.env.EMBEDDING_PROVIDER || 'gemini-embedding-2'
|
||||
const isGemma = provider === 'gemma-embedding-2-openai' || params.chunkKey.endsWith('.tar')
|
||||
|
||||
if (isGemma) {
|
||||
const { buffer: data } = await s3Service.getObject(
|
||||
process.env.S3_BUCKET || 'shumai',
|
||||
params.chunkKey,
|
||||
)
|
||||
|
||||
let filesMap: Map<string, File>
|
||||
try {
|
||||
const archive = new Bun.Archive(data)
|
||||
filesMap = await archive.files()
|
||||
} catch (err) {
|
||||
throw ApplicationFailure.create({
|
||||
message: `Failed to read chunk archive ${params.chunkKey}: ${err instanceof Error ? err.message : String(err)}`,
|
||||
nonRetryable: true,
|
||||
})
|
||||
}
|
||||
|
||||
const frameFiles = Array.from(filesMap.keys())
|
||||
.filter((f) => /\.(jpe?g|png|webp)$/i.test(f))
|
||||
.sort()
|
||||
|
||||
if (frameFiles.length === 0) {
|
||||
throw ApplicationFailure.create({
|
||||
message: `No extracted frames found in chunk archive ${params.chunkKey}`,
|
||||
nonRetryable: true,
|
||||
})
|
||||
}
|
||||
|
||||
let targetFrameCount = 14
|
||||
try {
|
||||
const firstFile = filesMap.get(frameFiles[0])
|
||||
if (firstFile) {
|
||||
const firstFileBytes = await firstFile.bytes()
|
||||
if (firstFileBytes.byteLength > 0) {
|
||||
const meta = await sharp(Buffer.from(firstFileBytes)).metadata()
|
||||
targetFrameCount = getGemmaEmbeddingTargetFrameCount(meta.width, meta.height)
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
targetFrameCount = 14
|
||||
}
|
||||
|
||||
let selectedFrames = frameFiles
|
||||
if (selectedFrames.length > targetFrameCount) {
|
||||
const step = (selectedFrames.length - 1) / (targetFrameCount - 1)
|
||||
selectedFrames = Array.from(
|
||||
{ length: targetFrameCount },
|
||||
(_, i) => selectedFrames[Math.round(i * step)],
|
||||
)
|
||||
}
|
||||
|
||||
const input: OpenAiEmbeddingItem[] = await Promise.all(
|
||||
selectedFrames.map(async (f) => {
|
||||
const file = filesMap.get(f)!
|
||||
const fileBytes = await file.bytes()
|
||||
if (fileBytes.byteLength === 0) {
|
||||
throw ApplicationFailure.create({
|
||||
message: `Extracted frame ${f} in chunk archive ${params.chunkKey} is empty (0 bytes)`,
|
||||
nonRetryable: true,
|
||||
})
|
||||
}
|
||||
return {
|
||||
type: 'image_url',
|
||||
// eslint-disable-next-line @typescript-eslint/naming-convention
|
||||
image_url: {
|
||||
url: `data:image/jpeg;base64,${Buffer.from(fileBytes).toString('base64')}`,
|
||||
},
|
||||
}
|
||||
}),
|
||||
)
|
||||
|
||||
return await callGemmaEmbeddingServer(input)
|
||||
}
|
||||
|
||||
const apiKey = process.env.GEMINI_API_KEY
|
||||
if (!apiKey) {
|
||||
throw ApplicationFailure.create({
|
||||
|
||||
@@ -610,11 +610,9 @@ import * as aiActivities from './activities/ai'
|
||||
import { agentAutofillMedia } from './workflows/agent-autofill'
|
||||
import { agentChat } from './workflows/agent-chat'
|
||||
import { agentEmbeddingMedia } from './workflows/agent-embedding'
|
||||
import { queryEmbeddingForSearch } from './workflows/query-embedding-for-search'
|
||||
|
||||
export function initAgentWorkflows() {
|
||||
registerWorkflow(WorkflowTaskType.ai_embedding, agentEmbeddingMedia)
|
||||
registerWorkflow(WorkflowTaskType.query_embedding_for_search, queryEmbeddingForSearch)
|
||||
registerWorkflow(WorkflowTaskType.ai_metadata_autofill, agentAutofillMedia)
|
||||
registerWorkflow(WorkflowTaskType.chat, agentChat)
|
||||
|
||||
@@ -625,21 +623,21 @@ export function initAgentWorkflows() {
|
||||
export * from './activities/agent'
|
||||
export {
|
||||
extractAiMetadataActivity,
|
||||
generateEmbeddingActivity,
|
||||
generateImageEmbeddingActivity,
|
||||
generateTextEmbeddingActivity,
|
||||
generateVideoChunkEmbeddingActivity,
|
||||
saveAssetEmbeddingsActivity,
|
||||
deleteAssetEmbeddingsActivity,
|
||||
type ExtractAiMetadataParams,
|
||||
type GeneratedEmbedding,
|
||||
type GenerateEmbeddingParams,
|
||||
type GenerateTextEmbeddingParams,
|
||||
type ExistingAssetEmbeddingInfo,
|
||||
} from './activities/ai'
|
||||
export * from './database-session-storage'
|
||||
export * from './context/serialize-context'
|
||||
export * from './workflows/agent-autofill'
|
||||
export * from './workflows/agent-chat'
|
||||
export * from './workflows/agent-embedding'
|
||||
export * from './workflows/query-embedding-for-search'
|
||||
export * from './tools/generate-image'
|
||||
export * from './tools/generate-video'
|
||||
export * from './tools/rename-asset'
|
||||
|
||||
@@ -38,6 +38,9 @@ describe('Agent Embedding Workflow', () => {
|
||||
saveAssetEmbeddingsActivity: Object.assign(vi.fn(), {
|
||||
_activityName: 'saveAssetEmbeddingsActivity',
|
||||
}),
|
||||
deleteAssetEmbeddingsActivity: Object.assign(vi.fn(), {
|
||||
_activityName: 'deleteAssetEmbeddingsActivity',
|
||||
}),
|
||||
updateTaskUsageActivity: Object.assign(vi.fn(), { _activityName: 'updateTaskUsageActivity' }),
|
||||
createCommentActivity: Object.assign(vi.fn(), { _activityName: 'createCommentActivity' }),
|
||||
updateCommentActivity: Object.assign(vi.fn(), { _activityName: 'updateCommentActivity' }),
|
||||
@@ -314,18 +317,21 @@ describe('Agent Embedding Workflow', () => {
|
||||
assetKey: 'test-transcoded.mp4',
|
||||
startTime: 0,
|
||||
endTime: 60,
|
||||
format: 'video',
|
||||
})
|
||||
expect(mockActivities.transcodeVideoChunkActivity).toHaveBeenNthCalledWith(2, {
|
||||
assetId: 'a1',
|
||||
assetKey: 'test-transcoded.mp4',
|
||||
startTime: 55,
|
||||
endTime: 115,
|
||||
format: 'video',
|
||||
})
|
||||
expect(mockActivities.transcodeVideoChunkActivity).toHaveBeenNthCalledWith(3, {
|
||||
assetId: 'a1',
|
||||
assetKey: 'test-transcoded.mp4',
|
||||
startTime: 110,
|
||||
endTime: 150,
|
||||
format: 'video',
|
||||
})
|
||||
|
||||
// Verify video chunk embedding generation (3 chunks) on agent queue
|
||||
@@ -357,14 +363,19 @@ describe('Agent Embedding Workflow', () => {
|
||||
model: 'gemini-embedding-2',
|
||||
})
|
||||
|
||||
// Verify embeddings saved
|
||||
expect(mockActivities.saveAssetEmbeddingsActivity).toHaveBeenCalledWith({
|
||||
// Verify embeddings saved per chunk immediately
|
||||
expect(mockActivities.saveAssetEmbeddingsActivity).toHaveBeenCalledTimes(3)
|
||||
expect(mockActivities.saveAssetEmbeddingsActivity).toHaveBeenNthCalledWith(1, {
|
||||
assetId: 'a1',
|
||||
embeddings: [
|
||||
{ embedding: [0.3, 0.4], startTime: 0, endTime: 60 },
|
||||
{ embedding: [0.3, 0.4], startTime: 55, endTime: 115 },
|
||||
{ embedding: [0.3, 0.4], startTime: 110, endTime: 150 },
|
||||
],
|
||||
embeddings: [{ embedding: [0.3, 0.4], startTime: 0, endTime: 60 }],
|
||||
})
|
||||
expect(mockActivities.saveAssetEmbeddingsActivity).toHaveBeenNthCalledWith(2, {
|
||||
assetId: 'a1',
|
||||
embeddings: [{ embedding: [0.3, 0.4], startTime: 55, endTime: 115 }],
|
||||
})
|
||||
expect(mockActivities.saveAssetEmbeddingsActivity).toHaveBeenNthCalledWith(3, {
|
||||
assetId: 'a1',
|
||||
embeddings: [{ embedding: [0.3, 0.4], startTime: 110, endTime: 150 }],
|
||||
})
|
||||
})
|
||||
|
||||
@@ -450,6 +461,49 @@ describe('Agent Embedding Workflow', () => {
|
||||
})
|
||||
})
|
||||
|
||||
it('should save completed chunks to DB immediately before subsequent chunk failure', async () => {
|
||||
mockActivities.getEmbeddingContextActivity.mockResolvedValue({
|
||||
agent: { id: 'b1' },
|
||||
asset: {
|
||||
id: 'a1',
|
||||
mediaType: 'video/mp4',
|
||||
storageKey: { key: 'test.mp4' },
|
||||
media: {
|
||||
duration: 150.0,
|
||||
videoTranscodes: [{ key: 'test-transcoded.mp4' }],
|
||||
},
|
||||
},
|
||||
chunkDuration: 60.0,
|
||||
chunkOverlap: 5.0,
|
||||
})
|
||||
|
||||
// Chunk 1 succeeds, Chunk 2 fails
|
||||
mockActivities.generateVideoChunkEmbeddingActivity
|
||||
.mockResolvedValueOnce({
|
||||
embedding: [0.1, 0.2],
|
||||
usage: { inputTokens: 3, outputTokens: 3, model: 'gemini-embedding-2' },
|
||||
})
|
||||
.mockRejectedValueOnce(new Error('Rate limit exceeded at chunk 2'))
|
||||
|
||||
const task = await prisma.workflowTask.create({
|
||||
data: {
|
||||
type: 'ai_embedding',
|
||||
status: 'pending',
|
||||
assetId: 'a1',
|
||||
teamId: 't1',
|
||||
},
|
||||
})
|
||||
|
||||
await expect(agentEmbeddingMedia(task)).rejects.toThrow('Rate limit exceeded at chunk 2')
|
||||
|
||||
// Chunk 1 MUST have been saved to the database before chunk 2 failed
|
||||
expect(mockActivities.saveAssetEmbeddingsActivity).toHaveBeenCalledTimes(1)
|
||||
expect(mockActivities.saveAssetEmbeddingsActivity).toHaveBeenCalledWith({
|
||||
assetId: 'a1',
|
||||
embeddings: [{ embedding: [0.1, 0.2], startTime: 0, endTime: 60 }],
|
||||
})
|
||||
})
|
||||
|
||||
it('should not create redundant chunks when video exactly matches chunk duration', async () => {
|
||||
mockActivities.getEmbeddingContextActivity.mockResolvedValue({
|
||||
agent: { id: 'b1' },
|
||||
@@ -485,4 +539,137 @@ describe('Agent Embedding Workflow', () => {
|
||||
}),
|
||||
)
|
||||
})
|
||||
|
||||
it('should run agent embedding workflow for video with gemma-embedding-2-openai requesting frames format', async () => {
|
||||
mockActivities.getEmbeddingContextActivity.mockResolvedValue({
|
||||
agent: { id: 'b1' },
|
||||
asset: {
|
||||
id: 'a1',
|
||||
mediaType: 'video/mp4',
|
||||
storageKey: { key: 'test.mp4' },
|
||||
media: {
|
||||
duration: 30.0,
|
||||
},
|
||||
},
|
||||
chunkDuration: 60.0,
|
||||
chunkOverlap: 5.0,
|
||||
embeddingProvider: 'gemma-embedding-2-openai',
|
||||
})
|
||||
|
||||
const task = await prisma.workflowTask.create({
|
||||
data: {
|
||||
type: 'ai_embedding',
|
||||
status: 'pending',
|
||||
assetId: 'a1',
|
||||
teamId: 't1',
|
||||
},
|
||||
})
|
||||
|
||||
await agentEmbeddingMedia(task)
|
||||
|
||||
expect(mockActivities.transcodeVideoChunkActivity).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
startTime: 0,
|
||||
endTime: 30,
|
||||
format: 'frames',
|
||||
}),
|
||||
)
|
||||
expect(mockActivities.updateTaskUsageActivity).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
model: 'gemma-embedding-2-openai',
|
||||
}),
|
||||
)
|
||||
})
|
||||
|
||||
it('should skip already completed chunks when resuming a video embedding task', async () => {
|
||||
mockActivities.getEmbeddingContextActivity.mockResolvedValue({
|
||||
agent: { id: 'b1' },
|
||||
asset: {
|
||||
id: 'a1',
|
||||
mediaType: 'video/mp4',
|
||||
storageKey: { key: 'test.mp4' },
|
||||
media: {
|
||||
duration: 150.0,
|
||||
videoTranscodes: [{ key: 'test-transcoded.mp4' }],
|
||||
},
|
||||
},
|
||||
chunkDuration: 60.0,
|
||||
chunkOverlap: 5.0,
|
||||
existingEmbeddings: [
|
||||
{
|
||||
id: 'emb-1',
|
||||
startTime: 0,
|
||||
endTime: 60,
|
||||
has1536: true,
|
||||
has768: false,
|
||||
},
|
||||
],
|
||||
})
|
||||
|
||||
const task = await prisma.workflowTask.create({
|
||||
data: {
|
||||
type: 'ai_embedding',
|
||||
status: 'pending',
|
||||
assetId: 'a1',
|
||||
teamId: 't1',
|
||||
},
|
||||
})
|
||||
|
||||
await agentEmbeddingMedia(task)
|
||||
|
||||
// Chunk 1 (0-60) should be skipped; only chunk 2 (55-115) and chunk 3 (110-150) should be transcoded and embedded
|
||||
expect(mockActivities.transcodeVideoChunkActivity).toHaveBeenCalledTimes(2)
|
||||
expect(mockActivities.transcodeVideoChunkActivity).toHaveBeenNthCalledWith(
|
||||
1,
|
||||
expect.objectContaining({ startTime: 55, endTime: 115 }),
|
||||
)
|
||||
expect(mockActivities.transcodeVideoChunkActivity).toHaveBeenNthCalledWith(
|
||||
2,
|
||||
expect.objectContaining({ startTime: 110, endTime: 150 }),
|
||||
)
|
||||
|
||||
expect(mockActivities.saveAssetEmbeddingsActivity).toHaveBeenCalledTimes(2)
|
||||
})
|
||||
|
||||
it('should purge incompatible embeddings before processing', async () => {
|
||||
mockActivities.getEmbeddingContextActivity.mockResolvedValue({
|
||||
agent: { id: 'b1' },
|
||||
asset: {
|
||||
id: 'a1',
|
||||
mediaType: 'image/png',
|
||||
storageKey: { key: 'test.png' },
|
||||
},
|
||||
embeddingProvider: 'gemma-embedding-2-openai',
|
||||
existingEmbeddings: [
|
||||
{
|
||||
id: 'old-gemini-emb',
|
||||
startTime: null,
|
||||
endTime: null,
|
||||
has1536: true,
|
||||
has768: false,
|
||||
},
|
||||
],
|
||||
})
|
||||
|
||||
const task = await prisma.workflowTask.create({
|
||||
data: {
|
||||
type: 'ai_embedding',
|
||||
status: 'pending',
|
||||
assetId: 'a1',
|
||||
teamId: 't1',
|
||||
},
|
||||
})
|
||||
|
||||
await agentEmbeddingMedia(task)
|
||||
|
||||
// Should purge incompatible old-gemini-emb
|
||||
expect(mockActivities.deleteAssetEmbeddingsActivity).toHaveBeenCalledWith({
|
||||
assetId: 'a1',
|
||||
ids: ['old-gemini-emb'],
|
||||
})
|
||||
|
||||
// Should generate and save new embedding
|
||||
expect(mockActivities.generateImageEmbeddingActivity).toHaveBeenCalled()
|
||||
expect(mockActivities.saveAssetEmbeddingsActivity).toHaveBeenCalled()
|
||||
})
|
||||
})
|
||||
|
||||
@@ -13,6 +13,14 @@ export interface GeneratedEmbedding {
|
||||
endTime?: number
|
||||
}
|
||||
|
||||
export interface ExistingAssetEmbeddingInfo {
|
||||
id: string
|
||||
startTime: number | null
|
||||
endTime: number | null
|
||||
has1536: boolean
|
||||
has768: boolean
|
||||
}
|
||||
|
||||
export async function agentEmbeddingMedia(task: WorkflowTask): Promise<void> {
|
||||
const {
|
||||
updateTaskStatusActivity,
|
||||
@@ -20,6 +28,7 @@ export async function agentEmbeddingMedia(task: WorkflowTask): Promise<void> {
|
||||
generateImageEmbeddingActivity,
|
||||
generateVideoChunkEmbeddingActivity,
|
||||
saveAssetEmbeddingsActivity,
|
||||
deleteAssetEmbeddingsActivity,
|
||||
updateTaskUsageActivity,
|
||||
createCommentActivity,
|
||||
updateCommentActivity,
|
||||
@@ -62,13 +71,40 @@ export async function agentEmbeddingMedia(task: WorkflowTask): Promise<void> {
|
||||
}
|
||||
|
||||
// 1. Fetch Agent Context
|
||||
const context = await executeActivity(agentWorkerQueue, getEmbeddingContextActivity, {
|
||||
const context = (await executeActivity(agentWorkerQueue, getEmbeddingContextActivity, {
|
||||
teamId: task.teamId,
|
||||
assetId: task.assetId,
|
||||
})
|
||||
})) as unknown as {
|
||||
asset: {
|
||||
id: string
|
||||
mediaType: string | null
|
||||
storageKey?: { key: string } | null
|
||||
media?: unknown
|
||||
}
|
||||
chunkDuration: number
|
||||
chunkOverlap: number
|
||||
embeddingProvider: string
|
||||
existingEmbeddings?: ExistingAssetEmbeddingInfo[]
|
||||
}
|
||||
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const { asset, chunkDuration, chunkOverlap } = context as any
|
||||
const { asset, chunkDuration, chunkOverlap, embeddingProvider, existingEmbeddings } = context
|
||||
const isGemmaProvider = embeddingProvider === 'gemma-embedding-2-openai'
|
||||
|
||||
const existing = existingEmbeddings || []
|
||||
|
||||
// Purge incompatible embeddings (e.g. 1536d when Gemma is used, or 768d when Gemini is used)
|
||||
const incompatibleIds = existing
|
||||
.filter((e) => (isGemmaProvider ? e.has1536 || !e.has768 : e.has768 || !e.has1536))
|
||||
.map((e) => e.id)
|
||||
|
||||
if (incompatibleIds.length > 0) {
|
||||
await executeActivity(agentWorkerQueue, deleteAssetEmbeddingsActivity, {
|
||||
assetId: task.assetId,
|
||||
ids: incompatibleIds,
|
||||
})
|
||||
}
|
||||
|
||||
const validExistingChunks = existing.filter((e) => !incompatibleIds.includes(e.id))
|
||||
|
||||
const mediaInfo = asset.media as PrismaJson.MediaInfo | null
|
||||
const is3dWithViews =
|
||||
@@ -113,23 +149,30 @@ export async function agentEmbeddingMedia(task: WorkflowTask): Promise<void> {
|
||||
throw ApplicationFailure.create({ message: 'asset has no key', nonRetryable: true })
|
||||
}
|
||||
|
||||
const results: GeneratedEmbedding[] = []
|
||||
const usage = {
|
||||
model: 'gemini-embedding-2',
|
||||
model: isGemmaProvider ? 'gemma-embedding-2-openai' : 'gemini-embedding-2',
|
||||
inputTokens: 0,
|
||||
outputTokens: 0,
|
||||
}
|
||||
|
||||
if (isImage) {
|
||||
const result = await executeActivity(agentWorkerQueue, generateImageEmbeddingActivity, {
|
||||
teamId: task.teamId,
|
||||
assetKey: embeddingKey,
|
||||
mediaType: embeddingMediaType,
|
||||
})
|
||||
results.push({ embedding: result.embedding })
|
||||
if (result.usage) {
|
||||
usage.inputTokens += result.usage.inputTokens || 0
|
||||
usage.outputTokens += result.usage.outputTokens || 0
|
||||
const alreadyEmbedded = validExistingChunks.some(
|
||||
(e) => e.startTime === null && e.endTime === null,
|
||||
)
|
||||
if (!alreadyEmbedded) {
|
||||
const result = await executeActivity(agentWorkerQueue, generateImageEmbeddingActivity, {
|
||||
teamId: task.teamId,
|
||||
assetKey: embeddingKey,
|
||||
mediaType: embeddingMediaType,
|
||||
})
|
||||
await executeActivity(agentWorkerQueue, saveAssetEmbeddingsActivity, {
|
||||
assetId: task.assetId,
|
||||
embeddings: [{ embedding: result.embedding }],
|
||||
})
|
||||
if (result.usage) {
|
||||
usage.inputTokens += result.usage.inputTokens || 0
|
||||
usage.outputTokens += result.usage.outputTokens || 0
|
||||
}
|
||||
}
|
||||
} else if (isVideo) {
|
||||
// Get video duration
|
||||
@@ -145,12 +188,30 @@ export async function agentEmbeddingMedia(task: WorkflowTask): Promise<void> {
|
||||
end = duration
|
||||
}
|
||||
|
||||
// Check if this chunk is already completed
|
||||
const chunkExists = validExistingChunks.some(
|
||||
(c) =>
|
||||
c.startTime !== null &&
|
||||
c.endTime !== null &&
|
||||
Math.abs(c.startTime - start) < 0.1 &&
|
||||
Math.abs(c.endTime - end) < 0.1,
|
||||
)
|
||||
|
||||
if (chunkExists) {
|
||||
// If this chunk reached the end of the video, stop
|
||||
if (end >= duration) {
|
||||
break
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
// Step A: Slice chunk on transcode worker and upload to S3
|
||||
const chunkRes = await executeActivity(transcodeWorkerQueue, transcodeVideoChunkActivity, {
|
||||
assetId: asset.id,
|
||||
assetKey: embeddingKey,
|
||||
startTime: start,
|
||||
endTime: end,
|
||||
format: isGemmaProvider ? 'frames' : 'video',
|
||||
})
|
||||
const { chunkKey } = chunkRes
|
||||
|
||||
@@ -165,10 +226,16 @@ export async function agentEmbeddingMedia(task: WorkflowTask): Promise<void> {
|
||||
},
|
||||
)
|
||||
|
||||
results.push({
|
||||
embedding: chunkResult.embedding,
|
||||
startTime: start,
|
||||
endTime: end,
|
||||
// Step C: Save chunk embedding to database immediately
|
||||
await executeActivity(agentWorkerQueue, saveAssetEmbeddingsActivity, {
|
||||
assetId: task.assetId,
|
||||
embeddings: [
|
||||
{
|
||||
embedding: chunkResult.embedding,
|
||||
startTime: start,
|
||||
endTime: end,
|
||||
},
|
||||
],
|
||||
})
|
||||
|
||||
if (chunkResult.usage) {
|
||||
@@ -176,7 +243,7 @@ export async function agentEmbeddingMedia(task: WorkflowTask): Promise<void> {
|
||||
usage.outputTokens += chunkResult.usage.outputTokens || 0
|
||||
}
|
||||
} finally {
|
||||
// Step C: Delete temporary chunk from S3
|
||||
// Step D: Delete temporary chunk from S3
|
||||
await executeActivity(transcodeWorkerQueue, deleteS3ObjectActivity, {
|
||||
key: chunkKey,
|
||||
})
|
||||
@@ -189,14 +256,6 @@ export async function agentEmbeddingMedia(task: WorkflowTask): Promise<void> {
|
||||
}
|
||||
}
|
||||
|
||||
// Save computed embeddings
|
||||
if (results.length > 0) {
|
||||
await executeActivity(agentWorkerQueue, saveAssetEmbeddingsActivity, {
|
||||
assetId: task.assetId,
|
||||
embeddings: results,
|
||||
})
|
||||
}
|
||||
|
||||
// Update Usage
|
||||
await executeActivity(agentWorkerQueue, updateTaskUsageActivity, {
|
||||
taskId: task.id,
|
||||
|
||||
@@ -1,145 +0,0 @@
|
||||
import { describe, it, expect, vi, beforeEach } from 'vitest'
|
||||
import { queryEmbeddingForSearch } from './query-embedding-for-search'
|
||||
import { prisma } from '@shumai/db'
|
||||
import { setupTestDbHooks } from '@shumai/db/test'
|
||||
import * as workflowUtils from '@shumai/workflow-core'
|
||||
|
||||
vi.mock('@shumai/workflow-core', async () => {
|
||||
const actual = await vi.importActual('@shumai/workflow-core')
|
||||
return {
|
||||
...actual,
|
||||
getActivities: vi.fn(),
|
||||
executeActivity: vi.fn(),
|
||||
}
|
||||
})
|
||||
|
||||
describe('Query Embedding For Search Workflow', () => {
|
||||
setupTestDbHooks()
|
||||
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- mockActivities holds mock functions cast to expected types
|
||||
let mockActivities: any
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
|
||||
mockActivities = {
|
||||
updateWorkflowTaskActivity: Object.assign(vi.fn(), {
|
||||
_activityName: 'updateWorkflowTaskActivity',
|
||||
}),
|
||||
generateTextEmbeddingActivity: Object.assign(vi.fn(), {
|
||||
_activityName: 'generateTextEmbeddingActivity',
|
||||
}),
|
||||
getAgentWorkerQueueActivity: Object.assign(vi.fn(), {
|
||||
_activityName: 'getAgentWorkerQueueActivity',
|
||||
}),
|
||||
}
|
||||
|
||||
mockActivities.getAgentWorkerQueueActivity.mockResolvedValue('agent_queue')
|
||||
mockActivities.updateWorkflowTaskActivity.mockResolvedValue({})
|
||||
mockActivities.generateTextEmbeddingActivity.mockResolvedValue({
|
||||
embedding: [0.1, 0.2, 0.3],
|
||||
usage: { inputTokens: 4, outputTokens: 0, model: 'gemini-embedding-2' },
|
||||
})
|
||||
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- mockActivities contains vi.fn mock functions which are cast to expected activity proxy types
|
||||
vi.mocked(workflowUtils.getActivities).mockReturnValue(mockActivities as any)
|
||||
vi.mocked(workflowUtils.executeActivity).mockImplementation(async (_queue, act, ...args) => {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- act is one of the mocked activities
|
||||
return (act as any)(...args)
|
||||
})
|
||||
})
|
||||
|
||||
it('should run query embedding workflow successfully', async () => {
|
||||
const task = await prisma.workflowTask.create({
|
||||
data: {
|
||||
type: 'query_embedding_for_search',
|
||||
status: 'pending',
|
||||
teamId: 't1',
|
||||
assetId: 'a1',
|
||||
payload: {
|
||||
projectId: 'p1',
|
||||
queryEmbeddingForSearch: {
|
||||
text: 'search query',
|
||||
},
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
await queryEmbeddingForSearch(task)
|
||||
|
||||
// Verify queue discovery
|
||||
expect(mockActivities.getAgentWorkerQueueActivity).toHaveBeenCalled()
|
||||
|
||||
// Verify task processing status
|
||||
expect(mockActivities.updateWorkflowTaskActivity).toHaveBeenCalledWith({
|
||||
taskId: task.id,
|
||||
status: 'processing',
|
||||
heartbeat: true,
|
||||
})
|
||||
|
||||
// Verify embedding generation
|
||||
expect(mockActivities.generateTextEmbeddingActivity).toHaveBeenCalledWith({
|
||||
text: 'search query',
|
||||
teamId: 't1',
|
||||
})
|
||||
|
||||
// Verify completed status and details
|
||||
expect(mockActivities.updateWorkflowTaskActivity).toHaveBeenCalledWith({
|
||||
taskId: task.id,
|
||||
status: 'completed',
|
||||
output: { embedding: [0.1, 0.2, 0.3] },
|
||||
inputTokens: 4,
|
||||
outputTokens: 0,
|
||||
model: 'gemini-embedding-2',
|
||||
})
|
||||
})
|
||||
|
||||
it('should throw immediately if text is missing', async () => {
|
||||
const task = await prisma.workflowTask.create({
|
||||
data: {
|
||||
type: 'query_embedding_for_search',
|
||||
status: 'pending',
|
||||
teamId: 't1',
|
||||
assetId: 'a1',
|
||||
},
|
||||
})
|
||||
|
||||
await expect(queryEmbeddingForSearch(task)).rejects.toThrow(
|
||||
'Missing text in query_embedding_for_search task payload',
|
||||
)
|
||||
|
||||
// Should not call any activities because it throws before queue discovery
|
||||
expect(mockActivities.getAgentWorkerQueueActivity).not.toHaveBeenCalled()
|
||||
expect(mockActivities.updateWorkflowTaskActivity).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('should handle activity failure and update task status to failed', async () => {
|
||||
mockActivities.generateTextEmbeddingActivity.mockRejectedValue(
|
||||
new Error('Embedding API failed'),
|
||||
)
|
||||
|
||||
const task = await prisma.workflowTask.create({
|
||||
data: {
|
||||
type: 'query_embedding_for_search',
|
||||
status: 'pending',
|
||||
teamId: 't1',
|
||||
assetId: 'a1',
|
||||
payload: {
|
||||
projectId: 'p1',
|
||||
queryEmbeddingForSearch: {
|
||||
text: 'search query',
|
||||
},
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
await expect(queryEmbeddingForSearch(task)).rejects.toThrow('Embedding API failed')
|
||||
|
||||
// Verify it updates task to failed
|
||||
expect(mockActivities.updateWorkflowTaskActivity).toHaveBeenCalledWith({
|
||||
taskId: task.id,
|
||||
status: 'failed',
|
||||
output: { error: 'Embedding API failed' },
|
||||
})
|
||||
})
|
||||
})
|
||||
@@ -1,56 +0,0 @@
|
||||
import type { WorkflowTask } from '@shumai/db'
|
||||
import { getActivities, executeActivity, TaskQueueAgent } from '@shumai/workflow-core'
|
||||
|
||||
export async function queryEmbeddingForSearch(task: WorkflowTask): Promise<void> {
|
||||
const { generateTextEmbeddingActivity, updateWorkflowTaskActivity, getAgentWorkerQueueActivity } =
|
||||
getActivities()
|
||||
|
||||
let agentWorkerQueue = ''
|
||||
|
||||
const payload = task.payload as PrismaJson.WorkflowTaskPayload | null
|
||||
const text = payload?.queryEmbeddingForSearch?.text
|
||||
|
||||
if (!text) {
|
||||
throw new Error('Missing text in query_embedding_for_search task payload')
|
||||
}
|
||||
|
||||
try {
|
||||
// 0. Discover queue
|
||||
agentWorkerQueue = await executeActivity(TaskQueueAgent, getAgentWorkerQueueActivity)
|
||||
|
||||
// 1. Mark as processing
|
||||
await executeActivity(agentWorkerQueue, updateWorkflowTaskActivity, {
|
||||
taskId: task.id,
|
||||
status: 'processing',
|
||||
heartbeat: true,
|
||||
})
|
||||
|
||||
// 2. Generate embedding
|
||||
const result = await executeActivity(agentWorkerQueue, generateTextEmbeddingActivity, {
|
||||
text,
|
||||
teamId: task.teamId!,
|
||||
})
|
||||
|
||||
// 3. Save output and complete
|
||||
await executeActivity(agentWorkerQueue, updateWorkflowTaskActivity, {
|
||||
taskId: task.id,
|
||||
status: 'completed',
|
||||
output: { embedding: result.embedding },
|
||||
inputTokens: result.usage.inputTokens,
|
||||
outputTokens: result.usage.outputTokens,
|
||||
model: result.usage.model,
|
||||
})
|
||||
} catch (err) {
|
||||
console.error(`queryEmbeddingForSearch failed for task ${task.id}:`, err)
|
||||
if (agentWorkerQueue) {
|
||||
await executeActivity(agentWorkerQueue, updateWorkflowTaskActivity, {
|
||||
taskId: task.id,
|
||||
status: 'failed',
|
||||
output: {
|
||||
error: err instanceof Error ? err.message : String(err),
|
||||
},
|
||||
})
|
||||
}
|
||||
throw err
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@shumai/api",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "./src/index.ts",
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { describe, it, expect, vi, beforeEach } from 'vitest'
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'
|
||||
import { HTTPException } from 'hono/http-exception'
|
||||
import { agentService } from '@shumai/core/src/agent/agent'
|
||||
import { authzService } from '@shumai/core/src/authz/authz'
|
||||
@@ -210,6 +210,30 @@ describe('Agent API', () => {
|
||||
itemId: 'agent1',
|
||||
})
|
||||
})
|
||||
|
||||
it('returns 403 when updating embedding agent and ENABLE_EMBEDDING_AGENT is true', async () => {
|
||||
process.env.ENABLE_EMBEDDING_AGENT = 'true'
|
||||
try {
|
||||
vi.mocked(agentService.getAgent).mockResolvedValueOnce({
|
||||
id: 'agent1',
|
||||
teamId: 'team1',
|
||||
type: 'embedding',
|
||||
} as unknown as Awaited<ReturnType<typeof agentService.getAgent>>)
|
||||
|
||||
const res = await app.request('/agents/agent1', {
|
||||
method: 'PUT',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
name: 'Updated Embedding Agent',
|
||||
type: 'embedding',
|
||||
}),
|
||||
})
|
||||
|
||||
expect(res.status).toBe(403)
|
||||
} finally {
|
||||
delete process.env.ENABLE_EMBEDDING_AGENT
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
describe('DELETE /agents/:agentId', () => {
|
||||
@@ -227,6 +251,25 @@ describe('Agent API', () => {
|
||||
itemId: 'agent1',
|
||||
})
|
||||
})
|
||||
|
||||
it('returns 403 when deleting embedding agent and ENABLE_EMBEDDING_AGENT is true', async () => {
|
||||
process.env.ENABLE_EMBEDDING_AGENT = 'true'
|
||||
try {
|
||||
vi.mocked(agentService.getAgent).mockResolvedValueOnce({
|
||||
id: 'agent1',
|
||||
teamId: 'team1',
|
||||
type: 'embedding',
|
||||
} as unknown as Awaited<ReturnType<typeof agentService.getAgent>>)
|
||||
|
||||
const res = await app.request('/agents/agent1', {
|
||||
method: 'DELETE',
|
||||
})
|
||||
|
||||
expect(res.status).toBe(403)
|
||||
} finally {
|
||||
delete process.env.ENABLE_EMBEDDING_AGENT
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
describe('PATCH /agents/:agentId/permission', () => {
|
||||
@@ -271,6 +314,27 @@ describe('Agent API', () => {
|
||||
})
|
||||
})
|
||||
|
||||
it('returns 403 when modifying embedding agent permission and ENABLE_EMBEDDING_AGENT is true', async () => {
|
||||
process.env.ENABLE_EMBEDDING_AGENT = 'true'
|
||||
try {
|
||||
vi.mocked(agentService.getAgent).mockResolvedValueOnce({
|
||||
id: 'agent1',
|
||||
teamId: 'team1',
|
||||
type: 'embedding',
|
||||
} as unknown as Awaited<ReturnType<typeof agentService.getAgent>>)
|
||||
|
||||
const res = await app.request('/agents/agent1/permission', {
|
||||
method: 'PATCH',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ permission: 'owner' }),
|
||||
})
|
||||
|
||||
expect(res.status).toBe(403)
|
||||
} finally {
|
||||
delete process.env.ENABLE_EMBEDDING_AGENT
|
||||
}
|
||||
})
|
||||
|
||||
it('denies access when user is not admin', async () => {
|
||||
vi.mocked(authzService.hasPermission).mockRejectedValue(
|
||||
new HTTPException(403, { message: 'Forbidden' }),
|
||||
@@ -410,4 +474,138 @@ describe('Agent API', () => {
|
||||
expect(agentService.listProjectChatAgents).not.toHaveBeenCalled()
|
||||
})
|
||||
})
|
||||
|
||||
describe('read-only embedding agent when ENABLE_EMBEDDING_AGENT is true', () => {
|
||||
beforeEach(() => {
|
||||
process.env.ENABLE_EMBEDDING_AGENT = 'true'
|
||||
vi.mocked(agentService.getAgent).mockResolvedValue({
|
||||
id: 'emb1',
|
||||
teamId: 'team1',
|
||||
type: 'embedding',
|
||||
} as unknown as Awaited<ReturnType<typeof agentService.getAgent>>)
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
delete process.env.ENABLE_EMBEDDING_AGENT
|
||||
})
|
||||
|
||||
it('rejects PUT /agents/:agentId with 403', async () => {
|
||||
const res = await app.request('/agents/emb1', {
|
||||
method: 'PUT',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
name: 'Updated Embedding',
|
||||
type: 'embedding',
|
||||
enabled: false,
|
||||
}),
|
||||
})
|
||||
|
||||
expect(res.status).toBe(403)
|
||||
expect(agentService.updateAgent).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('rejects DELETE /agents/:agentId with 403', async () => {
|
||||
const res = await app.request('/agents/emb1', {
|
||||
method: 'DELETE',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
})
|
||||
|
||||
expect(res.status).toBe(403)
|
||||
expect(agentService.deleteAgent).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('rejects PATCH /agents/:agentId/permission with 403', async () => {
|
||||
const res = await app.request('/agents/emb1/permission', {
|
||||
method: 'PATCH',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
permission: 'owner',
|
||||
}),
|
||||
})
|
||||
|
||||
expect(res.status).toBe(403)
|
||||
expect(agentService.updateAgentPermission).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('rejects POST /teams/:teamId/agents with 403 for embedding agent', async () => {
|
||||
const res = await app.request('/teams/team1/agents', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
name: 'Manual Embedding',
|
||||
type: 'embedding',
|
||||
}),
|
||||
})
|
||||
|
||||
expect(res.status).toBe(403)
|
||||
expect(agentService.createAgent).not.toHaveBeenCalled()
|
||||
})
|
||||
})
|
||||
|
||||
describe('read-only embedding agent when ENABLE_EMBEDDING_AGENT is false', () => {
|
||||
beforeEach(() => {
|
||||
process.env.ENABLE_EMBEDDING_AGENT = 'false'
|
||||
vi.mocked(agentService.getAgent).mockResolvedValue({
|
||||
id: 'emb1',
|
||||
teamId: 'team1',
|
||||
type: 'embedding',
|
||||
} as unknown as Awaited<ReturnType<typeof agentService.getAgent>>)
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
delete process.env.ENABLE_EMBEDDING_AGENT
|
||||
})
|
||||
|
||||
it('rejects PUT /agents/:agentId with 403', async () => {
|
||||
const res = await app.request('/agents/emb1', {
|
||||
method: 'PUT',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
name: 'Updated Embedding',
|
||||
type: 'embedding',
|
||||
enabled: false,
|
||||
}),
|
||||
})
|
||||
|
||||
expect(res.status).toBe(403)
|
||||
expect(agentService.updateAgent).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('rejects DELETE /agents/:agentId with 403', async () => {
|
||||
const res = await app.request('/agents/emb1', {
|
||||
method: 'DELETE',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
})
|
||||
|
||||
expect(res.status).toBe(403)
|
||||
expect(agentService.deleteAgent).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('rejects PATCH /agents/:agentId/permission with 403', async () => {
|
||||
const res = await app.request('/agents/emb1/permission', {
|
||||
method: 'PATCH',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
permission: 'owner',
|
||||
}),
|
||||
})
|
||||
|
||||
expect(res.status).toBe(403)
|
||||
expect(agentService.updateAgentPermission).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('rejects POST /teams/:teamId/agents with 403 for embedding agent', async () => {
|
||||
const res = await app.request('/teams/team1/agents', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
name: 'Manual Embedding',
|
||||
type: 'embedding',
|
||||
}),
|
||||
})
|
||||
|
||||
expect(res.status).toBe(403)
|
||||
expect(agentService.createAgent).not.toHaveBeenCalled()
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { Hono } from 'hono'
|
||||
import { HTTPException } from 'hono/http-exception'
|
||||
import { zValidator } from '@hono/zod-validator'
|
||||
import { agentService } from '@shumai/core/src/agent/agent'
|
||||
import { authzService, Permission, ResourceType } from '@shumai/core/src/authz/authz'
|
||||
@@ -62,6 +63,16 @@ const route = new Hono<{ Variables: { user: User } }>()
|
||||
id: teamId,
|
||||
})
|
||||
|
||||
const isEmbeddingManagedByEnv =
|
||||
process.env.ENABLE_EMBEDDING_AGENT === 'true' ||
|
||||
process.env.ENABLE_EMBEDDING_AGENT === 'false'
|
||||
|
||||
if (req.type === 'embedding' && isEmbeddingManagedByEnv) {
|
||||
throw new HTTPException(403, {
|
||||
message: 'Embedding agent is managed by environment configuration and cannot be created',
|
||||
})
|
||||
}
|
||||
|
||||
const agent = await agentService.createAgent({
|
||||
teamId,
|
||||
...req,
|
||||
@@ -92,6 +103,17 @@ const route = new Hono<{ Variables: { user: User } }>()
|
||||
id: agentId,
|
||||
})
|
||||
|
||||
const isEmbeddingManagedByEnv =
|
||||
process.env.ENABLE_EMBEDDING_AGENT === 'true' ||
|
||||
process.env.ENABLE_EMBEDDING_AGENT === 'false'
|
||||
|
||||
const existingAgent = await agentService.getAgent({ agentId })
|
||||
if (existingAgent?.type === 'embedding' && isEmbeddingManagedByEnv) {
|
||||
throw new HTTPException(403, {
|
||||
message: 'Embedding agent is managed by environment configuration and cannot be modified',
|
||||
})
|
||||
}
|
||||
|
||||
const agent = await agentService.updateAgent({
|
||||
agentId,
|
||||
...req,
|
||||
@@ -123,6 +145,17 @@ const route = new Hono<{ Variables: { user: User } }>()
|
||||
id: agentId,
|
||||
})
|
||||
|
||||
const isEmbeddingManagedByEnv =
|
||||
process.env.ENABLE_EMBEDDING_AGENT === 'true' ||
|
||||
process.env.ENABLE_EMBEDDING_AGENT === 'false'
|
||||
|
||||
const existingAgent = await agentService.getAgent({ agentId })
|
||||
if (existingAgent?.type === 'embedding' && isEmbeddingManagedByEnv) {
|
||||
throw new HTTPException(403, {
|
||||
message: 'Embedding agent is managed by environment configuration and cannot be modified',
|
||||
})
|
||||
}
|
||||
|
||||
const agent = await agentService.updateAgentPermission(agentId, req.permission)
|
||||
|
||||
await auditLogService.logAction({
|
||||
@@ -148,7 +181,16 @@ const route = new Hono<{ Variables: { user: User } }>()
|
||||
id: agentId,
|
||||
})
|
||||
|
||||
const isEmbeddingManagedByEnv =
|
||||
process.env.ENABLE_EMBEDDING_AGENT === 'true' ||
|
||||
process.env.ENABLE_EMBEDDING_AGENT === 'false'
|
||||
|
||||
const existingAgent = await agentService.getAgent({ agentId })
|
||||
if (existingAgent?.type === 'embedding' && isEmbeddingManagedByEnv) {
|
||||
throw new HTTPException(403, {
|
||||
message: 'Embedding agent is managed by environment configuration and cannot be deleted',
|
||||
})
|
||||
}
|
||||
|
||||
await agentService.deleteAgent({
|
||||
agentId,
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@shumai/core",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "./src/index.ts",
|
||||
@@ -29,6 +29,7 @@
|
||||
"@aws-sdk/s3-request-presigner": "3.1121.0",
|
||||
"@earendil-works/pi-agent-core": "0.82.1",
|
||||
"@earendil-works/pi-ai": "0.82.1",
|
||||
"@google/genai": "2.28.0",
|
||||
"@modelcontextprotocol/client": "2.0.0",
|
||||
"@shumai/db": "workspace:*",
|
||||
"@shumai/dtos": "workspace:*",
|
||||
|
||||
@@ -976,4 +976,171 @@ describe('AgentService', () => {
|
||||
expect(ids).toHaveLength(0)
|
||||
})
|
||||
})
|
||||
|
||||
describe('ensureEmbeddingAgent', () => {
|
||||
test('creates embedding agent when ENABLE_EMBEDDING_AGENT is true', async () => {
|
||||
const db = prisma
|
||||
const svc = new AgentService()
|
||||
const { team } = await setupTestData(db)
|
||||
|
||||
process.env.ENABLE_EMBEDDING_AGENT = 'true'
|
||||
try {
|
||||
await svc.ensureEmbeddingAgentForTeam(team.id)
|
||||
const agent = await db.agent.findFirst({
|
||||
where: { teamId: team.id, type: 'embedding' },
|
||||
include: { user: true, skills: { include: { skill: true } }, mcpServers: true },
|
||||
})
|
||||
expect(agent).toBeDefined()
|
||||
const info = await svc.toAgentInfo(agent!)
|
||||
expect(info.name).toBe('embedding-agent')
|
||||
expect(info.type).toBe('embedding')
|
||||
expect(info.enabled).toBe(true)
|
||||
expect(info.avatarPreset).toBe('avatar-1')
|
||||
expect(info.avatar).toContain('avatar-1.webp')
|
||||
expect(info.readOnly).toBe(true)
|
||||
|
||||
// Ensure idempotency
|
||||
await svc.ensureEmbeddingAgentForTeam(team.id)
|
||||
const count = await db.agent.count({
|
||||
where: { teamId: team.id, type: 'embedding' },
|
||||
})
|
||||
expect(count).toBe(1)
|
||||
} finally {
|
||||
delete process.env.ENABLE_EMBEDDING_AGENT
|
||||
}
|
||||
})
|
||||
|
||||
test('enables existing embedding agent when ENABLE_EMBEDDING_AGENT is true', async () => {
|
||||
const db = prisma
|
||||
const svc = new AgentService()
|
||||
const { team } = await setupTestData(db)
|
||||
|
||||
const bot = await db.user.create({
|
||||
data: { name: 'Existing Embedding', email: 'existing-emb@shumai.ai', type: 'agent' },
|
||||
})
|
||||
await db.agent.create({
|
||||
data: {
|
||||
id: bot.id,
|
||||
teamId: team.id,
|
||||
type: 'embedding',
|
||||
enabled: false,
|
||||
config: { provider: '', model: '' },
|
||||
},
|
||||
})
|
||||
|
||||
process.env.ENABLE_EMBEDDING_AGENT = 'true'
|
||||
try {
|
||||
await svc.ensureEmbeddingAgentForTeam(team.id)
|
||||
const agent = await db.agent.findFirst({
|
||||
where: { teamId: team.id, type: 'embedding' },
|
||||
include: { user: true, skills: { include: { skill: true } }, mcpServers: true },
|
||||
})
|
||||
expect(agent?.id).toBe(bot.id)
|
||||
expect(agent?.enabled).toBe(true)
|
||||
const info = await svc.toAgentInfo(agent!)
|
||||
expect(info.readOnly).toBe(true)
|
||||
} finally {
|
||||
delete process.env.ENABLE_EMBEDDING_AGENT
|
||||
}
|
||||
})
|
||||
|
||||
test('does nothing when ENABLE_EMBEDDING_AGENT is not set', async () => {
|
||||
const db = prisma
|
||||
const svc = new AgentService()
|
||||
const team = await db.team.create({ data: { name: 'Team No Agent' } })
|
||||
|
||||
delete process.env.ENABLE_EMBEDDING_AGENT
|
||||
await svc.ensureEmbeddingAgentForTeam(team.id)
|
||||
const agent = await db.agent.findFirst({
|
||||
where: { teamId: team.id, type: 'embedding' },
|
||||
})
|
||||
expect(agent).toBeNull()
|
||||
})
|
||||
|
||||
test('does not create embedding agent when ENABLE_EMBEDDING_AGENT is false and agent does not exist', async () => {
|
||||
const db = prisma
|
||||
const svc = new AgentService()
|
||||
const { team } = await setupTestData(db)
|
||||
|
||||
process.env.ENABLE_EMBEDDING_AGENT = 'false'
|
||||
try {
|
||||
await svc.ensureEmbeddingAgentForTeam(team.id)
|
||||
const agent = await db.agent.findFirst({
|
||||
where: { teamId: team.id, type: 'embedding' },
|
||||
})
|
||||
expect(agent).toBeNull()
|
||||
} finally {
|
||||
delete process.env.ENABLE_EMBEDDING_AGENT
|
||||
}
|
||||
})
|
||||
|
||||
test('disables existing embedding agent and marks readOnly: true when ENABLE_EMBEDDING_AGENT is false', async () => {
|
||||
const db = prisma
|
||||
const svc = new AgentService()
|
||||
const { team } = await setupTestData(db)
|
||||
|
||||
const bot = await db.user.create({
|
||||
data: { name: 'Existing Embedding', email: 'existing-emb-false@shumai.ai', type: 'agent' },
|
||||
})
|
||||
await db.agent.create({
|
||||
data: {
|
||||
id: bot.id,
|
||||
teamId: team.id,
|
||||
type: 'embedding',
|
||||
enabled: true,
|
||||
config: { provider: '', model: '' },
|
||||
},
|
||||
})
|
||||
|
||||
process.env.ENABLE_EMBEDDING_AGENT = 'false'
|
||||
try {
|
||||
await svc.ensureEmbeddingAgentForTeam(team.id)
|
||||
const agent = await db.agent.findFirst({
|
||||
where: { teamId: team.id, type: 'embedding' },
|
||||
include: { user: true, skills: { include: { skill: true } }, mcpServers: true },
|
||||
})
|
||||
expect(agent?.id).toBe(bot.id)
|
||||
expect(agent?.enabled).toBe(false)
|
||||
const info = await svc.toAgentInfo(agent!)
|
||||
expect(info.readOnly).toBe(true)
|
||||
expect(info.enabled).toBe(false)
|
||||
} finally {
|
||||
delete process.env.ENABLE_EMBEDDING_AGENT
|
||||
}
|
||||
})
|
||||
|
||||
test('ensureSystemEmbeddingAgents updates all teams when ENABLE_EMBEDDING_AGENT is false', async () => {
|
||||
const db = prisma
|
||||
const svc = new AgentService()
|
||||
const { team: team1 } = await setupTestData(db)
|
||||
const team2 = await db.team.create({ data: { name: 'Team 2' } })
|
||||
|
||||
// Create enabled embedding agent on team1
|
||||
const bot1 = await db.user.create({
|
||||
data: { name: 'Bot 1', email: 'b1-false@shumai.ai', type: 'agent' },
|
||||
})
|
||||
await db.agent.create({
|
||||
data: {
|
||||
id: bot1.id,
|
||||
teamId: team1.id,
|
||||
type: 'embedding',
|
||||
enabled: true,
|
||||
config: { provider: '', model: '' },
|
||||
},
|
||||
})
|
||||
|
||||
process.env.ENABLE_EMBEDDING_AGENT = 'false'
|
||||
try {
|
||||
await svc.ensureSystemEmbeddingAgents()
|
||||
|
||||
const a1 = await db.agent.findFirst({ where: { teamId: team1.id, type: 'embedding' } })
|
||||
const a2 = await db.agent.findFirst({ where: { teamId: team2.id, type: 'embedding' } })
|
||||
|
||||
expect(a1?.enabled).toBe(false)
|
||||
expect(a2).toBeNull()
|
||||
} finally {
|
||||
delete process.env.ENABLE_EMBEDDING_AGENT
|
||||
}
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
@@ -43,8 +43,8 @@ interface AgentInfoSource {
|
||||
config: unknown
|
||||
soul: string | null
|
||||
user: { name: string; image: string | null }
|
||||
skills: Array<{ id: string; skillId: string; skill: { id: string; name: string } }>
|
||||
mcpServers: Array<{ mcpServerId: string }>
|
||||
skills?: Array<{ id: string; skillId: string; skill: { id: string; name: string } }>
|
||||
mcpServers?: Array<{ mcpServerId: string }>
|
||||
}
|
||||
|
||||
function extractS3Key(urlOrKey?: string | null): string | null | undefined {
|
||||
@@ -462,6 +462,11 @@ export class AgentService {
|
||||
*/
|
||||
async toAgentInfo(agent: AgentInfoSource): Promise<AgentInfo> {
|
||||
const config = agent.config as unknown as PrismaJson.AgentConfig
|
||||
const isManagedByEnv =
|
||||
process.env.ENABLE_EMBEDDING_AGENT === 'true' ||
|
||||
process.env.ENABLE_EMBEDDING_AGENT === 'false'
|
||||
const isReadOnly = agent.type === 'embedding' && isManagedByEnv
|
||||
|
||||
return {
|
||||
id: agent.id,
|
||||
name: agent.user.name,
|
||||
@@ -475,13 +480,14 @@ export class AgentService {
|
||||
thinkingLevel: config.thinkingLevel || 'off',
|
||||
systemPrompt: config.systemPrompt,
|
||||
soul: agent.soul || undefined,
|
||||
skills: agent.skills.map((s) => ({
|
||||
skills: (agent.skills || []).map((s) => ({
|
||||
id: s.id,
|
||||
skillId: s.skillId,
|
||||
skill: s.skill,
|
||||
})),
|
||||
mcpServerIds: (agent.mcpServers || []).map((m) => m.mcpServerId),
|
||||
deniedTools: config.deniedTools || [],
|
||||
readOnly: isReadOnly,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -718,6 +724,60 @@ export class AgentService {
|
||||
params,
|
||||
)
|
||||
}
|
||||
|
||||
async ensureSystemEmbeddingAgents(): Promise<void> {
|
||||
const envVal = process.env.ENABLE_EMBEDDING_AGENT
|
||||
if (envVal !== 'true' && envVal !== 'false') {
|
||||
return
|
||||
}
|
||||
|
||||
const teams = await this.prismaClient.team.findMany({
|
||||
select: { id: true },
|
||||
})
|
||||
|
||||
for (const team of teams) {
|
||||
await this.ensureEmbeddingAgentForTeam(team.id)
|
||||
}
|
||||
}
|
||||
|
||||
async ensureEmbeddingAgentForTeam(teamId: string): Promise<void> {
|
||||
const envVal = process.env.ENABLE_EMBEDDING_AGENT
|
||||
if (envVal !== 'true' && envVal !== 'false') {
|
||||
return
|
||||
}
|
||||
|
||||
const existing = await this.prismaClient.agent.findFirst({
|
||||
where: {
|
||||
teamId,
|
||||
type: 'embedding',
|
||||
},
|
||||
})
|
||||
|
||||
if (envVal === 'true') {
|
||||
if (!existing) {
|
||||
await this.createAgent({
|
||||
teamId,
|
||||
name: 'embedding-agent',
|
||||
type: 'embedding',
|
||||
avatar: 'avatar-1',
|
||||
thinkingLevel: 'off',
|
||||
enabled: true,
|
||||
})
|
||||
} else if (!existing.enabled) {
|
||||
await this.prismaClient.agent.update({
|
||||
where: { id: existing.id },
|
||||
data: { enabled: true },
|
||||
})
|
||||
}
|
||||
} else if (envVal === 'false') {
|
||||
if (existing && existing.enabled) {
|
||||
await this.prismaClient.agent.update({
|
||||
where: { id: existing.id },
|
||||
data: { enabled: false },
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export const agentService = new AgentService()
|
||||
|
||||
@@ -10,6 +10,8 @@ export * from './project/project'
|
||||
export * from './provider/provider'
|
||||
export * from './s3/s3'
|
||||
export * from './search/search'
|
||||
export * from './search/text-embedding'
|
||||
export * from './search/embedding-service'
|
||||
export * from './share/share'
|
||||
export * from './skill/skill'
|
||||
export * from './team/team'
|
||||
|
||||
@@ -0,0 +1,256 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { prisma, AssetType, WorkflowTaskStatus, WorkflowTaskType } from '@shumai/db'
|
||||
import { setupTestDbHooks } from '@shumai/db/test'
|
||||
import { agentService } from '@shumai/core/src/agent/agent'
|
||||
import { embeddingService } from './embedding-service'
|
||||
|
||||
describe('EmbeddingService', () => {
|
||||
setupTestDbHooks()
|
||||
|
||||
it('should skip teams without an enabled embedding agent', async () => {
|
||||
const team = await prisma.team.create({
|
||||
data: { name: 'Team Without Agent' },
|
||||
})
|
||||
const project = await prisma.project.create({
|
||||
data: { name: 'Project 1', teamId: team.id },
|
||||
})
|
||||
await prisma.asset.create({
|
||||
data: {
|
||||
name: 'video.mp4',
|
||||
type: AssetType.file,
|
||||
status: 'processed',
|
||||
mediaType: 'video/mp4',
|
||||
projectId: project.id,
|
||||
},
|
||||
})
|
||||
|
||||
const result = await embeddingService.regenerateEmbeddings()
|
||||
expect(result.tasksCreated).toBe(0)
|
||||
expect(result.totalAssets).toBe(0)
|
||||
expect(result.skippedTeams).toContain('Team Without Agent')
|
||||
})
|
||||
|
||||
it('should skip teams where embedding agent exists but is disabled', async () => {
|
||||
const team = await prisma.team.create({
|
||||
data: { name: 'Team Disabled Agent' },
|
||||
})
|
||||
await agentService.createAgent({
|
||||
teamId: team.id,
|
||||
name: 'embedding-agent',
|
||||
type: 'embedding',
|
||||
avatar: 'avatar-1',
|
||||
thinkingLevel: 'off',
|
||||
enabled: false,
|
||||
})
|
||||
const project = await prisma.project.create({
|
||||
data: { name: 'Project Disabled', teamId: team.id },
|
||||
})
|
||||
await prisma.asset.create({
|
||||
data: {
|
||||
name: 'photo.jpg',
|
||||
type: AssetType.file,
|
||||
status: 'processed',
|
||||
mediaType: 'image/jpeg',
|
||||
projectId: project.id,
|
||||
},
|
||||
})
|
||||
|
||||
const result = await embeddingService.regenerateEmbeddings()
|
||||
expect(result.tasksCreated).toBe(0)
|
||||
expect(result.totalAssets).toBe(0)
|
||||
expect(result.skippedTeams).toContain('Team Disabled Agent')
|
||||
})
|
||||
|
||||
it('should regenerate embeddings for eligible media assets and clean up old data', async () => {
|
||||
const team = await prisma.team.create({
|
||||
data: { name: 'Active Embedding Team' },
|
||||
})
|
||||
const agent = await agentService.createAgent({
|
||||
teamId: team.id,
|
||||
name: 'embedding-agent',
|
||||
type: 'embedding',
|
||||
avatar: 'avatar-1',
|
||||
thinkingLevel: 'off',
|
||||
enabled: true,
|
||||
})
|
||||
|
||||
const project = await prisma.project.create({
|
||||
data: { name: 'Active Project', teamId: team.id },
|
||||
})
|
||||
|
||||
// 1. Eligible video asset
|
||||
const videoAsset = await prisma.asset.create({
|
||||
data: {
|
||||
name: 'sample.mp4',
|
||||
type: AssetType.file,
|
||||
status: 'processed',
|
||||
mediaType: 'video/mp4',
|
||||
projectId: project.id,
|
||||
},
|
||||
})
|
||||
|
||||
// 2. Eligible image asset
|
||||
const imageAsset = await prisma.asset.create({
|
||||
data: {
|
||||
name: 'sample.png',
|
||||
type: AssetType.file,
|
||||
status: 'processed',
|
||||
mediaType: 'image/png',
|
||||
projectId: project.id,
|
||||
},
|
||||
})
|
||||
|
||||
// 3. Eligible 3D asset with rendered views
|
||||
const asset3d = await prisma.asset.create({
|
||||
data: {
|
||||
name: 'model.glb',
|
||||
type: AssetType.file,
|
||||
status: 'processed',
|
||||
mediaType: 'model/gltf-binary',
|
||||
media: {
|
||||
proxyType: '3d',
|
||||
imageTranscodes: [
|
||||
{ key: 'files/3d/view.webp', format: 'webp', width: 100, height: 100, quality: 80 },
|
||||
],
|
||||
} as unknown as PrismaJson.MediaInfo,
|
||||
projectId: project.id,
|
||||
},
|
||||
})
|
||||
|
||||
// 4. Ineligible asset (non-media document)
|
||||
await prisma.asset.create({
|
||||
data: {
|
||||
name: 'document.pdf',
|
||||
type: AssetType.file,
|
||||
status: 'processed',
|
||||
mediaType: 'application/pdf',
|
||||
projectId: project.id,
|
||||
},
|
||||
})
|
||||
|
||||
// 5. Ineligible asset (trashed / deleted)
|
||||
await prisma.asset.create({
|
||||
data: {
|
||||
name: 'deleted-video.mp4',
|
||||
type: AssetType.file,
|
||||
status: 'processed',
|
||||
isDeleted: true,
|
||||
mediaType: 'video/mp4',
|
||||
projectId: project.id,
|
||||
},
|
||||
})
|
||||
|
||||
// 6. Ineligible asset (status not processed)
|
||||
await prisma.asset.create({
|
||||
data: {
|
||||
name: 'uploading.mp4',
|
||||
type: AssetType.file,
|
||||
status: 'uploading',
|
||||
mediaType: 'video/mp4',
|
||||
projectId: project.id,
|
||||
},
|
||||
})
|
||||
|
||||
// Create preexisting embeddings and workflow task for videoAsset to verify cleanup
|
||||
const dummyVec768 = `[${new Array(768).fill(0.1).join(',')}]`
|
||||
await prisma.$executeRaw`
|
||||
INSERT INTO asset_embeddings (id, asset_id, embedding_768, updated_at)
|
||||
VALUES (gen_random_uuid()::text, ${videoAsset.id}, ${dummyVec768}::vector, NOW())
|
||||
`
|
||||
await prisma.workflowTask.create({
|
||||
data: {
|
||||
assetId: videoAsset.id,
|
||||
type: WorkflowTaskType.ai_embedding,
|
||||
status: WorkflowTaskStatus.processing,
|
||||
teamId: team.id,
|
||||
projectId: project.id,
|
||||
},
|
||||
})
|
||||
|
||||
const initialEmbeddings = await prisma.assetEmbedding.count({
|
||||
where: { assetId: videoAsset.id },
|
||||
})
|
||||
expect(initialEmbeddings).toBe(1)
|
||||
|
||||
// Run regeneration
|
||||
const result = await embeddingService.regenerateEmbeddings()
|
||||
|
||||
expect(result.totalAssets).toBe(3)
|
||||
expect(result.tasksCreated).toBe(3)
|
||||
expect(result.skippedTeams).not.toContain('Active Embedding Team')
|
||||
|
||||
// Verify old embeddings were wiped upfront
|
||||
const remainingEmbeddings = await prisma.assetEmbedding.count({
|
||||
where: { assetId: videoAsset.id },
|
||||
})
|
||||
expect(remainingEmbeddings).toBe(0)
|
||||
|
||||
// Verify tasks created
|
||||
const createdTasks = await prisma.workflowTask.findMany({
|
||||
where: {
|
||||
type: WorkflowTaskType.ai_embedding,
|
||||
assetId: { in: [videoAsset.id, imageAsset.id, asset3d.id] },
|
||||
},
|
||||
})
|
||||
expect(createdTasks).toHaveLength(3)
|
||||
for (const task of createdTasks) {
|
||||
expect(task.status).toBe(WorkflowTaskStatus.pending)
|
||||
expect(task.teamId).toBe(team.id)
|
||||
expect(task.projectId).toBe(project.id)
|
||||
const payload = task.payload as { agent?: { agentId?: string } }
|
||||
expect(payload?.agent?.agentId).toBe(agent!.id)
|
||||
}
|
||||
})
|
||||
|
||||
it('should handle multiple teams where only some have active embedding agents', async () => {
|
||||
// Team 1: has active agent
|
||||
const team1 = await prisma.team.create({ data: { name: 'Team One' } })
|
||||
await agentService.createAgent({
|
||||
teamId: team1.id,
|
||||
name: 'embedding-agent',
|
||||
type: 'embedding',
|
||||
avatar: 'avatar-1',
|
||||
thinkingLevel: 'off',
|
||||
enabled: true,
|
||||
})
|
||||
const project1 = await prisma.project.create({ data: { name: 'P1', teamId: team1.id } })
|
||||
await prisma.asset.create({
|
||||
data: {
|
||||
name: 'v1.mp4',
|
||||
type: AssetType.file,
|
||||
status: 'processed',
|
||||
mediaType: 'video/mp4',
|
||||
projectId: project1.id,
|
||||
},
|
||||
})
|
||||
|
||||
// Team 2: no active agent
|
||||
const team2 = await prisma.team.create({ data: { name: 'Team Two' } })
|
||||
const project2 = await prisma.project.create({ data: { name: 'P2', teamId: team2.id } })
|
||||
await prisma.asset.create({
|
||||
data: {
|
||||
name: 'v2.mp4',
|
||||
type: AssetType.file,
|
||||
status: 'processed',
|
||||
mediaType: 'video/mp4',
|
||||
projectId: project2.id,
|
||||
},
|
||||
})
|
||||
|
||||
const result = await embeddingService.regenerateEmbeddings()
|
||||
|
||||
expect(result.totalAssets).toBe(1)
|
||||
expect(result.tasksCreated).toBe(1)
|
||||
expect(result.skippedTeams).toContain('Team Two')
|
||||
expect(result.skippedTeams).not.toContain('Team One')
|
||||
|
||||
const tasks = await prisma.workflowTask.findMany({
|
||||
where: {
|
||||
type: WorkflowTaskType.ai_embedding,
|
||||
teamId: { in: [team1.id, team2.id] },
|
||||
},
|
||||
})
|
||||
expect(tasks).toHaveLength(1)
|
||||
expect(tasks[0].teamId).toBe(team1.id)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,135 @@
|
||||
import { prisma, WorkflowTaskStatus, WorkflowTaskType } from '@shumai/db'
|
||||
import { logger } from '@shumai/core/src/logger'
|
||||
|
||||
export interface RegenerateEmbeddingsResult {
|
||||
tasksCreated: number
|
||||
totalAssets: number
|
||||
skippedTeams: string[]
|
||||
}
|
||||
|
||||
export class EmbeddingService {
|
||||
/**
|
||||
* Regenerates vector embeddings across all teams.
|
||||
*
|
||||
* Skips teams that do not have an active (enabled) embedding agent and logs a warning.
|
||||
* For teams with an active embedding agent, purges existing embeddings and pending/processing
|
||||
* embedding tasks upfront, then creates fresh ai_embedding workflow tasks for all eligible assets.
|
||||
*/
|
||||
async regenerateEmbeddings(): Promise<RegenerateEmbeddingsResult> {
|
||||
const teams = await prisma.team.findMany({
|
||||
select: { id: true, name: true },
|
||||
})
|
||||
|
||||
const activeAgents = await prisma.agent.findMany({
|
||||
where: {
|
||||
type: 'embedding',
|
||||
enabled: true,
|
||||
},
|
||||
select: { id: true, teamId: true },
|
||||
})
|
||||
|
||||
const activeAgentByTeamId = new Map(
|
||||
activeAgents
|
||||
.filter((a): a is typeof a & { teamId: string } => typeof a.teamId === 'string')
|
||||
.map((a) => [a.teamId, a]),
|
||||
)
|
||||
|
||||
const activeTeamIds: string[] = []
|
||||
const skippedTeams: string[] = []
|
||||
|
||||
for (const team of teams) {
|
||||
if (activeAgentByTeamId.has(team.id)) {
|
||||
activeTeamIds.push(team.id)
|
||||
} else {
|
||||
skippedTeams.push(team.name || team.id)
|
||||
logger.warn(
|
||||
{ teamId: team.id, teamName: team.name },
|
||||
'Skipping embedding regeneration for team: no active embedding agent found',
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
if (activeTeamIds.length === 0) {
|
||||
logger.info('No teams found with an active embedding agent. 0 tasks created.')
|
||||
return { tasksCreated: 0, totalAssets: 0, skippedTeams }
|
||||
}
|
||||
|
||||
const candidateAssets = await prisma.asset.findMany({
|
||||
where: {
|
||||
isDeleted: false,
|
||||
status: 'processed',
|
||||
type: { in: ['file', 'version_stack'] },
|
||||
project: { teamId: { in: activeTeamIds } },
|
||||
},
|
||||
include: { project: true },
|
||||
})
|
||||
|
||||
const eligibleAssets = candidateAssets.filter((asset) => {
|
||||
const isVideo = asset.mediaType?.startsWith('video/')
|
||||
const mediaInfo = asset.media as PrismaJson.MediaInfo | null
|
||||
const isImage =
|
||||
asset.mediaType?.startsWith('image/') ||
|
||||
(mediaInfo?.proxyType === '3d' && (mediaInfo.imageTranscodes?.length ?? 0) > 0)
|
||||
return isVideo || isImage
|
||||
})
|
||||
|
||||
const eligibleAssetIds = eligibleAssets.map((a) => a.id)
|
||||
|
||||
if (eligibleAssetIds.length > 0) {
|
||||
// Clean up previous embedding tasks to avoid duplicate execution
|
||||
await prisma.workflowTask.deleteMany({
|
||||
where: {
|
||||
assetId: { in: eligibleAssetIds },
|
||||
type: WorkflowTaskType.ai_embedding,
|
||||
},
|
||||
})
|
||||
|
||||
// Purge existing embeddings so everything regenerates cleanly with the configured provider
|
||||
await prisma.assetEmbedding.deleteMany({
|
||||
where: {
|
||||
assetId: { in: eligibleAssetIds },
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
let tasksCreated = 0
|
||||
for (const asset of eligibleAssets) {
|
||||
const teamId = asset.project?.teamId
|
||||
if (!teamId) continue
|
||||
const agent = activeAgentByTeamId.get(teamId)
|
||||
if (!agent) continue
|
||||
|
||||
await prisma.workflowTask.create({
|
||||
data: {
|
||||
assetId: asset.id,
|
||||
type: WorkflowTaskType.ai_embedding,
|
||||
status: WorkflowTaskStatus.pending,
|
||||
teamId,
|
||||
projectId: asset.projectId,
|
||||
payload: {
|
||||
projectId: asset.projectId ?? '',
|
||||
agent: { agentId: agent.id },
|
||||
},
|
||||
},
|
||||
})
|
||||
tasksCreated++
|
||||
}
|
||||
|
||||
logger.info(
|
||||
{
|
||||
totalAssets: eligibleAssets.length,
|
||||
tasksCreated,
|
||||
skippedTeamsCount: skippedTeams.length,
|
||||
},
|
||||
'Successfully queued embedding regeneration tasks',
|
||||
)
|
||||
|
||||
return {
|
||||
tasksCreated,
|
||||
totalAssets: eligibleAssets.length,
|
||||
skippedTeams,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export const embeddingService = new EmbeddingService()
|
||||
@@ -4,7 +4,7 @@ import { setupTestDbHooks } from '@shumai/db/test'
|
||||
import { SearchService } from './search'
|
||||
import { SearchConditionOperator } from '@shumai/dtos'
|
||||
import { metadataService } from '@shumai/core/src/metadata/metadata'
|
||||
import { workflowService } from '@shumai/workflow-core'
|
||||
import { textEmbeddingService } from './text-embedding'
|
||||
|
||||
import { AssetType } from '@shumai/db'
|
||||
import { assetService } from '@shumai/core/src/asset/asset'
|
||||
@@ -805,20 +805,14 @@ describe('SearchService', () => {
|
||||
)
|
||||
`
|
||||
|
||||
// Mock the embedding generation workflow task.
|
||||
// Mock the embedding generation
|
||||
// Our query vector points along the first dimension [1, 0, 0...]
|
||||
const queryVector = Array(1536).fill(0)
|
||||
queryVector[0] = 1.0
|
||||
const mockExecuteWait = vi
|
||||
.spyOn(workflowService, 'executeWait')
|
||||
.mockImplementation(async (task) => {
|
||||
return {
|
||||
...task,
|
||||
status: 'completed',
|
||||
output: {
|
||||
embedding: queryVector,
|
||||
},
|
||||
} as unknown as typeof task
|
||||
const mockGenerateEmbedding = vi
|
||||
.spyOn(textEmbeddingService, 'generateEmbedding')
|
||||
.mockResolvedValue({
|
||||
embedding: queryVector,
|
||||
})
|
||||
|
||||
// Search using semantic mode
|
||||
@@ -855,7 +849,248 @@ describe('SearchService', () => {
|
||||
expect(result.data[2].startTime).toBe(10)
|
||||
expect(result.data[2].endTime).toBe(20)
|
||||
|
||||
mockExecuteWait.mockRestore()
|
||||
mockGenerateEmbedding.mockRestore()
|
||||
})
|
||||
|
||||
it('matches and sorts 768-dimensional embeddings using embedding_768 column', async () => {
|
||||
const user = await prisma.user.create({
|
||||
data: { name: 'semantic-user-768', email: 'semantic-user-768@example.com', type: 'human' },
|
||||
})
|
||||
|
||||
const team = await prisma.team.create({
|
||||
data: { name: 'semantic-team-768' },
|
||||
})
|
||||
|
||||
const project = await prisma.project.create({
|
||||
data: { name: 'semantic-project-768', teamId: team.id },
|
||||
})
|
||||
|
||||
const botUser = await prisma.user.create({
|
||||
data: { name: 'Embedding Bot 768', email: 'emb-768@example.com', type: 'agent' },
|
||||
})
|
||||
const provider = await prisma.provider.create({
|
||||
data: {
|
||||
name: 'gemma',
|
||||
teamId: team.id,
|
||||
config: { api: 'openai', apiKey: 'key' } as unknown as PrismaJson.ProviderConfig,
|
||||
},
|
||||
})
|
||||
const model = await prisma.model.create({
|
||||
data: {
|
||||
modelId: 'embeddinggemma-2',
|
||||
name: 'EmbeddingGemma 2',
|
||||
providerId: provider.id,
|
||||
config: {
|
||||
reasoning: false,
|
||||
input: ['text'],
|
||||
contextWindow: 1000,
|
||||
maxTokens: 1000,
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
} as unknown as PrismaJson.ModelConfig,
|
||||
},
|
||||
})
|
||||
await prisma.agent.create({
|
||||
data: {
|
||||
id: botUser.id,
|
||||
teamId: team.id,
|
||||
type: 'embedding',
|
||||
enabled: true,
|
||||
providerId: provider.id,
|
||||
modelId: model.id,
|
||||
config: { provider: 'gemma-embedding-2-openai', model: 'embeddinggemma-2' },
|
||||
},
|
||||
})
|
||||
|
||||
const rootFolder = await prisma.asset.create({
|
||||
data: {
|
||||
name: 'root-768',
|
||||
type: AssetType.folder,
|
||||
projectId: project.id,
|
||||
creatorId: user.id,
|
||||
status: 'uploaded',
|
||||
},
|
||||
})
|
||||
|
||||
const videoAssetA = await prisma.asset.create({
|
||||
data: {
|
||||
name: 'VideoA_768.mp4',
|
||||
type: AssetType.file,
|
||||
mediaType: 'video/mp4',
|
||||
projectId: project.id,
|
||||
parentId: rootFolder.id,
|
||||
creatorId: user.id,
|
||||
status: 'uploaded',
|
||||
},
|
||||
})
|
||||
|
||||
const videoAssetB = await prisma.asset.create({
|
||||
data: {
|
||||
name: 'VideoB_768.mp4',
|
||||
type: AssetType.file,
|
||||
mediaType: 'video/mp4',
|
||||
projectId: project.id,
|
||||
parentId: rootFolder.id,
|
||||
creatorId: user.id,
|
||||
status: 'uploaded',
|
||||
},
|
||||
})
|
||||
|
||||
const arrA1 = Array(768).fill(0)
|
||||
arrA1[0] = 1.0
|
||||
const vectorA1 = JSON.stringify(arrA1)
|
||||
|
||||
const arrB1 = Array(768).fill(0)
|
||||
arrB1[0] = 1.0
|
||||
arrB1[1] = 1.0
|
||||
const vectorB1 = JSON.stringify(arrB1)
|
||||
|
||||
await prisma.$executeRaw`
|
||||
INSERT INTO asset_embeddings (id, asset_id, embedding_768, start_time, end_time, created_at, updated_at)
|
||||
VALUES (
|
||||
'chunk-768-a1',
|
||||
${videoAssetA.id},
|
||||
${vectorA1}::vector,
|
||||
0.0,
|
||||
10.0,
|
||||
NOW(),
|
||||
NOW()
|
||||
)
|
||||
`
|
||||
await prisma.$executeRaw`
|
||||
INSERT INTO asset_embeddings (id, asset_id, embedding_768, start_time, end_time, created_at, updated_at)
|
||||
VALUES (
|
||||
'chunk-768-b1',
|
||||
${videoAssetB.id},
|
||||
${vectorB1}::vector,
|
||||
5.0,
|
||||
15.0,
|
||||
NOW(),
|
||||
NOW()
|
||||
)
|
||||
`
|
||||
|
||||
// Query vector with 768 dims pointing along dimension 0
|
||||
const queryVector = Array(768).fill(0)
|
||||
queryVector[0] = 1.0
|
||||
const mockGenerateEmbedding = vi
|
||||
.spyOn(textEmbeddingService, 'generateEmbedding')
|
||||
.mockResolvedValue({
|
||||
embedding: queryVector,
|
||||
})
|
||||
|
||||
const result = await searchService.search(rootFolder.id, {
|
||||
recursively: true,
|
||||
assetType: 'file',
|
||||
operator: 'AND',
|
||||
conditions: [],
|
||||
query: 'find 768 dim match',
|
||||
isSemantic: true,
|
||||
})
|
||||
|
||||
expect(result.data.length).toBeGreaterThanOrEqual(2)
|
||||
expect(result.data[0].id).toBe(videoAssetA.id)
|
||||
expect(result.data[1].id).toBe(videoAssetB.id)
|
||||
|
||||
mockGenerateEmbedding.mockRestore()
|
||||
})
|
||||
|
||||
it('throws 502 if embedding generation fails during semantic search', async () => {
|
||||
const user = await prisma.user.create({
|
||||
data: { name: 'semantic-user-fail', email: 'semantic-fail@example.com', type: 'human' },
|
||||
})
|
||||
const team = await prisma.team.create({
|
||||
data: { name: 'semantic-team-fail' },
|
||||
})
|
||||
const project = await prisma.project.create({
|
||||
data: { name: 'semantic-project-fail', teamId: team.id },
|
||||
})
|
||||
const botUser = await prisma.user.create({
|
||||
data: { name: 'Embedding Bot Fail', email: 'emb-fail@example.com', type: 'agent' },
|
||||
})
|
||||
await prisma.agent.create({
|
||||
data: {
|
||||
id: botUser.id,
|
||||
teamId: team.id,
|
||||
type: 'embedding',
|
||||
enabled: true,
|
||||
config: { provider: 'gemini-embedding-2', model: 'gemini-embedding-2' },
|
||||
},
|
||||
})
|
||||
const rootFolder = await prisma.asset.create({
|
||||
data: {
|
||||
name: 'root-fail',
|
||||
type: AssetType.folder,
|
||||
projectId: project.id,
|
||||
creatorId: user.id,
|
||||
status: 'uploaded',
|
||||
},
|
||||
})
|
||||
|
||||
const mockGenerateEmbedding = vi
|
||||
.spyOn(textEmbeddingService, 'generateEmbedding')
|
||||
.mockRejectedValue(new Error('Rate limit exceeded'))
|
||||
|
||||
await expect(
|
||||
searchService.search(rootFolder.id, {
|
||||
recursively: true,
|
||||
assetType: 'file',
|
||||
operator: 'AND',
|
||||
conditions: [],
|
||||
query: 'fail query',
|
||||
isSemantic: true,
|
||||
}),
|
||||
).rejects.toThrow('Failed to generate query embedding: Rate limit exceeded')
|
||||
|
||||
mockGenerateEmbedding.mockRestore()
|
||||
})
|
||||
|
||||
it('rejects semantic search when ENABLE_EMBEDDING_AGENT is false', async () => {
|
||||
const user = await prisma.user.create({
|
||||
data: { name: 'u-emb-false', email: 'u-emb-false@example.com', type: 'human' },
|
||||
})
|
||||
const team = await prisma.team.create({ data: { name: 't-emb-false' } })
|
||||
const project = await prisma.project.create({
|
||||
data: { name: 'p-emb-false', teamId: team.id },
|
||||
})
|
||||
const rootFolder = await prisma.asset.create({
|
||||
data: {
|
||||
name: 'root-false',
|
||||
type: AssetType.folder,
|
||||
projectId: project.id,
|
||||
creatorId: user.id,
|
||||
status: 'uploaded',
|
||||
},
|
||||
})
|
||||
|
||||
// Even if an embedding agent is enabled in db
|
||||
const bot = await prisma.user.create({
|
||||
data: { name: 'Bot', email: 'b-emb-false@example.com', password: 'pw', type: 'agent' },
|
||||
})
|
||||
await prisma.agent.create({
|
||||
data: {
|
||||
id: bot.id,
|
||||
teamId: team.id,
|
||||
type: 'embedding',
|
||||
enabled: true,
|
||||
config: { provider: '', model: '' },
|
||||
},
|
||||
})
|
||||
|
||||
process.env.ENABLE_EMBEDDING_AGENT = 'false'
|
||||
try {
|
||||
await expect(
|
||||
searchService.search(rootFolder.id, {
|
||||
recursively: true,
|
||||
assetType: 'file',
|
||||
operator: 'AND',
|
||||
conditions: [],
|
||||
query: 'test query',
|
||||
isSemantic: true,
|
||||
}),
|
||||
).rejects.toThrow('Embedding agent not configured or disabled for this team.')
|
||||
} finally {
|
||||
delete process.env.ENABLE_EMBEDDING_AGENT
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
|
||||
@@ -1,18 +1,19 @@
|
||||
import { prisma } from '@shumai/db'
|
||||
import { Prisma, AssetType, WorkflowTaskType } from '@shumai/db'
|
||||
import { Prisma, AssetType } from '@shumai/db'
|
||||
import { AssetService, assetService } from '@shumai/core/src/asset/asset'
|
||||
import { AssetInfo } from '@shumai/dtos'
|
||||
import { SearchRequest } from '@shumai/dtos'
|
||||
import { PaginatedData, decodeCursor, encodeCursor, PageInfo } from '@shumai/core/src/pagination'
|
||||
import { generateSearchNgrams } from '@shumai/core/src/utils/ngram'
|
||||
import { workflowService } from '@shumai/workflow-core'
|
||||
import { HTTPException } from 'hono/http-exception'
|
||||
import { SqlQueryBuilder } from './sql-query-builder'
|
||||
import { TextEmbeddingService, textEmbeddingService } from './text-embedding'
|
||||
|
||||
export class SearchService {
|
||||
constructor(
|
||||
private readonly prismaClient: typeof prisma = prisma,
|
||||
private readonly assetSvc: AssetService = assetService,
|
||||
private readonly embeddingSvc: TextEmbeddingService = textEmbeddingService,
|
||||
) {}
|
||||
|
||||
async search(folderId: string, req: SearchRequest): Promise<PaginatedData<AssetInfo[]>> {
|
||||
@@ -34,6 +35,12 @@ export class SearchService {
|
||||
const teamId = team?.project?.teamId
|
||||
if (!teamId) throw new Error('Team ID not found for folder')
|
||||
|
||||
if (process.env.ENABLE_EMBEDDING_AGENT === 'false') {
|
||||
throw new HTTPException(422, {
|
||||
message: 'Embedding agent not configured or disabled for this team.',
|
||||
})
|
||||
}
|
||||
|
||||
const embeddingAgent = await this.prismaClient.agent.findFirst({
|
||||
where: { teamId, type: 'embedding', enabled: true },
|
||||
})
|
||||
@@ -44,36 +51,43 @@ export class SearchService {
|
||||
})
|
||||
}
|
||||
|
||||
// 2. Generate query embedding via workflow
|
||||
const task = await this.prismaClient.workflowTask.create({
|
||||
data: {
|
||||
type: WorkflowTaskType.query_embedding_for_search,
|
||||
// 2. Generate query embedding directly using configured environment provider
|
||||
let queryVector: number[]
|
||||
try {
|
||||
const result = await this.embeddingSvc.generateEmbedding(req.query, {
|
||||
teamId,
|
||||
assetId: folderId,
|
||||
payload: {
|
||||
projectId: '',
|
||||
queryEmbeddingForSearch: { text: req.query },
|
||||
} as PrismaJson.WorkflowTaskPayload,
|
||||
status: 'pending',
|
||||
},
|
||||
})
|
||||
})
|
||||
queryVector = result.embedding
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err)
|
||||
throw new HTTPException(502, {
|
||||
message: `Failed to generate query embedding: ${msg}`,
|
||||
})
|
||||
}
|
||||
|
||||
const completedTask = await workflowService.executeWait(task)
|
||||
const output = completedTask.output as Record<string, unknown> | null
|
||||
const queryVector = output?.embedding as number[] | undefined
|
||||
|
||||
if (!queryVector) {
|
||||
throw new Error('Failed to generate query embedding')
|
||||
if (!queryVector || queryVector.length === 0) {
|
||||
throw new HTTPException(502, {
|
||||
message: 'Failed to generate query embedding: empty vector returned',
|
||||
})
|
||||
}
|
||||
|
||||
// 3. Construct raw SQL using SqlQueryBuilder
|
||||
const vectorJson = JSON.stringify(queryVector)
|
||||
const is768 = queryVector.length === 768
|
||||
const distanceExpr = is768
|
||||
? Prisma.sql`(ae.embedding_768 <=> ${vectorJson}::vector)`
|
||||
: Prisma.sql`(ae.embedding <=> ${vectorJson}::vector)`
|
||||
const notNullCond = is768
|
||||
? Prisma.sql`ae.embedding_768 IS NOT NULL`
|
||||
: Prisma.sql`ae.embedding IS NOT NULL`
|
||||
|
||||
const builder = new SqlQueryBuilder()
|
||||
.select(
|
||||
Prisma.sql`a.id as "assetId", ae.start_time as "startTime", ae.end_time as "endTime", (ae.embedding <=> ${vectorJson}::vector) as "distance"`,
|
||||
Prisma.sql`a.id as "assetId", ae.start_time as "startTime", ae.end_time as "endTime", ${distanceExpr} as "distance"`,
|
||||
)
|
||||
.from(Prisma.sql`assets a JOIN asset_embeddings ae ON a.id = ae.asset_id`)
|
||||
.addWhere(Prisma.sql`a.is_deleted = false`)
|
||||
.addWhere(notNullCond)
|
||||
|
||||
if (targetFolderIds.length > 0) {
|
||||
builder.addWhere(Prisma.sql`a.parent_id = ANY(${targetFolderIds})`)
|
||||
@@ -158,6 +172,7 @@ export class SearchService {
|
||||
.select(Prisma.sql`COUNT(*)`)
|
||||
.from(Prisma.sql`assets a JOIN asset_embeddings ae ON a.id = ae.asset_id`)
|
||||
.addWhere(Prisma.sql`a.is_deleted = false`)
|
||||
.addWhere(notNullCond)
|
||||
|
||||
if (targetFolderIds.length > 0) {
|
||||
countBuilder.addWhere(Prisma.sql`a.parent_id = ANY(${targetFolderIds})`)
|
||||
|
||||
@@ -0,0 +1,236 @@
|
||||
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'
|
||||
import { TextEmbeddingService } from './text-embedding'
|
||||
import { logger } from '@shumai/core/src/logger'
|
||||
|
||||
const mockEmbedContent = vi.fn()
|
||||
|
||||
vi.mock('@google/genai', () => {
|
||||
return {
|
||||
// eslint-disable-next-line @typescript-eslint/naming-convention
|
||||
GoogleGenAI: class {
|
||||
models = {
|
||||
embedContent: mockEmbedContent,
|
||||
}
|
||||
},
|
||||
}
|
||||
})
|
||||
|
||||
describe('TextEmbeddingService', () => {
|
||||
const originalEnv = { ...process.env }
|
||||
let service: TextEmbeddingService
|
||||
|
||||
beforeEach(() => {
|
||||
process.env = { ...originalEnv }
|
||||
process.env.GEMINI_API_KEY = 'test-gemini-key'
|
||||
delete process.env.EMBEDDING_PROVIDER
|
||||
delete process.env.GEMMA_EMBEDDING_OPENAI_BASE_URL
|
||||
mockEmbedContent.mockReset()
|
||||
service = new TextEmbeddingService(500, 3600 * 1000)
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
process.env = originalEnv
|
||||
vi.restoreAllMocks()
|
||||
})
|
||||
|
||||
it('throws an error if text is empty or whitespace', async () => {
|
||||
await expect(service.generateEmbedding('')).rejects.toThrow('Text to embed must not be empty')
|
||||
await expect(service.generateEmbedding(' ')).rejects.toThrow(
|
||||
'Text to embed must not be empty',
|
||||
)
|
||||
})
|
||||
|
||||
it('generates Gemini embedding (1536d) by default', async () => {
|
||||
const dummyEmbedding = Array(1536).fill(0.05)
|
||||
mockEmbedContent.mockResolvedValueOnce({
|
||||
embeddings: [{ values: dummyEmbedding }],
|
||||
})
|
||||
|
||||
const result = await service.generateEmbedding('beautiful sunset')
|
||||
|
||||
expect(result.embedding).toEqual(dummyEmbedding)
|
||||
expect(result.usage?.model).toBe('gemini-embedding-2')
|
||||
expect(result.cached).toBeUndefined()
|
||||
expect(mockEmbedContent).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
model: 'gemini-embedding-2',
|
||||
contents: [
|
||||
expect.objectContaining({
|
||||
parts: [{ text: 'beautiful sunset' }],
|
||||
}),
|
||||
],
|
||||
config: {
|
||||
outputDimensionality: 1536,
|
||||
},
|
||||
}),
|
||||
)
|
||||
})
|
||||
|
||||
it('returns cached result on repeated call without re-invoking the provider', async () => {
|
||||
const dummyEmbedding = Array(1536).fill(0.05)
|
||||
mockEmbedContent.mockResolvedValueOnce({
|
||||
embeddings: [{ values: dummyEmbedding }],
|
||||
})
|
||||
|
||||
const first = await service.generateEmbedding('ocean breeze')
|
||||
expect(first.embedding).toEqual(dummyEmbedding)
|
||||
expect(mockEmbedContent).toHaveBeenCalledTimes(1)
|
||||
|
||||
// Second call with same text and extra whitespace
|
||||
const second = await service.generateEmbedding(' ocean breeze ')
|
||||
expect(second.embedding).toEqual(dummyEmbedding)
|
||||
expect(second.cached).toBe(true)
|
||||
expect(mockEmbedContent).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
|
||||
it('preserves case distinctions in cache keys', async () => {
|
||||
const dummyApple = Array(1536).fill(0.1)
|
||||
const dummyLowerApple = Array(1536).fill(0.2)
|
||||
mockEmbedContent
|
||||
.mockResolvedValueOnce({
|
||||
embeddings: [{ values: dummyApple }],
|
||||
})
|
||||
.mockResolvedValueOnce({
|
||||
embeddings: [{ values: dummyLowerApple }],
|
||||
})
|
||||
|
||||
const upper = await service.generateEmbedding('Apple')
|
||||
expect(upper.embedding).toEqual(dummyApple)
|
||||
expect(mockEmbedContent).toHaveBeenCalledTimes(1)
|
||||
|
||||
const lower = await service.generateEmbedding('apple')
|
||||
expect(lower.embedding).toEqual(dummyLowerApple)
|
||||
expect(lower.cached).toBeUndefined()
|
||||
expect(mockEmbedContent).toHaveBeenCalledTimes(2)
|
||||
})
|
||||
|
||||
it('does not log raw query text on cache hits', async () => {
|
||||
const dummyEmbedding = Array(1536).fill(0.05)
|
||||
mockEmbedContent.mockResolvedValue({
|
||||
embeddings: [{ values: dummyEmbedding }],
|
||||
})
|
||||
const debugSpy = vi.spyOn(logger, 'debug')
|
||||
|
||||
await service.generateEmbedding('confidential search query')
|
||||
await service.generateEmbedding('confidential search query')
|
||||
|
||||
expect(debugSpy).toHaveBeenCalledWith(
|
||||
expect.not.objectContaining({
|
||||
key: expect.anything(),
|
||||
text: expect.anything(),
|
||||
}),
|
||||
'Cache hit for text embedding',
|
||||
)
|
||||
})
|
||||
|
||||
it('clears cache when clearCache is called', async () => {
|
||||
const dummyEmbedding = Array(1536).fill(0.05)
|
||||
mockEmbedContent.mockResolvedValue({
|
||||
embeddings: [{ values: dummyEmbedding }],
|
||||
})
|
||||
|
||||
await service.generateEmbedding('query')
|
||||
expect(mockEmbedContent).toHaveBeenCalledTimes(1)
|
||||
|
||||
service.clearCache()
|
||||
|
||||
await service.generateEmbedding('query')
|
||||
expect(mockEmbedContent).toHaveBeenCalledTimes(2)
|
||||
})
|
||||
|
||||
it('throws if GEMINI_API_KEY is not configured', async () => {
|
||||
delete process.env.GEMINI_API_KEY
|
||||
await expect(service.generateEmbedding('test')).rejects.toThrow(
|
||||
'GEMINI_API_KEY environment variable is not configured',
|
||||
)
|
||||
})
|
||||
|
||||
it('throws if Gemini returns empty embedding values', async () => {
|
||||
mockEmbedContent.mockResolvedValueOnce({
|
||||
embeddings: [{ values: null }],
|
||||
})
|
||||
await expect(service.generateEmbedding('test')).rejects.toThrow(
|
||||
'No embedding values returned from gemini-embedding-2',
|
||||
)
|
||||
})
|
||||
|
||||
it('generates Gemma embedding (768d) when provider is gemma-embedding-2-openai', async () => {
|
||||
const dummyEmbedding = Array(768).fill(0.12)
|
||||
const mockFetch = vi.fn().mockResolvedValue({
|
||||
ok: true,
|
||||
json: async () => ({
|
||||
model: 'embeddinggemma-2',
|
||||
data: [{ embedding: dummyEmbedding }],
|
||||
// eslint-disable-next-line @typescript-eslint/naming-convention
|
||||
usage: { prompt_tokens: 12 },
|
||||
}),
|
||||
})
|
||||
vi.stubGlobal('fetch', mockFetch)
|
||||
|
||||
const result = await service.generateEmbedding('drone footage', {
|
||||
provider: 'gemma-embedding-2-openai',
|
||||
})
|
||||
|
||||
expect(result.embedding).toEqual(dummyEmbedding)
|
||||
expect(result.usage?.model).toBe('embeddinggemma-2')
|
||||
expect(result.usage?.inputTokens).toBe(12)
|
||||
expect(mockFetch).toHaveBeenCalledWith(
|
||||
'http://localhost:9379/v1/embeddings',
|
||||
expect.objectContaining({
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
model: 'embeddinggemma-2',
|
||||
input: 'task: search query | text: drone footage',
|
||||
}),
|
||||
}),
|
||||
)
|
||||
})
|
||||
|
||||
it('respects GEMMA_EMBEDDING_OPENAI_BASE_URL without /v1 suffix', async () => {
|
||||
process.env.GEMMA_EMBEDDING_OPENAI_BASE_URL = 'http://custom-host:9379'
|
||||
const dummyEmbedding = Array(768).fill(0.12)
|
||||
const mockFetch = vi.fn().mockResolvedValue({
|
||||
ok: true,
|
||||
json: async () => ({
|
||||
data: [{ embedding: dummyEmbedding }],
|
||||
}),
|
||||
})
|
||||
vi.stubGlobal('fetch', mockFetch)
|
||||
|
||||
await service.generateEmbedding('test query', {
|
||||
provider: 'gemma-embedding-2-openai',
|
||||
})
|
||||
|
||||
expect(mockFetch).toHaveBeenCalledWith(
|
||||
'http://custom-host:9379/v1/embeddings',
|
||||
expect.anything(),
|
||||
)
|
||||
})
|
||||
|
||||
it('throws descriptive error when Gemma server returns non-ok status', async () => {
|
||||
const mockFetch = vi.fn().mockResolvedValue({
|
||||
ok: false,
|
||||
status: 503,
|
||||
text: async () => 'Service Unavailable',
|
||||
})
|
||||
vi.stubGlobal('fetch', mockFetch)
|
||||
|
||||
await expect(
|
||||
service.generateEmbedding('test query', {
|
||||
provider: 'gemma-embedding-2-openai',
|
||||
}),
|
||||
).rejects.toThrow('Embedding server returned status 503: Service Unavailable')
|
||||
})
|
||||
|
||||
it('throws error when abort signal is triggered', async () => {
|
||||
const controller = new AbortController()
|
||||
controller.abort(new Error('Operation cancelled by user'))
|
||||
|
||||
await expect(
|
||||
service.generateEmbedding('test query', {
|
||||
signal: controller.signal,
|
||||
}),
|
||||
).rejects.toThrow('Operation cancelled by user')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,196 @@
|
||||
import { GoogleGenAI } from '@google/genai'
|
||||
import { LruTtlCache } from '@shumai/core/src/cache/lru-ttl-cache'
|
||||
import { logger } from '@shumai/core/src/logger'
|
||||
|
||||
export interface TextEmbeddingUsage {
|
||||
model: string
|
||||
inputTokens: number
|
||||
outputTokens: number
|
||||
}
|
||||
|
||||
export interface TextEmbeddingResult {
|
||||
embedding: number[]
|
||||
usage?: TextEmbeddingUsage
|
||||
cached?: boolean
|
||||
}
|
||||
|
||||
export interface GenerateTextEmbeddingOptions {
|
||||
provider?: string
|
||||
model?: string
|
||||
signal?: AbortSignal
|
||||
teamId?: string
|
||||
skipCache?: boolean
|
||||
}
|
||||
|
||||
export class TextEmbeddingService {
|
||||
private cache: LruTtlCache<string, number[]>
|
||||
|
||||
constructor(
|
||||
cacheCapacity: number = 500,
|
||||
private defaultTtlMs: number = 3600 * 1000,
|
||||
) {
|
||||
this.cache = new LruTtlCache<string, number[]>(cacheCapacity, 50)
|
||||
}
|
||||
|
||||
/**
|
||||
* Generates a text embedding vector directly using the configured provider.
|
||||
* If a cached embedding exists for the text, provider, and model, it is returned immediately.
|
||||
*/
|
||||
async generateEmbedding(
|
||||
text: string,
|
||||
options?: GenerateTextEmbeddingOptions,
|
||||
): Promise<TextEmbeddingResult> {
|
||||
const trimmedText = text.trim()
|
||||
if (!trimmedText) {
|
||||
throw new Error('Text to embed must not be empty')
|
||||
}
|
||||
|
||||
const provider = options?.provider || process.env.EMBEDDING_PROVIDER || 'gemini-embedding-2'
|
||||
const model =
|
||||
options?.model ||
|
||||
(provider === 'gemma-embedding-2-openai' ? 'embeddinggemma-2' : 'gemini-embedding-2')
|
||||
|
||||
const cacheKey = `${provider}:${model}:${trimmedText}`
|
||||
if (!options?.skipCache) {
|
||||
const cachedVector = this.cache.get(cacheKey)
|
||||
if (cachedVector) {
|
||||
logger.debug({ provider, model }, 'Cache hit for text embedding')
|
||||
return {
|
||||
embedding: cachedVector,
|
||||
usage: { model, inputTokens: 0, outputTokens: 0 },
|
||||
cached: true,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const signal = options?.signal ?? AbortSignal.timeout(15000)
|
||||
|
||||
let result: TextEmbeddingResult
|
||||
if (provider === 'gemma-embedding-2-openai') {
|
||||
result = await this.callGemmaEmbeddingServer(trimmedText, model, signal)
|
||||
} else {
|
||||
result = await this.callGeminiEmbedding(trimmedText, model, signal)
|
||||
}
|
||||
|
||||
if (!options?.skipCache) {
|
||||
this.cache.set(cacheKey, result.embedding, this.defaultTtlMs)
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
private async callGeminiEmbedding(
|
||||
text: string,
|
||||
model: string,
|
||||
signal: AbortSignal,
|
||||
): Promise<TextEmbeddingResult> {
|
||||
const apiKey = process.env.GEMINI_API_KEY
|
||||
if (!apiKey) {
|
||||
throw new Error('GEMINI_API_KEY environment variable is not configured')
|
||||
}
|
||||
|
||||
if (signal.aborted) {
|
||||
throw signal.reason ?? new Error('Embedding generation aborted or timed out')
|
||||
}
|
||||
|
||||
const resolvedApiKey = process.env[apiKey] || apiKey
|
||||
const ai = new GoogleGenAI({ apiKey: resolvedApiKey })
|
||||
|
||||
const embedPromise = ai.models.embedContent({
|
||||
model,
|
||||
contents: [
|
||||
{
|
||||
parts: [{ text }],
|
||||
role: 'user',
|
||||
},
|
||||
],
|
||||
config: {
|
||||
outputDimensionality: 1536,
|
||||
},
|
||||
})
|
||||
|
||||
const abortPromise = new Promise<never>((_, reject) => {
|
||||
if (signal.aborted) {
|
||||
reject(signal.reason ?? new Error('Embedding generation aborted or timed out'))
|
||||
return
|
||||
}
|
||||
signal.addEventListener(
|
||||
'abort',
|
||||
() => {
|
||||
reject(signal.reason ?? new Error('Embedding generation aborted or timed out'))
|
||||
},
|
||||
{ once: true },
|
||||
)
|
||||
})
|
||||
|
||||
const response = await Promise.race([embedPromise, abortPromise])
|
||||
const values = response.embeddings?.[0]?.values
|
||||
if (!values || !Array.isArray(values)) {
|
||||
throw new Error(`No embedding values returned from ${model}`)
|
||||
}
|
||||
|
||||
return {
|
||||
embedding: values,
|
||||
usage: {
|
||||
model,
|
||||
inputTokens: 0,
|
||||
outputTokens: 0,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
private async callGemmaEmbeddingServer(
|
||||
text: string,
|
||||
model: string,
|
||||
signal: AbortSignal,
|
||||
): Promise<TextEmbeddingResult> {
|
||||
const rawBaseUrl = process.env.GEMMA_EMBEDDING_OPENAI_BASE_URL || 'http://localhost:9379/v1'
|
||||
const normalized = rawBaseUrl.replace(/\/+$/, '')
|
||||
const endpoint = normalized.endsWith('/embeddings')
|
||||
? normalized
|
||||
: normalized.endsWith('/v1')
|
||||
? `${normalized}/embeddings`
|
||||
: `${normalized}/v1/embeddings`
|
||||
|
||||
const response = await fetch(endpoint, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
model,
|
||||
input: `task: search query | text: ${text}`,
|
||||
}),
|
||||
signal,
|
||||
})
|
||||
|
||||
if (!response.ok) {
|
||||
const errorText = await response.text()
|
||||
throw new Error(`Embedding server returned status ${response.status}: ${errorText}`)
|
||||
}
|
||||
|
||||
const json = (await response.json()) as {
|
||||
model?: string
|
||||
data?: Array<{ embedding?: number[] }>
|
||||
// eslint-disable-next-line @typescript-eslint/naming-convention
|
||||
usage?: { prompt_tokens?: number }
|
||||
}
|
||||
|
||||
const embedding = json.data?.[0]?.embedding
|
||||
if (!embedding || !Array.isArray(embedding)) {
|
||||
throw new Error('Invalid embedding returned from embedding server')
|
||||
}
|
||||
|
||||
return {
|
||||
embedding,
|
||||
usage: {
|
||||
model: json.model || model,
|
||||
inputTokens: json.usage?.prompt_tokens ?? 0,
|
||||
outputTokens: 0,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
clearCache(): void {
|
||||
this.cache.clear()
|
||||
}
|
||||
}
|
||||
|
||||
export const textEmbeddingService = new TextEmbeddingService()
|
||||
@@ -605,4 +605,32 @@ describe('TeamService', () => {
|
||||
expect(settings.appearance?.hideAgent).toBe(true)
|
||||
expect((settings as Record<string, unknown>).emailNotification).toBeUndefined()
|
||||
})
|
||||
|
||||
it('should report semanticSearchEnabled false when ENABLE_EMBEDDING_AGENT is false even if agent is enabled in db', async () => {
|
||||
const owner = await prisma.user.create({
|
||||
data: { name: 'Owner', email: 'owner-emb-false@example.com', password: 'pw' },
|
||||
})
|
||||
const team = await teamService.createTeam(owner, { name: 'Emb False Team' })
|
||||
|
||||
const bot = await prisma.user.create({
|
||||
data: { name: 'Bot', email: 'bot-emb-false@example.com', password: 'pw', type: 'agent' },
|
||||
})
|
||||
await prisma.agent.create({
|
||||
data: {
|
||||
id: bot.id,
|
||||
teamId: team.id,
|
||||
type: 'embedding',
|
||||
enabled: true,
|
||||
config: { provider: '', model: '' },
|
||||
},
|
||||
})
|
||||
|
||||
process.env.ENABLE_EMBEDDING_AGENT = 'false'
|
||||
try {
|
||||
const settings = await teamService.getSettings(team.id)
|
||||
expect(settings.semanticSearchEnabled).toBe(false)
|
||||
} finally {
|
||||
delete process.env.ENABLE_EMBEDDING_AGENT
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
@@ -6,6 +6,7 @@ import { notificationService } from '@shumai/core/src/notification/notification'
|
||||
import { getAvatarUrl } from '@shumai/core/src/user/avatar'
|
||||
import { HTTPException } from 'hono/http-exception'
|
||||
import { getAllowedAgentRoles } from '@shumai/core/src/agent/permissions'
|
||||
import { AgentService, agentService } from '@shumai/core/src/agent/agent'
|
||||
import { sandboxService } from '@shumai/core/src/sandbox/sandbox-service'
|
||||
import { turntableService } from '@shumai/core/src/turntable/turntable'
|
||||
import { gotenbergService } from '@shumai/core/src/gotenberg/gotenberg'
|
||||
@@ -46,6 +47,7 @@ export class TeamService {
|
||||
},
|
||||
})
|
||||
await providerService.initBuiltinProviders(team.id)
|
||||
await agentService.ensureEmbeddingAgentForTeam(team.id)
|
||||
}
|
||||
|
||||
return team
|
||||
@@ -86,10 +88,12 @@ export class TeamService {
|
||||
},
|
||||
})
|
||||
|
||||
// We use 'any' here because ProviderService expects the full PrismaClient,
|
||||
// We use 'any' here because ProviderService/AgentService expect the full PrismaClient,
|
||||
// but the TransactionClient is compatible with the methods we use.
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
await new ProviderService(tx as any).initBuiltinProviders(team.id)
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
await new AgentService(tx as any).ensureEmbeddingAgentForTeam(team.id)
|
||||
|
||||
return this.toTeamInfo(updatedTeam)
|
||||
})
|
||||
@@ -307,9 +311,12 @@ export class TeamService {
|
||||
delete rawSettings.emailNotification
|
||||
const settings = rawSettings as TeamSettingsResponse
|
||||
|
||||
const embeddingAgent = await prisma.agent.findFirst({
|
||||
where: { teamId, type: 'embedding', enabled: true },
|
||||
})
|
||||
const embeddingAgent =
|
||||
process.env.ENABLE_EMBEDDING_AGENT === 'false'
|
||||
? null
|
||||
: await prisma.agent.findFirst({
|
||||
where: { teamId, type: 'embedding', enabled: true },
|
||||
})
|
||||
settings.semanticSearchEnabled = !!embeddingAgent
|
||||
|
||||
if (!settings.transcode) {
|
||||
|
||||
@@ -501,7 +501,6 @@ describe('UploadService', () => {
|
||||
where: { assetId: asset.id, type: WorkflowTaskType.transcode_pdf },
|
||||
})
|
||||
expect(workflowTask).not.toBeNull()
|
||||
expect(workflowTask?.status).toBe(TaskStatus.pending)
|
||||
|
||||
// Test HTML document as well
|
||||
const htmlTask = await prisma.task.create({
|
||||
@@ -530,7 +529,6 @@ describe('UploadService', () => {
|
||||
where: { assetId: htmlAsset.id, type: WorkflowTaskType.transcode_pdf },
|
||||
})
|
||||
expect(htmlWorkflowTask).not.toBeNull()
|
||||
expect(htmlWorkflowTask?.status).toBe(TaskStatus.pending)
|
||||
} finally {
|
||||
if (originalEnv !== undefined) {
|
||||
process.env.GOTENBERG_URL = originalEnv
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@shumai/db",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "./src/index.ts",
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
-- AlterTable
|
||||
ALTER TABLE "asset_embeddings" ADD COLUMN "embedding_768" vector(768);
|
||||
|
||||
-- CreateIndex: HNSW index on asset_embeddings.embedding_768
|
||||
CREATE INDEX "AssetEmbedding_embedding_768_idx"
|
||||
ON "asset_embeddings"
|
||||
USING hnsw (embedding_768 vector_cosine_ops)
|
||||
WITH (m = 16, ef_construction = 64);
|
||||
@@ -706,17 +706,19 @@ model AssetCommentReaction {
|
||||
}
|
||||
|
||||
model AssetEmbedding {
|
||||
id String @id @default(ulid())
|
||||
embedding Unsupported("vector(1536)")?
|
||||
startTime Float? @map("start_time")
|
||||
endTime Float? @map("end_time")
|
||||
createdAt DateTime @default(now()) @map("created_at")
|
||||
updatedAt DateTime @updatedAt @map("updated_at")
|
||||
id String @id @default(ulid())
|
||||
embedding Unsupported("vector(1536)")?
|
||||
embedding768 Unsupported("vector(768)")? @map("embedding_768")
|
||||
startTime Float? @map("start_time")
|
||||
endTime Float? @map("end_time")
|
||||
createdAt DateTime @default(now()) @map("created_at")
|
||||
updatedAt DateTime @updatedAt @map("updated_at")
|
||||
|
||||
assetId String @map("asset_id")
|
||||
asset Asset @relation("AssetToEmbedding", fields: [assetId], references: [id], onDelete: Cascade)
|
||||
assetId String @map("asset_id")
|
||||
asset Asset @relation("AssetToEmbedding", fields: [assetId], references: [id], onDelete: Cascade)
|
||||
|
||||
@@index([embedding], name: "AssetEmbedding_embedding_idx")
|
||||
@@index([embedding768], name: "AssetEmbedding_embedding_768_idx")
|
||||
@@map("asset_embeddings")
|
||||
}
|
||||
|
||||
|
||||
+1
-1
File diff suppressed because one or more lines are too long
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@shumai/dtos",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "./src/index.ts",
|
||||
|
||||
@@ -42,6 +42,7 @@ export const agentInfoSchema = z.object({
|
||||
skills: z.array(agentSkillSchema).optional(),
|
||||
mcpServerIds: z.array(z.string()).optional(),
|
||||
deniedTools: z.array(z.string()).optional(),
|
||||
readOnly: z.boolean().optional(),
|
||||
})
|
||||
export type AgentInfo = z.infer<typeof agentInfoSchema>
|
||||
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
{
|
||||
"name": "@shumai/e2e",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"dependencies": {
|
||||
"@earendil-works/pi-agent-core": "0.82.1",
|
||||
"@google/genai": "2.19.0",
|
||||
"@google/genai": "2.28.0",
|
||||
"@shumai/agent": "workspace:*",
|
||||
"@shumai/core": "workspace:*",
|
||||
"@shumai/db": "workspace:*",
|
||||
|
||||
@@ -1,127 +0,0 @@
|
||||
import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest'
|
||||
import { prisma } from '@shumai/db'
|
||||
import { setupTestDbHooks } from '@shumai/db/test'
|
||||
import { workflowService, TaskQueueAgent } from '@shumai/workflow-core'
|
||||
import { initAgentWorkflows } from '@shumai/agent'
|
||||
import { s3Service } from '@shumai/core/src/s3/s3'
|
||||
const mockEmbedContent = vi.fn().mockResolvedValue({
|
||||
embeddings: [
|
||||
{
|
||||
values: Array(1536).fill(0.35),
|
||||
},
|
||||
],
|
||||
})
|
||||
|
||||
vi.mock('@google/genai', () => {
|
||||
return {
|
||||
// eslint-disable-next-line @typescript-eslint/naming-convention
|
||||
GoogleGenAI: class {
|
||||
models = {
|
||||
embedContent: mockEmbedContent,
|
||||
}
|
||||
},
|
||||
}
|
||||
})
|
||||
|
||||
import { fileURLToPath } from 'url'
|
||||
import * as path from 'path'
|
||||
|
||||
const currentDir = path.dirname(fileURLToPath(import.meta.url))
|
||||
const agentWorkflowsPath = path.resolve(currentDir, '../../../apps/agent/src/workflows.ts')
|
||||
|
||||
describe.each(['local', 'temporal'] as const)(
|
||||
'Workflow E2E - queryEmbeddingForSearch (executor: %s)',
|
||||
(mode) => {
|
||||
setupTestDbHooks()
|
||||
|
||||
let agentWorkerPromise: Promise<void> | null = null
|
||||
|
||||
beforeAll(async () => {
|
||||
process.env.S3_BUCKET = 'shumai-e2e-test-bucket-query'
|
||||
process.env.GEMINI_API_KEY = 'dummy-key'
|
||||
|
||||
workflowService.setExecutorType(mode)
|
||||
initAgentWorkflows()
|
||||
|
||||
if (mode === 'temporal') {
|
||||
console.log('Starting background worker for query embedding Temporal E2E tests...')
|
||||
agentWorkerPromise = workflowService.startWorkers(TaskQueueAgent, {
|
||||
workflowsPath: agentWorkflowsPath,
|
||||
})
|
||||
await new Promise((resolve) => setTimeout(resolve, 2000))
|
||||
} else {
|
||||
console.log('Starting local workflow service polling...')
|
||||
workflowService.start()
|
||||
}
|
||||
})
|
||||
|
||||
afterAll(async () => {
|
||||
if (mode === 'temporal') {
|
||||
console.log('Shutting down Temporal workers...')
|
||||
await workflowService.shutdownWorkers()
|
||||
await Promise.all([agentWorkerPromise].filter(Boolean))
|
||||
}
|
||||
workflowService.close()
|
||||
vi.restoreAllMocks()
|
||||
try {
|
||||
console.log('Cleaning up local E2E storage files...')
|
||||
await s3Service.deletePrefix('shumai-e2e-test-bucket-query', '')
|
||||
} catch (err) {
|
||||
console.error('Failed to clean up E2E storage folder:', err)
|
||||
}
|
||||
})
|
||||
|
||||
it('should run queryEmbeddingForSearch workflow and return embedding successfully', async () => {
|
||||
// 1. Seed Database
|
||||
const team = await prisma.team.create({
|
||||
data: { name: 'E2E Query Embedding Team' },
|
||||
})
|
||||
|
||||
const project = await prisma.project.create({
|
||||
data: { name: 'E2E Query Embedding Project', teamId: team.id },
|
||||
})
|
||||
|
||||
const asset = await prisma.asset.create({
|
||||
data: {
|
||||
name: 'dummy-search-asset',
|
||||
type: 'file',
|
||||
status: 'uploaded',
|
||||
mediaType: 'text/plain',
|
||||
projectId: project.id,
|
||||
},
|
||||
})
|
||||
|
||||
// 2. Create Workflow Task
|
||||
const task = await prisma.workflowTask.create({
|
||||
data: {
|
||||
type: 'query_embedding_for_search',
|
||||
status: 'pending',
|
||||
teamId: team.id,
|
||||
projectId: project.id,
|
||||
assetId: asset.id,
|
||||
payload: {
|
||||
projectId: project.id,
|
||||
queryEmbeddingForSearch: {
|
||||
text: 'test search query',
|
||||
},
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
// 3. Wait for workflow to complete
|
||||
console.log(
|
||||
`Submitted E2E Query Embedding Workflow Task. ID: ${task.id}. Awaiting completion...`,
|
||||
)
|
||||
const completedTask = await workflowService.executeWait(task, 45000)
|
||||
|
||||
// 4. Verification
|
||||
expect(completedTask.status).toBe('completed')
|
||||
|
||||
// Verify output payload contains correct embedding values and usage metadata
|
||||
const output = completedTask.output as unknown as { embedding: number[] }
|
||||
expect(output).toBeDefined()
|
||||
expect(output.embedding).toEqual(Array(1536).fill(0.35))
|
||||
expect(completedTask.model).toBe('gemini-embedding-2')
|
||||
}, 50000)
|
||||
},
|
||||
)
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@shumai/tableprinter",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "./src/index.ts",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@shumai/transcode",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "./src/index.ts",
|
||||
|
||||
@@ -1230,6 +1230,100 @@ describe('Transcode Activities', () => {
|
||||
expect(res.chunkKey).toMatch(/^files\/asset-abc\/tmp-embedding-chunks\/chunk-10-25-.*\.mp4$/)
|
||||
})
|
||||
|
||||
it('should extract frames archive when format is frames', async () => {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
;(child_process.execFile as any).mockImplementation(
|
||||
(
|
||||
_file: string,
|
||||
args: string[],
|
||||
cb: (err: Error | null, result: { stdout: string; stderr: string }) => void,
|
||||
) => {
|
||||
if (_file === 'ffmpeg' && args) {
|
||||
const outPattern = args[args.length - 1]
|
||||
if (outPattern && outPattern.includes('frame_%04d.jpg')) {
|
||||
// eslint-disable-next-line @typescript-eslint/no-require-imports
|
||||
const fsActual = require('fs')
|
||||
// eslint-disable-next-line @typescript-eslint/no-require-imports
|
||||
const pathActual = require('path')
|
||||
const dir = pathActual.dirname(outPattern)
|
||||
fsActual.writeFileSync(
|
||||
pathActual.join(dir, 'frame_0001.jpg'),
|
||||
Buffer.from('mock-frame'),
|
||||
)
|
||||
}
|
||||
}
|
||||
cb(null, { stdout: '', stderr: '' })
|
||||
},
|
||||
)
|
||||
|
||||
const res = await transcodeVideoChunkActivity({
|
||||
assetId: 'asset-abc',
|
||||
filePath: '/tmp/full-video.mp4',
|
||||
startTime: 10,
|
||||
endTime: 25,
|
||||
format: 'frames',
|
||||
})
|
||||
|
||||
expect(child_process.execFile).toHaveBeenCalledWith(
|
||||
'ffmpeg',
|
||||
expect.arrayContaining(['-vf', 'fps=1', '-q:v', '2']),
|
||||
expect.any(Function),
|
||||
)
|
||||
expect(child_process.execFile).not.toHaveBeenCalledWith(
|
||||
'tar',
|
||||
expect.anything(),
|
||||
expect.anything(),
|
||||
)
|
||||
expect(s3Service.putObject).toHaveBeenCalledWith(
|
||||
'shumai',
|
||||
expect.stringMatching(/^files\/asset-abc\/tmp-embedding-chunks\/frames-10-25-.*\.tar$/),
|
||||
expect.any(Buffer),
|
||||
expect.any(Number),
|
||||
'application/x-tar',
|
||||
)
|
||||
expect(res.chunkKey).toMatch(/^files\/asset-abc\/tmp-embedding-chunks\/frames-10-25-.*\.tar$/)
|
||||
|
||||
const putCall = vi.mocked(s3Service.putObject).mock.calls[0]
|
||||
const uploadedBuffer = putCall[2] as Buffer
|
||||
const uploadedArchive = new Bun.Archive(uploadedBuffer)
|
||||
const archiveFiles = await uploadedArchive.files()
|
||||
expect(archiveFiles.has('frame_0001.jpg')).toBe(true)
|
||||
const uploadedFile = archiveFiles.get('frame_0001.jpg')!
|
||||
expect(uploadedFile.size).toBeGreaterThan(0)
|
||||
const uploadedFileBytes = await uploadedFile.bytes()
|
||||
expect(uploadedFileBytes.byteLength).toBeGreaterThan(0)
|
||||
})
|
||||
|
||||
it('should sample uniform 16 frames when chunk duration > 16s', async () => {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- child_process.execFile mockImplementation returns void rather than ChildProcess in unit tests
|
||||
;(child_process.execFile as any).mockImplementation(
|
||||
(
|
||||
_cmd: unknown,
|
||||
_args: unknown,
|
||||
cb: (err: null, res: { stdout: string; stderr: string }) => void,
|
||||
) => {
|
||||
cb(null, {
|
||||
stdout: '',
|
||||
stderr: '',
|
||||
})
|
||||
},
|
||||
)
|
||||
|
||||
await transcodeVideoChunkActivity({
|
||||
assetId: 'asset-abc',
|
||||
filePath: '/tmp/full-video.mp4',
|
||||
startTime: 0,
|
||||
endTime: 60,
|
||||
format: 'frames',
|
||||
})
|
||||
|
||||
expect(child_process.execFile).toHaveBeenCalledWith(
|
||||
'ffmpeg',
|
||||
expect.arrayContaining(['-vf', `fps=${16 / 60}`, '-vframes', '16', '-q:v', '2']),
|
||||
expect.any(Function),
|
||||
)
|
||||
})
|
||||
|
||||
it('should throw non-retryable ApplicationFailure when neither filePath nor assetKey is provided', async () => {
|
||||
expect.assertions(2)
|
||||
try {
|
||||
@@ -1828,5 +1922,67 @@ describe('Transcode Activities', () => {
|
||||
})
|
||||
expect(task).toBeNull()
|
||||
})
|
||||
|
||||
it('should NOT create ai_embedding task when ENABLE_EMBEDDING_AGENT is false', async () => {
|
||||
const user = await prisma.user.create({
|
||||
data: { name: 'Test User Embedding 3', email: 'test-embedding-agent-3@test.com' },
|
||||
})
|
||||
const team = await prisma.team.create({
|
||||
data: { name: 'Test Team Embedding 3' },
|
||||
})
|
||||
await prisma.teamMember.create({
|
||||
data: { teamId: team.id, userId: user.id, role: 'owner' },
|
||||
})
|
||||
const project = await prisma.project.create({
|
||||
data: { name: 'Test Project Embedding 3', teamId: team.id },
|
||||
})
|
||||
await prisma.agent.create({
|
||||
data: {
|
||||
id: user.id,
|
||||
type: 'embedding',
|
||||
enabled: true,
|
||||
teamId: team.id,
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
config: { provider: 'test', model: 'test' } as any,
|
||||
},
|
||||
})
|
||||
const asset = await prisma.asset.create({
|
||||
data: {
|
||||
name: 'robot.glb',
|
||||
mediaType: 'model/gltf-binary',
|
||||
type: 'file',
|
||||
status: 'processed',
|
||||
projectId: project.id,
|
||||
media: {
|
||||
proxyType: '3d',
|
||||
imageTranscodes: [
|
||||
{
|
||||
key: 'files/asset-1/views.webp',
|
||||
width: 2880,
|
||||
height: 2880,
|
||||
quality: 85,
|
||||
format: 'webp',
|
||||
},
|
||||
],
|
||||
} as unknown as PrismaJson.MediaInfo,
|
||||
},
|
||||
})
|
||||
|
||||
process.env.ENABLE_EMBEDDING_AGENT = 'false'
|
||||
try {
|
||||
await createEmbeddingTaskIfEnabledActivity({
|
||||
assetId: asset.id,
|
||||
teamId: team.id,
|
||||
projectId: project.id,
|
||||
})
|
||||
|
||||
const task = await prisma.workflowTask.findFirst({
|
||||
where: { assetId: asset.id, type: 'ai_embedding' },
|
||||
})
|
||||
expect(task).toBeNull()
|
||||
} finally {
|
||||
delete process.env.ENABLE_EMBEDDING_AGENT
|
||||
}
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
@@ -1372,7 +1372,7 @@ export async function createEmbeddingTaskIfEnabledActivity(
|
||||
const teamId = params.teamId || asset.project?.teamId || null
|
||||
const projectId = params.projectId || asset.projectId
|
||||
|
||||
if (!teamId) {
|
||||
if (!teamId || process.env.ENABLE_EMBEDDING_AGENT === 'false') {
|
||||
return
|
||||
}
|
||||
|
||||
@@ -1490,6 +1490,7 @@ export interface TranscodeVideoChunkParams {
|
||||
filePath?: string
|
||||
startTime: number
|
||||
endTime: number
|
||||
format?: 'video' | 'frames'
|
||||
}
|
||||
|
||||
export async function transcodeVideoChunkActivity(
|
||||
@@ -1502,6 +1503,82 @@ export async function transcodeVideoChunkActivity(
|
||||
})
|
||||
}
|
||||
|
||||
if (params.format === 'frames') {
|
||||
const tmpDir = transcodeService.createTempDir('video-chunk-frames-')
|
||||
try {
|
||||
const bucket = process.env.S3_BUCKET || 'shumai'
|
||||
const inputSource =
|
||||
params.filePath ||
|
||||
(params.assetKey ? await s3Service.resolveInput(bucket, params.assetKey) : '')
|
||||
if (!inputSource) {
|
||||
throw ApplicationFailure.create({
|
||||
message: 'Neither filePath nor assetKey was provided for video chunk transcoding',
|
||||
nonRetryable: true,
|
||||
})
|
||||
}
|
||||
const isRemote = inputSource.startsWith('http://') || inputSource.startsWith('https://')
|
||||
|
||||
const duration = Math.max(0.1, params.endTime - params.startTime)
|
||||
const fpsFilter = duration > 16 ? `fps=${16 / duration}` : 'fps=1'
|
||||
const vframesArgs = duration > 16 ? ['-vframes', '16'] : []
|
||||
|
||||
await execFileAsync('ffmpeg', [
|
||||
'-y',
|
||||
'-loglevel',
|
||||
'warning',
|
||||
...(isRemote
|
||||
? ['-reconnect', '1', '-reconnect_streamed', '1', '-reconnect_delay_max', '5']
|
||||
: []),
|
||||
'-ss',
|
||||
params.startTime.toString(),
|
||||
'-i',
|
||||
inputSource,
|
||||
'-map',
|
||||
'0:V:0',
|
||||
'-t',
|
||||
duration.toString(),
|
||||
'-vf',
|
||||
fpsFilter,
|
||||
...vframesArgs,
|
||||
'-q:v',
|
||||
'2',
|
||||
path.join(tmpDir, 'frame_%04d.jpg'),
|
||||
])
|
||||
|
||||
const frameFiles = fs
|
||||
.readdirSync(tmpDir)
|
||||
.filter((f) => /\.(jpe?g|png|webp)$/i.test(f))
|
||||
.sort()
|
||||
|
||||
const archiveEntries: Record<string, Buffer> = {}
|
||||
for (const file of frameFiles) {
|
||||
archiveEntries[file] = fs.readFileSync(path.join(tmpDir, file))
|
||||
}
|
||||
|
||||
const archive = new Bun.Archive(archiveEntries)
|
||||
const archiveBytes = await archive.bytes()
|
||||
const archiveData = Buffer.from(
|
||||
archiveBytes.buffer,
|
||||
archiveBytes.byteOffset,
|
||||
archiveBytes.byteLength,
|
||||
)
|
||||
const key = `files/${params.assetId}/tmp-embedding-chunks/frames-${params.startTime}-${params.endTime}-${ulid()}.tar`
|
||||
await s3Service.putObject(bucket, key, archiveData, archiveData.length, 'application/x-tar')
|
||||
|
||||
return { chunkKey: key }
|
||||
} catch (err) {
|
||||
if (err instanceof ApplicationFailure) {
|
||||
throw err
|
||||
}
|
||||
throw ApplicationFailure.create({
|
||||
message: `Failed to extract video chunk frames for ${params.startTime}-${params.endTime}: ${err instanceof Error ? err.message : String(err)}`,
|
||||
nonRetryable: false,
|
||||
})
|
||||
} finally {
|
||||
transcodeService.removeDir(tmpDir)
|
||||
}
|
||||
}
|
||||
|
||||
const chunkTmp = path.join(os.tmpdir(), `video-chunk-${Date.now()}-${ulid()}.mp4`)
|
||||
try {
|
||||
const bucket = process.env.S3_BUCKET || 'shumai'
|
||||
|
||||
@@ -441,4 +441,39 @@ describe('FileCard', () => {
|
||||
expect(tooltip.textContent).toContain('FFmpeg transcode failed: exit code 1')
|
||||
vi.useRealTimers()
|
||||
})
|
||||
|
||||
it('does not render selection checkbox when hideCheckbox is true', () => {
|
||||
renderComponent({ item: fileItem, hideCheckbox: true })
|
||||
expect(screen.queryByRole('checkbox')).toBeNull()
|
||||
})
|
||||
|
||||
it('renders selection checkbox by default when hideCheckbox is false or omitted', () => {
|
||||
renderComponent({ item: fileItem })
|
||||
expect(screen.getByRole('checkbox')).toBeTruthy()
|
||||
})
|
||||
|
||||
it('does not render 3-dot dropdown menu when hideActions is true', () => {
|
||||
renderComponent({ item: fileItem, hideActions: true })
|
||||
// No dropdown trigger button rendered
|
||||
expect(screen.queryByRole('button')).toBeNull()
|
||||
})
|
||||
|
||||
it('renders matched duration chunk in preview when startTime and endTime are present', () => {
|
||||
const videoWithMatch: AssetInfo = {
|
||||
...fileItem,
|
||||
startTime: 65,
|
||||
endTime: 125,
|
||||
preview: {
|
||||
proxyType: 'video',
|
||||
thumbnailUrl: 'https://example.com/poster.webp',
|
||||
duration: 300,
|
||||
},
|
||||
} as AssetInfo
|
||||
|
||||
renderComponent({ item: videoWithMatch })
|
||||
|
||||
const matchedDuration = screen.getByTestId('file-preview-matched-duration')
|
||||
expect(matchedDuration).toBeTruthy()
|
||||
expect(matchedDuration.textContent).toBe('01:05 - 02:05')
|
||||
})
|
||||
})
|
||||
|
||||
@@ -48,18 +48,18 @@ import { FIELD_TYPE_ICONS } from '../fields-manager'
|
||||
interface FileCardProps {
|
||||
teamId?: string
|
||||
item: AssetInfo
|
||||
isSelected: boolean
|
||||
isChecked: boolean
|
||||
isEditing: boolean
|
||||
onSelect: (item: AssetInfo, event: React.MouseEvent) => void
|
||||
onDoubleClick: (item: AssetInfo) => void
|
||||
onContextMenu: (e: React.MouseEvent, item: AssetInfo) => void
|
||||
isSelected?: boolean
|
||||
isChecked?: boolean
|
||||
isEditing?: boolean
|
||||
onSelect?: (item: AssetInfo, event: React.MouseEvent) => void
|
||||
onDoubleClick?: (item: AssetInfo) => void
|
||||
onContextMenu?: (e: React.MouseEvent, item: AssetInfo) => void
|
||||
// Deprecated manual drag handlers replaced by dnd-kit
|
||||
onDragStart: (item: AssetInfo) => void
|
||||
onDrop: (target: AssetInfo) => void
|
||||
onRename: (newName: string) => void
|
||||
onFinishEditing: () => void
|
||||
onSaveField: (fieldId: string, value: unknown) => void
|
||||
onDragStart?: (item: AssetInfo) => void
|
||||
onDrop?: (target: AssetInfo) => void
|
||||
onRename?: (newName: string) => void
|
||||
onFinishEditing?: () => void
|
||||
onSaveField?: (fieldId: string, value: unknown) => void
|
||||
dragState?: DragState
|
||||
disabled?: boolean
|
||||
onAction?: (
|
||||
@@ -69,39 +69,43 @@ interface FileCardProps {
|
||||
isRecentlyDeleted?: boolean
|
||||
isRecents?: boolean
|
||||
selectedCount?: number
|
||||
fields: MetadataFieldInfo[]
|
||||
fields?: MetadataFieldInfo[]
|
||||
isShareView?: boolean
|
||||
canEdit?: boolean
|
||||
/** When false, hides the download action. Defaults to true. */
|
||||
allowDownload?: boolean
|
||||
hideCheckbox?: boolean
|
||||
hideActions?: boolean
|
||||
}
|
||||
|
||||
export function FileCard({
|
||||
teamId,
|
||||
item,
|
||||
isSelected,
|
||||
isChecked,
|
||||
isEditing,
|
||||
onSelect,
|
||||
onDoubleClick,
|
||||
onContextMenu,
|
||||
isSelected = false,
|
||||
isChecked = false,
|
||||
isEditing = false,
|
||||
onSelect = () => {},
|
||||
onDoubleClick = () => {},
|
||||
onContextMenu = () => {},
|
||||
// eslint-disable-next-line @typescript-eslint/no-unused-vars
|
||||
onDragStart: _onDragStart,
|
||||
// eslint-disable-next-line @typescript-eslint/no-unused-vars
|
||||
onDrop: _onDrop,
|
||||
onRename,
|
||||
onFinishEditing,
|
||||
onSaveField,
|
||||
onRename = () => {},
|
||||
onFinishEditing = () => {},
|
||||
onSaveField = () => {},
|
||||
dragState,
|
||||
disabled,
|
||||
onAction,
|
||||
isRecentlyDeleted,
|
||||
isRecents,
|
||||
selectedCount,
|
||||
fields,
|
||||
fields = [],
|
||||
isShareView,
|
||||
canEdit = true,
|
||||
allowDownload = true,
|
||||
hideCheckbox = false,
|
||||
hideActions = false,
|
||||
}: FileCardProps) {
|
||||
const [name, setName] = useState(item.name)
|
||||
const inputRef = useRef<HTMLInputElement>(null)
|
||||
@@ -170,10 +174,12 @@ export function FileCard({
|
||||
id: displayItem.id,
|
||||
item: displayItem,
|
||||
},
|
||||
disabled: disabled,
|
||||
})
|
||||
|
||||
// Combine refs
|
||||
const setNodeRef = (node: HTMLElement | null) => {
|
||||
if (disabled) return
|
||||
setDraggableRef(node)
|
||||
setDroppableRef(node)
|
||||
}
|
||||
@@ -332,17 +338,19 @@ export function FileCard({
|
||||
showDropFeedback && 'border-primary outline-1 outline-primary',
|
||||
)}
|
||||
>
|
||||
<div className="absolute left-2 top-2 z-10 flex items-center gap-1.5">
|
||||
<Checkbox
|
||||
checked={isChecked}
|
||||
onCheckedChange={() => {}}
|
||||
onClick={(e) => {
|
||||
e.stopPropagation()
|
||||
onSelect(displayItem, e)
|
||||
}}
|
||||
className="h-4 w-4 bg-white/20 dark:bg-white/20 border-2 data-[state=checked]:bg-primary data-[state=checked]:border-primary border-foreground/15"
|
||||
/>
|
||||
</div>
|
||||
{!hideCheckbox && (
|
||||
<div className="absolute left-2 top-2 z-10 flex items-center gap-1.5">
|
||||
<Checkbox
|
||||
checked={isChecked}
|
||||
onCheckedChange={() => {}}
|
||||
onClick={(e) => {
|
||||
e.stopPropagation()
|
||||
onSelect(displayItem, e)
|
||||
}}
|
||||
className="h-4 w-4 bg-white/20 dark:bg-white/20 border-2 data-[state=checked]:bg-primary data-[state=checked]:border-primary border-foreground/15"
|
||||
/>
|
||||
</div>
|
||||
)}
|
||||
|
||||
<div className="absolute right-2 top-2 z-10 flex items-center gap-1.5">
|
||||
{displayItem.agent && (
|
||||
@@ -395,7 +403,15 @@ export function FileCard({
|
||||
data-testid="file-card-preview-media"
|
||||
className={cn('h-full w-full', isProcessing && 'animate-pulse')}
|
||||
>
|
||||
<FilePreview item={displayItem} showDuration={!isProcessing && !isFailed} />
|
||||
<FilePreview
|
||||
item={displayItem}
|
||||
showDuration={!isProcessing && !isFailed}
|
||||
matchDuration={
|
||||
displayItem.startTime != null && displayItem.endTime != null
|
||||
? { startTime: displayItem.startTime, endTime: displayItem.endTime }
|
||||
: null
|
||||
}
|
||||
/>
|
||||
</div>
|
||||
{previewBadges}
|
||||
</>
|
||||
@@ -435,7 +451,15 @@ export function FileCard({
|
||||
</div>
|
||||
) : (
|
||||
<>
|
||||
<FilePreview item={displayItem} showDuration />
|
||||
<FilePreview
|
||||
item={displayItem}
|
||||
showDuration
|
||||
matchDuration={
|
||||
displayItem.startTime != null && displayItem.endTime != null
|
||||
? { startTime: displayItem.startTime, endTime: displayItem.endTime }
|
||||
: null
|
||||
}
|
||||
/>
|
||||
{previewBadges}
|
||||
</>
|
||||
)}
|
||||
@@ -487,7 +511,7 @@ export function FileCard({
|
||||
</Tooltip>
|
||||
</TooltipProvider>
|
||||
</div>
|
||||
{!isRecents && (
|
||||
{!hideActions && !isRecents && (
|
||||
<DropdownMenu modal={false}>
|
||||
<DropdownMenuTrigger asChild>
|
||||
<Button
|
||||
|
||||
@@ -20,9 +20,13 @@ type FilePreviewItem = {
|
||||
interface FilePreviewProps {
|
||||
item: FilePreviewItem
|
||||
showDuration?: boolean
|
||||
matchDuration?: {
|
||||
startTime?: number | null
|
||||
endTime?: number | null
|
||||
} | null
|
||||
}
|
||||
|
||||
export const FilePreview = ({ item, showDuration = false }: FilePreviewProps) => {
|
||||
export const FilePreview = ({ item, showDuration = false, matchDuration }: FilePreviewProps) => {
|
||||
const proxyType = item.preview?.proxyType || item.proxyType
|
||||
const isVideo = proxyType === 'video'
|
||||
const isAudio = proxyType === 'audio'
|
||||
@@ -32,16 +36,25 @@ export const FilePreview = ({ item, showDuration = false }: FilePreviewProps) =>
|
||||
const duration = item.preview?.duration
|
||||
const pageCount = item.preview?.pageCount
|
||||
|
||||
const overlay =
|
||||
showDuration && hasDuration && typeof duration === 'number' && duration > 0 ? (
|
||||
<span className="pointer-events-none absolute bottom-1 right-1 rounded bg-black/60 px-1 py-0.5 text-xs font-medium tabular-nums text-white">
|
||||
{formatTime(duration)}
|
||||
</span>
|
||||
) : showDuration && isPdf && typeof pageCount === 'number' && pageCount > 0 ? (
|
||||
<span className="pointer-events-none absolute bottom-1 right-1 rounded bg-black/60 px-1 py-0.5 text-xs font-medium tabular-nums text-white">
|
||||
{pageCount} P
|
||||
</span>
|
||||
) : null
|
||||
const hasMatchedChunk =
|
||||
matchDuration != null && matchDuration.startTime != null && matchDuration.endTime != null
|
||||
|
||||
const overlay = hasMatchedChunk ? (
|
||||
<span
|
||||
data-testid="file-preview-matched-duration"
|
||||
className="pointer-events-none absolute bottom-1 right-1 rounded bg-black/60 px-1 py-0.5 text-xs font-medium tabular-nums text-white"
|
||||
>
|
||||
{formatTime(matchDuration.startTime!)} - {formatTime(matchDuration.endTime!)}
|
||||
</span>
|
||||
) : showDuration && hasDuration && typeof duration === 'number' && duration > 0 ? (
|
||||
<span className="pointer-events-none absolute bottom-1 right-1 rounded bg-black/60 px-1 py-0.5 text-xs font-medium tabular-nums text-white">
|
||||
{formatTime(duration)}
|
||||
</span>
|
||||
) : showDuration && isPdf && typeof pageCount === 'number' && pageCount > 0 ? (
|
||||
<span className="pointer-events-none absolute bottom-1 right-1 rounded bg-black/60 px-1 py-0.5 text-xs font-medium tabular-nums text-white">
|
||||
{pageCount} P
|
||||
</span>
|
||||
) : null
|
||||
|
||||
const isMediaWithSprite =
|
||||
(isVideo || isPdf || is3D) &&
|
||||
|
||||
@@ -96,4 +96,19 @@ describe('FolderCard', () => {
|
||||
renderComponent({ item: folderItem, isRecentlyDeleted: true })
|
||||
expect(screen.queryByTestId('folder-card-days-left')).toBeNull()
|
||||
})
|
||||
|
||||
it('does not render selection checkbox when hideCheckbox is true', () => {
|
||||
renderComponent({ item: folderItem, hideCheckbox: true })
|
||||
expect(screen.queryByRole('checkbox')).toBeNull()
|
||||
})
|
||||
|
||||
it('renders selection checkbox by default when hideCheckbox is false or omitted', () => {
|
||||
renderComponent({ item: folderItem })
|
||||
expect(screen.getByRole('checkbox')).toBeTruthy()
|
||||
})
|
||||
|
||||
it('does not render 3-dot dropdown menu when hideActions is true', () => {
|
||||
renderComponent({ item: folderItem, hideActions: true })
|
||||
expect(screen.queryByRole('button')).toBeNull()
|
||||
})
|
||||
})
|
||||
|
||||
@@ -20,17 +20,17 @@ import { getAllFilesFromEntries } from '@/ui/lib/dnd-utils'
|
||||
|
||||
interface FolderCardProps {
|
||||
item: AssetInfo
|
||||
isSelected: boolean
|
||||
isChecked: boolean
|
||||
isEditing: boolean
|
||||
onSelect: (item: AssetInfo, event: React.MouseEvent) => void
|
||||
onDoubleClick: (item: AssetInfo) => void
|
||||
onContextMenu: (e: React.MouseEvent, item: AssetInfo) => void
|
||||
isSelected?: boolean
|
||||
isChecked?: boolean
|
||||
isEditing?: boolean
|
||||
onSelect?: (item: AssetInfo, event: React.MouseEvent) => void
|
||||
onDoubleClick?: (item: AssetInfo) => void
|
||||
onContextMenu?: (e: React.MouseEvent, item: AssetInfo) => void
|
||||
// Deprecated manual drag props replaced by dnd-kit
|
||||
onDragStart: (item: AssetInfo) => void
|
||||
onDrop: (target: AssetInfo) => void
|
||||
onRename: (newName: string) => void
|
||||
onFinishEditing: () => void
|
||||
onDragStart?: (item: AssetInfo) => void
|
||||
onDrop?: (target: AssetInfo) => void
|
||||
onRename?: (newName: string) => void
|
||||
onFinishEditing?: () => void
|
||||
dragState?: DragState
|
||||
disabled?: boolean
|
||||
onAction?: (
|
||||
@@ -49,6 +49,8 @@ interface FolderCardProps {
|
||||
onExternalDrop?: (files: File[], folderId: string) => void
|
||||
/** When false, hides the download action. Defaults to true. */
|
||||
allowDownload?: boolean
|
||||
hideCheckbox?: boolean
|
||||
hideActions?: boolean
|
||||
}
|
||||
|
||||
const FolderPreviewGrid = ({ items }: { items: ChildPreview[] }) => {
|
||||
@@ -106,18 +108,18 @@ const FolderPreviewGrid = ({ items }: { items: ChildPreview[] }) => {
|
||||
|
||||
export function FolderCard({
|
||||
item,
|
||||
isSelected,
|
||||
isChecked,
|
||||
isEditing,
|
||||
onSelect,
|
||||
onDoubleClick,
|
||||
onContextMenu,
|
||||
isSelected = false,
|
||||
isChecked = false,
|
||||
isEditing = false,
|
||||
onSelect = () => {},
|
||||
onDoubleClick = () => {},
|
||||
onContextMenu = () => {},
|
||||
// eslint-disable-next-line @typescript-eslint/no-unused-vars
|
||||
onDragStart: _onDragStart,
|
||||
// eslint-disable-next-line @typescript-eslint/no-unused-vars
|
||||
onDrop: _onDrop,
|
||||
onRename,
|
||||
onFinishEditing,
|
||||
onRename = () => {},
|
||||
onFinishEditing = () => {},
|
||||
dragState,
|
||||
disabled,
|
||||
onAction,
|
||||
@@ -132,6 +134,8 @@ export function FolderCard({
|
||||
resetExternalDragState,
|
||||
onExternalDrop,
|
||||
allowDownload = true,
|
||||
hideCheckbox = false,
|
||||
hideActions = false,
|
||||
}: FolderCardProps) {
|
||||
const [name, setName] = useState(item.name || '')
|
||||
const inputRef = useRef<HTMLInputElement>(null)
|
||||
@@ -154,9 +158,11 @@ export function FolderCard({
|
||||
id: item.id,
|
||||
item: item,
|
||||
},
|
||||
disabled: disabled,
|
||||
})
|
||||
|
||||
const setNodeRef = (node: HTMLElement | null) => {
|
||||
if (disabled) return
|
||||
setDraggableRef(node)
|
||||
setDroppableRef(node)
|
||||
}
|
||||
@@ -333,13 +339,15 @@ export function FolderCard({
|
||||
|
||||
<div className="shadow-[0_-4px_6px_rgba(0,0,0,0.08)] relative w-full flex flex-1 flex-col p-2 mt-5 bg-muted dark:bg-muted/50 rounded-sm">
|
||||
<div className="relative w-full aspect-video shrink-0 overflow-hidden">
|
||||
<div className="absolute top-1 left-1 z-30 transition-all duration-200">
|
||||
<Checkbox
|
||||
checked={isChecked}
|
||||
onClick={(e) => e.stopPropagation()} // Let parent handler deal with it
|
||||
className="w-5 h-5 border-1 border-foreground/20 bg-background/50 data-[state=checked]:bg-primary data-[state=checked]:border-primary"
|
||||
/>
|
||||
</div>
|
||||
{!hideCheckbox && (
|
||||
<div className="absolute top-1 left-1 z-30 transition-all duration-200">
|
||||
<Checkbox
|
||||
checked={isChecked}
|
||||
onClick={(e) => e.stopPropagation()} // Let parent handler deal with it
|
||||
className="w-5 h-5 border-1 border-foreground/20 bg-background/50 data-[state=checked]:bg-primary data-[state=checked]:border-primary"
|
||||
/>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{item.latestChildren && item.latestChildren.length > 0 && (
|
||||
<div data-testid="folder-preview-grid" className="h-full w-full">
|
||||
@@ -376,7 +384,7 @@ export function FolderCard({
|
||||
? m.n_items_singular({ count: item.fileCount || 0 })
|
||||
: m.n_items_plural({ count: item.fileCount || 0 })}
|
||||
</span>
|
||||
{!isRecents && (
|
||||
{!hideActions && !isRecents && (
|
||||
<DropdownMenu modal={false}>
|
||||
<DropdownMenuTrigger asChild>
|
||||
<button
|
||||
|
||||
@@ -0,0 +1,328 @@
|
||||
// @vitest-environment happy-dom
|
||||
import { cleanup, fireEvent, render, screen, waitFor } from '@testing-library/react'
|
||||
import React from 'react'
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import { QueryClient, QueryClientProvider } from '@tanstack/react-query'
|
||||
import type { AssetInfo } from '@shumai/dtos'
|
||||
import { SearchFilterDialog } from './search-filter-dialog'
|
||||
|
||||
vi.mock('@dnd-kit/react', () => ({
|
||||
useDraggable: () => ({
|
||||
ref: vi.fn(),
|
||||
isDragging: false,
|
||||
}),
|
||||
useDroppable: () => ({
|
||||
ref: vi.fn(),
|
||||
isDropTarget: false,
|
||||
}),
|
||||
}))
|
||||
|
||||
const mockSearchPost = vi.fn()
|
||||
const mockSettingsGet = vi.fn().mockResolvedValue({
|
||||
ok: true,
|
||||
json: async () => ({ semanticSearchEnabled: true }),
|
||||
})
|
||||
|
||||
vi.mock('@/ui/api/client', () => ({
|
||||
client: {
|
||||
api: {
|
||||
teams: {
|
||||
':teamId': {
|
||||
settings: {
|
||||
$get: (...args: unknown[]) => mockSettingsGet(...args),
|
||||
},
|
||||
},
|
||||
},
|
||||
folders: {
|
||||
':folderId': {
|
||||
search: {
|
||||
$post: (...args: unknown[]) => mockSearchPost(...args),
|
||||
},
|
||||
},
|
||||
},
|
||||
files: {
|
||||
':fileId': {
|
||||
$get: vi.fn().mockResolvedValue({
|
||||
ok: true,
|
||||
json: async () => ({}),
|
||||
}),
|
||||
},
|
||||
},
|
||||
projects: {
|
||||
':projectId': {
|
||||
collections: {
|
||||
$post: vi.fn().mockResolvedValue({
|
||||
ok: true,
|
||||
json: async () => ({ id: 'col-1' }),
|
||||
}),
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}))
|
||||
|
||||
vi.mock('@tanstack/react-router', () => ({
|
||||
useNavigate: () => vi.fn(),
|
||||
useRouterState: () => '/projects/proj-1',
|
||||
}))
|
||||
|
||||
describe('SearchFilterDialog', () => {
|
||||
let queryClient: QueryClient
|
||||
let windowOpenSpy: ReturnType<typeof vi.spyOn>
|
||||
|
||||
beforeEach(() => {
|
||||
queryClient = new QueryClient({
|
||||
defaultOptions: { queries: { retry: false } },
|
||||
})
|
||||
windowOpenSpy = vi.spyOn(window, 'open').mockImplementation(() => null)
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
cleanup()
|
||||
vi.clearAllMocks()
|
||||
windowOpenSpy.mockRestore()
|
||||
})
|
||||
|
||||
const defaultProps = {
|
||||
open: true,
|
||||
onOpenChange: vi.fn(),
|
||||
teamId: 'team-1',
|
||||
projectId: 'proj-1',
|
||||
assetId: 'folder-root',
|
||||
fields: [],
|
||||
initialConditions: [],
|
||||
onApply: vi.fn(),
|
||||
}
|
||||
|
||||
it('renders initial ready to search state when open and no query', () => {
|
||||
render(
|
||||
<QueryClientProvider client={queryClient}>
|
||||
<SearchFilterDialog {...defaultProps} />
|
||||
</QueryClientProvider>,
|
||||
)
|
||||
|
||||
expect(screen.getByText(/Ready to search|准备搜索/i)).toBeTruthy()
|
||||
})
|
||||
|
||||
it('renders search results in a responsive grid using FileCard and FolderCard without checkboxes or action menus', async () => {
|
||||
const mockFile: AssetInfo = {
|
||||
id: 'file-101',
|
||||
name: 'video_clip.mp4',
|
||||
type: 'file',
|
||||
sizeByte: 1048576,
|
||||
status: 'processed',
|
||||
createdAt: '2026-01-01T00:00:00Z',
|
||||
updatedAt: '2026-01-01T00:00:00Z',
|
||||
preview: {
|
||||
proxyType: 'video',
|
||||
thumbnailUrl: 'https://example.com/clip.jpg',
|
||||
duration: 120,
|
||||
},
|
||||
} as AssetInfo
|
||||
|
||||
const mockFolder: AssetInfo = {
|
||||
id: 'folder-202',
|
||||
name: 'B-Roll Folder',
|
||||
type: 'folder',
|
||||
fileCount: 4,
|
||||
status: 'processed',
|
||||
createdAt: '2026-01-01T00:00:00Z',
|
||||
updatedAt: '2026-01-01T00:00:00Z',
|
||||
} as AssetInfo
|
||||
|
||||
mockSearchPost.mockResolvedValueOnce({
|
||||
ok: true,
|
||||
json: async () => ({
|
||||
data: [mockFile, mockFolder],
|
||||
pageInfo: { total: 2 },
|
||||
}),
|
||||
})
|
||||
|
||||
render(
|
||||
<QueryClientProvider client={queryClient}>
|
||||
<SearchFilterDialog
|
||||
{...defaultProps}
|
||||
initialConditions={[{ field: 'name', operator: 'contains', value: 'video' }]}
|
||||
/>
|
||||
</QueryClientProvider>,
|
||||
)
|
||||
|
||||
await waitFor(() => {
|
||||
expect(screen.getByText('video_clip.mp4')).toBeTruthy()
|
||||
expect(screen.getByText('B-Roll Folder')).toBeTruthy()
|
||||
})
|
||||
|
||||
// Neither card renders checkboxes or 3-dots action menus in search dialog
|
||||
expect(screen.queryByRole('checkbox')).toBeNull()
|
||||
expect(screen.queryByRole('button', { name: /more/i })).toBeNull()
|
||||
|
||||
// Grid container is present
|
||||
const fileCard = screen.getByText('video_clip.mp4').closest('[data-testid="file-card"]')
|
||||
expect(fileCard).toBeTruthy()
|
||||
const gridContainer = fileCard?.parentElement
|
||||
expect(gridContainer?.className).toContain('grid')
|
||||
})
|
||||
|
||||
it('single-clicking a file opens detail page in a new tab seeking to startTime without closing dialog', async () => {
|
||||
const onOpenChange = vi.fn()
|
||||
const mockSemanticFile: AssetInfo = {
|
||||
id: 'file-sem-1',
|
||||
name: 'interview.mp4',
|
||||
type: 'file',
|
||||
status: 'processed',
|
||||
startTime: 45,
|
||||
endTime: 90,
|
||||
createdAt: '2026-01-01T00:00:00Z',
|
||||
updatedAt: '2026-01-01T00:00:00Z',
|
||||
preview: {
|
||||
proxyType: 'video',
|
||||
thumbnailUrl: 'https://example.com/interview.jpg',
|
||||
duration: 300,
|
||||
},
|
||||
} as AssetInfo
|
||||
|
||||
mockSearchPost.mockResolvedValueOnce({
|
||||
ok: true,
|
||||
json: async () => ({
|
||||
data: [mockSemanticFile],
|
||||
pageInfo: { total: 1 },
|
||||
}),
|
||||
})
|
||||
|
||||
render(
|
||||
<QueryClientProvider client={queryClient}>
|
||||
<SearchFilterDialog
|
||||
{...defaultProps}
|
||||
onOpenChange={onOpenChange}
|
||||
initialConditions={[{ field: 'name', operator: 'contains', value: 'interview' }]}
|
||||
/>
|
||||
</QueryClientProvider>,
|
||||
)
|
||||
|
||||
await waitFor(() => {
|
||||
expect(screen.getByText('interview.mp4')).toBeTruthy()
|
||||
})
|
||||
|
||||
// Matched chunk duration replaces total duration in preview
|
||||
const matchedBadge = screen.getByTestId('file-preview-matched-duration')
|
||||
expect(matchedBadge.textContent).toBe('00:45 - 01:30')
|
||||
|
||||
// Single click on file card
|
||||
const card = screen.getByText('interview.mp4').closest('[data-testid="file-card"]')!
|
||||
fireEvent.click(card)
|
||||
|
||||
expect(windowOpenSpy).toHaveBeenCalledWith(
|
||||
'/projects/proj-1/files/file-sem-1?start=45',
|
||||
'_blank',
|
||||
'noopener,noreferrer',
|
||||
)
|
||||
// Dialog should NOT be closed
|
||||
expect(onOpenChange).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('single-clicking a folder opens folder page in a new tab without closing dialog', async () => {
|
||||
const onOpenChange = vi.fn()
|
||||
const mockFolder: AssetInfo = {
|
||||
id: 'folder-sub-1',
|
||||
name: 'Subfolder Alpha',
|
||||
type: 'folder',
|
||||
status: 'processed',
|
||||
createdAt: '2026-01-01T00:00:00Z',
|
||||
updatedAt: '2026-01-01T00:00:00Z',
|
||||
} as AssetInfo
|
||||
|
||||
mockSearchPost.mockResolvedValueOnce({
|
||||
ok: true,
|
||||
json: async () => ({
|
||||
data: [mockFolder],
|
||||
pageInfo: { total: 1 },
|
||||
}),
|
||||
})
|
||||
|
||||
render(
|
||||
<QueryClientProvider client={queryClient}>
|
||||
<SearchFilterDialog
|
||||
{...defaultProps}
|
||||
onOpenChange={onOpenChange}
|
||||
initialConditions={[{ field: 'name', operator: 'contains', value: 'Subfolder' }]}
|
||||
/>
|
||||
</QueryClientProvider>,
|
||||
)
|
||||
|
||||
await waitFor(() => {
|
||||
expect(screen.getByText('Subfolder Alpha')).toBeTruthy()
|
||||
})
|
||||
|
||||
const card = screen.getByText('Subfolder Alpha').closest('[data-testid="file-card"]')!
|
||||
fireEvent.click(card)
|
||||
|
||||
expect(windowOpenSpy).toHaveBeenCalledWith(
|
||||
'/projects/proj-1/folders/folder-sub-1',
|
||||
'_blank',
|
||||
'noopener,noreferrer',
|
||||
)
|
||||
expect(onOpenChange).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('renders multiple distinct chunk matches for the same video file with unique timestamps', async () => {
|
||||
const chunk1: AssetInfo = {
|
||||
id: 'video-same',
|
||||
name: 'drone_shots.mp4',
|
||||
type: 'file',
|
||||
status: 'processed',
|
||||
startTime: 10,
|
||||
endTime: 30,
|
||||
createdAt: '2026-01-01T00:00:00Z',
|
||||
updatedAt: '2026-01-01T00:00:00Z',
|
||||
preview: {
|
||||
proxyType: 'video',
|
||||
thumbnailUrl: 'https://example.com/drone.jpg',
|
||||
duration: 200,
|
||||
},
|
||||
} as AssetInfo
|
||||
|
||||
const chunk2: AssetInfo = {
|
||||
id: 'video-same',
|
||||
name: 'drone_shots.mp4',
|
||||
type: 'file',
|
||||
status: 'processed',
|
||||
startTime: 120,
|
||||
endTime: 150,
|
||||
createdAt: '2026-01-01T00:00:00Z',
|
||||
updatedAt: '2026-01-01T00:00:00Z',
|
||||
preview: {
|
||||
proxyType: 'video',
|
||||
thumbnailUrl: 'https://example.com/drone.jpg',
|
||||
duration: 200,
|
||||
},
|
||||
} as AssetInfo
|
||||
|
||||
mockSearchPost.mockResolvedValueOnce({
|
||||
ok: true,
|
||||
json: async () => ({
|
||||
data: [chunk1, chunk2],
|
||||
pageInfo: { total: 2 },
|
||||
}),
|
||||
})
|
||||
|
||||
render(
|
||||
<QueryClientProvider client={queryClient}>
|
||||
<SearchFilterDialog
|
||||
{...defaultProps}
|
||||
initialConditions={[{ field: 'name', operator: 'contains', value: 'drone' }]}
|
||||
/>
|
||||
</QueryClientProvider>,
|
||||
)
|
||||
|
||||
await waitFor(() => {
|
||||
const titles = screen.getAllByText('drone_shots.mp4')
|
||||
expect(titles).toHaveLength(2)
|
||||
})
|
||||
|
||||
const badges = screen.getAllByTestId('file-preview-matched-duration')
|
||||
expect(badges).toHaveLength(2)
|
||||
expect(badges[0].textContent).toBe('00:10 - 00:30')
|
||||
expect(badges[1].textContent).toBe('02:00 - 02:30')
|
||||
})
|
||||
})
|
||||
@@ -8,23 +8,15 @@ import { ScrollArea } from '@/ui/components/ui/scroll-area'
|
||||
import { Input } from '@/ui/components/ui/input'
|
||||
import { Label } from '@/ui/components/ui/label'
|
||||
import { Switch } from '@/ui/components/ui/switch'
|
||||
import { formatSize } from '@/ui/lib/format'
|
||||
import { cn } from '@/ui/lib/utils'
|
||||
import { useInfiniteQuery, useMutation, useQuery, useQueryClient } from '@tanstack/react-query'
|
||||
import { useNavigate } from '@tanstack/react-router'
|
||||
import {
|
||||
AlertCircle,
|
||||
ChevronDown,
|
||||
ChevronUp,
|
||||
FileIcon,
|
||||
FolderIcon,
|
||||
Loader2,
|
||||
Search,
|
||||
X,
|
||||
} from 'lucide-react'
|
||||
import { AlertCircle, ChevronDown, ChevronUp, Loader2, Search, X } from 'lucide-react'
|
||||
import { useEffect, useMemo, useState } from 'react'
|
||||
import { useInView } from 'react-intersection-observer'
|
||||
import { toast } from 'sonner'
|
||||
import { FileCard } from '@/ui/components/file-browser/file-card'
|
||||
import { FolderCard } from '@/ui/components/file-browser/folder-card'
|
||||
import { FilterPanel } from './filter-panel'
|
||||
import { m } from '@/ui/paraglide/messages.js'
|
||||
|
||||
@@ -225,36 +217,25 @@ export function SearchFilterDialog({
|
||||
setConditions([])
|
||||
}
|
||||
|
||||
const formatTimestamp = (seconds: number) => {
|
||||
const m = Math.floor(seconds / 60)
|
||||
const s = Math.floor(seconds % 60)
|
||||
return `${m.toString().padStart(2, '0')}:${s.toString().padStart(2, '0')}`
|
||||
}
|
||||
|
||||
const handleResultClick = (item: AssetInfo) => {
|
||||
if (item.type === 'folder') {
|
||||
navigate({
|
||||
to: '/projects/$projectId/folders/$folderId',
|
||||
params: { projectId, folderId: item.id! },
|
||||
})
|
||||
window.open(`/projects/${projectId}/folders/${item.id}`, '_blank', 'noopener,noreferrer')
|
||||
} else {
|
||||
navigate({
|
||||
to: '/projects/$projectId/files/$fileId',
|
||||
params: { projectId, fileId: item.versionStack ? item.versionStack.id : item.id! },
|
||||
search: {
|
||||
version: undefined,
|
||||
start: item.startTime && item.startTime > 0 ? item.startTime : undefined,
|
||||
},
|
||||
})
|
||||
const fileId = item.versionStack ? item.versionStack.id : item.id!
|
||||
const searchParams = new URLSearchParams()
|
||||
if (item.startTime !== undefined && item.startTime !== null && item.startTime > 0) {
|
||||
searchParams.set('start', String(item.startTime))
|
||||
}
|
||||
const query = searchParams.toString() ? `?${searchParams.toString()}` : ''
|
||||
window.open(`/projects/${projectId}/files/${fileId}${query}`, '_blank', 'noopener,noreferrer')
|
||||
}
|
||||
onOpenChange(false)
|
||||
}
|
||||
|
||||
const activeFiltersCount = conditions.filter((c) => c.value !== '').length
|
||||
|
||||
return (
|
||||
<Dialog open={open} onOpenChange={onOpenChange}>
|
||||
<DialogContent className="sm:max-w-[900px] p-0 gap-0 overflow-hidden bg-background border-border rounded-2xl shadow-2xl flex flex-col h-[85vh]">
|
||||
<DialogContent className="sm:max-w-4xl lg:max-w-5xl p-0 gap-0 overflow-hidden bg-background border-border rounded-2xl shadow-2xl flex flex-col h-[85vh]">
|
||||
<DialogHeader className="px-6 pt-3 pb-0 text-left">
|
||||
<DialogTitle className="text-lg font-bold">{m.search_dialog_title()}</DialogTitle>
|
||||
</DialogHeader>
|
||||
@@ -394,121 +375,94 @@ export function SearchFilterDialog({
|
||||
)}
|
||||
</div>
|
||||
|
||||
<div className="flex flex-col border border-border rounded-xl divide-y divide-border overflow-hidden min-w-0">
|
||||
{!hasActiveCriteria ? (
|
||||
<div className="py-20 text-center flex flex-col items-center justify-center bg-muted/20 px-4">
|
||||
<Search className="w-10 h-10 text-muted-foreground/30 mb-2" />
|
||||
<h3 className="text-sm font-semibold text-foreground/80">
|
||||
{m.ready_to_search()}
|
||||
</h3>
|
||||
<p className="text-xs text-muted-foreground max-w-sm mt-1 mx-auto">
|
||||
{m.type_query_instruction()}
|
||||
{!hasActiveCriteria ? (
|
||||
<div className="border border-border rounded-xl py-20 text-center flex flex-col items-center justify-center bg-muted/20 px-4">
|
||||
<Search className="w-10 h-10 text-muted-foreground/30 mb-2" />
|
||||
<h3 className="text-sm font-semibold text-foreground/80">{m.ready_to_search()}</h3>
|
||||
<p className="text-xs text-muted-foreground max-w-sm mt-1 mx-auto">
|
||||
{m.type_query_instruction()}
|
||||
</p>
|
||||
{isSemantic && (
|
||||
<p className="text-xs text-muted-foreground/80 max-w-md mt-4 mx-auto border-t border-border/60 pt-3">
|
||||
{m.semantic_search_note()}
|
||||
</p>
|
||||
{isSemantic && (
|
||||
<p className="text-xs text-muted-foreground/80 max-w-md mt-4 mx-auto border-t border-border/60 pt-3">
|
||||
{m.semantic_search_note()}
|
||||
</p>
|
||||
)}
|
||||
</div>
|
||||
) : searchError ? (
|
||||
<div className="py-12 px-4 text-center flex flex-col items-center justify-center bg-muted/20">
|
||||
<AlertCircle className="w-10 h-10 text-destructive/50 mb-2" />
|
||||
<h3 className="text-sm font-semibold text-foreground/80">{m.search_failed()}</h3>
|
||||
<p className="text-xs text-muted-foreground max-w-sm mt-1 mx-auto">
|
||||
{searchError.message.includes('Embedding agent not configured')
|
||||
? m.enable_embedding_agent_hint()
|
||||
: searchError.message}
|
||||
</p>
|
||||
</div>
|
||||
) : isLoading ? (
|
||||
Array.from({ length: 3 }).map((_, idx) => (
|
||||
)}
|
||||
</div>
|
||||
) : searchError ? (
|
||||
<div className="border border-border rounded-xl py-12 px-4 text-center flex flex-col items-center justify-center bg-muted/20">
|
||||
<AlertCircle className="w-10 h-10 text-destructive/50 mb-2" />
|
||||
<h3 className="text-sm font-semibold text-foreground/80">{m.search_failed()}</h3>
|
||||
<p className="text-xs text-muted-foreground max-w-sm mt-1 mx-auto">
|
||||
{searchError.message.includes('Embedding agent not configured')
|
||||
? m.enable_embedding_agent_hint()
|
||||
: searchError.message}
|
||||
</p>
|
||||
</div>
|
||||
) : isLoading ? (
|
||||
<div className="grid grid-cols-2 sm:grid-cols-3 lg:grid-cols-4 gap-4">
|
||||
{Array.from({ length: 8 }).map((_, idx) => (
|
||||
<div
|
||||
key={idx}
|
||||
className="p-3.5 flex items-center gap-3 animate-pulse bg-muted/10"
|
||||
className="flex flex-col rounded-xl border border-border bg-card p-2 space-y-2 animate-pulse"
|
||||
>
|
||||
<div className="w-11 h-11 rounded-xl bg-muted shrink-0" />
|
||||
<div className="flex-1 space-y-2">
|
||||
<div className="h-4 bg-muted rounded w-1/3" />
|
||||
<div className="h-3 bg-muted/50 rounded w-1/4" />
|
||||
<div className="aspect-square w-full rounded-lg bg-muted/60" />
|
||||
<div className="p-1 space-y-1.5">
|
||||
<div className="h-4 bg-muted/60 rounded w-3/4" />
|
||||
<div className="h-3 bg-muted/40 rounded w-1/2" />
|
||||
</div>
|
||||
</div>
|
||||
))
|
||||
) : flattenedResults.length === 0 ? (
|
||||
<div className="py-12 px-4 text-center flex flex-col items-center justify-center bg-muted/20">
|
||||
<AlertCircle className="w-10 h-10 text-muted-foreground/30 mb-2" />
|
||||
<h3 className="text-sm font-semibold text-foreground/80">
|
||||
{m.no_records_matched()}
|
||||
</h3>
|
||||
<p className="text-xs text-muted-foreground max-w-sm mt-1 mx-auto">
|
||||
{m.try_adjusting_spelling()}
|
||||
</p>
|
||||
))}
|
||||
</div>
|
||||
) : flattenedResults.length === 0 ? (
|
||||
<div className="border border-border rounded-xl py-12 px-4 text-center flex flex-col items-center justify-center bg-muted/20">
|
||||
<AlertCircle className="w-10 h-10 text-muted-foreground/30 mb-2" />
|
||||
<h3 className="text-sm font-semibold text-foreground/80">
|
||||
{m.no_records_matched()}
|
||||
</h3>
|
||||
<p className="text-xs text-muted-foreground max-w-sm mt-1 mx-auto">
|
||||
{m.try_adjusting_spelling()}
|
||||
</p>
|
||||
</div>
|
||||
) : (
|
||||
<>
|
||||
<div className="grid grid-cols-2 sm:grid-cols-3 lg:grid-cols-4 gap-4">
|
||||
{flattenedResults.map((record) => {
|
||||
const itemKey =
|
||||
record.startTime != null && record.endTime != null
|
||||
? `${record.id}-${record.startTime}-${record.endTime}`
|
||||
: record.id
|
||||
if (record.type === 'folder') {
|
||||
return (
|
||||
<FolderCard
|
||||
key={itemKey}
|
||||
item={record}
|
||||
hideCheckbox
|
||||
hideActions
|
||||
disabled
|
||||
onSelect={() => handleResultClick(record)}
|
||||
/>
|
||||
)
|
||||
}
|
||||
return (
|
||||
<FileCard
|
||||
key={itemKey}
|
||||
teamId={teamId}
|
||||
item={record}
|
||||
hideCheckbox
|
||||
hideActions
|
||||
disabled
|
||||
onSelect={() => handleResultClick(record)}
|
||||
/>
|
||||
)
|
||||
})}
|
||||
</div>
|
||||
) : (
|
||||
<>
|
||||
{flattenedResults.map((record) => (
|
||||
<div
|
||||
key={record.id}
|
||||
onClick={() => handleResultClick(record)}
|
||||
className="p-3.5 flex items-center gap-3 hover:bg-muted/40 transition-colors group/record cursor-pointer min-w-0"
|
||||
>
|
||||
<div
|
||||
className={cn(
|
||||
'shrink-0 w-11 h-11 rounded-xl shadow-xs flex items-center justify-center text-white font-bold relative group-hover/record:scale-105 transition-transform bg-muted overflow-hidden',
|
||||
record.type === 'folder' ? 'bg-primary/5' : '',
|
||||
)}
|
||||
>
|
||||
{record.preview?.thumbnailUrl ? (
|
||||
<img
|
||||
src={record.preview.thumbnailUrl}
|
||||
alt=""
|
||||
className="w-full h-full object-cover rounded-xl"
|
||||
/>
|
||||
) : record.type === 'folder' ? (
|
||||
<FolderIcon className="h-6 w-6 text-primary/70" />
|
||||
) : (
|
||||
<FileIcon className="h-6 w-6 text-muted-foreground" />
|
||||
)}
|
||||
</div>
|
||||
|
||||
<div className="flex-1 min-w-0">
|
||||
<div className="flex items-center justify-between gap-3 min-w-0">
|
||||
<div className="flex-1 min-w-0">
|
||||
<span className="font-medium text-sm text-foreground group-hover/record:text-primary transition-colors block truncate">
|
||||
{record.name}
|
||||
</span>
|
||||
<span className="text-xs text-muted-foreground block mt-0.5 truncate">
|
||||
{record.type === 'folder'
|
||||
? m.folder()
|
||||
: `${m.size_prefix()} ${formatSize(record.sizeByte || 0)}`}
|
||||
</span>
|
||||
{record.startTime !== undefined &&
|
||||
record.startTime !== null &&
|
||||
record.endTime !== undefined &&
|
||||
record.endTime !== null && (
|
||||
<span className="text-[10px] font-semibold text-primary bg-primary/10 border border-primary/20 px-1.5 py-0.5 rounded mt-1 inline-block max-w-full truncate">
|
||||
{m.match_found_at()} {formatTimestamp(record.startTime)} -{' '}
|
||||
{formatTimestamp(record.endTime)}
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
<span className="text-xs text-muted-foreground font-mono shrink-0">
|
||||
{new Date(record.createdAt).toLocaleDateString(undefined, {
|
||||
year: 'numeric',
|
||||
month: 'short',
|
||||
})}
|
||||
</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
))}
|
||||
{(hasNextPage || isFetchingNextPage) && (
|
||||
<div ref={ref} className="p-4 flex justify-center items-center">
|
||||
<Loader2 className="h-6 w-6 animate-spin text-muted-foreground" />
|
||||
</div>
|
||||
)}
|
||||
</>
|
||||
)}
|
||||
</div>
|
||||
{(hasNextPage || isFetchingNextPage) && (
|
||||
<div ref={ref} className="p-4 flex justify-center items-center">
|
||||
<Loader2 className="h-6 w-6 animate-spin text-muted-foreground" />
|
||||
</div>
|
||||
)}
|
||||
</>
|
||||
)}
|
||||
</div>
|
||||
</ScrollArea>
|
||||
|
||||
|
||||
@@ -92,4 +92,39 @@ describe('AgentFormDialog', () => {
|
||||
expect(screen.getByText('Move Assets')).toBeDefined()
|
||||
expect(screen.getByText('Delete Asset')).toBeDefined()
|
||||
})
|
||||
|
||||
it('renders read-only state with system managed hint and only Close button', () => {
|
||||
render(
|
||||
<QueryClientProvider client={queryClient}>
|
||||
<AgentFormDialog
|
||||
isOpen={true}
|
||||
readOnly={true}
|
||||
onClose={vi.fn()}
|
||||
teamId="team-123"
|
||||
type="embedding"
|
||||
title="Edit Embedding Agent"
|
||||
initialValues={{
|
||||
id: 'agent-1',
|
||||
name: 'embedding-agent',
|
||||
type: 'embedding',
|
||||
enabled: true,
|
||||
permission: 'reviewer',
|
||||
thinkingLevel: 'off',
|
||||
skills: [],
|
||||
mcpServerIds: [],
|
||||
deniedTools: [],
|
||||
readOnly: true,
|
||||
}}
|
||||
/>
|
||||
</QueryClientProvider>,
|
||||
)
|
||||
|
||||
expect(
|
||||
screen.getByText(
|
||||
'This agent is managed by the server environment configuration and cannot be modified or deleted.',
|
||||
),
|
||||
).toBeDefined()
|
||||
expect(screen.getAllByRole('button', { name: 'Close' }).length).toBe(2)
|
||||
expect(screen.queryByRole('button', { name: 'Save Changes' })).toBeNull()
|
||||
})
|
||||
})
|
||||
|
||||
@@ -48,6 +48,7 @@ interface AgentFormDialogProps {
|
||||
type?: AgentType
|
||||
initialValues?: AgentInfo
|
||||
title: string
|
||||
readOnly?: boolean
|
||||
}
|
||||
|
||||
const agentFormSchema = z.object({
|
||||
@@ -137,6 +138,7 @@ export function AgentFormDialog({
|
||||
type,
|
||||
initialValues,
|
||||
title,
|
||||
readOnly = false,
|
||||
}: AgentFormDialogProps) {
|
||||
const queryClient = useQueryClient()
|
||||
const { canAdmin } = usePermissions()
|
||||
@@ -368,6 +370,11 @@ export function AgentFormDialog({
|
||||
>
|
||||
<ScrollArea className="flex-1 min-h-0">
|
||||
<div className="p-6 pt-2 space-y-6">
|
||||
{readOnly && (
|
||||
<div className="p-3 bg-muted text-muted-foreground text-sm rounded-lg border border-border">
|
||||
{m.agent_managed_by_system_hint()}
|
||||
</div>
|
||||
)}
|
||||
<div className="grid grid-cols-1 md:grid-cols-2 gap-6">
|
||||
<form.Field
|
||||
name="name"
|
||||
@@ -382,6 +389,7 @@ export function AgentFormDialog({
|
||||
onChange={(e) => field.handleChange(e.target.value)}
|
||||
placeholder={m.agent_name_placeholder()}
|
||||
aria-invalid={isInvalid}
|
||||
disabled={readOnly}
|
||||
/>
|
||||
{isInvalid && <FieldError errors={mapErrors(field.state.meta.errors)} />}
|
||||
</Field>
|
||||
@@ -423,7 +431,7 @@ export function AgentFormDialog({
|
||||
onValueChange={(val) =>
|
||||
field.handleChange(val as 'owner' | 'editor' | 'reviewer')
|
||||
}
|
||||
disabled={!canAdmin}
|
||||
disabled={readOnly || !canAdmin}
|
||||
>
|
||||
<SelectTrigger>
|
||||
<SelectValue />
|
||||
@@ -453,12 +461,15 @@ export function AgentFormDialog({
|
||||
<button
|
||||
key={preset.id}
|
||||
type="button"
|
||||
onClick={() => field.handleChange(preset.id)}
|
||||
disabled={readOnly}
|
||||
onClick={() => !readOnly && field.handleChange(preset.id)}
|
||||
className={cn(
|
||||
'relative w-14 h-14 rounded-full overflow-hidden border-2 transition-all hover:scale-105',
|
||||
'relative w-14 h-14 rounded-full overflow-hidden border-2 transition-all',
|
||||
!readOnly && 'hover:scale-105',
|
||||
isSelected
|
||||
? 'border-primary ring-2 ring-primary ring-offset-2'
|
||||
: 'border-border hover:border-muted-foreground',
|
||||
readOnly && 'cursor-default opacity-80',
|
||||
)}
|
||||
>
|
||||
<img
|
||||
@@ -473,7 +484,8 @@ export function AgentFormDialog({
|
||||
!AVAILABLE_AVATARS.some((p) => p.id === field.state.value) && (
|
||||
<button
|
||||
type="button"
|
||||
className="relative w-14 h-14 rounded-full overflow-hidden border-2 border-primary ring-2 ring-primary ring-offset-2 scale-105"
|
||||
disabled={true}
|
||||
className="relative w-14 h-14 rounded-full overflow-hidden border-2 border-primary ring-2 ring-primary ring-offset-2 scale-105 cursor-default"
|
||||
>
|
||||
<img
|
||||
src={field.state.value}
|
||||
@@ -1024,29 +1036,37 @@ export function AgentFormDialog({
|
||||
</ScrollArea>
|
||||
|
||||
<DialogFooter className="p-6 pt-4 border-t border-border flex-shrink-0">
|
||||
<Button variant="outline" onClick={onClose} type="button">
|
||||
{m.cancel()}
|
||||
</Button>
|
||||
<form.Subscribe
|
||||
selector={(state) => [state.canSubmit, state.isSubmitting]}
|
||||
children={([canSubmit, isSubmitting]) => (
|
||||
<Button
|
||||
type="submit"
|
||||
className="bg-primary hover:bg-primary/90 text-primary-foreground min-w-[120px]"
|
||||
disabled={
|
||||
!canSubmit ||
|
||||
isSubmitting ||
|
||||
createMutation.isPending ||
|
||||
updateMutation.isPending
|
||||
}
|
||||
>
|
||||
{(isSubmitting || createMutation.isPending || updateMutation.isPending) && (
|
||||
<Loader2 className="w-4 h-4 animate-spin mr-2" />
|
||||
)}
|
||||
{initialValues ? m.save_changes() : m.create_agent()}
|
||||
{readOnly ? (
|
||||
<Button variant="outline" onClick={onClose} type="button">
|
||||
{m.close()}
|
||||
</Button>
|
||||
) : (
|
||||
<>
|
||||
<Button variant="outline" onClick={onClose} type="button">
|
||||
{m.cancel()}
|
||||
</Button>
|
||||
)}
|
||||
/>
|
||||
<form.Subscribe
|
||||
selector={(state) => [state.canSubmit, state.isSubmitting]}
|
||||
children={([canSubmit, isSubmitting]) => (
|
||||
<Button
|
||||
type="submit"
|
||||
className="bg-primary hover:bg-primary/90 text-primary-foreground min-w-[120px]"
|
||||
disabled={
|
||||
!canSubmit ||
|
||||
isSubmitting ||
|
||||
createMutation.isPending ||
|
||||
updateMutation.isPending
|
||||
}
|
||||
>
|
||||
{(isSubmitting || createMutation.isPending || updateMutation.isPending) && (
|
||||
<Loader2 className="w-4 h-4 animate-spin mr-2" />
|
||||
)}
|
||||
{initialValues ? m.save_changes() : m.create_agent()}
|
||||
</Button>
|
||||
)}
|
||||
/>
|
||||
</>
|
||||
)}
|
||||
</DialogFooter>
|
||||
</form>
|
||||
</DialogContent>
|
||||
|
||||
@@ -189,6 +189,7 @@ export function AgentsSettings({ teamId }: AgentsSettingsProps) {
|
||||
{(isCreateDialogOpen || editingAgent) && (
|
||||
<AgentFormDialog
|
||||
isOpen={true}
|
||||
readOnly={editingAgent?.readOnly}
|
||||
title={
|
||||
editingAgent
|
||||
? m.edit_item({ name: editingAgent.name })
|
||||
@@ -340,29 +341,38 @@ export function AgentsSettings({ teamId }: AgentsSettingsProps) {
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<DropdownMenu modal={false}>
|
||||
<DropdownMenuTrigger asChild onClick={(e) => e.stopPropagation()}>
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="icon"
|
||||
className="h-8 w-8 text-muted-foreground hover:text-foreground"
|
||||
>
|
||||
<MoreVertical className="w-4 h-4" />
|
||||
</Button>
|
||||
</DropdownMenuTrigger>
|
||||
<DropdownMenuContent align="end" className="w-32">
|
||||
<DropdownMenuItem
|
||||
onClick={(e) => {
|
||||
e.stopPropagation()
|
||||
setAgentToDelete(agent)
|
||||
setIsDeleteDialogOpen(true)
|
||||
}}
|
||||
className="text-red-500 focus:text-red-500 gap-2 cursor-pointer"
|
||||
>
|
||||
<Trash2 className="w-4 h-4" /> {m.delete()}
|
||||
</DropdownMenuItem>
|
||||
</DropdownMenuContent>
|
||||
</DropdownMenu>
|
||||
{agent.readOnly ? (
|
||||
<Badge
|
||||
variant="outline"
|
||||
className="text-xs text-muted-foreground border-border"
|
||||
>
|
||||
{m.agent_readonly()}
|
||||
</Badge>
|
||||
) : (
|
||||
<DropdownMenu modal={false}>
|
||||
<DropdownMenuTrigger asChild onClick={(e) => e.stopPropagation()}>
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="icon"
|
||||
className="h-8 w-8 text-muted-foreground hover:text-foreground"
|
||||
>
|
||||
<MoreVertical className="w-4 h-4" />
|
||||
</Button>
|
||||
</DropdownMenuTrigger>
|
||||
<DropdownMenuContent align="end" className="w-32">
|
||||
<DropdownMenuItem
|
||||
onClick={(e) => {
|
||||
e.stopPropagation()
|
||||
setAgentToDelete(agent)
|
||||
setIsDeleteDialogOpen(true)
|
||||
}}
|
||||
className="text-red-500 focus:text-red-500 gap-2 cursor-pointer"
|
||||
>
|
||||
<Trash2 className="w-4 h-4" /> {m.delete()}
|
||||
</DropdownMenuItem>
|
||||
</DropdownMenuContent>
|
||||
</DropdownMenu>
|
||||
)}
|
||||
</div>
|
||||
|
||||
<div className="flex items-center justify-between gap-2 pt-2 border-t border-border/50">
|
||||
@@ -408,6 +418,7 @@ export function AgentsSettings({ teamId }: AgentsSettingsProps) {
|
||||
<div onClick={(e) => e.stopPropagation()}>
|
||||
<Switch
|
||||
checked={agent.enabled}
|
||||
disabled={agent.readOnly}
|
||||
onCheckedChange={(enabled) => {
|
||||
updateAgentMutation.mutate({
|
||||
teamId,
|
||||
|
||||
@@ -155,4 +155,19 @@ describe('SpriteScrubber', () => {
|
||||
fireEvent.mouseMove(scrubberVideo, { clientX: 25 })
|
||||
expect(scrubberVideo.getAttribute('data-frame')).toBe('25')
|
||||
})
|
||||
|
||||
it('renders with cursor-pointer instead of cursor-crosshair for cards', () => {
|
||||
const { getByTestId } = render(
|
||||
<SpriteScrubber
|
||||
spriteUrl="https://example.com/sprite.webp"
|
||||
thumbnailUrl="https://example.com/thumb.webp"
|
||||
videoWidth={1920}
|
||||
videoHeight={1080}
|
||||
/>,
|
||||
)
|
||||
|
||||
const scrubber = getByTestId('sprite-scrubber')
|
||||
expect(scrubber.className).toContain('cursor-pointer')
|
||||
expect(scrubber.className).not.toContain('cursor-crosshair')
|
||||
})
|
||||
})
|
||||
|
||||
@@ -165,7 +165,7 @@ export const SpriteScrubber: React.FC<SpriteScrubberProps> = ({
|
||||
ref={containerRef}
|
||||
data-testid="sprite-scrubber"
|
||||
data-frame={currentFrame}
|
||||
className={`relative w-full h-full overflow-hidden cursor-crosshair group bg-black select-none ${className}`}
|
||||
className={`relative w-full h-full overflow-hidden cursor-pointer group bg-black select-none ${className}`}
|
||||
onMouseEnter={handleMouseEnter}
|
||||
onMouseLeave={handleMouseLeave}
|
||||
onMouseMove={handleMouseMove}
|
||||
|
||||
@@ -633,6 +633,8 @@
|
||||
"chat_agent": "Chat Agent",
|
||||
"chat_agents": "Chat Agents",
|
||||
"embedding_agent": "Embedding Agent",
|
||||
"agent_readonly": "Read-only",
|
||||
"agent_managed_by_system_hint": "This agent is managed by the server environment configuration and cannot be modified or deleted.",
|
||||
"skills_count_value": "{count} Skills",
|
||||
"autofill_agent": "Autofill Agent",
|
||||
"ai_provider": "AI Provider",
|
||||
|
||||
@@ -633,6 +633,8 @@
|
||||
"chat_agent": "对话智能体",
|
||||
"chat_agents": "对话智能体",
|
||||
"embedding_agent": "嵌入智能体",
|
||||
"agent_readonly": "只读",
|
||||
"agent_managed_by_system_hint": "此智能体由服务器环境配置管理,无法进行修改或删除。",
|
||||
"skills_count_value": "{count} 个技能",
|
||||
"autofill_agent": "自动填充智能体",
|
||||
"ai_provider": "AI 提供商",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@shumai/webui",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "./index.html",
|
||||
|
||||
@@ -1 +1 @@
|
||||
export const APP_VERSION = '0.5.2'
|
||||
export const APP_VERSION = '0.5.3'
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@shumai/workflow-core",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "./src/index.ts",
|
||||
|
||||
@@ -8,7 +8,6 @@ import { WorkflowTaskStatus, WorkflowTaskType } from '@shumai/db'
|
||||
// Mock the individual workflows
|
||||
const mocks = {
|
||||
agentEmbeddingMedia: vi.fn(),
|
||||
queryEmbeddingForSearch: vi.fn(),
|
||||
agentAutofillMedia: vi.fn(),
|
||||
agentChat: vi.fn(),
|
||||
transcodeMedia: vi.fn(),
|
||||
@@ -26,9 +25,6 @@ describe('LocalExecutor Integration Tests', () => {
|
||||
|
||||
// Register workflows dynamically for testing
|
||||
registerWorkflow(WorkflowTaskType.ai_embedding, async (task) => mocks.agentEmbeddingMedia(task))
|
||||
registerWorkflow(WorkflowTaskType.query_embedding_for_search, async (task) =>
|
||||
mocks.queryEmbeddingForSearch(task),
|
||||
)
|
||||
registerWorkflow(WorkflowTaskType.ai_metadata_autofill, async (task) =>
|
||||
mocks.agentAutofillMedia(task),
|
||||
)
|
||||
|
||||
@@ -22,10 +22,6 @@ export class TemporalExecutor implements Executor {
|
||||
workflowName = 'agentEmbeddingWorkflow'
|
||||
taskQueue = TaskQueueAgent
|
||||
break
|
||||
case WorkflowTaskType.query_embedding_for_search:
|
||||
workflowName = 'queryEmbeddingForSearch'
|
||||
taskQueue = TaskQueueAgent
|
||||
break
|
||||
case WorkflowTaskType.ai_metadata_autofill:
|
||||
workflowName = 'agentAutofillWorkflow'
|
||||
taskQueue = TaskQueueAgent
|
||||
|
||||
Reference in New Issue
Block a user