docs(agents): add image and video generation settings documentation (#389)

This commit is contained in:
Yiling-J
2026-09-06 21:21:16 +08:00
committed by GitHub
parent 659a0af2c5
commit bc85dc4541
3 changed files with 158 additions and 4 deletions
+1
View File
@@ -42,6 +42,7 @@
"group": "Shumai Agent Settings",
"pages": [
"using-shumai/agent-providers",
"using-shumai/image-video-generation",
"using-shumai/agents",
"using-shumai/agents-md",
"using-shumai/agent-skills",
+16 -4
View File
@@ -55,22 +55,34 @@ The following tools can be individually toggled in the agent configuration dialo
- **Description:** Allows the agent to upload and stack a new version of an existing file in Shumai.
- **Why it matters:** Allows the agent to update files in a clean, non-destructive way by stacking revisions rather than creating duplicate files.
### 8. Download Asset (`download_asset`)
### 8. Generate Image (`generate_image`)
- **Description:** Allows the agent to generate still images from text prompts and reference images using configured AI image models.
- **Why it matters:** Enables agents to create concept art, mockups, textures, and illustrations directly inside Shumai.
- **Prerequisite:** This tool requires at least one image model to be configured and enabled in [Image / Video Generation](/using-shumai/image-video-generation). If no image models are enabled under a configured provider, this tool is omitted from the agent session even if enabled in the agent settings dialog.
### 9. Generate Video (`generate_video`)
- **Description:** Allows the agent to generate video clips from text prompts, image animation, or transition keyframes using configured AI video models.
- **Why it matters:** Enables agents to produce short animated clips, camera motion shots, and video transitions.
- **Prerequisite:** This tool requires at least one video model to be configured and enabled in [Image / Video Generation](/using-shumai/image-video-generation). If no video models are enabled under a configured provider, this tool is omitted from the agent session even if enabled in the agent settings dialog.
### 10. Download Asset (`download_asset`)
- **Description:** Allows the agent to download a workspace asset or specific storage S3 key to the local `.pi` directory for processing with local scripts or bash commands.
- **Why it matters:** Enables agents executing custom skill scripts to fetch files or intermediate transcode artifacts to local disk.
### 9. Rename Asset (`rename_asset`)
### 11. Rename Asset (`rename_asset`)
- **Description:** Allows the agent to rename an existing file or folder in Shumai.
- **Why it matters:** Enables agents to keep project files neatly named and organized according to workspace naming conventions. Enabled by default.
### 10. Move Assets (`move_assets`)
### 12. Move Assets (`move_assets`)
- **Description:** Allows the agent to move one or more assets to a destination parent folder.
- **Why it matters:** Enables agents to reorganize workspace structures, relocate generated outputs, or tidy up project folders. Enabled by default.
### 11. Delete Asset (`delete_asset`)
### 13. Delete Asset (`delete_asset`)
- **Description:** Allows the agent to delete a single file or folder by moving it to the Trash bin.
- **Why it matters:** Provides agents with the ability to clean up outdated or redundant assets. To prevent accidental data loss, this tool is **disabled by default** and strictly limited to deleting one asset at a time. Team Owners must explicitly enable this tool for an agent.
@@ -0,0 +1,141 @@
---
title: Image / Video Generation
description: Configure image and video generation providers and models for built-in AI agent tools.
---
Shumai agents can generate creative visual assets directly within your workspace using built-in agent tools: **Generate Image** (`generate_image`) and **Generate Video** (`generate_video`).
Because generating visual media requires specialized foundation models, you must configure upstream media providers and enable specific models in your team settings before agents can use these capabilities.
> 🔒 **Access Control:** Only **Team Owners** can view and configure media generation settings. To access this page, click your avatar, navigate to **Settings**, and select **Image / Video Generation** from the left sidebar (or navigate to `teams/<teamId>/settings#image-video`).
---
## Why This Setting Is Needed
Conversational AI models (configured under **AI Providers**) handle text reasoning and chat interactions, but image and video generation require dedicated multimodal generative models (such as DALL-E, FLUX, Gemini Image, Veo, Kling AI, Luma Ray, Wan, and others).
The **Image / Video Generation** settings page allows team owners to:
1. Authenticate with one or more supported image and video generation providers.
2. Select which curated models to enable for your team, or register custom model IDs.
3. Establish default models for agents when generating visual assets.
---
## Tool Availability & Activation Rules
<Warning>
**Important:** Enabling `generate_image` or `generate_video` in an agent's configuration dialog is
**not enough** on its own. The tool will only be available to the agent if all three of the
following conditions are met:
</Warning>
1. **Provider Configured:** The provider must have a valid API key (either set via server environment variables or configured with a custom team key).
2. **At Least One Model Enabled:** There must be **at least one enabled model** for that modality (`image` or `video`) under an active provider.
3. **Tool Permitted for Agent:** The tool must not be disabled in the agent's tool permissions list under **Team Settings** > **Agents** > **Tools**.
### What Happens If No Models Are Configured?
If no image models are enabled under a configured provider, the `generate_image` tool is **completely omitted** from the agent's runtime environment. Similarly, if no video models are enabled, the `generate_video` tool is omitted.
In this state:
- The agent **cannot call** the generation tool.
- The tool will not appear in the agent's tool catalog or prompt definition.
- Even if a Team Owner has toggled **Generate Image** or **Generate Video** ON in the agent's settings dialog, the agent will not have access to the tool until a model is enabled.
---
## Supported Providers
Shumai provides built-in support for 18 media generation providers:
| Provider | Supported Modalities | Default Environment Variable |
| :-------------------- | :------------------- | :--------------------------- |
| **OpenAI** | Image | `OPENAI_API_KEY` |
| **Google** | Image, Video | `GEMINI_API_KEY` |
| **Google Vertex AI** | Image, Video | `GOOGLE_CLOUD_API_KEY` |
| **xAI** | Image, Video | `XAI_API_KEY` |
| **Fal** | Image, Video | `FAL_KEY` |
| **Replicate** | Image, Video | `REPLICATE_API_TOKEN` |
| **Black Forest Labs** | Image, Video | `BFL_API_KEY` |
| **Kling AI** | Video | `KLINGAI_API_KEY` |
| **Together AI** | Image | `TOGETHER_API_KEY` |
| **Fireworks AI** | Image | `FIREWORKS_API_KEY` |
| **DeepInfra** | Image | `DEEPINFRA_API_KEY` |
| **Luma** | Image | `LUMA_API_KEY` |
| **Amazon Bedrock** | Image | `AWS_ACCESS_KEY_ID` |
| **Alibaba** | Video | `ALIBABA_API_KEY` |
| **ByteDance** | Image, Video | `ARK_API_KEY` |
| **MiniMax** | Video | `MINIMAX_API_KEY` |
| **Prodia** | Image, Video | `PRODIA_TOKEN` |
| **Azure OpenAI** | Image | `AZURE_API_KEY` |
---
## Configuring Providers & API Keys
Clicking on any provider in the **Media Providers** list opens its configuration dialog:
### Provider Statuses
- **Configured (Custom)**: A team-specific API key or custom environment variable name was entered in this dialog.
- **Configured (Environment)**: The system detected a server-level default environment variable (e.g. `OPENAI_API_KEY`). No team key is required unless you want to override it.
- **Not Configured**: No API key is available. You must provide an API key or an environment variable name to use this provider.
### Overriding API Keys
In the **API Key** tab of the provider dialog:
- Enter an API key literal (e.g. `sk-...`) or a custom environment variable name available on the host system.
- Click **Clear Custom Key** to revert back to using the default environment variable if one exists.
---
## Managing Models
In the **Models** tab of the provider dialog:
### Curated Models
Shumai comes pre-configured with popular curated models (such as `dall-e-3`, `fal-ai/flux/dev`, `veo-3.1-generate`, `kling-v3.0-t2v`, etc.). Toggle the switch next to any model to enable or disable it for your team.
### Custom Models
If your provider supports a new or fine-tuned model not listed in the curated models:
1. Select the modality (**Image** or **Video**).
2. Enter the exact **Model ID** recognized by the upstream provider.
3. Click **Add**.
### Default Model Selection
The **first enabled model** of each type (`image` or `video`) automatically becomes the default model. When an agent calls `generate_image` or `generate_video` without explicitly specifying a `model` parameter, Shumai automatically directs the generation request to this default model.
---
## How Agents Use Generation Tools
Once enabled, agents can invoke the tools during conversations or automated workflows:
### 1. Generate Image (`generate_image`)
- **Prompt:** Detailed text description of the image.
- **Model Selection:** Automatically defaults to your first enabled image model, or the agent can choose any enabled model in `provider:modelId` format (e.g. `openai:dall-e-3` or `fal:fal-ai/flux/dev`).
- **Image References:** Agents can use existing workspace assets as style or composition references by passing S3 storage keys obtained from the `read_asset` tool (`s3KeyOnly: true`).
- **Inpainting:** Supports mask-guided localized image editing using an S3 mask key.
- **Parameters:** Custom aspect ratios (`1:1`, `16:9`, `9:16`, `4:3`, `3:4`, `2:3`, `3:2`), dimensions, and seeds.
### 2. Generate Video (`generate_video`)
- **Generation Modes:**
- `text_to_video`: Generate a clip from a descriptive motion prompt.
- `image_to_video`: Animate an existing still image asset.
- `first_last_frame`: Seamlessly interpolate and transition between two keyframe assets.
- `reference_to_video`: Use reference images for consistent characters or environments.
- **Parameters:** Aspect ratio (`16:9`, `9:16`, `1:1`, `adaptive`), resolution, duration, frame rate (FPS), and audio generation.
### Asset Creation Workflow
All generated media files are initially saved to the local sandboxed directory (`.pi/`). Agents can then use **Create File** (`create_file`) to upload the output as a new project asset or **Create Version** (`create_version`) to stack it onto an existing revision history.