diff --git a/CHANGELOG.md b/CHANGELOG.md index 3b699f66..f61208f5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added - **storage**: Add optional archive storage support (`local` folder/NAS or `s3` with configurable storage class and Glacier restore) allowing editors and owners to archive and restore original files while keeping previews, proxy streams, comments, and metadata available in main storage +- **storage**: Add team-level automatic asset archival based on age policies (creation time or last modified time) with configurable post-restore grace periods, an hourly background scanning worker, and a dedicated Storage settings page displaying read-only archive backend status and policy configurations ### Fixed diff --git a/README.md b/README.md index 80379de7..c835f303 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,7 @@ Shumai is an open-source Frame.io alternative built for creative teams to collaborate and share media with ease: * **S3-Compatible & Local Storage**: Store and serve creative assets securely using the local filesystem or any S3-compatible storage service, including AWS S3, Cloudflare R2, and MinIO. -* **Cold Archive Storage**: Offload original master media files to secondary local storage or cold S3 storage to save on storage costs while keeping proxies for instant browsing and playback. See [Archive Storage documentation](https://docs.shumai.one/configuration/archive-storage). +* **Archive Storage & Auto-Archival**: Offload original master media files to secondary local storage or cold S3 storage—manually or automatically via team lifecycle policies—to save on storage costs while keeping proxies for instant browsing and playback. See [Archive Storage documentation](https://docs.shumai.one/configuration/archive-storage). * **Frame-by-Frame Annotations & Comments**: Leave precise feedback with frame-specific drawing tools and timestamped comments on both video and image assets. * **Secure Sharing & Collections**: Create private share links and curated media collections for clients, stakeholders, and collaborators. * **Granular Access Control**: Manage workspace permissions with team-level and project-level role-based access control. diff --git a/apps/web/src/index.ts b/apps/web/src/index.ts index 7e9db8cd..c2ec12e4 100644 --- a/apps/web/src/index.ts +++ b/apps/web/src/index.ts @@ -20,6 +20,7 @@ import { authService } from '@shumai/core/src/auth/auth' import { embeddingService } from '@shumai/core/src/search/embedding-service' import { sandboxService } from '@shumai/core' import { notificationJobService } from '@shumai/core/src/notification/notification-job' +import { autoArchiveJobService } from '@shumai/core/src/asset/auto-archive-job' if (process.argv.includes('--check')) { console.log('✅ Web app evaluated successfully!') @@ -91,6 +92,7 @@ async function run() { await agentService.ensureSystemEmbeddingAgents().catch(console.error) assetService.startCleanupJob() notificationJobService.start() + autoArchiveJobService.start() const args = process.argv.slice(2) let workersOption = process.env.SHUMAI_WORKERS || 'all' for (let i = 0; i < args.length; i++) { diff --git a/docs/configuration/archive-storage.mdx b/docs/configuration/archive-storage.mdx index 54166c01..b01e03dd 100644 --- a/docs/configuration/archive-storage.mdx +++ b/docs/configuration/archive-storage.mdx @@ -39,11 +39,12 @@ Configure archive storage by adding the following environment variables to your ### General Configuration -| Variable | Description | Required | Default | -| :------------------------ | :------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------- | :--------------- | -| `ARCHIVE_STORAGE_BACKEND` | Storage backend for archives: `local` or `s3`. When unset, archive functionality is disabled in the WebUI. | No | Unset (Disabled) | -| `CONCURRENCY_ARCHIVE` | Maximum concurrent archive and restore tasks executed by background workers. | No | `5` | -| `SHUMAI_WORKERS` | Worker queues listened to by the application server: `all` (default), `none`, `agent` (or `ai`), `transcode`, `archive`. Set to `all` or `archive` to process tasks. | No | `all` | +| Variable | Description | Required | Default | +| :------------------------------ | :------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------- | :--------------- | +| `ARCHIVE_STORAGE_BACKEND` | Storage backend for archives: `local` or `s3`. When unset, archive functionality is disabled in the WebUI. | No | Unset (Disabled) | +| `CONCURRENCY_ARCHIVE` | Maximum concurrent archive and restore tasks executed by background workers. | No | `5` | +| `SHUMAI_WORKERS` | Worker queues listened to by the application server: `all` (default), `none`, `agent` (or `ai`), `transcode`, `archive`. Set to `all` or `archive` to process tasks. | No | `all` | +| `AUTO_ARCHIVE_INTERVAL_MINUTES` | Frequency (in minutes) of the background auto-archive scanner job. Accepts decimal numbers (e.g. `0.02` for testing). | No | `60` (1 hour) | ### Local Archive Storage @@ -198,6 +199,48 @@ ARCHIVE_S3_STORAGE_CLASS=COLDLINE --- +## Automatic Asset Archival + +In addition to manual on-demand archival, Shumai allows teams to automate cold storage tiering based on asset age lifecycle policies. This ensures heavy original master files are automatically offloaded to secondary cold storage without requiring manual housekeeping from creative staff or administrators. + +### WebUI Configuration + +Team owners can configure automatic archival under **Team Settings > Storage**: + +1. **Archive Storage Status Overview**: + - Displays the current system archive storage backend (`local` or `s3`), endpoint, bucket, or root directory configured on the server. + - If archive storage is not configured in server environment variables, a warning alert is displayed, and auto-archive controls are disabled. + - For security, credentials and private server settings are not exposed to client browsers. + +2. **Automatic Asset Archival Policy**: + - **Enable Auto Archive**: Toggle automated background archival on or off for the team (disabled by default; requires an active archive storage backend). + - **Archive Criteria (`method`)**: + - **Not modified for specified days (`not_updated`)**: Evaluates the last modification timestamp (`updatedAt`). Ideal for active production workflows—files stay in fast primary storage as long as team members are actively commenting, annotating, or modifying them. Once no modifications have occurred for the configured duration, the asset is moved to cold storage. + - **After creation for specified days (`after_created`)**: Evaluates the initial upload/creation timestamp (`createdAt`). Ideal for strict retention or compliance pipelines where footage is tiered based on its age (e.g., 60 or 90 days after delivery) regardless of recent review comments. + - **Age Threshold (`days`)**: The threshold in days (between 1 and 3,650 days) after which eligible assets are queued for archival. + - **Post-Restore Grace Period (`postRestoreGraceDays`)**: + - When an asset is restored from archive back to primary storage, its modification date is refreshed to the restore time. + - When using the **After creation** rule, an asset older than the threshold (e.g. 100 days old) would otherwise be immediately picked up and re-archived by the very next hourly scan. + - The post-restore grace period (default: 30 days) guarantees that restored files remain in primary storage for at least the specified number of days before becoming eligible for automated archival again. This value should be set smaller than the Age Threshold. Under this policy, only files that are older than the Age Threshold and have also surpassed the Post-Restore Grace Period since restoration are auto-archived. + +### Sibling Deduplication & Safety + +Shumai uses content-addressable storage keys (`storage_keys`) to deduplicate identical files across projects and folders: + +- **Sibling Protection**: If multiple asset records reference the same underlying storage object (e.g., copied or duplicated files across folders or projects), the auto-archive scanner inspects all sibling assets. +- **Strict Consensus**: The underlying storage file is **only** archived if **all active sibling files** satisfy the team's auto-archive criteria and are in a stable `processed` state. +- **Starvation-Free Scheduling**: If any sibling copy was modified recently or is currently in an active workflow, the storage key is safely skipped without blocking other eligible assets from being archived. + +### Background Scanner & Execution Semantics + +The auto-archive job runs periodically in the background: + +- **Scan Frequency**: Configured via the `AUTO_ARCHIVE_INTERVAL_MINUTES` environment variable (defaults to `60` minutes; accepts decimal values such as `0.02` for automated testing). +- **Distributed Leader Election**: Uses PostgreSQL transaction-scoped advisory locks (`pg_try_advisory_xact_lock`) during the candidate discovery phase. In multi-pod or multi-replica container deployments, only one server instance performs the scan per interval while other replicas cleanly skip. +- **Race Condition Prevention**: During archival, candidate assets transition atomically via conditional SQL (`UPDATE assets SET status = 'archiving' WHERE status = 'processed'`) protected by row-level `storage_keys FOR UPDATE` locks. This guarantees zero race conditions with concurrent manual user archival, restoration, or file deletion. + +--- + ## WebUI Behavior & Permissions - **Permissions**: Only team **Owners** and **Editors** can archive or restore assets. diff --git a/docs/configuration/env-variables.mdx b/docs/configuration/env-variables.mdx index 5456c78a..5bbe4f2d 100644 --- a/docs/configuration/env-variables.mdx +++ b/docs/configuration/env-variables.mdx @@ -78,6 +78,7 @@ Shumai supports configuring an optional **Archive Storage** tier (`local` NAS/di | `ARCHIVE_S3_USE_RESTORE_OBJECT` | Controls whether S3 `RestoreObject` is called before reading archived objects (`auto`, `true`, or `false`). | `auto` (default) | | `ARCHIVE_S3_RESTORE_DAYS` | Number of days a restored copy of a cold S3 object remains readable during restoration. | `1` (default) | | `ARCHIVE_S3_RESTORE_TIER` | Optional S3 Glacier retrieval tier (`Expedited`, `Standard`, `Bulk`). Omitted when unset for S3-compatible providers that reject `GlacierJobParameters`. | Unset (default) | +| `AUTO_ARCHIVE_INTERVAL_MINUTES` | Frequency (in minutes) of the background auto-archive scanner job. Accepts decimal values (e.g. `0.02` for testing). | `60` (1 hour default) | --- diff --git a/packages/api/src/team.test.ts b/packages/api/src/team.test.ts index 4e9bba8a..7164f94f 100644 --- a/packages/api/src/team.test.ts +++ b/packages/api/src/team.test.ts @@ -244,7 +244,23 @@ describe('team api', () => { permission: Permission.Read, }), ) - expect(mockGetSettings).toHaveBeenCalledWith('t1') + expect(mockGetSettings).toHaveBeenCalledWith('t1', { isAdmin: true }) + }) + + it('GET /teams/:teamId/settings passes isAdmin: false when user lacks admin permission', async () => { + vi.mocked(authzService.hasPermission).mockImplementation(async (params) => { + if (params.permission === Permission.Admin) { + throw new Error('Forbidden') + } + }) + mockGetSettings.mockResolvedValue({ + archiveStorage: { enabled: true }, + }) + + const res = await app.request('/teams/t1/settings') + + expect(res.status).toBe(200) + expect(mockGetSettings).toHaveBeenCalledWith('t1', { isAdmin: false }) }) it('GET /teams/:teamId/settings does not return emailNotification to readers', async () => { @@ -381,6 +397,65 @@ describe('team api', () => { expect(mockUpdateSettings).toHaveBeenCalledWith('t1', 'appearance.hideAgent', true) }) + it('PATCH /teams/:teamId/settings updates autoArchive settings object', async () => { + mockUpdateSettings.mockResolvedValue({ + autoArchive: { + enabled: true, + method: 'not_updated', + days: 30, + postRestoreGraceDays: 30, + }, + }) + + const res = await app.request('/teams/t1/settings', { + method: 'PATCH', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ + key: 'autoArchive', + value: { + enabled: true, + method: 'not_updated', + days: 30, + postRestoreGraceDays: 30, + }, + }), + }) + + expect(res.status).toBe(200) + const data = await res.json() + expect(data.autoArchive.enabled).toBe(true) + expect(data.autoArchive.days).toBe(30) + expect(mockUpdateSettings).toHaveBeenCalledWith('t1', 'autoArchive', { + enabled: true, + method: 'not_updated', + days: 30, + postRestoreGraceDays: 30, + }) + }) + + it('PATCH /teams/:teamId/settings updates autoArchive.enabled individual field', async () => { + mockUpdateSettings.mockResolvedValue({ + autoArchive: { + enabled: false, + method: 'not_updated', + days: 30, + postRestoreGraceDays: 30, + }, + }) + + const res = await app.request('/teams/t1/settings', { + method: 'PATCH', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ + key: 'autoArchive.enabled', + value: false, + }), + }) + + expect(res.status).toBe(200) + expect(mockUpdateSettings).toHaveBeenCalledWith('t1', 'autoArchive.enabled', false) + }) + it('GET /teams/:teamId/user-metadata returns all metadata for the user in the team', async () => { mockListUserMetadata.mockResolvedValue([ { key: 'key1', value: 'value1' }, diff --git a/packages/api/src/team.ts b/packages/api/src/team.ts index 96708c68..8d2d903e 100644 --- a/packages/api/src/team.ts +++ b/packages/api/src/team.ts @@ -210,7 +210,17 @@ const route = new Hono<{ Variables: { user: User } }>() id: teamId, }) - const settings = await teamService.getSettings(teamId) + const isAdmin = await authzService + .hasPermission({ + user, + permission: Permission.Admin, + type: ResourceType.Team, + id: teamId, + }) + .then(() => true) + .catch(() => false) + + const settings = await teamService.getSettings(teamId, { isAdmin }) return c.json(settings) }) .patch( diff --git a/packages/core/src/asset/asset-archive.ts b/packages/core/src/asset/asset-archive.ts index f02e358e..eaa09580 100644 --- a/packages/core/src/asset/asset-archive.ts +++ b/packages/core/src/asset/asset-archive.ts @@ -82,7 +82,17 @@ export class AssetArchiveService { }) } - async archiveAssets(assetIds: string[]): Promise { + async archiveAssets( + assetIds: string[], + options?: { + suppressBusyErrors?: boolean + autoArchivePolicy?: { + method: 'not_updated' | 'after_created' + cutoffDate: Date + restoreCutoffDate: Date + } + }, + ): Promise { if (!isArchiveStorageEnabled()) { throw new HTTPException(400, { message: 'Archive storage is not configured' }) } @@ -187,15 +197,17 @@ export class AssetArchiveService { } if (eligibleByStorageKey.size === 0) { - if (alreadyArchivedCount === 0 && busyCount > 0) { - throw new HTTPException(409, { - message: 'Cannot archive asset while a workflow task is in progress', - }) - } - if (alreadyArchivedCount === 0 && invalidStateCount > 0) { - throw new HTTPException(409, { - message: 'Only processed assets can be archived', - }) + if (!options?.suppressBusyErrors) { + if (alreadyArchivedCount === 0 && busyCount > 0) { + throw new HTTPException(409, { + message: 'Cannot archive asset while a workflow task is in progress', + }) + } + if (alreadyArchivedCount === 0 && invalidStateCount > 0) { + throw new HTTPException(409, { + message: 'Only processed assets can be archived', + }) + } } return { archivedAssetIds: [], taskIds: [], skippedCount } } @@ -206,28 +218,65 @@ export class AssetArchiveService { for (const [storageKeyId, filesForKey] of eligibleByStorageKey.entries()) { const primaryFile = filesForKey[0] await this.prismaClient.$transaction(async (tx) => { - const sharedAssets = await tx.asset.findMany({ - where: { - storageKeyId, - isDeleted: false, - type: AssetType.file, - status: AssetStatus.processed, - }, - select: { id: true }, - }) - const sharedAssetIds = sharedAssets.map((a) => a.id) + // 1. Lock the storageKey row to serialize concurrent archive requests + const lockedKeys = await tx.$queryRaw>` + SELECT id, is_archived AS "isArchived" + FROM storage_keys + WHERE id = ${storageKeyId} + FOR UPDATE + ` + if (!lockedKeys || lockedKeys.length === 0 || lockedKeys[0].isArchived) { + return + } - if (sharedAssetIds.length > 0) { - await tx.asset.updateMany({ - where: { id: { in: sharedAssetIds } }, - data: { status: AssetStatus.archiving }, + // If an auto-archive policy was specified, re-verify under lock that all sibling assets meet the age policy + if (options?.autoArchivePolicy) { + const policy = options.autoArchivePolicy + const ineligible = await tx.asset.findFirst({ + where: { + storageKeyId, + isDeleted: false, + OR: [ + { type: { not: AssetType.file } }, + { status: { not: AssetStatus.processed } }, + ...(policy.method === 'not_updated' + ? [{ updatedAt: { gt: policy.cutoffDate } }] + : [ + { + OR: [ + { createdAt: { gt: policy.cutoffDate } }, + { updatedAt: { gt: policy.restoreCutoffDate } }, + ], + }, + ]), + ], + }, + select: { id: true }, }) - - for (const id of sharedAssetIds) { - archivedAssetIdsSet.add(id) + if (ineligible) { + return } } + // 2. Atomic Filter + Update: only files that are currently 'processed' transition to 'archiving' + const updatedAssets = await tx.$queryRaw>` + UPDATE assets + SET status = 'archiving'::"AssetStatus", updated_at = NOW() + WHERE storage_key_id = ${storageKeyId} + AND type = 'file'::"AssetType" + AND status = 'processed'::"AssetStatus" + AND is_deleted = false + RETURNING id + ` + + if (!updatedAssets || updatedAssets.length === 0) { + return + } + + for (const row of updatedAssets) { + archivedAssetIdsSet.add(row.id) + } + const existingTask = await tx.workflowTask.findFirst({ where: { assetId: { in: filesForKey.map((f) => f.id) }, diff --git a/packages/core/src/asset/asset.ts b/packages/core/src/asset/asset.ts index f17f58aa..d23773d0 100644 --- a/packages/core/src/asset/asset.ts +++ b/packages/core/src/asset/asset.ts @@ -2991,8 +2991,18 @@ export class AssetService { return s3Service.presign(process.env.S3_BUCKET || 'shumai', key, 'GET', true, downloadFilename) } - async archiveAssets(assetIds: string[]): Promise { - return assetArchiveService.archiveAssets(assetIds) + async archiveAssets( + assetIds: string[], + options?: { + suppressBusyErrors?: boolean + autoArchivePolicy?: { + method: 'not_updated' | 'after_created' + cutoffDate: Date + restoreCutoffDate: Date + } + }, + ): Promise { + return assetArchiveService.archiveAssets(assetIds, options) } async restoreArchivedAssets(assetIds: string[]): Promise { diff --git a/packages/core/src/asset/auto-archive-job.test.ts b/packages/core/src/asset/auto-archive-job.test.ts new file mode 100644 index 00000000..4c408be6 --- /dev/null +++ b/packages/core/src/asset/auto-archive-job.test.ts @@ -0,0 +1,464 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import * as fs from 'fs' +import * as path from 'path' +import { + AssetStatus, + prisma, + Prisma, + WorkflowTaskType, + WorkflowTaskStatus, + type User, +} from '@shumai/db' +import { setupTestDbHooks } from '@shumai/db/test' +import { projectService } from '@shumai/core/src/project/project' +import { logger } from '@shumai/core/src/logger' +import { assetService } from './asset' +import { assetArchiveService } from './asset-archive' +import { resetArchiveStorageServiceForTests } from '../s3/archive-storage' +import { + AutoArchiveJobService, + autoArchiveJobService, + getAutoArchiveIntervalMs, +} from './auto-archive-job' + +describe('AutoArchiveJobService', () => { + setupTestDbHooks() + + const archiveRoot = path.join(process.cwd(), 'data-test-auto-archive') + let originalEnv: NodeJS.ProcessEnv + let owner: User + let teamId: string + let projectId: string + let rootFolderId: string + + beforeEach(async () => { + originalEnv = { ...process.env } + process.env.ARCHIVE_STORAGE_BACKEND = 'local' + process.env.ARCHIVE_LOCAL_ROOT = archiveRoot + resetArchiveStorageServiceForTests() + + if (fs.existsSync(archiveRoot)) { + fs.rmSync(archiveRoot, { recursive: true, force: true }) + } + + owner = await prisma.user.create({ + data: { name: 'Auto Archive Owner', email: 'auto-archive-owner@test.com', password: 'pw' }, + }) + const team = await prisma.team.create({ + data: { + name: 'Auto Archive Team', + settings: { + autoArchive: { + enabled: true, + method: 'not_updated', + days: 30, + postRestoreGraceDays: 30, + }, + }, + }, + }) + teamId = team.id + await prisma.teamMember.create({ data: { teamId, userId: owner.id, role: 'owner' } }) + + const project = await projectService.createProject(owner, { teamId, name: 'Archive Project' }) + projectId = project.id + rootFolderId = project.rootFolder! + }) + + afterEach(() => { + process.env = originalEnv + resetArchiveStorageServiceForTests() + if (fs.existsSync(archiveRoot)) { + fs.rmSync(archiveRoot, { recursive: true, force: true }) + } + autoArchiveJobService.stop() + }) + + async function createTestFile(params: { + name: string + key: string + createdAt?: Date + updatedAt?: Date + status?: AssetStatus + }) { + const asset = await assetService.createAsset({ + name: params.name, + type: 'file', + parentId: rootFolderId, + projectId, + key: params.key, + sizeByte: 2048, + contentType: 'video/mp4', + creatorId: owner.id, + }) + + const updated = await prisma.asset.update({ + where: { id: asset.id }, + data: { + status: params.status ?? AssetStatus.processed, + createdAt: params.createdAt ?? new Date(), + updatedAt: params.updatedAt ?? new Date(), + }, + include: { storageKey: true }, + }) + + return updated + } + + describe('getAutoArchiveIntervalMs', () => { + it('returns default 1 hour (3600000ms) when env not specified', () => { + delete process.env.AUTO_ARCHIVE_INTERVAL_MINUTES + expect(getAutoArchiveIntervalMs()).toBe(3600000) + }) + + it('parses integer minute strings', () => { + process.env.AUTO_ARCHIVE_INTERVAL_MINUTES = '30' + expect(getAutoArchiveIntervalMs()).toBe(1800000) + }) + + it('parses decimal floating point minute strings', () => { + process.env.AUTO_ARCHIVE_INTERVAL_MINUTES = '0.02' + expect(getAutoArchiveIntervalMs()).toBe(1200) + }) + }) + + it('skips scan when archive storage is not configured', async () => { + delete process.env.ARCHIVE_STORAGE_BACKEND + resetArchiveStorageServiceForTests() + + const job = new AutoArchiveJobService() + const res = await job.processAutoArchive() + expect(res.archivedCount).toBe(0) + }) + + it('skips teams where autoArchive is disabled', async () => { + await prisma.team.update({ + where: { id: teamId }, + data: { + settings: { + autoArchive: { + enabled: false, + method: 'not_updated', + days: 30, + postRestoreGraceDays: 30, + }, + }, + }, + }) + + const oldDate = new Date(Date.now() - 40 * 86_400_000) + await createTestFile({ + name: 'old-file.mp4', + key: 'files/test/old-file.mp4', + updatedAt: oldDate, + }) + + const job = new AutoArchiveJobService() + const res = await job.processAutoArchive() + expect(res.archivedCount).toBe(0) + }) + + it('archives assets exceeding not_updated threshold while leaving recent assets untouched', async () => { + const oldDate = new Date(Date.now() - 35 * 86_400_000) + const recentDate = new Date(Date.now() - 5 * 86_400_000) + + const oldFile = await createTestFile({ + name: 'old-file.mp4', + key: 'files/test/old-file.mp4', + updatedAt: oldDate, + }) + + const recentFile = await createTestFile({ + name: 'recent-file.mp4', + key: 'files/test/recent-file.mp4', + updatedAt: recentDate, + }) + + const job = new AutoArchiveJobService() + const res = await job.processAutoArchive() + + expect(res.archivedCount).toBe(1) + + const refreshedOld = await prisma.asset.findUniqueOrThrow({ where: { id: oldFile.id } }) + const refreshedRecent = await prisma.asset.findUniqueOrThrow({ where: { id: recentFile.id } }) + + expect(refreshedOld.status).toBe(AssetStatus.archiving) + expect(refreshedRecent.status).toBe(AssetStatus.processed) + + const task = await prisma.workflowTask.findFirst({ + where: { assetId: oldFile.id, type: WorkflowTaskType.archive_asset }, + }) + expect(task).not.toBeNull() + expect(task?.status).toBe(WorkflowTaskStatus.pending) + }) + + it('archives assets exceeding after_created threshold and respects postRestoreGraceDays', async () => { + await prisma.team.update({ + where: { id: teamId }, + data: { + settings: { + autoArchive: { + enabled: true, + method: 'after_created', + days: 30, + postRestoreGraceDays: 14, + }, + }, + }, + }) + + const oldCreatedDate = new Date(Date.now() - 45 * 86_400_000) + const recentlyRestoredDate = new Date(Date.now() - 2 * 86_400_000) + const oldRestoredDate = new Date(Date.now() - 20 * 86_400_000) + + // Old creation, but restored 2 days ago (within 14-day grace period) -> should NOT archive + const restoredRecently = await createTestFile({ + name: 'restored-recently.mp4', + key: 'files/test/restored-recently.mp4', + createdAt: oldCreatedDate, + updatedAt: recentlyRestoredDate, + }) + + // Old creation, restored 20 days ago (> 14-day grace period) -> SHOULD archive + const restoredLongAgo = await createTestFile({ + name: 'restored-long-ago.mp4', + key: 'files/test/restored-long-ago.mp4', + createdAt: oldCreatedDate, + updatedAt: oldRestoredDate, + }) + + const job = new AutoArchiveJobService() + const res = await job.processAutoArchive() + + expect(res.archivedCount).toBe(1) + + const refreshedRecently = await prisma.asset.findUniqueOrThrow({ + where: { id: restoredRecently.id }, + }) + const refreshedLongAgo = await prisma.asset.findUniqueOrThrow({ + where: { id: restoredLongAgo.id }, + }) + + expect(refreshedRecently.status).toBe(AssetStatus.processed) + expect(refreshedLongAgo.status).toBe(AssetStatus.archiving) + }) + + it('skips storage key if any sibling file sharing the key was recently modified', async () => { + const oldDate = new Date(Date.now() - 40 * 86_400_000) + const recentDate = new Date() + + const sharedKey = 'files/test/shared-video.mp4' + + const primaryFile = await createTestFile({ + name: 'primary.mp4', + key: sharedKey, + updatedAt: oldDate, + }) + + // Create a sibling copy sharing the same storageKeyId but modified recently + const siblingCopy = await assetService.createAsset({ + name: 'sibling-copy.mp4', + type: 'file', + parentId: rootFolderId, + projectId, + key: sharedKey, + sizeByte: 2048, + contentType: 'video/mp4', + creatorId: owner.id, + }) + + await prisma.asset.update({ + where: { id: siblingCopy.id }, + data: { + status: AssetStatus.processed, + updatedAt: recentDate, + }, + }) + + const job = new AutoArchiveJobService() + const res = await job.processAutoArchive() + + expect(res.archivedCount).toBe(0) + + const refreshedPrimary = await prisma.asset.findUniqueOrThrow({ + where: { id: primaryFile.id }, + }) + expect(refreshedPrimary.status).toBe(AssetStatus.processed) + }) + + it('handles race conditions: concurrent archive attempts cleanly serialize without duplicate tasks', async () => { + const oldDate = new Date(Date.now() - 50 * 86_400_000) + const file = await createTestFile({ + name: 'concurrent-test.mp4', + key: 'files/test/concurrent-test.mp4', + updatedAt: oldDate, + }) + + const job = new AutoArchiveJobService() + + // Trigger auto-archive and manual archive concurrently + const [jobRes, manualRes] = await Promise.all([ + job.processAutoArchive(), + assetService.archiveAssets([file.id], { suppressBusyErrors: true }), + ]) + + // Together they should archive the asset exactly once + const totalArchived = jobRes.archivedCount + manualRes.archivedAssetIds.length + expect(totalArchived).toBe(1) + + // Verify exactly ONE WorkflowTask was created + const tasks = await prisma.workflowTask.findMany({ + where: { assetId: file.id, type: WorkflowTaskType.archive_asset }, + }) + expect(tasks).toHaveLength(1) + + const refreshedFile = await prisma.asset.findUniqueOrThrow({ where: { id: file.id } }) + expect(refreshedFile.status).toBe(AssetStatus.archiving) + }) + + it('prevents candidate starvation: storage keys with stuck siblings are ignored so eligible candidates are not blocked', async () => { + const oldDate = new Date(Date.now() - 40 * 86_400_000) + + // Key 1: has an old file AND a stuck sibling with status 'failed' + const stuckKey = 'files/test/stuck-sibling.mp4' + const fileWithStuckSibling = await createTestFile({ + name: 'stuck-primary.mp4', + key: stuckKey, + updatedAt: oldDate, + }) + const stuckSibling = await assetService.createAsset({ + name: 'stuck-sibling-failed.mp4', + type: 'file', + parentId: rootFolderId, + projectId, + key: stuckKey, + sizeByte: 2048, + contentType: 'video/mp4', + creatorId: owner.id, + }) + await prisma.asset.update({ + where: { id: stuckSibling.id }, + data: { + status: AssetStatus.failed, + updatedAt: oldDate, + }, + }) + + // Key 2: fully eligible old file + const eligibleFile = await createTestFile({ + name: 'eligible-siblingless.mp4', + key: 'files/test/eligible-siblingless.mp4', + updatedAt: oldDate, + }) + + const job = new AutoArchiveJobService() + // With batchSize 1, the query must pick the eligible file instead of getting stuck on Key 1 + const res = await job.processAutoArchive(1) + + expect(res.archivedCount).toBe(1) + + const refreshedEligible = await prisma.asset.findUniqueOrThrow({ + where: { id: eligibleFile.id }, + }) + expect(refreshedEligible.status).toBe(AssetStatus.archiving) + + const refreshedStuck = await prisma.asset.findUniqueOrThrow({ + where: { id: fileWithStuckSibling.id }, + }) + expect(refreshedStuck.status).toBe(AssetStatus.processed) + }) + + it('skips scan when another process holds the transaction advisory lock', async () => { + const queryRawSpy = vi.spyOn(prisma, '$queryRaw').mockImplementation((query: unknown) => { + const sqlText = String((query as { strings?: string[] })?.strings || '') + if (sqlText.includes('pg_try_advisory_xact_lock')) { + return Promise.resolve([{ locked: false }]) as unknown as Prisma.PrismaPromise + } + queryRawSpy.mockRestore() + return prisma.$queryRaw(query as TemplateStringsArray) + }) + + try { + const job = new AutoArchiveJobService() + const res = await job.processAutoArchive() + expect(res.archivedCount).toBe(0) + } finally { + queryRawSpy.mockRestore() + } + }) + + it('logs info level message on job start and finish with totalArchived count', async () => { + const infoSpy = vi.spyOn(logger, 'info') + + const oldDate = new Date(Date.now() - 40 * 86_400_000) + await createTestFile({ + name: 'log-test.mp4', + key: 'files/test/log-test.mp4', + updatedAt: oldDate, + }) + + const job = new AutoArchiveJobService() + const res = await job.processAutoArchive() + + expect(res.archivedCount).toBe(1) + expect(infoSpy).toHaveBeenCalledWith('Auto-archive job started') + expect(infoSpy).toHaveBeenCalledWith({ totalArchived: 1 }, 'Auto-archive job finished') + + infoSpy.mockRestore() + }) + + it('does not log info level messages when no files are eligible to archive', async () => { + const infoSpy = vi.spyOn(logger, 'info') + + const job = new AutoArchiveJobService() + const res = await job.processAutoArchive() + + expect(res.archivedCount).toBe(0) + expect(infoSpy).not.toHaveBeenCalledWith('Auto-archive job started') + expect(infoSpy).not.toHaveBeenCalledWith(expect.anything(), 'Auto-archive job finished') + + infoSpy.mockRestore() + }) + + it('does not archive non-file assets even if their status is processed and age criteria met', async () => { + const oldDate = new Date(Date.now() - 50 * 86_400_000) + + const folder = await assetService.createAsset({ + name: 'Old Folder', + type: 'folder', + projectId, + parentId: rootFolderId, + creatorId: owner.id, + }) + await prisma.asset.update({ + where: { id: folder.id }, + data: { + status: AssetStatus.processed, + updatedAt: oldDate, + createdAt: oldDate, + }, + }) + + const res = await assetArchiveService.archiveAssets([folder.id], { + suppressBusyErrors: true, + autoArchivePolicy: { + method: 'not_updated', + cutoffDate: new Date(), + restoreCutoffDate: new Date(), + }, + }) + + expect(res.archivedAssetIds).toHaveLength(0) + + const refreshedFolder = await prisma.asset.findUniqueOrThrow({ + where: { id: folder.id }, + }) + expect(refreshedFolder.status).toBe(AssetStatus.processed) + }) + + it('can start and stop without errors', () => { + const job = new AutoArchiveJobService() + job.start(1000, 5000) + job.stop() + }) +}) diff --git a/packages/core/src/asset/auto-archive-job.ts b/packages/core/src/asset/auto-archive-job.ts new file mode 100644 index 00000000..f14a38a5 --- /dev/null +++ b/packages/core/src/asset/auto-archive-job.ts @@ -0,0 +1,197 @@ +import { prisma, Prisma } from '@shumai/db' +import { isArchiveStorageEnabled } from '@shumai/core/src/s3/archive-storage' +import { assetArchiveService } from '@shumai/core/src/asset/asset-archive' +import { logger } from '@shumai/core/src/logger' + +export function getAutoArchiveIntervalMs(): number { + const raw = process.env.AUTO_ARCHIVE_INTERVAL_MINUTES + if (raw) { + const parsed = Number.parseFloat(raw) + if (!Number.isNaN(parsed) && parsed > 0) { + return Math.round(parsed * 60 * 1000) + } + } + return 60 * 60 * 1000 // default 1 hour +} + +interface TeamArchiveBatch { + teamId: string + assetIds: string[] + policy: { + method: 'not_updated' | 'after_created' + cutoffDate: Date + restoreCutoffDate: Date + } +} + +export class AutoArchiveJobService { + private isRunning = false + private isProcessing = false + private timer: NodeJS.Timeout | null = null + + start(initialDelayMs = 60_000, intervalMs?: number): void { + if (this.isRunning) return + this.isRunning = true + logger.info('Starting auto-archive background job') + + const run = async () => { + if (!this.isRunning) return + try { + await this.processAutoArchive() + } catch (err: unknown) { + logger.error({ err }, 'Error in auto-archive background job') + } + + if (this.isRunning) { + const nextInterval = intervalMs ?? getAutoArchiveIntervalMs() + this.timer = setTimeout(run, nextInterval) + } + } + + this.timer = setTimeout(run, initialDelayMs) + } + + stop(): void { + this.isRunning = false + if (this.timer) { + clearTimeout(this.timer) + this.timer = null + } + logger.info('Stopped auto-archive background job') + } + + async processAutoArchive(batchSize = 100): Promise<{ archivedCount: number }> { + if (!isArchiveStorageEnabled()) { + logger.debug('Archive storage is not configured; skipping auto-archive run') + return { archivedCount: 0 } + } + + if (this.isProcessing) { + logger.debug('Auto-archive job turn already in progress, skipping') + return { archivedCount: 0 } + } + + this.isProcessing = true + + try { + let totalArchived = 0 + + // Phase 1: Query & Scan under transaction-scoped advisory lock. + // Automatically released as soon as this transaction completes. + const teamBatches = await prisma.$transaction(async (tx) => { + const lockRes = await tx.$queryRaw>` + SELECT pg_try_advisory_xact_lock(hashtext('auto_archive_job')) AS locked + ` + if (!lockRes || !lockRes[0]?.locked) { + logger.debug( + 'Could not acquire transaction advisory lock for auto-archive scan, skipping', + ) + return null + } + + // Query only teams where autoArchive is enabled via JSON filter + const eligibleTeams = await tx.team.findMany({ + where: { + settings: { + path: ['autoArchive', 'enabled'], + equals: true, + }, + }, + select: { id: true, settings: true }, + }) + + if (eligibleTeams.length === 0) { + return [] + } + + const batches: TeamArchiveBatch[] = [] + + for (const team of eligibleTeams) { + const settings = (team.settings || {}) as PrismaJson.Settings + const autoArchive = settings.autoArchive + const method = autoArchive?.method || 'not_updated' + const days = Math.max(1, autoArchive?.days || 30) + const postRestoreGraceDays = Math.max(1, autoArchive?.postRestoreGraceDays || 30) + + const now = Date.now() + const cutoffDate = new Date(now - days * 86_400_000) + const restoreCutoffDate = new Date(now - postRestoreGraceDays * 86_400_000) + + // Query candidate assets whose storage keys have NO ineligible sibling. + // NOT EXISTS eliminates starvation from stuck/recent siblings at the database level. + const candidateRows = await tx.$queryRaw>` + SELECT a.id + FROM assets a + JOIN projects p ON a.project_id = p.id + JOIN storage_keys sk ON a.storage_key_id = sk.id + WHERE p.team_id = ${team.id} + AND a.type = 'file'::"AssetType" + AND a.status = 'processed'::"AssetStatus" + AND a.is_deleted = false + AND sk.is_archived = false + AND ${ + method === 'not_updated' + ? Prisma.sql`a.updated_at <= ${cutoffDate}` + : Prisma.sql`a.created_at <= ${cutoffDate} AND a.updated_at <= ${restoreCutoffDate}` + } + AND NOT EXISTS ( + SELECT 1 + FROM assets sib + WHERE sib.storage_key_id = a.storage_key_id + AND sib.is_deleted = false + AND ( + sib.type != 'file'::"AssetType" + OR sib.status != 'processed'::"AssetStatus" + OR ${ + method === 'not_updated' + ? Prisma.sql`sib.updated_at > ${cutoffDate}` + : Prisma.sql`sib.created_at > ${cutoffDate} OR sib.updated_at > ${restoreCutoffDate}` + } + ) + ) + ORDER BY a.id ASC + LIMIT ${batchSize}; + ` + + if (candidateRows.length > 0) { + batches.push({ + teamId: team.id, + assetIds: candidateRows.map((r) => r.id), + policy: { method, cutoffDate, restoreCutoffDate }, + }) + } + } + + return batches + }) + + if (!teamBatches || teamBatches.length === 0) { + return { archivedCount: 0 } + } + + logger.info('Auto-archive job started') + + // Phase 2: Execute archival in chunks per team batch. + // Each asset/storageKey transition is protected by row-level lock and re-verifies policy. + const chunkSize = 50 + + for (const batch of teamBatches) { + for (let i = 0; i < batch.assetIds.length; i += chunkSize) { + const chunk = batch.assetIds.slice(i, i + chunkSize) + const result = await assetArchiveService.archiveAssets(chunk, { + suppressBusyErrors: true, + autoArchivePolicy: batch.policy, + }) + totalArchived += result.archivedAssetIds.length + } + } + + logger.info({ totalArchived }, 'Auto-archive job finished') + return { archivedCount: totalArchived } + } finally { + this.isProcessing = false + } + } +} + +export const autoArchiveJobService = new AutoArchiveJobService() diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 5820b3c3..49a3d25e 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -1,6 +1,7 @@ export * from './agent/agent' export * from './asset/asset' export * from './asset/asset-archive' +export * from './asset/auto-archive-job' export * from './auth/auth' export * from './authz/authz' export * from './collection/collection' diff --git a/packages/core/src/team/team.test.ts b/packages/core/src/team/team.test.ts index 3d79b183..02ee0c94 100644 --- a/packages/core/src/team/team.test.ts +++ b/packages/core/src/team/team.test.ts @@ -1,3 +1,4 @@ +import * as path from 'path' import { prisma } from '@shumai/db' import { setupTestDbHooks } from '@shumai/db/test' import { teamService } from '@shumai/core/src/team/team' @@ -388,6 +389,13 @@ describe('TeamService', () => { username: undefined, }, appearance: { hideAgent: false }, + archiveStorage: { enabled: false }, + autoArchive: { + enabled: false, + method: 'not_updated', + days: 30, + postRestoreGraceDays: 30, + }, }) await teamService.updateSettings(team.id, 'theme', 'dark') @@ -417,6 +425,13 @@ describe('TeamService', () => { username: undefined, }, appearance: { hideAgent: false }, + archiveStorage: { enabled: false }, + autoArchive: { + enabled: false, + method: 'not_updated', + days: 30, + postRestoreGraceDays: 30, + }, }) await teamService.updateSettings(team.id, 'transcode.videoStrategy', 'multi') @@ -456,6 +471,13 @@ describe('TeamService', () => { }, appearance: { hideAgent: true }, semanticSearchEnabled: false, + archiveStorage: { enabled: false }, + autoArchive: { + enabled: false, + method: 'not_updated', + days: 30, + postRestoreGraceDays: 30, + }, }) }) @@ -633,4 +655,66 @@ describe('TeamService', () => { delete process.env.ENABLE_EMBEDDING_AGENT } }) + + it('rejects enabling auto-archive when archive storage is not configured', async () => { + delete process.env.ARCHIVE_STORAGE_BACKEND + const owner = await prisma.user.create({ + data: { name: 'Owner AA', email: 'owner-aa@example.com', password: 'pw' }, + }) + const team = await teamService.createTeam(owner, { name: 'AA Team' }) + + await expect( + teamService.updateSettings(team.id, 'autoArchive', { + enabled: true, + method: 'not_updated', + days: 30, + postRestoreGraceDays: 30, + }), + ).rejects.toThrow('Cannot enable auto archive when archive storage is not configured') + + await expect(teamService.updateSettings(team.id, 'autoArchive.enabled', true)).rejects.toThrow( + 'Cannot enable auto archive when archive storage is not configured', + ) + }) + + it('allows updating auto-archive settings when archive storage is configured', async () => { + const saved = { ...process.env } + process.env.ARCHIVE_STORAGE_BACKEND = 'local' + process.env.ARCHIVE_LOCAL_ROOT = '/tmp/shumai-archive-test-team' + + try { + const owner = await prisma.user.create({ + data: { name: 'Owner AA 2', email: 'owner-aa2@example.com', password: 'pw' }, + }) + const team = await teamService.createTeam(owner, { name: 'AA Team 2' }) + + await teamService.updateSettings(team.id, 'autoArchive', { + enabled: true, + method: 'after_created', + days: 45, + postRestoreGraceDays: 15, + }) + + const settings = await teamService.getSettings(team.id) + expect(settings.autoArchive).toEqual({ + enabled: true, + method: 'after_created', + days: 45, + postRestoreGraceDays: 15, + }) + expect(settings.archiveStorage).toEqual({ + enabled: true, + backend: 'local', + localRoot: path.resolve('/tmp/shumai-archive-test-team'), + }) + + // Non-admin call should mask backend/root details + const readerSettings = await teamService.getSettings(team.id, { isAdmin: false }) + expect(readerSettings.archiveStorage).toEqual({ + enabled: true, + }) + } finally { + process.env = saved + } + }) }) diff --git a/packages/core/src/team/team.ts b/packages/core/src/team/team.ts index 131afed1..6990b3d4 100644 --- a/packages/core/src/team/team.ts +++ b/packages/core/src/team/team.ts @@ -10,6 +10,10 @@ import { AgentService, agentService } from '@shumai/core/src/agent/agent' import { sandboxService } from '@shumai/core/src/sandbox/sandbox-service' import { turntableService } from '@shumai/core/src/turntable/turntable' import { gotenbergService } from '@shumai/core/src/gotenberg/gotenberg' +import { + getArchiveStorageConfig, + isArchiveStorageEnabled, +} from '@shumai/core/src/s3/archive-storage' import { ServiceCreateTeamRequest, ServiceGetUserTeamsRequest, @@ -23,6 +27,9 @@ import { SandboxSettings, UpdateSandboxSettingsRequest, TeamSettingsResponse, + AutoArchiveMethod, + AutoArchiveSettings, + DEFAULT_AUTO_ARCHIVE_SETTINGS, } from '@shumai/dtos' export class TeamService { @@ -300,7 +307,10 @@ export class TeamService { }) } - async getSettings(teamId: string): Promise { + async getSettings( + teamId: string, + options?: { isAdmin?: boolean }, + ): Promise { const team = await prisma.team.findUnique({ where: { id: teamId }, }) @@ -355,6 +365,33 @@ export class TeamService { settings.turntable = await turntableService.getSettings(teamId) settings.gotenberg = await gotenbergService.getSettings(teamId) + const archiveConfig = getArchiveStorageConfig() + if (!archiveConfig.enabled) { + settings.archiveStorage = { enabled: false } + } else if (options?.isAdmin === false) { + settings.archiveStorage = { enabled: true } + } else if (archiveConfig.backend === 'local') { + settings.archiveStorage = { + enabled: true, + backend: 'local', + localRoot: archiveConfig.localRoot, + } + } else { + settings.archiveStorage = { + enabled: true, + backend: 's3', + endpoint: archiveConfig.endpoint, + bucket: archiveConfig.bucket, + } + } + + settings.autoArchive = { + enabled: settings.autoArchive?.enabled ?? false, + method: settings.autoArchive?.method ?? 'not_updated', + days: settings.autoArchive?.days ?? 30, + postRestoreGraceDays: settings.autoArchive?.postRestoreGraceDays ?? 30, + } + return settings } @@ -421,6 +458,49 @@ export class TeamService { } settings.appearance.hideAgent = Boolean(value) delete settings['appearance.hideAgent'] + } else if (key === 'autoArchive') { + const autoArchiveVal = value as AutoArchiveSettings + if (autoArchiveVal.enabled && !isArchiveStorageEnabled()) { + throw new HTTPException(400, { + message: 'Cannot enable auto archive when archive storage is not configured', + }) + } + settings.autoArchive = { + enabled: Boolean(autoArchiveVal.enabled), + method: autoArchiveVal.method || 'not_updated', + days: Number(autoArchiveVal.days) || 30, + postRestoreGraceDays: Number(autoArchiveVal.postRestoreGraceDays) || 30, + } + } else if (key === 'autoArchive.enabled') { + const enabled = Boolean(value) + if (enabled && !isArchiveStorageEnabled()) { + throw new HTTPException(400, { + message: 'Cannot enable auto archive when archive storage is not configured', + }) + } + if (!settings.autoArchive) { + settings.autoArchive = { ...DEFAULT_AUTO_ARCHIVE_SETTINGS } + } + settings.autoArchive.enabled = enabled + delete settings['autoArchive.enabled'] + } else if (key === 'autoArchive.method') { + if (!settings.autoArchive) { + settings.autoArchive = { ...DEFAULT_AUTO_ARCHIVE_SETTINGS } + } + settings.autoArchive.method = value as AutoArchiveMethod + delete settings['autoArchive.method'] + } else if (key === 'autoArchive.days') { + if (!settings.autoArchive) { + settings.autoArchive = { ...DEFAULT_AUTO_ARCHIVE_SETTINGS } + } + settings.autoArchive.days = Number(value) + delete settings['autoArchive.days'] + } else if (key === 'autoArchive.postRestoreGraceDays') { + if (!settings.autoArchive) { + settings.autoArchive = { ...DEFAULT_AUTO_ARCHIVE_SETTINGS } + } + settings.autoArchive.postRestoreGraceDays = Number(value) + delete settings['autoArchive.postRestoreGraceDays'] } else { settings[key] = value } diff --git a/packages/db/src/prisma-json-types.ts b/packages/db/src/prisma-json-types.ts index c9cad020..bffe9ebb 100644 --- a/packages/db/src/prisma-json-types.ts +++ b/packages/db/src/prisma-json-types.ts @@ -122,6 +122,7 @@ declare global { mediaGeneration?: MediaGenerationSettings appearance?: AppearanceSettings emailNotification?: EmailNotificationSettings + autoArchive?: import('@shumai/dtos').AutoArchiveSettings } // ---------------------------------------------------------------------- diff --git a/packages/dtos/src/team.ts b/packages/dtos/src/team.ts index 8770a6b5..b61e9318 100644 --- a/packages/dtos/src/team.ts +++ b/packages/dtos/src/team.ts @@ -110,6 +110,32 @@ export type HlsResolutionLadder = (typeof HlsResolutionLadder)[keyof typeof HlsR export const DEFAULT_HLS_RESOLUTIONS: HlsResolutionLadder[] = ['480p', '720p', '1080p'] +export const autoArchiveMethodSchema = z.enum(['not_updated', 'after_created']) +export type AutoArchiveMethod = z.infer + +export const autoArchiveSettingsSchema = z.object({ + enabled: z.boolean(), + method: autoArchiveMethodSchema, + days: z.number().int().min(1).max(3650), + postRestoreGraceDays: z.number().int().min(1).max(3650).default(30), +}) +export type AutoArchiveSettings = z.infer + +export const DEFAULT_AUTO_ARCHIVE_SETTINGS: AutoArchiveSettings = { + enabled: false, + method: 'not_updated', + days: 30, + postRestoreGraceDays: 30, +} + +export interface ArchiveStorageInfo { + enabled: boolean + backend?: 'local' | 's3' + endpoint?: string + bucket?: string + localRoot?: string +} + export const updateTeamSettingsRequestSchema = z.union([ z.object({ key: z.literal('transcode.videoStrategy'), @@ -139,6 +165,26 @@ export const updateTeamSettingsRequestSchema = z.union([ key: z.literal('appearance.hideAgent'), value: z.boolean(), }), + z.object({ + key: z.literal('autoArchive'), + value: autoArchiveSettingsSchema, + }), + z.object({ + key: z.literal('autoArchive.enabled'), + value: z.boolean(), + }), + z.object({ + key: z.literal('autoArchive.method'), + value: autoArchiveMethodSchema, + }), + z.object({ + key: z.literal('autoArchive.days'), + value: z.number().int().min(1).max(3650), + }), + z.object({ + key: z.literal('autoArchive.postRestoreGraceDays'), + value: z.number().int().min(1).max(3650), + }), ]) export type UpdateTeamSettingsRequest = z.infer @@ -189,6 +235,8 @@ export interface TeamSettingsResponse { gotenberg?: GotenbergSettingsResponse appearance?: TeamAppearanceSettings semanticSearchEnabled?: boolean + archiveStorage?: ArchiveStorageInfo + autoArchive?: AutoArchiveSettings [key: string]: unknown } diff --git a/packages/transcode/src/activities/archive.ts b/packages/transcode/src/activities/archive.ts index 7aa0902d..b984036d 100644 --- a/packages/transcode/src/activities/archive.ts +++ b/packages/transcode/src/activities/archive.ts @@ -489,7 +489,7 @@ export const commitRestoreDbStateActivity = defineStep( type: 'file', status: AssetStatus.restoring, }, - data: { status: AssetStatus.processed }, + data: { status: AssetStatus.processed, updatedAt: new Date() }, }), ]) }, diff --git a/packages/webui/components/settings/StorageSettings.test.tsx b/packages/webui/components/settings/StorageSettings.test.tsx new file mode 100644 index 00000000..96da2416 --- /dev/null +++ b/packages/webui/components/settings/StorageSettings.test.tsx @@ -0,0 +1,234 @@ +// @vitest-environment happy-dom +import { cleanup, render, screen, fireEvent, waitFor } from '@testing-library/react' +import React from 'react' +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { QueryClient, QueryClientProvider } from '@tanstack/react-query' +import { StorageSettings } from './StorageSettings' +import { toast } from 'sonner' + +const mockGetSettings = vi.fn() +const mockPatchSettings = vi.fn() + +vi.mock('@/ui/api/client', () => ({ + client: { + api: { + teams: { + ':teamId': { + settings: { + $get: () => mockGetSettings(), + $patch: (args: unknown) => mockPatchSettings(args), + }, + }, + }, + }, + }, +})) + +vi.mock('sonner', () => ({ + toast: { + success: vi.fn(), + error: vi.fn(), + }, +})) + +describe('StorageSettings', () => { + let queryClient: QueryClient + + beforeEach(() => { + vi.clearAllMocks() + queryClient = new QueryClient({ + defaultOptions: { + queries: { retry: false }, + }, + }) + mockGetSettings.mockResolvedValue({ + ok: true, + json: async () => ({ + archiveStorage: { + enabled: true, + backend: 's3', + bucket: 'my-archive-bucket', + endpoint: 'https://s3.custom.com', + }, + autoArchive: { + enabled: false, + method: 'not_updated', + days: 30, + postRestoreGraceDays: 30, + }, + }), + }) + mockPatchSettings.mockResolvedValue({ + ok: true, + json: async () => ({ + autoArchive: { + enabled: true, + method: 'not_updated', + days: 30, + postRestoreGraceDays: 30, + }, + }), + }) + }) + + afterEach(() => { + cleanup() + }) + + const renderComponent = () => + render( + + + , + ) + + it('renders archive storage details for S3 backend when configured', async () => { + renderComponent() + + await waitFor(() => { + expect(screen.getByText('my-archive-bucket')).toBeDefined() + expect(screen.getByText('https://s3.custom.com')).toBeDefined() + }) + }) + + it('renders archive storage details for local backend', async () => { + mockGetSettings.mockResolvedValueOnce({ + ok: true, + json: async () => ({ + archiveStorage: { + enabled: true, + backend: 'local', + localRoot: '/var/data/archive', + }, + autoArchive: { + enabled: false, + method: 'not_updated', + days: 30, + postRestoreGraceDays: 30, + }, + }), + }) + + renderComponent() + + await waitFor(() => { + expect(screen.getByText('/var/data/archive')).toBeDefined() + }) + }) + + it('renders warning alert and disables auto-archive form when archive storage is not configured', async () => { + mockGetSettings.mockResolvedValueOnce({ + ok: true, + json: async () => ({ + archiveStorage: { + enabled: false, + }, + autoArchive: { + enabled: false, + method: 'not_updated', + days: 30, + postRestoreGraceDays: 30, + }, + }), + }) + + renderComponent() + + await waitFor(() => { + const switchElement = screen.getByRole('switch') + expect(switchElement.hasAttribute('disabled')).toBe(true) + }) + }) + + it('submits updated auto-archive configuration when Save is clicked', async () => { + renderComponent() + + const switchElement = await screen.findByRole('switch') + expect(switchElement.getAttribute('data-state')).toBe('unchecked') + fireEvent.click(switchElement) + + const daysInput = await screen.findByLabelText(/Age Threshold/i) + fireEvent.change(daysInput, { target: { value: '60' } }) + + const saveButton = screen.getByRole('button', { name: /Save/i }) + fireEvent.click(saveButton) + + await waitFor(() => { + expect(mockPatchSettings).toHaveBeenCalledWith({ + param: { teamId: 'team-storage-123' }, + json: { + key: 'autoArchive', + value: { + enabled: true, + method: 'not_updated', + days: 60, + postRestoreGraceDays: 30, + }, + }, + }) + expect(toast.success).toHaveBeenCalled() + }) + }) + + it('handles error when saving auto-archive configuration fails', async () => { + mockPatchSettings.mockResolvedValueOnce({ + ok: false, + json: async () => ({ message: 'Cannot save settings' }), + }) + + renderComponent() + + const saveButton = await screen.findByRole('button', { name: /Save/i }) + fireEvent.click(saveButton) + + await waitFor(() => { + expect(toast.error).toHaveBeenCalledWith('Cannot save settings') + }) + }) + + it('allows clearing the days input and clamps value on blur', async () => { + renderComponent() + + const daysInput = (await screen.findByLabelText(/Age Threshold/i)) as HTMLInputElement + expect(daysInput.value).toBe('30') + + // User clears the field + fireEvent.change(daysInput, { target: { value: '' } }) + expect(daysInput.value).toBe('') + + // User types new value + fireEvent.change(daysInput, { target: { value: '45' } }) + expect(daysInput.value).toBe('45') + + // Blur clamps if out of range + fireEvent.change(daysInput, { target: { value: '99999' } }) + fireEvent.blur(daysInput) + expect(daysInput.value).toBe('3650') + }) + + it('renders error state and retries on button click', async () => { + mockGetSettings.mockResolvedValueOnce({ + ok: false, + }) + + renderComponent() + + await waitFor(() => { + expect(screen.getByRole('button', { name: /Retry/i })).toBeDefined() + }) + + mockGetSettings.mockResolvedValueOnce({ + ok: true, + json: async () => ({ + archiveStorage: { enabled: true, backend: 'local', localRoot: '/tmp/retry-path' }, + autoArchive: { enabled: false, method: 'not_updated', days: 30, postRestoreGraceDays: 30 }, + }), + }) + + fireEvent.click(screen.getByRole('button', { name: /Retry/i })) + + await waitFor(() => { + expect(screen.getByText('/tmp/retry-path')).toBeDefined() + }) + }) +}) diff --git a/packages/webui/components/settings/StorageSettings.tsx b/packages/webui/components/settings/StorageSettings.tsx new file mode 100644 index 00000000..ba6f2f44 --- /dev/null +++ b/packages/webui/components/settings/StorageSettings.tsx @@ -0,0 +1,358 @@ +import React, { useState, useEffect } from 'react' +import { useQuery, useMutation, useQueryClient } from '@tanstack/react-query' +import { client } from '@/ui/api/client' +import { + Card, + CardHeader, + CardTitle, + CardDescription, + CardContent, + CardFooter, +} from '@/ui/components/ui/card' +import { Input } from '@/ui/components/ui/input' +import { Button } from '@/ui/components/ui/button' +import { Badge } from '@/ui/components/ui/badge' +import { Switch } from '@/ui/components/ui/switch' +import { + Select, + SelectContent, + SelectItem, + SelectTrigger, + SelectValue, +} from '@/ui/components/ui/select' +import { Loader2, HardDrive, CheckCircle2, AlertTriangle, Info, AlertCircle } from 'lucide-react' +import { toast } from 'sonner' +import { m } from '@/ui/paraglide/messages.js' +import type { TeamSettingsResponse, AutoArchiveMethod } from '@shumai/dtos' + +interface StorageSettingsProps { + teamId: string +} + +function clampDays(val: string | number, defaultVal = 30): number { + const parsed = typeof val === 'number' ? val : parseInt(val, 10) + if (isNaN(parsed)) return defaultVal + return Math.min(3650, Math.max(1, parsed)) +} + +export const StorageSettings: React.FC = ({ teamId }) => { + const queryClient = useQueryClient() + + const { + data: settings, + isLoading, + isError, + error, + refetch, + } = useQuery({ + queryKey: ['teams', teamId, 'settings'], + queryFn: async () => { + const res = await client.api.teams[':teamId'].settings.$get({ + param: { teamId }, + }) + if (!res.ok) throw new Error(m.failed_load_settings()) + return (await res.json()) as TeamSettingsResponse + }, + enabled: !!teamId, + }) + + const [enabled, setEnabled] = useState(false) + const [method, setMethod] = useState('not_updated') + const [daysInput, setDaysInput] = useState('30') + const [postRestoreGraceDaysInput, setPostRestoreGraceDaysInput] = useState('30') + const [isDirty, setIsDirty] = useState(false) + + useEffect(() => { + if (settings?.autoArchive && !isDirty) { + setEnabled(settings.autoArchive.enabled ?? false) + setMethod(settings.autoArchive.method ?? 'not_updated') + setDaysInput(String(settings.autoArchive.days ?? 30)) + setPostRestoreGraceDaysInput(String(settings.autoArchive.postRestoreGraceDays ?? 30)) + } + }, [settings?.autoArchive, isDirty]) + + const archiveStorage = settings?.archiveStorage + const isArchiveConfigured = Boolean(archiveStorage?.enabled) + + const { mutate: saveAutoArchiveSettings, isPending: isSaving } = useMutation({ + mutationFn: async () => { + const days = clampDays(daysInput, 30) + const postRestoreGraceDays = clampDays(postRestoreGraceDaysInput, 30) + + const res = await client.api.teams[':teamId'].settings.$patch({ + param: { teamId }, + json: { + key: 'autoArchive', + value: { + enabled, + method, + days, + postRestoreGraceDays, + }, + }, + }) + if (!res.ok) { + const err = (await res.json().catch(() => null)) as { message?: string } | null + throw new Error(err?.message || m.auto_archive_settings_save_failed()) + } + return await res.json() + }, + onSuccess: () => { + setIsDirty(false) + queryClient.invalidateQueries({ + queryKey: ['teams', teamId, 'settings'], + }) + toast.success(m.auto_archive_settings_saved()) + }, + onError: (err: unknown) => { + toast.error(err instanceof Error ? err.message : m.auto_archive_settings_save_failed()) + }, + }) + + if (isLoading) { + return ( +
+ +
+ ) + } + + if (isError) { + return ( +
+ + + + + {m.failed_to_load_storage_settings()} + + + {error instanceof Error ? error.message : m.failed_load_settings()} + + + + + + +
+ ) + } + + return ( +
+ {/* Read-only Archive Storage Config Card */} + + +
+
+ + + {m.archive_storage_config_card()} + + {m.archive_storage_config_card_description()} +
+ {isArchiveConfigured ? ( + + + {m.archive_storage_configured()} + + ) : ( + + + {m.archive_storage_not_configured()} + + )} +
+
+ + {!isArchiveConfigured ? ( +
+ +
+

{m.archive_storage_not_configured()}

+

+ {m.archive_storage_not_configured_alert()} +

+
+
+ ) : ( +
+
+ +

+ {archiveStorage?.backend === 's3' + ? m.archive_storage_backend_s3() + : m.archive_storage_backend_local()} +

+
+ + {archiveStorage?.backend === 's3' ? ( + <> +
+ +

{archiveStorage?.bucket || '-'}

+
+
+ +

+ {archiveStorage?.endpoint || m.archive_storage_default_endpoint()} +

+
+ + ) : ( +
+ +

+ {archiveStorage?.localRoot || '-'} +

+
+ )} +
+ )} +
+
+ + {/* Auto Archive Config Card */} + + + {m.auto_archive_card()} + {m.auto_archive_card_description()} + + + {!isArchiveConfigured && ( +
+ + {m.auto_archive_disabled_hint()} +
+ )} + + {/* Toggle Auto Archive */} +
+
+ +

{m.enable_auto_archive_description()}

+
+ { + setIsDirty(true) + setEnabled(val) + }} + disabled={!isArchiveConfigured || isSaving} + /> +
+ + {/* Method and Days Configuration */} +
+
+ + +

{m.auto_archive_method_description()}

+
+ +
+ + { + setIsDirty(true) + setDaysInput(e.target.value) + }} + onBlur={() => { + setDaysInput(String(clampDays(daysInput, 30))) + }} + disabled={!isArchiveConfigured || isSaving} + className="w-full md:w-64" + /> +

{m.auto_archive_days_description()}

+
+ + {method === 'after_created' && ( +
+ + { + setIsDirty(true) + setPostRestoreGraceDaysInput(e.target.value) + }} + onBlur={() => { + setPostRestoreGraceDaysInput(String(clampDays(postRestoreGraceDaysInput, 30))) + }} + disabled={!isArchiveConfigured || isSaving} + className="w-full md:w-64" + /> +

+ {m.auto_archive_post_restore_grace_days_description()} +

+
+ )} +
+
+ + + +
+
+ ) +} diff --git a/packages/webui/messages/en.json b/packages/webui/messages/en.json index ddb75b87..e749479f 100644 --- a/packages/webui/messages/en.json +++ b/packages/webui/messages/en.json @@ -274,6 +274,38 @@ "turntable_testing": "Testing...", "integrations": "Integrations", "integrations_settings_description": "Configure third-party service integrations for media rendering and document processing.", + "storage": "Storage", + "storage_settings": "Storage Settings", + "storage_settings_description": "View archive storage status and configure automatic asset archival policies.", + "archive_storage_config_card": "Archive Storage Configuration", + "archive_storage_config_card_description": "Current system archive storage backend details configured through server environment variables.", + "archive_storage_configured": "Configured", + "archive_storage_not_configured": "Not Configured", + "archive_storage_backend": "Storage Backend", + "archive_storage_backend_s3": "Amazon S3 / S3-compatible", + "archive_storage_backend_local": "Local File System", + "archive_storage_endpoint": "Endpoint", + "archive_storage_bucket": "Bucket", + "archive_storage_local_root": "Root Path", + "archive_storage_default_endpoint": "Default AWS S3", + "archive_storage_not_configured_alert": "Archive storage is not configured in environment variables (ARCHIVE_STORAGE_BACKEND). Automatic archival cannot be enabled until archive storage is set.", + "auto_archive_card": "Automatic Asset Archival", + "auto_archive_card_description": "Automatically transfer aging files to archive storage to optimize primary storage usage.", + "enable_auto_archive": "Enable Auto Archive", + "enable_auto_archive_description": "Automatically scan and archive assets in this team that meet the age criteria.", + "auto_archive_method": "Archive Criteria", + "auto_archive_method_description": "Choose whether asset age is evaluated from creation time or last modification time.", + "auto_archive_method_not_updated": "Not modified for specified days", + "auto_archive_method_after_created": "After creation for specified days", + "auto_archive_days": "Age Threshold (days)", + "auto_archive_days_description": "Assets older than this number of days will be moved to archive storage.", + "auto_archive_post_restore_grace_days": "Post-Restore Grace Period (days)", + "auto_archive_post_restore_grace_days_description": "After an asset is restored from archive, wait at least this many days before allowing it to be auto-archived again. This should be smaller than the age threshold; only assets that exceed the age threshold and have also passed this grace period since restoration will be auto-archived.", + "auto_archive_disabled_hint": "Archive storage must be configured on the server before auto archive can be enabled.", + "auto_archive_settings_saved": "Auto archive settings saved successfully", + "auto_archive_settings_save_failed": "Failed to save auto archive settings", + "failed_to_load_storage_settings": "Failed to load storage settings", + "retry": "Retry", "gotenberg_settings": "Gotenberg Document Converter", "gotenberg_settings_description": "Configure the external Gotenberg service for converting Office documents, Markdown, and HTML to PDF.", "gotenberg_server_url": "Server URL", diff --git a/packages/webui/messages/zh.json b/packages/webui/messages/zh.json index 34abbacf..998c277f 100644 --- a/packages/webui/messages/zh.json +++ b/packages/webui/messages/zh.json @@ -274,6 +274,38 @@ "turntable_testing": "正在测试...", "integrations": "集成", "integrations_settings_description": "配置用于媒体渲染与文档处理的第三方服务集成。", + "storage": "存储", + "storage_settings": "存储设置", + "storage_settings_description": "查看归档存储状态并配置资产自动归档策略。", + "archive_storage_config_card": "归档存储配置", + "archive_storage_config_card_description": "通过服务端环境变量配置的当前系统归档存储后端信息。", + "archive_storage_configured": "已配置", + "archive_storage_not_configured": "未配置", + "archive_storage_backend": "存储后端", + "archive_storage_backend_s3": "Amazon S3 / S3 兼容存储", + "archive_storage_backend_local": "本地文件系统", + "archive_storage_endpoint": "终端节点", + "archive_storage_bucket": "存储桶", + "archive_storage_local_root": "根路径", + "archive_storage_default_endpoint": "默认 AWS S3", + "archive_storage_not_configured_alert": "服务端环境变量中未配置归档存储(ARCHIVE_STORAGE_BACKEND)。在配置归档存储前无法启用自动归档。", + "auto_archive_card": "资产自动归档", + "auto_archive_card_description": "自动将达到一定时间的原始文件转移至归档存储,以优化主存储空间。", + "enable_auto_archive": "启用自动归档", + "enable_auto_archive_description": "自动扫描并归档本团队中符合时间条件的资产。", + "auto_archive_method": "归档判定条件", + "auto_archive_method_description": "选择根据文件的创建时间还是最近修改时间来判定资产年龄。", + "auto_archive_method_not_updated": "指定天数内未修改", + "auto_archive_method_after_created": "创建后达到指定天数", + "auto_archive_days": "时间阈值(天)", + "auto_archive_days_description": "超出此天数的资产将被转移至归档存储。", + "auto_archive_post_restore_grace_days": "恢复后宽限期(天)", + "auto_archive_post_restore_grace_days_description": "从归档恢复资产后,至少等待指定天数后才允许再次被自动归档。此值应小于时间阈值,实际被自动归档的资产需同时满足创建时间超过时间阈值且恢复后已超出宽限期。", + "auto_archive_disabled_hint": "必须在服务端配置归档存储后才能启用自动归档。", + "auto_archive_settings_saved": "自动归档设置保存成功", + "auto_archive_settings_save_failed": "保存自动归档设置失败", + "failed_to_load_storage_settings": "加载存储设置失败", + "retry": "重试", "gotenberg_settings": "Gotenberg 文档转换服务", "gotenberg_settings_description": "配置用于将 Office 文档、Markdown 及 HTML 转换为 PDF 的外部 Gotenberg 服务。", "gotenberg_server_url": "服务地址", diff --git a/packages/webui/routes/teams/$teamId/settings.lazy.tsx b/packages/webui/routes/teams/$teamId/settings.lazy.tsx index 7b60334b..664bac4d 100644 --- a/packages/webui/routes/teams/$teamId/settings.lazy.tsx +++ b/packages/webui/routes/teams/$teamId/settings.lazy.tsx @@ -25,6 +25,7 @@ import { EmailNotificationSettings } from '@/ui/components/settings/EmailNotific import { ImageVideoGenerationSettings } from '@/ui/components/settings/ImageVideoGenerationSettings' import { TurntableSettingsCard } from '@/ui/components/settings/TurntableSettingsCard' import { GotenbergSettingsCard } from '@/ui/components/settings/GotenbergSettingsCard' +import { StorageSettings } from '@/ui/components/settings/StorageSettings' import { Card, CardContent, CardDescription, CardHeader, CardTitle } from '@/ui/components/ui/card' import { Slider } from '@/ui/components/ui/slider' import { cn } from '@/ui/lib/utils' @@ -45,6 +46,7 @@ import { Key, Gauge, Palette, + HardDrive, } from 'lucide-react' import { useState, useRef, useEffect } from 'react' import { Avatar, AvatarFallback, AvatarImage } from '@/ui/components/ui/avatar' @@ -70,6 +72,7 @@ type SettingsTab = | 'appearance' | 'email' | 'integrations' + | 'storage' | 'skills' | 'mcp' | 'providers' @@ -86,6 +89,7 @@ const VALID_SETTINGS_TABS: readonly SettingsTab[] = [ 'appearance', 'email', 'integrations', + 'storage', 'skills', 'mcp', 'providers', @@ -301,6 +305,8 @@ export function TeamSettingsPage() { 'quotas', 'appearance', 'email', + 'integrations', + 'storage', 'providers', 'image-video', 'skills', @@ -667,6 +673,22 @@ export function TeamSettingsPage() {
)} + + )} @@ -789,6 +811,7 @@ export function TeamSettingsPage() { {activeTab === 'appearance' && m.appearance_settings()} {activeTab === 'email' && m.email_notifications()} {activeTab === 'integrations' && m.integrations()} + {activeTab === 'storage' && m.storage_settings()} {activeTab === 'skills' && m.skills_management()} {activeTab === 'mcp' && m.mcp_servers()} {activeTab === 'providers' && m.ai_providers()} @@ -805,6 +828,7 @@ export function TeamSettingsPage() { {activeTab === 'appearance' && m.appearance_settings_description()} {activeTab === 'email' && m.email_notifications_description()} {activeTab === 'integrations' && m.integrations_settings_description()} + {activeTab === 'storage' && m.storage_settings_description()} {activeTab === 'skills' && m.skills_description()} {activeTab === 'mcp' && m.mcp_servers_description()} {activeTab === 'providers' && m.providers_description()} @@ -1217,6 +1241,8 @@ export function TeamSettingsPage() {
)} + + {activeTab === 'storage' && } diff --git a/packages/webui/routes/teams/$teamId/settings.test.tsx b/packages/webui/routes/teams/$teamId/settings.test.tsx index db2c3c31..f96e5243 100644 --- a/packages/webui/routes/teams/$teamId/settings.test.tsx +++ b/packages/webui/routes/teams/$teamId/settings.test.tsx @@ -556,4 +556,22 @@ describe('TeamSettingsPage Transcode Settings', () => { // Redundant "Transcode Settings" card title should not be rendered expect(screen.queryByRole('heading', { level: 3, name: 'Transcode Settings' })).toBeNull() }) + + it('redirects non-owners attempting to access storage tab to general', async () => { + window.location.hash = '#storage' + mockGetMe.mockResolvedValueOnce({ + ok: true, + json: async () => ({ role: 'member' }), + }) + + render( + + + , + ) + + await waitFor(() => { + expect(screen.queryByText('Storage Settings')).toBeNull() + }) + }) })