Compare commits

..
8 Commits
Author SHA1 Message Date
yiling 49ef48a74a Release v0.3.5 2026-08-09 20:58:35 +08:00
Yiling-J a680fccb12 fix(webui): enable rename and delete actions in file detail top bar breadcrumb (#288)
* fix(webui): enable rename and delete actions in file detail top bar breadcrumb

* fix(webui): select only base file name when renaming from file detail breadcrumb
2026-08-09 20:49:55 +08:00
Yiling-J e67552d8bd fix(webui): batch download no longer flashes a tab per file (#287)
startDownload() used <a target="_blank" download> anchors, which made
Chrome open a new tab for every file that then auto-closed as soon as the
download started (Content-Disposition: attachment). Replace with hidden
iframes: downloads start in the background without opening tabs, and a
failed/expired link renders inside the hidden iframe instead of
navigating the SPA tab away.
2026-08-09 17:09:01 +08:00
Yiling-J 8080acfb2e fix: use renamed asset name in download Content-Disposition (#286)
* fix(core): use renamed asset name in download Content-Disposition

Downloads of renamed assets (e.g. test.png renamed to foo.png) previously
saved under the storage-key basename because the presigned URLs / local
file handler only set Content-Disposition: attachment without a filename,
and the HTML download attribute is ignored for cross-origin URLs.

- Add buildContentDisposition() helper that emits a sanitized
  'attachment; filename=...; filename*=UTF-8...' header (RFC 5987) so
  non-ASCII names survive.
- Thread an optional filename through S3Service.presign():
  * S3 backend passes it via contentDisposition (response-content-disposition)
  * Local backend carries it as a ?filename= query param consumed by the
    /files serveStatic onFound handler
- getDownloadLinks / getDownloadUrl now pass the renamed asset name from
  the DB, so all download flows (file browser, viewers, breadcrumb, share)
  save files under the current asset name.

* fix(core): keep Content-Disposition header ASCII for non-ASCII names

Bun's Headers API rejects header values containing non-ASCII bytes, so a
download of a renamed file with a non-ASCII name (e.g. 幻境边界.pptx) threw
'TypeError: Header Content-Disposition has invalid value' on the local
storage backend.

Per RFC 5987, non-ASCII names belong only in the percent-encoded
filename* parameter. buildContentDisposition now emits the quoted
filename="..." fallback only when the name is pure ASCII; otherwise the
header carries just 'attachment; filename*=UTF-8''...' (all-ASCII, accepted
by Bun/Node Headers and all modern browsers).
2026-08-09 16:39:15 +08:00
Yiling-J dbd30a815f fix(webui): exclude file extension when selecting file name during inline rename (#285) 2026-08-09 16:06:44 +08:00
Yiling-J 442568de00 fix(webui): resolve upload card freeze for unsupported files (#284) 2026-08-09 15:16:41 +08:00
yiling d15fc5c6be update doc 2026-08-09 14:49:07 +08:00
Yiling-J aa6d5ed626 feat(transcode): add Gotenberg support for Office, HTML, CSV, and Markdown PDF proxies (#283)
* feat(transcode): add Gotenberg support for Office, HTML, CSV, and Markdown PDF proxies

* fix(transcode): use Gotenberg chromium markdown endpoint with index.html wrapper for markdown conversion

* test(transcode): add complete unit test coverage for HTML, CSV landscape/portrait, TXT, and Gotenberg fallback branches

* fix(gotenberg): force .md extension for Gotenberg markdown conversion endpoint compatibility

* refactor(gotenberg): standardize basic auth environment variables on GOTENBERG_BASIC_AUTH_USERNAME and GOTENBERG_BASIC_AUTH_PASSWORD

* test(e2e): add real Gotenberg testcontainers workflow E2E test for docx, pptx, and xlsx

* docs: add Gotenberg document PDF conversion documentation and update features in README

* docs(gotenberg): add Docker quickstart commands and remove obsolete GOTENBERG_USERNAME note
2026-08-09 14:44:13 +08:00
50 changed files with 2027 additions and 117 deletions
+1
View File
@@ -26,6 +26,7 @@ Shumai is an open-source Frame.io alternative built for creative teams to collab
* **Custom Asset Metadata**: Define dynamic metadata fields tailored to your production pipeline.
* **Share Link Watermarking**: Protect shared media with customizable text and image watermarks, and save watermark as reusable templates.
* **Wide File Format Support**: Preview common creative file formats, including images (with PSD support), videos, audio, and documents, while allowing any file type to be uploaded and stored.
* **Office & Document PDF Proxying via Gotenberg**: Convert Word, PowerPoint, Excel, Markdown, HTML, and CSV documents to high-quality PDF proxies for in-browser previewing using [Gotenberg](https://docs.shumai.one/using-shumai/gotenberg).
But Shumai goes beyond being a Frame.io alternative. With modern models like GPT-Image-2 and Seedance 2.5, AI can already handle a surprising amount of creative work. That means you can create or update videos and images from a simple prompt without having to jump back into Photoshop, Premiere or Houdini. That is why Shumai includes a powerful agent system:
+1
View File
@@ -27,6 +27,7 @@ Shumai 是一个开源的 Frame.io 替代方案,专为创意团队打造,让
* **自定义素材元数据**:可以按你的制作流程,自由定义动态元数据字段。
* **分享链接水印**:通过可自定义的文字和图片水印保护共享媒体,并将水印配置保存为模板以便重复使用。
* **丰富的文件格式支持**:支持预览常见创意文件格式,包括图片(支持 PSD)、视频、音频和文档,同时可上传和存储任意类型的文件。
* **基于 Gotenberg 的文档 PDF 转码代理**:通过集成 [Gotenberg](https://docs.shumai.one/using-shumai/gotenberg) 支持将 Word、PowerPoint、Excel、Markdown、HTML 和 CSV 文档高质量转换为 PDF 代理文件进行在线预览。
但 Shumai 不仅仅只满足于 Frame.io 的开源替代。随着 GPT-Image-2、Seedance 2.5 这类强大模型的问世,AI已经生成产品级的图片/视频。很多时候,你只要给一句提示词,就能直接生成或修改图片和视频,而不用再打开 Photoshop、Premiere 或者 Houdini。也正因为这样,Shumai 才把 AI Agent 做成了核心能力:
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/agent-app",
"version": "0.3.4",
"version": "0.3.5",
"private": true,
"type": "module",
"bin": {
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/cli",
"version": "0.3.4",
"version": "0.3.5",
"private": true,
"type": "module",
"bin": {
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/transcode-app",
"version": "0.3.4",
"version": "0.3.5",
"private": true,
"type": "module",
"bin": {
+1 -1
View File
@@ -68,7 +68,7 @@ apps/web/e2e/
| Test | File |
|---|---|
| Owner creates a project with a cover image | `tests/project/create-project.spec.ts` |
| Owner uploads a file via the "Upload File" context menu action | `tests/project/upload-file.spec.ts` |
| Owner uploads a file or an unsupported file via the "Upload File" context menu action | `tests/project/upload-file.spec.ts` |
| Owner deletes a file, finds it in recently deleted, and restores it | `tests/project/delete-restore.spec.ts` |
| Owner creates a share link from a file context menu and lands on the share page | `tests/project/create-share-link.spec.ts` |
| Owner protects a share with a password; guests must enter it to view | `tests/project/share-password.spec.ts` |
@@ -47,3 +47,39 @@ test('owner uploads a file via the "Upload File" context menu action', async ({
const projectRow = await prisma.project.findUnique({ where: { id: projectId } })
expect(asset!.parentId).toBe(projectRow?.rootFolderId)
})
test('owner uploads an unsupported file via context menu action', async ({ project, prisma }) => {
const { page, projectId } = project
const fileName = `e2e-upload-${Date.now()}.bin`
const binBuffer = Buffer.from([0x00, 0x01, 0x02, 0x03])
await page.goto(`/projects/${projectId}`)
await expect(page.getByText('This folder is empty')).toBeVisible()
await page.getByText('This folder is empty').click({ button: 'right' })
const [fileChooser] = await Promise.all([
page.waitForEvent('filechooser'),
page.getByRole('menuitem', { name: 'Upload File' }).click(),
])
await fileChooser.setFiles({
name: fileName,
mimeType: 'application/octet-stream',
buffer: binBuffer,
})
const card = fileCard(page, fileName)
await expect(card).toBeVisible()
await expect
.poll(
async () => {
const asset = await prisma.asset.findFirst({ where: { name: fileName } })
return asset?.status
},
{ timeout: 30_000 },
)
.toBe('processed')
// Verify that the file card shows the generic file icon and is not stuck in uploading state
await expect(card.locator('svg.lucide-file')).toBeVisible({ timeout: 10_000 })
})
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/web-app",
"version": "0.3.4",
"version": "0.3.5",
"private": true,
"type": "module",
"dependencies": {
+2 -1
View File
@@ -34,7 +34,8 @@
"using-shumai/notifications",
"using-shumai/metadata",
"using-shumai/collections",
"using-shumai/share-links"
"using-shumai/share-links",
"using-shumai/gotenberg"
]
},
{
+82
View File
@@ -0,0 +1,82 @@
---
title: Document PDF Conversion with Gotenberg
description: Convert Office documents, HTML, Markdown, CSV, and TXT files into high-fidelity PDF proxies for in-browser previewing.
---
Office documents, HTML pages, Markdown files, spreadsheets, and plain-text files cannot be rendered natively in web browsers. Shumai uses [Gotenberg](https://gotenberg.dev), an open-source, Docker powered conversion API — to turn these files into PDF proxy assets for seamless in-browser previewing.
Gotenberg performs the conversion using:
- **LibreOffice** for Office documents, spreadsheets, and plain text.
- **Headless Chromium** for HTML and Markdown files.
---
## Supported File Types
When Gotenberg is configured, Shumai automatically generates a PDF proxy for the following uploads:
| Category | Extensions | Converter |
| ------------------ | ---------------------- | ----------- |
| Word | `.doc`, `.docx` | LibreOffice |
| Excel | `.xls`, `.xlsx` | LibreOffice |
| PowerPoint | `.ppt`, `.pptx` | LibreOffice |
| OpenDocument | `.odt`, `.ods`, `.odp` | LibreOffice |
| Rich Text | `.rtf` | LibreOffice |
| HTML | `.html`, `.htm` | Chromium |
| Markdown | `.md`, `.markdown` | Chromium |
| Spreadsheets / CSV | `.csv` | LibreOffice |
| Plain text | `.txt` | LibreOffice |
---
## Configuration
Gotenberg support is opt-in and controlled by the following environment variables on the machine running the transcode workers:
| Variable | Required | Description |
| ------------------------------- | --------------- | ---------------------------------------------------------------- |
| `GOTENBERG_URL` | Yes (to enable) | Base URL of the Gotenberg service, e.g. `http://gotenberg:3000`. |
| `GOTENBERG_BASIC_AUTH_USERNAME` | No | Username when Basic Authentication is enabled on Gotenberg. |
| `GOTENBERG_BASIC_AUTH_PASSWORD` | No | Password when Basic Authentication is enabled on Gotenberg. |
### Docker Quickstart
You can start a Gotenberg container with a single command:
```bash
docker run -d --name gotenberg -p 3000:3000 gotenberg/gotenberg:8
```
If you wish to enable Basic Authentication:
```bash
docker run -d --name gotenberg -p 3000:3000 gotenberg/gotenberg:8 gotenberg --gotenberg-basic-auth-username=admin --gotenberg-basic-auth-password=secret
```
### Docker Compose Setup
Add a Gotenberg service to your `docker-compose.yaml`:
```yaml
gotenberg:
image: gotenberg/gotenberg:8
restart: unless-stopped
ports:
- '3000:3000'
```
Then point Shumai at it:
```bash
GOTENBERG_URL=http://localhost:3000
```
---
## Behavior Without Gotenberg
If Gotenberg is not configured or unreachable, Shumai degrades gracefully:
- **Office documents and HTML**: the upload completes as `processed` without a PDF proxy.
- **CSV, Markdown, and TXT**: a lightweight built-in PDF generator produces the proxy instead, with reduced fidelity.
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "shumai",
"version": "0.3.4",
"version": "0.3.5",
"private": true,
"type": "module",
"repository": {
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/agent",
"version": "0.3.4",
"version": "0.3.5",
"private": true,
"type": "module",
"main": "./src/index.ts",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/api",
"version": "0.3.4",
"version": "0.3.5",
"private": true,
"type": "module",
"main": "./src/index.ts",
+21 -4
View File
@@ -39,7 +39,7 @@ describe('S3 API', () => {
}
})
it('GET /files/* sets Content-Disposition: attachment only when download=1 query param is present', async () => {
it('GET /files/* sets Content-Disposition with filename when download=1 & filename query params are present', async () => {
// We need to capture the onFound callback from serveStatic options
// eslint-disable-next-line @typescript-eslint/no-explicit-any
let onFoundCallback: ((path: string, c: any) => void) | undefined
@@ -54,19 +54,36 @@ describe('S3 API', () => {
const { default: route } = await import('./s3')
const app = new Hono().route('/files', route)
await app.request('/files/b1/test.webp?download=1')
await app.request('/files/b1/test.webp?download=1&filename=foo.png')
expect(onFoundCallback).toBeDefined()
// Test with download=1
// Test with download=1 and filename
const mockContextWithDownload = {
req: { query: (key: string) => (key === 'download' ? '1' : undefined) },
req: {
query: (key: string) =>
key === 'download' ? '1' : key === 'filename' ? 'foo.png' : undefined,
},
header: vi.fn(),
}
if (onFoundCallback) {
onFoundCallback('/files/b1/test.webp', mockContextWithDownload)
expect(mockContextWithDownload.header).toHaveBeenCalledWith(
'Content-Disposition',
'attachment; filename="foo.png"; filename*=UTF-8\'\'foo.png',
)
}
// Test with download=1 but no filename
const mockContextDownloadNoFilename = {
req: { query: (key: string) => (key === 'download' ? '1' : undefined) },
header: vi.fn(),
}
if (onFoundCallback) {
onFoundCallback('/files/b1/test.webp', mockContextDownloadNoFilename)
expect(mockContextDownloadNoFilename.header).toHaveBeenCalledWith(
'Content-Disposition',
'attachment',
)
+2 -2
View File
@@ -1,5 +1,5 @@
import { Hono } from 'hono'
import { s3Service } from '@shumai/core/src/s3/s3'
import { buildContentDisposition, s3Service } from '@shumai/core/src/s3/s3'
import { serveStatic } from 'hono/bun'
const route = new Hono()
@@ -10,7 +10,7 @@ const route = new Hono()
rewriteRequestPath: (path) => path.replace(/^\/files\//, ''),
onFound: (_path, c) => {
if (c.req.query('download') === '1') {
c.header('Content-Disposition', 'attachment')
c.header('Content-Disposition', buildContentDisposition(c.req.query('filename')))
}
},
}),
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/core",
"version": "0.3.4",
"version": "0.3.5",
"private": true,
"type": "module",
"main": "./src/index.ts",
+47
View File
@@ -4,6 +4,7 @@ import { setupTestDbHooks } from '@shumai/db/test'
import { HTTPException } from 'hono/http-exception'
import { AssetType, AssetStatus, Prisma } from '@shumai/db'
import { s3Service } from '@shumai/core/src/s3/s3'
import { AssetService } from './asset'
vi.mock('@shumai/core/src/s3/s3', () => ({
@@ -2229,6 +2230,18 @@ describe('AssetService — natural sort by name', () => {
for (const item of result) {
expect(item.url).toBe('http://mock-s3-url')
}
// Renamed file names must be passed to presign so the download
// Content-Disposition header uses the asset name, not the storage key
for (const name of ['file1.txt', 'file2.txt', 'version2.txt']) {
expect(s3Service.presign).toHaveBeenCalledWith(
expect.any(String),
expect.any(String),
'GET',
true,
name,
)
}
})
it('resolves top-level symlinks to their targets when passed as direct starting IDs', async () => {
@@ -2539,6 +2552,40 @@ describe('AssetService — natural sort by name', () => {
)
expect(url).toBe('http://mock-s3-url')
expect(s3Service.presign).toHaveBeenCalledWith(
expect.any(String),
'files/01TESTULID000000000000000/test-720p.mp4',
'GET',
true,
'test.mp4',
)
})
it('should use the renamed asset name as the download filename', async () => {
const asset = await prisma.asset.create({
data: {
name: 'foo.png',
type: AssetType.file,
status: AssetStatus.uploaded,
storageKey: {
create: { key: 'files/01RENAMEDULID0000000000000/test.png' },
},
},
})
const url = await assetService.getDownloadUrl(
asset.id,
'files/01RENAMEDULID0000000000000/test.png',
)
expect(url).toBe('http://mock-s3-url')
expect(s3Service.presign).toHaveBeenCalledWith(
expect.any(String),
'files/01RENAMEDULID0000000000000/test.png',
'GET',
true,
'foo.png',
)
})
it('should return a presigned download URL for the original key', async () => {
+15 -3
View File
@@ -1966,7 +1966,13 @@ export class AssetService {
const key = file.storageKeyId ? storageKeyMap.get(file.storageKeyId) : null
if (!key) return null
const url = await s3Service.presign(process.env.S3_BUCKET || 'shumai', key, 'GET', true)
const url = await s3Service.presign(
process.env.S3_BUCKET || 'shumai',
key,
'GET',
true,
file.name,
)
return {
id: file.id,
@@ -1988,7 +1994,7 @@ export class AssetService {
const targetAssetId = await this.resolveLatestVersionId(assetId)
const asset = await this.prismaClient.asset.findUnique({
where: { id: targetAssetId },
select: { id: true, parentId: true, storageKey: { select: { key: true } } },
select: { id: true, name: true, parentId: true, storageKey: { select: { key: true } } },
})
if (!asset?.storageKey?.key) {
@@ -2022,7 +2028,13 @@ export class AssetService {
throw new Error('Key does not belong to this asset')
}
return s3Service.presign(process.env.S3_BUCKET || 'shumai', key, 'GET', true)
return s3Service.presign(
process.env.S3_BUCKET || 'shumai',
key,
'GET',
true,
asset.name ?? undefined,
)
}
private async toPreviewInfo(asset: Asset | AssetWithIncludes): Promise<PreviewInfo | null> {
@@ -0,0 +1,219 @@
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'
import * as fs from 'fs'
import * as path from 'path'
import * as os from 'os'
import { GotenbergService } from './gotenberg'
describe('GotenbergService', () => {
const originalEnv = process.env
let tmpDir: string
let testFilePath: string
beforeEach(() => {
process.env = { ...originalEnv }
tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gotenberg-test-'))
testFilePath = path.join(tmpDir, 'test.docx')
fs.writeFileSync(testFilePath, 'fake docx content')
})
afterEach(() => {
process.env = originalEnv
fs.rmSync(tmpDir, { recursive: true, force: true })
vi.restoreAllMocks()
})
describe('isAvailable', () => {
it('should return false if GOTENBERG_URL is not set', async () => {
delete process.env.GOTENBERG_URL
const service = new GotenbergService()
expect(await service.isAvailable()).toBe(false)
})
it('should return false if GOTENBERG_URL is empty', async () => {
process.env.GOTENBERG_URL = ' '
const service = new GotenbergService()
expect(await service.isAvailable()).toBe(false)
})
it('should return true when health check returns 200 OK', async () => {
process.env.GOTENBERG_URL = 'http://gotenberg:3000'
const fetchSpy = vi.spyOn(globalThis, 'fetch').mockResolvedValue({
ok: true,
status: 200,
} as Response)
const service = new GotenbergService()
const available = await service.isAvailable()
expect(available).toBe(true)
expect(fetchSpy).toHaveBeenCalledWith(
'http://gotenberg:3000/health',
expect.objectContaining({
method: 'GET',
}),
)
})
it('should include Basic Auth header if username and password are provided', async () => {
process.env.GOTENBERG_URL = 'http://gotenberg:3000'
process.env.GOTENBERG_BASIC_AUTH_USERNAME = 'user'
process.env.GOTENBERG_BASIC_AUTH_PASSWORD = 'pass'
const fetchSpy = vi.spyOn(globalThis, 'fetch').mockResolvedValue({
ok: true,
status: 200,
} as Response)
const service = new GotenbergService()
await service.isAvailable()
const expectedBasic = Buffer.from('user:pass').toString('base64')
expect(fetchSpy).toHaveBeenCalledWith(
'http://gotenberg:3000/health',
expect.objectContaining({
headers: {
Authorization: `Basic ${expectedBasic}`,
},
}),
)
})
it('should return false when fetch fails', async () => {
process.env.GOTENBERG_URL = 'http://gotenberg:3000'
vi.spyOn(globalThis, 'fetch').mockRejectedValue(new Error('Network error'))
const service = new GotenbergService()
expect(await service.isAvailable()).toBe(false)
})
})
describe('convertDocumentToPdf', () => {
it('should throw error if GOTENBERG_URL is not set', async () => {
delete process.env.GOTENBERG_URL
const service = new GotenbergService()
await expect(service.convertDocumentToPdf(testFilePath, 'test.docx')).rejects.toThrow(
'Gotenberg URL is not configured',
)
})
it('should call libreoffice convert endpoint and return buffer', async () => {
process.env.GOTENBERG_URL = 'http://gotenberg:3000/'
const mockPdfBuffer = Buffer.from('%PDF-1.7 mock output')
const fetchSpy = vi.spyOn(globalThis, 'fetch').mockResolvedValue({
ok: true,
status: 200,
arrayBuffer: async () => mockPdfBuffer.buffer,
} as unknown as Response)
const service = new GotenbergService()
const result = await service.convertDocumentToPdf(testFilePath, 'test.docx', {
landscape: true,
})
expect(result).toBeDefined()
expect(result.toString()).toContain('%PDF-1.7')
expect(fetchSpy).toHaveBeenCalledWith(
'http://gotenberg:3000/forms/libreoffice/convert',
expect.objectContaining({
method: 'POST',
body: expect.any(FormData),
}),
)
})
it('should throw error if gotenberg returns non-200', async () => {
process.env.GOTENBERG_URL = 'http://gotenberg:3000'
vi.spyOn(globalThis, 'fetch').mockResolvedValue({
ok: false,
status: 500,
statusText: 'Internal Server Error',
text: async () => 'Conversion failed',
} as unknown as Response)
const service = new GotenbergService()
await expect(service.convertDocumentToPdf(testFilePath, 'test.docx')).rejects.toThrow(
'Gotenberg conversion failed with status 500: Conversion failed',
)
})
})
describe('convertHtmlToPdf', () => {
it('should call chromium html convert endpoint and return buffer', async () => {
process.env.GOTENBERG_URL = 'http://gotenberg:3000'
const mockPdfBuffer = Buffer.from('%PDF-1.4 html output')
const fetchSpy = vi.spyOn(globalThis, 'fetch').mockResolvedValue({
ok: true,
status: 200,
arrayBuffer: async () => mockPdfBuffer.buffer,
} as unknown as Response)
const htmlPath = path.join(tmpDir, 'test.html')
fs.writeFileSync(htmlPath, '<h1>Hello</h1>')
const service = new GotenbergService()
const result = await service.convertHtmlToPdf(htmlPath, 'test.html')
expect(result).toBeDefined()
expect(fetchSpy).toHaveBeenCalledWith(
'http://gotenberg:3000/forms/chromium/convert/html',
expect.objectContaining({
method: 'POST',
}),
)
})
})
describe('convertMarkdownToPdf', () => {
it('should call chromium markdown convert endpoint with index.html wrapper and return buffer', async () => {
process.env.GOTENBERG_URL = 'http://gotenberg:3000'
const mockPdfBuffer = Buffer.from('%PDF-1.4 markdown output')
const fetchSpy = vi.spyOn(globalThis, 'fetch').mockResolvedValue({
ok: true,
status: 200,
arrayBuffer: async () => mockPdfBuffer.buffer,
} as unknown as Response)
const mdPath = path.join(tmpDir, 'test.md')
fs.writeFileSync(mdPath, '# Hello Gotenberg')
const service = new GotenbergService()
const result = await service.convertMarkdownToPdf(mdPath, 'test.md')
expect(result).toBeDefined()
expect(fetchSpy).toHaveBeenCalledWith(
'http://gotenberg:3000/forms/chromium/convert/markdown',
expect.objectContaining({
method: 'POST',
body: expect.any(FormData),
}),
)
})
it('should rename non-.md filenames like README.markdown to file.md for Gotenberg endpoint compatibility', async () => {
process.env.GOTENBERG_URL = 'http://gotenberg:3000'
const mockPdfBuffer = Buffer.from('%PDF-1.4 markdown output')
let capturedFormData: FormData | null = null
vi.spyOn(globalThis, 'fetch').mockImplementation(async (_url, options) => {
capturedFormData = options?.body as FormData
return {
ok: true,
status: 200,
arrayBuffer: async () => mockPdfBuffer.buffer,
} as unknown as Response
})
const mdPath = path.join(tmpDir, 'README.markdown')
fs.writeFileSync(mdPath, '# Hello Gotenberg')
const service = new GotenbergService()
await service.convertMarkdownToPdf(mdPath, 'README.markdown')
expect(capturedFormData).not.toBeNull()
const indexBlob = (capturedFormData as unknown as FormData).get('files') as File
const indexHtmlText = await indexBlob.text()
expect(indexHtmlText).toContain('{{ toHTML "file.md" }}')
})
})
})
+178
View File
@@ -0,0 +1,178 @@
import * as fs from 'fs'
export interface GotenbergConvertOptions {
landscape?: boolean
}
export class GotenbergService {
private getUrl(): string | undefined {
const url = process.env.GOTENBERG_URL?.trim()
return url ? url.replace(/\/+$/, '') : undefined
}
private getAuthHeader(): string | undefined {
const username = process.env.GOTENBERG_BASIC_AUTH_USERNAME
const password = process.env.GOTENBERG_BASIC_AUTH_PASSWORD
if (username && password) {
const credentials = Buffer.from(`${username}:${password}`).toString('base64')
return `Basic ${credentials}`
}
return undefined
}
async isAvailable(): Promise<boolean> {
const baseUrl = this.getUrl()
if (!baseUrl) {
return false
}
try {
const headers: Record<string, string> = {}
const authHeader = this.getAuthHeader()
if (authHeader) {
headers['Authorization'] = authHeader
}
const response = await fetch(`${baseUrl}/health`, {
method: 'GET',
headers,
signal: AbortSignal.timeout(3000),
})
return response.ok
} catch {
return false
}
}
async convertDocumentToPdf(
filePath: string,
filename: string,
options: GotenbergConvertOptions = {},
): Promise<Buffer> {
const baseUrl = this.getUrl()
if (!baseUrl) {
throw new Error('Gotenberg URL is not configured.')
}
const fileBytes = fs.readFileSync(filePath)
const formData = new FormData()
formData.append('files', new Blob([fileBytes]), filename)
if (options.landscape) {
formData.append('landscape', 'true')
}
const headers: Record<string, string> = {}
const authHeader = this.getAuthHeader()
if (authHeader) {
headers['Authorization'] = authHeader
}
const response = await fetch(`${baseUrl}/forms/libreoffice/convert`, {
method: 'POST',
headers,
body: formData,
signal: AbortSignal.timeout(60000),
})
if (!response.ok) {
const errorText = await response.text().catch(() => '')
throw new Error(
`Gotenberg conversion failed with status ${response.status}: ${errorText || response.statusText}`,
)
}
const arrayBuffer = await response.arrayBuffer()
return Buffer.from(arrayBuffer)
}
async convertHtmlToPdf(filePath: string, filename: string): Promise<Buffer> {
const baseUrl = this.getUrl()
if (!baseUrl) {
throw new Error('Gotenberg URL is not configured.')
}
const fileBytes = fs.readFileSync(filePath)
const formData = new FormData()
const targetFilename =
filename.toLowerCase().endsWith('.html') || filename.toLowerCase().endsWith('.htm')
? filename
: 'index.html'
formData.append('files', new Blob([fileBytes]), targetFilename)
const headers: Record<string, string> = {}
const authHeader = this.getAuthHeader()
if (authHeader) {
headers['Authorization'] = authHeader
}
const response = await fetch(`${baseUrl}/forms/chromium/convert/html`, {
method: 'POST',
headers,
body: formData,
signal: AbortSignal.timeout(60000),
})
if (!response.ok) {
const errorText = await response.text().catch(() => '')
throw new Error(
`Gotenberg HTML conversion failed with status ${response.status}: ${errorText || response.statusText}`,
)
}
const arrayBuffer = await response.arrayBuffer()
return Buffer.from(arrayBuffer)
}
async convertMarkdownToPdf(filePath: string, filename: string): Promise<Buffer> {
const baseUrl = this.getUrl()
if (!baseUrl) {
throw new Error('Gotenberg URL is not configured.')
}
const mdFilename = filename.toLowerCase().endsWith('.md') ? filename : 'file.md'
const htmlTemplate = `<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<title>PDF</title>
</head>
<body>
{{ toHTML "${mdFilename}" }}
</body>
</html>`
const fileBytes = fs.readFileSync(filePath)
const formData = new FormData()
formData.append('files', new Blob([htmlTemplate], { type: 'text/html' }), 'index.html')
formData.append('files', new Blob([fileBytes]), mdFilename)
const headers: Record<string, string> = {}
const authHeader = this.getAuthHeader()
if (authHeader) {
headers['Authorization'] = authHeader
}
const response = await fetch(`${baseUrl}/forms/chromium/convert/markdown`, {
method: 'POST',
headers,
body: formData,
signal: AbortSignal.timeout(60000),
})
if (!response.ok) {
const errorText = await response.text().catch(() => '')
throw new Error(
`Gotenberg Markdown conversion failed with status ${response.status}: ${errorText || response.statusText}`,
)
}
const arrayBuffer = await response.arrayBuffer()
return Buffer.from(arrayBuffer)
}
}
export const gotenbergService = new GotenbergService()
+1
View File
@@ -24,3 +24,4 @@ export * from './pagination'
export * from './chat/chat'
export * from './ai-usage/ai-usage'
export * from './watermark/watermark'
export * from './gotenberg/gotenberg'
+81 -1
View File
@@ -1,7 +1,7 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import * as fs from 'fs'
import * as path from 'path'
import { LocalStorageService, S3StorageService } from './s3'
import { buildContentDisposition, LocalStorageService, S3StorageService } from './s3'
// Mock Bun S3Client
const s3ClientConstructorSpy = vi.fn()
@@ -140,6 +140,32 @@ describe('S3Service implementations', () => {
)
})
it('should set contentDisposition with the filename when download filename is provided', async () => {
const s3 = new S3StorageService('http://localhost:9000', 'key', 'secret', 'test-bucket')
await s3.presign('bucket', 'key', 'GET', true, 'foo.png')
expect(s3PresignSpy).toHaveBeenCalledWith(
'key',
expect.objectContaining({
contentDisposition: 'attachment; filename="foo.png"; filename*=UTF-8\'\'foo.png',
}),
)
})
it('should include an RFC 5987 filename* for non-ASCII filenames', async () => {
const s3 = new S3StorageService('http://localhost:9000', 'key', 'secret', 'test-bucket')
await s3.presign('bucket', 'key', 'GET', true, '报告.png')
expect(s3PresignSpy).toHaveBeenCalledWith(
'key',
expect.objectContaining({
// Non-ASCII names are conveyed only via filename* (percent-encoded),
// keeping the header value pure ASCII (RFC 7230).
contentDisposition: "attachment; filename*=UTF-8''%E6%8A%A5%E5%91%8A.png",
}),
)
})
it('should not cache download presign URLs', async () => {
const s3 = new S3StorageService('http://localhost:9000', 'key', 'secret', 'test-bucket')
@@ -248,6 +274,18 @@ describe('S3Service implementations', () => {
expect(url).toBe('http://localhost:3000/files/my-bucket/dir/file.txt?download=1')
})
it('should include an encoded filename in the local download URL', async () => {
const url = await localS3.presign('my-bucket', 'dir/file.txt', 'GET', true, 'foo bar.png')
expect(url).toBe(
'http://localhost:3000/files/my-bucket/dir/file.txt?download=1&filename=foo%20bar.png',
)
})
it('should not append a filename param when none is provided', async () => {
const url = await localS3.presign('my-bucket', 'dir/file.txt', 'GET', true)
expect(url).not.toContain('filename=')
})
it('should throw an error for unsupported presign methods', async () => {
await expect(localS3.presign('b', 'k', 'POST')).rejects.toThrow()
})
@@ -287,4 +325,46 @@ describe('S3Service implementations', () => {
expect(url).not.toContain('4567')
})
})
describe('buildContentDisposition', () => {
it('returns plain attachment when no filename is provided', () => {
expect(buildContentDisposition()).toBe('attachment')
expect(buildContentDisposition('')).toBe('attachment')
expect(buildContentDisposition(null)).toBe('attachment')
})
it('builds a disposition with quoted filename and RFC 5987 filename*', () => {
expect(buildContentDisposition('foo.png')).toBe(
'attachment; filename="foo.png"; filename*=UTF-8\'\'foo.png',
)
})
it('strips control characters and quotes to prevent header injection', () => {
expect(buildContentDisposition('a\r\nb"c\\d')).toBe(
'attachment; filename="abcd"; filename*=UTF-8\'\'abcd',
)
})
it('percent-encodes reserved RFC 5987 characters', () => {
expect(buildContentDisposition("it's (final).png")).toBe(
"attachment; filename=\"it's (final).png\"; filename*=UTF-8''it%27s%20%28final%29.png",
)
})
it('percent-encodes the * character in filename*', () => {
expect(buildContentDisposition('foo*bar.png')).toBe(
'attachment; filename="foo*bar.png"; filename*=UTF-8\'\'foo%2Abar.png',
)
})
it('omits the quoted filename fallback for non-ASCII names to keep the header ASCII', () => {
expect(buildContentDisposition('幻境边界.pptx')).toBe(
"attachment; filename*=UTF-8''%E5%B9%BB%E5%A2%83%E8%BE%B9%E7%95%8C.pptx",
)
// The whole header value must remain pure ASCII so Bun/Node Headers accept it
expect(
[...buildContentDisposition('幻境边界.pptx')].every((ch) => ch.charCodeAt(0) <= 0x7f),
).toBe(true)
})
})
})
+57 -4
View File
@@ -27,6 +27,38 @@ export function verifyLocalUrlSignature(bucket: string, key: string, signature:
}
}
/**
* Build a Content-Disposition header value for downloads.
* - Falls back to plain `attachment` when no filename is provided.
* - Strips control characters (CR/LF header injection), quotes, and backslashes.
* - Header values must be ASCII (RFC 7230): non-ASCII names are conveyed only
* via the percent-encoded RFC 5987 `filename*` parameter; the quoted
* `filename="..."` fallback is included only when the name is pure ASCII.
*/
export function buildContentDisposition(filename?: string | null): string {
if (!filename) return 'attachment'
// Strip C0 control characters (incl. CR/LF header injection) and DEL, then
// quotes/backslashes which would break the quoted-string form.
const sanitized = [...filename]
.filter((ch) => {
const code = ch.charCodeAt(0)
return code > 0x1f && code !== 0x7f && ch !== '"' && ch !== '\\'
})
.join('')
.trim()
if (!sanitized) return 'attachment'
// RFC 5987 attr-char excludes ' ( ) * — percent-encode them explicitly
const encoded = encodeURIComponent(sanitized).replace(
/['()*]/g,
(c) => `%${c.charCodeAt(0).toString(16).toUpperCase()}`,
)
const isAscii = [...sanitized].every((ch) => ch.charCodeAt(0) <= 0x7f)
if (!isAscii) {
return `attachment; filename*=UTF-8''${encoded}`
}
return `attachment; filename="${sanitized}"; filename*=UTF-8''${encoded}`
}
export interface S3Object {
buffer: Buffer
contentType: string
@@ -55,7 +87,13 @@ export interface S3Service {
listObjects: (bucket: string, prefix: string) => Promise<string[]>
uploadFile: (filePath: string, contentType: string) => Promise<string>
uploadFileToKey: (filePath: string, key: string, contentType: string) => Promise<void>
presign: (bucket: string, key: string, method: string, download?: boolean) => Promise<string>
presign: (
bucket: string,
key: string,
method: string,
download?: boolean,
filename?: string,
) => Promise<string>
}
export class S3StorageService implements S3Service {
@@ -214,7 +252,13 @@ export class S3StorageService implements S3Service {
})
}
async presign(bucket: string, key: string, method: string, download?: boolean): Promise<string> {
async presign(
bucket: string,
key: string,
method: string,
download?: boolean,
filename?: string,
): Promise<string> {
const expireHours = parseInt(process.env.PRESIGNED_URL_EXPIRES_IN || '5', 10)
const expiresInSeconds = expireHours * 3600
// Cache time is 2/3 of expire time, rounded to minute
@@ -237,7 +281,7 @@ export class S3StorageService implements S3Service {
}
if (download) {
options.contentDisposition = 'attachment'
options.contentDisposition = buildContentDisposition(filename)
}
const url = this.client.presign(key, options)
@@ -461,7 +505,13 @@ export class LocalStorageService implements S3Service {
await fs.promises.copyFile(filePath, destPath)
}
async presign(bucket: string, key: string, method: string, download?: boolean): Promise<string> {
async presign(
bucket: string,
key: string,
method: string,
download?: boolean,
filename?: string,
): Promise<string> {
if (method !== 'GET' && method !== 'PUT') {
throw new Error(`Invalid method: ${method}`)
}
@@ -471,6 +521,9 @@ export class LocalStorageService implements S3Service {
let url = `${this.endpoint}/files/${bucket}/${key}`
if (download) {
url += '?download=1'
if (filename) {
url += `&filename=${encodeURIComponent(filename)}`
}
}
return url
}
+127
View File
@@ -2,6 +2,7 @@ import { describe, expect, it, vi, beforeEach } from 'vitest'
import { prisma } from '@shumai/db'
import { setupTestDbHooks } from '@shumai/db/test'
import { uploadService } from './upload'
import { gotenbergService } from '@shumai/core/src/gotenberg/gotenberg'
import { s3Service } from '@shumai/core/src/s3/s3'
import { AssetStatus, AssetType, WorkflowTaskType } from '@shumai/db'
@@ -262,6 +263,132 @@ describe('UploadService', () => {
expect(workflowTask).toBeNull()
})
it('should confirm file upload and create transcode_pdf for office document when Gotenberg is available', async () => {
vi.spyOn(gotenbergService, 'isAvailable').mockResolvedValue(true)
const task = await prisma.task.create({
data: { creatorId: userId, total: 1, uploaded: 0, type: 'upload' },
})
const asset = await prisma.asset.create({
data: {
name: 'document.docx',
type: AssetType.file,
project: { connect: { id: projectId } },
parent: { connect: { id: parentId } },
status: AssetStatus.uploading,
storageKey: {
connectOrCreate: {
where: { key: 'test-key' },
create: { key: 'test-key' },
},
},
mediaType: 'application/vnd.openxmlformats-officedocument.wordprocessingml.document',
},
})
await uploadService.confirmFileUpload(userId, task.id, { fileId: asset.id })
const workflowTask = await prisma.workflowTask.findFirst({
where: { assetId: asset.id, type: WorkflowTaskType.transcode_pdf },
})
expect(workflowTask).toBeDefined()
})
it('should mark asset processed without creating transcode_pdf for office document when Gotenberg is unavailable', async () => {
vi.spyOn(gotenbergService, 'isAvailable').mockResolvedValue(false)
const task = await prisma.task.create({
data: { creatorId: userId, total: 1, uploaded: 0, type: 'upload' },
})
const asset = await prisma.asset.create({
data: {
name: 'document.docx',
type: AssetType.file,
project: { connect: { id: projectId } },
parent: { connect: { id: parentId } },
status: AssetStatus.uploading,
storageKey: {
connectOrCreate: {
where: { key: 'test-key' },
create: { key: 'test-key' },
},
},
mediaType: 'application/vnd.openxmlformats-officedocument.wordprocessingml.document',
},
})
await uploadService.confirmFileUpload(userId, task.id, { fileId: asset.id })
const updatedAsset = await prisma.asset.findUnique({ where: { id: asset.id } })
expect(updatedAsset?.status).toBe(AssetStatus.processed)
const workflowTask = await prisma.workflowTask.findFirst({
where: { assetId: asset.id, type: WorkflowTaskType.transcode_pdf },
})
expect(workflowTask).toBeNull()
})
it('should confirm file upload and create transcode_pdf for HTML document when Gotenberg is available', async () => {
vi.spyOn(gotenbergService, 'isAvailable').mockResolvedValue(true)
const task = await prisma.task.create({
data: { creatorId: userId, total: 1, uploaded: 0, type: 'upload' },
})
const asset = await prisma.asset.create({
data: {
name: 'index.html',
type: AssetType.file,
project: { connect: { id: projectId } },
parent: { connect: { id: parentId } },
status: AssetStatus.uploading,
storageKey: {
connectOrCreate: {
where: { key: 'test-key-html-1' },
create: { key: 'test-key-html-1' },
},
},
mediaType: 'text/html',
},
})
await uploadService.confirmFileUpload(userId, task.id, { fileId: asset.id })
const workflowTask = await prisma.workflowTask.findFirst({
where: { assetId: asset.id, type: WorkflowTaskType.transcode_pdf },
})
expect(workflowTask).toBeDefined()
})
it('should mark asset processed without creating transcode_pdf for HTML document when Gotenberg is unavailable', async () => {
vi.spyOn(gotenbergService, 'isAvailable').mockResolvedValue(false)
const task = await prisma.task.create({
data: { creatorId: userId, total: 1, uploaded: 0, type: 'upload' },
})
const asset = await prisma.asset.create({
data: {
name: 'index.html',
type: AssetType.file,
project: { connect: { id: projectId } },
parent: { connect: { id: parentId } },
status: AssetStatus.uploading,
storageKey: {
connectOrCreate: {
where: { key: 'test-key-html-2' },
create: { key: 'test-key-html-2' },
},
},
mediaType: 'text/html',
},
})
await uploadService.confirmFileUpload(userId, task.id, { fileId: asset.id })
const updatedAsset = await prisma.asset.findUnique({ where: { id: asset.id } })
expect(updatedAsset?.status).toBe(AssetStatus.processed)
const workflowTask = await prisma.workflowTask.findFirst({
where: { assetId: asset.id, type: WorkflowTaskType.transcode_pdf },
})
expect(workflowTask).toBeNull()
})
it('should confirm file upload and resolve empty mediaType using Bun resolver for audio', async () => {
const task = await prisma.task.create({
data: { creatorId: userId, total: 1, uploaded: 0, type: 'upload' },
+16 -1
View File
@@ -13,8 +13,9 @@ import {
import { ImageTranscoder, PdfTranscoder, VideoTranscoder } from '@shumai/transcode'
import { generateKeyBetween } from 'jittered-fractional-indexing'
import { ulid } from 'ulid'
import { gotenbergService } from '@shumai/core/src/gotenberg/gotenberg'
import { sanitizeFilename } from '@shumai/core/src/utils/filename'
import { getProxyType } from '@shumai/core/src/utils/mime'
import { getProxyType, isHtmlDocument, isOfficeDocument } from '@shumai/core/src/utils/mime'
export class UploadService {
constructor(private readonly prismaClient: typeof prisma = prisma) {}
@@ -315,6 +316,20 @@ export class UploadService {
// eslint-disable-next-line @typescript-eslint/no-explicit-any
await new ImageTranscoder(tx as any, asset.id, team.id, projectId).withThumbnail().submit()
} else if (isPdf) {
const isOffice = isOfficeDocument(asset.mediaType, asset.name)
const isHtml = isHtmlDocument(asset.mediaType, asset.name)
if (isOffice || isHtml) {
const gotenbergAvailable = await gotenbergService.isAvailable()
if (!gotenbergAvailable) {
await tx.asset.update({
where: { id: asset.id },
data: { status: AssetStatus.processed },
})
return
}
}
// eslint-disable-next-line @typescript-eslint/no-explicit-any
await new PdfTranscoder(tx as any, asset.id, team.id, projectId)
.withSprite()
+46 -8
View File
@@ -1,5 +1,12 @@
import { describe, it, expect } from 'vitest'
import { detectSupportedMimeType, getProxyType } from './mime'
import {
detectSupportedMimeType,
getProxyType,
isOfficeDocument,
isHtmlDocument,
isMarkdownDocument,
isCsvDocument,
} from './mime'
describe('detectSupportedMimeType', () => {
it('should detect JPEG', () => {
@@ -45,6 +52,38 @@ describe('detectSupportedMimeType', () => {
})
})
describe('document helpers', () => {
it('isOfficeDocument should correctly identify office files', () => {
expect(isOfficeDocument('application/msword', 'letter.doc')).toBe(true)
expect(
isOfficeDocument(
'application/vnd.openxmlformats-officedocument.wordprocessingml.document',
'file.docx',
),
).toBe(true)
expect(isOfficeDocument(null, 'sheet.xlsx')).toBe(true)
expect(isOfficeDocument(null, 'slides.pptx')).toBe(true)
expect(isOfficeDocument(null, 'notes.rtf')).toBe(true)
expect(isOfficeDocument('text/plain', 'notes.txt')).toBe(false)
})
it('isHtmlDocument should correctly identify html files', () => {
expect(isHtmlDocument('text/html', 'page.html')).toBe(true)
expect(isHtmlDocument(null, 'index.htm')).toBe(true)
expect(isHtmlDocument('text/plain', 'notes.txt')).toBe(false)
})
it('isMarkdownDocument should correctly identify markdown files', () => {
expect(isMarkdownDocument('text/markdown', 'README.md')).toBe(true)
expect(isMarkdownDocument(null, 'doc.markdown')).toBe(true)
})
it('isCsvDocument should correctly identify csv files', () => {
expect(isCsvDocument('text/csv', 'data.csv')).toBe(true)
expect(isCsvDocument(null, 'data.csv')).toBe(true)
})
})
describe('getProxyType', () => {
it('should detect image proxyType', () => {
expect(getProxyType('image/png', 'test.png')).toBe('image')
@@ -61,21 +100,20 @@ describe('getProxyType', () => {
expect(getProxyType('audio/mpeg', 'song.mp3')).toBe('audio')
})
it('should detect pdf proxyType only for pdf, csv, txt, and markdown files', () => {
it('should detect pdf proxyType for pdf, csv, txt, markdown, html, and office files', () => {
expect(getProxyType('application/pdf', 'doc.pdf')).toBe('pdf')
expect(getProxyType('text/plain', 'notes.txt')).toBe('pdf')
expect(getProxyType('text/csv', 'data.csv')).toBe('pdf')
expect(getProxyType('text/markdown', 'README.md')).toBe('pdf')
expect(getProxyType('text/x-markdown', 'doc.markdown')).toBe('pdf')
expect(getProxyType(null, 'file.txt')).toBe('pdf')
expect(getProxyType(null, 'README.md')).toBe('pdf')
expect(getProxyType(null, 'doc.markdown')).toBe('pdf')
expect(getProxyType('text/html', 'index.html')).toBe('pdf')
expect(getProxyType('application/msword', 'letter.doc')).toBe('pdf')
expect(getProxyType(null, 'sheet.xlsx')).toBe('pdf')
expect(getProxyType(null, 'slides.pptx')).toBe('pdf')
})
it('should return null for unsupported files including office files', () => {
it('should return null for unsupported files', () => {
expect(getProxyType('application/zip', 'archive.zip')).toBeNull()
expect(getProxyType('application/msword', 'letter.doc')).toBeNull()
expect(getProxyType(null, 'sheet.xlsx')).toBeNull()
expect(getProxyType(null, 'unknown.bin')).toBeNull()
})
})
+70 -6
View File
@@ -1,5 +1,71 @@
export type ProxyType = 'image' | 'video' | 'audio' | 'pdf'
export function isOfficeDocument(mediaType?: string | null, filename?: string | null): boolean {
const lowerMediaType = mediaType?.toLowerCase() || ''
const lowerFilename = filename?.toLowerCase() || ''
if (
lowerMediaType === 'application/msword' ||
lowerMediaType === 'application/vnd.openxmlformats-officedocument.wordprocessingml.document' ||
lowerMediaType === 'application/vnd.ms-excel' ||
lowerMediaType === 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet' ||
lowerMediaType === 'application/vnd.ms-powerpoint' ||
lowerMediaType ===
'application/vnd.openxmlformats-officedocument.presentationml.presentation' ||
lowerMediaType === 'application/vnd.oasis.opendocument.text' ||
lowerMediaType === 'application/vnd.oasis.opendocument.spreadsheet' ||
lowerMediaType === 'application/vnd.oasis.opendocument.presentation' ||
lowerMediaType === 'application/rtf' ||
lowerMediaType === 'text/rtf'
) {
return true
}
const officeExtensions = [
'.doc',
'.docx',
'.xls',
'.xlsx',
'.ppt',
'.pptx',
'.odt',
'.ods',
'.odp',
'.rtf',
]
return officeExtensions.some((ext) => lowerFilename.endsWith(ext))
}
export function isHtmlDocument(mediaType?: string | null, filename?: string | null): boolean {
const lowerMediaType = mediaType?.toLowerCase() || ''
const lowerFilename = filename?.toLowerCase() || ''
return (
lowerMediaType === 'text/html' ||
lowerFilename.endsWith('.html') ||
lowerFilename.endsWith('.htm')
)
}
export function isMarkdownDocument(mediaType?: string | null, filename?: string | null): boolean {
const lowerMediaType = mediaType?.toLowerCase() || ''
const lowerFilename = filename?.toLowerCase() || ''
return (
lowerMediaType === 'text/markdown' ||
lowerMediaType === 'text/x-markdown' ||
lowerFilename.endsWith('.md') ||
lowerFilename.endsWith('.markdown')
)
}
export function isCsvDocument(mediaType?: string | null, filename?: string | null): boolean {
const lowerMediaType = mediaType?.toLowerCase() || ''
const lowerFilename = filename?.toLowerCase() || ''
return lowerMediaType === 'text/csv' || lowerFilename.endsWith('.csv')
}
export function getProxyType(
mediaType?: string | null,
filename?: string | null,
@@ -14,14 +80,12 @@ export function getProxyType(
if (
lowerMediaType === 'application/pdf' ||
lowerMediaType === 'text/plain' ||
lowerMediaType === 'text/csv' ||
lowerMediaType === 'text/markdown' ||
lowerMediaType === 'text/x-markdown' ||
lowerFilename.endsWith('.pdf') ||
lowerFilename.endsWith('.txt') ||
lowerFilename.endsWith('.csv') ||
lowerFilename.endsWith('.md') ||
lowerFilename.endsWith('.markdown')
isCsvDocument(mediaType, filename) ||
isMarkdownDocument(mediaType, filename) ||
isHtmlDocument(mediaType, filename) ||
isOfficeDocument(mediaType, filename)
) {
return 'pdf'
}
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/db",
"version": "0.3.4",
"version": "0.3.5",
"private": true,
"type": "module",
"main": "./src/index.ts",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/dtos",
"version": "0.3.4",
"version": "0.3.5",
"private": true,
"type": "module",
"main": "./src/index.ts",
Binary file not shown.
Binary file not shown.
Binary file not shown.
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/e2e",
"version": "0.3.4",
"version": "0.3.5",
"private": true,
"type": "module",
"dependencies": {
@@ -0,0 +1,321 @@
import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest'
import { prisma, AssetStatus } from '@shumai/db'
import { setupTestDbHooks } from '@shumai/db/test'
import { workflowService, TaskQueueTranscode } from '@shumai/workflow-core'
import { initTranscodeWorkflows } from '@shumai/transcode'
import { s3Service } from '@shumai/core/src/s3/s3'
import { fileURLToPath } from 'url'
import * as path from 'path'
import * as fs from 'fs'
import { GenericContainer, StartedTestContainer } from 'testcontainers'
const currentDir = path.dirname(fileURLToPath(import.meta.url))
const transcodeWorkflowsPath = path.resolve(currentDir, '../../../apps/transcode/src/workflows.ts')
const fixturesDir = path.resolve(currentDir, '../fixtures')
describe.each(['local', 'temporal'] as const)(
'Workflow E2E - Real Gotenberg Transcode (executor: %s)',
(mode) => {
setupTestDbHooks()
let transcodeWorkerPromise: Promise<void> | null = null
let gotenbergContainer: StartedTestContainer | null = null
beforeAll(async () => {
process.env.S3_BUCKET = 'shumai-e2e-test-bucket-gotenberg'
console.log('Starting Gotenberg container via testcontainers...')
try {
gotenbergContainer = await new GenericContainer('gotenberg/gotenberg:8')
.withExposedPorts(3000)
.start()
const host = gotenbergContainer.getHost()
const port = gotenbergContainer.getMappedPort(3000)
process.env.GOTENBERG_URL = `http://${host}:${port}`
console.log(`Gotenberg container ready at ${process.env.GOTENBERG_URL}`)
} catch (err) {
console.error('Failed to start Gotenberg container:', err)
throw err
}
workflowService.setExecutorType(mode)
initTranscodeWorkflows()
if (mode === 'temporal') {
console.log('Starting background worker for transcode Temporal E2E tests...')
transcodeWorkerPromise = workflowService.startWorkers(TaskQueueTranscode, {
workflowsPath: transcodeWorkflowsPath,
})
await new Promise((resolve) => setTimeout(resolve, 2000))
} else {
console.log('Starting local workflow service polling...')
workflowService.start()
}
}, 120000)
afterAll(async () => {
if (mode === 'temporal') {
console.log('Shutting down Temporal workers...')
await workflowService.shutdownWorkers()
await Promise.all([transcodeWorkerPromise].filter(Boolean))
}
workflowService.close()
vi.restoreAllMocks()
if (gotenbergContainer) {
console.log('Stopping Gotenberg container...')
await gotenbergContainer.stop()
}
try {
console.log('Cleaning up E2E storage files...')
await s3Service.deletePrefix('shumai-e2e-test-bucket-gotenberg', '')
} catch (err) {
console.error('Failed to clean up E2E storage folder:', err)
}
}, 120000)
it('should process real office Word document (.docx) transcode PDF task using Gotenberg', async () => {
const team = await prisma.team.create({
data: { name: 'E2E Gotenberg DOCX Team' },
})
const project = await prisma.project.create({
data: { name: 'E2E Gotenberg DOCX Project', teamId: team.id },
})
const storageKey = await prisma.storageKey.create({
data: {
key: 'projects/e2e/test.docx',
status: 'active',
},
})
const asset = await prisma.asset.create({
data: {
name: 'test.docx',
type: 'file',
status: 'uploaded',
mediaType: 'application/vnd.openxmlformats-officedocument.wordprocessingml.document',
projectId: project.id,
storageKeyId: storageKey.id,
},
})
const docxPath = path.join(fixturesDir, 'test.docx')
const docxBuffer = fs.readFileSync(docxPath)
await s3Service.putObject(
'shumai-e2e-test-bucket-gotenberg',
'projects/e2e/test.docx',
docxBuffer,
docxBuffer.length,
'application/vnd.openxmlformats-officedocument.wordprocessingml.document',
)
const task = await prisma.workflowTask.create({
data: {
type: 'transcode_pdf',
status: 'pending',
assetId: asset.id,
projectId: project.id,
teamId: team.id,
payload: {
projectId: project.id,
transcode: {
poster: true,
sprite: true,
},
},
},
})
console.log(
`Submitted E2E Gotenberg DOCX Workflow Task. ID: ${task.id}. Awaiting completion...`,
)
const completedTask = await workflowService.executeWait(task, 60000)
expect(completedTask.status).toBe('completed')
const updatedAsset = await prisma.asset.findUnique({
where: { id: asset.id },
})
expect(updatedAsset?.status).toBe(AssetStatus.processed)
const mediaInfo = updatedAsset?.media as unknown as {
proxyType?: string
pdfTranscode?: { key: string }
poster?: { key: string }
sprite?: { key: string }
frames?: number
}
expect(mediaInfo).toBeDefined()
expect(mediaInfo.proxyType).toBe('pdf')
expect(mediaInfo.pdfTranscode?.key).toBe(`files/${asset.id}/proxy.pdf`)
expect(mediaInfo.poster?.key).toContain('poster.webp')
expect(mediaInfo.sprite?.key).toContain('sprite.webp')
expect(mediaInfo.frames).toBeGreaterThan(0)
}, 90000)
it('should process real office PowerPoint presentation (.pptx) transcode PDF task using Gotenberg', async () => {
const team = await prisma.team.create({
data: { name: 'E2E Gotenberg PPTX Team' },
})
const project = await prisma.project.create({
data: { name: 'E2E Gotenberg PPTX Project', teamId: team.id },
})
const storageKey = await prisma.storageKey.create({
data: {
key: 'projects/e2e/test.pptx',
status: 'active',
},
})
const asset = await prisma.asset.create({
data: {
name: 'test.pptx',
type: 'file',
status: 'uploaded',
mediaType: 'application/vnd.openxmlformats-officedocument.presentationml.presentation',
projectId: project.id,
storageKeyId: storageKey.id,
},
})
const pptxPath = path.join(fixturesDir, 'test.pptx')
const pptxBuffer = fs.readFileSync(pptxPath)
await s3Service.putObject(
'shumai-e2e-test-bucket-gotenberg',
'projects/e2e/test.pptx',
pptxBuffer,
pptxBuffer.length,
'application/vnd.openxmlformats-officedocument.presentationml.presentation',
)
const task = await prisma.workflowTask.create({
data: {
type: 'transcode_pdf',
status: 'pending',
assetId: asset.id,
projectId: project.id,
teamId: team.id,
payload: {
projectId: project.id,
transcode: {
poster: true,
sprite: true,
},
},
},
})
console.log(
`Submitted E2E Gotenberg PPTX Workflow Task. ID: ${task.id}. Awaiting completion...`,
)
const completedTask = await workflowService.executeWait(task, 60000)
expect(completedTask.status).toBe('completed')
const updatedAsset = await prisma.asset.findUnique({
where: { id: asset.id },
})
expect(updatedAsset?.status).toBe(AssetStatus.processed)
const mediaInfo = updatedAsset?.media as unknown as {
proxyType?: string
pdfTranscode?: { key: string }
poster?: { key: string }
sprite?: { key: string }
frames?: number
}
expect(mediaInfo).toBeDefined()
expect(mediaInfo.proxyType).toBe('pdf')
expect(mediaInfo.pdfTranscode?.key).toBe(`files/${asset.id}/proxy.pdf`)
expect(mediaInfo.poster?.key).toContain('poster.webp')
expect(mediaInfo.sprite?.key).toContain('sprite.webp')
expect(mediaInfo.frames).toBeGreaterThan(0)
}, 90000)
it('should process real office Excel spreadsheet (.xlsx) transcode PDF task using Gotenberg', async () => {
const team = await prisma.team.create({
data: { name: 'E2E Gotenberg XLSX Team' },
})
const project = await prisma.project.create({
data: { name: 'E2E Gotenberg XLSX Project', teamId: team.id },
})
const storageKey = await prisma.storageKey.create({
data: {
key: 'projects/e2e/test.xlsx',
status: 'active',
},
})
const asset = await prisma.asset.create({
data: {
name: 'test.xlsx',
type: 'file',
status: 'uploaded',
mediaType: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
projectId: project.id,
storageKeyId: storageKey.id,
},
})
const xlsxPath = path.join(fixturesDir, 'test.xlsx')
const xlsxBuffer = fs.readFileSync(xlsxPath)
await s3Service.putObject(
'shumai-e2e-test-bucket-gotenberg',
'projects/e2e/test.xlsx',
xlsxBuffer,
xlsxBuffer.length,
'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
)
const task = await prisma.workflowTask.create({
data: {
type: 'transcode_pdf',
status: 'pending',
assetId: asset.id,
projectId: project.id,
teamId: team.id,
payload: {
projectId: project.id,
transcode: {
poster: true,
sprite: true,
},
},
},
})
console.log(
`Submitted E2E Gotenberg XLSX Workflow Task. ID: ${task.id}. Awaiting completion...`,
)
const completedTask = await workflowService.executeWait(task, 60000)
expect(completedTask.status).toBe('completed')
const updatedAsset = await prisma.asset.findUnique({
where: { id: asset.id },
})
expect(updatedAsset?.status).toBe(AssetStatus.processed)
const mediaInfo = updatedAsset?.media as unknown as {
proxyType?: string
pdfTranscode?: { key: string }
poster?: { key: string }
sprite?: { key: string }
frames?: number
}
expect(mediaInfo).toBeDefined()
expect(mediaInfo.proxyType).toBe('pdf')
expect(mediaInfo.pdfTranscode?.key).toBe(`files/${asset.id}/proxy.pdf`)
expect(mediaInfo.poster?.key).toContain('poster.webp')
expect(mediaInfo.sprite?.key).toContain('sprite.webp')
expect(mediaInfo.frames).toBeGreaterThan(0)
}, 90000)
},
)
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/tableprinter",
"version": "0.3.4",
"version": "0.3.5",
"private": true,
"type": "module",
"main": "./src/index.ts",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/transcode",
"version": "0.3.4",
"version": "0.3.5",
"private": true,
"type": "module",
"main": "./src/index.ts",
@@ -3,6 +3,7 @@ import { s3Service } from '@shumai/core/src/s3/s3'
import { prisma } from '@shumai/db'
import { setupTestDbHooks } from '@shumai/db/test'
import { beforeEach, describe, expect, it, vi } from 'vitest'
import { gotenbergService } from '@shumai/core/src/gotenberg/gotenberg'
import { transcodeService } from '@shumai/core/src/transcode/transcode'
import * as child_process from 'child_process'
@@ -37,39 +38,43 @@ vi.mock('@shumai/core/src/s3/s3', () => ({
},
}))
vi.mock('@shumai/core/src/transcode/transcode', () => ({
transcodeService: {
getVideoInfo: vi.fn(),
getAudioInfo: vi.fn(),
getImageInfo: vi.fn(),
getPdfInfo: vi.fn().mockResolvedValue({
originalWidth: 800,
originalHeight: 1000,
duration: 0,
bitRate: 0,
frameRate: 0,
totalFrames: 10,
hasAudio: false,
mimeType: 'application/pdf',
}),
transcodeVideo: vi.fn(),
transcodeAudio: vi.fn(),
transcodeImage: vi.fn(),
generateSprite: vi.fn(),
generatePdfSprite: vi.fn().mockResolvedValue({
pageCount: 10,
originalWidth: 800,
originalHeight: 1000,
}),
generatePdfFromText: vi.fn(),
generatePdfFromCsv: vi.fn(),
createTempDir: vi.fn().mockReturnValue('/tmp'),
removeDir: vi.fn(),
takeScreenshots: vi.fn(),
overlayAnnotations: vi.fn(),
renderPdfPages: vi.fn(),
},
}))
vi.mock('@shumai/core/src/transcode/transcode', async (importOriginal) => {
const actual = await importOriginal<typeof import('@shumai/core/src/transcode/transcode')>()
return {
...actual,
transcodeService: {
getVideoInfo: vi.fn(),
getAudioInfo: vi.fn(),
getImageInfo: vi.fn(),
getPdfInfo: vi.fn().mockResolvedValue({
originalWidth: 800,
originalHeight: 1000,
duration: 0,
bitRate: 0,
frameRate: 0,
totalFrames: 10,
hasAudio: false,
mimeType: 'application/pdf',
}),
transcodeVideo: vi.fn(),
transcodeAudio: vi.fn(),
transcodeImage: vi.fn(),
generateSprite: vi.fn(),
generatePdfSprite: vi.fn().mockResolvedValue({
pageCount: 10,
originalWidth: 800,
originalHeight: 1000,
}),
generatePdfFromText: vi.fn(),
generatePdfFromCsv: vi.fn(),
createTempDir: vi.fn().mockReturnValue('/tmp'),
removeDir: vi.fn(),
takeScreenshots: vi.fn(),
overlayAnnotations: vi.fn(),
renderPdfPages: vi.fn(),
},
}
})
vi.mock('@shumai/core/src/metadata/metadata', () => ({
metadataService: {
@@ -92,8 +97,70 @@ vi.mock('fs', async (importOriginal) => {
describe('Transcode Activities', () => {
setupTestDbHooks()
beforeEach(() => {
beforeEach(async () => {
vi.clearAllMocks()
const fs = await import('fs')
vi.mocked(fs.readFileSync).mockImplementation(() => Buffer.from('fake data') as never)
})
it('should convert CSV via Gotenberg in landscape mode when maxCols > 5', async () => {
const asset = await prisma.asset.create({
data: { name: 'wide.csv', type: 'file', status: 'uploaded' },
})
const fs = await import('fs')
vi.mocked(fs.readFileSync).mockImplementation((_path: unknown, options?: unknown) => {
if (options) {
return 'c1,c2,c3,c4,c5,c6,c7\n1,2,3,4,5,6,7' as never
}
return Buffer.from('fake data') as never
})
vi.spyOn(gotenbergService, 'isAvailable').mockResolvedValue(true)
const convertSpy = vi
.spyOn(gotenbergService, 'convertDocumentToPdf')
.mockResolvedValue(Buffer.from('fake pdf landscape csv'))
const res = await generatePdfProxyActivity({
assetId: asset.id,
assetKey: 'files/asset1/wide.csv',
filePath: '/tmp/wide.csv',
mediaType: 'text/csv',
filename: 'wide.csv',
})
expect(res.pdfProxyKey).toBe(`files/${asset.id}/proxy.pdf`)
expect(convertSpy).toHaveBeenCalledWith('/tmp/wide.csv', 'wide.csv', { landscape: true })
vi.mocked(fs.readFileSync).mockImplementation(() => Buffer.from('fake data') as never)
})
it('should convert CSV via Gotenberg in portrait mode when maxCols <= 5', async () => {
const asset = await prisma.asset.create({
data: { name: 'narrow.csv', type: 'file', status: 'uploaded' },
})
const fs = await import('fs')
vi.mocked(fs.readFileSync).mockImplementation((_path: unknown, options?: unknown) => {
if (options) {
return 'c1,c2,c3\n1,2,3' as never
}
return Buffer.from('fake data') as never
})
vi.spyOn(gotenbergService, 'isAvailable').mockResolvedValue(true)
const convertSpy = vi
.spyOn(gotenbergService, 'convertDocumentToPdf')
.mockResolvedValue(Buffer.from('fake pdf portrait csv'))
const res = await generatePdfProxyActivity({
assetId: asset.id,
assetKey: 'files/asset1/narrow.csv',
filePath: '/tmp/narrow.csv',
mediaType: 'text/csv',
filename: 'narrow.csv',
})
expect(res.pdfProxyKey).toBe(`files/${asset.id}/proxy.pdf`)
expect(convertSpy).toHaveBeenCalledWith('/tmp/narrow.csv', 'narrow.csv', { landscape: false })
vi.mocked(fs.readFileSync).mockImplementation(() => Buffer.from('fake data') as never)
})
it('should call getVideoInfo and set file_type to video', async () => {
@@ -245,6 +312,7 @@ describe('Transcode Activities', () => {
data: { name: 'test.txt', type: 'file', status: 'uploaded' },
})
vi.spyOn(gotenbergService, 'isAvailable').mockResolvedValue(false)
const generatePdfFromTextSpy = vi
.spyOn(transcodeService, 'generatePdfFromText')
.mockImplementation(async (_inPath, outPath) => {
@@ -270,6 +338,7 @@ describe('Transcode Activities', () => {
data: { name: 'README.md', type: 'file', status: 'uploaded' },
})
vi.spyOn(gotenbergService, 'isAvailable').mockResolvedValue(false)
const generatePdfFromTextSpy = vi
.spyOn(transcodeService, 'generatePdfFromText')
.mockImplementation(async (_inPath, outPath) => {
@@ -293,6 +362,188 @@ describe('Transcode Activities', () => {
generatePdfFromTextSpy.mockRestore()
})
it('should convert markdown document via Gotenberg Chromium in generatePdfProxyActivity when available', async () => {
const asset = await prisma.asset.create({
data: { name: 'README.md', type: 'file', status: 'uploaded' },
})
vi.spyOn(gotenbergService, 'isAvailable').mockResolvedValue(true)
const convertMdSpy = vi
.spyOn(gotenbergService, 'convertMarkdownToPdf')
.mockResolvedValue(Buffer.from('fake pdf from gotenberg markdown'))
const res = await generatePdfProxyActivity({
assetId: asset.id,
assetKey: 'files/asset1/README.md',
filePath: '/tmp/README.md',
mediaType: 'text/markdown',
filename: 'README.md',
})
expect(res.pdfProxyKey).toBe(`files/${asset.id}/proxy.pdf`)
expect(convertMdSpy).toHaveBeenCalledWith('/tmp/README.md', 'README.md')
})
it('should convert office document via Gotenberg in generatePdfProxyActivity when available', async () => {
const asset = await prisma.asset.create({
data: { name: 'document.docx', type: 'file', status: 'uploaded' },
})
vi.spyOn(gotenbergService, 'isAvailable').mockResolvedValue(true)
const convertSpy = vi
.spyOn(gotenbergService, 'convertDocumentToPdf')
.mockResolvedValue(Buffer.from('fake pdf from gotenberg office'))
const res = await generatePdfProxyActivity({
assetId: asset.id,
assetKey: 'files/asset1/document.docx',
filePath: '/tmp/document.docx',
mediaType: 'application/vnd.openxmlformats-officedocument.wordprocessingml.document',
filename: 'document.docx',
})
expect(res.pdfProxyKey).toBe(`files/${asset.id}/proxy.pdf`)
expect(convertSpy).toHaveBeenCalledWith('/tmp/document.docx', 'document.docx')
})
it('should convert html document via Gotenberg Chromium in generatePdfProxyActivity when available', async () => {
const asset = await prisma.asset.create({
data: { name: 'index.html', type: 'file', status: 'uploaded' },
})
vi.spyOn(gotenbergService, 'isAvailable').mockResolvedValue(true)
const convertHtmlSpy = vi
.spyOn(gotenbergService, 'convertHtmlToPdf')
.mockResolvedValue(Buffer.from('fake pdf from gotenberg html'))
const res = await generatePdfProxyActivity({
assetId: asset.id,
assetKey: 'files/asset1/index.html',
filePath: '/tmp/index.html',
mediaType: 'text/html',
filename: 'index.html',
})
expect(res.pdfProxyKey).toBe(`files/${asset.id}/proxy.pdf`)
expect(convertHtmlSpy).toHaveBeenCalledWith('/tmp/index.html', 'index.html')
})
it('should throw ApplicationFailure when Office document conversion is attempted but Gotenberg is unavailable', async () => {
const asset = await prisma.asset.create({
data: { name: 'sheet.xlsx', type: 'file', status: 'uploaded' },
})
vi.spyOn(gotenbergService, 'isAvailable').mockResolvedValue(false)
await expect(
generatePdfProxyActivity({
assetId: asset.id,
assetKey: 'files/asset1/sheet.xlsx',
filePath: '/tmp/sheet.xlsx',
mediaType: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
filename: 'sheet.xlsx',
}),
).rejects.toThrow('Gotenberg service is required for Office document conversion')
})
it('should throw ApplicationFailure when HTML document conversion is attempted but Gotenberg is unavailable', async () => {
const asset = await prisma.asset.create({
data: { name: 'index.html', type: 'file', status: 'uploaded' },
})
vi.spyOn(gotenbergService, 'isAvailable').mockResolvedValue(false)
await expect(
generatePdfProxyActivity({
assetId: asset.id,
assetKey: 'files/asset1/index.html',
filePath: '/tmp/index.html',
mediaType: 'text/html',
filename: 'index.html',
}),
).rejects.toThrow('Gotenberg service is required for HTML document conversion')
})
it('should fallback to transcodeService.generatePdfFromCsv for CSV when Gotenberg is unavailable', async () => {
const asset = await prisma.asset.create({
data: { name: 'data.csv', type: 'file', status: 'uploaded' },
})
vi.spyOn(gotenbergService, 'isAvailable').mockResolvedValue(false)
const generatePdfFromCsvSpy = vi
.spyOn(transcodeService, 'generatePdfFromCsv')
.mockImplementation(async (_inPath, outPath) => {
const fs = await import('fs')
fs.writeFileSync(outPath, 'fake pdf from csv fallback')
})
const res = await generatePdfProxyActivity({
assetId: asset.id,
assetKey: 'files/asset1/data.csv',
filePath: '/tmp/data.csv',
mediaType: 'text/csv',
filename: 'data.csv',
})
expect(res.pdfProxyKey).toBe(`files/${asset.id}/proxy.pdf`)
expect(generatePdfFromCsvSpy).toHaveBeenCalledWith(
'/tmp/data.csv',
expect.stringContaining('proxy.pdf'),
)
generatePdfFromCsvSpy.mockRestore()
})
it('should convert TXT document via Gotenberg when available', async () => {
const asset = await prisma.asset.create({
data: { name: 'notes.txt', type: 'file', status: 'uploaded' },
})
vi.spyOn(gotenbergService, 'isAvailable').mockResolvedValue(true)
const convertSpy = vi
.spyOn(gotenbergService, 'convertDocumentToPdf')
.mockResolvedValue(Buffer.from('fake pdf from txt gotenberg'))
const res = await generatePdfProxyActivity({
assetId: asset.id,
assetKey: 'files/asset1/notes.txt',
filePath: '/tmp/notes.txt',
mediaType: 'text/plain',
filename: 'notes.txt',
})
expect(res.pdfProxyKey).toBe(`files/${asset.id}/proxy.pdf`)
expect(convertSpy).toHaveBeenCalledWith('/tmp/notes.txt', 'notes.txt')
})
it('should fallback to transcodeService.generatePdfFromText for TXT when Gotenberg is unavailable', async () => {
const asset = await prisma.asset.create({
data: { name: 'notes.txt', type: 'file', status: 'uploaded' },
})
vi.spyOn(gotenbergService, 'isAvailable').mockResolvedValue(false)
const generatePdfFromTextSpy = vi
.spyOn(transcodeService, 'generatePdfFromText')
.mockImplementation(async (_inPath, outPath) => {
const fs = await import('fs')
fs.writeFileSync(outPath, 'fake pdf from txt fallback')
})
const res = await generatePdfProxyActivity({
assetId: asset.id,
assetKey: 'files/asset1/notes.txt',
filePath: '/tmp/notes.txt',
mediaType: 'text/plain',
filename: 'notes.txt',
})
expect(res.pdfProxyKey).toBe(`files/${asset.id}/proxy.pdf`)
expect(generatePdfFromTextSpy).toHaveBeenCalledWith(
'/tmp/notes.txt',
expect.stringContaining('proxy.pdf'),
)
generatePdfFromTextSpy.mockRestore()
})
it('should set file_type to file for unknown types', async () => {
const asset = await prisma.asset.create({
data: { name: 'u.xyz', type: 'file', status: 'uploaded' },
+77 -15
View File
@@ -3,7 +3,15 @@ import { s3Service } from '@shumai/core/src/s3/s3'
import { transcodeService } from '@shumai/core/src/transcode/transcode'
import { metadataService } from '@shumai/core/src/metadata/metadata'
import { stemFromKey } from '@shumai/core/src/utils/filename'
import { getProxyType } from '@shumai/core/src/utils/mime'
import { gotenbergService } from '@shumai/core/src/gotenberg/gotenberg'
import { parseCsvContent } from '@shumai/core/src/transcode/transcode'
import {
getProxyType,
isCsvDocument,
isHtmlDocument,
isMarkdownDocument,
isOfficeDocument,
} from '@shumai/core/src/utils/mime'
import { ApplicationFailure } from '@temporalio/activity'
import * as path from 'path'
import * as fs from 'fs'
@@ -511,16 +519,13 @@ export interface GeneratePdfProxyActivityParams {
export async function generatePdfProxyActivity(
params: GeneratePdfProxyActivityParams,
): Promise<{ pdfProxyKey: string; pdfFilePath: string }> {
const isTxt =
params.mediaType === 'text/plain' ||
params.mediaType === 'text/markdown' ||
params.mediaType === 'text/x-markdown' ||
params.filename.toLowerCase().endsWith('.txt') ||
params.filename.toLowerCase().endsWith('.md') ||
params.filename.toLowerCase().endsWith('.markdown')
const isCsv = params.mediaType === 'text/csv' || params.filename.toLowerCase().endsWith('.csv')
const isOffice = isOfficeDocument(params.mediaType, params.filename)
const isHtml = isHtmlDocument(params.mediaType, params.filename)
const isMd = isMarkdownDocument(params.mediaType, params.filename)
const isCsv = isCsvDocument(params.mediaType, params.filename)
const isTxt = params.mediaType === 'text/plain' || params.filename.toLowerCase().endsWith('.txt')
if (!isTxt && !isCsv) {
if (!isOffice && !isHtml && !isMd && !isCsv && !isTxt) {
return { pdfProxyKey: params.assetKey, pdfFilePath: params.filePath }
}
@@ -530,10 +535,67 @@ export async function generatePdfProxyActivity(
const pdfFilePath = path.join(tmpDir, 'proxy.pdf')
try {
if (isCsv) {
await transcodeService.generatePdfFromCsv(params.filePath, pdfFilePath)
} else {
await transcodeService.generatePdfFromText(params.filePath, pdfFilePath)
const gotenbergAvailable = await gotenbergService.isAvailable()
if (isOffice) {
if (!gotenbergAvailable) {
throw new Error(
'Gotenberg service is required for Office document conversion but is unavailable.',
)
}
const pdfBuffer = await gotenbergService.convertDocumentToPdf(
params.filePath,
params.filename,
)
fs.writeFileSync(pdfFilePath, pdfBuffer)
} else if (isHtml) {
if (!gotenbergAvailable) {
throw new Error(
'Gotenberg service is required for HTML document conversion but is unavailable.',
)
}
const pdfBuffer = await gotenbergService.convertHtmlToPdf(params.filePath, params.filename)
fs.writeFileSync(pdfFilePath, pdfBuffer)
} else if (isCsv) {
if (gotenbergAvailable) {
let isLandscape = false
try {
const content = String(fs.readFileSync(params.filePath, 'utf-8'))
const rows = parseCsvContent(content)
const maxCols = Math.max(...rows.map((r) => r.length), 0)
isLandscape = maxCols > 5
} catch {
// Fallback landscape calculation
}
const pdfBuffer = await gotenbergService.convertDocumentToPdf(
params.filePath,
params.filename,
{ landscape: isLandscape },
)
fs.writeFileSync(pdfFilePath, pdfBuffer)
} else {
await transcodeService.generatePdfFromCsv(params.filePath, pdfFilePath)
}
} else if (isMd) {
if (gotenbergAvailable) {
const pdfBuffer = await gotenbergService.convertMarkdownToPdf(
params.filePath,
params.filename,
)
fs.writeFileSync(pdfFilePath, pdfBuffer)
} else {
await transcodeService.generatePdfFromText(params.filePath, pdfFilePath)
}
} else if (isTxt) {
if (gotenbergAvailable) {
const pdfBuffer = await gotenbergService.convertDocumentToPdf(
params.filePath,
params.filename,
)
fs.writeFileSync(pdfFilePath, pdfBuffer)
} else {
await transcodeService.generatePdfFromText(params.filePath, pdfFilePath)
}
}
const stat = fs.statSync(pdfFilePath)
@@ -544,7 +606,7 @@ export async function generatePdfProxyActivity(
} catch (err) {
const { message } = getErrorDetails(err)
throw ApplicationFailure.create({
message: `Failed to generate PDF proxy for text/csv/markdown: ${message}`,
message: `Failed to generate PDF proxy: ${message}`,
nonRetryable: true,
cause: err instanceof Error ? err : undefined,
})
+12 -7
View File
@@ -65,6 +65,8 @@ interface BreadcrumbNavProps {
shareId?: string
allowDownload?: boolean
onFolderClick?: (folderId: string) => void
onRename?: () => void
onDelete?: () => void
fileId?: string
downloadInfo?: {
originalKey?: string
@@ -103,6 +105,8 @@ export function BreadcrumbNav({
onChatbotToggle,
isPublic = false,
onFolderClick,
onRename,
onDelete,
fileId,
shareId,
allowDownload = true,
@@ -290,14 +294,15 @@ export function BreadcrumbNav({
{m.download()}
</DropdownMenuItem>
))}
{canEdit && (
<DropdownMenuItem onClick={() => console.log('Rename')}>
Rename
</DropdownMenuItem>
{canEdit && onRename && (
<DropdownMenuItem onClick={onRename}>{m.rename()}</DropdownMenuItem>
)}
{canEdit && (
<DropdownMenuItem onClick={() => console.log('Delete')}>
Delete
{canEdit && onDelete && (
<DropdownMenuItem
onClick={onDelete}
className="text-destructive focus:text-destructive"
>
{m.delete()}
</DropdownMenuItem>
)}
@@ -17,6 +17,7 @@ import { EditableText } from '@/ui/components/ui/editable-text'
import { ProgressCircle } from '@/ui/components/ui/progress-circle'
import { Skeleton } from '@/ui/components/ui/skeleton'
import { formatTimeAgo } from '@/ui/lib/time'
import { selectFileNameWithoutExtension } from '@/ui/lib/rename-utils'
import { cn } from '@/ui/lib/utils'
import { useUploadStore } from '@/ui/stores/upload'
import { useDraggable, useDroppable } from '@dnd-kit/react'
@@ -116,7 +117,15 @@ export function FileCard({
},
})
const displayItem = polledItem || item
const displayItem = useMemo(() => {
if (item.status === 'processed' || item.status === 'error') {
return item
}
if (polledItem?.status === 'processed' || polledItem?.status === 'error') {
return polledItem
}
return polledItem || item
}, [item, polledItem])
// dnd-kit hooks
const { ref: setDraggableRef, isDragging: isDraggableDragging } = useDraggable({
@@ -180,8 +189,9 @@ export function FileCard({
// the DOM has stabilized before focusing. Radix focus restoration
// can sometimes conflict with immediate focus calls.
const timeoutId = setTimeout(() => {
inputRef.current?.focus()
inputRef.current?.select()
if (inputRef.current) {
selectFileNameWithoutExtension(inputRef.current)
}
}, 150)
return () => clearTimeout(timeoutId)
}
@@ -7,6 +7,7 @@ import { File, Folder, MoreVertical, AudioLines } from 'lucide-react'
import React, { useEffect, useMemo, useRef, useState } from 'react'
import { cn } from '@/ui/lib/utils'
import { formatSize } from '@/ui/lib/format'
import { selectFileNameWithoutExtension } from '@/ui/lib/rename-utils'
import type { DragState } from '../dnd-types'
import FieldRenderer from '../field-renderer'
import { Badge } from '@/ui/components/ui/badge'
@@ -91,7 +92,15 @@ export function FileListItem({
},
})
const displayItem = polledItem || item
const displayItem = useMemo(() => {
if (item.status === 'processed' || item.status === 'error') {
return item
}
if (polledItem?.status === 'processed' || polledItem?.status === 'error') {
return polledItem
}
return polledItem || item
}, [item, polledItem])
// dnd-kit hooks
const { ref: setDraggableRef, isDragging: isDraggableDragging } = useDraggable({
@@ -125,8 +134,9 @@ export function FileListItem({
// the DOM has stabilized before focusing. Radix focus restoration
// can sometimes conflict with immediate focus calls.
const timeoutId = setTimeout(() => {
inputRef.current?.focus()
inputRef.current?.select()
if (inputRef.current) {
selectFileNameWithoutExtension(inputRef.current)
}
}, 150)
return () => clearTimeout(timeoutId)
}
@@ -303,13 +303,17 @@ export function useFileActions({
for (let i = 0; i < files.length; i += batchSize) {
const chunk = files.slice(i, i + batchSize)
for (const file of chunk) {
const a = document.createElement('a')
a.href = file.url
a.download = file.name
a.target = '_blank'
document.body.appendChild(a)
a.click()
document.body.removeChild(a)
// Use a hidden iframe instead of an anchor with target="_blank".
// Download URLs respond with Content-Disposition: attachment, so the
// browser starts the download in the background: no flash of a tab
// that opens and auto-closes. A failed/expired link renders inside
// the hidden iframe instead of navigating the SPA tab away.
const iframe = document.createElement('iframe')
iframe.style.display = 'none'
iframe.src = file.url
document.body.appendChild(iframe)
// Remove the iframe once the download has had time to start.
window.setTimeout(() => iframe.remove(), 60_000)
}
if (i + batchSize < files.length) {
await delay(600)
+4
View File
@@ -27,6 +27,8 @@ export function TopNav() {
shareId,
allowDownload,
onFolderClick,
onRename,
onDelete,
compareMode,
canCompareVersions,
onCompareVersions,
@@ -118,6 +120,8 @@ export function TopNav() {
downloadInfo={downloadInfo}
versions={versions}
onFolderClick={onFolderClick}
onRename={onRename}
onDelete={onDelete}
compareMode={compareMode}
canCompareVersions={canCompareVersions}
onCompareVersions={onCompareVersions}
+61
View File
@@ -0,0 +1,61 @@
import { describe, expect, it, vi } from 'vitest'
import { getFileNameBaseRange, selectFileNameWithoutExtension } from './rename-utils'
describe('rename-utils', () => {
describe('getFileNameBaseRange', () => {
it('returns range excluding file extension for standard filenames', () => {
expect(getFileNameBaseRange('test.png')).toEqual([0, 4])
expect(getFileNameBaseRange('document.pdf')).toEqual([0, 8])
expect(getFileNameBaseRange('archive.tar.gz')).toEqual([0, 11])
})
it('returns full length range for filenames without extension', () => {
expect(getFileNameBaseRange('README')).toEqual([0, 6])
expect(getFileNameBaseRange('my-folder')).toEqual([0, 9])
})
it('returns full length range for dotfiles starting with dot', () => {
expect(getFileNameBaseRange('.gitignore')).toEqual([0, 10])
expect(getFileNameBaseRange('.env')).toEqual([0, 4])
})
it('returns range excluding trailing dot for filenames ending with dot', () => {
expect(getFileNameBaseRange('file.')).toEqual([0, 4])
})
it('returns [0, 0] for empty string', () => {
expect(getFileNameBaseRange('')).toEqual([0, 0])
})
})
describe('selectFileNameWithoutExtension', () => {
it('calls focus and setSelectionRange on input element', () => {
const input = {
focus: vi.fn(),
value: 'image.png',
setSelectionRange: vi.fn(),
select: vi.fn(),
} as unknown as HTMLInputElement
selectFileNameWithoutExtension(input)
expect(input.focus).toHaveBeenCalled()
expect(input.setSelectionRange).toHaveBeenCalledWith(0, 5)
expect(input.select).not.toHaveBeenCalled()
})
it('falls back to select() if setSelectionRange is not a function', () => {
const input = {
focus: vi.fn(),
value: 'image.png',
setSelectionRange: undefined,
select: vi.fn(),
} as unknown as HTMLInputElement
selectFileNameWithoutExtension(input)
expect(input.focus).toHaveBeenCalled()
expect(input.select).toHaveBeenCalled()
})
})
})
+27
View File
@@ -0,0 +1,27 @@
/**
* Calculates the start and end selection range for a file name to exclude its extension.
* e.g. "test.png" -> [0, 4] ("test")
* e.g. "archive.tar.gz" -> [0, 11] ("archive.tar")
* e.g. "file_without_ext" -> [0, 16] ("file_without_ext")
* e.g. ".gitignore" -> [0, 10] (".gitignore")
*/
export function getFileNameBaseRange(name: string): [number, number] {
const lastDotIndex = name.lastIndexOf('.')
if (lastDotIndex > 0) {
return [0, lastDotIndex]
}
return [0, name.length]
}
/**
* Focuses the input element and selects the base file name, excluding the extension.
*/
export function selectFileNameWithoutExtension(input: HTMLInputElement) {
input.focus()
const [start, end] = getFileNameBaseRange(input.value)
if (typeof input.setSelectionRange === 'function') {
input.setSelectionRange(start, end)
} else {
input.select()
}
}
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/webui",
"version": "0.3.4",
"version": "0.3.5",
"private": true,
"type": "module",
"main": "./index.html",
@@ -18,6 +18,27 @@ import { useMutation } from '@tanstack/react-query'
import { createLazyFileRoute } from '@tanstack/react-router'
import { InferRequestType, InferResponseType } from 'hono/client'
import {
AlertDialog,
AlertDialogAction,
AlertDialogCancel,
AlertDialogContent,
AlertDialogDescription,
AlertDialogFooter,
AlertDialogHeader,
AlertDialogTitle,
} from '@/ui/components/ui/alert-dialog'
import { Button } from '@/ui/components/ui/button'
import {
Dialog,
DialogContent,
DialogFooter,
DialogHeader,
DialogTitle,
} from '@/ui/components/ui/dialog'
import { Input } from '@/ui/components/ui/input'
import { selectFileNameWithoutExtension } from '@/ui/lib/rename-utils'
import { toast } from 'sonner'
import type { MediaController } from '@/ui/components/viewers/types'
import type { AssetInfo, AssetInfoPaginatedList, CommentInfo } from '@shumai/dtos'
import { keepPreviousData, useQuery, useQueryClient } from '@tanstack/react-query'
@@ -147,6 +168,84 @@ function FileViewPage() {
const fileData = versionData || stackData
const versionsDataList = versionAssetId ? versionsList : stackData?.versionStack?.versions
const parentFolderId = fileData?.ancestorFolders?.[0]?.id ?? projectInfo?.rootFolder ?? ''
const [isRenameDialogOpen, setIsRenameDialogOpen] = useState(false)
const [renameInput, setRenameInput] = useState('')
const renameInputRef = useRef<HTMLInputElement>(null)
const renameSettledRef = useRef(false)
const [isDeleteDialogOpen, setIsDeleteDialogOpen] = useState(false)
const $renameFile = client.api.files[':fileId'].$put
const { mutate: renameFile, isPending: isRenaming } = useMutation<
InferResponseType<typeof $renameFile, 200>,
Error,
InferRequestType<typeof $renameFile>
>({
mutationFn: async (request) => {
const res = await $renameFile(request)
if (!res.ok) throw new Error('Failed to rename file')
return (await res.json()) as unknown as InferResponseType<typeof $renameFile, 200>
},
onSuccess: () => {
queryClient.invalidateQueries({ queryKey: ['files'] })
queryClient.invalidateQueries({ queryKey: ['version_stacks'] })
toast.success(m.renamed_successfully?.() || 'Renamed successfully')
setIsRenameDialogOpen(false)
},
onError: () => {
toast.error(m.failed_to_rename?.() || 'Failed to rename')
},
})
const $deleteFiles = client.api.files.$delete
const { mutate: deleteFiles, isPending: isDeleting } = useMutation<
void,
Error,
InferRequestType<typeof $deleteFiles>
>({
mutationFn: async (request) => {
const res = await $deleteFiles(request)
if (!res.ok) throw new Error('Failed to delete file')
},
onSuccess: () => {
queryClient.invalidateQueries({ queryKey: ['files'] })
queryClient.invalidateQueries({ queryKey: ['version_stacks'] })
toast.success(m.deleted?.() || 'Deleted')
setIsDeleteDialogOpen(false)
if (parentFolderId && parentFolderId !== projectInfo?.rootFolder) {
navigate({
to: '/projects/$projectId/folders/$folderId',
params: { projectId, folderId: parentFolderId },
})
} else {
navigate({
to: '/projects/$projectId',
params: { projectId },
})
}
},
onError: () => {
toast.error(m.failed_to_delete?.() || 'Failed to delete')
},
})
const handleRenameSubmit = (e?: React.FormEvent) => {
e?.preventDefault()
if (!renameInput.trim() || !activeFileId) return
renameFile({
param: { fileId: activeFileId },
json: { name: renameInput.trim() },
})
}
const handleDeleteSubmit = () => {
if (!activeFileId) return
deleteFiles({
json: { ids: [activeFileId] },
})
}
useEffect(() => {
if (fileData && projectInfo && teamId) {
setProjectState({
@@ -198,6 +297,14 @@ function FileViewPage() {
params: { projectId, folderId: id },
})
},
onRename: () => {
renameSettledRef.current = false
setRenameInput(fileData?.name ?? '')
setIsRenameDialogOpen(true)
},
onDelete: () => {
setIsDeleteDialogOpen(true)
},
})
}
@@ -216,7 +323,29 @@ function FileViewPage() {
navigate,
])
const parentFolderId = fileData?.ancestorFolders?.[0]?.id ?? projectInfo?.rootFolder ?? ''
useEffect(() => {
if (!isRenameDialogOpen) return
// Stop correcting the selection shortly after the dialog opens so user
// edits are not disturbed. The dropdown's focus restoration settles well
// within this window.
const timeoutId = setTimeout(() => {
renameSettledRef.current = true
}, 1000)
return () => clearTimeout(timeoutId)
}, [isRenameDialogOpen])
const handleRenameInputFocus = () => {
if (renameSettledRef.current) return
// The dialog focus scope calls select() synchronously right after the
// focus event, and the breadcrumb dropdown restores focus ~150ms later
// (after its exit animation), re-triggering the trap's select-all. Defer
// the base-name selection so it runs after those select-all calls.
setTimeout(() => {
if (renameInputRef.current) {
selectFileNameWithoutExtension(renameInputRef.current)
}
}, 0)
}
useEffect(() => {
if (!parentFolderId || !activeFileId) return
@@ -470,6 +599,58 @@ function FileViewPage() {
</>
)}
</div>
<Dialog open={isRenameDialogOpen} onOpenChange={setIsRenameDialogOpen}>
<DialogContent>
<form onSubmit={handleRenameSubmit}>
<DialogHeader>
<DialogTitle>{m.rename()}</DialogTitle>
</DialogHeader>
<div className="py-4">
<Input
ref={renameInputRef}
value={renameInput}
onChange={(e) => setRenameInput(e.target.value)}
placeholder={m.enter_new_name?.() || 'Enter new name'}
onFocus={handleRenameInputFocus}
autoFocus
/>
</div>
<DialogFooter>
<Button type="button" variant="outline" onClick={() => setIsRenameDialogOpen(false)}>
{m.cancel()}
</Button>
<Button type="submit" disabled={isRenaming || !renameInput.trim()}>
{isRenaming && <Loader2 className="mr-2 h-4 w-4 animate-spin" />}
{m.save()}
</Button>
</DialogFooter>
</form>
</DialogContent>
</Dialog>
<AlertDialog open={isDeleteDialogOpen} onOpenChange={setIsDeleteDialogOpen}>
<AlertDialogContent>
<AlertDialogHeader>
<AlertDialogTitle>{m.delete_asset_title?.() || 'Delete Asset?'}</AlertDialogTitle>
<AlertDialogDescription>
{m.delete_asset_description?.() ||
'Deleted items can be recovered for 30 days before being permanently deleted.'}
</AlertDialogDescription>
</AlertDialogHeader>
<AlertDialogFooter>
<AlertDialogCancel>{m.cancel()}</AlertDialogCancel>
<AlertDialogAction
onClick={handleDeleteSubmit}
disabled={isDeleting}
className="bg-destructive text-destructive-foreground hover:bg-destructive/90"
>
{isDeleting && <Loader2 className="mr-2 h-4 w-4 animate-spin" />}
{m.delete()}
</AlertDialogAction>
</AlertDialogFooter>
</AlertDialogContent>
</AlertDialog>
</div>
)
}
+2
View File
@@ -34,6 +34,8 @@ export interface TopNavProjectState {
/** When false, the public share hides download affordances. Defaults to true. */
allowDownload?: boolean
onFolderClick?: (folderId: string) => void
onRename?: () => void
onDelete?: () => void
isRightSidebarCollapsed?: boolean
onRightSidebarToggle?: () => void
// Compare Versions
+1 -1
View File
@@ -1 +1 @@
export const APP_VERSION = '0.3.4'
export const APP_VERSION = '0.3.5'
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shumai/workflow-core",
"version": "0.3.4",
"version": "0.3.5",
"private": true,
"type": "module",
"main": "./src/index.ts",