Files
shumai/packages/webui/docs/video_player_design.md
T
Yiling-J a84da8678e feat(webui): switch from video.js to hls.js and native video player (#437)
* feat(webui): switch from video.js to hls.js and native video player

* fix(webui): prevent stale seek hijacking and guard native HLS compare mode
2026-09-30 20:23:44 +08:00

19 KiB
Raw Blame History

Frame-Accurate Video Player & Comment System: Technical Design Document

This document outlines the technical design, mathematical formulas, synchronization loops, and database schemes that guarantee 100% frame-accurate playback, scrubbing, stepping, and comment navigation in Shumai.


1. Architectural Overview

Shumai's video player is a decoupled, frame-locked system that coordinates three primary components around a single state coordinator (useFramePlayer):

                     ┌───────────────────────────┐
                     │    useFramePlayer Hook    │
                     │  (State: currentFrame)    │
                     └──────┬─────────────▲──────┘
                            │             │
        1. Set currentTime  │             │ 2. Sync frame index
        (Frame Center Nudge)│             │ (Floor-with-Epsilon)
                            ▼             │
             ┌────────────────────────────┴──┐
             │   Native HTML5 Video Player   │
             │     (Hls.js / Native MSE)     │
             └───────────────────────────────┘
                     ▲                   ▲
                     │                   │
         Stepping /  │                   │ Comment Selection /
         Scrubbing   │                   │ Timestamp Navigation
                     │                   │
             ┌───────┴──────┐     ┌──────┴───────┐
             │   Seekbar    │     │   Comments   │
             │ (Progress)   │     │ (Panel/List) │
             └──────────────┘     └──────────────┘
  1. Native HTML5 Video Player (Hls.js / Native MSE): Manages the underlying browser media stream, buffering, audio clock, and presentation. Direct <video> element with zero library wrappers for MP4, and lightweight hls.js for adaptive HLS streams.
  2. Seekbar (Scrubber & Progress Bar): Visualizes playback progress and maps horizontal screen clicks/drags to playhead percentages.
  3. Comment Area (Sidebar): Renders comment list, displays timestamp labels, captures drawings, and triggers seeks to target frames.

2. Core Mathematical Foundations

To achieve frame-accurate synchronization across different browsers (which use floating-point double precision for playhead timestamps), we utilize three key mathematical formulas:

A. Frame-to-Time Boundary Formula

When generating timestamps for external storage (such as database comments), we calculate the exact start boundary time of the frame. This avoids carrying floating-point errors into the database:

\text{Time}_{\text{boundary}} = \frac{N}{\text{frameRate}}

Where N is the integer frame index ($0$-indexed).


B. Time-to-Frame Conversion (Floor with 0.45 Offset)

When converting a raw player timestamp (t) back to an integer frame index, we use a Floor with 0.45 Offset formula to absorb sub-millisecond clock drift, compositor presentation offsets, and V-Sync boundaries:

\text{Frame} = \lfloor(t \cdot \text{frameRate}) + 0.45\rfloor
  • Compositor Lag Guard: Since the browser compositor typically presents a frame slightly before the playhead crosses the exact mathematical start time (due to V-Sync ticks and decoding latency), the browser transitions visually around the half-frame mark. An offset of 0.45 shifts the transition boundary to 0.55 frames, matching user perception.
  • Seek Buffer Guard: When we seek to the safe center of a frame (N + 0.5 frames), using a 0.45 offset yields N + 0.95, which floors back to the correct frame index N even if the browser has minor floating-point seek target deviations.

C. Safe Seek Time (Half-Frame Offset Center)

Browsers can occasionally render the previous frame if the playhead lands exactly on a frame boundary due to float rounding. To guarantee that seeking always lands squarely inside the target frame's presentation window, we seek to the center of the frame:

\text{Time}_{\text{center}} = \left(N \cdot T_d\right) + \frac{T_d}{2} = \frac{N}{\text{frameRate}} + \frac{1}{2 \cdot \text{frameRate}}

Where T_d = \frac{1}{\text{frameRate}} is the duration of a single frame.


D. Effective Frame Count Derivation (Container vs. Video Stream)

The player's totalFrames governs the frame readout (N / M fr), the seekbar extent, frame-stepping bounds, and the playback clamp. It is not always the same as the container's frame count, because a container's duration can exceed its video stream — most commonly when the audio track is slightly longer than the video track (AAC padding and the fixed 1024-samples-per-frame granularity mean audio rarely ends on the exact video timestamp), or from container overhead.

Deriving the count naively from the container duration (\lfloor \text{containerDuration} \cdot \text{frameRate} \rceil) invents phantom trailing frames the video stream does not contain. Whether those extra frames should be counted depends on whether the playhead can actually reach them during playback, which is governed by the same stall threshold used by the playback loop (§3):

\text{Threshold}_{s} = \frac{\max\left(100, \frac{3000}{\text{frameRate}}\right)}{1000}\text{ seconds}

Let \text{tail} = \text{containerDuration} - \frac{\text{dbTotalFrames}}{\text{frameRate}} (where dbTotalFrames is the video stream's nb_frames). Then:

$$\text{totalFrames} = \begin{cases} \lfloor \text{containerDuration} \cdot \text{frameRate} \rceil & \text{if tail} > \text{Threshold}_{s}\ \text{(reachable — keep the tail)} \ \text{dbTotalFrames} & \text{otherwise (unreachable — drop phantom frames)} \ \lfloor \text{containerDuration} \cdot \text{frameRate} \rceil & \text{if dbTotalFrames} \le 0\ \text{(nb_frames missing)} \end{cases}$$

  • Small tail (tail ≤ threshold): The RAF fallback never activates (§3), so the playhead physically cannot enter the tail. Counting those frames would strand the readout short of its own maximum (e.g. a 268-frame video in a 270-frame container would show 267 / 269 and stop at 267). We therefore use the accurate video-stream count → 267 / 267. The trailing audio still plays out to the container end on the native element; it is simply not represented as extra scrubbable frames.
  • Large tail (tail > threshold): The playhead does traverse the audio-only region (the RAF fallback drives it from video.currentTime, §3), so the container-derived count is correct and the readout reaches its maximum at the natural end (e.g. a 150-frame video with a 6.5s audio tail shows 194 / 194).

This rule is implemented by resolveTotalFrames in viewers/utils.ts, which shares stallThresholdMs with useFramePlayer so the frame-count boundary and the runtime playhead-reachability boundary can never drift apart.


3. Active Playback Loops & Dual-Drive Coordination

When playing, the playhead loop coordinates two mechanisms in parallel to prevent timeline freezes and avoid visual jitter:

                            ┌───────────────────┐
                            │ Playback Started  │
                            └─────────┬─────────┘
                                      │
                         ┌────────────┴────────────┐
                         │   Start loops: RAF &    │
                         │   VFC (if supported)    │
                         └────────────┬────────────┘
                                      │
                       ┌──────────────┴──────────────┐
                       │  Is VFC Stalled (> Thresh)? │
                       └─────┬─────────────────┬─────┘
                             │ Yes             │ No
                             ▼                 ▼
                 ┌───────────────────────┐ ┌───────────────────────┐
                 │ Update UI using       │ │ Update UI using       │
                 │ currentTime (RAF)     │ │ mediaTime (VFC)       │
                 └───────────────────────┘ └───────────────────────┘
  1. requestVideoFrameCallback (rVFC): The primary driver when the video is actively decoding. It fires only when a new video frame is sent to the compositor, updating the UI frame-accurately based on metadata.mediaTime.

  2. requestAnimationFrame (RAF): Runs continuously at 60Hz. It acts as the fallback driver.

  3. Compositor-First Gating & Stall Detection: video.currentTime is the playback (audio) clock, and on some engines (notably Safari/WebKit) it runs ahead of the frame actually presented on screen right after playback starts. Trusting it before the compositor catches up would race the playhead forward and then snap it backward when the first rVFC (mediaTime) arrives. To prevent this, each play resets the per-session sync state, and the playhead is driven from video.currentTime only when requestVideoFrameCallback is unsupported, or rVFC has delivered a frame this session and has since stalled beyond the dynamic stall threshold:

    \text{Threshold} = \max\left(100, \frac{3000}{\text{frameRate}}\right)\text{ ms}

    the requestAnimationFrame loop may drive playhead updates from video.currentTime. As a safety net, this fallback also engages if rVFC never fires within an initial grace window (~1000 ms) after play, so the playhead can never freeze permanently. Until the first compositor frame is presented, the playhead follows mediaTime (rVFC) exclusively, so it never leads the on-screen image.

    This prevents the timeline from freezing when the video track ends but audio keeps playing (e.g. video track ends at 10s, audio at 15.4s).


4. Scenario Walkthroughs (User Operations)

A. Play Video

  1. User clicks Play.
  2. The play event listener resets the per-session sync state and triggers updateFrameLoop().
  3. If supported, requestVideoFrameCallback is registered. In parallel, requestAnimationFrame is scheduled recursively.
  4. Until the first rVFC frame is presented, the playhead is driven only by mediaTime; the requestAnimationFrame path does not advance it from video.currentTime (see §3), avoiding a forward race on engines whose playback clock leads presentation at startup.
  5. Thereafter the UI displays continuous updates aligned with compositor frames.

B. Pause Video

  1. User clicks Pause.
  2. The pause event listener triggers handlePause():
    • Cancels active VFC (cancelVideoFrameCallback) and RAF (cancelAnimationFrame) loops.
    • Chooses the synchronization time. When rVFC is fresh (it delivered a frame within the stall threshold), the last presented compositor time lastMediaTime is used — this is the frame actually on screen. Otherwise (rVFC stale or unsupported, e.g. an audio-only tail after the video track ends) it falls back to video.currentTime: \text{syncTime} = \begin{cases} \text{lastMediaTime} & \text{if rVFC is fresh} \\ video.currentTime & \text{otherwise} \end{cases}
    • Performs a final playhead synchronization: \text{clampedFrame} = \max\left(0, \min\left(\lfloor(\text{syncTime} \cdot frameRate) + 0.45\rfloor, \text{totalFrames} - 1\right)\right)
    • Calls setCurrentFrame(clampedFrame) to lock the UI to the frame the user is actually looking at.
    • Snaps the browser playhead to the calculated frame's center time to force the browser compositor to align: \text{currentTime} = \text{calculateFrameCenterTime}(\text{clampedFrame}, \text{frameRate})

Frame-accurate tradeoff: because pause locks to the presented frame (which on Safari lags the audio clock), snapping currentTime back to that frame's center means the small audio segment between the presented frame and the audio position is replayed on resume. This is intentional and matches other frame-accurate tools (e.g. frame.io): landing on the exact displayed frame takes priority over seamless audio. The effect is proportional to the engine's audio/video presentation offset — negligible on Chrome, ~100–300 ms on Safari.


C. Drag Seekbar Head to a Timestamp

  1. User clicks or drags on the seekbar track.
  2. The progress bar computes the click horizontal percentage (P).
  3. The player calculates the target frame: \text{targetFrame} = \text{Math.round}(P \cdot \text{totalFrames})
  4. The player calls seekToFrame(targetFrame):
    • Sets isSeekingRef.current = true to lock the synchronization loop.
    • Calculates the safe center time: safeTargetTime = calculateFrameCenterTime(targetFrame, frameRate).
    • Sets video.currentTime = safeTargetTime.
    • Listens for the browser's native seeked event, then resets isSeekingRef.current = false.

D. Step Frame-by-Frame (Arrow Keys)

  1. User presses ArrowRight (Next Frame) or ArrowLeft (Prev Frame).
  2. The keydown handler intercepts the event, calculates the target frame index (N \pm 1), and calls seekToFrame(targetFrame).
  3. The video seeks to the safe center time of the adjacent frame, rendering it frame-accurately.

E. Pause Video and Create a Comment

  1. User pauses the video (UI locks to integer currentFrame, e.g. 120).
  2. User writes a comment and clicks submit.
  3. The comment creation form reads the currentTime prop, which is computed strictly from the frame index: \text{currentTime} = \frac{\text{currentFrame}}{\text{frameRate}} = \frac{120}{23.976} = 5.005005\text{s}
  4. The API saves the exact frame boundary time 5.005005 in the database.

F. Click Comment to Jump to Comment-Timestamp

  1. User clicks a comment in the sidebar.
  2. The select handler calls:
    videoRef.current.currentTime(comment.second) // e.g., 5.005005s
    videoRef.current.pause()
    
  3. The browser seeks the media element. Once finished, it fires the native seeked event.
  4. The handleExternalSeeked listener in useFramePlayer triggers because isSeekingRef.current is false:
    • Maps video.currentTime back to the frame index: \text{finalFrame} = \lfloor(video.currentTime \cdot frameRate) + 0.45\rfloor = \lfloor(5.005005 \cdot 23.976) + 0.45\rfloor = 120
    • Computes the safe center time: safeCenterTime = calculateFrameCenterTime(120, 23.976) = 5.025859s.
    • Detects if the current time is offset from the center by more than a quarter-frame: \text{Math.abs}(5.005005 - 5.025859) = 0.020854\text{s} > \frac{0.041708}{4}\text{s}
    • Nudges the browser's playhead: video.currentTime = safeCenterTime (5.025859s). This forces the browser to render Frame 120 cleanly.

G. Switch Resolution

1. HLS Streams (hls.js)

  • Immediate Rendition Switch (currentLevel): When the user selects a specific rendition (e.g. 1080p), the player assigns hls.currentLevel = targetIndex rather than nextLevel. This causes hls.js to immediately flush the MSE buffer around currentTime and fetch fragments for the target rendition (matching Frame.io's responsive behavior).
  • Auto Mode: When selecting Auto, hls.currentLevel = -1 is set to re-enable adaptive bitrate (ABR). The player listens to Hls.Events.LEVEL_SWITCHED to update the control bar label dynamically (e.g., Auto (1080p)).
  • Paused Frame-Center Re-Nudge: If the user switches quality while paused on a frame (e.g. Frame 120), the player listens for the subsequent seeked event once the new fragment renders and re-nudges video.currentTime to calculateFrameCenterTime(currentFrame, frameRate). This guarantees zero frame drift across quality switches.

2. Normal MP4 Streams (Native Video)

  1. User selects a different resolution (e.g. 720p or Original).
  2. The player intercepts the click:
    • Captures wasPlaying = !video.paused and the current playhead time: currentT = video.currentTime.
    • Updates the source URL: video.src = res.url.
  3. The player registers a one-shot loadedmetadata event listener on the new source:
    • Restores playhead time: video.currentTime = currentT.
    • Restores playback state (play() if wasPlaying was true) and playback rate.
  4. The seek triggered by video.currentTime = currentT fires a seeked event, which automatically calls handleExternalSeeked, recalculating the frame index and nudging the playhead to the safe frame center on the new media stream.

H. Manual Zoom & Zoom Fit Reset

  1. User adjusts zoom manually using the Zoom In (+) or Zoom Out (-) buttons:
    • Sets hasManuallyZoomed to true.
    • Sets the new manual scale, overriding any auto-scale layout behavior.
  2. User clicks the Fit button:
    • Resets hasManuallyZoomed to false.
    • Triggers the responsive scale recalculation effect, mapping the video scale back to the layout container bounds: \text{scale} = \min\left(\frac{\text{containerSize.width}}{\text{originalWidth}}, \frac{\text{containerSize.height}}{\text{originalHeight}}\right)

5. Backend Specifications

A. Transcode & Metadata Extraction

When a video is uploaded, the @shumai/transcode worker executes an ffprobe command to analyze the source container and streams:

ffprobe -v error -select_streams v:0 -show_entries stream=r_frame_rate,duration,nb_frames -of default=noprint_wrappers=1:nocaveat=1 input.mp4

The extracted metadata details are saved to the AssetMetadata table:

  • frameRate (e.g. 23.976 or 30)
  • duration (the absolute container duration, e.g. 15.42)
  • totalFrames (the physical number of frames in the video stream, e.g. 240)

Note: totalFrames is always the physical video-stream frame count (nb_frames), while duration is the container duration (which can be longer, e.g. when the audio track outlasts the video). The frontend reconciles the two into an effective frame count via the rule in §2.D — it does not blindly trust either value alone.


B. Database Schema

Comments are represented in the Prisma database schema by the Comment model:

model Comment {
  id           String       @id @default(ulid())
  text         String
  second       Decimal?     @db.Decimal(10, 4) // Stores currentFrame / frameRate
  fileId       String
  file         Asset        @relation(fields: [fileId], references: [id], onDelete: Cascade)
  createdAt    DateTime     @default(now())
}
  • Precision: The second column uses Decimal(10, 4) to store sub-millisecond timestamps accurately without rounding or floating-point truncation.