* feat(webui): switch from video.js to hls.js and native video player * fix(webui): prevent stale seek hijacking and guard native HLS compare mode
19 KiB
Frame-Accurate Video Player & Comment System: Technical Design Document
This document outlines the technical design, mathematical formulas, synchronization loops, and database schemes that guarantee 100% frame-accurate playback, scrubbing, stepping, and comment navigation in Shumai.
1. Architectural Overview
Shumai's video player is a decoupled, frame-locked system that coordinates three primary components around a single state coordinator (useFramePlayer):
┌───────────────────────────┐
│ useFramePlayer Hook │
│ (State: currentFrame) │
└──────┬─────────────▲──────┘
│ │
1. Set currentTime │ │ 2. Sync frame index
(Frame Center Nudge)│ │ (Floor-with-Epsilon)
▼ │
┌────────────────────────────┴──┐
│ Native HTML5 Video Player │
│ (Hls.js / Native MSE) │
└───────────────────────────────┘
▲ ▲
│ │
Stepping / │ │ Comment Selection /
Scrubbing │ │ Timestamp Navigation
│ │
┌───────┴──────┐ ┌──────┴───────┐
│ Seekbar │ │ Comments │
│ (Progress) │ │ (Panel/List) │
└──────────────┘ └──────────────┘
- Native HTML5 Video Player (Hls.js / Native MSE): Manages the underlying browser media stream, buffering, audio clock, and presentation. Direct
<video>element with zero library wrappers for MP4, and lightweighthls.jsfor adaptive HLS streams. - Seekbar (Scrubber & Progress Bar): Visualizes playback progress and maps horizontal screen clicks/drags to playhead percentages.
- Comment Area (Sidebar): Renders comment list, displays timestamp labels, captures drawings, and triggers seeks to target frames.
2. Core Mathematical Foundations
To achieve frame-accurate synchronization across different browsers (which use floating-point double precision for playhead timestamps), we utilize three key mathematical formulas:
A. Frame-to-Time Boundary Formula
When generating timestamps for external storage (such as database comments), we calculate the exact start boundary time of the frame. This avoids carrying floating-point errors into the database:
\text{Time}_{\text{boundary}} = \frac{N}{\text{frameRate}}
Where N is the integer frame index ($0$-indexed).
B. Time-to-Frame Conversion (Floor with 0.45 Offset)
When converting a raw player timestamp (t) back to an integer frame index, we use a Floor with 0.45 Offset formula to absorb sub-millisecond clock drift, compositor presentation offsets, and V-Sync boundaries:
\text{Frame} = \lfloor(t \cdot \text{frameRate}) + 0.45\rfloor
- Compositor Lag Guard: Since the browser compositor typically presents a frame slightly before the playhead crosses the exact mathematical start time (due to V-Sync ticks and decoding latency), the browser transitions visually around the half-frame mark. An offset of
0.45shifts the transition boundary to0.55frames, matching user perception. - Seek Buffer Guard: When we seek to the safe center of a frame (
N + 0.5frames), using a0.45offset yieldsN + 0.95, which floors back to the correct frame indexNeven if the browser has minor floating-point seek target deviations.
C. Safe Seek Time (Half-Frame Offset Center)
Browsers can occasionally render the previous frame if the playhead lands exactly on a frame boundary due to float rounding. To guarantee that seeking always lands squarely inside the target frame's presentation window, we seek to the center of the frame:
\text{Time}_{\text{center}} = \left(N \cdot T_d\right) + \frac{T_d}{2} = \frac{N}{\text{frameRate}} + \frac{1}{2 \cdot \text{frameRate}}
Where T_d = \frac{1}{\text{frameRate}} is the duration of a single frame.
D. Effective Frame Count Derivation (Container vs. Video Stream)
The player's totalFrames governs the frame readout (N / M fr), the seekbar extent, frame-stepping bounds, and the playback clamp. It is not always the same as the container's frame count, because a container's duration can exceed its video stream — most commonly when the audio track is slightly longer than the video track (AAC padding and the fixed 1024-samples-per-frame granularity mean audio rarely ends on the exact video timestamp), or from container overhead.
Deriving the count naively from the container duration (\lfloor \text{containerDuration} \cdot \text{frameRate} \rceil) invents phantom trailing frames the video stream does not contain. Whether those extra frames should be counted depends on whether the playhead can actually reach them during playback, which is governed by the same stall threshold used by the playback loop (§3):
\text{Threshold}_{s} = \frac{\max\left(100, \frac{3000}{\text{frameRate}}\right)}{1000}\text{ seconds}
Let \text{tail} = \text{containerDuration} - \frac{\text{dbTotalFrames}}{\text{frameRate}} (where dbTotalFrames is the video stream's nb_frames). Then:
$$\text{totalFrames} = \begin{cases} \lfloor \text{containerDuration} \cdot \text{frameRate} \rceil & \text{if tail} > \text{Threshold}_{s}\ \text{(reachable — keep the tail)} \ \text{dbTotalFrames} & \text{otherwise (unreachable — drop phantom frames)} \ \lfloor \text{containerDuration} \cdot \text{frameRate} \rceil & \text{if dbTotalFrames} \le 0\ \text{(nb_frames missing)} \end{cases}$$
- Small tail (tail ≤ threshold): The RAF fallback never activates (§3), so the playhead physically cannot enter the tail. Counting those frames would strand the readout short of its own maximum (e.g. a 268-frame video in a 270-frame container would show
267 / 269and stop at 267). We therefore use the accurate video-stream count →267 / 267. The trailing audio still plays out to the container end on the native element; it is simply not represented as extra scrubbable frames. - Large tail (tail > threshold): The playhead does traverse the audio-only region (the RAF fallback drives it from
video.currentTime, §3), so the container-derived count is correct and the readout reaches its maximum at the natural end (e.g. a 150-frame video with a 6.5s audio tail shows194 / 194).
This rule is implemented by resolveTotalFrames in viewers/utils.ts, which shares stallThresholdMs with useFramePlayer so the frame-count boundary and the runtime playhead-reachability boundary can never drift apart.
3. Active Playback Loops & Dual-Drive Coordination
When playing, the playhead loop coordinates two mechanisms in parallel to prevent timeline freezes and avoid visual jitter:
┌───────────────────┐
│ Playback Started │
└─────────┬─────────┘
│
┌────────────┴────────────┐
│ Start loops: RAF & │
│ VFC (if supported) │
└────────────┬────────────┘
│
┌──────────────┴──────────────┐
│ Is VFC Stalled (> Thresh)? │
└─────┬─────────────────┬─────┘
│ Yes │ No
▼ ▼
┌───────────────────────┐ ┌───────────────────────┐
│ Update UI using │ │ Update UI using │
│ currentTime (RAF) │ │ mediaTime (VFC) │
└───────────────────────┘ └───────────────────────┘
-
requestVideoFrameCallback(rVFC): The primary driver when the video is actively decoding. It fires only when a new video frame is sent to the compositor, updating the UI frame-accurately based onmetadata.mediaTime. -
requestAnimationFrame(RAF): Runs continuously at 60Hz. It acts as the fallback driver. -
Compositor-First Gating & Stall Detection:
video.currentTimeis the playback (audio) clock, and on some engines (notably Safari/WebKit) it runs ahead of the frame actually presented on screen right after playback starts. Trusting it before the compositor catches up would race the playhead forward and then snap it backward when the first rVFC (mediaTime) arrives. To prevent this, eachplayresets the per-session sync state, and the playhead is driven fromvideo.currentTimeonly whenrequestVideoFrameCallbackis unsupported, or rVFC has delivered a frame this session and has since stalled beyond the dynamic stall threshold:\text{Threshold} = \max\left(100, \frac{3000}{\text{frameRate}}\right)\text{ ms}the
requestAnimationFrameloop may drive playhead updates fromvideo.currentTime. As a safety net, this fallback also engages if rVFC never fires within an initial grace window (~1000 ms) after play, so the playhead can never freeze permanently. Until the first compositor frame is presented, the playhead followsmediaTime(rVFC) exclusively, so it never leads the on-screen image.This prevents the timeline from freezing when the video track ends but audio keeps playing (e.g. video track ends at 10s, audio at 15.4s).
4. Scenario Walkthroughs (User Operations)
A. Play Video
- User clicks Play.
- The
playevent listener resets the per-session sync state and triggersupdateFrameLoop(). - If supported,
requestVideoFrameCallbackis registered. In parallel,requestAnimationFrameis scheduled recursively. - Until the first rVFC frame is presented, the playhead is driven only by
mediaTime; therequestAnimationFramepath does not advance it fromvideo.currentTime(see §3), avoiding a forward race on engines whose playback clock leads presentation at startup. - Thereafter the UI displays continuous updates aligned with compositor frames.
B. Pause Video
- User clicks Pause.
- The
pauseevent listener triggershandlePause():- Cancels active VFC (
cancelVideoFrameCallback) and RAF (cancelAnimationFrame) loops. - Chooses the synchronization time. When rVFC is fresh (it delivered a frame within the stall threshold), the last presented compositor time
lastMediaTimeis used — this is the frame actually on screen. Otherwise (rVFC stale or unsupported, e.g. an audio-only tail after the video track ends) it falls back tovideo.currentTime:\text{syncTime} = \begin{cases} \text{lastMediaTime} & \text{if rVFC is fresh} \\ video.currentTime & \text{otherwise} \end{cases} - Performs a final playhead synchronization:
\text{clampedFrame} = \max\left(0, \min\left(\lfloor(\text{syncTime} \cdot frameRate) + 0.45\rfloor, \text{totalFrames} - 1\right)\right) - Calls
setCurrentFrame(clampedFrame)to lock the UI to the frame the user is actually looking at. - Snaps the browser playhead to the calculated frame's center time to force the browser compositor to align:
\text{currentTime} = \text{calculateFrameCenterTime}(\text{clampedFrame}, \text{frameRate})
- Cancels active VFC (
Frame-accurate tradeoff: because pause locks to the presented frame (which on Safari lags the audio clock), snapping
currentTimeback to that frame's center means the small audio segment between the presented frame and the audio position is replayed on resume. This is intentional and matches other frame-accurate tools (e.g. frame.io): landing on the exact displayed frame takes priority over seamless audio. The effect is proportional to the engine's audio/video presentation offset — negligible on Chrome, ~100–300 ms on Safari.
C. Drag Seekbar Head to a Timestamp
- User clicks or drags on the seekbar track.
- The progress bar computes the click horizontal percentage (
P). - The player calculates the target frame:
\text{targetFrame} = \text{Math.round}(P \cdot \text{totalFrames}) - The player calls
seekToFrame(targetFrame):- Sets
isSeekingRef.current = trueto lock the synchronization loop. - Calculates the safe center time:
safeTargetTime = calculateFrameCenterTime(targetFrame, frameRate). - Sets
video.currentTime = safeTargetTime. - Listens for the browser's native
seekedevent, then resetsisSeekingRef.current = false.
- Sets
D. Step Frame-by-Frame (Arrow Keys)
- User presses
ArrowRight(Next Frame) orArrowLeft(Prev Frame). - The keydown handler intercepts the event, calculates the target frame index (
N \pm 1), and callsseekToFrame(targetFrame). - The video seeks to the safe center time of the adjacent frame, rendering it frame-accurately.
E. Pause Video and Create a Comment
- User pauses the video (UI locks to integer
currentFrame, e.g.120). - User writes a comment and clicks submit.
- The comment creation form reads the
currentTimeprop, which is computed strictly from the frame index:\text{currentTime} = \frac{\text{currentFrame}}{\text{frameRate}} = \frac{120}{23.976} = 5.005005\text{s} - The API saves the exact frame boundary time
5.005005in the database.
F. Click Comment to Jump to Comment-Timestamp
- User clicks a comment in the sidebar.
- The select handler calls:
videoRef.current.currentTime(comment.second) // e.g., 5.005005s videoRef.current.pause() - The browser seeks the media element. Once finished, it fires the native
seekedevent. - The
handleExternalSeekedlistener inuseFramePlayertriggers becauseisSeekingRef.currentisfalse:- Maps
video.currentTimeback to the frame index:\text{finalFrame} = \lfloor(video.currentTime \cdot frameRate) + 0.45\rfloor = \lfloor(5.005005 \cdot 23.976) + 0.45\rfloor = 120 - Computes the safe center time:
safeCenterTime = calculateFrameCenterTime(120, 23.976) = 5.025859s. - Detects if the current time is offset from the center by more than a quarter-frame:
\text{Math.abs}(5.005005 - 5.025859) = 0.020854\text{s} > \frac{0.041708}{4}\text{s} - Nudges the browser's playhead:
video.currentTime = safeCenterTime(5.025859s). This forces the browser to render Frame 120 cleanly.
- Maps
G. Switch Resolution
1. HLS Streams (hls.js)
- Immediate Rendition Switch (
currentLevel): When the user selects a specific rendition (e.g.1080p), the player assignshls.currentLevel = targetIndexrather thannextLevel. This causeshls.jsto immediately flush the MSE buffer aroundcurrentTimeand fetch fragments for the target rendition (matching Frame.io's responsive behavior). - Auto Mode: When selecting
Auto,hls.currentLevel = -1is set to re-enable adaptive bitrate (ABR). The player listens toHls.Events.LEVEL_SWITCHEDto update the control bar label dynamically (e.g.,Auto (1080p)). - Paused Frame-Center Re-Nudge: If the user switches quality while paused on a frame (e.g. Frame 120), the player listens for the subsequent
seekedevent once the new fragment renders and re-nudgesvideo.currentTimetocalculateFrameCenterTime(currentFrame, frameRate). This guarantees zero frame drift across quality switches.
2. Normal MP4 Streams (Native Video)
- User selects a different resolution (e.g.
720porOriginal). - The player intercepts the click:
- Captures
wasPlaying = !video.pausedand the current playhead time:currentT = video.currentTime. - Updates the source URL:
video.src = res.url.
- Captures
- The player registers a one-shot
loadedmetadataevent listener on the new source:- Restores playhead time:
video.currentTime = currentT. - Restores playback state (
play()ifwasPlayingwas true) and playback rate.
- Restores playhead time:
- The seek triggered by
video.currentTime = currentTfires aseekedevent, which automatically callshandleExternalSeeked, recalculating the frame index and nudging the playhead to the safe frame center on the new media stream.
H. Manual Zoom & Zoom Fit Reset
- User adjusts zoom manually using the
Zoom In(+) orZoom Out(-) buttons:- Sets
hasManuallyZoomedtotrue. - Sets the new manual scale, overriding any auto-scale layout behavior.
- Sets
- User clicks the Fit button:
- Resets
hasManuallyZoomedtofalse. - Triggers the responsive scale recalculation effect, mapping the video scale back to the layout container bounds:
\text{scale} = \min\left(\frac{\text{containerSize.width}}{\text{originalWidth}}, \frac{\text{containerSize.height}}{\text{originalHeight}}\right)
- Resets
5. Backend Specifications
A. Transcode & Metadata Extraction
When a video is uploaded, the @shumai/transcode worker executes an ffprobe command to analyze the source container and streams:
ffprobe -v error -select_streams v:0 -show_entries stream=r_frame_rate,duration,nb_frames -of default=noprint_wrappers=1:nocaveat=1 input.mp4
The extracted metadata details are saved to the AssetMetadata table:
frameRate(e.g.23.976or30)duration(the absolute container duration, e.g.15.42)totalFrames(the physical number of frames in the video stream, e.g.240)
Note:
totalFramesis always the physical video-stream frame count (nb_frames), whiledurationis the container duration (which can be longer, e.g. when the audio track outlasts the video). The frontend reconciles the two into an effective frame count via the rule in §2.D — it does not blindly trust either value alone.
B. Database Schema
Comments are represented in the Prisma database schema by the Comment model:
model Comment {
id String @id @default(ulid())
text String
second Decimal? @db.Decimal(10, 4) // Stores currentFrame / frameRate
fileId String
file Asset @relation(fields: [fileId], references: [id], onDelete: Cascade)
createdAt DateTime @default(now())
}
- Precision: The
secondcolumn usesDecimal(10, 4)to store sub-millisecond timestamps accurately without rounding or floating-point truncation.