13 KiB
Frame-Accurate Video Player & Comment System: Technical Design Document
This document outlines the technical design, mathematical formulas, synchronization loops, and database schemes that guarantee 100% frame-accurate playback, scrubbing, stepping, and comment navigation in Shumai.
1. Architectural Overview
Shumai's video player is a decoupled, frame-locked system that coordinates three primary components around a single state coordinator (useFramePlayer):
┌───────────────────────────┐
│ useFramePlayer Hook │
│ (State: currentFrame) │
└──────┬─────────────▲──────┘
│ │
1. Set currentTime │ │ 2. Sync frame index
(Frame Center Nudge)│ │ (Floor-with-Epsilon)
▼ │
┌────────────────────────────┴──┐
│ Native HTML5 Video Player │
│ (VideoJS Engine Wrapper) │
└───────────────────────────────┘
▲ ▲
│ │
Stepping / │ │ Comment Selection /
Scrubbing │ │ Timestamp Navigation
│ │
┌───────┴──────┐ ┌──────┴───────┐
│ Seekbar │ │ Comments │
│ (Progress) │ │ (Panel/List) │
└──────────────┘ └──────────────┘
- Native HTML5 Video Player (VideoJS Engine Wrapper): Manages the underlying browser media stream, buffering, audio clock, and presentation.
- Seekbar (Scrubber & Progress Bar): Visualizes playback progress and maps horizontal screen clicks/drags to playhead percentages.
- Comment Area (Sidebar): Renders comment list, displays timestamp labels, captures drawings, and triggers seeks to target frames.
2. Core Mathematical Foundations
To achieve frame-accurate synchronization across different browsers (which use floating-point double precision for playhead timestamps), we utilize three key mathematical formulas:
A. Frame-to-Time Boundary Formula
When generating timestamps for external storage (such as database comments), we calculate the exact start boundary time of the frame. This avoids carrying floating-point errors into the database:
\text{Time}_{\text{boundary}} = \frac{N}{\text{frameRate}}
Where N is the integer frame index ($0$-indexed).
B. Time-to-Frame Conversion (Floor with 0.45 Offset)
When converting a raw player timestamp (t) back to an integer frame index, we use a Floor with 0.45 Offset formula to absorb sub-millisecond clock drift, compositor presentation offsets, and V-Sync boundaries:
\text{Frame} = \lfloor(t \cdot \text{frameRate}) + 0.45\rfloor
- Compositor Lag Guard: Since the browser compositor typically presents a frame slightly before the playhead crosses the exact mathematical start time (due to V-Sync ticks and decoding latency), the browser transitions visually around the half-frame mark. An offset of
0.45shifts the transition boundary to0.55frames, matching user perception. - Seek Buffer Guard: When we seek to the safe center of a frame (
N + 0.5frames), using a0.45offset yieldsN + 0.95, which floors back to the correct frame indexNeven if the browser has minor floating-point seek target deviations.
C. Safe Seek Time (Half-Frame Offset Center)
Browsers can occasionally render the previous frame if the playhead lands exactly on a frame boundary due to float rounding. To guarantee that seeking always lands squarely inside the target frame's presentation window, we seek to the center of the frame:
\text{Time}_{\text{center}} = \left(N \cdot T_d\right) + \frac{T_d}{2} = \frac{N}{\text{frameRate}} + \frac{1}{2 \cdot \text{frameRate}}
Where T_d = \frac{1}{\text{frameRate}} is the duration of a single frame.
3. Active Playback Loops & Dual-Drive Coordination
When playing, the playhead loop coordinates two mechanisms in parallel to prevent timeline freezes and avoid visual jitter:
┌───────────────────┐
│ Playback Started │
└─────────┬─────────┘
│
┌────────────┴────────────┐
│ Start loops: RAF & │
│ VFC (if supported) │
└────────────┬────────────┘
│
┌──────────────┴──────────────┐
│ Is VFC Stalled (> Thresh)? │
└─────┬─────────────────┬─────┘
│ Yes │ No
▼ ▼
┌───────────────────────┐ ┌───────────────────────┐
│ Update UI using │ │ Update UI using │
│ currentTime (RAF) │ │ mediaTime (VFC) │
└───────────────────────┘ └───────────────────────┘
-
requestVideoFrameCallback(rVFC): The primary driver when the video is actively decoding. It fires only when a new video frame is sent to the compositor, updating the UI frame-accurately based onmetadata.mediaTime. -
requestAnimationFrame(RAF): Runs continuously at 60Hz. It acts as the fallback driver. -
Stall Detection: The playhead tracks when the last rVFC callback was fired. If it has been stalled for longer than the dynamic stall threshold:
\text{Threshold} = \max\left(100, \frac{3000}{\text{frameRate}}\right)\text{ ms}the RAF loop takes over playhead updates using
video.currentTime.This prevents the timeline from freezing when the video track ends but audio keeps playing (e.g. video track ends at 10s, audio at 15.4s).
4. Scenario Walkthroughs (User Operations)
A. Play Video
- User clicks Play.
- The
playevent listener triggersupdateFrameLoop(). - If supported,
requestVideoFrameCallbackis registered. In parallel,requestAnimationFrameis scheduled recursively. - UI displays continuous updates aligned with compositor frames.
B. Pause Video
- User clicks Pause.
- The
pauseevent listener triggershandlePause():- Cancels active VFC (
cancelVideoFrameCallback) and RAF (cancelAnimationFrame) loops. - Performs a final playhead synchronization:
\text{clampedFrame} = \max\left(0, \min\left(\lfloor(video.currentTime \cdot frameRate) + 0.45\rfloor, \text{totalFrames} - 1\right)\right) - Calls
setCurrentFrame(clampedFrame)to lock the UI to the correct frame. - Snaps the browser playhead to the calculated frame's center time to force the browser compositor to align:
\text{currentTime} = \text{calculateFrameCenterTime}(\text{clampedFrame}, \text{frameRate})
- Cancels active VFC (
C. Drag Seekbar Head to a Timestamp
- User clicks or drags on the seekbar track.
- The progress bar computes the click horizontal percentage (
P). - The player calculates the target frame:
\text{targetFrame} = \text{Math.round}(P \cdot \text{totalFrames}) - The player calls
seekToFrame(targetFrame):- Sets
isSeekingRef.current = trueto lock the synchronization loop. - Calculates the safe center time:
safeTargetTime = calculateFrameCenterTime(targetFrame, frameRate). - Sets
video.currentTime = safeTargetTime. - Listens for the browser's native
seekedevent, then resetsisSeekingRef.current = false.
- Sets
D. Step Frame-by-Frame (Arrow Keys)
- User presses
ArrowRight(Next Frame) orArrowLeft(Prev Frame). - The keydown handler intercepts the event, calculates the target frame index (
N \pm 1), and callsseekToFrame(targetFrame). - The video seeks to the safe center time of the adjacent frame, rendering it frame-accurately.
E. Pause Video and Create a Comment
- User pauses the video (UI locks to integer
currentFrame, e.g.120). - User writes a comment and clicks submit.
- The comment creation form reads the
currentTimeprop, which is computed strictly from the frame index:\text{currentTime} = \frac{\text{currentFrame}}{\text{frameRate}} = \frac{120}{23.976} = 5.005005\text{s} - The API saves the exact frame boundary time
5.005005in the database.
F. Click Comment to Jump to Comment-Timestamp
- User clicks a comment in the sidebar.
- The select handler calls:
videoRef.current.currentTime(comment.second) // e.g., 5.005005s videoRef.current.pause() - The browser seeks the media element. Once finished, it fires the native
seekedevent. - The
handleExternalSeekedlistener inuseFramePlayertriggers becauseisSeekingRef.currentisfalse:- Maps
video.currentTimeback to the frame index:\text{finalFrame} = \lfloor(video.currentTime \cdot frameRate) + 0.45\rfloor = \lfloor(5.005005 \cdot 23.976) + 0.45\rfloor = 120 - Computes the safe center time:
safeCenterTime = calculateFrameCenterTime(120, 23.976) = 5.025859s. - Detects if the current time is offset from the center by more than a quarter-frame:
\text{Math.abs}(5.005005 - 5.025859) = 0.020854\text{s} > \frac{0.041708}{4}\text{s} - Nudges the browser's playhead:
video.currentTime = safeCenterTime(5.025859s). This forces the browser to render Frame 120 cleanly.
- Maps
G. Switch Resolution
- User selects a different resolution (e.g.
720porOriginal). - The player intercepts the click:
- Captures
wasPlaying = !player.paused()and the current playhead time:currentT = player.currentTime(). - Updates the source URL:
player.src({ type: 'video/mp4', src: res.url }).
- Captures
- The player registers a one-shot
loadedmetadataevent listener on the new source:- Restores playhead time:
player.currentTime(currentT). - Restores playback state (
play()ifwasPlayingwas true) and playback rate.
- Restores playhead time:
- The seek triggered by
player.currentTime(currentT)fires aseekedevent, which automatically callshandleExternalSeeked, recalculating the frame index and nudging the playhead to the safe frame center on the new media stream.
H. Manual Zoom & Zoom Fit Reset
- User adjusts zoom manually using the
Zoom In(+) orZoom Out(-) buttons:- Sets
hasManuallyZoomedtotrue. - Sets the new manual scale, overriding any auto-scale layout behavior.
- Sets
- User clicks the Fit button:
- Resets
hasManuallyZoomedtofalse. - Triggers the responsive scale recalculation effect, mapping the video scale back to the layout container bounds:
\text{scale} = \min\left(\frac{\text{containerSize.width}}{\text{originalWidth}}, \frac{\text{containerSize.height}}{\text{originalHeight}}\right)
- Resets
5. Backend Specifications
A. Transcode & Metadata Extraction
When a video is uploaded, the @shumai/transcode worker executes an ffprobe command to analyze the source container and streams:
ffprobe -v error -select_streams v:0 -show_entries stream=r_frame_rate,duration,nb_frames -of default=noprint_wrappers=1:nocaveat=1 input.mp4
The extracted metadata details are saved to the AssetMetadata table:
frameRate(e.g.23.976or30)duration(the absolute container duration, e.g.15.42)totalFrames(the physical number of frames in the video stream, e.g.240)
B. Database Schema
Comments are represented in the Prisma database schema by the Comment model:
model Comment {
id String @id @default(ulid())
text String
second Decimal? @db.Decimal(10, 4) // Stores currentFrame / frameRate
fileId String
file Asset @relation(fields: [fileId], references: [id], onDelete: Cascade)
createdAt DateTime @default(now())
}
- Precision: The
secondcolumn usesDecimal(10, 4)to store sub-millisecond timestamps accurately without rounding or floating-point truncation.