74 lines
4.5 KiB
Markdown
74 lines
4.5 KiB
Markdown
# Object Matte: comparison plan (not measured yet)
|
||||
|
|
|
|||
|
|
**Status: untested.** There is no After Effects 26.2+ comparison of Object Matte yet. Nothing on
|
|||
|
|
this page is a measurement. It lists what to compare once an After Effects reference session
|
|||
|
|
is available, so the first run can be repeated and scored without guesswork.
|
|||
|
|
|
|||
|
|
EffectCraft side, as of #783: the Object Matte effect (`ec.matte.objectmatte`), the Object Matte
|
|||
|
|
tool in the Composition viewer, `objectMatte.apply` / `.propagate` / `.freeze` /
|
|||
|
|
`.status`, and segmentations kept in the disk cache. The point (and an optional box) is
|
|||
|
|
segmented by Roto Brush's engine: MobileSAM when it is installed (Settings ▸ Roto Brush, with
|
|||
|
|
the box as the model's box prompt), otherwise the classical graph cut. The result is tracked
|
|||
|
|
frame by frame with optical flow, and the model re-prompts each frame from the carried matte.
|
|||
|
|
Record which engine was active, because the two are separate cases.
|
|||
|
|
|
|||
|
|
## Test clips
|
|||
|
|
|
|||
|
|
All original, generated or openly licensed; none of them goes into the repository.
|
|||
|
|
|
|||
|
|
1. **Synthetic disk**: the moving textured disk over a textured background from
|
|||
|
|
`crates/engine/src/tests_roto.rs` (`draw(center(f))`, 320×240, 25 fps, 25 frames, radius
|
|||
|
|
40 px, about 4 px a frame to the right). Export it as a PNG sequence. Its ground truth is
|
|||
|
|
analytic (`gt(center(f))`).
|
|||
|
|
2. **Synthetic disk with a jump**: the same disk with `jumping(f)` (it jumps at frame 8). This
|
|||
|
|
checks how each tool handles losing the object.
|
|||
|
|
3. **Two objects**: the disk plus a second still, differently coloured square that touches it
|
|||
|
|
on some frames, prompted once with a point only and once with a box around the disk. This
|
|||
|
|
checks the box prompt.
|
|||
|
|
4. **Real footage**: one openly licensed clip (CC0 or CC BY) with a person or object moving
|
|||
|
|
against clutter, at least 1080p and 5 s long. Record its URL, licence and SHA-256 next to the
|
|||
|
|
results, and keep the clip itself out of the repository.
|
|||
|
|
|
|||
|
|
## Prompt and settings
|
|||
|
|
|
|||
|
|
- Reference frame 0. The point at the object's centre, in layer pixels (clip 1: `[90, 110]`).
|
|||
|
|
The box, where used, is `[centre − 55, centre + 55]` on each axis.
|
|||
|
|
- Track forward to the last frame (Frames After = clip length − 1, Frames Before = 0).
|
|||
|
|
- Matte refinement at its defaults on both sides: Feather 0, Contrast 0, Shift Edge 0, no
|
|||
|
|
Invert. Output the matte itself (View ▸ Matte in EffectCraft, the alpha channel in After
|
|||
|
|
Effects) as a full-resolution PNG sequence.
|
|||
|
|
- Record the After Effects build, the OS, the GPU, and the exact click position in After
|
|||
|
|
Effects. It is a viewer click, so convert it to layer pixels with the comp's zoom.
|
|||
|
|
|
|||
|
|
Keep the After Effects mattes in gitignored `plan/aftereffects/ref/object-matte/`. Only an
|
|||
|
|
aggregate results file (like `keyframe-ease.json`) and the comparison script may be committed.
|
|||
|
|
|
|||
|
|
## Metrics (per frame, then mean / worst)
|
|||
|
|
|
|||
|
|
- **IoU against After Effects**: binarize both mattes at alpha 0.5, then intersection over
|
|||
|
|
union. This is the headline number. `effectcraft_track::roto::iou` computes it, and
|
|||
|
|
`roto.status` already reports it for Roto Brush through `compareTo`; Object Matte needs the
|
|||
|
|
same for exported mattes.
|
|||
|
|
- **IoU against ground truth** for the synthetic clips, for both applications. It tells
|
|||
|
|
"differs from After Effects" apart from "worse than After Effects".
|
|||
|
|
- **Boundary error**: the mean and 95th-percentile distance (px) between the two mattes' 0.5
|
|||
|
|
contours, which shows edge disagreement that IoU dilutes on large objects.
|
|||
|
|
- **Soft-edge error**: the mean absolute alpha difference inside a 3 px band around the After
|
|||
|
|
Effects contour.
|
|||
|
|
- **Temporal stability**: the frame-to-frame IoU change of each matte on clip 1, where the
|
|||
|
|
object moves rigidly, so jitter shows.
|
|||
|
|
- **Drift**: the frame where IoU against ground truth first drops below 0.9 (clips 1 and 2).
|
|||
|
|
- **Time**: seconds per frame to track the clip, warm and cold, on the same machine, with the
|
|||
|
|
disk cache on and off.
|
|||
|
|
|
|||
|
|
Proposed pass bar (a maintainer decision, not yet agreed): mean IoU against After Effects
|
|||
|
|
≥ 0.95 and worst ≥ 0.90 on clips 1 and 3, with MobileSAM. With the classical engine the numbers
|
|||
|
|
are reported but not gated.
|
|||
|
|
|
|||
|
|
## Reproduce (to write)
|
|||
|
|
|
|||
|
|
The script does not exist yet. It should: generate the synthetic clips; run EffectCraft
|
|||
|
|
headless (`effectcraft-cli run … objectMatte.apply … objectMatte.propagate {"wait":true}`, then
|
|||
|
|
render the matte); drive After Effects through ExtendScript or AEsync to apply Object Matte with
|
|||
|
|
the same prompt and render its alpha; then compute the metrics above into a JSON results file.
|