R&D Report · Interim

AI Camera Match.

Integrating a real, keyed greenscreen actor into an AI-generated Jefferson Memorial background whose camera move matches the live plate — what was tried, what won, what died, what runs next.

Report · 2026-07-03 v0.6.0 · interim

3 stages solved 3 interim / pending 3 dead ends closed 8 research notes 34 evidence proxies · 17 MB

Executive Summary

The pipeline works end-to-end. Keying is broadcast-clean, the locked-shot composite is proven, and the camera problem is solved two ways: pushed to Seedance's interpretive ceiling for look, and hard-locked by construction with local depth-conditioned generation for accuracy. The central law learned:

Every reference-video model we tested repaints the subject (Seedance 2.0, confirmed independently on two platforms). "Keep the person, replace the background" is only achievable by keying the real subject and compositing — so the pipeline splits: real subject (key) + generated background (camera-matched), married in the comp. Seedance is the current BG recommend — a SOTA shootout (Kling, Runway, Veo) is planned before locking it.

Stage	State	Evidence
Keying (CorridorKey)	solved	All 3 plates keyed clean; true-alpha ProRes 4444 delivered (jib)
Locked-shot composite	proven	Statue-reframe comp v1 — §07
Camera match — look-first	model ceiling	Seedance start+end keyframe clamp — §05
Camera match — accuracy-first	proven	LTX depth render-to-real, trajectory locked by construction — §06
Dolly / jib moving comps	interim	Dolly comp over Seedance BG done; move-amount matching open — §07
Relight / harmonization	pending	Build-our-own stack scaffolded, not installed — §09

The Pipeline

Four workflows were tested (or designed) this cycle. W1 is the production path — everything else feeds it or was eliminated reaching it. Arrows show data direction; every generated background ultimately meets the real keyed subject in W1's composite.

Key + Composite — the production path

adopted

Render-to-Real — camera hard-lock

proven

Video-Ref Regenerate

dead end

Obscura Hybrid

planned

solved / adopted works with caveats pending / planned dead end

Strategy	Camera accuracy	Look	Use when
A · Seedance + keyframes	endpoint-clamped, mid-path drifts	10/10	Hero look, forgiving moves
B · Seedance empty-BG preset	interpretive (preset vocabulary)	10/10	Key+composite shots, art-directed plates
C · LTX depth render-to-real	exact, by construction	photoreal, below Seedance	Accuracy-critical camera, local, free

The Three Test Shots

One performer, three camera behaviors: a locked-off medium, a push-in dolly, and a crane-down jib. Together they cover the camera-matching problem space from trivial to hard.

LOCKED · 151f · 2D track confirms static (worst drift 1.56% ≈ handheld micro-motion)

DOLLY · 143f push-in · planar solve (1 marker) → feature-spread ramp measured 1.82×

JIB · 151f crane high→low · planar; move read from footage, no 3D solve

Keying

solved

CorridorKey on the 5090 (BiRefNet hint → neural key + despill → sharp hybrid composite) outputs the subject on flat gray-148 for clean matte extraction. Naive chroma keying failed outright — the warm dark-olive "green" had r≈g. A distance-ramp matte off the gray, plus interior hole-fill for near-gray dress folds, yields the delivered true-alpha ProRes 4444.

Locked key · gray-148 output

Dolly key · re-keyed after byte-verified transfer fix (stale-footage bug caught & killed)

Jib key · full crane range; alpha master: 03-keyed/jeff_jib_key_ALPHA.mov (ProRes 4444, 4K, 246 MB — local)

Keyer validation suite — the keyer was proven on harder cases before touching production footage: flying dark hair mid-motion, fine blonde strands against a dark suit, and full-body dance moves at 4K. Composited on a gradient specifically to expose edge artifacts.

before / afterGreen plate → keyed — split-screen validation, dancer at 4K

sharp hybridSharp-hybrid composite — the keyer's signature pass: neural matte + high-freq detail recovery

hard caseFlying dark hair, mid-motion — the classic keyer killer; strands survive

hard caseFine blonde strands + dark suit — low-contrast edges hold

Matte QC — the hole-fill fix. The silk camisole's near-gray folds sat close to the key gray and punched semi-transparent holes in the body interior. Interior hole-fill (solidify enclosed regions, preserve the soft hair rim) produced the delivered alpha:

first passRaw distance matte — dress folds read as background: interior holes

deliveredAfter interior hole-fill — solid body, hair edge still soft; this matte ships in the ProRes 4444

Pipeline bug worth remembering: the original key run silently keyed stale footage — scp to a fixed filename dropped without error. Fix: dd-over-ssh push with byte-count gate before every key. Upgrade path researched: MatAnyone 2 (−26% MAD) as the single highest-leverage keyer swap.

Camera Match — Seedance Track

model ceiling

Four iterations pushed Seedance 2.0 from "statue flung anywhere" to a pinned trajectory. The law that emerged: Seedance interprets a camera, it never hard-locks one — it is architecturally diffuse-follow. Composition obeys legibility; trajectory obeys nothing but keyframes.

dead endv1 · techhalla method — generic still reference; Seedance re-composed hard-right, wrong camera

leverTextured previz at the solved camera — legibility is the composition lever; grey clay gets ignored, legible geometry lands

partialv2 · legible previz reference — composition lands (statue centered, low angle) but depth direction inverted: our camera pushes in, Seedance widens

bestv4 · start+end keyframe clamp — trajectory pinned at both ends; residual mid-path drift = comp stabilize, not regen. This is Seedance's ceiling. Master: routeB_seedance4_keyframes.mp4

Stress test · previz camera vs clamped Seedance, side by side

Camera Match — Hard-Lock Track

The other side of the deciding axis: structure as conditioning. Two engines tested — one died on look, one is the proven accuracy path. (Naming note: the repo's research docs use "Route A" for Uni3C reference-copy — untested to date; the Wan test below was run as a hard-lock engine trial.)

dead endWan2.2 Fun Control (5090, depth) — camera locked by construction but look = 3/10, unusable for finals. Kept only as a possible guide layer in an untested fusion.

provenLTX-2 depth IC-LoRA render-to-real (5090) — Blender dolly previz in, photoreal rotunda out, trajectory identical frame-for-frame. 220s local, free. Statue blobby (crude proxy + 512×288) — levers identified. Reverses the Jun-27 LTX demotion only under depth conditioning — free generation stays demoted. See research note 2026-07-02.

Structural guide · Blender clay rotunda, 143f measured 1.82× push, real parallax baked in (start / mid / end)

Photorealized · same frames after depth-conditioned LTX — camera is the model's input, not its guess

Zoom vs dolly — the parallax test. Seedance's pushIn preset renders an optical zoom (uniform magnification, no parallax). Its superDollyIn preset — and the LTX depth path by construction — produce true forward translation: foreground columns slide past frame edges at a different rate than the back wall. Always inspect start/mid/end frames for parallax before accepting a "dolly."

zoomSeedance pushIn — magnification only; flagged in review, replaced

dollySeedance superDollyIn — genuine travel: camera passes between the foreground columns

Composites

interim

Real keyed subject over generated backgrounds. The locked shot is proven; the dolly comps exist in two BG flavors; the jib comp is next (its empty Seedance BG already exists — §08).

provenLocked comp v1 — statue-reframe: subject at native scale, Jefferson plate reframed behind her at human scale

interimDolly comp · static BG + ramp zoom — BG is a still given the measured 1.82× push; no BG parallax

interimDolly comp · Seedance BG — moving generated BG behind untouched subject; BG move amount > subject move (matching open)

Locked BG plate (Nano Banana, option A — chosen)

driftedLocked-shot Seedance BG — "static" generation drifts; needs 2D stabilize or regen before the locked comp upgrades to a moving-grade BG (open item)

Dolly wide eye-level plate — statue right, inscription wall legible (nailed first pass)

okJib end low-hero plate

failedJib start high-angle — Nano Banana snapped to eye-level and invented a person; stills won't hold composition (known law). Redo or derive from Blender.

The Regenerate Dead End — and Its Useful Half

Feeding Seedance the raw greenscreen video + a reference BG image + "replace the background, keep her exactly" produced a repainted stranger on a random stage. Tested on two platforms (magnific + Higgsfield manual runs) — same law both times.

dead endVideo-ref BG-replace (dolly) — output unrelated to the plate; subject repainted

confirms lawHiggsfield manual jib run 1 — she's in the rotunda with the crane move, but the model repainted her face/wardrobe. Second platform, same law.

usableHiggsfield manual jib run 2 — empty rotunda with the real crane move: a ready-made jib BG plate for the key+composite path

Planned test (not yet run): LTX-2.3 Obscura Remova — generate Seedance video-ref at 4K (accurate camera, repainted person), remove the AI person with Obscura, mask the 1080p reconstruction back into the native-4K frame (the real keyed subject re-covers almost all of it), composite the real performer. Best look + camera accuracy if Obscura's temporal reconstruction holds.

Relight

pending

Decision locked after a Beeble/SwitchLight build-vs-buy analysis: build our own. IC-Light v1 FBC + DiffusionLight (single-frame HDRI lock — the anti-shimmer trick for our lighting-static BGs) + Marigold, on the 5090. Open stack ≈ 80–90% of SwitchLight on stills; Beeble API ($0.10/gen) kept as the reference bar only. Scaffolded at relight/, ~15 GB install not yet run.

Model Capability Map

Model / method	Camera control	Preserves real footage?	Look	Where it fits
Seedance 2.0 (keyframes)	endpoint clamp; presets; interprets	no — repaints	10/10	Pipeline base for BG look
Seedance 2.0 (video ref)	copies plate move well	no — repaints subject	10/10	BG-only gen; Obscura candidate
LTX-2 depth IC-LoRA (local)	exact — structure conditioned	n/a (generates BG)	photoreal, sub-Seedance	Accuracy-critical BG
Wan2.2 Fun Control (local)	exact — depth conditioned	n/a	3/10 local	Rejected; possible guide layer
Uni3C (Wan2.2 cloud)	geometry-aware ref copy	no — re-renders	6–7 cloud	Repo "Route A" — untested fusion ingredient
Kling (Motion Control)	ref-follow	no — regenerates	high	Shootout planned — head-to-head vs Seedance
Runway (Gen-4 / Aleph)	prompt + ref modes	no — regenerates	high	Shootout planned
Veo 3.x	prompt-level	no — regenerates	high	Shootout planned
LTX-2.3 Obscura Remova	n/a (V2V removal)	yes — reconstructs behind occluder	1080p cap	Planned: strip AI person from Seedance ref output
CorridorKey (5090)	n/a	yes — it IS the footage	broadcast-clean	The subject track

Locked Decisions

Jun 26 — Route B (camera-sharing / render-to-real) is the primary architecture; background is AI-generated, not scanned (no license-clear Jefferson interior asset exists; splat = fallback).
Jun 27 — Local Wan2.2 Fun Control rejected for finals (look 3/10) despite perfect camera lock.
Jun 28 — Seedance 2.0 adopted as pipeline base (look 10/10); engineer the camera around it. Legible textured previz replaces grey clay as the composition lever.
Jun 29 — Camera match declared solved to model ceiling via start+end keyframe clamp; residual drift handled in comp, not regen. Relight = build-our-own (IC-Light FBC + DiffusionLight + Marigold), Beeble API as reference bar.
Jun 29 — "Keep the subject" = key + composite, never regenerate. Seedance used BG-only for moving shots.
Jun 30 — Camera presets audited for parallax: pushIn = zoom (rejected); superDollyIn = true dolly (adopted). Second platform confirmed video-ref repaints subjects.
Jul 2 — Local render-to-real proven: LTX-2 depth IC-LoRA hard-locks the measured dolly trajectory (512×288, 220s). Adopted as the accuracy-first BG strategy — under conditioning only; LTX free generation stays demoted per the Jun-27 benchmark.
Delivery standard — layered EXR package for After Effects (camera via Blender .jsx export); external composite in Fusion/Nuke/AE.

Open Items — What Runs Next

Item	State	Notes
SOTA BG-gen shootout	planned	Same dolly/jib BG brief across Kling, Runway, Veo vs Seedance — Seedance is the working recommend pending head-to-head
Obscura Remova test	planned	Strip AI person from Seedance video-ref output; mask-back into 4K; composite real subject
Jib composite	ready	Empty jib BG exists (Higgsfield run 2) + clean jib key + alpha master — assemble
Jib high-angle plate	redo	NB still failed; derive from Blender previz or regenerate
Dolly move-amount match	open	Seedance superDollyIn travels further than the plate's 1.82×; gentler re-gen or partial-range comp
LTX r2r quality pass	open	768×432+, full 143f, real Jefferson statue mesh (Sketchfab/Smithsonian) for legible silhouette
Relight install + first test	gated	`relight/SETUP-5090.md` (~15 GB); A/B CorridorKey ComfyUI node vs key.sh
Locked-shot Seedance BG	open	Static gen drifted (§07); 2D-stabilize or regen; then rebuild locked comp on moving-grade BG
Fusion experiment	untested	Cloud-Wan depth hard-lock guide + Seedance restyle — proposed lever if clamp ceiling isn't enough
Keyer upgrade	research	MatAnyone 2 (−26% MAD); verify weights/licenses
Higgsfield API auth	pending	Manual runs worked; automated path awaits auth callback

Assets & Masters

Everything on this page is a 720p H.264 proxy encoded for handoff (site filenames may shorten master names — each caption cites the master where they differ). Full-resolution masters live in the project repo on the workstation:

~/CodeMonkey/AI_Camera_Match/ — project root: plates (4K .mov), ISA.md (system of record), 8 research notes in research/
03-keyed/jeff_jib_key_ALPHA.mov — ProRes 4444 true-alpha, 4K, 246 MB
02-camera-matched/routeB_seedance4_keyframes.mp4 — best Seedance camera-match master (site proxy: routeB_v4_keyframes.mp4)
06-render-to-real/dolly_r2r_v1_workflow.json — reproducible ComfyUI workflow (LTX-2 depth IC-LoRA, 5090)
camera-solve/ — 2D tracks, dolly push-ramp JSON, Blender previz scenes (Mac + 5090 copies)
_manual_dolly_bgreplace/ · _manual_jib_bgreplace/ — self-serve Seedance test packs with prompts
Hardware dependency: keying, LTX render-to-real, and the pending relight stage run on a dedicated RTX 5090 node (ComfyUI, wake-on-LAN, one-workload-at-a-time GPU profiles). Runbooks: relight/SETUP-5090.md, workflows/BUILD-NOTES.md, camera-solve/pc/.
Evidence chain: verdicts trace to dated research notes in research/ — 06-25 cloud rev-2 · 06-26 render-to-real rev-3 · 06-26 real-locations · 06-27 Route A/Wan · 06-28 Seedance camera-lock · 06-29 Beeble build-vs-buy · 06-29 Blender low-poly · 07-02 LTX depth r2r (+ README Log).