How do you build a local AI VFX shot pipeline with mask + driving video + control nets + reference images (Wan VACE/Phantom merge), and what fails?

Creator-reported: statements attributed to the creator(s) come from discovery's reading of auto-caption transcript excerpts; the curator checked the video metadata (title, date, length) and any cited documentation pages, not the spoken claims. Nothing was tested; promotional or sponsored framing applies where stated. Mickmumpitz (2026-03-30) uses a community merge of Wan 2.1 VACE (in/outpainting guided by depth, Canny edges, pose) with Phantom (up to four reference images for identity), published as Inner-Reflections/Wan2.1_VACE_Phantom. Building blocks: (1) black-and-white mask (only white is regenerated), (2) driving video = modified plate + control signal, (3) reference images (one = start frame; more = style freedom), (4) detailed prompt (LLM-"proofed" but without camera description). Controls: tracking markers drawn as white squares let the model reproduce camera motion; Canny keeps exact detail, depth allows more new detail; OpenPose plus a back-view reference kept a creature consistent when turning; leaving the mask area without control yields clean plates. Practical settings: FP8 model where VRAM allows else GGUF; 720p max, 121 frames (~5 s) in the free version; ~10 minutes per shot on an RTX 4090; Sage Attention roughly halves time; preview with latent2RGB to abort bad runs early; NVIDIA RTX Super Resolution to 4K afterwards. Failure notes: full-frame mask improves interaction (splashes, reflections, shadows on him) but changes his identity, so exclude face/upper body and feather the feet; mask shape biases output (ball-shaped mask + ball reference -> literal fireball; fixed by drawing Canny flame lines); fully dark driving frames let the model invent a new camera move (fixed by Canny outlines blended at frame edges); a moving black mask crocodile floated and vanished when the mask stopped. Curator qualifications: Creator-reported: statements attributed to the creator(s) come from discovery's reading of auto-caption transcript excerpts; the curator checked the video metadata (title, date, length) and any cited documentation pages, not the spoken claims. Nothing was tested; promotional or sponsored framing applies where stated. Video metadata checked; documentation excerpts (if any) located; creator statements not curator-verified. Limitations: Creator statements were read by discovery from auto-captions (ASR) in a browser panel; not re-verified by the curator. ASR: "vase"=VACE, "Kenny"=Canny, "that sky model"=Phantom (inferred from the HF repo name). Merge licence not read (Wan 2.1 base Apache-2.0 per common knowledge, unverified here). Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.

{
  "citations": [
    {
      "url": "https://www.youtube.com/watch?v=_n0ir5V5tX4",
      "note": "0:00-2:00; 2:00; 10:03; 12:03; 16:07; search result"
    },
    {
      "url": "https://huggingface.co/api/models?search=vace-phantom",
      "note": "0:00-2:00; 2:00; 10:03; 12:03; 16:07; search result"
    }
  ]
}