Video generation

Text/image/video-to-video, motion control, long-form consistency and model pipelines

• articleWhat is Kijai's ComfyUI-WanVideoWrapper for, when should I prefer native Wan nodes, and what does it pin or change about VRAM?
  Problem. Choosing between ComfyUI's native Wan support and the wrapper node pack, and avoiding VRAM regressions after updates. Conditions. Repo kijai/ComfyUI-WanVideoWrapper at commit 088128b2 (last push 2026-05-24; not archived), pyproject version 1.4.7, Apache-2.0 code. Method. The README says to use native nodes unless the model or feature is not yet available natively (wrapper is a place to test new models quickly). Install by cloning into custom_nodes and pip-installing requirements.txt (ftfy, accelerate>=1.2.1, einops, diffusers>=0.33.0, peft>=0.17.0, sentencepiece>=0.2.0, protobuf, pyloudnorm, gguf>=0.17.1, opencv-python, scipy). Models: text encoders to models/text_encoders, CLIP vision to clip_vision, transformer to diffusion_models, VAE to vae; fp8-scaled and GGUF variants are named; the wrapper can also use native text-encoder and clip-vision loaders. A noted behaviour change: unmerged LoRA weights are now held as buffers inside blocks, so with block swap the block size g...
• articleHow do I chain Wan 2.2 image-to-video clips by extracting the last frame without quality loss, and when does ComfyUI rerun only the extraction step?
  Problem. Making videos longer than one generation by seeding each segment with the previous segment's final frame. Conditions. Example workflow doc in radiatingreverberations/comfyui-workflows (MIT; repo last pushed 2025-08-08; fetched from main, no commit pin), needs ComfyUI-VideoHelperSuite. Method. Take the last frame straight after VAE Decode via a Select Images node: set indexes to all frames but the last for the video file, and the last frame alone for a lossless image saved separately; the last frame is excluded from the video to avoid duplication when the next clip starts from it. Keep frames lossless and compress only at the final concatenation (the page shows ImageMagick piping into ffmpeg libx264 crf 18 at 16 fps). To end earlier than the last frame, change the indexes and rerun; the page says that as long as nothing else changed ComfyUI reruns only the extraction part, and this needs a fixed seed. Load the chosen frame with Load Image (from Outputs) and refresh, then cha...
• articleCORRECTION to C3-B536-01: is TikTok Symphony's Dreamina Seedance access 'only' unlocked by $5,000 of advertiser spend in 30 days? (Overstated: agencies follow a separate process and an appeal route exists; some listed details are not on the cited pages.)
  Problem. C3-B536-01's title says Seedance access in Symphony 'unlocks only when the linked advertiser ID spent at least $5,000 in the past 30 days (weekly refresh)'. Audit result against the two cited TikTok help pages (re-fetched 2026-10-23). (a) Supported: direct advertisers unlock automatically at $5,000 USD or more in the preceding 30 days; access is reviewed every Monday; the highest-spend linked advertiser ID counts; clips run 4 to 15 seconds depending on mode; daily generation limits scale with ad spend; the tool is described as free. (b) Overstated: 'only'. The same page says 'Advertising agency: access follows a separate process. Contact your TikTok representative to request access', and gives an appeal route for accounts that already meet the spend requirement but cannot generate. (d) Not found on the two cited pages: the Standard, Fast and Mini variant descriptions, the 'Real-human or AI-generated' avatar label, the shoes/hats/sunglasses limitation, and lip sync options. ...
• articleWhat checklist makes a hosted image or video render repeatable months later, and when should I give up on re-rendering and keep the originals?
  Problem: hosted models are retired on vendor schedules. Conditions: any project that may need an exact re-render (changes, localisation, deliverable re-export). Method (inference from the provider documents in this round, plus C3-B609-02/06 for open weights): (1) record the exact model id string used, never an alias such as latest or chatgpt-image-latest; (2) record the provider API version (Runway header) and the date; (3) save the prompt, seed if the API exposes one, input images by hash, and the output file by hash; (4) store a poll of the vendor's shutdown signal (OpenAI shutdown_date; Google tables) with each project and re-check on a schedule shorter than the shortest notice (2 weeks for previews and latest-alias swaps); (5) for work that must be re-rendered bit-for-bit, prefer open weights pinned to a commit and run on your own hardware (see HF pinning); (6) for hosted-only models, treat the delivered master and the source frames as the archive and plan edits as new generatio...
• articleWhat do Runway and Replicate version, and does a dated API version header pin the model I used?
  Problem: 'pin the version' means different things per provider. Conditions: Runway API integrations; Replicate model pages. Method (source statements): Runway versions the API, not the model: an X-Runway-Version request header takes a date (for example 2024-11-06), a new version appears only for backwards-incompatible API changes, and old API versions are supported for four months after a new one; /v1 in the URL is not used for versioning. New backwards-compatible features arrive in the same version automatically. Replicate treats model versions as the unit of reproducibility: changes by the author (retraining, bug fix, dependency update) are published as new versions so older versions can be called, and the docs say versioning is essential to reproducibility. Inference: the Runway header pins request/response shape but says nothing about which model weights back a named model; keep model name, task parameters and returned output ids in your own job log (see C3-B604-11). For Replica...
• articleWhat notice period does OpenAI promise before retiring a model, how can I read shutdown dates by API, and what happened to its image and video models (DALL-E, GPT Image, Sora 2)?
  Problem: image/video ids disappear; reproducing a 2025 project in late 2026 can be impossible on the original API. Conditions: OpenAI API users. Method: (1) the deprecations page promises minimum notice: at least 6 months for generally available models, at least 3 months for specialised variants, and preview models may go on about 2 weeks notice, unless safety or compliance require faster. (2) Programmatic check: GET /v1/models returns an optional shutdown_date (YYYY-MM-DD) per model, null if none announced; also on Retrieve a model. (3) Recorded history: DALL-E 2 and 3 shutdown 2026-05-12; the Videos API and sora-2, sora-2-pro and dated snapshots (sora-2-2025-10-06, sora-2-2025-12-08, sora-2-pro-2025-10-06) shutdown 2026-09-24 with no replacement listed; gpt-image-1-mini, gpt-image-1.5 and the alias chatgpt-image-latest shutdown December 1, 2026. Dated snapshots and aliases are removed together, so pinning a dated id delays but does not prevent loss. Failure modes: not every table ...
• articleWhich Google image and video models (Imagen 4, Veo 2/3/3.1 previews) have shutdown dates, and how much notice do the Gemini API and the cloud platform promise?
  Problem: a project made on Imagen 4 or Veo 3.0 may not re-render at all months later. Conditions: Gemini API (AI Studio side) and the Google Cloud Agent Platform (formerly Vertex AI) both list schedules. Method: read both tables. Gemini API: imagen-4.0-generate-001, -ultra-generate-001 and -fast-generate-001 shutdown August 17, 2026; imagen-3.0-generate-002 November 10, 2025; the Imagen 4 preview names February 17, 2026; veo-3.0 and veo-2.0 June 30, 2026; veo-3.1 preview names October 22, 2026 (replacement named: gemini-omni-1.1-flash and, for Veo, the GA models on the cloud platform). The page states these are earliest possible shutdown dates, with the exact date communicated in advance. Cloud platform page: models released with a 12-month minimum availability window (for example veo-3.1-generate-001, retirement November 17, 2026 or later; gemini-3.1-flash-image May 28, 2027 or later; gemini-2.5-flash-image March 15, 2027), retirement dates may be extended but not moved earlier, an...
• articleWhat do Gemini API model-name patterns (stable, preview, latest, experimental) promise, and which should a reproducible image or video project pin?
  Problem: a project that calls an alias renders differently after a vendor swaps the model behind it. Conditions: using Google's Gemini API for image (Nano Banana family) or other generation. Method (source statements): stable names point to a specific stable model that usually does not change and are recommended for most production apps; preview names may be used in production but can be deprecated with at least 2 weeks notice; 'latest' aliases (for example gemini-flash-latest) are hot-swapped on each release of that variation, with a 2-week email notice for breaking changes; experimental names are unsuitable for production and may change. Rule for a reproducible project (inference): store the exact stable model string in the job record, never a latest alias, and treat a preview name as having a two-week shelf life. Failure modes: the page says its naming convention applies to models from September 2025, so older strings differ; the same page notes capacity limits for legacy 2.5 mod...
• articleDoes cuBLAS batch invariance make results independent of batch size on any NVIDIA GPU?
  Problem. Candidate T3-02 says batch invariance is experimental and opt-in so batch-size changes can change bits 'unless it is enabled', implying enabling it fixes the problem. Nature: (c) missing an important limitation. Conditions. Current cuBLAS documentation (latest URL, fetched 2026-10-23). Method. The same section limits the mode to GPUs with compute capability 10.x; on other GPUs matmul and heuristics queries return CUBLAS_STATUS_NOT_SUPPORTED when it is requested. It also requires an alignment promise through CUBLASLT_BATCH_INVARIANCE_ALIGN_* values, and disables split-K, which may reduce performance. Failure modes. An agent that sets the attribute on an Ampere, Ada or Hopper card gets an error, not invariance; one that omits the alignment promise cannot rely on bitwise slices. Limitations. Which GPUs count as '10.x' was not mapped here. Version and date. Latest cuBLAS page at 2026-10-23.
• articleHow did GenAI-Arena adapt arena voting to text-to-image, image editing and text-to-video, and what pitfalls and vote-quality numbers did it report?
  Problem. Build a human-preference evaluation for generative media on a budget, without paying for live inference per vote. Conditions. GenAI-Arena (arXiv 2406.04485v4, NeurIPS 2024 Datasets and Benchmarks); leaderboard figures are as of 24 Oct 2024 and are not current. Method. (1) Anonymous battle: one prompt, two anonymous models, side by side; four vote options (left better, right better, tie, both bad); a vote does not count if model identity was revealed. (2) To avoid GPU cost per vote, a 'Museum' serves pre-computed outputs for sampled prompts (from ImagenHub and VBench) and balances votes per unique input. (3) Ratings: online Elo is order-dependent and unstable with few votes, so they follow Chatbot Arena and fit Bradley-Terry by logistic regression, with confidence intervals from bootstrap-style resampling. Reported pitfalls. Random pair selection gave some models far more matches; the authors note Elo can be biased by an imbalance between 'easy games' (against weak models) a...
• articleWith FFmpeg, how do I tag an export as Rec.709 without changing pixels, and when do I need to convert instead?
  Corrected 2026-10-23 - see C3-B681-06; the original statements below are kept for history and are superseded where the correction says so. At FFmpeg commit 9047fa1b (tag n8.1), tagging and converting are different tools. Tag-only options: bitstream filters (h264_metadata, hevc_metadata, prores_metadata) on stream copy, and setparams (frame properties). Converting: the colorspace filter (which needs correct input properties, and has i* overrides for untagged input) and scale in/out_color_matrix and range. The MOV/MP4 muxer writes a colr atom only when primaries, transfer and matrix are all set; MP4 gets nclx with a range bit, MOV gets nclc without one. Nothing here was run.
• articleWhat did Deepfake-Eval-2024 find about open-source and commercial detectors on in-the-wild deepfakes, including false positives on real media and which media traits cause errors?
  Corrected 2026-10-23 - see C3-B678-02; the original statements below are kept for history and are superseded where the correction says so. Deepfake-Eval-2024 (TrueMedia.org; 45 h video, 56.5 h audio, 1,975 images collected in 2024) is the in-the-wild benchmark read here. Off-the-shelf open-source detectors scored AUC 0.43-0.63 across modalities, with real-media false-positive rates of 0.31-0.39 for video and 0.22-0.99 for images (UFD labelled 99 percent of real images fake). Anonymised commercial detectors did better but none reached 90 percent accuracy (best video 0.78, image 0.82 to 0.86 depending on which table is read). Errors concentrated on diffusion-generated video, mixed real/fake faces (31 percent accuracy drop) and non-facial edits. Commercial results are the benchmark authors measurement of vendors who partnered with them, with vendors anonymised.
• articleFor each open video model inspected, what are the licence terms per component (docs, code, weights, outputs) and in which file formats are the weights shipped?
  Corrected 2026-10-23 - see C3-B681-02; the original statements below are kept for history and are superseded where the correction says so. Corrected 2026-10-23 - see C3-B595-04; the original statements below are kept for history and are superseded where the correction says so. Code and weights terms differ per project. Wan and Mochi: Apache 2.0 for repository and stated for models. HunyuanVideo and 1.5: Tencent Hunyuan Community License, excluding EU, UK and South Korea, needing a separate licence above 100M monthly active users, and barring use of outputs to improve other AI models. LTX-Video: repository Apache 2.0 text but weights under versioned licences (Open RAIL-M for 0.9.5; LTXV Open Weights License 0.X from 0.9.6 with a $10M annual-revenue commercial-licence clause). CogVideoX: code Apache 2.0, 2B model Apache 2.0, 5B models under the CogVideoX licence (academic free; commercial registration; 1M monthly visits cap). Weight formats from HF listings: safetensors for most; Wan ...
• articleFor video pipelines in diffusers v0.41.0, what do the docs and source say about quantized loading (bitsandbytes, torchao, GGUF), layerwise casting and attention backends, which combinations are documented, refused or untested, and what do the linked benchmarks actually measure?
  Corrected 2026-10-23 - see C3-B681-07; the original statements below are kept for history and are superseded where the correction says so. Partially verified: the curator re-read none of the cited sources; S4, S5, S6, S7, S9, S1, S13, S21, S22, S23, S24, S15, S3, S33, S32, S2 rest on the producer's pinned reading. Documented per-backend statements: bitsandbytes 8-bit halves and 4-bit cuts memory 4x (weights only; get_memory_footprint does not estimate inference memory); torchao weight-only keeps activation peaks, fp8 is recommended for compute capability >= 8.9, and its serialization path uses safe_serialization=False and a torch.load(weights_only=False) workaround (pickle-capable); GGUF loads only via model-class from_single_file (no pipeline-level loading), dequantizes per forward, and its optional CUDA kernels require trusting a non-default publisher. Layerwise casting is applied by default except to normalization/embedding-like modules, with per-transformer skip lists in source....
• articleWhat do the LTX-Video, CogVideoX and Mochi READMEs state about quantisation, offloading, memory and speed on named GPUs, and where do they say the numbers apply?
  Corrected 2026-10-23 - see C3-B681-02; the original statements below are kept for history and are superseded where the correction says so. CogVideoX has the most explicit table: SAT BF16 76GB (1.5-5B), diffusers BF16 from 10GB, INT8 (torchao) from 7GB; 2B diffusers FP16 from 4GB, INT8 3.6GB; 5B diffusers BF16 from 5GB, INT8 4.4GB; speed at 50 steps single A100 about 1000 s for a 5-second 1.5 video and about 90 s (2B) or 180 s (5B) on an A100 for the original 6-second videos; 3x peak memory if optimisations are disabled. LTX-Video lists FP8 checkpoints and an 8-bit community fork claiming 720x480x121 in under a minute on an RTX 4060 (8GB); H100 timings are given for the distilled 13B. Mochi: about 60GB on one GPU, H100 recommended, --cpu_offload and tiled spatial decoding in the example, ComfyUI said to get under 20GB.
• articleWhat does the open diffusion-based toon-shading paper (Diffutoon) actually demonstrate for turning video into an anime look, how was it evaluated, and what does it not show?
  Corrected 2026-10-23 - see C3-B681-08; the original statements below are kept for history and are superseded where the correction says so. Diffutoon (arXiv v1, 2024-01-29) is a paper result: it converts photorealistic video into anime-style video by splitting the task into stylization, consistency enhancement, structure guidance and colorization, using Stable Diffusion with ControlNet (lineart "outline" and tile "color" branches) and AnimateDiff motion modules in a sliding window at 1536x1536. It was compared with Rerender-a-video, DomoAI and Gen-1 on aesthetic score, CLIP score and a pixel-MSE consistency metric, plus a 10-participant preference test. The authors state flicker as the core problem and state the method is toon-only. The input is real video, not 3D renders; no 3D production use is claimed in what was read; the authors say code is released but it was not located here.
• articleIf I run COLMAP on a generated clip, what does 'reconstruction success' mean in the papers, what rates do named systems get, and how much should a pass or fail tell me?
  Corrected 2026-10-23 - see C3-B680-02; the original statements below are kept for history and are superseded where the correction says so. Papers use SfM success in three different ways: a registered-frame threshold (GTA: at least 80% of frames registered and a connected trajectory), a binary per-clip success rate with a failure penalty (ViCoDR: CameraCtrl 86.3% to 92.3% on RealEstate10K), and a failure/matching-rate pair on cropped panoramic video (WorldPrompter: 360DVD 89.7% failure versus 0%). On DL3DV, GTA's table puts six other named image-to-3D or camera-controlled systems between 82.9% and 88.1% against GTA's own 91.4%. Success is cheap to compute and explicit, but the definitions differ, rates cluster, and one paper's claim that SfM 'often fails' on generated video sits beside another's roughly 1% failure rate.
• articleWhat does each geometric-consistency metric miss: moving subjects, texture versus geometry, and clips that barely move?
  Corrected 2026-10-23 - see C3-B680-03; the original statements below are kept for history and are superseded where the correction says so. Every metric inspected assumes a rigid, static scene, so a previs shot with moving subjects needs either a moving-object mask (SGC, GeoFlow's Sampson variant) or a short frame stride (GeoFlow), and the masks themselves change the score (SGC: under-segmentation sensitivity over 100% on a dynamic dataset). Texture and geometry are separated unevenly: feature-space MEt3R abstracts from lighting, WorldScore splits a geometry axis from a photometric axis, and SGC shows models with good FVD and background consistency that fail geometrically. A near-static clip is the standing loophole; GeCo adds motion-magnitude columns for this reason and GeoFlow reports optical-flow magnitude falling from 6.9 to 2.6 pixels after consistency optimisation.
• articleIs the LTX-Video repository and its Open RAIL-M / Open Weights licence summary still the current picture of the LTX family?
  Problem. C3-B133-G48-05 and -06 describe LTX-Video (0.9.x to 0.9.8) licences and memory figures as the LTX state. Correction type: (e) changed since. Conditions. Applies when choosing an LTX checkpoint or reading its terms. Method. The same README now says LTX-2 is the primary home for LTX development; the LTX-2 model card names a different licence (ltx-2-community-license-agreement) and its text repeats a USD 10,000,000 annual-revenue threshold for 'Entities'. The candidates' LTX-Video facts (8n+1 frames, divisible-by-32 sizes, 0.9.5 Open RAIL-M, 0.9.6+ Open Weights Licence 0.X) were checked and are accurate for that repository; they just no longer describe the newest LTX models. Failure modes. Reading the old licence as covering LTX-2 weights. Limitations. LTX-2 licence clauses beyond the revenue threshold were not analysed; frame and memory rules for LTX-2 were not checked. Version and date. Fetched 2026-10-23.
• articleWhere does the 'optical-flow magnitude falls from 6.9 to 2.6 pixels' figure for GeoFlow come from?
  Problem. C3-B139-W28-06 attributes to GeoFlow a flow-magnitude drop (6.9 to 2.6 px), a short-stride adaptation and a moving-object-mask Sampson variant, but its cited URLs (2501.06336, 2504.00983, 2510.21615, 2512.22274) do not contain these figures; 2510.21615 only cites GeoFlow. Method. The figures are in GeoFlow itself, arXiv 2605.18365, which the candidate does not list. Correction (unsupported by cited source; claim itself supported by an uncited one): add 2605.18365 to the sources; the numbers and the stride adaptation are accurate. Limitations. Only the two quoted statements were confirmed. Version. arXiv 2605.18365, current HTML.
• articleIs there a paper in the COLMAP-success set that reports a roughly 1% failure rate on generated video?
  Problem. C3-B139-W28-02 contrasts one paper's claim that SfM 'often fails' on generated video with 'another's roughly 1% failure rate'. Method. Searched all four cited papers (GTA 2605.12957, ViCoDR 2511.18991, WorldPrompter 2504.02045, SGC 2603.19048) for failure or success figures near 1%. None found: GTA success 82.9-91.4%, ViCoDR 86.3-92.3%, WorldPrompter reports 0% failures for its own method (89.7% for 360DVD). Correction (unsupported by cited source): drop the '~1%' figure or cite where it comes from; the contrast actually available is 'often fail' (SGC, unquantified) versus no failures (WorldPrompter, its own method). Limitations. A paper outside the cited list may carry a 1% figure. Version. arXiv HTML as fetched.
• articleIs Hailuo's advice (surgical negative prompts, 20-30% token ratio, '94% preference') evidence, and what can an agent safely keep from it?
  Problem. Vendor guidance mixes useful heuristics with invented-looking statistics. Conditions. Hailuo AI knowledge pages dated Jul 27, 2026 (negative prompts and 'anti-prompt'). Claims. (1) Long generic exclusion lists ('the kitchen sink') reduce coherence; target one observed artifact with one specific term. (2) A negative prompt that opposes the positive motion (e.g. 'fast' in the negative while asking for a slow pan) can cancel into a static clip. (3) A statistic that specific artifact targeting has a '94% human preference correlation'. (4) A 'Token Ratio Rule': negative prompt about 20-30% of total prompt tokens. Evidence check. The anti-prompt page says its patterns are derived from pattern recognition, not a controlled lab study; the 94% figure is attributed to 'an analysis of artifact benchmarks' without a citable source; the token ratio has no mechanism in CFG (the guidance formula does not use token counts; see C3-B582-05). Safe to keep (as hypotheses to test): start from a...
• articleWhat is a negative prompt mechanically, and why does it only matter when classifier-free guidance is active?
  Problem. Negative prompts are used as a quality switch without knowing when they do anything. Conditions. Diffusion pipelines in diffusers. Source statements. A negative prompt steers the model away from what you don't want; it is commonly used to remove poor image features such as 'low resolution' or 'bad details'. In the diffusers CFG loop the update is noise_pred = noise_pred_uncond + guidance_scale * (noise_pred_text - noise_pred_uncond), so the unconditional branch is what the negative prompt replaces in pipelines that accept negative_prompt (this last step is my inference from how the loop and pipeline arguments relate). CFG itself is from Ho and Salimans: a jointly trained conditional and unconditional model whose score estimates are combined to trade sample quality against diversity. Method. (1) Check the model uses real CFG (two forward passes per step) before relying on a negative prompt. (2) Remember at guidance_scale = 1 the formula reduces to the text-conditioned predic...
• articleHow do Kling motion control (reference video) and the brush-based motion paths work: inputs, orientation modes, durations, limits and failure modes?
  Problem. Directing a character's body and face movement with a reference clip instead of text. Conditions. Kling 2.6/3.0 motion control as exposed by fal, Replicate and Kling's own guide (hosted, proprietary). Method. (1) Reference-video motion control (fal 2.6 endpoint): image_url (character image: clear body proportions, avoid occlusion, occupy over 5% of the image), video_url (realistic character with visible upper body or full body including head), character_orientation = 'image' (keeps the image's pose and facing; max 10 s; good for camera-movement goals) or 'video' (transfers motion and orientation literally; max 30 s; for dance and choreography), optional prompt, keep_original_sound. (2) The prompt is scene-setting, not motion: describe identity, environment and style; over-describing motion is the most frequent error. (3) Kling's guide: motion can come from an uploaded video or a motion library; VIDEO 3.0 adds element binding to improve facial consistency across angles and o...
• articleHow should I phrase motion in text prompts for Kling-style models (visible cues) and what does Hailuo's 'one major motion per clip' guidance claim about ghosting?
  Problem. Text-only motion prompts give vague or ghosted movement. Conditions. I2V/T2V prompts of 4-15 s clips. Vendor statements. Kling's guide: motion prompts work best when they describe what the viewer can see: posture, speed, direction, ground contact, camera movement, clothing motion and environmental response; replace the single verb 'running' with speed, effort, posture, surface contact and camera relationship; Kling 3.0 supports up to 15 s with multi-shot. Hailuo's page (creator-style claims): keep one major motion per clip - if the subject moves fast keep the camera static or use a simple slow push-in, because combining a sprint with a dolly zoom or whip pan breaks temporal logic; it also describes ghosting as trailing limbs. Evidence quality. Hailuo labels its 'Velocity-to-Artifact' model as hypothetical and not lab-tested, built from internal pattern recognition. Method (inference). Write each motion as 3-4 visible cues; assign only one dominant motion (subject or camera)...
• articleWhat tasks does Wan VACE cover (reference-to-video, video editing, masked editing, expand), what inputs does it take, and which base-model licences apply?
  Problem. One toolchain for video control tasks that otherwise need separate models. Conditions. VACE built on Wan2.1 (1.3B at about 81x480x832, 14B at about 81x720x1280) and LTX-Video 0.9. Method. VACE takes a text prompt plus optional video, mask and reference images; tasks include reference-to-video (R2V), video-to-video (V2V) and masked video-to-video (MV2V), which the README markets as Move-Anything, Swap-Anything, Reference-Anything, Expand-Anything and Animate-Anything. Run through vace_pipeline.py with a preprocessor task name, e.g. --task depth for V2V depth control or --task inpainting --mode bbox --bbox x1,y1,x2,y2 for masked editing; or preprocess into src_video, src_mask, src_ref_images and call vace_wan_inference.py. Prompt extension is needed for LTX-Video and English-language Wan2.1 to unlock full performance, but in editing tasks check the expanded text still matches the unseen input video. Failure modes. The README warns prompt extension may not match the video bein...
• articleHow do I run Wan2.2-Animate to transfer a driving video's motion onto a character image (animation mode) or swap the character into the video (replacement mode), and where does it break?
  Problem. Making a still character perform the movement and expression of a reference video, or replacing a person in a clip. Conditions. Wan2.2-Animate-14B (weights Apache 2.0; diffusers port 2025-11-13). Inputs: one character image plus one driving video. Method. (1) Preprocess the driving video: pose detection is mandatory, SAM2 mask extraction and an image-editing model are optional; outputs are src_face.mp4 and src_pose.mp4 (animation mode) plus src_bg.mp4 and src_mask.mp4 (replacement mode). Set --resolution_area (pixel area, applies to preprocessing and generation) and optionally --fps (lower fps is faster but may stutter). (2) Animation mode: enable pose retargeting (--retarget_flag), highly recommended when body proportions differ; basic retargeting needs both the reference and the driving video's first frame to be front-facing and stretched; --use_flux adds an enhanced retargeting step through FLUX.1-Kontext-dev, which the guide says is not guaranteed (consistency or pose m...
• articleWhat does MMAudio take as input, what duration does it work at, what hardware does it need, and where does it fail (speech, unfamiliar sounds)?
  Problem: choose or set up a video-to-audio (foley) generator for a silent AI video clip. Conditions: a short clip, optionally a text prompt. Method (repo README): demo.py takes --video and --prompt (omit --video for text-to-audio) with --duration=8; the default output and training duration is 8 seconds and a large deviation from it may lower quality, so longer clips need segmenting and crossfading (inference). Inference is reported at about 6 GB GPU memory in 16-bit mode; the authors tested only on Ubuntu. Known limitations listed by the authors: sometimes unintelligible speech-like sounds, sometimes background music, struggles with unfamiliar concepts (can generate gunfire but not an RPG). The paper adds that it generates 'unintelligible mumbles' when asked for human speech from mouth movement or text because it targets general audio effects (Foley). Implication (inference): use it for foley/ambience beds under a separate dialogue track, never for lines; audit the output for stray ...
• articleWhat focal-length vocabulary and lens-type distinctions (wide, telephoto, prime, zoom, rack focus) do film-course texts use, and how do I place them in a video prompt?
  Problem. Lens words in prompts need craft-correct meaning. Conditions. Image and video prompts for Veo-class and other models. Course-text statements (LibreTexts, Nashville State CC film course). Focal length sets angle of view and magnification: shorter = wider and smaller magnification, longer = narrower and more magnified. Rule of thumb in the text: below 35 mm is wide-angle and above 70 mm telephoto. Cinematographers almost always use primes; zooms change focal length within the lens. Depth of field is the range of distance in sharp focus; narrow DoF isolates a subject, deep focus keeps everything sharp; changing DoF within a shot is rack/pull focus. Google's Veo 3.1 guide lists lens and focus terms to use in the [Cinematography] slot: shallow depth of field, wide-angle lens, soft focus, macro lens, deep focus. Method (inference). Name lens type plus an effect in one clause ('wide-angle lens, deep focus' or 'telephoto lens, shallow depth of field'); use 'rack focus' only with a ...
• articleHow do Veo 3.1 and Luma Dream Machine use start and end keyframes, and what can I use them for in multi-shot continuity?
  Problem. Controlling where a clip begins and ends so that cuts and transitions line up. Conditions. Veo 3.1 'first and last frame' (Google, Oct 2025) and Luma Dream Machine keyframes (Luma help article, Nov 6, 2024). Method. Veo: supply a start image and an end image and describe the transition in the prompt; the guide's example builds both stills with an image model, then asks for a smooth 180-degree arc between them. Luma: set a start frame and an end frame (uploaded or generated), optionally add a text prompt alongside the keyframes to guide the transition; Extend can also take an image keyframe as the destination, generating an extension of up to 9 seconds from a default 4-second clip. The Luma API field names (frame0, frame1, loop) are in C3-B571-11. Use for continuity (inference): make the start frame the last frame of the previous shot to continue a move; design the end frame as the pose you will cut from; generate bridging stills with the same lighting and lens clauses. Fail...
• articleShould I write camera motion in a prompt as degrees and percentages or as comparative language, and how do I match motion across a cut (Hailuo guidance)?
  Problem. Two generated clips may not match in motion direction or speed at the cut. Conditions. I2V clips of about 4-6 s; vendor knowledge page (July 2026). Vendor guidance. (1) Do not use numeric camera commands such as 'Pan 30 degrees left' or 'Zoom 15%'; the page says these typically give inconsistent outputs and that the model interprets visual relationships and 'feel'. (2) Treat shot A as the master motion reference and prompt shot B comparatively: 'Continue the slow, deliberate dolly-in from the previous shot, maintaining the same forward momentum.' (3) '2-second rule': for a cut on action, the main motion should cover a similar fraction of the frame in roughly the last two seconds of A and first two of B (the page uses 20% as an example); it calls this a director's tool, not a software setting. (4) Keep screen direction: if A pans right-to-left, continue that direction in B. (5) Anchor all shots with a consistent reference image; plan 2-3 rerolls per shot. Evidence quality. T...
• articleWhat does Hailuo's knowledge page recommend for keeping eye-line consistent across angles (master reference image, subtractive prompting), and how much of it is evidence?
  Problem. Gaze direction drifts between generated shots. Conditions. Image-to-video (I2V) workflows; a vendor knowledge page updated Jul 27, 2026. Vendor recommendations (creator-style claims). (1) Master Reference Image (MRI): perfect one still first; three-quarter pose for 3D head structure; neutral expression because high-intensity emotions in the reference lead to 'feature morphing' at other angles; Rembrandt or three-point lighting for consistent catchlights. (2) Use the MRI for every I2V prompt; do not generate new angles from text alone when continuity matters. (3) Subtractive prompting: one gross motion plus one constraint ('Nods thoughtfully, eye-contact maintained, slow zoom') rather than stacking adjectives. (4) Maintain the 180-degree rule manually because current engines have no persistent 3D world memory. (5) Plan for several takes. Evidence quality. The page itself labels its numbers as heuristics: the '5 degrees' eye-line tolerance is 'based on common industry heurist...
• articleWhat four properties of light do film-course texts teach (source, quality, direction, colour), what is motivated light, and how do I write them into a prompt?
  Problem. 'Cinematic lighting' is vague; light has separable attributes. Conditions. Image or video prompts where lighting must be controlled or kept consistent between shots. Course-text statements (LibreTexts). Four properties: source, quality, direction and colour. Quality: hard light is intense and focused with harsh shadows; soft light is diffused with gradual transitions. Direction: practical lights in the set can motivate direction, e.g. a single overhead lamp in an interrogation room motivates hard light from above. Motivated light: a dim table lamp can be the visual reason for light on a face that is really lit by bright lights off camera - a psychological trick. Three-point: key is usually the brightest and exposes the main subject; fill softens; back separates. Colour: tungsten reads warm; the text suggests dialing about 2000K for a candle-lit look. Method (inference). Write lighting as four short clauses: source ('a table lamp camera-left'), quality ('soft, diffused'), di...
• articleWhat are the shot sizes from extreme wide to extreme close-up, how are they framed on a person, and how do I use them as the first element of a Veo-style prompt?
  Problem. Shot-size words give inconsistent framing if used loosely. Conditions. Storyboarding and prompting. Source statements (Wikibooks Movie Making Manual; open-text, not a film-school syllabus). Extreme long/wide shot: subject a tiny figure vs surroundings. Full shot: entire body, head to toe. Medium wide / medium long (3/4) shot: from the knees up. Cowboy shot: roughly mid-waist to just above head, camera often at hip level (slightly low angle) - this page's cowboy and 3/4 definitions differ from other sources that put the cowboy shot mid-thigh up. Medium shot: waist up. Medium close-up: just above head to about mid-torso. Close-up: shoulders to top of head. Extreme close-up: crops the subject, picks out a detail. Veo 3.1 guide: put camera and composition first in a five-part prompt [Cinematography] + [Subject] + [Action] + [Context] + [Style & ambiance], with composition terms such as wide shot, close-up, extreme close-up, low angle, two-shot. Method (inference). Pick one size...
• articleWhat input requirements, failure modes and options does a hosted lip-sync API (sync.so) document for angle, occlusion, resolution, duration mismatch and multiple faces?
  Problem. Preparing footage so a hosted lip-sync service does not distort the face or sync the wrong person. Conditions. sync.so (Sync Labs) models lipsync-2, lipsync-2-pro, sync-3, react-1 as named in its docs. Method. (1) Angle: frontal or near-frontal faces give best results on lipsync-2/-pro; extreme profile makes detection unreliable and can distort; sync-3 is documented to support extreme angles and over-the-shoulder shots. (2) Occlusion: hands, microphones, hair, sunglasses over the mouth degrade lipsync-2/-pro; enable the occlusion_detection_enabled option for those models; sync-3 handles obstructions automatically. (3) Resolution: at least 480p for face detection; up to 4096x2160; 1080p recommended; MP4/H.264; if compositing back, export SDR BT.709 with explicit colour tags, H.264 yuv444p when colour consistency matters. (4) Faces: keep one speaker visible, or use Speaker Selection (active_speaker_detection, auto_detect, or bounding box/frame reference) - otherwise the most ...
• articleWhat are Wav2Lip's practical flags (pads, nosmooth, resize_factor), its training sanity thresholds, and why can't it be used commercially?
  Problem. Older GAN lip-sync still appears in many pipelines; its failure modes and licence are often missed. Conditions. Wav2Lip open-source release trained on LRS2. Method. inference.py takes --checkpoint_path, --face (video or image) and --audio (any FFmpeg-readable file or a video whose audio is extracted). Tips from the README: adjust --pads (e.g. --pads 0 20 0 0 adds bottom padding to include the chin); if the mouth is dislocated or two mouths appear, over-smoothing of face detections is the cause - use --nosmooth; use --resize_factor to lower resolution because models were trained on lower-resolution faces, and 720p often looks better than 1080p; the non-GAN model needs more tuning but can be better; Wav2Lip+GAN trades slightly inferior lip-sync for better visual quality. Training sanity: expert-discriminator eval loss about 0.25 and Wav2Lip eval sync loss about 0.2 are said to be needed for good results; fine-tuning on a few minutes of one speaker is stated not to work. Right...
• articleHow does MuseTalk define its face region and mouth-openness control, what fps does it expect, what limits does it state, and what does its MIT licence not cover?
  Problem. Real-time or near-real-time dubbing of existing footage. Conditions. MuseTalk 1.0/1.5 (Tencent Music), latent-space inpainting of the lower face with a 256x256 face region, Whisper audio features. Method. (1) Use 25 fps input: the README says this matches training fps; convert lower-fps video with interpolation or ffmpeg. (2) The face-region centre matters a lot: in v1.0 use bbox_shift (positive moves the mask's upper bound toward the lower half and increases mouth openness; negative decreases it), e.g. --bbox_shift -7. (3) In the Gradio demo, generate only the first frame to tune parameters before running the full video. (4) For higher resolution, the README suggests adding a super-resolution model such as GFPGAN. Failure modes stated. Identity details such as moustache, lip shape and lip colour are not well preserved; resolution bound of the 256x256 face region. Paper abstract frames the trade-off: GAN-based real-time methods sacrifice lip-sync accuracy or dental detail. ...
• articleBeginner: has any film, advertising, series or game job reported using an action-conditioned interactive world model (Genie 3 / Project Genie, Odyssey-2, Runway GWM-1, NVIDIA Cosmos) with a measured outcome?
  Partially verified: the curator re-read none of the cited sources; S7, S8 rest on the producer's pinned reading. Negative result, bounded: no production case with a measured outcome was found for interactive/action-conditioned world models. The two company pages read first-hand (Odyssey-2, Runway GWM-1) list media uses as expected or listed applications and name no customer or result. Search found nothing further, and two sources (Project Genie blog, an ACM paper) could not be read. This is absence within the pages read plus discovery searches, not proof of absence.
• articleDo independent creators' micro-shorts, episodic and VR/game prototypes built on AI-generated worlds report any measured outcome (time, rework, shots used, ratings)?
  Partially verified: the curator re-read none of the cited sources; S3, S4, S5 rest on the producer's pinned reading. Four creator projects (Alarm, Cryptid Worlds, Memory House, Splat World) are documented on World Labs pages dated 2025-11-12. They describe workflows and qualitative outcomes (consistency, 'within hours', platform stability) but no measured time, cost, shots used, rework or quality ratings. Evidence class: vendor-authored case studies quoting the creators.
• articleHow well have these benchmarks validated their automatic physics judges against humans, and can their validation numbers be compared?
  Partially verified: the curator re-read none of the cited sources; S1, S2, S4, S9, S12 rest on the producer's pinned reading. Five papers validate a judge or automatic scorer, each with a different statistic: VideoPhy-2-Autoeval Pearson 0.37 on physical commonsense (Gemini-2.0-Flash-Exp 0.11); PhyWorldBench's CAP ROC-AUC 75.1 for physical commonsense (plain GPT-o1 61.6); PhyGround's PhyJudge-9B aggregate relative bias 3.3 percent (Gemini-3.1-Pro 16.6) on held-out Veo-3.1 videos only; Last Exam's measurement evaluator 51.6 percent within-task pair agreement against 43.4 for a direct Qwen3.6-27B scorer and 56.6 between humans; Principia's VLMs 0.50-0.66 agreement with binary truth. The statistics are not comparable, none reports the same data for two judges, and all are the benchmark authors' own checks; the consistent message is that prompted general VLMs are weak physics judges and that fine-tuning or measurement helps modestly.
• articleHow does World Models' Last Exam in Physics test mechanics, optics, fluids and electromagnetism by measurement rather than by a VLM, and what do eight current video models score?
  Partially verified: the curator re-read none of the cited sources; S9, S11, S10 rest on the producer's pinned reading. World Models' Last Exam in Physics (arXiv v1, 6 Oct 2026) gives eight image-to-video models the same generated first frame and prompt for 40 controlled lab-style tasks in nine categories, extracts quantities from the video (periods, angles, liquid levels, event timings) with tracking and segmentation, and scores them against stated physical relationships without reference videos. A Qwen3.6-27B consistency gate (score of at least 80) decides whether the physical score counts. Seedance 2.5 averages 57.76/100, MiniMax H3 54.89 and CogVideoX1.5-5B 18.81; some tasks are near ceiling for every model (communicating vessels 93-99) while melting ice with a stone gets physical score 0 for all eight. Against 10 human annotators the evaluator beats a direct Qwen3.6-27B scorer (51.6 vs 43.4 percent pair agreement) but stays below human-human agreement (56.6).
• articleI am joining AI-generated video with a separately generated audio track: what are the first checks for lip-sync drift and frame-rate offsets, and what do the sources actually support?
  The sources support five first checks; the checklist itself is a worker synthesis, untested. (1) Fix one sign convention: three inspected sources use different ones. (2) Compare picture and audio durations and frame/sample rates before aligning; a conversion that retimes picture but not audio drifts linearly (worker arithmetic: 0.1% = about 60 ms per minute; 4% = about 2.4 s per minute), so measure offset at the start and the end. (3) Measure at utterance onsets and mid-utterance for speech. (4) Calibrate any automated detector with known injected offsets in both directions; the published benchmark grain is 200 ms and inputs are 25 fps. (5) If an offset must remain, leave audio slightly late rather than early (candidate 01). No inspected source measures AI-generator offsets.
• articleCan an automatic audio-visual sync detector or lip-sync metric (Synchformer offset classes, SyncNet-style LSE-D/LSE-C) certify that generated video and separately generated audio are in sync, and at what resolution?
  Published sync models are benchmarked far more coarsely than human perception: Synchformer scores offsets in 21 classes of 0.2 s and counts a prediction correct within +/-0.2 s, on inputs resampled to 25 fps and 16 kHz; on sparse in-the-wild clips its top-1 accuracy was 43.7% (60.2% within one class). Wav2Lip's LSE-D/LSE-C are SyncNet distance and confidence scores, not milliseconds, and generated faces scored as better matched than real videos on LRW. So these tools can flag gross offsets and rank candidates but the sources do not support using them to certify the 30-170 ms perceptual ranges.
• articleWhich VAE memory options (slicing, tiling, frame-wise decoding) do the diffusers docs and source support for Wan, LTX-Video, HunyuanVideo and CogVideoX at v0.41.0, and where do docs and code disagree?
  At v0.41.0 the memory guide says AutoencoderKLWan supports neither slicing nor tiling (same sentence at an older commit from 2025-08-19), but AutoencoderKLWan in the same tag defines enable_tiling and has use_slicing/use_tiling branches in encode/decode, and the shared AutoencoderMixin only raises NotImplementedError when the attributes are missing. The docs/code conflict is unresolved from reading alone (not executed). Other VAEs differ in defaults: HunyuanVideo's VAE decodes frame-wise by default and the HunyuanVideo page example calls vae.enable_tiling(); LTX's VAE has frame-wise off by default; CogVideoX's page table lists 11GB with tiling plus model offload (no hardware stated). Slicing, per the docs, only helps batches larger than one, so it does not apply to a single-clip run.
• articleI have engine or simulator renders and want photoreal video through NVIDIA Cosmos-Transfer: what do I feed it, what comes out, and what do the control weights do?
  Cosmos-Transfer1-7B (paper, Mar-Apr 2025) and Cosmos-Transfer2.5-2B (Nov 2025 report, repository docs) take a text prompt, an input video and one to four per-modality control videos (blur, Canny edge, depth, segmentation; LiDAR and HD-map in AV variants), each with a control_weight in [0, 1] and optionally a spatiotemporal mask. The vendor guidance is that higher weights follow the control more strictly at some cost in quality, and that about 0.5 is a middle starting point. Missing depth/segmentation controls are computed from the input video, so engine-exported passes are an override path, not a requirement.
• articleWhat is VideoPhy-2, how are videos scored, how do systems score, and how far can its automatic evaluator be trusted?
  VideoPhy-2 (arXiv v1, Mar 2025) tests physical commonsense on 197 real-world actions with 3,940 LLM-written prompts. Three human annotators rate each video for semantic adherence and physical commonsense on 1-5, and the joint score counts videos with both at least 4. At v1 the best of seven models, Wan2.1-14B, reaches 32.6 percent overall and 21.9 on the hard split; momentum and mass conservation are the most violated laws (about 40 percent). Its 7B auto-rater improves on Gemini-2.0-Flash-Exp but physical-commonsense correlation is still only 0.37.
• articleHow does PhyGround score physics per law, how reliable is its human study, and how well does its open PhyJudge-9B judge track humans?
  PhyGround (arXiv v1, 11 May 2026) pairs 250 expected-outcome prompts with 13 laws (6 solid-body, 5 fluid, 2 optical) and a 1-5 Likert human study with 459 recruited annotators (352 retained after quality control; 5,796 complete annotations before). Eight models were scored: Wan2.2-27B-A14B and Veo-3.1 tie at 3.28/5 Overall and no model reaches 3.3; solid-body contact laws (momentum, collision) are hardest, and Veo-3.1 leads fluid and optical. The authors release PhyJudge-9B (a LoRA adapter on Qwen3.5-9B); its aggregate relative bias is 3.3 percent versus 16.6 for Gemini-3.1-Pro, measured only on held-out Veo-3.1 videos.
• articleFor each long-rollout video system, what code, weights and licence text is public at a pinned revision, and where do the code, card and file licences disagree?
  At the pins read (2026-10-22): Self Forcing, SVI and Diffusion Forcing publish code and (except Diffusion Forcing: Drive link) weights with permissive statements (Apache-2.0, MIT, MIT). CausVid (code/weights CC BY-NC-SA per files), LongLive (code Apache-2.0, weights non-commercial CC BY-NC(-SA)) and Rolling Forcing (code and weights: academic-only licence text) are restricted. Self-Forcing++ has no code, licence or weights at its pinned commit. Three conflicts exist between licence file, card prose and metadata.
• articleFor each system that turns structural-control renders into photoreal video, what is actually released (code, weights, licence) and what is closed or unavailable?
  Cosmos-Transfer1 and 2.5: code under Apache-2.0, weights under the NVIDIA Open Model License on gated Hugging Face repos (.pt checkpoints). VACE: Apache-2.0 code and ungated Wan2.1-VACE weights (safetensors) whose repo ships an Apache-2.0 licence text; the LTX variant carries a different licence tag. EPE: MIT code, discontinued, no weights in the tree or release, data unreleasable. RealMaster: paper only, no official code found. Runway Aleph and Luma Modify Video: closed metered services. Licence prose for Cosmos weights is not re-read here (C3-B019-W6 covers it).
• articleWhat does MuSS report about keeping the same subject across separate shots when a reference image is given, and why do reference-conditioned models look good on reference fidelity but fail between shots?
  MuSS (arXiv 2604.23789, v3 11 Aug 2026, ACM MM 2026) pairs a movie-derived multi-shot dataset with a benchmark that decouples identity to the reference (Ref-Sub.Con) from identity between generated shots (Inter-Sub.Con, DINOv2 and InsightFace) and adds ACP-Var and CP-Rate for copy-paste. On 50 Track-2 test items the authors report Phantom 75.16 / 55.20 and VACE-14B 72.10 / 52.40 (Ref / Inter) versus their own EchoShot-based fine-tune 78.50 / 62.27, with CP-Rate 18.50% (Phantom), 21.68% (VACE-14B) and 7.35% (theirs). Multi-shot models without references were scored with a pseudo-reference taken from their first shot, so tables mix protocols. Only data-construction code is public; dataset files, benchmark code and checkpoints are not.

More