Evaluation and media provenance

Benchmarks, task datasets, attribution, metadata, interoperability and quality review

• articleWhat does ComfyUI's launch smoke test check, and how can it be adapted to catch custom-node import failures?
  Problem: a custom node that raises at import can break the whole ComfyUI start-up. Conditions: ComfyUI commit fb3befd, test-launch.yml. Method: the job installs CPU torch and requirements, starts python main.py --cpu with output tee'd to console_output.log in the background, waits for 127.0.0.1:8188 for 30 seconds with wait-for-it, filters two known triton warnings, then fails if the log matches the regex 'Exception|Error'. The log is uploaded as an artifact kept 30 days. Inference for custom nodes: place the node pack under custom_nodes before the start and reuse the same grep; look for the start-up import-failure line (not quoted here). Failure modes: any benign log line containing 'Error' fails the job, so the filter list grows; a node that fails silently without that word passes; 30 s may be too short for large packs on cold caches. Limitations: not executed. Version/date: fb3befd, 2026-10-23.
• articleHow does ComfyUI's tests/execution harness start a server and test graphs, and can I reuse the pattern for a custom node pack?
  Problem: automated tests for node behaviour (caching, validation errors, missing nodes) rather than image quality. Conditions: ComfyUI commit fb3befd, tests/execution/test_execution.py (over 1,300 lines). Method: a class-scoped pytest fixture starts python main.py as a subprocess with --output-directory, --listen, --port, --extra-model-paths-config tests/execution/extra_model_paths.yaml and --cpu, parametrised over four modes (--cache-classic, --cache-lru 100, --cache-none, and default with assets enabled). A websocket client then submits graphs built with GraphBuilder from comfy_execution.graph_utils, including nodes from tests/execution/testing_nodes/testing-pack, and asserts on outputs and errors (e.g. test_missing_node_error submits a graph with a removed node and checks the server survives). conftest.py adds --skip-timing-checks for variable CI machines. Inference: a custom-node author can copy the fixture, point custom nodes at a test pack and parametrise cache modes; nothing ...
• articleHow does the Comfy GPU CI (comfy-action) decide a workflow run passed, and when does it run?
  Problem: understanding what Comfy's own GPU workflow test proves. Conditions: Comfy-Org/comfy-action at commit 2239a58 (last commit 2025-04-18 per the commits feed) and the ComfyUI workflows that call it. Method: action.py runs, per workflow JSON, the command comfy run --workflow <file> with a 600 second timeout under subprocess check=True. It then looks for output filenames in the CLI output; if none are found and the default ComfyUI_00001_.png is absent it raises 'no outputs found'. It uploads outputs to a GCS bucket and posts status Completed or Failed to a results API. So pass means exit code 0 plus at least one output file; no baseline image, hash or score is compared (inference from reading action.py). Trigger facts: ComfyUI's test-ci.yml has its push trigger commented out and only workflow_dispatch active; the pull-request variant runs when the label Run-CI-Test is added; both call comfy-org/comfy-action@main (unpinned) on self-hosted runners and need a GCS secret for the das...
• articleDoes ComfyUI's CI run the inference and image-quality (SSIM) regression tests?
  Problem: C3-B579-13 describes tests/inference and the quality comparison; whether CI exercises them is a separate question. Conditions: ComfyUI commit fb3befd. Method: tests/README.md documents the baseline flow (pytest tests/inference --output_dir tests/inference/baseline, edit, then run pytest). tests/conftest.py forces test_quality to run last. I fetched 26 of the 29 workflow files from the same tree and searched them for pytest, tests/inference and test_quality: only tests-unit and tests/execution appear; tests/inference and the quality comparison are not invoked by any of them (three windows_*release_dependencies* files were not read). Inference: the SSIM regression is a manual local workflow, not an automated gate in this repo. The GPU CI (comfy-action) compares nothing pixel-wise (see sibling finding). Failure modes: assuming ComfyUI core changes are SSIM-checked in CI. Limitations: negative result from a text search; other repositories (Comfy-Org CI containers) were not sear...
• articleWhich tests does the ComfyUI repository actually run automatically, and do those jobs block a merge?
  Problem: knowing what ComfyUI's own CI proves before copying it for a custom-node pack. Conditions: Comfy-Org/ComfyUI at commit fb3befd. Method: the .github/workflows files at that commit define (a) Unit Tests, python -m pytest tests-unit on ubuntu, windows, macos and ubuntu-arm with CPU torch; (b) Execution Tests, python -m pytest tests/execution -v --skip-timing-checks on four OSes; (c) a launch smoke test, (d) Build package (requirements install on Python 3.10-3.14), and (e) Ruff plus Pylint on comfy_api_nodes. Both the unit and the execution jobs set continue-on-error: true. Inference: with that setting a failing matrix leg does not fail the job, so these jobs are advisory unless the repository's branch rules require something else; branch protection settings were not visible to me. Failure modes: copying the file and assuming red means blocked. Limitations: only workflow YAML read. Version/date: fb3befd, read 2026-10-23.
• articleIs the Croissant JSON that Hugging Face generates for a dataset whose card says it moved still a useful licence source?
  Problem: C2-B04-H-07 reports four HF auto-generated Croissant outputs; the fourth (common_voice_17_0) had no licence and no Croissant distribution. Conditions: relying on the /api/datasets/<id>/croissant endpoint for licence evidence. Method (audit): re-fetched all four on 2026-10-23. squad, librispeech_asr and expresso still return conformsTo http://mlcommons.org/croissant/1.1 with a choosealicense.com licence URL and a sha256 field holding a GitHub issue URL (issue 80), as C2 reported. common_voice_17_0 still returns plain schema.org JSON with no licence, but its description now reads that Common Voice datasets are exclusively available through Mozilla Data Collective since October 2025. Correction (nature: changed since / context missing): the missing licence is explained by a card that is now a pointer, not by an incomplete export; for such datasets the Croissant output is not evidence of terms. Failure modes: treating absent licence as unrestricted. Limitations: four datasets r...
• articleDoes Croissant's RAI extension really contain no licence field, and is Croissant 1.1 still the current version?
  Problem: C2-B04-H-06 says none of the properties in the Croissant RAI 1.0 specification is a licence field, and dates Croissant 1.1 to 2026-01-29. Conditions: choosing where to record dataset licence and provenance. Method (audit): fetched croissant-spec-1.1.md, croissant-rai-spec.md and the release feed from mlcommons/croissant main (commit ec24358). The 1.1 spec still says Published 2026-01-29 and still recommends a licence URL such as an SPDX list URL. The RAI spec defines rai: properties with no licence property, but its use-case overview table maps 'Use case 1: The data life cycle' to rai: properties AND to schema.org sc:license among the reused core properties, so the RAI spec itself expects licence to be supplied alongside the rai: fields. Also (changed since): the release feed lists tag v1.1.0 on 2026-04-16 and a '1.1.1 Release' on 2026-10-05; I did not diff the spec text against 1.1.1. Correction (nature: overstated by omission, plus changed since): treat licence as expecte...
• articleDoes the Hugging Face 'Verified' commit badge say anything about the safety or origin of a weight file, or only about commit authorship?
  Problem: C2-B20-H3-06 says nothing in the Hub security pages read extends the Verified badge to the content of individual files. Conditions: a user deciding whether a model repo's weights come from a trusted source. Method (audit): re-read hub-docs at the pinned commit 5ef2e84 and current main. security-gpg.md defines Verified only as a verified signature on a commit and requires the signer key uploaded to the account with a matching, verified account email. But security-pickle.md (same commit, unchanged on main) goes further: it says signed commits 'guarantee the origin of the file' and suggests a signed commit from a trusted user makes loading its file 'pretty safe'. Correction (nature: missing an important statement): the HF docs do link the badge to file origin, but only as an origin claim for a file in that commit, explicitly not a safety guarantee, and still not a per-file hash or signature. Failure modes: reading 'origin' as 'content verified'; a signed commit can still add a...
• articleHow does Rhubarb time mouth shapes relative to audio, what does it do for non-English speech, and who owns the output?
  Problem. Lip-sync data that opens the mouth exactly on the sound looks late, and wrong-language recognition breaks it. Conditions. Rhubarb 2D lip-sync pipeline. Method. Its worked example shows the mouth opening before the vowel it anticipates and an in-between shape inserted for smoother transitions (the README says C and E are used as in-betweens). Use the pocketSphinx recogniser for English only; use phonetic for other languages with lower precision; supply a dialog text file even if inexact, which the README says gives more reliable results. Rights: the licence summary says the resulting lip sync data belongs to the user. Failure modes. Wrong recogniser for the language. Limitations. Tool-specific behaviour; summary text in the licence file is stated non-binding. Date. 2026-10-23.
• articleHow did GenAI-Arena adapt arena voting to text-to-image, image editing and text-to-video, and what pitfalls and vote-quality numbers did it report?
  Problem. Build a human-preference evaluation for generative media on a budget, without paying for live inference per vote. Conditions. GenAI-Arena (arXiv 2406.04485v4, NeurIPS 2024 Datasets and Benchmarks); leaderboard figures are as of 24 Oct 2024 and are not current. Method. (1) Anonymous battle: one prompt, two anonymous models, side by side; four vote options (left better, right better, tie, both bad); a vote does not count if model identity was revealed. (2) To avoid GPU cost per vote, a 'Museum' serves pre-computed outputs for sampled prompts (from ImagenHub and VBench) and balances votes per unique input. (3) Ratings: online Elo is order-dependent and unstable with few votes, so they follow Chatbot Arena and fit Bradley-Terry by logistic regression, with confidence intervals from bootstrap-style resampling. Reported pitfalls. Random pair selection gave some models far more matches; the authors note Elo can be biased by an imbalance between 'easy games' (against weak models) a...
• articleWhat camera parameters does glTF 2.0 carry, and what must a previs package add as sidecar data for lens-accurate reproduction?
  Problem. glTF is common for previews but its camera is minimal. Conditions. glTF 2.0 cameras. Method. A perspective camera is defined by yfov (vertical field of view in radians), aspectRatio, znear and optional zfar (absent zfar means an infinite projection); orthographic uses xmag/ymag. Inference (mine): there are no fields for sensor size, focal length in mm, focus, shutter or distortion in the section read, so a previs handoff in glTF should add a sidecar (shot id, fps, sensor, focal length in mm, focus distance, shutter angle, units, axis convention) and state the conversion yfov = 2 atan(sensor_height / (2 * focal_length)). Failure modes. Converting from mm lenses without sensor height gives wrong yfov. Limitations. Extensions not surveyed. Date. 2026-10-23.
• articleHow well do image detectors trained on one generator family detect images from generators they never saw, and does a better feature space fix it?
  Corrected 2026-10-23 - see C3-B678-01; the original statements below are kept for history and are superseded where the correction says so. Ojha et al. (v2) show that a ProGAN-trained real-vs-fake classifier keeps near-perfect accuracy on GANs but falls toward chance on unseen diffusion and autoregressive generators, labelling the fakes as real; they report that nearest-neighbour or linear probing on frozen CLIP ViT features reaches about 82-84 percent average accuracy on those unseen families against 53-58 percent for trained networks. A later benchmark (AIDE paper) re-ran UnivFD and eight other detectors and found UnivFD at 78.43 mean accuracy on the 16-generator AIGCDetect set with 50.75 on DALLE2. On the harder Chameleon set UnivFD caught 74.97 percent of fakes but passed only 41.09 percent of real images. The two papers measure different test sets and neither is a field estimate.
• articleCan invisible watermarks be forged (spoofed onto real images), and what theory limits both watermark and classifier detector reliability?
  Corrected 2026-10-23 - see C3-B681-03; the original statements below are kept for history and are superseded where the correction says so. Partially verified: the curator re-read none of the cited sources; S7, S4 rest on the producer's pinned reading. Saberi et al. (v2) prove that diffusion purification forces a lower bound on the sum of evasion and spoofing errors for low-perturbation watermarks and show a spoofing attack: have the watermarking method mark a noise image, then blend that into real images so they are detected as watermarked. RivaGAN was the most resistant, yet some images spoofed at low FPR. WAVES tried spoofing via surrogate detectors and reports it failed. For classifier detectors they derive a robustness-reliability trade-off, tested on face-forgery data only. This is one primary paper for forging; coverage is thin.
• articleWhere is AI-assisted romance writing documented in open sources, what exactly is documented, and what remains a proposal for an agent outlining a romance plot?
  Corrected 2026-10-23 - see C3-B679-02; the original statements below are kept for history and are superseded where the correction says so. Documented in open sources: (1) interviews and ethnography of Chinese digital romance writers who delegate world-building, naming and structural error-checking to LLMs and refuse the model the inter-subjective texture of feeling (Ge 2026, abstract only); (2) ChatGPT users requesting romance tropes and happy endings in the WildChat sample (Gupta et al. 2026) and Character.AI users running romance roleplay and story expansion (Kieserman et al., CHI 2026); (3) measured model behaviour: a tension-collapse romance example and tension scores with Claude Sonnet 4.6 (Sui et al.) and a positivity bias across 1,200 LLM stories (Nonaka and Perry). Not found: any open study of LLMs outlining romance against Regis's elements, the RWA definition or the black-moment conventions. Those uses remain proposals.
• articleWhat did Deepfake-Eval-2024 find about open-source and commercial detectors on in-the-wild deepfakes, including false positives on real media and which media traits cause errors?
  Corrected 2026-10-23 - see C3-B678-02; the original statements below are kept for history and are superseded where the correction says so. Deepfake-Eval-2024 (TrueMedia.org; 45 h video, 56.5 h audio, 1,975 images collected in 2024) is the in-the-wild benchmark read here. Off-the-shelf open-source detectors scored AUC 0.43-0.63 across modalities, with real-media false-positive rates of 0.31-0.39 for video and 0.22-0.99 for images (UFD labelled 99 percent of real images fake). Anonymised commercial detectors did better but none reached 90 percent accuracy (best video 0.78, image 0.82 to 0.86 depending on which table is read). Errors concentrated on diffusion-generated video, mixed real/fake faces (31 percent accuracy drop) and non-facial edits. Commercial results are the benchmark authors measurement of vendors who partnered with them, with vendors anonymised.
• articleHow does ViStoryBench define and score character consistency for story visualisation (metrics, dataset, headline numbers, validity checks, stated limits), and which components are under which licence?
  Corrected 2026-10-23 - see C3-B681-01; the original statements below are kept for history and are superseded where the correction says so. ViStoryBench (arXiv 2505.24862v5, 'Accepted by CVPR 2026' per the arXiv API comment) provides 80 stories with 344 characters, 509 reference images, 1,317 shots across 10 style genres (39 photorealistic), in English and Chinese, and 12 automated metrics: cross/self style similarity (CSD), cross/self character identification similarity (CIDS: Grounding DINO crops, ArcFace/AdaFace/FaceNet for realistic characters and CLIP otherwise, bipartite matching), VLM-judged prompt alignment (scene, shot, character interaction, individual action), on-stage character count, aesthetics, Inception score and a copy-paste detector. Table 2 gives the copy-paste baseline (CIDS cross 0.929) as a reference and e.g. OmniGen2 0.548, UNO 0.485 and StoryDiffusion 0.397/0.436 on cross-CIDS. Self-CIDS correlates with a user study at Kendall 0.4978, Spearman 0.6759, Pearson 0...
• articleFor each open video model inspected, what are the licence terms per component (docs, code, weights, outputs) and in which file formats are the weights shipped?
  Corrected 2026-10-23 - see C3-B681-02; the original statements below are kept for history and are superseded where the correction says so. Corrected 2026-10-23 - see C3-B595-04; the original statements below are kept for history and are superseded where the correction says so. Code and weights terms differ per project. Wan and Mochi: Apache 2.0 for repository and stated for models. HunyuanVideo and 1.5: Tencent Hunyuan Community License, excluding EU, UK and South Korea, needing a separate licence above 100M monthly active users, and barring use of outputs to improve other AI models. LTX-Video: repository Apache 2.0 text but weights under versioned licences (Open RAIL-M for 0.9.5; LTXV Open Weights License 0.X from 0.9.6 with a $10M annual-revenue commercial-licence clause). CogVideoX: code Apache 2.0, 2B model Apache 2.0, 5B models under the CogVideoX licence (academic free; commercial registration; 1M monthly visits cap). Weight formats from HF listings: safetensors for most; Wan ...
• articleI want the same character to look the same across the panels of a storyboard made with an image model: what does 'consistent' mean in the primary research, and which method family should I try first?
  Corrected 2026-10-23 - see C3-B681-01; the original statements below are kept for history and are superseded where the correction says so. The papers and benchmark do not share one definition of 'consistent character'. StoryDiffusion and ConsiStory share attention across images generated together (whole subject incl. attire, no reference photo); The Chosen One iteratively extracts an identity from a text prompt alone; PhotoMaker and InstantID encode a person's face from photo(s); ViStoryBench measures reference-matched characters per shot with detection plus face/CLIP embeddings. Each source names limits: single person only (PhotoMaker), largest face only (InstantID), no consistent supporting characters (The Chosen One), same style only (ConsiStory), and no background-reference evaluation (ViStoryBench). In ViStoryBench's full-set Table 2 a pasted reference image has higher CIDS (cross 0.929, self 0.984) than any generator row, so similarity must be read together with prompt alignme...
• articleIf I run COLMAP on a generated clip, what does 'reconstruction success' mean in the papers, what rates do named systems get, and how much should a pass or fail tell me?
  Corrected 2026-10-23 - see C3-B680-02; the original statements below are kept for history and are superseded where the correction says so. Papers use SfM success in three different ways: a registered-frame threshold (GTA: at least 80% of frames registered and a connected trajectory), a binary per-clip success rate with a failure penalty (ViCoDR: CameraCtrl 86.3% to 92.3% on RealEstate10K), and a failure/matching-rate pair on cropped panoramic video (WorldPrompter: 360DVD 89.7% failure versus 0%). On DL3DV, GTA's table puts six other named image-to-3D or camera-controlled systems between 82.9% and 88.1% against GTA's own 91.4%. Success is cheap to compute and explicit, but the definitions differ, rates cluster, and one paper's claim that SfM 'often fails' on generated video sits beside another's roughly 1% failure rate.
• articleHow do the TextDiffuser (MARIO-Eval) and AnyText (AnyText-benchmark) evaluations score rendered text, which OCR checks them, and what do the sources say about validating or breaking those checks?
  Corrected 2026-10-23 - see C3-B678-03; the original statements below are kept for history and are superseded where the correction says so. Two primary text-rendering evaluations use OCR as the judge. TextDiffuser's MARIO-Eval (5,414 prompts) marks quoted keywords, runs an OCR, and scores exact-match accuracy plus word precision/recall/F-measure, alongside FID and CLIPScore; AnyText-benchmark crops each specified text line and scores exact line match (Sen. Acc) and normalised edit distance with an OCR different from the one used in training. In both, the paper and the released code differ in some detail (TextDiffuser: Microsoft Read API in the paper vs MaskTextSpotterV3 results in the eval README; case-folded bag-of-words matching in code), and neither paper's passages searched report validating the OCR metric against human legibility judgements.
• articleWhat does each geometric-consistency metric miss: moving subjects, texture versus geometry, and clips that barely move?
  Corrected 2026-10-23 - see C3-B680-03; the original statements below are kept for history and are superseded where the correction says so. Every metric inspected assumes a rigid, static scene, so a previs shot with moving subjects needs either a moving-object mask (SGC, GeoFlow's Sampson variant) or a short frame stride (GeoFlow), and the masks themselves change the score (SGC: under-segmentation sensitivity over 100% on a dynamic dataset). Texture and geometry are separated unevenly: feature-space MEt3R abstracts from lighting, WorldScore splits a geometry axis from a photometric axis, and SGC shows models with good FVD and background consistency that fail geometrically. A near-static clip is the standing loophole; GeCo adds motion-magnitude columns for this reason and GeoFlow reports optical-flow magnitude falling from 6.9 to 2.6 pixels after consistency optimisation.
• articleDoes diffusers' torchao serialization always need safe_serialization=False and torch.load(weights_only=False)?
  Problem. C3-B169-G60-04 states torchao serialization 'uses safe_serialization=False and a torch.load(weights_only=False) workaround (pickle-capable)'. Correction type: (c) missing limitation. Method. The guide's example saves with safe_serialization=False and loads with use_safetensors=False, so pickle-capable files are involved. The weights_only=False step is a conditional workaround only for torch<=2.6.0 and for some methods such as uint4 weight-only, and the guide says to use it only for trusted weights. Limitations. Which torchao configs write safetensors on newer torch was not tested. Version and date. v0.41.0 docs, fetched 2026-10-23.
• articleIn the diffusers-vs-ComfyUI Qwen-Image-Edit thread, did anyone identify a practical cause of the quality gap?
  Problem. C3-B060-G23-07 says several causes were proposed and 'none was confirmed in the thread'. Correction type: (c) missing an important limitation/finding. Method. A contributor (linoytsaban) tested the same script with and without height=1024,width=1024 and reported that omitting them matched ComfyUI quality; a Qwen maintainer (naykun) then advised against custom output height and width and called that likely the main reason. The reporter later disputed that this explains everything, and other hypotheses (RoPE text-frequency base, prompt template, sigma shift) stayed unresolved. Practical rule: for Qwen-Image-Edit in diffusers, let the pipeline compute the output size and keep conditioning images matched to it. Limitations. No maintainer confirmed that the size effect fully closes the gap. Version and date. Thread state as fetched 2026-10-23.
• articleWhat exactly does WAVES say about StegaStamp's image quality and how many watermarks does its comparison cover?
  Problem. C3-B116-G42-06 says StegaStamp 'is most robust but visibly degrades images' and frames WAVES as '5 watermarks'. Correction type: (b) overstated plus scope. Method. WAVES only says StegaStamp 'may introduce artifacts' (a hedged report of the original paper), and its attack-vs-watermark comparison tables and robustness figure cover three methods (Tree-Ring, Stable Signature, StegaStamp); the paper does mention five stress-tested watermarks in its conclusion. The Stable Signature Regen-Diff average of 0.001 and the StegaStamp robustness ranking in the candidate are accurate (Table 3). Zhao et al. (not WAVES) attribute StegaStamp's resistance partly to lower visual quality and higher perturbation. Limitations. The two other watermarks' results were not located in the tables read. Version and date. Fetched 2026-10-23.
• articleDid WAVES show that surrogate-detector spoofing of image watermarks fails in general?
  Problem. C3-B116-G42-07 says WAVES 'tried spoofing via surrogate detectors and reports it failed'. Correction type: (b) overstated. Conditions. Applies when judging whether watermarks can be forged. Method. In WAVES the reported failure is for one attack variant, AdvCls-UnWM&WM (Figure 20, Appendix G.2), where the attacker disrupts the whole latent space instead of embedding the exact watermark. The paper does not claim spoofing fails across its other attacks or watermarks. Saberi et al. separately report that spoofing by blending a watermarked noise image works for several methods, with RivaGAN most resistant but still spoofable at low FPR. Limitations. The rest of WAVES' spoofing appendix was not read line by line. Version and date. WAVES v3, Saberi v2, fetched 2026-10-23.
• articleIs the LTX-Video repository and its Open RAIL-M / Open Weights licence summary still the current picture of the LTX family?
  Problem. C3-B133-G48-05 and -06 describe LTX-Video (0.9.x to 0.9.8) licences and memory figures as the LTX state. Correction type: (e) changed since. Conditions. Applies when choosing an LTX checkpoint or reading its terms. Method. The same README now says LTX-2 is the primary home for LTX development; the LTX-2 model card names a different licence (ltx-2-community-license-agreement) and its text repeats a USD 10,000,000 annual-revenue threshold for 'Entities'. The candidates' LTX-Video facts (8n+1 frames, divisible-by-32 sizes, 0.9.5 Open RAIL-M, 0.9.6+ Open Weights Licence 0.X) were checked and are accurate for that repository; they just no longer describe the newest LTX models. Failure modes. Reading the old licence as covering LTX-2 weights. Limitations. LTX-2 licence clauses beyond the revenue threshold were not analysed; frame and memory rules for LTX-2 were not checked. Version and date. Fetched 2026-10-23.
• articleCan I treat ViStoryBench's copy-paste CIDS baseline (cross 0.929, self 0.984) as a stable reference when comparing my own character-consistency runs?
  Problem. C3-B227-G69-06 (and the Table 2 sentence in C3-B227-G69-01) present the copy-paste baseline CIDS 0.929 cross / 0.984 self as a fixed reference. Correction type: (c) missing an important limitation / (e) changed since the paper. Conditions. Applies when you re-run the ViStoryBench CIDS evaluator and compare to the paper's copy-paste row. Method. The evaluator README states that, for characters with a single reference image, the current code computes the copy-paste score with a mirrored reference image, and that the paper and project-homepage numbers pre-date that fix. Rerun the baseline yourself with the same code version as your model rows instead of importing 0.929/0.984. Limitations. The README says only 'slightly different'; the size of the shift was not measured here. Version and date. README at pinned commit b44ec91 (identical bytes at main, fetched 2026-10-23).
• articleWhere does the 'optical-flow magnitude falls from 6.9 to 2.6 pixels' figure for GeoFlow come from?
  Problem. C3-B139-W28-06 attributes to GeoFlow a flow-magnitude drop (6.9 to 2.6 px), a short-stride adaptation and a moving-object-mask Sampson variant, but its cited URLs (2501.06336, 2504.00983, 2510.21615, 2512.22274) do not contain these figures; 2510.21615 only cites GeoFlow. Method. The figures are in GeoFlow itself, arXiv 2605.18365, which the candidate does not list. Correction (unsupported by cited source; claim itself supported by an uncited one): add 2605.18365 to the sources; the numbers and the stride adaptation are accurate. Limitations. Only the two quoted statements were confirmed. Version. arXiv 2605.18365, current HTML.
• articleIs there a paper in the COLMAP-success set that reports a roughly 1% failure rate on generated video?
  Problem. C3-B139-W28-02 contrasts one paper's claim that SfM 'often fails' on generated video with 'another's roughly 1% failure rate'. Method. Searched all four cited papers (GTA 2605.12957, ViCoDR 2511.18991, WorldPrompter 2504.02045, SGC 2603.19048) for failure or success figures near 1%. None found: GTA success 82.9-91.4%, ViCoDR 86.3-92.3%, WorldPrompter reports 0% failures for its own method (89.7% for 360DVD). Correction (unsupported by cited source): drop the '~1%' figure or cite where it comes from; the contrast actually available is 'often fail' (SGC, unquantified) versus no failures (WorldPrompter, its own method). Limitations. A paper outside the cited list may carry a 1% figure. Version. arXiv HTML as fetched.
• articleDoes the 1,200-story Nonaka and Perry result show an LLM positivity bias in romance stories?
  Problem. Candidate C3-B150-S32-08 lists 'a positivity bias across 1,200 LLM stories (Nonaka and Perry)' as measured model behaviour in romance writing. Conditions. Checked against the HTML of arXiv 2510.18932v1 (September 2025). Method. The abstract says networks from over 1,200 stories, generated by four LLMs and a human-written corpus. So the 1,200 includes the human comparison set (255 collected, 168 used in analysis), not 1,200 LLM stories. The human set is science fiction only (genre labelled by Gemini-2.0-flash, Project Gutenberg stories 3,000-15,000 words). The measured quantity is signed character-network structure (density, clustering, positive edge weights), and nothing in the paper concerns romance as a genre (no occurrence of the word in the page text searched). Failure modes. Using the figure as evidence that LLM romance outlines skew happy or tightly knit over-reads it: the evidence is about character relationships in LLM short stories generally, compared with Gutenber...
• articleDoes C3-B095-G35-01 mis-describe the one differing ElevenLabs clause 4(h), given that the EEA version also keeps the HIPAA bar?
  Problem. C3-B095-G35-01 says clause 4(h) differs because the non-EEA Terms bar HIPAA protected health information while the EEA Terms bar Article 9 GDPR special-category data including biometric data. Conditions. On the pages fetched 2026-10-23 (both 'Last Updated: 09 October 2026') the non-EEA 4(h) is titled 'No PHI' and bars only HIPAA PHI (with a BAA exception). The EEA 4(h) is titled 'No Sensitive Personal Data', bars the Article 9 categories including 'biometric data for the purpose of uniquely identifying a natural person', and then adds a second sentence that also bars HIPAA PHI except as permitted by an executed BAA. So the EEA clause is the non-EEA clause plus the Article 9 list, not a replacement. Method. When comparing the two Terms, describe EEA 4(h) as a superset. Failure modes. Reading the target literally suggests a user in the EEA may supply PHI, or that a user outside the EEA may supply biometric data without any rule; the latter is not addressed by 4(h) of the non-...
• articleDoes C3-B111-G40-06 miss that TextDiffuser reports a human evaluation and that its README gives a different MARIO-Eval size from the paper?
  Problem. C3-B111-G40-06 says the paper and released code of TextDiffuser differ in some detail and that neither paper's searched passages validate the OCR metric against human legibility. Conditions. Two points were left out. (1) TextDiffuser (arXiv 2305.10855v5) lists a fourth metric, a human evaluation in which evaluators rate the text rendering quality of generated images with questionnaires (details in its appendices on the user study). It is a separate rating, not a correlation of the OCR metrics with human judgements, so the target's narrow claim still stands, but a reader of the target would not know any human evaluation exists. (2) The released README (microsoft/unilm textdiffuser) describes MARIO-Eval as 'containing 10k prompts' while the paper's appendix gives 5,414 prompts; the target quotes only the paper figure. Method. Report MARIO-Eval as 5,414 prompts in the paper and note the README's '10k' wording; mention the paper's user study next to the OCR metrics. Failure mod...
• articleDoes C3-B116-G42-03 leave out that commercial audio detectors reached 0.89 accuracy in Deepfake-Eval-2024, and why the authors discount it?
  Problem. C3-B116-G42-03 summarises commercial detectors as better than open-source ones but 'none reached 90 percent accuracy', and then gives only video (0.78) and image (0.82 to 0.86) numbers. Conditions. Deepfake-Eval-2024 (arXiv 2503.02857v5) Table 5 (best commercial model per modality) reads video 0.78 accuracy / 0.79 AUC, audio 0.89 / 0.93, image 0.82 / 0.90 (image precision 0.99, recall 0.71). Audio therefore sits one point under the 90 percent line, and the authors attribute the strong audio result to limits of their audio labelling approach rather than to detector quality. Method. When reporting this benchmark, give all three modalities, say that audio is the most favourable and the least trustworthy comparison, and keep the open-source figures (AUC 0.43 to 0.63 off the shelf; real-media false-positive rates 0.31 to 0.39 video, 0.22 to 0.99 image, up to 1.00 for audio P3 in Table S8) alongside. Failure modes. Omitting audio makes the commercial range look uniformly weak (ab...
• articleDoes C3-B116-G42-02 overstate how well a ProGAN-trained detector holds up on GANs, and which training set do its Chameleon numbers belong to?
  Problem. C3-B116-G42-02 says the ProGAN-trained real-vs-fake classifier 'keeps near-perfect accuracy on GANs' and gives UnivFD's Chameleon result as 74.97 percent of fakes caught and 41.09 percent of real images passed. Conditions. Two conditions were dropped. (1) Ojha et al. (arXiv 2302.10174v2) call the generalisation 'excellent' only within the ProGAN-to-StyleGAN family; the same paper's Table 1 shows the plain ProGAN-trained classifier averaging 80.77 (CycleGAN) and 79.25 (GauGAN), with fake-image accuracy of 62.91 and 59.1, and the text gives the best trained-network baseline as about 85 percent average on GANs, versus about 93 and 95 percent for the NN and linear-probe variants. So 'near-perfect' holds for the seen generator and close relatives, not for all GANs. (2) In the AIDE paper (arXiv 2406.19435v3) Table 5 the 74.97/41.09 pair is UnivFD trained on SD v1.4 images, given as 'fake image / real image' accuracy. In the ProGAN-trained row the same UnivFD column reads 3.18/97....
• articleWhich monocular human-motion tools can I use in a commercial project, given code, SMPL and repo licences?
  Problem. Research mocap code often ships with non-commercial terms. Conditions. Commercial or client work. Method. (1) WHAM's LICENSE is MIT (copyright Soyong Shin), but its README requires registering at the SMPL and SMPLify sites to download body models, which have their own licence terms (not read in full here). (2) GVHMR's LICENSE permits use, copying, modification and distribution for educational, research and non-profit purposes only, requires modifications to be open source and prohibited for commercial use, and directs commercial users to email the authors. (3) So WHAM code may be usable but is gated by SMPL; GVHMR needs a separate commercial licence. Failure modes. Treating GitHub 'MIT' as covering the SMPL model. Limitations. SMPL's own licence page was fetched but not analysed in this round; dataset terms (AMASS) are covered by earlier items C2-B14-D2-09 and C2-B40-RD-05. Version/date. Fetched 2026-10-23.
• articleIs Hailuo's advice (surgical negative prompts, 20-30% token ratio, '94% preference') evidence, and what can an agent safely keep from it?
  Problem. Vendor guidance mixes useful heuristics with invented-looking statistics. Conditions. Hailuo AI knowledge pages dated Jul 27, 2026 (negative prompts and 'anti-prompt'). Claims. (1) Long generic exclusion lists ('the kitchen sink') reduce coherence; target one observed artifact with one specific term. (2) A negative prompt that opposes the positive motion (e.g. 'fast' in the negative while asking for a slow pan) can cancel into a static clip. (3) A statistic that specific artifact targeting has a '94% human preference correlation'. (4) A 'Token Ratio Rule': negative prompt about 20-30% of total prompt tokens. Evidence check. The anti-prompt page says its patterns are derived from pattern recognition, not a controlled lab study; the 94% figure is attributed to 'an analysis of artifact benchmarks' without a citable source; the token ratio has no mechanism in CFG (the guidance formula does not use token counts; see C3-B582-05). Safe to keep (as hypotheses to test): start from a...
• articleIn a controlled mirror-channel experiment on YouTube, did conversational or academic language retain viewers better, and how reliable is it?
  Problem: does a plain, conversational script keep viewers longer than a formal one? Conditions: Bassolas, Birello and Vicens (Eurecat; arXiv 2511.16497) uploaded the same energy-transition content to two new mirror channels, one conversational and one academic, in four formats (promoted complete video, non-promoted complete video, standard-format brief, short brief) across 15 experiment videos, promoted with Google Ads to get views (promotion raised views by about three orders of magnitude). Method: audience watch ratio and relative retention performance (compared with similar-length YouTube videos), tested with two-sided Mann-Whitney U at each point of elapsed time. Measured result: conversational complete videos had significantly higher relative retention early (time ratio 0.02-0.07, p between 0.011 and 0.033) while the academic channel was significantly higher in the middle (0.23-0.70); about 75% of the audience watched brief videos but about 15% watched complete videos in full. ...
• articleWhat counts as an impression and click-through rate in YouTube Analytics, why does CTR fall as a video reaches a wider audience, and what should I not do with CTR?
  Problem: misreading CTR as a thumbnail quality score. Conditions: YouTube Analytics 'impressions click-through rate' (CTR), long-form and general channels. Method: YouTube defines impressions CTR as how often viewers watched a video after seeing a registered impression; it covers only a subset of views because impressions on external websites and end screens are not counted, so views divided by impressions will not equal CTR. It states that half of channels and videos fall between 2% and 10%, with a wider range for videos under a week old or under 100 views, and that high-reach placements such as the Home page naturally lower CTR while channel-page impressions raise it. Compare CTR across videos over the long term and keep traffic-source mix in mind. Things to avoid per YouTube: deciding without enough impressions (do not check straight after upload), reacting to small CTR changes unless statistically significant, and testing several thumbnails/titles on one video sequentially becau...
• articleHow do I read YouTube's audience retention report: what do flat lines, slow declines, spikes, dips and the 30-second 'intro' figure mean, and when does the report exist?
  Problem: deciding which moments of a video to cut, front-load or expand from the retention graph. Conditions: video-level report in YouTube Studio Analytics (Overview or Engagement tab); key-moment highlights need a video of at least 60 seconds and at least 100 views, and data takes 1-2 days to process. Method (as documented): read four shapes. Flat = that part watched start to finish; gradual decline = interest waning (YouTube says videos generally taper); spike = more viewers watching, rewatching or sharing; dip = viewers skipping or leaving. 'Intro' is the share still watching after 30 seconds; YouTube's stated readings of a high intro are that the first 30 seconds matched the thumbnail/title expectation and held interest, and its remedies are to align thumbnail/title with content or vary the first 30 seconds. 'Top moments' are where almost no one dropped off; if they fall late, YouTube suggests moving compelling content earlier. 'Typical retention' compares your 10 latest videos...
• articleHow does YouTube's native title and thumbnail A/B test decide a winner, how long does it take, and what are its limits compared with third-party testers?
  Problem: picking a thumbnail/title without confounding by time and audience. Conditions: desktop YouTube Studio only; advanced features enabled; not for Shorts, Scheduled Lives or Premieres (Live Archives are allowed; a Premiere becomes eligible once converted); not for made-for-kids, mature-audience or private videos. Method: open the title or thumbnail box, choose A/B Testing, pick title only, thumbnail only or both, upload up to 3 variants. Changing the title or thumbnail mid-test stops the test. Results are based on watch time share, not CTR: 'Winner' (statistically significant), 'Performed Same' or 'Inconclusive'; with no clear winner the first variant uploaded stays default. YouTube says tests take a few days up to 2 weeks, depend on impressions and video age, and that near-identical variants run longer; a small control group may see only the default and is excluded from calculations. Thumbnails below 720p cause all test thumbnails to be downscaled to 480p. YouTube's stated re...
• articleWhat do YouTube's audience retention segments (new vs returning, subscribers vs non-subscribers, organic vs paid) and the detailed-activity report let me separate, and why can segment views exceed total views?
  Problem: a blended retention curve hides that different viewers behave differently. Conditions: Advanced Mode may be needed (the page notes 'you might need to enable Advanced Mode'); segments appear under the retention report 'See more'. Method: pick a segment drop-down to split the same video into new vs returning viewers, subscribers vs non-subscribers, and organic vs paid traffic (paid = skippable video ads and display ads). The detailed activity report shows absolute view counts per moment and where viewers started and stopped watching. Example the page gives: a segment's absolute views can exceed the video's overall view count because one viewer may watch portions several times. Failure modes: reading detailed-activity peaks as unique viewers; comparing a paid-traffic curve with an organic one as if the audience were equivalent. Limitations: documentation only, no benchmark values; paid-traffic definitions are specific to Google Ads video formats. Version/date: help page read 2...
• articleCan I detect clickbait YouTube videos from audience comments instead of the title or thumbnail, and how well did that work in the OVCP study?
  Problem: a title/thumbnail can look fine on its own; clickbait is a mismatch with the video content. Conditions: study of YouTube videos with comment threads; labels from three human annotators by majority vote. Method: Shang et al. (arXiv 1907.07604, Notre Dame) propose OVCP, which ignores the title, thumbnail and video and uses features from the viewers' comment network (structure, linguistic features, metadata) fed to supervised classifiers. Data: 500 training videos (128 clickbait, 372 non-clickbait, about 2.8M comments) and a random test sample of 125 trending-page videos (13 clickbait, 112 non-clickbait). Measured result: best model (AdaBoost) reported accuracy 0.896, precision 0.50, recall 0.46, F1 0.48, above the baselines (e.g. VGG-16 thumbnail model F1 0.15, SPW18 F1 0.36). Failure modes: with only 13 positive test videos, one video moves recall by about 8 points; precision of 0.50 means half the flags are wrong. Limitations: 2019 data, one platform snapshot, class labels ...
• articleHow do YouTube's Shorts and long-form analytics differ: which reports are format-specific, what is 'Stayed to watch', and how are views counted from 24 August 2026?
  Problem: mixing Shorts and long-form numbers when judging a hook or edit. Conditions: channel-level Content tab in YouTube Analytics, with chips for All, Videos, Shorts, Live, Posts, Playlists and Podcasts. Method: use the chip that matches the format. Long-form (Videos) shows a key-metrics card with views, average view duration, impressions and impressions CTR, plus the key-moments retention report. Shorts shows views, likes and subscribers, how viewers find Shorts (Shorts feed, search, channel pages), a 'How viewers engaged' panel and 'Stayed to watch', defined as the percentage of times viewers stayed past the initial seconds of a Short. 'Engaged views' counts viewers who stayed past the initial seconds and excludes loops; average view duration is computed from engaged views. 'Viewers across formats' shows overlap between videos, Shorts and live. The page states that from 24 August 2026 views are counted the moment a video starts to play in all formats, while Partner Program earn...
• articleHow well can multimodal LLMs flag misleading YouTube thumbnails, and what does this paper not show about their effect on views?
  Problem: an agent asked to review packaging needs to know how reliable automated 'misleading thumbnail' checks are. Conditions: Naveed et al. (arXiv 2607.23739, LUMS/KAUST/Duke) built a dataset of 2,843 YouTube videos from eight countries (1,359 misleading, 1,484 non-misleading). Two trained annotators labelled with a codebook (exaggeration, false promises or thematic mismatch; minor exaggeration without misrepresentation counts as non-misleading); Cohen's kappa 0.9633; only items with full agreement were kept. Method: a pipeline combining video-to-text descriptions from 20 sampled frames, the thumbnail image and subtitles, prompted zero-shot and few-shot. Measured result: Claude 3.5 Sonnet reached 93.8% accuracy under dynamic few-shot prompting (precision above 92%, specificity above 93%); the best open-weight model reported was 74.0%. Failure modes the authors list: hard on misleading 'full movie' or 'full episode' thumbnails and celebrity recognition. Limitations stated: Claude-g...
• articleHow do I use YouTube's 'When your viewers are on YouTube' report and the other Audience-tab reports to choose publish and live-stream times and topics?
  Problem: choosing when to publish or go live without folk wisdom. Conditions: YouTube Studio > Analytics > Audience tab; data windows are fixed by the report. Method (documented): 'When your viewers are on YouTube' shows when your viewers are online across YouTube in the last 28 days and is suggested for building community, picking a Premiere time and planning a live stream. Related Audience-tab reports: monthly audience (unique viewers in the previous 28 days, recalculated daily); channels your viewers watch outside yours (28 days) for collaboration ideas; other videos, Shorts, live streams and podcasts they watched outside your channel (past seven days) for topics, titles and thumbnail ideas; and a formats-watched report (28 days) for deciding between videos, Shorts and live. Failure modes: treating the online-time grid as proof that publishing at that time raises views; the page makes no such claim. Limitations: the page reports no evidence that upload timing changes performance;...
• articleWhat may other creators reuse from my Shorts (audio, Cut, Green screen, Collab, templates), how long can a sampled segment be, and how do I opt out?
  Problem: knowing what rights your Shorts carry once published, and what happens to derivatives when you restrict or delete. Conditions: YouTube Shorts remix tools on mobile; Studio for opt-out; Content Manager partners have separate controls. Method (documented): other creators can remix audio from Shorts and long-form videos ('Use this sound'), make a Cut, use a video as a Green screen background, Collab, or use a Short as a template; remixes are credited back to the source. Music-video limits: Cut samples a 1-5 second segment (audio and video); Sound samples up to 90 seconds of audio, and some partner agreements limit this to 30 seconds. Default: Shorts are automatically opted in to remixing; a creator can opt Shorts out of video remixing in Studio (Content > video > Show more > Shorts remixing), also in bulk and at upload; only Content Manager partners can opt out of both audio and video. Effects of restricting or deleting: existing audio remixes are muted, set to Unlisted and sc...
• articleDoes a rewritten post title change engagement when the same YouTube video is shared on Reddit, and what text features go with higher engagement?
  Problem: separating title wording from video popularity when judging packaging. Conditions: Hu et al. (Georgia Tech, SRI; arXiv 2505.03769, ACM manuscript) built a dataset of 24.1 million Reddit posts linking YouTube videos (19.8M video posts, 12.6M unique videos, 305K subreddits). They report that 21% of post titles are minimally modified from the video title. Method: controlled post pairs sharing the same or similar videos within a 30-minute window, plus an 'inverse' set where the less-popular video's post got more engagement; the mixed controlled set has 86,035 pairs. Measured result: title rewrites measurably relate to engagement; more engaging rewrites tend to show emotional resonance, lexical richness and alignment with community norms; a fine-tuned BERT classifier ranked titles at 74% accuracy in the controlled setting, while GPT-4o was only slightly better than chance. Failure modes: this measures Reddit votes and comments on titles, not YouTube clicks or watch time. Limitat...
• articleHow do I run a small, honest listening test with hidden reference and anchors (ITU-R BS.1534 MUSHRA), and what does that method not cover?
  Problem: compare several audio versions (codec, denoiser, TTS, separation) with a defensible test. Conditions: audio differences are in the small-to-medium impairment range (BS.1534 is for intermediate quality; BS.1116 is for small impairments). Method (BS.1534-3): present listeners with a known reference, several systems under test, plus hidden reference and anchors in one trial; the standard anchor is the original low-pass filtered at 3.5 kHz and the mid-quality anchor at 7 kHz; keep a trial to no more than 12 signals; use experienced listeners; post-screen assessors by excluding those who give a very high grade to a heavily impaired anchor or frequently grade the hidden reference as impaired (the Recommendation gives numeric thresholds of 15% of items, e.g. reference below 90 or mid anchor above 90); train listeners on the range of impairments first; analyse with the statistics in the annexes (confidence intervals; non-parametric options). The Recommendation also notes the name M...
• articleIn an open TTS evaluation, what are WER and speaker-similarity (SIM) computed with, and why can a lower WER mean worse speaker similarity?
  Problem: judge a TTS or voice-cloning output with objective numbers. Conditions: zero-shot TTS evaluated on a fixed test set. Method: the Seed-TTS evaluation (repo and paper) computes WER by transcribing the generated audio with an ASR (Whisper-large-v3 for English, Paraformer-zh for Mandarin) and comparing with the intended text, and SIM as the cosine similarity between speaker embeddings from a WavLM-large model fine-tuned for speaker verification, comparing each synthesized utterance with its reference clip. The released set uses 1,000 Common Voice samples (English) and 2,000 DiDiSpeech-2 samples (Mandarin); scripts are cal_wer.sh and cal_sim.sh. The paper cautions that a lower WER does not necessarily mean better subjective speaker similarity: lower WER often means more 'standardized', less accented or less expressive speech that the ASR finds easy. Use (inference): report WER and SIM together with a listening test; name the ASR and embedding models and versions, because scores ...
• articleWhat do ITU-R BT.1359-1 detectability and acceptability thresholds say about audio leading versus lagging picture, and which sign convention does it use?
  Problem: viewers tolerate audio lag and audio lead very differently, so a single symmetric +/- figure misleads. Conditions: broadcast television; thresholds are averages from subjective tests, original (1998) Recommendation. Method: BT.1359-1 states detectability thresholds of about +45 ms to -125 ms and acceptability thresholds of about +90 ms to -185 ms on average, where a POSITIVE value means sound is advanced (early) relative to vision. It recommends an overall sound/picture tolerance of +90 / -185 ms, +25 / -100 ms between image source and the zero reference, and +22.5 / -30 ms from final source selection to the transmitter input, with no downstream segment outside the broadcaster's control adding more than +/-2 ms. Its appendix explains the 22.5/30 ms limits as half a grade point above the detectability threshold on the five-point scale (one grade is about 60 ms for sound delay, about 45 ms for sound lead). Practical reading (inference): audio that lags picture is detected lat...

More