Evaluation and media provenance

Benchmarks, task datasets, attribution, metadata, interoperability and quality review

• articleWhen a paper says its audio is "in sync", what do DeSync, IB-score, AV-Align and the human scores actually measure, and how is each computed?
  Five different things are called "synchronization" in the sources. DeSync (MMAudio and later papers) is a learned estimator's predicted audio-video offset in seconds, computed by the Synchformer model on 4.8 s windows. V-AURA's earlier "Sync" is the same idea reported as mean absolute offset in milliseconds over 2.56 s clips. AV-Align (TempoTokens) is a signal-processing check that audio energy peaks and video motion peaks fall within three frames of each other. MMAudio also reports a model-free onset accuracy on Greatest Hits. Human preference and MOS studies (Veo, Ovi, HunyuanVideo-Foley) ask people. IB-score is not a timing metric: it is semantic similarity.
• articleI want to know whether a text-to-image model follows prompts: which benchmark covers object presence, counting, colour binding, spatial relations, dense prompts or rendered text, and what judges each one?
  Five primary sources, five different judges: GenEval (6 template tasks, detector + CLIP colour classifier), T2I-CompBench++ (8 sub-categories, B-VQA / UniDet / CLIP / MLLM per category), DPG-Bench (1,065 dense prompts, GPT-4 questions answered by mPLUG), MARIO-Eval (5,414 prompts, OCR keyword match) and AnyText-benchmark (positioned lines, OCR exact match and edit distance). None replaces human inspection; their numbers use different prompts and judges and cannot be compared across benchmarks.
• articleWhat does GenEval actually check in a text-to-image result, and how does it decide an image is right or wrong?
  GenEval (arXiv v1) turns a template prompt into object-level checks: it counts and locates objects with a COCO-trained instance-segmentation detector and checks colour of each cropped, background-masked object with CLIP zero-shot classification. Every image gets a binary pass/fail (all prompt elements correct), averaged per task and then across six tasks (single object, two object, counting, colours, position, attribute binding). It can only verify what the detector's 80 COCO classes and the chosen thresholds can see.
• articleWhat exactly does R2M-Bench compute for revisit memory (pair mining, MemoryGain, NMR), how well does it track humans and resist the slow-motion shortcut, what did it find, and what can be reused?
  R2M-Bench defines revisit memory as behaviour: after a commanded return the output should recover the earlier scene more than comparable non-revisit moments in the same video. For each revisit pair it samples a gap-matched non-revisit pair and a short-range pair from the same rollout, reports MemoryGain (revisit minus baseline) and a Normalized Memory Ratio (gain divided by short-range minus baseline) over five families (appearance, scene identity, object identity, local geometry, persistent state). On 100 scenes x 3 trajectories, DreamX-World-Memo leads (Overall NMR 0.706); raw similarity tracks humans better (0.594) than Overall NMR (0.547) but is coupled to low motion. The benchmark team and the top model share an organisation. Code and 100 reference images are MIT-labelled; you generate the videos.
• articleHow can an agent test whether a generated world 'remembers' a place it left: what revisit protocol, which measures, which failure cases, and what can be reused with original synthetic camera paths?
  The inspected papers converge on one shape of test: script a camera path that leaves a start view and returns to it (out-and-back, square loop, left-right yaw cycles, A-B-C-D-A loops), find the frame where the camera is back, and compare it with the first-visit frame. Measures range from PSNR/SSIM/LPIPS and DINO/CLIP/place-descriptor similarity to feature-match inlier ratios, depth-order correlation, pose-recovery error and VLM checklists. Three cautions recur: absolute similarity is inflated by a rollout that barely moves (R2M-Bench), the camera may not actually have returned (WorldExam, WorldMemBench), and one return is not enough (WorldMemBench found decay by the third). The reuse proposals below are labelled proposals; nothing was executed.
• articleThe ComfyUI repository ships openapi.yaml: can I generate a client from it for my local server, or does it describe something else?
  openapi.yaml at the pin is a union spec that describes more than the local server.py. Of its 79 operations, 20 match a server.py decorator; 59 do not (assets, workflows, settings, billing, tasks, /health for a GKE ingress, etc.). It documents authenticated, cloud-style semantics (402/403/429 statuses, ApiKeyAuth/BearerAuth, 'authenticated user' scoping) and omits client_id and prompt_id from PromptRequest, which the local /prompt reads and the official examples send. It also says /api/interrupt cannot target a job, but the local handler accepts a prompt_id. A generated client should be treated as a draft: verify each call against server.py and prefer /api/jobs over the deprecated history routes only where the local server implements them.
• articleWhat does a more detailed hierarchical outline plus a controller (DOC) add over Re3, how was it evaluated, and how did humans interact with it?
  DOC (Yang et al., v3 June 2023) expands a brief outline into a hierarchical depth-3 outline (2-5 children per node; settings and characters attached; relevance/coherence filtering) and uses an OPT-350m FUDGE-style controller to keep passages faithful to the outline; the drafting model is OPT-175B. In pairwise Surge AI ratings on passages from 20 stories, DOC beat Re3 on coherence (67.6 vs 45.1), relevance (65.3 vs 37.1) and interest (60.1 vs 39.4); ablations credit the outline for interest and both parts for relevance; in a 20-run human-interactive test authors strongly preferred DOC. Leaf-level outline faithfulness remained 58.5% with the controller and 37.8% without.
• articleWhat does ConStory-Bench measure about consistency errors in long LLM-written stories, how does its checker work, and how far can its checker be trusted compared with human readers?
  ConStory-Bench (arXiv v1, March 2026) gives 2,000 prompts for 8,000-10,000-word stories, a 5-category / 19-subtype taxonomy and an o4-mini based four-stage checker that outputs quoted contradictory spans with character offsets. The authors report that errors cluster in factual and timeline dimensions and near 40-60% of the narrative, and that the checker found 55% of 1,000 injected errors against 17.1% for two professional novel writers. That human comparison used LLM-injected errors and two annotators paid $1.00 per story, so it supports checker-as-triage, not checker-as-ground-truth.
• articleHow was FlawedFictions built and annotated, how well did models and humans do at finding injected continuity errors, and what failure modes did the authors report?
  FlawedFictions (arXiv v3, Dec 2025) injects a continuity error into a human-written short story by negating a first-act proposition in later acts, filters with an LLM judge and majority human verification, and scores binary detection plus two-way sentence localization. On 414 stories (average 731 words) the best model rows reach human level (accuracy 0.76) but most models are near chance; on 200 longer stories (average 2,703 words) the best CEEval-Full is 0.53, barely above the always-no-error baseline (0.51). Reported failure modes: over-reading motivations and genre conventions, miscounting entity states, and no gain from more reasoning tokens.
• articleWhat did each of these open interactive world models actually evaluate (metrics, baselines, raters, test sets), and what do the evaluations not support?
  The evaluations are author-run and use different protocols. Matrix-Game 1.0 uses its own GameWorld Score on Minecraft plus a double-blind human comparison against Oasis and MineWorld. Matrix-Game 2.0 reports the same score against YUME on wild scenes and is lower on motion smoothness, object consistency and scenario consistency. Matrix-Game 4.0's report and card give latency and throughput but no image-quality metric, and the card says it claims no benchmark. Hunyuan-GameCraft reports FVD, relative pose error, image quality and a 30-evaluator ranking on 150 images and 12 action signals, compared with camera-control baselines and Matrix-Game 1.0. LingBot-World 1 uses VBench on 100 generated videos over 30 s; LingBot 2.0 relies on comparison figures and an hour-long session. None measures persistence or revisit consistency quantitatively.
• articleWhat terms govern the non-text guardrail parts shipped in the Cosmos-Transfer1-7B snapshot: SigLIP so400m, the Resnet50_Final.pth face-blur model, NVIDIA's safety_filter.pt and blocklists, and nltk_data (wordnet, punkt_tab)?
  SigLIP carries an Apache-2.0 declaration in its card header (no licence file in the repo). The face-blur file Resnet50_Final.pth shares its name with a weights file that the upstream Pytorch_Retinaface project distributes (that repository is MIT-licensed; NVIDIA code cites it, but the origin of the bundled bytes is not verified); neither that licence nor its README states terms for the weights or the WIDERFACE training data, and the bundle file was not compared to upstream. NLTK data is governed per package: wordnet carries a Princeton permission notice, while punkt_tab is listed by the NLTK maintainers as an unclarified licence. NVIDIA's own safety_filter.pt and blocklist files fall under the NVIDIA Open Model License metadata of Cosmos-Guardrail1 (card unreadable). NVIDIA's ATTRIBUTIONS.md lists neither the SigLIP weights, the face-blur weights nor the NLTK data.
• articleIs the Hugging Face snapshot nvidia/Cosmos-Transfer1-7B@ef4dc8e3 the same release as GitHub nvidia-cosmos/cosmos-transfer1@5005e823, and where do they differ?
  Metadata does not show them to be one release. They are different artefact kinds (weights repo vs code repo) with no cross-reference by commit or tag, and 5005e823 (2026-06-07, a README-redirect merge) postdates every HF weight file (latest 2025-07-11) and the HF head (2025-11-19, README only). They are compatible by file name: all eight checkpoint paths the code names exist in the HF snapshot. They differ on guardrails: the code at 5005e823 loads Cosmos-Guardrail1 and Llama-Guard-3-8B from separate repos, while the HF snapshot bundles a guardrail/ folder (Cosmos-Guardrail1 contents plus an Aegis/LlamaGuard-7b folder) that predates GitHub commit b7b20fec, which replaced Aegis with Llama Guard 3.
• articleFor released weights of Real-ESRGAN, FILM, RIFE and Practical-RIFE, what do the repositories say or not say about terms inherited from the training datasets, per component?
  At the pinned commits, no repository README or training doc mentions inherited dataset terms for the weights. Code licences are explicit (Real-ESRGAN BSD-3-Clause, RIFE and Practical-RIFE MIT, FILM Apache-2.0). Only the Practical-RIFE README says its linked model files carry the same MIT licence as the project, and it names no training dataset. Real-ESRGAN names DIV2K as training data while the DIV2K page says academic research purpose only; the Real-ESRGAN files read do not mention that sentence. Whether one affects the other is a legal question and is left to curation.
• articleHow does TripoSG's MIT declaration sit alongside the Tencent agreements and the BRIA licence that its own files and NOTICE point to, and where do the texts conflict?
  At commit fc5c40990181e2a756c4e0b1c2f4d6b5202faf8c TripoSG declares MIT at the root while one file states that its use is subject to the Tencent Hunyuan Community License Agreement (with the agreement text embedded), a second licence file in triposg/ is the FlashVDM agreement, and NOTICE adds BRIA and Tencent code. The Tencent template lets modifiers add other licence terms 'except' where its sections 3 and 5 apply, and both agreements restrict territory and other-model improvement; whether and how those survive in an MIT-labelled release is not stated by any of the texts. Separately, FlashVDM's own source files name a Non-Commercial agreement while its LICENSE and NOTICE name the Community agreement, and RMBG-1.4's licence URL is dead. No conclusion on legal effect is drawn.
• articleIs TripoSG just MIT, or does its code include third-party parts under other agreements, and which files come from where?
  At commit fc5c40990181e2a756c4e0b1c2f4d6b5202faf8c TripoSG's root licence is MIT, but its NOTICE lists four third-party sources: BRIA RMBG-1.4 (licence bria-rmbg-1.4), Diffusers (Apache 2.0), HunyuanDiT (Tencent Hunyuan Community License) and FlashVDM (Tencent Hunyuan FlashVDM Community License). In the tree, RMBG is the file scripts/briarmbg.py; HunyuanDiT is declared in the header of triposg_transformer.py, which embeds the agreement text; FlashVDM's agreement is a second licence file in triposg/, and the flash decoder functions in inference_utils.py and attention_processor.py share much text with FlashVDM's files. The README has no licence section.
• articleWhat licence text covers the code and the weights of Real-ESRGAN, RIFE, Practical-RIFE and FILM, and where is it silent?
  Code licences are explicit: Real-ESRGAN BSD-3-Clause, RIFE and Practical-RIFE MIT, FILM Apache-2.0. Weight terms differ: Practical-RIFE states that the linked model files are under the same MIT licence as the project; the Real-ESRGAN, original RIFE and FILM READMEs and LICENSE files read make no separate statement about weights, which are distributed as release assets or Google Drive/Baidu links in pickle-capable or SavedModel formats. FILM additionally requires a third-party VGG .mat file whose terms were not read. No output-use terms were found in the documents read.
• articleHow do the papers measure quality degradation over rollout time (curves, drift deltas, judges), what do the numbers show, and where are the metrics unreliable?
  Four measurement styles appear: (1) an imaging-quality-versus-time curve over 30 s (CausVid Fig. 8); (2) a first-5-s versus last-5-s imaging-quality delta, "ΔDriftQuality", on 30 s videos (Rolling Forcing); (3) whole-video VBench-Long dimensions at 50/75/100 s plus a Gemini-2.5-Pro "visual stability" score (Self-Forcing++); (4) VBench dimensions at 50/250 s with an explicit warning that copied clips fool consistency metrics (SVI). The authors of Self-Forcing++ and SVI both say standard image/consistency metrics can reward degraded or static video, so motion (dynamic degree) must be read with them.
• articleHow long have these autoregressive video systems been shown to run, and which lengths are measured in a paper versus only shown in a figure or demo?
  Measured (table) lengths in the primary papers are 30 s (CausVid, LongLive, Rolling Forcing), 50/75/100 s (Self-Forcing++) and 50/250/300 s (Stable Video Infinity). Longer claims are qualitative figures or demos: 255 s (Self-Forcing++ Fig. 6), 240 s (LongLive appendix + demo page), over 2 min (Rolling Forcing Fig. 4), 1000 frames at 128 px (Diffusion Forcing, qualitative) and 8-14 min (SVI README). Self Forcing itself reports no quality measure beyond 5 s and states that degradation remains past the training length.
• articleWhat is the architecture of the "Generative Agents" LLM characters (memory, reflection, planning), stated plainly?
  Park et al. (arXiv 2304.03442, pinned v2, UIST 2023) describe an LLM wrapper with a memory stream of natural-language observations, a retrieval function that scores memories by recency, importance and relevance, periodic reflection that writes higher-level memories, and planning that turns the agent's summary and memories into daily plans. It was instantiated with gpt3.5-turbo in a 25-agent sandbox. These are the authors' reported implementation values, not a production game recipe.
• articleWhen an NLE's AI feature changes footage, what do the vendors' pages document about what is stored, what is lost on interchange, and what should an agent record in an EDL or project log (documented versus proposed)?
  Adobe documents that Generative Extend output is a new clip file with an AI-generated label, zero-based timecode, no transferred logging metadata or markers, and incomplete support for XML/AAF/EDL/OMF/OTIO until each extension is baked with Render and Replace. Resolve 21's guide says Motion DeBlur renders new media and keeps the originals; Avid keeps the original AI transcript beside edits; Resolve IntelliScript puts alternates on disabled tracks. The recommended log fields below are a synthesis proposal assembled from those documented behaviours, not a vendor or standard requirement.
• articleWhat does OpenS2V-Eval measure about keeping a given subject's identity in generated video, and what scores did closed and open subject-to-video systems get?
  OpenS2V-Nexus (arXiv 2505.20292, v3 read) is a single-clip subject-to-video benchmark, not a cross-shot one: 180 prompts in seven categories, one generated video per sample at seed 42, scored with NexusScore (detector-cropped subject versus reference), NaturalScore (GPT-based naturalness), GmeScore (text relevance), FaceSim, aesthetics and motion. In the human domain Hailuo has the top total (60.20%; FaceSim 57.79, NaturalScore 74.52) while VideoMaker has the highest FaceSim (76.45) but a NaturalScore of 47.08, and the authors say ConsisID and Concat-ID have relatively strong FaceSim yet significant copy-paste issues. EntityBench characterises OpenS2V as evaluating single-subject preservation within individual shots. Code is Apache-2.0; the dataset and leaderboard are on Hugging Face.
• articleWhat does EntityBench report about keeping characters, objects and locations consistent across up to 50 shots, under which metrics and settings, and which systems were compared?
  EntityBench (arXiv 2605.15199v1, 14 May 2026, ByteDance/Rice) is a benchmark of 140 episodes and 2,491 shots with per-shot schedules for characters, objects and locations, scored on 51 metrics in three pillars. Four systems were compared: HoloCine, CineTrans, StoryMem and the authors' EntityMem. The authors report that cross-shot consistency drops with the number of shots between appearances, that DINOv2 face similarity and LLM identity accuracy rank the systems differently, and that EntityMem (their own per-entity reference memory on top of StoryMem, no extra training) has the highest LLM character-identity scores (llm_face_accuracy 0.406 vs 0.226 StoryMem, 0.228 HoloCine, 0.091 CineTrans). Everything is authors-run; no code or data link was found in the paper bytes.
• articleWhat do the identity and consistency scores in video-generation papers (subject consistency, FaceSim, NexusScore, Inter-Sub.Con, ACP-Var, LLM-judge accuracy) actually measure, and what can each one miss?
  The papers use at least six different 'consistency' numbers that are not interchangeable: VBench-style subject consistency (DINO similarity of adjacent frames inside one shot), face-embedding similarity (FaceSim, InsightFace/ArcFace) to a reference or between shots, NexusScore (detector-cropped subject versus reference by a retrieval model), Ref-Sub.Con versus Inter-Sub.Con (to the reference image versus between generated shots), anti-copy-paste scores (ACP-Var, CP-Rate), and VLM or human judgements (EntityBench same/different accuracy, PersonaShot MOS-aligned evaluators). Each can be high while identity is wrong: within-shot DINO scores ignore cuts, embeddings can reward generic faces, face scores ignore costume, and reference-fidelity scores can reward pasting the reference. Scores from different papers use different data, models, and settings and must not be compared across tables.
• articleHow can an agent check whether a character or location stayed the same across the shots of a generated sequence, and which checks mislead?
  Published benchmarks give a concrete checking recipe an agent can follow: detect and crop each scheduled entity per shot from a written registry description, compare crops by DINOv2 embedding to an order-invariant centroid, add a face-ID model for faces, ask a vision-language model a same/different question per crop against an anchor, admit only crops that passed a per-shot fidelity check, and plot similarity against the number of shots between appearances. EntityBench reports that embedding similarity and LLM identity judgement can rank methods in opposite order (DINOv2 face similarity favoured StoryMem 0.792 vs 0.737 while LLM face accuracy favoured EntityMem 0.406 vs 0.226), so one number is not enough. All of this is the benchmark authors' protocol; nothing here was run by this worker.
• articleWhat could an agent use to check a long draft or a series against its own established facts, and which parts are measured results versus proposals?
  Measured: LLM checkers can find injected contradictions in short stories at roughly human level for the best model and drop toward chance beyond about 1,200-4,000 words (FlawedFictions); a four-stage extract-pair-evidence checker with released prompts found 55% of planted errors in about 8,000-10,000-word stories (ConStory). A 2023 KG+GNN approach on synthetic negation did not beat humans. Proposed only: an explicit fact ledger that is filled from earlier text and queried when new text is written; no inspected paper tests it. This candidate separates the two and defines an experiment.
• articleWhich checks on LLM-written game or story state are actually supported by published tests, and which are only suggested?
  Four checks have a published test in the sources read: (1) rules in the prompt (Wang: game progress 61.5% to 92.1%); (2) deterministic script checks of event conditions and variable updates (RPGBench; they measured errors but did not test whether adding the checker as a retry loop reduced them); (3) a BFS reachability check on a generated game definition (RPGBench game creation; 125 of 500 LLM-generated games valid); (4) an LLM auditor with an explicit fact ledger (NCP-Bench; measures failure, auditor shifts progress numbers). Giving the narrator an explicit state table, and a validator-in-the-loop retry for narrative, are proposals; the only validator-corrected pipeline is Guan's PDDL (planning, not narration).
• articleHow often do LLM narrators contradict established story facts or plot commitments over many turns, and what checks did the benchmark use?
  NCP-Bench (arXiv 2608.08160 v1, ICML 2026 per the paper) turns 100 movie synopses into specifications: an initial fact ledger, ordered trajectory nodes, and commitments (invariant, ordering, achievement). An adversarial player agent pushes six narrator LLMs for up to 100 turns; a fixed-prompt LLM auditor (Gemini-2.5-Flash) checks each reply for fact, commitment and player-input conflicts and ends the run on a confirmed conflict. Best model GPT-5.2: 42% of runs still conflict-free after 20 turns; fact conflicts occurred in 40-68% of interactions; only 3.5% of all 600 runs lasted 100 turns. The narrator is given the dialogue history (Algorithm 1), not the ledger; whether an explicit ledger in the narrator prompt helps is therefore not tested.
• articleHow comparable are the evaluation protocols (datasets, subsets, alignment, baselines) across TripoSR, SF3D, SPAR3D, TRELLIS, TRELLIS.2, Hunyuan3D and TripoSG?
  Protocols differ enough that absolute numbers should not be compared across papers. TripoSR, SF3D and SPAR3D share GSO and OmniObject3D but use different subsets (about 300 each; 278 and 308; about 250 each) and different GPUs; SF3D re-measured TripoSR at CD 0.111 / FS@0.1 0.645 on GSO while TripoSR reports CD 0.111 / 0.651 on its own subset. TRELLIS uses Toys4k with depth-unprojected points; TRELLIS.2 adds 90 curated Sketchfab assets and 100 AI-generated prompts; Hunyuan 2.0 anonymises several baselines; TripoSG evaluates shape-only with Normal-FID and an LMM judge. Only SF3D, SPAR3D and TripoSR share a metric and datasets closely enough for rough comparison, and even they were each run by the authors of the newer method.
• articleFor open music/audio models (Stable Audio Open, MusicGen), what do code, weights, training data and outputs each permit?
  Components differ. Stable Audio Open 1.0 and Small: code in stable-audio-tools is MIT; weights are gated on Hugging Face and the public card names the Stability AI Community License (commercial registration, revenue threshold, attribution) with output ownership stated 'as between' Stability and the user; training data are stated as CC0, CC BY or CC Sampling+ audio with content-detection screening. MusicGen (AudioCraft at commit 896ec7c47f5e5d1e5aa1e4b260c4405328bf009d): code MIT, weights CC BY-NC 4.0, training data stated as licensed (Meta Music Initiative, Shutterstock, Pond5). Both model cards say they should not be used downstream without further risk evaluation. SAO's own card says vocals are not realistic and MusicGen's says the same.
• articleHow well do AI-generated video detectors trained on some generators work on other generators and on compressed, cropped or watermarked video?
  GenVideo (arXiv 2405.19707 v3) trains 15 model configurations (Table 2) on over a million real and generated clips and tests on a 10-generator out-of-domain set built from MSR-VTT real videos. The authors DeMamba-XCLIP-FT reaches average recall/F1/AP 0.9392/0.9020/0.9710 when trained on many generators, but only 0.7572/0.7263/0.8166 when trained on a single generator (Pika), with Sora at F1 0.43. Degradations hurt unevenly: cropping dropped its recall by 0.2244 and JPEG dropped F1 by 0.2197 while CRF=28 compression cost 0.0250 F1. Real-video false positives are not separately reported in the tables read, and the real test videos come from one source.
• articleHow much does JPEG compression, blur, resizing or re-encoding reduce image-detector accuracy, according to benchmark papers?
  For images the pinned papers report JPEG and blur tests only; no paper read tested resizing or re-encoding of image detectors. In the AIDE authors run on AIGCDetectBenchmark, JPEG at quality 95 cut LNP accuracy from 83.84 to 53.58 and AIDE from 92.77 to 75.54, and quality 50 cut AIDE to 69.60; blur effects were mixed and some detectors improved. Ojha et al. report their detectors, trained with blur+JPEG augmentation, as generally robust and argue extreme blur is not worth defending against. Both results are on benchmarks where the fakes are not necessarily post-processed the way real platforms do.
• articleCan I treat a score from an AI-image or AI-video detector as a verdict that a file is AI-generated or authentic?
  No. In the benchmark papers read, detectors that score near 1.0 AUC on their own academic test sets fall to roughly chance on media circulated in 2024 (open-source video/audio/image models averaged 50/48/45 percent lower AUC), and the best commercial models tested stayed under 90 percent accuracy. Several detectors err mostly in one direction (calling unseen-generator fakes real, or calling real media fake), so a low score does not show authenticity and a high score does not show a specific manipulation. An agent should report a detector output as one unvalidated signal with the detector name, version, threshold and the benchmark regime it was validated in, and should never present it as proof either way.
• articleWhen an LLM writes several characters' lines, how can I choose among candidates so characters stay distinct from each other?
  TalkMatrix (arXiv v1, 20 Jul 2026) generates k=16 candidate lines per character-situation cell with Qwen3.6-27B-FP8 (Japanese) and picks one per cell jointly to maximise the worst of four embedding-based objectives, one of which is character diversity (mean pairwise cosine distance between different characters' lines in a situation, using a content embedding plus a style embedding). On 25 curated board-game scenarios an LLM judge (GLM-4.7-AWQ) scored the minimax configuration +0.98 over random (6.44 vs 5.46, 17 wins/3 ties/5 losses); independent per-cell reranking by character consistency gave +0.02. Claims are limited to Japanese role-playing scenarios, a model judge and no human ratings.
• articleWhat does generating one minute of 1080p AI video cost per model on vendor APIs, and how does price relate to measured quality (Oct 2026)?
  Artificial Analysis video boards (rendered 2026-10-23; Elo anchored at Kling 3.0 1080p Pro = 1000; prices = 1 minute of 1080p on the creator's API at default settings). Text-to-video: Wan 3.0 1175 ($12.00/min), HeyGen Video 1 1160 ($2.40), MiniMax H3 768p 1158 ($4.80), Seedance 2.5 1158 ($34.12), FLUX 3 Max 720p 1155 ($2.55), MiniMax H3 Max 1154 ($4.80), FLUX 3 1145 ($17.40), Seedance 2.0 1136 ($22.45), Gemini Omni Flash 1.1 1130 ($9.12), Grok Imagine 1.5 1056 ($15.00), Kling 3.0 Omni 1020 ($8.40), Kling 3.0 1000 ($10.08), Veo 3.1 963 ($24.00), Veo 3.1 Fast 960 ($7.20), LTX-2.5 Fast 949 ($7.80), LTX-2.3 Pro 888 ($4.80). Image-to-video: MiniMax H3 768p 1140 ($4.80), FLUX 3 Max 1138 ($2.55), Seedance 2.5 1138 ($34.12), Gemini Omni Flash 1.1 1132 ($9.12), Wan 3.0 1127 ($12), ... Veo 3.1 1028 ($24), Kling 3.0 1000 ($10.08). CIs are about +/-15 Elo, so the top 6-8 are statistically close while prices differ by >10x (Seedance 2.5 vs FLUX 3 Max / MiniMax H3). This quantifies creator report...
• articleWhich TTS voices and music models lead blind preference tests in Oct 2026, and at what API price?
  Artificial Analysis Speech Arena, provider-voice board (rendered 2026-10-23): Eleven v4 Turbo 1335 (+/-17; Sept 2026; $40/1M chars), Eleven v4 1329 ($80), Qwen-Audio-3.1-TTS-Plus 1301 ($19.3), Sonic 3.6 1289 (Aug 2026; $49), Gemini 3.8 Flash TTS 1278 ($16.5), Realtime TTS-2 1267 ($20.8), Qwen-Audio-3.0-TTS-Plus 1265, Gemini 3.8 Flash-Lite TTS 1251 ($11), Fish Audio Drama 3 Preview 1248 (Oct 2026; $15), Simba 3.2 1245 ($6.6), Voxtral 3 TTS 1218 ($13); ~2,000-3,400 samples each, 8 voices per model. AA-Music-Instrumental v1.1 (anchor MiniMax Music 3.0 = 1000): Suno v6 1148 (+/-22), Suno v6-mini 1114, Mureka V9.5 1084, Mureka V9 1083, Lyria 3 Pro 1080, Lyria 3.5 1078, Eleven Music 1021, Stable Audio 3 Large 1014, Eleven Music v2 1011, MiniMax Music 2.5+ 1007, ..., Lyria 2 926, MusicGen 868 (~2,100-2,300 samples each; separate vocal board exists, not captured). Use: price/quality trade-offs are large (e.g. Gemini 3.8 Flash TTS within ~57 Elo of Eleven v4 at about a fifth of the price); F...
• articleHow could an agent check one generated image against its prompt, using what these benchmarks do, and how far can it trust each check?
  Proposal (synthesis, untested): split the prompt into elements; check counts, positions and colours with detector/crop-classifier rules, quoted text with an independent OCR and exact match, and open relations with per-element yes/no VLM questions that carry parent dependencies; repeat the VLM judge and keep a human-labelled sample to measure agreement. Each piece is documented in the benchmarks (GenEval, T2I-CompBench++, DPG-Bench, MARIO-Eval, AnyText) but the combined agent procedure has not been reported or tested in these sources.
• articleCan I say current video models understand physics? What do the newer benchmarks support me saying, and what must I avoid claiming?
  After reading six 2025-2026 benchmark papers, a safe statement is: on each paper's own tests, current video generators score well below ceiling on many physical behaviours, do better on some (reflection, buoyancy, communicating vessels, sand-pile angles) than others (momentum, collisions, melting with state change, persistence after the camera looks away), and visual quality does not predict physical consistency. Unsafe statements: that any model 'has learned the laws of physics', that a score is comparable across benchmarks, or that open or closed models are generally ahead. The papers do not test training-distribution generalisation, so they neither confirm nor refute 'learn' in the mechanistic sense.
• articleIf I re-run the same ComfyUI workflow with the same seed, which parts of the code and the docs decide whether I get the same image, and what should I write down so the run can be repeated or its drift explained?
  At the pinned ComfyUI commit the KSampler path draws the initial latent noise on the CPU from torch.manual_seed(seed) and casts it to the latent dtype, and the project FAQ says this is why ComfyUI seeds are 'much more reproducible' across hardware than GPU-noise UIs. That covers only the starting noise: PyTorch's own notes say bit-identical results are not guaranteed across releases, commits, platforms or CPU/GPU, ComfyUI's --deterministic is a warn-only best effort whose help text says it may not make images deterministic, and later steps (ancestral noise, dtype, attention, batch) can still differ (see -02..-04). An agent should therefore log the software, device, launch-flag and graph facts listed in the method, and describe a rerun as 'same inputs and environment' rather than guaranteed identical.
• articleGive me a step-by-step check for whether a draft's unreliable narration (prose, voice-over or flashback) is fair to the audience.
  No inspected source supplies a validated fairness test. This candidate assembles an eight-step agent check from the sources (labels, ground truth table, signal audit, rival-explanation audit, image-lie audit, hindsight audit, role-stereotype audit, fact audit) and marks each step with its source support. It is a synthesis proposal; the thresholds are not sourced. LLM classifiers for unreliability are reported to be weak outside within-narration tics, so an agent should do the audit with explicit tables and use a model label only as a hint.
• articleIf an image has no detectable invisible watermark, is it human-made, and if it has one, does that prove who made it?
  No to both, on the papers read. Absence of a watermark is weak evidence: regeneration attacks removed pixel-level watermarks in over 99 percent of cases (Zhao et al.) and Stable Signature almost entirely (WAVES), and a generator may not mark its output at all. Presence is also not proof: one paper shows real images can be made to read as watermarked using a marked noise image, and the better-resisting schemes still have attack settings that work. What a vendor says it marks is in C3-B033-G12-08 and C3-B046-G16-07; this candidate covers the independent tests.
• articleBefore an agent hands an AI-generated roto/matte to compositing, what should it check, which of those checks do the sources actually support, and which have no source-given test?
  Synthesised from the pinned sources: record tool and model version; decide mask versus alpha; scrub frames at the failure scenarios the authors name (shot changes, crowds, long occlusions, thin fast structure, similar objects in background); check core holes and background specks; compare the edge band with an outline view; judge temporal chatter on a static subject; check the delivery resolution against the model input; bake the matte to pixels; and record licence status. The sources support the failure list and the controls but give no pass/fail threshold; the one ground-truth-free quality metric found (MatAnyone 2 unreliable-pixel rates) depends on an evaluator checkpoint that is not released at the pinned README. Every checklist item below is the lead's synthesis from vendor and paper statements, not a published standard.
• articleWhich provenance records (C2PA, IPTC/XMP, SPDX, REUSE) answer which intake question, what do they not record, and what happens to them when the asset is copied or re-encoded?
  Origin records and licence records answer different questions. C2PA 2.4 describes provenance and admits a manifest can be separated from its asset when metadata is stripped (fingerprints may help recovery); for assets without a manifest it uses XMP DocumentID/InstanceID as identifiers. IPTC digital source type values say how media was made (for example trainedAlgorithmicMedia) and say nothing about reuse permission. REUSE 3.3 records copyright and licence per file via comment headers, .license sidecars (allowed for non-text files such as images) or REUSE.toml, but scopes itself to software projects, so media use is by analogy. SPDX 3.0.1 AIPackage describes AI packages. Synthesis (producer, supported): keep provenance and licence records alongside the file (sidecar or database) with the file hash at receipt, and never read an AI-generated source type as a licence statement.
• articleWhich of these open 3D generators can be used commercially, and what territory, user-count or revenue limits do their licence texts state?
  Three licence families appear. TRELLIS, TRELLIS.2, TripoSR and TripoSG state MIT for code and models (README/LICENSE text; Hugging Face metadata agrees), with listed exceptions for submodules or dependencies. Hunyuan3D 2.0 and 2.1 use dated Tencent community licences whose text excludes the European Union, United Kingdom and South Korea from the licensed Territory, requires a separate licence above 1 million monthly active users at the release date, and forbids using the works or outputs to improve another AI model. SF3D and SPAR3D use the Stability AI Community License (Last Updated July 5, 2024): free for research/non-commercial; commercial use needs registration and ends above USD 1,000,000 annual revenue. Texts address use, distribution and outputs differently; none of this is legal clearance.
• articleWhat failure modes (identity drift, geometry violations, flicker) do sources report when photoreal video is generated from engine renders or structural controls?
  Reported failure modes come from papers and vendor caveats, not studio accounts. RealMaster (research, GTA-V) reports identity drift and colour shifts in an Aleph baseline, flicker and identity fluctuation from multiple image-edit anchors, face-structure loss under depth conditioning, and temporal artifacts under fast motion. NVIDIA reports hallucinated or misoriented vehicles for Transfer1 AV variants and error accumulation over long clips, and Transfer1 papers show geometry errors for single controls (segmentation-only headings). Vendor docs warn that high control weight can cost quality and that image-sourced subjects may be inconsistent.
• articleWhich adherence and temporal-consistency metrics do the Cosmos-Transfer, VACE, RealMaster and Enhancing Photorealism Enhancement papers report, and can their numbers be compared?
  Adherence is reported only by the Cosmos-Transfer papers (control re-extraction metrics: Blur SSIM, Edge F1, depth si-RMSE, mask mIoU). Temporal consistency is reported by VACE and RealMaster through VBench Temporal Flickering/Motion Smoothness, and by Transfer2.5 as a DOVER-based error-accumulation curve; the Transfer1 paper text read defines no temporal-consistency metric. The same Transfer1-7B configuration scores differently in the Transfer1 paper (TransferBench) and the Cosmos 2.5 report (PAIBench-Transfer), so numbers do not carry across papers.
• articleIs there a benchmark for whether a generated world keeps an object or event going after the camera looks away (object permanence), and what do 23 models score?
  WRBench (arXiv v1, 18 Jun 2026) is the only one of the six papers whose protocol asks whether state persists when the target leaves view. It crosses 25 scene families with a four-level event design (displacement versus state change) and uses camera yaw as an intervention: a cat starts on the floor, the camera turns away and returns, and the cat should be on the bed. Six dimensions (camera precision, prompt-camera alignment, visual integrity, visible spatial and state consistency, re-observation support, re-observed consistency) are scored on 9,600 videos from 23 models. The authors report that re-observed event state stays about 0.62 to 0.66 even when the object returns, does not improve from Wan 1.3B to 14B (0.66 to 0.62), and is only judgeable in 2,073 of 9,600 videos.
• articleHow does Principia test gravity, friction, momentum and similar laws without knowing scale or camera, and what do six generators and four VLMs score?
  Principia (arXiv v1, 3 Sep 2026; project page says accepted to NeurIPS 2026) filmed paired objects under matched conditions (two blocks of different mass on identical inclines, hollow versus solid cylinders, pendulums of different lengths) and asks whether generated videos keep the pair's invariant (equal acceleration, equal period ratio, distance ordering). A unit-free score S = 1 - |F1 - F2| / (|F1| + |F2|) is computed in pixel space after SAM3 tracking. Six generators score between 0.148 and 0.419 Overall (Wan2.2-14B best, ahead of closed Veo-3.1 at 0.379) while all score about 0.8 on VBench; momentum is near zero for every model. Four VLMs asked to judge PASS/FAIL on real and synthetic anti-physics clips average 0.50 to 0.66 agreement.
• articleWhat does PhyWorldBench test beyond ordinary physics (anti-physics prompts), how does its MLLM judge compare with humans, and did its numbers change between v1 and v3?
  PhyWorldBench v3 (revised 26 May 2026) has 1,050 prompts over 10 physics categories (kinematics, interaction dynamics, energy, fluids and particles, lighting and shadows, deformation and others, plus human/animal motion and an Anti-Physics level) and scores videos with three-annotator Yes/No majority votes on semantic adherence (SA), physical commonsense (PC) and both. Twelve text-to-video models were scored; the best, Pika 2.0, reaches 0.262 on 'Both' and anti-physics prompts are far lower (0.000 to 0.042 for most). All twelve overall rows in Table 3 are identical between v1 (Jul 2025) and v3 by my own comparison, while v3 adds precision/recall tables and a CAP neutrality check. The authors' Context-Aware Prompt (CAP) judge reaches ROC-AUC 75.1 for physical commonsense versus 61.6 for plain GPT-o1.
• articleWhich physics benchmarks for video and world models appeared after early 2025, what does each test, how is each scored, and what do current systems score?
  Six papers read at pinned arXiv versions (VideoPhy-2 v1, PhyWorldBench v3, PhyGround v1, World Models' Last Exam in Physics v1, Principia v1, WRBench v1) define post-early-2025 physics tests for video generators. They differ in what is tested (everyday actions; 50 sub-categories with anti-physics prompts; 13 laws; 40 lab-style tasks; paired-object invariants; persistence after the camera looks away), in who scores (crowd humans, VLM judges, or measurements extracted from video), and in scale, so their headline numbers cannot be compared. Every paper reports that the evaluated systems fall well short of its ceiling; none of them contradicts the earlier conclusion, and none tests training-distribution generalisation.
• articleHow can an agent measure whether a mystery's reveal is both surprising and fair (set up so the clues add up in hindsight), and what do the open metrics and their human check actually show?
  Wagner, Keydar and Abend (arXiv v2, May 2026) define surprise as how badly a "gullible" reader (an LLM told to take events at face value) predicts the culprit paragraph by paragraph, and coherence as how well clue-competent readers predict it, and argue theoretically that fair play needs a hindsight gap. On whodunit stories their metrics rank Christie above Conan Doyle for surprise and fair play, and LLM-generated mysteries rarely reach both; a 20-person human check gives small correlations (r about 0.1 to 0.25). It is a usable check design for detective outlines and a research result, not a validated test, and code and data were not yet released.

More