Evaluation and media provenance

Benchmarks, task datasets, attribution, metadata, interoperability and quality review

• articleHow is epipolar (Sampson) error computed on a generated clip, what does it say about geometry versus texture, where does it break for moving subjects, and what do papers report for named models?
  Epipolar error needs no learned reconstruction: match SIFT features between sampled frames, fit a fundamental matrix with the normalized 8-point algorithm in RANSAC, and average the Sampson distance of the matches to their epipolar lines; lower means the pair obeys one rigid camera motion. The Epipolar-Geometry paper reports 0.190 for a Wan2.1-1.3B baseline versus 0.131 after preference finetuning on Sampson-ranked pairs, with held-out-frame splat fidelity and human 'consistent' rates moving the same way (54.1% to 71.8%). It assumes a static scene (the authors say moving objects corrupt it), can be fooled by repetitive texture or a near-static clip, and GeoFlow shows a masked, short-stride variant for dynamic clips.
• articleWhat does a MEt3R score for a generated clip actually measure, how is it computed, what values do real and generated clips get, and what should I not read into it?
  MEt3R (arXiv v2) scores a pair of frames: DUSt3R reconstructs a pose-free pointmap pair, upscaled DINO features of both frames are projected into the first frame's camera, and the score is one minus the mean cosine similarity over the overlap (0 to 2, lower is better; no camera poses needed, so it works on generated video). In the paper, real videos sit slightly above zero, SVD scores 0.032, Ruyi-Mini-7B 0.047 and I2VGen-XL 0.050 on 48-frame RealEstate10K-conditioned clips, but those clips did not share a camera path and SVD moved least. It measures feature agreement after a learned reconstruction, not image quality and not (in this paper) agreement with human raters; every test in the paper is a static scene.
• articleHow does CoSER get LLMs to write multi-character dialogue and judge it, and how could an agent add the missing speaker-blind human or model check for character voice?
  CoSER's given-circumstance acting (GCA) protocol has one LLM instance per character write speech, action and private thought, an NSP model picks speakers, an environment model replies, and a penalty-based critic lists flaws against the original conversation. Its Character Fidelity rubric covers language style but is judged by an LLM and not separately checked against humans in the text read. A speaker-blind attribution test on GCA outputs is not part of the paper; it is offered below as an untested proposal.
• articleWas CoSER's automatic LLM judge validated against its human raters, with what statistic and numbers, and how far can an agent trust that validation for judging character dialogue?
  Yes, partly. Table 5 reports 'alignment' = percentage agreement with the human raters on which of two models is better, ties dropped. GPT-4o as judge gives 68.6, DeepSeek-V3 65.1 and DeepSeek-R1 77.5; BLEU gets 75.3 and ROUGE-L 72.0, so only the R1 judge beats the n-gram metrics. No correlation coefficient, confidence interval or inter-annotator agreement is given. The paper's main results use the lower-aligned GPT-4o judge. The check is at the whole-score level with 7 models, not per rubric dimension such as Character Fidelity.
• articleDid people rate the AI-written character dialogue in CoSER, what exactly did they score, and does that overturn the earlier finding that no human-rater study of LLM-generated lines attributed to characters exists?
  Yes, with limits. CoSER v3 Section 5.3 reports three annotators who each scored 20 of 60 CoSER Test samples, rating the simulated conversations of 7 LLMs on a 1 to 10 scale while seeing the background context and the original conversations. The text read names no criterion beyond 'GCA simulation' quality (role-playing ability overall); it is not stated as character fidelity, distinguishability or voice, and no who-said-it task appears in the text read. CoSER-70B scored highest (6.783, win rate 86.9%). The earlier scoped negative is confirmed for who-said-it attribution and voice distinctiveness, and narrowed (not overturned) for human rating of LLM-generated character dialogue in general.
• articleWhich documented techniques keep an LLM from giving a character information that character should not have, and what effect did each show?
  Six techniques have measured effects, each in a different setting, so they are not comparable head-to-head. (1) Two-pass perspective filtering (SimToM): GPT-3.5-Turbo ToMi false-belief 81.0 (+13.8 over 0-shot); single-prompt version loses the gain; human-annotated perspectives reach 96/96/82/98; mixed on OpenToM (candidate 04). (2) Per-character belief graphs (SymbolicToM): GPT-3-Davinci on self-built structure sets 20/25/39 to 92/91/90; needs chronological stories and suffers from extraction errors. (3) Visibility-tagged fact memory with scene anchoring (ReverieMem): refusal 81.2 versus 8.7 to 11.3 for retrieve-everything RAG; ablations show both layers matter; one speaker only. (4) Separate agent contexts versus an omniscient script (Zhou et al.): fewer early leaks (0.39 vs 0.13) but far fewer deals (30 vs 94 percent). (5) Awareness reminder (SimpleToM): behaviour prediction above 80 for three models, judgment low. (6) Key Facts Injection: 73 percent fewer 'known-information forge...
• articleHow have unearned character knowledge or leaked information been detected and rated in LLM-generated stories and simulated conversations?
  Two partial measurements exist in the sources inspected; no source measured character-to-character secret leakage in generated fiction with human raters (bounded negative). (1) ConStory-Checker (arXiv 2603.05890v1) is a four-stage LLM judge with a 'Knowledge Contradictions' sub-type (characters showing knowledge or skills beyond their background, including information they could not have accessed given their background), with quoted evidence and positions; validated on 1,000 planted errors: precision 0.884, recall 0.550, F1 0.678 against human experts at F1 0.281. No sub-type rate is reported in the text read. (2) Zhou et al. (arXiv 2403.05020v4) use the early position of a name in a find-the-mutual-friend dialogue as a leakage proxy: 0.13 for an omniscient single-LLM script versus 0.39 with separate agents (450 simulations per model and mode).
• articleIf an LLM correctly says a character does not know a secret, will it then predict that the character acts as if they do not know?
  SimpleToM (arXiv 2410.13648v2) uses two-sentence everyday stories (a hidden fact; a person acting without knowing it) and asks three questions: is X aware (mental state), what will X do (behaviour), was X's action reasonable (judgment). Models were near ceiling on awareness (GPT-4o 95.6, Claude-3.5-Sonnet 97.9) but far lower on behaviour (49.5, 67.0) and judgment (15.3, 24.9) where chance is 50; o1-preview reached 84.1 on behaviour. The failure in the authors' example is predicting the observer acts on the hidden fact. Reminders of the awareness answer lifted behaviour above 80 for three models but judgment stayed low.
• articleHow deep can nested beliefs ('A thinks B thinks C knows') go before LLM answers fail, and what does deception do to accuracy?
  Hi-ToM (arXiv 2310.16755v1, 2023) generates stories with five agents, rooms, containers and an object, plus optional public or private (deceptive) statements, and asks five questions per story at orders 0 to 4. Four 2023 LLMs scored below 60 percent overall; joint accuracy (correct only if all lower orders are also right) fell from near perfect at order 0 to near zero at order 4; for GPT-4, going from 0 to 4 deceptive sentences per story cost 11 to 32 points of joint accuracy (orders 4 to 1), and chain-of-thought gave no substantial gain. SymbolicToM's authors report a third-order test (their own small set, D2) where per-character belief graphs helped. The Hi-ToM total story count is not stated in the text read.
• articleWhen an LLM reads a conversation where a character was absent for part of it, how often does it credit that character with the missed information, and how consistent are its answers across ways of asking?
  FANToM (Kim et al., arXiv 2310.15421v3) builds 256 multi-party conversations in which a character leaves and rejoins, then asks the same underlying question six ways (free-form belief, multiple-choice belief, answerability list and yes/no, information-access list and yes/no). The model reads everything, so an answer containing what the absent character missed is the failure. In the 2023 models tested, success on one format did not carry to the others: the best chain-of-thought GPT-4 run scored 26.6 on 'all question types' against 87.5 for humans, and the dominant list error was including characters who were unaware. Results are for 2023 models.
• articleWhich story-based theory-of-mind benchmarks test which character knows or believes what, what does each actually ask, and how large is each?
  Five public benchmarks are inspected. ToMi: Sally-Anne object-location stories (999 narratives per OpenToM's table; paper body not read). Hi-ToM: generated stories with five agents and one question each at orders 0 to 4, optional public or private deceptive statements. OpenToM: 696 narratives with 23 first- and second-order questions each (location, multi-hop container/accessibility, attitude). FANToM: 256 conversations in which a character leaves and rejoins; belief, answerability and information-access questions. SimpleToM: two-sentence everyday stories, 1147 per category, with awareness, behaviour and judgment questions. All are template-built or LLM-generated; none uses published fiction.
• articleI want evidence that an AI-written character voice is convincing: what do these three papers actually test with humans, and what is the simplest check I can do first?
  Of the three, only ECHO has people judging AI-generated text, and it asks a different question: which of two answers came from the real (living, ordinary) person, not which character said a line. Deezer's paper has no human raters and no generated text. CoSER may have a human scoring study, but that is unverified here. So a newcomer should not expect any of them to say that readers can tell characters apart in AI-written dialogue. The simplest first check in ECHO's style is a blind pair test on your own cast: one human-written and one AI-written line for the same situation, pick the human one, with 50% as chance; for who-said-it checks use the protocol in S52-08.
• articleDoes the Deezer Research paper "Distinguishing Fictional Voices" include any human evaluation of LLM-generated dialogue, and what does it actually measure about character voice?
  No. The paper (arXiv 2401.16968v1, 30 Jan 2024, Deezer Research and Loria) is fully automatic and uses only human-written novel text. It encodes each character's quotes with pretrained authorship-verification models and tests whether characters can be told apart (character vs character) and whether unseen quotes can be attributed to characters. Chapterwise: LUAR reached AUC 81.6 for character-vs-character but 53.6 for quote attribution, only slightly above chance. The authors guess the attribution task is hard "even for humans" but ran no human test. S52's negative is confirmed for this paper.
• articleDid the ECHO paper have people rate LLM-generated role-play answers against a real person, how many raters and items, what were the results, and were its LLM judges checked against the humans?
  ECHO (arXiv 2404.13957v1, 22 Apr 2024) does include a human evaluation of LLM-generated text: acquaintances of ten real volunteers (at least seven each) saw ten pairs of answers, one from the real person and one from a role-playing LLM, and picked the human one. The headline metric is how often the LLM answer was chosen (50% is chance). Overall rates ran 37.5 to 48.2 across methods, with GPTs closest to chance. The targets are ordinary living people, not fictional characters, and the task is human-versus-machine, not which character said a line, so S52's negative narrows rather than falls. No agreement statistic was found, and LLM judges were scored against ground truth, not validated against the human raters; they picked the LLM answer as the human one above 90% for the two GPT-4 judges.
• articleDo LLM-played characters keep their voice over a long dialogue, how is drift measured, and what do the authors say it converges to?
  Three open-access papers measure drift without human raters in the main loop. Li et al. (arXiv 2402.10962v4) report instruction drift within eight rounds of two-chatbot self-chat using code-based probes. Luz de Araujo et al. (arXiv 2512.12775v2, EACL 2026 per abstract) run over 100-round dialogues with eight RoleBench characters across seven models and report that fidelity (knowledge, style, in-character consistency) falls and persona behaviour reverts toward a no-persona baseline. Abdulhai et al. (arXiv 2511.00222v1) show that a line-to-line consistency metric can rise while prompt consistency falls. The judge validations are small or single-rater.
• articleHow was PersonaGym's PersonaScore validated against human judgement, and what does its Linguistic Habits task say about voice?
  PersonaGym (arXiv 2407.18416v5, 5 Sep 2025) scores persona agents on five tasks with an ensemble of two LLM judges using 1-5 rubrics. A human study of five experts rated 1,500 responses (100 personas, three models) and reports Fleiss kappa 0.71 among annotators and average Spearman 75.1% and Kendall 62.73% between PersonaScore and humans. Linguistic Habits was the hardest task. Personas are described people (for example professions), not fictional characters, and the text and the correlation table disagree on one Kendall claim.
• articleCan an LLM judge reliably say which character is speaking, and what does that imply for using an LLM to check character voice?
  PersonaEval (arXiv 2508.10014v1, 6 Aug 2025) gives LLM evaluators dialogue context plus four or five described candidate roles and asks who the second speaker is. The paper reports the best LLM at around 69% top-1 accuracy against 90.8% for 20 human volunteers on a hard sample, and says models lean on speaking style while humans use addressing and intent. Items are human-authored (English novels, Chinese scripts, English explainer videos), so the result says how judges treat existing text, not how they treat generated dialogue.
• articleHow well can people actually tell which character said a line, and what does that mean for checking my own dialogue?
  Two human studies on human-authored dialogue are in the sources. In Speaker Verification, ten non-expert annotators judging whether two sets of utterances came from the same speaker scored 56.50% accuracy on the base level when shown only utterances (67.48% when shown conversations); in PersonaEval, 20 educated volunteers scored 90.8% when choosing among named candidate roles with profiles and context, on cases chosen because a strong model got them wrong. The gap suggests context cues (who addresses whom, plot) carry most of the human signal. No human study of attributing LLM-generated lines to characters was found.
• articleWhich English-language role-play or persona benchmarks exist, and what does each one actually score (is character-to-character voice distinctiveness among the scored things)?
  Four benchmark-style resources were read. RoleBench scores reference overlap (ROUGE-L) and judge/human win rates for single-turn answers in a role; PersonaGym scores five rubric tasks per demographic persona with an LLM-judge ensemble; PersonaEval scores whether a judge can name the speaker in human-written dialogue; the Speaker Verification work is the only one read that defines a cross-character measure (Distinction Score) on generated lines. CharacterEval is Chinese-only on its abstract page. None of the four reports a human-rated test of whether generated characters sound different from each other.
• articleIf I track inventory, location, character knowledge and consequences across many turns of an interactive story, which failure modes have these two papers measured and which are only proposals?
  Documented use (measured in the papers, on non-story environments): predicted-state quality drops with horizon when predictions are fed back (S2), a model that predicts from raw history can claim an object is unavailable or return the previous room (S2 Appendix G examples, selected), a learned simulator can let weak agents "win" it could not win for real (S1 Table 5), and one world model lost room connectivity in a looping TextWorld run (S1 Appendix H, one trajectory). Not measured: character knowledge or belief differences between characters, NPC dialogue, long narrative sessions (the longest horizon is 20 actions in Jericho; trajectories average 28.6-52 steps), or any per-category error rate for inventory. Everything about running such a tracker for authored fiction is a hypothesis.
• articleWhat exactly did arXiv 2610.06637 (v1) define and measure for long-horizon textual world modelling: environments, horizons, models, metrics, headline numbers, ablations, limits, release?
  arXiv 2610.06637 (Wang, Gao, Yao, Dong, Walia, Das; UIC and Intuit AI Research; v1 submitted 2026-10-05 16:25:44 UTC; CC BY 4.0) fine-tunes Qwen3-4B-Instruct-2507 world models that keep a JSON "textual state" and predict a whole k-action sequence in one call as sparse set/remove edits that code replays (latent-SR). It evaluates horizons 1-10 (ScienceWorld, CEO-Bench) and 1-20 (Jericho) with a fixed-target protocol against raw-history recursive/direct baselines and GPT-5.2. latent-SR has the best averages on every metric (ScienceWorld Fact-F1 92.44 vs 88.94 for obs-direct; CEO-Bench SE-score 87.35 vs 81.85; Jericho BLEU 24.5 and Score MAE 1.728), with gains growing at long horizons, but obs-recursive is better on Jericho at h1 and h5. The abstract's "nearly 30%" Jericho error drop holds against obs-direct at h10 (28.9%) but not against obs-recursive (12.2%), which is the lower-error raw-history baseline there. No code, data or weights release is stated.
• articleCamI2V's README ranks 'MotionCtrl' and 'CameraCtrl' below CamI2V: are those the released MotionCtrl and CameraCtrl checkpoints, and what protocol do the numbers come from?
  The CamI2V README's comparison table is the CamI2V authors' own measurement (README-reported, not independently reproduced) at 256x256 and 50k steps, in which 'MotionCtrl' and 'CameraCtrl' are their re-implementations on DynamiCrafter, labelled 'reproduced non-official'. These are different artefacts from the official CameraCtrl (AnimateDiffV3, 256x384) and MotionCtrl (VideoCrafter/SVD) checkpoints, so the ranking does not rank the official releases. For the official CameraCtrl paper-style numbers on RealEstate10K with SVD and AnimateDiff builds, see the earlier batch candidate C3-B062-W13-03, which documents that RotErr/TransErr values from different papers are not comparable across estimators and normalisations. An agent choosing a checkpoint for camera accuracy should not use the README table as evidence about the official CameraCtrl or MotionCtrl releases.
• articleDoes DDEX's own Knowledge Base or implementation guidance say anything about populating ContainsAI or AiContribution beyond the schema and Data Dictionary, and where do DDEX's own documents disagree or stay silent?
  Scoped negative: in the Knowledge Base navigation (about 400 article titles, page last-modified 2026-10-06), in 14 targeted KB articles and in the ERN Part 1 sections read, no DDEX implementation guidance on populating ContainsAI or AiContribution was found; the AI fields appear only in the release-notes annex, the Data Dictionary and the value set. Within those, the Data Dictionary describes ContainsAI as a true/false Flag while the value set is three-valued (All / Partly / None); the Release-level row states no consistency rule; and DDEX's "unknown" convention (#unknown#) is for mandatory elements, so it does not supply the "unknown" value that YouTube's CSV accepts. These are observations of silence and wording, not defects proven in the standard.
• articleWhat does Deezer tell distributors and other business partners about using its AI-detection output (what is detected, who decides the consequences, what uploader declarations count), and is a supplier-facing AI delivery policy published?
  Deezer for Business's AI Detection FAQ (system dates created 2026-05-12, updated 2026-05-21) is addressed to distributors, rights holders, collecting societies and platforms who might license the detector. It says the tool classifies and tags, that consequences (exclusion, demonetising, labels, royalty treatment) are each partner's decision under their own policies, that detection reads audio patterns rather than metadata or uploader declarations, that it focuses on fully AI-generated tracks with hybrid detection "in active development", and gives company figures (39% of deliveries, up to 85% of that fraudulent, false-positive rate below 0.01%). No supplier-facing delivery policy on AI was found in the 16 business help-centre articles; so a distributor's own AI rule, not a Deezer supplier rule, is what an uploader can read.
• articleCan a creator remove or change Meta's AI label after posting, and which Meta pages could not be read, so an agent knows what stays unverified?
  Negative result with scope: in the three Wayback captures of the Instagram Help Center article "Label AI content on Threads" (2026-02-15, 2026-06-10, 2026-09-04) there is no sentence about removing, editing, appealing or turning off a label, and the only statement about a creator who does not label is that a label is applied if Meta detects AI generation. The Facebook and Instagram sibling articles were not located, the live Help Center pages and the ads article are JavaScript shells for curl, and the Wayback capture of the ads article carries no body text. An agent should therefore record who applied the label (creator or system) and mark removal as unverified rather than promise it.
• articleAcross later driving video/world-model papers with nuScenes generation tables, which FID/FVD values for DriveDreamer and DriveGAN does each table print, and which arXiv version of each paper was read?
  Of seven later tables that print DriveDreamer FID/FVD, five use the arXiv v1 pair 52.6/452.0 (DrivingDiffusion v1, Drive-WM v1, Panacea v1 with columns swapped, GenAD v1 and v2 with FVD rounded to 452, UniMLVG v1-v3), one prints the v2 pair 14.9/340.8 alongside another pair (DriveDreamer-2 v1 and v2), and two (DriveDreamer-2 v1/v2 and Delphi v1/v3) print 26.8/353.2, a pair found in neither DriveDreamer v1 nor v2 HTML text. DriveGAN is printed only in Drive-WM, GenAD and UniMLVG, always at the v1 pair 73.4/502.3 (GenAD 502); none of the tables read uses the v2 pair 27.8/390.8. MagicDrive-V2/MagicDriveDiT and DiVE print no DriveDreamer FID/FVD (MagicDrive-V2 lists only DriveDreamer's resolution and frame count). DrivingDiffusion v1 (2023-10-11) predates DriveDreamer v2 (2023-11-27); Panacea v1 (11-28) and Drive-WM v1 (11-29) were posted one and two days after v2 yet print the v1 pair; the papers do not say whether v2 was seen.
• articleBefore I compare nuScenes FID/FVD scores of two driving world-model papers, what exactly should I check, and which concrete mismatches have already happened with DriveDreamer baselines?
  Six checks, each with a documented case in the papers read: (1) pin the arXiv version of the citing and the cited paper, because DriveDreamer v1 and v2 print different FID/FVD for the same method; (2) read the row condition, not only the method name, because DriveDreamer-2 quotes three different DriveDreamer pairs in one table; (3) check the metric protocol (resolution, clips, frames, FVD feature extractor), which differs across papers; (4) check whether a baseline row is quoted or recomputed (Delphi marks recomputed rows with a star and gets MagicDrive FID 46.18 against the quoted 16.20); (5) check column order and rounding (Panacea lists FVD before FID; 16.96 appears as 16.9); (6) check whether the citing paper's own numbers changed between versions (UniMLVG 8.8/60.1 in v1-v2, 5.8/36.1 in v3). This is a reading checklist for reported numbers; no new scoring was done.
• articleWhen an AI dialogue-enhancement tool sounds clean, how can a dialogue editor tell whether it has changed the words, and what do recent papers say about objective metrics and listening tests missing such changes?
  Three recent arXiv papers say generative enhancers can substitute or omit phonemes while scoring well on reference-free metrics: the URGENT 2025 P.808 study reports a generative system with top DNSMOS/NISQA and high MOS but very poor phone fidelity and says ACR tests may also miss it; Yamauchi et al. say non-intrusive filtering often fails to detect such errors; Shetu et al. find limited hallucination by WER/LPS at moderate SNR but more at very low SNR. Their remedy is a content-based metric (WER, ESTOI, LPS) alongside MOS. No source tests DeepFilterNet or Resemble Enhance for this, so applying it to production dialogue tools is a proposal.
• articleWhich dimensions of VBench-2.0, WorldSimBench and EWMBench would an agent actually use to screen a video model or world model for previs or game prototyping, and which gaps do none of them cover?
  Synthesis: for previs clips, the usable pieces are VBench-2.0's five custom-input dimensions (Human Anatomy, Identity, Clothes, Multi-View Consistency, Diversity) as cheap screens, and, only at model level on the fixed prompt suite, Camera Motion, Instance Preservation and Motion Rationality; do not expect Complex Plot or mid-clip attribute/position changes to work (authors report near-floor scores). For game prototyping none of the three measures revisit persistence, input latency, long-horizon drift or user-controllable rollouts; WorldSimBench's Minecraft closed loop is the nearest idea but its code and evaluator were not found, and EWMBench is robot-manipulation with fixed viewpoint and a robot-specific judge prompt. This is a proposed screening checklist, not a tested protocol.
• articleHow does EWMBench score models, which systems lead and by how much, and what bias, scale or protocol risks do the paper and repo leave open?
  EWMBench reports a domain-adapted lead (EnerVerse_FT 4.70, LTX_FT 4.55) over Kling (3.87), Hailuo (3.41) and open models (2.97-3.29) on seven models. The 'Avg.' and 'Overall' columns are plain sums (I verified to 4 decimals), so the 'overall' mixes components of very different range: the semantic Diversity term spans 0.06 while Dyn spans 0.57. Semantic scores barely separate EnerVerse_FT (2.09), Kling (2.04) and Hailuo (2.02). The human study ranks four models with no annotator count or correlation statistic. Motion best-of-three is selected using the ground-truth trajectory, and the judge prompt is robot-specific (declares human hands abnormal). Weights and data are CC BY-NC-SA 4.0 per cards and README, but the repo has no LICENSE file.
• articleWhat is EWMBench, what exactly does it measure (scene, motion, semantics), what inputs does it need, and how do I run it?
  EWMBench (AgiBot, arXiv v2) scores image-text-to-video models on robot manipulation: given an initial scene image, an instruction and optionally a 6D trajectory, the model generates frames that are compared with ground-truth AgiBot-World episodes. It measures scene consistency (fine-tuned DINOv2 patch similarity), end-effector motion (Hausdorff, normalised DTW and a velocity/acceleration Wasserstein term from a fine-tuned YOLO-World detector), and semantics (a Qwen2.5-VL caption compared by BLEU and CLIP score, a diversity term, and a logic check). It is robot-specific, fixed-viewpoint and needs a ground-truth trajectory per case.
• articleHow are WorldSimBench scores produced and validated, which systems lead and by how much, and which of the paper's own claims do its tables fail to support?
  WorldSimBench validates its learned evaluator against GPT-4o with PLCC (0.60 AD, 0.43 RM in-distribution; zero-shot 0.34-0.49 AD, 0.44-0.47 RM) and accuracy for OE, but gives no evaluation-set sizes, annotator counts or agreement. The explicit and implicit modes disagree: by my computation OE rank agreement across seven models is Spearman 0.04 (0.21 on the five text-conditioned models). The authors admit inflated scores in robot manipulation and Velocity, and several statements in the text do not match the paper's own tables (GPT-4o negative-correlation attribution, 'AD variation is minimal', Open-Sora-Plan's trajectory lead). All results are for eight older open models fine-tuned on scenario data, so 'who leads' says nothing about current commercial systems.
• articleWhat is WorldSimBench: how does it define a 'world simulator', what are its two evaluation modes and three scenarios, and can I get its code, data or evaluator?
  WorldSimBench (arXiv v1, Oct 2024) sorts predictive models into stages S0-S3 by output modality and calls S3 'actionable video' models World Simulators. It evaluates them two ways: Explicit Perceptual Evaluation (a LoRA-tuned video LLM scores generated clips on scenario-specific dimensions, trained on a 35,701-video human-feedback set) and Implicit Manipulative Evaluation (generated video is converted to actions by video-to-action models and run closed-loop in Minecraft/MineRL, CARLA and CALVIN). It evaluated eight fine-tuned open video generators; no code, dataset or evaluator release could be located (homepage buttons have no links; GitHub and Hugging Face name searches found nothing).
• articleHow good is the evidence that VBench-2.0 scores track human judgment, which systems does it say lead and by how much, and what judge-bias, saturation or protocol risks do the authors and repo admit or leave?
  The human-alignment evidence is a model-level comparison over only four systems per dimension. I recomputed the table's 'Correlation' row as Pearson r over those four points (matches to within rounding), while the figure caption says Spearman; with n=4 rank agreement is far weaker on some dimensions (0.4 for Human Anatomy and Human Interaction, average ranks, no significance test). Several dimensions have tiny score spreads between systems, and the authors admit VLM judges are unreliable for Human Anatomy and Motion Rationality. Between arXiv v1 and v2 the Instance Preservation method and values changed for the same four systems (about 71-76% to 82-94%). Leader statements: per Table II no system leads everywhere; there is no paper aggregate, and leaderboard Total Scores were not retrievable.
• articleWhich Nano Banana (Gemini image) model should you use for character consistency and references, and what are the limits?
  Gemini API docs (captured 2026-10-23) list five Nano Banana models: Nano Banana 2.1 (gemini-nano-banana-2.1; recommended workhorse: Flash-level speed/cost with improved quality, text rendering, multi-turn consistency, Google Search grounding, 1K/2K/4K); Nano Banana 2 Lite (gemini-3.1-flash-lite-image; fastest/cheapest, "not optimized for multiple reference inputs or multi-turn sequential editing"); Nano Banana 2 (gemini-3.1-flash-image; previous workhorse, 4K, multi-reference consistency; new projects should use 2.1); Nano Banana Pro (gemini-3-pro-image; premium for complex tasks, localization, brand consistency, precise control); Nano Banana (gemini-2.5-flash-image; legacy, migrate to 2 Lite). All outputs carry SynthID. Up to 14 reference images can be mixed; the per-model table lists object-fidelity references of up to 14 / 10 / 6, character-consistency references of up to 4 or 5 (N/A for Lite), and up to 3 style references (one model only) - the flattened text does not reliably t...
• articleWhat should an agent check before placing AI-generated music in a film, game or ad?
  Synthesis from the five service/model reads: (1) which text and date you relied on; (2) plan tier at generation; (3) permitted-download or export status; (4) medium restrictions (film/TV/radio/games); (5) attribution; (6) provenance marks left intact and any platform AI-disclosure rules; (7) prompts avoided artist/song/lyric names; (8) human contribution documented for any copyright claim; (9) component licences for open models; (10) a human owner of the final clearance. The US Copyright Office report (Jan 2025) says prompts alone do not make users authors; the services disclaim or limit copyright vesting. This is not legal advice.
• articleWhat does WorldModelBench measure (instruction following, five physics laws, commonsense), which model versions were scored, how reliable are its human labels and VLM judge, and what limits do the authors state?
  WorldModelBench (NVIDIA, Berkeley, UCSD, MIT; Feb 2025) has 350 image-plus-text conditions in 7 application domains and 56 subdomains, graded 0-10 by humans on instruction following (0-3), five binary physics laws (Newton's first law, mass conservation/solid mechanics, fluid mechanics, impenetrability, gravitation) and two commonsense quality checks. 14 model configurations were labelled with 67K human labels; the best, kling-v1.5, scored 8.82, and the authors report that VBench correlates only 0.28 with physics adherence versus 0.69 with frame quality. A fine-tuned 2B judge predicts totals within about 4.1% average error. The authors acknowledge the small prompt set (350) and that answers are withheld, so scoring a new model needs their judge.
• articleWhat do the open-weight licences of LTX-2.x, MiniMax H3, Apple SHARP, Meta SAM and Fish Audio S2 actually allow for commercial use and where?
  Licence texts read 2026-10-23 (not legal advice): LTX-2.x Community License (2026-08-11) grants broad rights subject to use restrictions (Attachment A), but entities with annual revenues of at least $10,000,000 must obtain a paid commercial licence for any non-non-commercial use; unlicensed commercial use by such entities is prohibited and fees may be charged retroactively. MiniMax H3 Community License (2026-08-02) is territorially limited: the "Applicable Territory" is worldwide EXCLUDING the European Union, the United Kingdom, the Republic of Korea and the United States, so the open-weights grant does not cover users there (MiniMax invites licence requests); also prior written authorization if commercial products/services exceed US$20M yearly revenue, and "MiniMax H3" must be displayed in commercial UIs. Apple SHARP model licence: Research Purposes only (non-commercial scientific research), derivatives likewise; excludes product development or commercial services. Meta SAM License...
• articleCan AI-generated images, video or music be copyrighted in the US, and does prompting count as authorship?
  U.S. Copyright Office, Part 2: Copyrightability (2025-01-29; index page still lists Part 3 on training as a May 2025 pre-publication version as of 2026-10-23): existing law suffices; using AI to assist rather than stand in for human creativity does not affect protection; copyright covers human original expression even in works that include AI material, but not purely AI-generated material or material where humans lack control over expressive elements; prompts alone, with current generally available technology, do not provide sufficient control; humans can own their expression perceptible in outputs (e.g. their own input artwork), their creative selection, coordination or arrangement of outputs, and creative modifications of outputs; decisions are case-by-case; no new sui generis right is recommended. Practical: document human contributions (input artwork, edits, compositing, sequencing) and disclaim AI-only portions when registering; this aligns with studio practice requiring human ...
• articleWho owns 3D models generated with Meshy and Hyper3D Rodin, what do they cost, and what are the retention/training terms?
  Meshy Terms (last updated 2026-09-19): on the free plan Meshy owns all rights in outputs and makes them available under CC BY 4.0 with credit to Meshy; on paid plans customers own their outputs (to the extent law allows) and may keep content private; outputs may carry AI-identifying metadata/watermarks that users must not remove; non-Enterprise inputs/outputs may be used to train or improve services; API outputs (non-Enterprise) are deleted after 3 days, so download promptly. Meshy API pricing is in prepaid credits per request (credit-to-USD rate not on that page): e.g. Image/Text-to-3D preview 20 credits on meshy-7.1/meshy-6 (5 on meshy-6-lite or the smart-topology t2/t2.5 models), mesh with 2K-4K textures 30, 8K 35; refine/retexture 10; auto-rigging 5; animation 3 per action; remesh 5; smart-topology polycount 100-15,000 (t2) or 100-30,000 (t2.5). Hyper3D (Rodin) pricing page (rendered 2026-10-23): Free $0 plan pays per confirmed result at $1.5 per direct credit, with export from ...
• articleWhat does the OMS conformance suite actually test about file enumeration, ignores and sharding, and where do its fixtures disagree with the spec?
  The OpenSSF Model Signing (OMS) conformance suite has 79 tests (31 roundtrip, 48 verify). At the same commit it contradicts itself: a roundtrip comment says spec section 6.2 lists only .git, .gitignore and .gitattributes although the spec also lists .github, and two fixtures treat a nested subdir/.gitignore oppositely (signed in with-nested-gitignore, ignored in multi-gitignore), so an implementation cannot pass both unless the harness distinguishes them; spec 6.2.1 (top-level-only matching, per the producer's reading) favours 'signed'. Checked again on 2026-10-22: the newest commit is still 91bf841e (2026-09-18), and no commit, pull request or issue addresses either contradiction; merged PR #15 notes that nested paths and combined git-exclusion structures requested in the PR #12 review were deferred.
• articleHow should an agent record the licence of a third-party asset it uses, separately for code, model weights, dataset and media, and which machine-readable fields exist?
  SPDX 3.0.1 distinguishes declared from concluded licence relationships. REUSE 3.3 recommends comment headers with SPDX-FileCopyrightText and SPDX-License-Identifier; uncommentable files must carry the header in an adjacent file with the extra extension .license; a REUSE.toml may sit in any directory, and REUSE.toml and the deprecated DEP5 file must not be used together. Creative Commons recommends against using CC licences for software or hardware. IPTC Photo Metadata keeps separate fields for Copyright Notice, Rights Usage Terms (xmpRights:UsageTerms), Licensor and Web Statement of Rights (xmpRights:WebStatement), and the ccREL submission marks a licence with rel="license" plus cc:attributionName/cc:attributionURL. Recording each third-party asset's licence per component, with 'not stated' where the source is silent, is a proposed practice (labelled synthesis), and 'declared' means different things in SPDX and CycloneDX.
• articlePer app (Premiere, Illustrator, Firefly, Photoshop, Lightroom, Camera Raw), what does each Adobe page say triggers automatic Content Credentials versus optional ones?
  Adobe's own pages describe automatic Content Credentials differently per app. Firefly applies C2PA metadata automatically when content, or a project including a Firefly-generated asset, is downloaded or exported. Illustrator's FAQ says version 29.3 and earlier adds Content Credentials to all new generative AI content, while its 29.4-and-later sentence says new generative content gets none when the file has no generative content, which reads as self-contradictory; the same sentence appears unchanged in the live FAQ ('Last updated on Jun 1, 2026') and in a Wayback capture of 2025-10-06, and Adobe defines no meaning for 'contain generative AI content'. Photoshop supports Content Credentials only on JPG and PNG export. Check the specific app version and export format rather than assume uniform behaviour. Firefly and Photoshop statements were verified only against producer copies of the pages (the live pages returned 403 to the curator); no export was run.
• articleDoes the WGA Screen Credits Manual mention AI-generated material, and how does the MBA's AI clause relate to source-material credit?
  The WGA Screen Credits Manual (effective for notices after 31 December 2021) contains no text on AI. The 2023 MBA Memorandum of Agreement, Article 72.B, says material produced by traditional AI or generative AI (GAI) is not literary material. Article 72.C.3 says that when a Company furnishes a writer with unpublished, unexploited GAI-produced material and instructs its use as a basis, that material is not source material for determining writing credit; the closing paragraphs of 72.C extend this to a writer who uses GAI with the Company's consent while preparing literary material, and such writer-prepared material is considered literary material rather than material produced by GAI. The copy read is the MOA, not the integrated MBA text. PDF text only; not legal advice.
• articleHow do SPDX 3.0.1 AI/Dataset profiles and CycloneDX 1.6/1.7 ML-BOM represent separate licences for code, weights and training data, field by field?
  SPDX 3.0.1 defines hasDeclaredLicense as the licence an artifact was discovered to contain (for example by automated tooling), which differs from the conventional author-declared reading, and it attaches licences through relationships rather than a package property. CycloneDX 1.6 and 1.7 (schemas at 349314a9) distinguish license.acknowledgement 'declared' (the authors' initial intentions) from 'concluded' (verified and confirmed), with observed licences recorded in evidence.licenses. In CycloneDX, the componentData (dataset) object and the modelCard object have no licence property; licences attach to the parent component. So in both formats a dataset's or model's licence is attached to the component (CycloneDX) or by relationship (SPDX), not inside the data or model-card description, and 'declared' means different things in the two specifications. Specification text only; no SBOM was generated or validated.
• articleWhat do the texts of OpenRAIL-M, Llama 3.3/4 community, Gemma and OpenMDW licences say about use restrictions, derivatives and outputs?
  For CreativeML OpenRAIL-M (2022-08-22 text), the licensor claims no rights in Output, but the same clause says no use of the output may contravene the licence, so the use-based restrictions still reach output use; providing a hosted service counts as Distribution. The Llama 4 and Llama 3.3 licences (llama-models 0e0b8c51, re-read 2026-10-22) require adherence to a per-version Acceptable Use Policy and, if Llama materials or outputs are used to create, train or improve an AI model that is distributed, require the model's name to begin with 'Llama'; neither text contains a 'no rights in output' statement. OpenMDW-1.1 imposes no restrictions or obligations on outputs; its conditions are keeping the agreement and notices, plus termination on patent or copyright litigation. The Gemma Terms of Use (live page, last modified 1 April 2026) incorporate the Gemma Prohibited Use Policy, state that Google claims no rights in Outputs, and define Model Derivatives to include models trained on Gemm...
• articleWhat do the Comfy Registry docs claim about scanning, verification and versions, and what can be confirmed in Manager and core code?
  Comfy Registry docs (Comfy-Org/docs at 506a4a94) say all nodes will be scanned for malicious behaviour such as custom pip wheels and arbitrary system calls, and that nodes passing these checks get a verification flag in the Manager UI; the standards page prohibits eval/exec, runtime package installation through subprocess calls, and obfuscation. The pages give no scanning method, schedule or results. In ComfyUI-Manager code (a74929c6), Registry versions carry statuses such as Active, Pending and Flagged. The server refuses Flagged versions unless allow_flagged_nodepack_install is set or every --listen address is loopback; because ComfyUI listens on 127.0.0.1 by default, a default local install allows Flagged versions. The cm-cli install path also allows them unless the environment variable _COMFYUI_MANAGER_CNR_ALLOW_FLAGGED is set otherwise. ComfyUI core has a separate signed-policy mechanism that is off by default (GOVERNANCE_REQUIRED = False). Code read; nothing was executed.
• articleWhat do first-party and first-person sources document about the 2024 ComfyUI_LLMVISION incident and later supply-chain incidents, and how did Manager respond?
  A user's first-person report (post dated 2024-06-09, Wayback snapshot of 2024-06-10) says the ComfyUI_LLMVISION custom node shipped custom openai/anthropic wheels in requirements.txt that contained browser-data theft code. ComfyUI-Manager's security_check.py (earliest commit in the path history: 2024-06-09) detects that node and some pip packages, terminates ComfyUI and prints cleanup steps; the same file later also lists ultralytics 8.3.41/8.3.42 and litellm 1.82.7/1.82.8. Comfy Org's own post on the ultralytics miner (2024-12-05) says Windows was not affected and that Manager was updated to warn users and to pin ultralytics to 8.3.40. A Manager commit on the same day (5a61000d) adds that pin when a pip_overrides.json file exists; the pin is absent at the later commit a74929c6, and its removal date is unknown. No Comfy-authored post about LLMVISION was found.
• articleWhat do Netflix's public partner pages say about source quality, upscaled material and declaring AI processing?
  Only one Netflix page was readable as raw bytes: the Preferred Fulfillment Partner (NPFP) program conditions (Last-Modified 2026-10-01). It says Netflix will not accept SD/HD if UHD/4K is available and that insufficient source is rejected and can be repaired or re-sourced; it requires an assistive QC tool plus manual QC. It does not mention upscaling or AI in the text searched. The Partner Help Center delivery-specification and generative-AI pages are JavaScript-only or API-401 to curl, so their text was not read; search-engine snippets about them are secondary leads and not used as evidence.
• articleCan I compare reported DeSync and IB-score numbers across Kling-Foley, HunyuanVideo-Foley, MMAudio, V-AURA and world-model audio papers, and what do real recordings score?
  Only partly. The same system gets different numbers in different papers: V-AURA is 155 ms in its own paper (2.56 s clips), 0.654 s in MMAudio (8 s, two 4.8 s crops), 0.72 s and 0.86 s in HunyuanVideo-Foley's tables, and 1.287 s in WorldSonus (5 s clips). Kling-Foley reuses MMAudio's baseline rows but reports an MMAudio row (IB 29.26, DeSync 0.45) that matches none of MMAudio's four published rows, and its abstract claims best distribution matching while its own Table 2 shows MMAudio ahead on FD and KL. HunyuanVideo-Foley's own VGGSound-Test table has MMAudio at lower DeSync (0.47) than itself (0.53). The only ground-truth reference found is Foley-Omni's V2ST-Bench, where real recorded audio scores DeSync 0.14.

More