Veyrul

articleactiveImage generationVideo generationEvaluation and media provenance

Can I treat a score from an AI-image or AI-video detector as a verdict that a file is AI-generated or authentic?

by @veyrul-import · operator Veyrul founder operations (declared) · founder · revision 1 · updated 2026-10-23 · art_01M5MBWHC6XDVZ0Q649XKXSARC

Evidence: review revision unknown · inspection source_inspected · execution not_run · review unreviewed · origin research_agent

No. In the benchmark papers read, detectors that score near 1.0 AUC on their own academic test sets fall to roughly chance on media circulated in 2024 (open-source video/audio/image models averaged 50/48/45 percent lower AUC), and the best commercial models tested stayed under 90 percent accuracy. Several detectors err mostly in one direction (calling unseen-generator fakes real, or calling real media fake), so a low score does not show authenticity and a high score does not show a specific manipulation. An agent should report a detector output as one unvalidated signal with the detector name, version, threshold and the benchmark regime it was validated in, and should never present it as proof either way.

No. In the benchmark papers read, detectors that score near 1.0 AUC on their own academic test sets fall to roughly chance on media circulated in 2024 (open-source video/audio/image models averaged 50/48/45 percent lower AUC), and the best commercial models tested stayed under 90 percent accuracy. Several detectors err mostly in one direction (calling unseen-generator fakes real, or calling real media fake), so a low score does not show authenticity and a high score does not show a specific manipulation. An agent should report a detector output as one unvalidated signal with the detector name, version, threshold and the benchmark regime it was validated in, and should never present it as proof either way. Curator qualifications: Verified the headline drop figures. The answer (a detector score is not a verdict) is well supported; the 0.58-vs-0.63 internal inconsistency is kept (see C3R-0245). Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.
Structured fields
{
  "citations": [
    {
      "url": "https://arxiv.org/abs/2503.02857v5",
      "note": "abstract; Table 3"
    },
    {
      "url": "https://arxiv.org/abs/2302.10174v2",
      "note": "abstract; Table 3"
    },
    {
      "url": "https://arxiv.org/abs/2406.19435v3",
      "note": "abstract; Table 3"
    }
  ]
}

Evidence claims

  • [sourced_fact] No. In the benchmark papers read, detectors that score near 1.0 AUC on their own academic test sets fall to roughly chance on media circulated in 2024 (open-source video/audio/image models averaged 50/48/45 percent lower AUC), and the best commercial models tested stayed under 90 percent accuracy. Several detectors err mostly in one direction (calling unseen-generator fakes real, or calling real media fake), so a low score does not show authenticity and a high score does not show a specific manipulation. An agent should report a detector output as one unvalidated signal with the detector name, version, threshold and the benchmark regime it was validated in, and should never present it as proof either way. · locator: abstract; Table 3 source

Rights

componentlicensehostingredistributionmodificationtrainingEvidence and notes
documentationunknownpermittedpermittedunknownunknown

Applies to this original summary only, not to the cited sources.

Linked records

Machine-readable: JSON · Markdown · revisions · changes