What did Deepfake-Eval-2024 find about open-source and commercial detectors on in-the-wild deepfakes, including false positives on real media and which media traits cause errors?

Deepfake-Eval-2024 (TrueMedia.org; 45 h video, 56.5 h audio, 1,975 images collected in 2024) is the in-the-wild benchmark read here. Off-the-shelf open-source detectors scored AUC 0.43-0.63 across modalities, with real-media false-positive rates of 0.31-0.39 for video and 0.22-0.99 for images (UFD labelled 99 percent of real images fake). Anonymised commercial detectors did better but none reached 90 percent accuracy (best video 0.78, image 0.82 to 0.86 depending on which table is read). Errors concentrated on diffusion-generated video, mixed real/fake faces (31 percent accuracy drop) and non-facial edits. Commercial results are the benchmark authors measurement of vendors who partnered with them, with vendors anonymised. Curator correction 2026-10-23 (discovery audit record C3-B678-02; sources re-fetched 2026-10-23, excerpts re-located by curator GET): Problem. C3-B116-G42-03 summarises commercial detectors as better than open-source ones but 'none reached 90 percent accuracy', and then gives only video (0.78) and image (0.82 to 0.86) numbers. Conditions. Deepfake-Eval-2024 (arXiv 2503.02857v5) Table 5 (best commercial model per modality) reads video 0.78 accuracy / 0.79 AUC, audio 0.89 / 0.93, image 0.82 / 0.90 (image precision 0.99, recall 0.71). Audio therefore sits one point under the 90 percent line, and the authors attribute the strong audio result to limits of their audio labelling approach rather than to detector quality. Method. When reporting this benchmark, give all three modalities, say that audio is the most favourable and the least trustworthy comparison, and keep the open-source figures (AUC 0.43 to 0.63 off the shelf; real-media false-positive rates 0.31 to 0.39 video, 0.22 to 0.99 image, up to 1.00 for audio P3 in Table S8) alongside. Failure modes. Omitting audio makes the commercial range look uniformly weak (about 0.8) when one modality is within a point of the human-analyst bound the authors set at 90 percent. Limitations. The 0.82 versus 0.86 image figures in the target were not both located; Table 5 gives 0.82 and the 0.86 in the text belongs to the finetuned audio model P3. Version/date. arXiv v5, fetched 2026-10-23; vendors are anonymised and partnered with the authors. Curator qualifications: Benchmark facts verified. Of the producer's flagged inconsistencies: (1) the text's max open-source AUC 0.58 vs Table 3's GenConViT 0.63 is a real text/table mismatch; (2) FTCN F1 0.41 is inconsistent with its P 0.51 / R 0.67 (the curator computes F1 0.579); (3) Table 5's 'best' image row equals S10 rank #2. That row has the higher AUC (0.90 vs 0.88) while S10 ranks appear to follow accuracy, so 'best' was likely selected by AUC. This is a selection-criterion difference, not a transcription error. Item (3): explain as metric-dependent 'best' (AUC-best vs accuracy-best image model); quote both and name the metric. Items (1) and (2): confirmed internal inconsistencies; cite table values and flag the text figure and FTCN F1. Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.

{
  "citations": [
    {
      "url": "https://arxiv.org/abs/2503.02857v5",
      "note": "work/html_2503.02857v5.html Deepfake-Eval-2024 arXiv v5 HTML: sec 4.1, Table 3, Table 5, Table S10"
    },
    {
      "url": "https://arxiv.org/html/2503.02857v5",
      "note": "correction source (record C3-B678-02): Table 5 (flattened); Sec. 4.3; Sec. 4.3"
    }
  ]
}