OmniEdit-Bench scoring: accuracy gates the total, and skipped edits change the evaluated sample

OmniEdit-Bench's pinned README f6662c5371c21392b80939710ed61adec68531be describes instruction-based video-edit evaluation, not a tested editor interface. This guide follows the complete text captured on 2026-10-28. No videos, task rows, evaluator code, prompt file or API were run or inspected. The documented dataset contains 790 tasks: Spatial240, Temporal200, Reasoning50, Audio100 and Reference200. Most source videos are described as 720p in either landscape1280x720 or portrait720x1280. A task supplies video_id, source_video and editing_instruction; Reference tasks also supply a relative reference image/video path. The taxonomy includes camera, motion, audio and reference edits, but it does not establish any generator's conditioning interface, mask convention or measured editing ability. Outputs are expected at eval_720p_10s/<model_name>/<video_id>.mp4. Use the task ID, not the source-video filename, as the output stem. The directory name alone does not establish that every clip is exactly ten seconds or specify FPS, codec, alignment and normalization. Those need actual task data and implementation inspection. According to the README, gemini_eval.py sends the source video, edited video, editing instruction and, for Reference tasks, the reference asset to Gemini. The listed rubric file is eval_configs/prompts.json; its actual prompt text was not inspected. The requested JSON contains accuracy, preservation, realism, consistency and a reason. Ratings are on a 0 to 5 scale: accuracy concerns the requested edit, preservation concerns unrelated objects/regions/actions/sounds, realism concerns plausibility, and consistency covers spatial/temporal/identity/audio-visual stability. Reference accuracy must also match the supplied reference. These are documented judge dimensions, not validated error detectors or independent human ratings. For valid scores from 0 to 5, the displayed formula uses penalty=accuracy/5 and overall=0.50*accuracy + penalty*(0.20*preservation + 0.15*realism + 0.15*consistency). It is not a simple50/20/15/15 weighted average: every other contribution is gated by instruction accuracy. As an arithmetic consequence, an accuracy0 yields overall0 even if the other scores are5. With accuracy3 and all others5, overall is3; the total cannot exceed accuracy within the documented score range. These are illustrations of the printed formula, not benchmark runs. Whether the script validates/clamps malformed ratings or implements the equation exactly remains uninspected. Coverage is part of the result. Missing edited videos are skipped, and Count means successfully evaluated tasks; failed API calls may also reduce it. Missing outputs are not automatically zero-score failures. Avg_Overall can therefore describe a different evaluated subset for each model. The README recommends reporting each track's average and Count plus the Gemini model/endpoint and missing-output handling. For a fair comparison, additionally preserve the exact evaluated IDs and missing/failed cases and use a matched subset or explicitly report the coverage difference. Do not present a partial average as completion of all790 tasks or silently change the benchmark to zero-fill skips. The README describes per-task details files containing the four ratings, reason, overall_score, video_id and instruction, plus track-level full_summary JSON. It describes resuming by skipping IDs already present in a details file. A changed output, instruction, rubric, judge version or endpoint must therefore be considered when interpreting reused results; an existing ID is not evidence that a new run configuration was evaluated. Binding saved results to the exact inputs and judge configuration is proposed review guidance, not verified cache behavior. The documented evaluator uses temperature0.0, two workers and up to three retry rounds. Temperature0 does not guarantee deterministic or human-equivalent judgments. This README supplies no exact immutable Gemini model/endpoint release, complete rubric implementation, human-agreement validation protocol or measured model leaderboard. The sample Avg_Overall in its output example is a format illustration, not a measured result for a named model. Before execution, inspect and pin gemini_eval.py, prompts.json, the five task JSON files and actual media provenance; record judge identity, request/response metadata, file hashes, frame/audio alignment and all failures. Sending source/edited/reference media to an external judge also requires applicable input and service permissions, which this capture does not establish. The card declares CC-BY-NC-4.0. Full license text, per-video/reference permissions and judge-service conditions were not inspected; no commercial-use or media-redistribution clearance is inferred. This is original source-linked analysis, not copied task/media content. Evidence is source_inspected/not_run/unreviewed. The local Qwen extractor and its support screens share the founder operator and provide no independent verification.

{
  "citations": [
    {
      "url": "https://huggingface.co/datasets/OmniEdit-Bench/OmniEdit-Bench/raw/f6662c5371c21392b80939710ed61adec68531be/README.md",
      "note": "Exact full README; scoring and coverage locators in source pins. Code, rubric, tasks and media not inspected."
    }
  ]
}