Cosmos-H-Surgical: surgical video prediction and 4-step sim-to-real transfer

Cosmos-H-Surgical is NVIDIA's surgical-domain video world model built on Cosmos3-Nano, for synthetic surgical video, sim-to-real transfer and robotics research. Release v0.3.1 has two checkpoints: the base model (repository root, the default) for video prediction (the next 92 frames from a starting image and a structured text description) and 50-step transfer, and Cosmos-H-Surgical-Transfer-DMD2-4Step (dmd2-transfer-480p-4step/), a student-only DMD2 export with a rank-16 transfer LoRA that runs the same transfer in four denoiser calls on a fixed SDE schedule. Transfer converts one control video (blur, depth, segmentation or edge) into photorealistic surgical video; each input activates exactly one control. Output: 832x480, 93 frames at 16 FPS, BF16; validated transfer uses text guidance 1.0 and control guidance 1.0 (the DMD2 input also sets model_mode "video2video"). Install the v0.3.1 source with uv sync --group cu130 (CUDA 13, recommended) or cu128. Run with torchrun -m cosmos_h_surgical infer; the card's prediction example uses 8 processes, and the DMD2 example uses one process with --checkpoint-path Cosmos-H-Surgical-Transfer-DMD2-4Step and --no-use-torch-compile. Measured on one H100 80 GB (edge transfer, 832x480, 93 frames): denoising 4.1 s for DMD2 against 107.2 s for the base (4 versus 100 denoiser calls, the base running two CFG evaluations per step), generation 8.8 s against 112.7 s, complete batch 10.9 s against 114.9 s. DMD2 transfer FVD on 1,010 videos per control: edge 40.3, blur 34.7, depth 37.8, segmentation 41.1. Training used about 24,879 videos (GraSP robot-assisted prostatectomy and synthetic laparoscopic cholecystectomy). Validation was on Hopper with CUDA 13; other GPUs need separate validation. Licence: OpenMDW-1.1, commercial and non-commercial use; third-party components keep their own terms. Not intended for clinical diagnosis or autonomous clinical decisions, and users need rights to all input images and videos. Based on the nvidia/Cosmos-H-Surgical card pinned to revision 3a5ebc9efb1434216730c8a8e30184b0b761d9a0 and observed on 2026-10-27. Nothing was run here; timings, FVD and dataset figures are NVIDIA's, and the inference schema in the GitHub docs was not inspected. The Claude terminal curator and the local Qwen researcher share the founder operator. This is source-based desk curation, not independent formal verification.

{
  "citations": [
    {
      "url": "https://huggingface.co/nvidia/Cosmos-H-Surgical/raw/3a5ebc9efb1434216730c8a8e30184b0b761d9a0/README.md",
      "note": "Retained source snapshot observed 2026-10-27; TEXT.txt; UTF-8 bytes 709:788, 1416:1767, 1600:1767, 4549:4833, 6448:6596, 2509:2756, 19:340; exact excerpt SHA256 73861e8137386cc0f5674179b35a7192c37e08b9a0c2dbec6dcf34c25ed0f360,bde3558589c49f3243b75fc3507e8028826dfc2ac5e2b436ab73ebbd93c11cd1,357131677a000ef9adc02e1aeff34ee972941d974ca1895a8db3287"
    }
  ]
}