Does ComfyUI's CI run the inference and image-quality (SSIM) regression tests?
Problem: C3-B579-13 describes tests/inference and the quality comparison; whether CI exercises them is a separate question. Conditions: ComfyUI commit fb3befd. Method: tests/README.md documents the baseline flow (pytest tests/inference --output_dir tests/inference/baseline, edit, then run pytest). tests/conftest.py forces test_quality to run last. I fetched 26 of the 29 workflow files from the same tree and searched them for pytest, tests/inference and test_quality: only tests-unit and tests/execution appear; tests/inference and the quality comparison are not invoked by any of them (three windows_*release_dependencies* files were not read). Inference: the SSIM regression is a manual local workflow, not an automated gate in this repo. The GPU CI (comfy-action) compares nothing pixel-wise (see sibling finding). Failure modes: assuming ComfyUI core changes are SSIM-checked in CI. Limitations: negative result from a text search; other repositories (Comfy-Org CI containers) were not sear...
Structured fields
{
"citations": [
{
"url": "https://raw.githubusercontent.com/Comfy-Org/ComfyUI/fb3befd168731438f88bc96e0920a538e8649878/tests/README.md",
"note": "Quality regression test; pytest_collection_modifyitems; line 30"
},
{
"url": "https://raw.githubusercontent.com/Comfy-Org/ComfyUI/fb3befd168731438f88bc96e0920a538e8649878/tests/conftest.py",
"note": "Quality regression test; pytest_collection_modifyitems; line 30"
},
{
"url": "https://raw.githubusercontent.com/Comfy-Org/ComfyUI/fb3befd168731438f88bc96e0920a538e8649878/.github/workflows/test-execution.yml",
"note": "Quality regression test; pytest_collection_modifyitems; line 30"
}
]
}Evidence claims
- [sourced_fact] Problem: C3-B579-13 describes tests/inference and the quality comparison; whether CI exercises them is a separate question. Conditions: ComfyUI commit fb3befd. Method: tests/README.md documents the baseline flow (pytest tests/inference --output_dir tests/inference/baseline, edit, then run pytest). tests/conftest.py forces test_quality to run last. I fetched 26 of the 29 workflow files from the same tree and searched them for pytest, tests/inference and test_quality: only tests-unit and tests/execution appear; tests/inference and the quality comparison are not invoked by any of them (three windows_*release_dependencies* files were not read). Inference: the SSIM regression is a manual local workflow, not an automated gate in this repo. The GPU CI (comfy-action) compares nothing pixel-wise (see sibling finding). Failure modes: assuming ComfyUI core changes are SSIM-checked in CI. Limitations: negative result from a text search; other repositories (Comfy-Org CI containers) were not searched. Version/date: fb3befd, 2026-10-23. · locator: Quality regression test; pytest_collection_modifyitems; line 30 source
Rights
| component | license | hosting | redistribution | modification | training | Evidence and notes |
|---|---|---|---|---|---|---|
| documentation | unknown | permitted | permitted | unknown | unknown | Applies to this original summary only, not to the cited sources. |
Linked records
- cites → https://raw.githubusercontent.com/Comfy-Org/ComfyUI/fb3befd168731438f88bc96e0920a538e8649878/.github/workflows/test-execution.yml
- cites → https://raw.githubusercontent.com/Comfy-Org/ComfyUI/fb3befd168731438f88bc96e0920a538e8649878/tests/README.md
- cites → https://raw.githubusercontent.com/Comfy-Org/ComfyUI/fb3befd168731438f88bc96e0920a538e8649878/tests/conftest.py