What would a regression gate for a ComfyUI workflow change look like, and which parts are documented versus my own design?
Problem: a workflow edit (new node, changed sampler, swapped model) can silently change output or break on another machine. Conditions: you can run a ComfyUI server (CPU or GPU) in CI and keep a baseline output set. Method (INFERENCE, assembled from documented parts): (1) static portability layer: compare the workflow's node types and recorded version fields with the target install (see C3-B590-04..06). (2) start-up layer: launch with the node pack installed and fail on import errors, as in ComfyUI's launch smoke test. (3) execution layer: submit the API-format graph with comfy run --wait --timeout and require exit 0 and expected output files, which is all Comfy's GPU CI checks. (4) quality layer: fixed seed, baseline directory and image-similarity threshold as described in C3-B579-13; keep it on the same hardware and software pins as the baseline because the repo provides no cross-device guarantee. (5) drift layer: scheduled re-check of pins. (6) release layer: publish only after l...
Structured fields
{
"citations": [
{
"url": "https://raw.githubusercontent.com/Comfy-Org/ComfyUI/fb3befd168731438f88bc96e0920a538e8649878/tests/README.md",
"note": "Quality regression test; line 357; line 15"
},
{
"url": "https://raw.githubusercontent.com/Comfy-Org/comfy-cli/0b49c624571571eaef36ed6e8c8d892cf2d700c4/tests/e2e/test_e2e.py",
"note": "Quality regression test; line 357; line 15"
},
{
"url": "https://raw.githubusercontent.com/Comfy-Org/ComfyUI/fb3befd168731438f88bc96e0920a538e8649878/.github/workflows/test-unit.yml",
"note": "Quality regression test; line 357; line 15"
}
]
}Evidence claims
- [sourced_fact] Problem: a workflow edit (new node, changed sampler, swapped model) can silently change output or break on another machine. Conditions: you can run a ComfyUI server (CPU or GPU) in CI and keep a baseline output set. Method (INFERENCE, assembled from documented parts): (1) static portability layer: compare the workflow's node types and recorded version fields with the target install (see C3-B590-04..06). (2) start-up layer: launch with the node pack installed and fail on import errors, as in ComfyUI's launch smoke test. (3) execution layer: submit the API-format graph with comfy run --wait --timeout and require exit 0 and expected output files, which is all Comfy's GPU CI checks. (4) quality layer: fixed seed, baseline directory and image-similarity threshold as described in C3-B579-13; keep it on the same hardware and software pins as the baseline because the repo provides no cross-device guarantee. (5) drift layer: scheduled re-check of pins. (6) release layer: publish only after layers 1-4 pass. Failure modes: advisory jobs (continue-on-error) hiding red; GPU-nondeterminism false alarms; baselines refreshed without review. Limitations: this combination was not built or run; thresholds are not given by sources. Version/date: sources at ComfyUI fb3befd, comfy-cli 0b49c62, 2026-10-23. · locator: Quality regression test; line 357; line 15 source
Rights
| component | license | hosting | redistribution | modification | training | Evidence and notes |
|---|---|---|---|---|---|---|
| documentation | unknown | permitted | permitted | unknown | unknown | Applies to this original summary only, not to the cited sources. |
Linked records
- cites → https://raw.githubusercontent.com/Comfy-Org/ComfyUI/fb3befd168731438f88bc96e0920a538e8649878/.github/workflows/test-unit.yml
- cites → https://raw.githubusercontent.com/Comfy-Org/ComfyUI/fb3befd168731438f88bc96e0920a538e8649878/tests/README.md
- cites → https://raw.githubusercontent.com/Comfy-Org/comfy-cli/0b49c624571571eaef36ed6e8c8d892cf2d700c4/tests/e2e/test_e2e.py