When does dynamic batching help or hurt a diffusion/video model server, and which Triton settings trade latency for throughput?
Problem. Deciding whether to batch concurrent image/video requests on one GPU or run them one at a time. Conditions. NVIDIA Triton Inference Server docs/user_guide/batcher.md at commit d13379e3 (2026-10-22). Method. The docs say the dynamic batcher combines requests into batches to typically increase throughput and should be used for stateless models; it is configured per model (dynamic_batching { }). Tuning loop from the docs: measure latency and throughput with Performance Analyzer under the default config; if latency is within budget, raise the maximum batch size and/or set max_queue_delay_microseconds to a non-zero value, increasing it until the latency budget is exceeded. The delay makes the batcher wait for a full (or preferred) batch but never longer than the configured delay for any request. Preferred batch sizes are discouraged for most models and only worthwhile if a size is much faster. Queue policy lets you cap queue size and set timeouts with reject or delay actions, an...
Problem. Deciding whether to batch concurrent image/video requests on one GPU or run them one at a time. Conditions. NVIDIA Triton Inference Server docs/user_guide/batcher.md at commit d13379e3 (2026-10-22). Method. The docs say the dynamic batcher combines requests into batches to typically increase throughput and should be used for stateless models; it is configured per model (dynamic_batching { }). Tuning loop from the docs: measure latency and throughput with Performance Analyzer under the default config; if latency is within budget, raise the maximum batch size and/or set max_queue_delay_microseconds to a non-zero value, increasing it until the latency budget is exceeded. The delay makes the batcher wait for a full (or preferred) batch but never longer than the configured delay for any request. Preferred batch sizes are discouraged for most models and only worthwhile if a size is much faster. Queue policy lets you cap queue size and set timeouts with reject or delay actions, and priority levels, so a service can shed load rather than build unbounded latency. Sequence batching (different feature) is for stateful models. Inference for diffusion: a multi-step sampler holds the GPU for many seconds, so batching requests that differ in resolution, step count or LoRAs is only possible when tensor shapes match, and for large video models a batch of one may already saturate memory; ComfyUI's own server runs one prompt at a time per process (C3-B574-08), so scale-out is more instances. Failure modes. Adding queue delay on interactive paths; batching mixed seeds/prompts then expecting identical per-image results (see C3-B060-G23-02 on batch position); unbounded queues. Limitations. Triton docs are model-agnostic; no diffusion-specific measurements were read, so the diffusion advice is inference. Version and date. Commit d13379e3.
Curator qualifications: All cited documentation excerpts located on the live pages by curator GET; no video sources.
Limitations: Diffusion applicability is inference.
Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.
Structured fields
{
"citations": [
{
"url": "https://raw.githubusercontent.com/triton-inference-server/server/d13379e3720617551e9b1fed0b1c2ba59f65b2e1/docs/user_guide/batcher.md",
"note": "Dynamic Batcher; tuning bullets; Delayed batching"
}
]
}Evidence claims
- [sourced_fact] Problem. Deciding whether to batch concurrent image/video requests on one GPU or run them one at a time. Conditions. NVIDIA Triton Inference Server docs/user_guide/batcher.md at commit d13379e3 (2026-10-22). Method. The docs say the dynamic batcher combines requests into batches to typically increase throughput and should be used for stateless models; it is configured per model (dynamic_batching { }). Tuning loop from the docs: measure latency and throughput with Performance Analyzer under the default config; if latency is within budget, raise the maximum batch size and/or set max_queue_delay_microseconds to a non-zero value, increasing it until the latency budget is exceeded. The delay makes the batcher wait for a full (or preferred) batch but never longer than the configured delay for any request. Preferred batch sizes are discouraged for most models and only worthwhile if a size is much faster. Queue policy lets you cap queue size and set timeouts with reject or delay actions, and priority levels, so a service can shed load rather than build unbounded latency. Sequence batching (different feature) is for stateful models. Inference for diffusion: a multi-step sampler holds the GPU for many seconds, so batching requests that differ in resolution, step count or LoRAs is only possible when tensor shapes match, and for large video models a batch of one may already saturate memory; ComfyUI's own server runs one prompt at a time per process (C3-B574-08), so scale-out is more instances. Failure modes. Adding queue delay on interactive paths; batching mixed seeds/prompts then expecting identical per-image results (see C3-B060-G23-02 on batch position); unbounded queues. Limitations. Triton docs are model-agnostic; no diffusion-specific measurements were read, so the diffusion advice is inference. Version and date. Commit d13379e3. · locator: Dynamic Batcher; tuning bullets; Delayed batching source
Rights
| component | license | hosting | redistribution | modification | training | Evidence and notes |
|---|---|---|---|---|---|---|
| documentation | unknown | permitted | permitted | unknown | unknown | Applies to this original summary only, not to the cited sources. |