How strongly should I set IP-Adapter's image-prompt scale, how do I limit it to style or layout blocks, and what does it do to non-square references?

Problem. A reference image should steer look or subject without overriding the text prompt or ignoring the rest of the frame. Conditions. Stable Diffusion 1.5 / SDXL pipelines with an IP-Adapter checkpoint (h94/IP-Adapter), via the original repo or diffusers. Method. (1) Image-only prompting: scale=1.0 with an empty or generic text prompt; for image plus text, start near scale=0.5 and move up for more reference fidelity, down for more diversity and text control (README best practice; diffusers: set_ip_adapter_scale(0.5) balances). (2) In diffusers, scale can be a dict keyed by UNet block so the adapter only acts in selected blocks, e.g. {'down': {'block_2': [0.0, 1.0]}, 'up': {'block_0': [0.0, 1.0, 0.0]}} for layout plus style (the InstantStyle pattern); omitted layers default to 0. Style-only activation uses just up block_0. (3) Multiple adapters or reference images take a list of scales, e.g. set_ip_adapter_scale([0.7, 0.3]). (4) CLIP's default processor centre-crops, so pre-crop or resize references to a square or the outer parts are lost. Failure modes. All-layer activation follows the reference more strongly and can reduce diversity (diffusers doc); content outside the centre crop is ignored for non-square images. Limitations. Docs give starting values, not tuned ranges per base model; the repo README still pins diffusers 0.22.1 in its install block, so API details differ from current diffusers. Version/date. IP-Adapter repo README (last release note 2024/01/19); diffusers docs main branch fetched 2026-10-23. Curator qualifications: All cited documentation excerpts located on the live pages by curator GET; no video sources. Limitations: Per-block scale semantics come from the diffusers doc; I did not run any generation, so no measured effect sizes. The number 0.5 is a documented starting point, not an optimum. Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.

{
  "citations": [
    {
      "url": "https://raw.githubusercontent.com/tencent-ailab/IP-Adapter/main/README.md",
      "note": "Best Practice (SD_1.5); IP-Adapter for non-square images; intro to set_ip_adapter_scale; Style and layout control; Style and layout control"
    },
    {
      "url": "https://raw.githubusercontent.com/huggingface/diffusers/main/docs/source/en/using-diffusers/ip_adapter.md",
      "note": "Best Practice (SD_1.5); IP-Adapter for non-square images; intro to set_ip_adapter_scale; Style and layout control; Style and layout control"
    }
  ]
}