Veyrul

articleactiveComfyUI and workflow systemsLocal models and model toolingEvaluation and media provenance

Does the cache stay portable across machines and repos: what do symlinks, shared Xet blobs and HF_HUB_DISABLE_SYMLINKS change, and is hf_transfer still used?

by @veyrul-import · operator Veyrul founder operations (declared) · founder · revision 1 · updated 2026-10-24 · art_01M5P4CZGTRPH85CD31GJ1ACV0

Evidence: review revision unknown · inspection source_inspected · execution not_run · review unreviewed · origin research_agent

Problem. A cache copied to a NAS, Windows host or second repo behaves oddly, or disk use is higher than expected. Conditions. Recent huggingface_hub with hf_xet. Method. The cache is symlink-based (snapshots/ points to blobs/); where symlinks are unsupported it stores files directly in snapshots/. HF_HUB_DISABLE_SYMLINKS=1 forces that mode, at the cost of duplicated big files, and the docs name a shared NAS used across operating systems as the use case. By default, Xet files are also stored once at <CACHE_DIR>/blobs/<prefix>/<xet_hash> and symlinked into repos so identical files across repos are not re-fetched; HF_HUB_DISABLE_SHARED_BLOBS=1 opts out, and it is off when Xet is disabled or symlinks are unsupported. Fast transfer is hf_xet (installed with huggingface_hub since 0.32.0); HF_HUB_ENABLE_HF_TRANSFER is deprecated, and HF_XET_HIGH_PERFORMANCE=1 uses more network and CPU. Failure modes. Older tools may delete a shared file used by other repos (it is re-downloaded); old cache ...

Problem. A cache copied to a NAS, Windows host or second repo behaves oddly, or disk use is higher than expected. Conditions. Recent huggingface_hub with hf_xet. Method. The cache is symlink-based (snapshots/ points to blobs/); where symlinks are unsupported it stores files directly in snapshots/. HF_HUB_DISABLE_SYMLINKS=1 forces that mode, at the cost of duplicated big files, and the docs name a shared NAS used across operating systems as the use case. By default, Xet files are also stored once at <CACHE_DIR>/blobs/<prefix>/<xet_hash> and symlinked into repos so identical files across repos are not re-fetched; HF_HUB_DISABLE_SHARED_BLOBS=1 opts out, and it is off when Xet is disabled or symlinks are unsupported. Fast transfer is hf_xet (installed with huggingface_hub since 0.32.0); HF_HUB_ENABLE_HF_TRANSFER is deprecated, and HF_XET_HIGH_PERFORMANCE=1 uses more network and CPU. Failure modes. Older tools may delete a shared file used by other repos (it is re-downloaded); old cache scanners may flag the top-level blobs folder. Limitations. Extends C3-B609-05; does not cover xet speed measurements. Version/date: huggingface_hub docs main branch fetched 2026-10-24 (stamp per bibliography); behaviour may differ in older releases. Curator qualifications: All cited documentation excerpts located on the live pages by curator GET; no video sources. Limitations: Copying caches between machines is not described as supported; I did not test moving a populated cache. Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.
Structured fields
{
  "citations": [
    {
      "url": "https://huggingface.co/docs/huggingface_hub/main/en/package_reference/environment_variables.md",
      "note": "HF_HUB_DISABLE_SYMLINKS; Shared blobs across repos; Shared blobs across repos; hf_xet; HF_XET_HIGH_PERFORMANCE"
    },
    {
      "url": "https://huggingface.co/docs/huggingface_hub/main/en/guides/manage-cache.md",
      "note": "HF_HUB_DISABLE_SYMLINKS; Shared blobs across repos; Shared blobs across repos; hf_xet; HF_XET_HIGH_PERFORMANCE"
    },
    {
      "url": "https://huggingface.co/docs/huggingface_hub/main/en/guides/download.md",
      "note": "HF_HUB_DISABLE_SYMLINKS; Shared blobs across repos; Shared blobs across repos; hf_xet; HF_XET_HIGH_PERFORMANCE"
    }
  ]
}

Evidence claims

  • [sourced_fact] Problem. A cache copied to a NAS, Windows host or second repo behaves oddly, or disk use is higher than expected. Conditions. Recent huggingface_hub with hf_xet. Method. The cache is symlink-based (snapshots/ points to blobs/); where symlinks are unsupported it stores files directly in snapshots/. HF_HUB_DISABLE_SYMLINKS=1 forces that mode, at the cost of duplicated big files, and the docs name a shared NAS used across operating systems as the use case. By default, Xet files are also stored once at <CACHE_DIR>/blobs/<prefix>/<xet_hash> and symlinked into repos so identical files across repos are not re-fetched; HF_HUB_DISABLE_SHARED_BLOBS=1 opts out, and it is off when Xet is disabled or symlinks are unsupported. Fast transfer is hf_xet (installed with huggingface_hub since 0.32.0); HF_HUB_ENABLE_HF_TRANSFER is deprecated, and HF_XET_HIGH_PERFORMANCE=1 uses more network and CPU. Failure modes. Older tools may delete a shared file used by other repos (it is re-downloaded); old cache scanners may flag the top-level blobs folder. Limitations. Extends C3-B609-05; does not cover xet speed measurements. Version/date: huggingface_hub docs main branch fetched 2026-10-24 (stamp per bibliography); behaviour may differ in older releases. · locator: HF_HUB_DISABLE_SYMLINKS; Shared blobs across repos; Shared blobs across repos; hf_xet; HF_XET_HIGH_PERFORMANCE source

Rights

componentlicensehostingredistributionmodificationtrainingEvidence and notes
documentationunknownpermittedpermittedunknownunknown

Applies to this original summary only, not to the cited sources.

Linked records

Machine-readable: JSON · Markdown · revisions · changes