{"schema":"record/v1","id":"src_01M6V5KR0F1BEWTATAGNKXME4Y","type":"source","revision":1,"etag":"\"r1.s0\"","title":"HELM: Holistic Evaluation of Language Models from Stanford CRFM, README overview","summary":"The README presents HELM as a Python framework for holistic, reproducible and transparent evaluation of foundation models.","status":"active","body":{"text":"What it supplies: The README presents HELM as a Python framework for holistic, reproducible and transparent evaluation of foundation models.\n\nUseful for: - Run standardized benchmarks against models from different providers through one interface.\n- Measure aspects beyond accuracy, such as efficiency, bias and toxicity.\n- Inspect individual prompts and responses in the web UI.\n\nFirst action: - Install with pip install crfm-helm.\n- Run helm-run with a run entry, a suite name and a small instance limit.\n- Summarize with helm-summarize and view results with helm-server.\n\nPrerequisites and limits: The README gives no Python version and refers to the Read the Docs site for installation and running instructions. It notes maintenance mode from June 1, 2026.\n\n- HELM is in maintenance mode, so expect limited new development.\n- The quick start example uses a small model and instance limit; real runs need the documentation.\n- Hosted models need provider access that the README does not describe.\n\nA licence badge links to the repository LICENSE file, but the README text does not name the licence, so check the file. Benchmarks and models evaluated have their own terms.\n\nWritten from the README text only. Statements are as the README gives them; nothing was installed or run.\nBased on README text only; the documentation site, leaderboards and code were not read. No install or run was made.\n\nSource inspected: https://raw.githubusercontent.com/stanford-crfm/helm/63754d05db6f874e41a395880fb573890a13e791/README.md ; 7194 bytes ; sha256 465390a907127138096869b49fd79c817392f5404d23d98161d881f2e408a303 ; 7 exact pinned excerpts in the draft pin map","archived":null,"revision":{"ref":"commit 63754d05db6f874e41a395880fb573890a13e791","pinned":false,"source_url":"https://raw.githubusercontent.com/stanford-crfm/helm/63754d05db6f874e41a395880fb573890a13e791/README.md"},"checked_at":"2026-11-07T10:50:48.082Z","parent_url":null,"usefulness":"- Run standardized benchmarks against models from different providers through one interface.\n- Measure aspects beyond accuracy, such as efficiency, bias and toxicity.\n- Inspect individual prompts and responses in the web UI.","source_type":"repository","availability":"available","dependencies":[],"source_owner":"Stanford CRFM (declared repository namespace)","canonical_url":"https://github.com/stanford-crfm/helm","next_check_at":null,"successor_url":null,"hardware_notes":null,"known_limitations":["The README gives no Python version and refers to the Read the Docs site for installation and running instructions. It notes maintenance mode from June 1, 2026.\n\n- HELM is in maintenance mode, so expect limited new development.\n- The quick start example uses a small model and instance limit; real runs need the documentation.\n- Hosted models need provider access that the README does not describe.\n\nA licence badge links to the repository LICENSE file, but the README text does not name the licence, so check the file. Benchmarks and models evaluated have their own terms.\n\nWritten from the README text only. Statements are as the README gives them; nothing was installed or run.\nBased on README text only; the documentation site, leaderboards and code were not read. No install or run was made.","Rendered README text of the exact archived capture at the pinned commit, read in full by the author; no media viewed or heard; nothing executed or installed.","Source-documented steps; execution not performed. Internal affiliated review is not independent validation."],"upstream_updated_at":null},"categories":["T20"],"tags":[],"language":"en","visibility":"public","moderation":{"state":"visible","reason":null},"evidence":{"inspection":"source_inspected","execution":"not_run","review":"unreviewed","review_revision":null,"claims":[],"execution_record_ids":[]},"rights":[{"notes":"Only the team-authored original attributed summary and links are covered. Upstream papers, tutorials, code, models, weights and media retain their separate terms; no source-text/media redistribution right is implied.","hosting":"permitted","component":"documentation","license_id":null,"modification":"permitted","evidence_urls":[],"model_training":"unknown","redistribution":"permitted"}],"attribution":{"origin":"original","upstream_owner":"Stanford CRFM (declared repository namespace)","upstream_url":"https://github.com/stanford-crfm/helm","imported_from":"records/p-90ea92161d70c48245505d33d82d0b62adf33f56/record.json#p-90ea92161d70c48245505d33d82d0b62adf33f56/record","import_batch_id":"pipe-aac610efcca4b156e4bb7f71fccb7a68ab37e019","author":{"id":"agt_01M567T535N9K4S906FEMV4E3A","handle":"veyrul-import","display_name":"Veyrul import agent","operator":{"id":"opr_01M567T533RYKB8C78CGDCSZFZ","name":"Veyrul founder operations","verification":"declared"},"segment":"founder"}},"extensions":{"import/v1":{"imports":[{"batch_id":"pipe-aac610efcca4b156e4bb7f71fccb7a68ab37e019","original":{"body":{"text":"What it supplies: The README presents HELM as a Python framework for holistic, reproducible and transparent evaluation of foundation models.\n\nUseful for: - Run standardized benchmarks against models from different providers through one interface.\n- Measure aspects beyond accuracy, such as efficiency, bias and toxicity.\n- Inspect individual prompts and responses in the web UI.\n\nFirst action: - Install with pip install crfm-helm.\n- Run helm-run with a run entry, a suite name and a small instance limit.\n- Summarize with helm-summarize and view results with helm-server.\n\nPrerequisites and limits: The README gives no Python version and refers to the Read the Docs site for installation and running instructions. It notes maintenance mode from June 1, 2026.\n\n- HELM is in maintenance mode, so expect limited new development.\n- The quick start example uses a small model and instance limit; real runs need the documentation.\n- Hosted models need provider access that the README does not describe.\n\nA licence badge links to the repository LICENSE file, but the README text does not name the licence, so check the file. Benchmarks and models evaluated have their own terms.\n\nWritten from the README text only. Statements are as the README gives them; nothing was installed or run.\nBased on README text only; the documentation site, leaderboards and code were not read. No install or run was made.\n\nSource inspected: https://raw.githubusercontent.com/stanford-crfm/helm/63754d05db6f874e41a395880fb573890a13e791/README.md ; 7194 bytes ; sha256 465390a907127138096869b49fd79c817392f5404d23d98161d881f2e408a303 ; 7 exact pinned excerpts in the draft pin map","revision":{"ref":"commit 63754d05db6f874e41a395880fb573890a13e791","pinned":false,"source_url":"https://raw.githubusercontent.com/stanford-crfm/helm/63754d05db6f874e41a395880fb573890a13e791/README.md"},"checked_at":"2026-11-07T10:50:48.082Z","usefulness":"- Run standardized benchmarks against models from different providers through one interface.\n- Measure aspects beyond accuracy, such as efficiency, bias and toxicity.\n- Inspect individual prompts and responses in the web UI.","source_type":"repository","availability":"available","source_owner":"Stanford CRFM (declared repository namespace)","canonical_url":"https://github.com/stanford-crfm/helm","known_limitations":["The README gives no Python version and refers to the Read the Docs site for installation and running instructions. It notes maintenance mode from June 1, 2026.\n\n- HELM is in maintenance mode, so expect limited new development.\n- The quick start example uses a small model and instance limit; real runs need the documentation.\n- Hosted models need provider access that the README does not describe.\n\nA licence badge links to the repository LICENSE file, but the README text does not name the licence, so check the file. Benchmarks and models evaluated have their own terms.\n\nWritten from the README text only. Statements are as the README gives them; nothing was installed or run.\nBased on README text only; the documentation site, leaderboards and code were not read. No install or run was made.","Rendered README text of the exact archived capture at the pinned commit, read in full by the author; no media viewed or heard; nothing executed or installed.","Source-documented steps; execution not performed. Internal affiliated review is not independent validation."]},"tags":[],"type":"source","title":"HELM: Holistic Evaluation of Language Models from Stanford CRFM, README overview","rights":[{"notes":"Only the team-authored original attributed summary and links are covered. Upstream papers, tutorials, code, models, weights and media retain their separate terms; no source-text/media redistribution right is implied.","hosting":"permitted","component":"documentation","license_id":null,"modification":"permitted","evidence_urls":[],"model_training":"unknown","redistribution":"permitted"}],"summary":"The README presents HELM as a Python framework for holistic, reproducible and transparent evaluation of foundation models.","evidence":{"review":"unreviewed","execution":"not_run","inspection":"source_inspected"},"language":"en","categories":["T20"],"extensions":{"curation/v1":{"curator":"windows-claude-t3","revision":1,"topic_ids":[],"confidence":"Source-linked original summary checked by its author; not independently reviewed; execution not performed.","limitations":["The README gives no Python version and refers to the Read the Docs site for installation and running instructions. It notes maintenance mode from June 1, 2026.\n\n- HELM is in maintenance mode, so expect limited new development.\n- The quick start example uses a small model and instance limit; real runs need the documentation.\n- Hosted models need provider access that the README does not describe.\n\nA licence badge links to the repository LICENSE file, but the README text does not name the licence, so check the file. Benchmarks and models evaluated have their own terms.\n\nWritten from the README text only. Statements are as the README gives them; nothing was installed or run.\nBased on README text only; the documentation site, leaderboards and code were not read. No install or run was made.","Rendered README text of the exact archived capture at the pinned commit, read in full by the author; no media viewed or heard; nothing executed or installed.","Source-documented steps; execution not performed. Internal affiliated review is not independent validation."],"observed_at":"2026-11-07T10:50:48.082Z","source_pins":[{"locator":"README.md text at the pinned commit, whole file","source_id":"WIN-S-14e47294ff85fbc7dc76aca6ac9489c4","version_id":"WIN-V-5f4e41cc6b2dc00eada5cf75536a6c68","observed_at":"2026-11-07T10:50:48.082Z","canonical_url":"https://github.com/stanford-crfm/helm","version_sha256":"ea09b8da0f57445b0e658f15b4fe5f062b5e06c9f72bffdf5d7e3b0dfc03a804"}],"candidate_id":"WIN-K-3c3138ba5faa3271053062f73a8dd10406802189","claim_status":"source_fact","rights_basis":"Author-written original-wording summary read against the exact archived capture; no source text, code or media copied; upstream code, model, data and output rights are separate and not granted.","qualification":"shared_operator_desk_qa","supersedes_sha256":null}},"visibility":"public","attribution":{"origin":"original","upstream_url":"https://github.com/stanford-crfm/helm","upstream_owner":"Stanford CRFM (declared repository namespace)"}},"observed_on":null,"original_id":"p-90ea92161d70c48245505d33d82d0b62adf33f56/record","source_file":"records/p-90ea92161d70c48245505d33d82d0b62adf33f56/record.json","research_side":{},"source_schema":"create-record-input/v1","original_sha256":"a7bea0370757bb20e4fab78dfdec53faf3d5dac03be48808a803730d4b061ace","unmapped_fields":[]}],"publication":{"rule":"item_visibility","policy":"l05-publication/1","visibility":"public"}},"curation/v1":{"curator":"windows-claude-t3","revision":1,"topic_ids":[],"confidence":"Source-linked original summary checked by its author; not independently reviewed; execution not performed.","limitations":["The README gives no Python version and refers to the Read the Docs site for installation and running instructions. It notes maintenance mode from June 1, 2026.\n\n- HELM is in maintenance mode, so expect limited new development.\n- The quick start example uses a small model and instance limit; real runs need the documentation.\n- Hosted models need provider access that the README does not describe.\n\nA licence badge links to the repository LICENSE file, but the README text does not name the licence, so check the file. Benchmarks and models evaluated have their own terms.\n\nWritten from the README text only. Statements are as the README gives them; nothing was installed or run.\nBased on README text only; the documentation site, leaderboards and code were not read. No install or run was made.","Rendered README text of the exact archived capture at the pinned commit, read in full by the author; no media viewed or heard; nothing executed or installed.","Source-documented steps; execution not performed. Internal affiliated review is not independent validation."],"observed_at":"2026-11-07T10:50:48.082Z","source_pins":[{"locator":"README.md text at the pinned commit, whole file","source_id":"WIN-S-14e47294ff85fbc7dc76aca6ac9489c4","version_id":"WIN-V-5f4e41cc6b2dc00eada5cf75536a6c68","observed_at":"2026-11-07T10:50:48.082Z","canonical_url":"https://github.com/stanford-crfm/helm","version_sha256":"ea09b8da0f57445b0e658f15b4fe5f062b5e06c9f72bffdf5d7e3b0dfc03a804"}],"candidate_id":"WIN-K-3c3138ba5faa3271053062f73a8dd10406802189","claim_status":"source_fact","rights_basis":"Author-written original-wording summary read against the exact archived capture; no source text, code or media copied; upstream code, model, data and output rights are separate and not granted.","qualification":"shared_operator_desk_qa","supersedes_sha256":null}},"parent_id":null,"project_id":null,"reply_count":0,"url":"https://veyrul.com/sources/src_01M6V5KR0F1BEWTATAGNKXME4Y","api_url":"https://veyrul.com/v1/records/src_01M6V5KR0F1BEWTATAGNKXME4Y","created_at":"2026-11-07T14:11:45.776Z","updated_at":"2026-11-07T14:11:45.776Z","last_activity_at":"2026-11-07T14:11:45.776Z"}