OpenAI Evals: framework and registry for evaluating LLMs, README overview
The README presents Evals as a framework and registry for testing how language models behave on your use case.
Reusable resources
Locations saved with this record. Check access, licensing and suitability at the source.
- commit 8eac7a7de5215c907fbddc30efdaf316913eccdd · source_revision · · availability: unknown
- OpenAI Evals: framework and registry for evaluating LLMs, README overview · repository · · availability: available (checked 2026-11-07)
Source: https://github.com/openai/evals
What it supplies: The README presents Evals as a framework and registry for testing how language models behave on your use case.
Useful for: - Run registry evals against a model.
- Write your own model-graded or templated evals.
First action: - Set OPENAI_API_KEY and use Python 3.9 or later.
- Fetch the registry data with Git-LFS.
- Install with pip install evals, or pip install -e . to develop evals.
Prerequisites and limits: Python 3.9 or later and an OpenAI API key; registry data comes through Git-LFS.
- Custom-code evals are not accepted as contributions.
- The workflow depends on an OpenAI API key.
Source-documented: a contribution disclaimer names the MIT licence for the repository. Running evals uses the paid OpenAI API.
Written from the README text only. Statements are as the README gives them; nothing was installed or run.
Based on intro, setup, running, writing, FAQ and disclaimer README text; linked docs were not read. Nothing was installed or run.
Source inspected: https://raw.githubusercontent.com/openai/evals/8eac7a7de5215c907fbddc30efdaf316913eccdd/README.md ; 6463 bytes ; sha256 f0b50bd53e2fbc81be774a6bfb0cf20997aa7509bd22925ae779a87d57c7c914 ; 9 exact pinned excerpts in the draft pin map
Structured fields
{
"revision": {
"ref": "commit 8eac7a7de5215c907fbddc30efdaf316913eccdd",
"pinned": false,
"source_url": "https://raw.githubusercontent.com/openai/evals/8eac7a7de5215c907fbddc30efdaf316913eccdd/README.md"
},
"checked_at": "2026-11-07T12:17:28.862Z",
"usefulness": "- Run registry evals against a model.\n- Write your own model-graded or templated evals.",
"source_type": "repository",
"availability": "available",
"source_owner": "OpenAI (evals) (declared repository namespace)",
"known_limitations": [
"Python 3.9 or later and an OpenAI API key; registry data comes through Git-LFS.\n\n- Custom-code evals are not accepted as contributions.\n- The workflow depends on an OpenAI API key.\n\nSource-documented: a contribution disclaimer names the MIT licence for the repository. Running evals uses the paid OpenAI API.\n\nWritten from the README text only. Statements are as the README gives them; nothing was installed or run.\nBased on intro, setup, running, writing, FAQ and disclaimer README text; linked docs were not read. Nothing was installed or run.",
"Rendered README text of the exact archived capture at the pinned commit, read in full by the author; no media viewed or heard; nothing executed or installed.",
"Source-documented steps; execution not performed. Internal affiliated review is not independent validation."
]
}Rights
| component | license | hosting | redistribution | modification | training | Evidence and notes |
|---|---|---|---|---|---|---|
| documentation | unknown | permitted | permitted | permitted | unknown | Only the team-authored original attributed summary and links are covered. Upstream papers, tutorials, code, models, weights and media retain their separate terms; no source-text/media redistribution right is implied. |