What exactly does RoleBench score for "speaking style", how were the human and GPT-4 judgements made, and what can its English data and licences be used for?
RoleLLM (arXiv 2310.00746v3, 18 Jun 2024) builds RoleBench: 95 English and 5 Chinese roles, 168,093 samples. Speaking style is scored as ROUGE-L overlap with GPT-4-generated reference answers (CUS), alongside GPT-4 and human win rates against a GPT-4 prompted baseline (RoleGPT). The human check used three NLP graduate students on a 500-instruction subset, with no agreement statistic in the text read. The setup is single-turn and per-role; the data card says apache-2.0 but the repository has no licence file and the scripts behind the data were not examined.
RoleLLM (arXiv 2310.00746v3, 18 Jun 2024) builds RoleBench: 95 English and 5 Chinese roles, 168,093 samples. Speaking style is scored as ROUGE-L overlap with GPT-4-generated reference answers (CUS), alongside GPT-4 and human win rates against a GPT-4 prompted baseline (RoleGPT). The human check used three NLP graduate students on a 500-instruction subset, with no agreement statistic in the text read. The setup is single-turn and per-role; the data card says apache-2.0 but the repository has no licence file and the scripts behind the data were not examined.
Curator qualifications: RoleBench: style scored as ROUGE-L vs GPT-4 references; human check by three graduate students on 500 instructions, no agreement statistic.
Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.
Structured fields
{
"citations": [
{
"url": "https://arxiv.org/html/2310.00746v3",
"note": "candidate evidence excerpts"
},
{
"url": "https://huggingface.co/datasets/ZenMoore/RoleBench/raw/a57ed54f9613921e4a5f1b63601a558cd5acf971/README.md",
"note": "candidate evidence excerpts"
},
{
"url": "https://raw.githubusercontent.com/InteractiveNLP-Team/RoleLLM-public/131a157c9962f46d36a29bcec6962b5acfa7644a/README.md",
"note": "candidate evidence excerpts"
}
]
}Evidence claims
- [sourced_fact] RoleLLM (arXiv 2310.00746v3, 18 Jun 2024) builds RoleBench: 95 English and 5 Chinese roles, 168,093 samples. Speaking style is scored as ROUGE-L overlap with GPT-4-generated reference answers (CUS), alongside GPT-4 and human win rates against a GPT-4 prompted baseline (RoleGPT). The human check used three NLP graduate students on a 500-instruction subset, with no agreement statistic in the text read. The setup is single-turn and per-role; the data card says apache-2.0 but the repository has no licence file and the scripts behind the data were not examined. · locator: candidate evidence excerpts source
Rights
| component | license | hosting | redistribution | modification | training | Evidence and notes |
|---|---|---|---|---|---|---|
| documentation | unknown | permitted | permitted | unknown | unknown | Applies to this original summary only, not to the cited sources. |