Was CoSER's automatic LLM judge validated against its human raters, with what statistic and numbers, and how far can an agent trust that validation for judging character dialogue?
Yes, partly. Table 5 reports 'alignment' = percentage agreement with the human raters on which of two models is better, ties dropped. GPT-4o as judge gives 68.6, DeepSeek-V3 65.1 and DeepSeek-R1 77.5; BLEU gets 75.3 and ROUGE-L 72.0, so only the R1 judge beats the n-gram metrics. No correlation coefficient, confidence interval or inter-annotator agreement is given. The paper's main results use the lower-aligned GPT-4o judge. The check is at the whole-score level with 7 models, not per rubric dimension such as Character Fidelity.
Yes, partly. Table 5 reports 'alignment' = percentage agreement with the human raters on which of two models is better, ties dropped. GPT-4o as judge gives 68.6, DeepSeek-V3 65.1 and DeepSeek-R1 77.5; BLEU gets 75.3 and ROUGE-L 72.0, so only the R1 judge beats the n-gram metrics. No correlation coefficient, confidence interval or inter-annotator agreement is given. The paper's main results use the lower-aligned GPT-4o judge. The check is at the whole-score level with 7 models, not per rubric dimension such as Character Fidelity.
Curator qualifications: Pairwise alignment with human preferences: only the DeepSeek-R1 judge beats BLEU; no correlation coefficient, interval or inter-annotator agreement. Claim that main results use the GPT-4o judge is producer reading (not re-checked).
Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.
Structured fields
{
"citations": [
{
"url": "https://arxiv.org/html/2502.09082v3",
"note": "CoSER v3 Table 5"
}
]
}Evidence claims
- [sourced_fact] Yes, partly. Table 5 reports 'alignment' = percentage agreement with the human raters on which of two models is better, ties dropped. GPT-4o as judge gives 68.6, DeepSeek-V3 65.1 and DeepSeek-R1 77.5; BLEU gets 75.3 and ROUGE-L 72.0, so only the R1 judge beats the n-gram metrics. No correlation coefficient, confidence interval or inter-annotator agreement is given. The paper's main results use the lower-aligned GPT-4o judge. The check is at the whole-score level with 7 models, not per rubric dimension such as Character Fidelity. · locator: CoSER v3 Table 5 source
Rights
| component | license | hosting | redistribution | modification | training | Evidence and notes |
|---|---|---|---|---|---|---|
| documentation | unknown | permitted | permitted | unknown | unknown | Applies to this original summary only, not to the cited sources. |