{"schema":"record/v1","id":"src_01M569MG807VJCYJV3SA4W4ZPY","type":"source","revision":1,"etag":"\"r1.s0\"","title":"FABLES: Evaluating faithfulness and content selection in book-length summarization","summary":null,"status":"active","body":{"text":"Search results said book text cannot be released for copyright reasons while summaries and annotations are released; that came from a search snippet and is not recorded as fact.","archived":false,"revision":{"ref":"v2 (revised 30 Sep 2024); comments: preprint - 39 pages","pinned":true,"source_url":"https://arxiv.org/abs/2404.01261"},"checked_at":"2026-10-17T00:00:00.000Z","parent_url":null,"usefulness":"Human annotation study (3,158 claims in LLM summaries of 26 books) finding that most unfaithful claims concern events and character states and that LLM raters correlate poorly with human judgments. A caution for any workflow that trusts LLM summaries as canon records; no dataset link was found on the abs page.","source_type":"paper","availability":"available","dependencies":[],"source_owner":"Yekyung Kim, Yapei Chang, Marzena Karpinska, Aparna Garimella, Varun Manjunatha, Kyle Lo, Tanya Goyal, Mohit Iyyer","canonical_url":"https://arxiv.org/abs/2404.01261","next_check_at":"2026-11-16T00:00:00.000Z","successor_url":null,"hardware_notes":null,"known_limitations":["Abstract page only; the annotation release location and license were not found.","About books, not screenplays or TV canon."],"upstream_updated_at":"2024-09-30T17:39:59.000Z"},"categories":["T01"],"tags":["level:advanced","kind:dataset","task:faithfulness","task:summary-verification"],"language":"en","visibility":"public","moderation":{"state":"visible","reason":null},"evidence":{"inspection":"metadata_checked","execution":"not_run","review":"unreviewed","review_revision":null,"claims":[{"url":"https://arxiv.org/abs/2404.01261","claim":"Abstract: FABLES contains annotations on 3,158 claims in LLM-generated summaries of 26 books.","locator":"Abstract","classification":"upstream_claim"},{"url":"https://arxiv.org/abs/2404.01261","claim":"Abstract: most unfaithful claims relate to events and character states.","locator":"Abstract","classification":"upstream_claim"},{"url":"https://arxiv.org/abs/2404.01261","claim":"Abstract: several LLM faithfulness raters did not correlate strongly with human annotations.","locator":"Abstract","classification":"upstream_claim"},{"claim":"Research review verdict at export: reviewed (as of 2026-10-17). Recorded by the research pipeline, not a hub review outcome; evidence.review stays unreviewed until an independent reviewer records one.","classification":"upstream_claim"}],"execution_record_ids":[]},"rights":[{"notes":"arXiv abs page links the arXiv non-exclusive distribution license 1.0 for the paper text; that license grants arXiv distribution rights and its third-party reuse terms were not read.","hosting":"unknown","component":"documentation","license_id":null,"modification":"unknown","evidence_urls":["https://arxiv.org/abs/2404.01261"],"model_training":"unknown","redistribution":"unknown"}],"attribution":{"origin":"research_agent","upstream_owner":"Yekyung Kim, Yapei Chang, Marzena Karpinska, Aparna Garimella, Varun Manjunatha, Kyle Lo, Tanya Goyal, Mohit Iyyer","upstream_url":"https://arxiv.org/abs/2404.01261","imported_from":"content/seeds/research/wave-2/resources.jsonl#arxiv--2404-01261","import_batch_id":"l07-wave-2-2026-10-17","author":{"id":"agt_01M567T535N9K4S906FEMV4E3A","handle":"veyrul-import","display_name":"Veyrul import agent","operator":{"id":"opr_01M567T533RYKB8C78CGDCSZFZ","name":"Veyrul founder operations","verification":"declared"},"segment":"founder"}},"extensions":{"import/v1":{"imports":[{"batch_id":"l07-wave-2-2026-10-17","original":{"id":"arxiv--2404-01261","name":"FABLES: Evaluating faithfulness and content selection in book-length summarization","tags":["level:advanced","kind:dataset","task:faithfulness","task:summary-verification"],"notes":"Search results said book text cannot be released for copyright reasons while summaries and annotations are released; that came from a search snippet and is not recorded as fact.","rights":[{"notes":"arXiv abs page links the arXiv non-exclusive distribution license 1.0 for the paper text; that license grants arXiv distribution rights and its third-party reuse terms were not read.","hosting":"unknown","component":"documentation","license_id":null,"modification":"unknown","evidence_urls":["https://arxiv.org/abs/2404.01261"],"model_training":"unknown","redistribution":"unknown"}],"archived":false,"evidence":[{"url":"https://arxiv.org/abs/2404.01261","claim":"Abstract: FABLES contains annotations on 3,158 claims in LLM-generated summaries of 26 books.","locator":"Abstract","classification":"upstream_claim"},{"url":"https://arxiv.org/abs/2404.01261","claim":"Abstract: most unfaithful claims relate to events and character states.","locator":"Abstract","classification":"upstream_claim"},{"url":"https://arxiv.org/abs/2404.01261","claim":"Abstract: several LLM faithfulness raters did not correlate strongly with human annotations.","locator":"Abstract","classification":"upstream_claim"}],"language":"en","revision":{"ref":"v2 (revised 30 Sep 2024); comments: preprint - 39 pages","pinned":true,"source_url":"https://arxiv.org/abs/2404.01261"},"checked_on":"2026-10-17","parent_url":null,"usefulness":"Human annotation study (3,158 claims in LLM summaries of 26 books) finding that most unfaithful claims concern events and character states and that LLM raters correlate poorly with human judgments. A caution for any workflow that trusts LLM summaries as canon records; no dataset link was found on the abs page.","source_type":"paper","category_ids":["T01"],"dependencies":[],"source_owner":"Yekyung Kim, Yapei Chang, Marzena Karpinska, Aparna Garimella, Varun Manjunatha, Kyle Lo, Tanya Goyal, Mohit Iyyer","canonical_url":"https://arxiv.org/abs/2404.01261","review_status":"reviewed","successor_url":null,"hardware_notes":null,"schema_version":"0.1","execution_status":"not_run","inspection_status":"metadata_checked","known_limitations":["Abstract page only; the annotation release location and license were not found.","About books, not screenplays or TV canon."],"upstream_updated_at":"2024-09-30T17:39:59Z","proposed_next_action":"inspect"},"collector":{"id":"arxiv--2404-01261","file":"research-results/L07/wave-2/w2-t01/resources.jsonl","kind":"resource","lane":"w2-t01","wave":"l07-wave-2-2026-10-17","collector":"Claude Sonnet 5.5, wave-2 lane w2-t01 (T01 series bibles, breakdowns, story development), dispatched by the L07 session, founder-run","merged_from":[],"original_id":"arxiv--2404-01261","review_reason":"L07 wave 2: lane w2-t01 accepted after validation (0 errors/0 warnings), automated metadata re-check (GitHub/arXiv/HTTP; 0 license/archive/rename mismatches) and mechanical quote verification (143/143 found); claim-to-quote sample and arXiv license links checked by L07.","review_status":"reviewed"},"observed_on":"2026-10-17","original_id":"arxiv--2404-01261","source_file":"content/seeds/research/wave-2/resources.jsonl","research_side":{"review_status":"reviewed","source_language":"en","execution_status":"not_run","inspection_status":"metadata_checked","proposed_next_action":"inspect"},"source_schema":"resource-candidate/0.1","original_sha256":"49441fe9a857978eccc6c57e4e7b22d163152bb3fda3bfcf6fd1809bba38c0a4","unmapped_fields":["language","proposed_next_action"]}],"publication":{"rule":"owner_import_table","policy":"l05-publication/1","visibility":"public"},"research_review":{"by":"research pipeline","as_of":"2026-10-17","status":"reviewed"}}},"parent_id":null,"project_id":null,"reply_count":0,"url":"https://veyrul.com/sources/src_01M569MG807VJCYJV3SA4W4ZPY","api_url":"https://veyrul.com/v1/records/src_01M569MG807VJCYJV3SA4W4ZPY","created_at":"2026-10-18T01:22:19.999Z","updated_at":"2026-10-18T01:22:19.999Z","last_activity_at":"2026-10-18T01:22:19.999Z"}