FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction
(2403.02270)Abstract
Recent advancements in text summarization, particularly with the advent of LLMs, have shown remarkable performance. However, a notable challenge persists as a substantial number of automatically-generated summaries exhibit factual inconsistencies, such as hallucinations. In response to this issue, various approaches for the evaluation of consistency for summarization have emerged. Yet, these newly-introduced metrics face several limitations, including lack of interpretability, focus on short document summaries (e.g., news articles), and computational impracticality, especially for LLM-based metrics. To address these shortcomings, we propose Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction (FENICE), a more interpretable and efficient factuality-oriented metric. FENICE leverages an NLI-based alignment between information in the source document and a set of atomic facts, referred to as claims, extracted from the summary. Our metric sets a new state of the art on AGGREFACT, the de-facto benchmark for factuality evaluation. Moreover, we extend our evaluation to a more challenging setting by conducting a human annotation process of long-form summarization.
Overview
-
The paper provides detailed guidelines for authors submitting manuscripts to *ACL conferences using the LaTeX document preparation system.
-
It emphasizes the importance of uniform formatting, referencing, and document structuring to adhere to *ACL proceedings standards.
-
Instructions include using provided style files (
acl.sty) and template (acl.tex), formatting for non-Latin scripts with XeLaTeX, and handling specialized elements like footnotes, tables, figures, and citations. -
The guidelines aim to streamline the review process, enhance readability, and facilitate research dissemination within the computational linguistics community.
Guidelines for Submitting Papers to *ACL Conferences Using LaTeX
Introduction to the Guidelines
The paper provides a comprehensive set of instructions for authors intending to submit their manuscripts to *ACL conferences using the LaTeX document preparation system. It details the essential aspects of formatting, referencing, and structuring documents in accordance with the specified requirements for *ACL proceedings. The document serves both as a guideline and a template that authors can directly employ to prepare their manuscripts, ensuring uniformity and adherence to the publication standards set by the Association for Computational Linguistics (ACL).
LaTeX Template and Styling Details
A major focus of the document is on the LaTeX style files (acl.sty) and template (acl.tex) provided for author use. These files are designed to simplify the process of formatting by automatically applying the correct styles to various elements of the manuscript, including the title, author list, abstract, main text, references, and appendices. The authors are directed to use the PDFLaTeX engine for generating PDF files due to its wide support and compatibility with the typesetting requirements of *ACL conferences. Additionally, the document mentions the suitability of XeLaTeX for manuscripts that contain non-Latin scripts, thus accommodating a broader range of linguistics research.
Structuring the Manuscript
The guidelines outline the structure that manuscripts should follow, beginning with the declaration of the document class and the loading of the acl style file with or without the review option, depending on the stage of submission. Authors are instructed on the formatting of the document's preamble, including the specification of the title and author information. The document emphasizes the importance of adhering to the default font (Times Roman) and the prohibition against modifying default caption sizes for tables and figures.
Handling Special Elements
Several sections of the guidelines are devoted to the correct handling of specialized elements within a manuscript:
- Footnotes and Hyperlinks: Instructions on creating footnotes and addressing common compilation errors with hyperlinks in PDFLaTeX.
- Tables and Figures: Detailed formatting rules for tables and figures, including an example table that demonstrates how to correctly input accented characters.
- Citations and References: Guidance on using the Natbib package for citations within the text and the formatting of the references section. Authors are encouraged to include DOIs or URLs for referenced works to enhance the accessibility and discoverability of cited sources.
Appendices and Acknowledgements
The document provides information on including appendices by switching the section numbering to letters and crediting contributions in the acknowledgements section. It also reflects on the collaborative effort in evolving the manuscript guidelines across various ACL-related conferences, acknowledging contributions from several individuals over the years.
Conclusion and Implications for Future Submissions
The guidelines presented in the document are critical for authors submitting to *ACL conferences, ensuring that all manuscripts maintain a consistent format and meet the required standards. Looking ahead, adherence to these guidelines will simplify the review process, enhance the readability of conference proceedings, and facilitate the dissemination of research findings within the computational linguistics community. As the field continues to evolve, it may be anticipated that these guidelines will be periodically updated to reflect new typesetting technologies, changing publication practices, and the growing diversity of research topics in computational linguistics.
Create an account to read this summary for free: