ChroniclingAmericaQA: A Large-scale Question Answering Dataset based on Historical American Newspaper Pages

ChroniclingAmericaQA: Advancing Question Answering Research with Historical Newspaper Collections

Introduction to ChroniclingAmericaQA

The field of Question Answering (QA) has experienced notable advances owing to the advent of deep learning and, more specifically, the development of LLMs. However, a prevalent limitation within current QA research is its focus on modern textual data, overlooking the rich tapestry of historical documents available. The ChroniclingAmericaQA dataset seeks to address this gap by leveraging the Chronicling America historical newspaper collection, resulting in a novel QA dataset comprising 485K question-answer pairs derived from documents spanning 120 years (1800-1920). This dataset not only expands the temporal scope of QA research but also introduces the challenge of working with noisy OCR text, a common issue in digital historical document collections.

Dataset Construction and Challenges

Creating the ChroniclingAmericaQA dataset involved a meticulous process to convert the historical newspapers into a format suitable for QA research. Key challenges included the handling of noisy OCR text, which often hampers the extraction of accurate information. To mitigate this, a hybrid approach was adopted that involved both raw and corrected OCR text, allowing for comprehensive model testing. Moreover, by incorporating scanned images of the newspaper pages, the dataset promotes research into multimodal QA systems capable of interpreting both textual and visual data.

The dataset's construction process can be summarized into three critical steps:

1. Data Collection: A diverse selection of newspaper pages was curated from the Chronicling America project, ensuring a wide geographic and temporal coverage.
2. Data Preparation: The OCR text underwent a crucial correction phase employing GPT 3.5 Turbo, enhancing the text's quality for better question generation.
3. Question Generation Module: Utilizing the T5-base model, question-answer pairs were generated, highlighting the ability of generative models to produce coherent and relevant QA pairs even from complex historical texts.

Dataset Characteristics

ChroniclingAmericaQA distinguishes itself by its longitudinal coverage and the inclusion of noisy OCR text scenarios, offering a unique resource for QA research. It stands as the most extended QA dataset of its kind, spanning over a century of content. This breadth not only introduces the challenge of language evolution over time but also tests a model's ability to discern information amidst the inherent inaccuracies of historical OCR text.

Evaluation and Insights

Evaluation of ChroniclingAmericaQA involved testing with various models including, but not limited to BERT, RoBERTa, and T5, alongside the emerging LLMs like LLaMA2 and Mistral. Highlighting a few key insights:

• Performance Degradation with Noisy OCR: There's a noticeable performance drop when models are tested with raw OCR text compared to corrected text, underscoring the importance of text quality in historical QA tasks.
• Model Adaptability: Models fine-tuned on both the ChroniclingAmericaQA and other QA datasets like SQuAD demonstrated superior performance, suggesting the benefit of a diverse training regimen that includes both modern and historical texts.
• Value of LLMs: Advanced models like LLaMA2 showcased remarkable resilience against the challenges posed by the dataset, indicating the potential of LLMs in historical document QA research.

Practical Implications and Future Directions

The introduction of ChroniclingAmericaQA paves the way for a new direction in QA research, emphasizing the untapped potential of historical document collections. Beyond academia, this dataset has practical applications in digital humanities, archival science, and education, facilitating access and understanding of historical documents through advanced QA systems.

Future endeavors may extend the ChroniclingAmericaQA framework to other historical document collections, further enriching the resources available for QA research. Moreover, tackling the challenge of bias and ethical considerations in historical texts through advanced model training presents a crucial area for further investigation.

Conclusion

In summary, the ChroniclingAmericaQA dataset marks a significant step forward in the quest to extend QA and MRC tasks to historical documents. By bridging the gap between modern textual analyses and the rich informational content of historical archives, it lays the groundwork for a more inclusive and comprehensive approach to QA research.