FATURA: A Multi-Layout Invoice Image Dataset for Document Analysis and Understanding (2311.11856v1)

Abstract: Document analysis and understanding models often require extensive annotated data to be trained. However, various document-related tasks extend beyond mere text transcription, requiring both textual content and precise bounding-box annotations to identify different document elements. Collecting such data becomes particularly challenging, especially in the context of invoices, where privacy concerns add an additional layer of complexity. In this paper, we introduce FATURA, a pivotal resource for researchers in the field of document analysis and understanding. FATURA is a highly diverse dataset featuring multi-layout, annotated invoice document images. Comprising $10,000$ invoices with $50$ distinct layouts, it represents the largest openly accessible image dataset of invoice documents known to date. We also provide comprehensive benchmarks for various document analysis and understanding tasks and conduct experiments under diverse training and evaluation scenarios. The dataset is freely accessible at https://zenodo.org/record/8261508, empowering researchers to advance the field of document analysis and understanding.

• The paper presents FATURA, a new dataset with 10,000 invoice images featuring 50 distinct layouts for enhanced document analysis.
• It details a comprehensive methodology including multi-format annotations (COCO, LayoutLMv3, and standard) and evaluation strategies across intra- and inter-template settings.
• Experiments with visual (YOLOS), multi-modal (LayoutLMv3), and hybrid approaches demonstrate improved structured data extraction despite OCR and segmentation challenges.

"FATURA: A Multi-Layout Invoice Image Dataset for Document Analysis and Understanding" (2311.11856)

Overview

The paper introduces FATURA, a comprehensive dataset designed as a resource for document analysis and understanding, particularly for invoices. The dataset includes 10,000 invoices with 50 unique layouts, making it the most extensive open invoice image dataset available. Key to document analysis is the need for annotated data that encompasses both text transcription and precise bounding-box annotations, which FATURA provides freely for researchers. Additionally, the paper discusses the development of benchmarks and evaluates various strategies for document understanding tasks utilizing the dataset.

Dataset Construction and Features

FATURA is structured to address the challenges associated with analyzing invoices, such as varying formats and privacy concerns. The dataset generation process involves several steps:

• Invoice Template Collection: Real invoice images serve as templates, and their layouts are annotated using VGG Image Annotator, excluding textual data at this stage for privacy reasons.
• Logo and Text Generation: Unique logos are created using a pre-trained Text-to-Image Latent Diffusion model, while textual content for invoice fields is generated randomly to mimic real-world diversity.
• Data Diversity: The dataset includes 50 distinct templates with different font styles, placements, and graphical elements, enhancing its applicability across various industries.
• Annotation Formats: The dataset provides annotations in COCO format, a version integrated for use with the LayoutLMv3 architecture, and a standard format.

  Figure 1: Examples of annotated images from different templates.

  

  Figure 2: Class occurrence distribution in the FATURA Dataset.

Evaluation Strategies

The paper outlines two evaluation strategies to test model performance using FATURA:

1. Intra-Template Evaluation: Models are trained and tested on images from the same templates, exposing them to multiple content variations but familiar layouts.
2. Inter-Template Evaluation: Models are trained on certain templates and tested on entirely different ones, evaluating their ability to generalize across various layouts.

Experiments and Results

Several approaches were tested on FATURA:

Visual-Based Approach with YOLOS

YOLOS was used for object detection, showing high performance at recognizing text regions within familiar templates, but struggling with unseen templates due to its focus on layout rather than textual comprehension.

• Intra-Template Success: Demonstrated by high mAP scores on familiar templates.
• Inter-Template Challenge: Performance dropped significantly when models were trained and tested on dissimilar templates.

Multi-Modal Approach with LayoutLMv3

LayoutLMv3 leverages both visual and textual information for a token classification task.

• Employed at the region level to mitigate reliance on precise word-level bounding boxes and OCR errors.
• Exhibited robust classification capabilities across various document structures.

Hybrid Approach

A novel combination of YOLOS and LayoutLMv3 for improved document understanding.

• Strengths: This approach effectively combines visual and textual cues, outperforming purely visual methods in specific field extractions.
• Weaknesses: Suffered from OCR inaccuracies affecting token-level classification.

  Figure 3: Comparison between ground-truth (left), YOLOS predictions (center), and hybrid approach predictions (right).

Conclusion

The FATURA dataset addresses existing gaps in document analysis, providing high-quality, diverse invoice data essential for advancing AI models in this domain. The various evaluation strategies and hybrid approaches highlight both the potential and limitations of current models, informing future research directions.

Ongoing efforts to expand this dataset to include multi-lingual invoices will further broaden its utility, presenting opportunities to refine models capable of handling diverse document types across languages. Lastly, while the hybrid approach presents notable improvements, the importance of high-precision OCR and segmentation remains a critical focus for future studies.