Benchmarking LLMs for Environmental Review and Permitting (2407.07321v3)

Abstract: The National Environment Policy Act (NEPA) stands as a foundational piece of environmental legislation in the United States, requiring federal agencies to consider the environmental impacts of their proposed actions. The primary mechanism for achieving this is through the preparation of Environmental Assessments (EAs) and, for significant impacts, comprehensive Environmental Impact Statements (EIS). LLMs' effectiveness in specialized domains like NEPA remains untested for adoption in federal decision-making processes. To address this gap, we present NEPA Question and Answering Dataset (NEPAQuAD), the first comprehensive benchmark derived from EIS documents, along with a modular and transparent evaluation pipeline, MAPLE, to assess LLM performance on NEPA-focused regulatory reasoning tasks. Our benchmark leverages actual EIS documents to create diverse question types, ranging from factual to complex problem-solving ones. We built a modular and transparent evaluation pipeline to test both closed- and open-source models in zero-shot or context-driven QA benchmarks. We evaluate five state-of-the-art LLMs using our framework to assess both their prior knowledge and their ability to process NEPA-specific information. The experimental results reveal that all the models consistently achieve their highest performance when provided with the gold passage as context. While comparing the other context-driven approaches for each model, Retrieval Augmented Generation (RAG)-based approaches substantially outperform PDF document contexts, indicating that neither model is well suited for long-context question-answering tasks. Our analysis suggests that NEPA-focused regulatory reasoning tasks pose a significant challenge for LLMs, particularly in terms of understanding the complex semantics and effectively processing the lengthy regulatory documents.

• The paper demonstrates that Retrieval-Augmented Generation (RAG) significantly improves answer accuracy in environmental review tasks.
• It introduces the NEPAQuAD1.0 benchmark comprising 1,599 expertly validated Q&A pairs from NEPA documents.
• Comparative evaluation reveals that LLM performance varies by context type, highlighting strengths in no-context, full PDF, RAG, and gold passage setups.

Examination of LLMs for Environmental Review Document Comprehension

The paper "RAG vs. Long Context: Examining Frontier LLMs for Environmental Review Document Comprehension" presents an evaluation of various state-of-the-art LLMs—Claude Sonnet, Gemini, and GPT-4—in a domain-specific task centered on Environmental Impact Statements (EIS) under the National Environmental Policy Act (NEPA). The paper aims to investigate these models' capabilities in understanding and answering questions derived from lengthy NEPA documents, emphasizing the nuances of legal, technical, and compliance-related information.

Benchmark and Methodology

To facilitate the evaluation, the authors introduce the NEPAQuAD1.0 benchmark, designed specifically for assessing LLMs on NEPA documents. Building this benchmark involved a multi-step process: selecting pertinent excerpts from EIS documents, identifying relevant question types, using GPT-4 to generate question-answer pairs, and validating these pairs through NEPA experts.

Key Contributions

1. Creation of NEPAQuAD1.0:
   • The benchmark is generated using a semi-supervised method, leveraging GPT-4 to produce contextual questions and answers from selected EIS document excerpts. The final dataset comprises 1,599 question-answer pairs validated by NEPA experts.
2. Comparative Evaluation:
   • The paper compares the performance of LLMs in different contextual settings: no context, full PDF documents, silver passages (using RAG), and gold passages.
   • The evaluation involves metrics that focus on both factual and semantic correctness, driven by the RAGAs score.

Performance Analysis

Contextual Influence

The results illustrate that Retrieval-Augmented Generation (RAG) models, which leverage relevant passages from the documents, significantly outperform long context LLMs that process entire documents. The RAG setup enhances answer accuracy, demonstrating the importance of retrieving pertinent information over processing extensive, potentially noisy document contexts.

• No Context: Gemini leads in performance when no additional context is provided, indicating a strong baseline prior knowledge.
• Full PDF: Surprisingly, providing full PDF context did not yield the expected performance improvements for Gemini, whereas GPT-4 performed better with this setup.
• RAG Context: Here, Claude excels, suggesting that passage retrieval effectively supports accurate answer generation across models.
• Gold Passage: When provided with highly relevant passages (gold data), models, including Claude and GPT-4, show optimal performance.

Question Type Specifics

The analysis reveals that the models struggle with more complex and divergent questions:

• Closed Questions: These are answered most accurately, particularly when using RAG or gold passages.
• Problem-solving and Divergent Questions: Performance is notably lower, especially without context-specific support.

Positional Knowledge

Position of the context within documents also plays a role. Models perform better on earlier document sections, although problem-solving questions exhibit better results sourced from latter document parts. This suggests potential limitations in current LLM architectures in maintaining attention over long sequences.

Implications and Future Directions

This research reveals several implications:

1. Retrieval over Long Context: The observed advantage of RAG highlights the potential for hybrid models combining retrieval techniques with generative LLMs to handle domain-specific, lengthy documents efficiently.
2. Context Sensitivity: Understanding the context type and content relevance is critical for model performance, as demonstrated by varied results across different context scenarios and question types.
3. Need for Enhanced Reasoning: Addressing the models' difficulties with complex questions necessitates further work. Future research might explore advanced reranking techniques or adaptive retrieval mechanisms that cater to different question complexities and types.

Conclusion

The paper underscores the challenges and opportunities presented by domain-specific LLM applications. Key findings advocate for the adoption of RAG methodologies, emphasizing their superiority in generating accurate responses in niche areas like environmental review documents. Despite the current LLMs' limitations, particularly in handling extensive and complex contexts, this research sets a significant precedent for future improvements in LLMs tailored to specialized domains. The introduction of NEPAQuAD1.0 serves as a valuable resource for rigorously evaluating these models, laying a foundation for continued advancements in the field.