On the State of the Art of Evaluation in Neural Language Models (1707.05589v2)

Abstract: Ongoing innovations in recurrent neural network architectures have provided a steady influx of apparently state-of-the-art results on LLMling benchmarks. However, these have been evaluated using differing code bases and limited computational resources, which represent uncontrolled sources of experimental variation. We reevaluate several popular architectures and regularisation methods with large-scale automatic black-box hyperparameter tuning and arrive at the somewhat surprising conclusion that standard LSTM architectures, when properly regularised, outperform more recent models. We establish a new state of the art on the Penn Treebank and Wikitext-2 corpora, as well as strong baselines on the Hutter Prize dataset.

• The paper reevaluates neural language model performance by using large-scale, automated hyperparameter tuning to reveal that standard LSTMs can outperform more complex architectures.
• The study demonstrates that performance variability stems largely from hyperparameter dependence rather than inherent model differences, highlighting replication challenges.
• It emphasizes that rigorous, controlled evaluation methods are essential for reliable benchmarking and practical improvements in AI research.

On the State of the Art of Evaluation in Neural LLMs

The paper "On the State of the Art of Evaluation in Neural LLMs" investigates the evaluation methodologies used for recurrent neural network architectures in the context of LLMing. It identifies significant variability in empirical results due to differing experimental setups, particularly in hyperparameter dependencies. The authors address this by utilizing a large-scale, automated black-box hyperparameter tuning process to reassess several prominent architectures and regularization strategies.

Key Findings

1. Evaluation Reappraisal: The paper revises the performance of neural architectures, specifically LSTMs, Recurrent Highway Networks (RHNs), and architectures derived via Neural Architecture Search (NAS). Contrary to prevalent claims, the standard LSTMs, when meticulously regularized, perform competitively, if not superiorly, to more recent models on benchmarks like the Penn Treebank and Wikitext-2 corpora. This challenges previous state-of-the-art assertions.
2. Replication and Hyperparameters: The paper highlights the risks of replication failures stemming from inadequate control over hyperparameters. It aligns with previous findings indicating that emphasized performance differences often arise from variations in hyperparameter settings rather than intrinsic architectural merits.
3. Model Comparisons: Using parameterized model families and adjustable regularization and learning hyperparameters, various models are compared. The LSTM variants, particularly in parameter-efficient configurations, consistently demonstrate superior performance, with a 24M parameter LSTM achieving a test perplexity of 4.065 on the Penn Treebank dataset.
4. Role of Hyperparameter Optimization: Employing a black-box hyperparameter tuner, Google Vizier, the research underscores the necessity of automated hyperparameter optimization to achieve robust results. The paper reports that controlled evaluation is computationally costly but crucial.
5. Analysis of Experimental Variables: The paper identifies down-projection, dropout techniques, and shared embeddings as impactful features, contributing variably across datasets.

Implications and Future Directions

The findings are significant for both theoretical and practical applications in AI. They suggest that the perceived advantage of novel architectures may often be attributed to experimental conditions rather than structural innovation. This calls for a paradigm shift in model evaluation practices, advocating for standardized methodologies that balance thoroughness with computational feasibility.

The results imply that improved model evaluation methodologies could lead to:

• Enhanced reproducibility and reliability of AI research.
• A better understanding of model architecture utility and real-world performance implications.
• Development of resource-efficient evaluation frameworks that can reduce experimental noise and overfitting to particular hyperparameter setups.

Future work may focus on streamlining model evaluation processes by reducing hyperparameter sensitivity, improving hyperparameter optimization techniques, and establishing computational leagues that ensure equitable conditions for model assessment. This would better allocate computational resources while maintaining or improving the rigor of architectural evaluations.

In conclusion, while the paper highlights significant methodological issues in LLM evaluation, it also provides a clear path forward for enhancing the reliability of AI research outcomes. The emphasis on hyperparameter control as a key variable in empirical outcomes presents a valuable framework for both current and future research endeavors.