A Contrastive Framework for Neural Text Generation (2202.06417v3)

Abstract: Text generation is of great importance to many natural language processing applications. However, maximization-based decoding methods (e.g. beam search) of neural LLMs often lead to degenerate solutions -- the generated text is unnatural and contains undesirable repetitions. Existing approaches introduce stochasticity via sampling or modify training objectives to decrease probabilities of certain tokens (e.g., unlikelihood training). However, they often lead to solutions that lack coherence. In this work, we show that an underlying reason for model degeneration is the anisotropic distribution of token representations. We present a contrastive solution: (i) SimCTG, a contrastive training objective to calibrate the model's representation space, and (ii) a decoding method -- contrastive search -- to encourage diversity while maintaining coherence in the generated text. Extensive experiments and analyses on three benchmarks from two languages demonstrate that our proposed approach significantly outperforms current state-of-the-art text generation methods as evaluated by both human and automatic metrics.

• The paper presents a contrastive training objective, SimCTG, that recalibrates token representations to reduce degeneration in text generation.
• It introduces contrastive search, a decoding method that blends deterministic and probabilistic strategies to ensure outputs are both coherent and diverse.
• Experiments reveal improvements in perplexity, prediction accuracy, and text quality, demonstrating the method’s potential to enhance natural language generation.

A Contrastive Framework for Neural Text Generation

The paper "A Contrastive Framework for Neural Text Generation" addresses the well-recognized issue of degeneration in neural text generation models like GPT-2. Models often produce outputs that are repetitive and lack natural diversity when decoded via traditional maximization strategies such as beam search. Existing methods attempt to mitigate this by introducing stochastic sampling approaches or alternative training objectives. However, these methods can compromise text coherence, leaving room for improvement.

Core Contributions

This paper introduces a novel dual-faceted approach to tackle the text degeneration problem:

1. SimCTG - Contrastive Training Objective: The researchers pinpoint the anisotropic distribution of token representations as a core issue leading to degeneration. To remedy this, they propose SimCTG, a contrastive training objective that calibrates the model's representation space to be more discriminative and isotropic. This adjustment aims to prevent the tight clustering that often leads to repeated text generation.
2. Contrastive Search - Decoding Method: Aimed at balancing coherence and diversity, this new decoding strategy incorporates elements of both deterministic and probabilistic methods. Specifically, it encourages diversity through a degeneration penalty while maintaining semantic closeness to the prompt by selecting amongst the top-k predictions. This mechanism aims to avoid the pitfalls of semantically inconsistent generation often observed in stochastic methods.

Evaluation

The authors conducted comprehensive experiments across multiple benchmarks to validate their approach. The results present a significant performance improvement over existing state-of-the-art methods. Below are some highlighted metrics:

• LLMling Quality: SimCTG shows an improvement in perplexity (23.82) and prediction accuracy (40.91%) on the Wikitext-103 dataset, indicating a better capability in modeling natural language over traditional MLE (24.32 and 39.63%, respectively) and unlikelihood training (28.57 and 38.41%).
• Generation Quality: In terms of generation diversity and semantically consistent coherence metrics, SimCTG paired with contrastive search consistently outperforms other models. Notably, it achieves a higher MAUVE score of 0.94, suggesting that the generated text is closer to human-written text distributions.

Implications and Future Directions

This work advances the field by providing an alternative pathway to balance coherence and diversity in neural text generation. The proposed contrastive training and search methods open up new possibilities in enhancing model representations and decoding processes without significant computational overhead.

The practical implications include improvements not only in generating more natural and varied text but also in informing training and decoding practices across other domains, such as machine translation and dialogue generation. Future research could further explore integrating these approaches in broader applications, larger models, and other language contexts to validate scalability and robustness.

Overall, the application of contrastive learning to text generation, as proposed in this paper, provides a feasible approach to address fundamental issues inherent in current models and offers a promising direction for further exploration and refinement.