Gemma: Open Models Based on Gemini Research and Technology (2403.08295v4)

Abstract: This work introduces Gemma, a family of lightweight, state-of-the art open models built from the research and technology used to create Gemini models. Gemma models demonstrate strong performance across academic benchmarks for language understanding, reasoning, and safety. We release two sizes of models (2 billion and 7 billion parameters), and provide both pretrained and fine-tuned checkpoints. Gemma outperforms similarly sized open models on 11 out of 18 text-based tasks, and we present comprehensive evaluations of safety and responsibility aspects of the models, alongside a detailed description of model development. We believe the responsible release of LLMs is critical for improving the safety of frontier models, and for enabling the next wave of LLM innovations.

• The paper introduces the Gemma family of open-source 2B and 7B language models built on Gemini technology, demonstrating strong NLP capabilities.
• The methodology includes a transformer decoder enhanced by multi-query attention, RoPE embeddings, and GeGLU activations, trained on trillions of tokens.
• Evaluation results show Gemma 7B achieving superior scores on MMLU, GSM8K, and safety benchmarks while maintaining low memorization rates.

Gemma: An Open Model Based on Gemini Technology

The paper "Gemma: Open Models Based on Gemini Research and Technology" (2403.08295) introduces Gemma, a family of open-source LLMs derived from the same research and technology used to develop the Gemini models. The Gemma family includes two sizes: 2B and 7B parameter models, with both pretrained and fine-tuned versions available. These models exhibit strong performance across a range of NLP tasks, including language understanding, reasoning, and safety.

Model Architecture and Training

The Gemma models are based on the transformer decoder architecture [DBLP:journals/corr/VaswaniSPUJGKP17] and incorporate several improvements, including multi-query attention (MQA) [mqa] in the 2B model, RoPE embeddings [rope], GeGLU activations [geglu], and RMSNorm [rmsnorm]. The models were trained on 3T (2B) and 6T (7B) tokens of primarily English data, using TPUv5e infrastructure. The training process leverages JAX [bradburyJAX] and Pathways [barham2022pathways] for distributed training and optimization.

Pretraining and Instruction Tuning

The pretraining data was filtered to remove unwanted or unsafe utterances and to minimize the risk of memorization. Instruction tuning was performed using supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) [christiano2017deep, ouyang2022training]. The SFT data mixtures were selected based on LM-based side-by-side evaluations [zheng2023judging], and the RLHF reward model was trained on English-only preference data.

Evaluation Results

The Gemma models were evaluated across a range of benchmarks, including automated benchmarks and human preference evaluations. The results indicate that Gemma models outperform similarly sized open models on a variety of tasks. For example, the Gemma 7B model achieves 64.3% on MMLU [mmlu], surpassing other open-source alternatives. In human evaluations, Gemma 7B IT demonstrates a 61.2% win rate against Mistral v0.2 7B Instruct on instruction following and a 63.5% win rate on safety.

Figure 1: Language understanding and generation performance of Gemma 7B across different capabilities compared to similarly sized open models, showcasing strong performance across a range of NLP tasks.

The models also exhibit strong performance on mathematics and coding benchmarks, outperforming other models by at least 10 points on GSM8K [gsm8k] and MATH [hendrycksmath2021].

Memorization Analysis

The paper includes an analysis of memorization in the Gemma pretrained models, using the methodology described in [anil2023palm]. The results show that Gemma models have low rates of memorization, comparable to PaLM [chowdhery2022palm] and PaLM 2 [anil2023palm]. Furthermore, the models do not appear to memorize sensitive data.

Figure 2: Comparing average memorization rates across model families, demonstrating that Gemma models have similarly low rates of memorization compared to PaLM and PaLM 2 models of comparable size.

Figure 3: Measuring personal and sensitive data memorization rates, showing no cases of memorized sensitive data.

Figure 4: Comparing exact and approximate memorization, finding that roughly 50% more data is approximately memorized compared to exact memorization.

Responsible Deployment

The paper outlines a structured approach to responsible development and deployment of the Gemma models, including filtering and measuring biases in pre-training data, assessing safety through standardized AI safety benchmarks, and internal red teaming. The authors acknowledge the risks associated with malicious uses of LLMs but argue that the benefits of open access to these models outweigh the risks.

Conclusion

The Gemma models represent a significant advancement in open-source LLMs, offering strong performance, safety, and responsible development practices. The release of these models is intended to encourage further AI safety research, community innovation, and the development of beneficial applications in various domains. While limitations exist, the authors express confidence that Gemma models will provide a net benefit to the AI community.