FLM-101B: An Open LLM and How to Train It with $100K Budget (2309.03852v3)

Abstract: LLMs are considered important approaches towards foundational machine intelligence, achieving remarkable success in Natural Language Processing and multimodal tasks, among others. However, the carbon footprints and financial costs originating from heavy pre-training computation is a non-negligible issue. Progressive training methods, inspired by the neurogenesis process that grows neural structures, have shown potential to accelerate LLM pre-training. However, the algorithms, implementation, and practices for progressively training LLMs beyond 100B parameters remain underexplored. In this paper, we show that our model, namely FLM-101B, trained with our growth strategy under a budget of \$100K, reaches 80\% of the baselines' performances with only 10\% of their floating-point operations. We believe that further studies on progressive training will benefit the community by cutting down the costs and promoting green AI. The checkpoint of FLM-101B is released at https://huggingface.co/CofeAI/FLM-101B.

• The paper presents a novel growth strategy that reduces training costs for a 101B-parameter LLM to just $100K.
• It evaluates FLM-101B using IQ-inspired assessments that test reasoning and pattern recognition beyond standard benchmarks.
• Experimental results show competitive performance compared to models like GPT-3 while significantly lowering computational demands.

An Examination of FLM-101B: Training a 101B-Parameter LLM with Budget Constraints

The paper "FLM-101B: An Open LLM and How to Train It with a \$100K Budget&quot; investigates the development and training of a LLM with over 100 billion parameters under a constrained budget. This paper addresses two pivotal challenges in LLM development: reducing high training costs and establishing fair evaluations that extend beyond mere memorization.</p> <h3 class='paper-heading' id='training-cost-reduction-through-a-growth-strategy'>Training Cost Reduction Through a Growth Strategy</h3> <p>A significant contribution of the paper is the introduction of a growth strategy to minimize training cost. Traditionally, LLMs like GPT-3 and the LLAMA series have high computational demands. The researchers present a methodology to train a 101B-parameter LLM, termed FLM-101B, using only \$100,000. The key innovation is the "growth" strategy where model parameters are not fixed and grow throughout training. This approach theoretically reduces the number of floating-point operations required, as FLOPs generally scale with the number of model parameters. Through this method, the training cost benefits directly by maximizing computational savings across different growth strategies evaluated in the text.

Performance Evaluation and IQ-Based Assessment

FLM-101B is evaluated not only through conventional knowledge-based assessments but also through intelligence quotient (IQ)-like tests. The paper identifies limitations in standard benchmarks that may not fully reflect a model's true reasoning and problem-solving abilities. Therefore, the IQ-inspired evaluations focus on symbolic mapping, rule understanding, pattern mining, and anti-interference, offering a diversified approach to evaluating LLM capabilities beyond simple knowledge recall. Notably, FLM-101B provides strong performance comparable to existing models such as GPT-3 and GLM-130B in these varied contextual evaluations.

Contributions and Experimental Results

The paper claims that FLM-101B, aside from being a cost-efficient model, offers competitive results on several evaluation tasks while using significantly fewer computational resources. The model undergoes extensive evaluations across a wide range of tasks, demonstrating skills in both knowledge-oriented benchmarks and less conventional IQ tests. The paper asserts that the model's effective use of the growth strategy provides a promising direction for future research in reducing the computational demands of training expansive LLMs.

Implications and Future Directions

The paper’s implications are twofold. Practically, it emphasizes cost-effective methodologies for scaling LLMs. Theoretically, it provokes a rethinking of evaluation methodologies in AI, highlighting the potential of IQ-inspired assessments. Future developments could explore further optimizations in the growth strategy, potentially applying these principles to even larger models, potentially beyond the trillion-parameter mark. Additionally, with the release of FLM-101B model checkpoints, this research supports burgeoning efforts in bilingual LLM development and experimentation.

In conclusion, the paper effectively illustrates strategies to address the dual challenges of cost and evaluation in LLM training, advocating for both innovative technical approaches and reconsidered evaluative frameworks. The combination of explicit cost constraints with a broader evaluation capacity aligns with emerging needs in the AI community for scalable, cost-efficient, and robust LLMs.