Many NLP tasks benefit from using LLMs that often have more than 100 billion parameters. With the release of BLOOM-176B and OPT-175B, everyone can download pretrained models of this scale. Still, using these models requires high-end hardware unavailable to many researchers. In some cases, LLMs can be used more affordably via RAM offloading or hosted APIs. However, these techniques have innate limitations: offloading is too slow for interactive inference, while APIs are not flexible enough for research that requires access to weights, attention or logits. In this work, we propose Petals - a system for inference and fine-tuning of large models collaboratively by joining the resources of multiple parties. We demonstrate that this strategy outperforms offloading for very large models, running inference of BLOOM-176B on consumer GPUs with $\approx$ 1 step per second, which is enough for many interactive LLM applications. Unlike most inference APIs, Petals also natively exposes hidden states of served models, allowing to train and share custom model extensions based on efficient fine-tuning methods.
Petals introduces a decentralized system to collaboratively perform inference and fine-tuning on LLMs using distributed computational resources.
The framework includes features such as distributed model layers, fault tolerance, load balancing, adaptive routing, and quantization to optimize performance and accessibility.
Benchmark results display significant efficiency and robustness, making high-performance NLP capabilities accessible to users with consumer-grade hardware.
The increasing scale of LLMs, exemplified by models like BLOOM-176B and OPT-175B, presents both opportunities and challenges for the NLP community. These models, with parameter counts exceeding 100 billion, have demonstrated remarkable abilities in solving various NLP tasks through fine-tuning or prompting. However, their computational and memory demands restrict their usability to institutions equipped with extensive hardware resources. The paper titled "Petals: Collaborative Inference and Fine-tuning of Large Models" addresses this bottleneck by proposing a novel, decentralized approach to leverage distributed computational resources.
The core innovation presented in Petals is a decentralized system enabling multiple users to collaboratively perform inference and fine-tuning of LLMs. The system is designed to distribute model layers across multiple servers, allowing for democratized access to high-performance NLP capabilities.
Key components and design principles of Petals include:
The performance of Petals has been rigorously benchmarked under various network conditions and hardware setups. Key findings include:
The theoretical contributions of Petals span several domains, particularly in distributed systems and collaborative computing. Practically, the system opens up new possibilities for researchers and practitioners who were previously constrained by hardware limitations. It enables scalable, efficient, and democratized access to state-of-the-art LLMs, facilitating a wider range of NLP research and applications.
Looking ahead, the approach introduced by Petals could lead to several advancements:
"Petals: Collaborative Inference and Fine-tuning of Large Models" presents a robust framework to surmount the computational barriers associated with large-scale language models. By distributing model computations across multiple servers and implementing sophisticated optimization techniques, Petals offers a scalable solution to make advanced NLP models more accessible. The system's practical utility, coupled with its theoretical contributions, holds promise for future research and applications in artificial intelligence.
The authors acknowledge the assistance and discussions from individuals and institutions that contributed to the development and testing of Petals. The collaborative nature of this development reflects the ethos of the system itself, emphasizing distributed effort and shared advancement.
This overview captures the technical sophistication and practical implications of Petals while adhering to a formal and professional tone suitable for experienced researchers in the field.