Emergent Mind

Large Language Models as Optimizers

(2309.03409)

Published Sep 7, 2023 in cs.LG , cs.AI , and cs.CL

Abstract

Optimization is ubiquitous. While derivative-based algorithms have been powerful tools for various problems, the absence of gradient imposes challenges on many real-world applications. In this work, we propose Optimization by PROmpting (OPRO), a simple and effective approach to leverage LLMs as optimizers, where the optimization task is described in natural language. In each optimization step, the LLM generates new solutions from the prompt that contains previously generated solutions with their values, then the new solutions are evaluated and added to the prompt for the next optimization step. We first showcase OPRO on linear regression and traveling salesman problems, then move on to our main application in prompt optimization, where the goal is to find instructions that maximize the task accuracy. With a variety of LLMs, we demonstrate that the best prompts optimized by OPRO outperform human-designed prompts by up to 8% on GSM8K, and by up to 50% on Big-Bench Hard tasks. Code at https://github.com/google-deepmind/opro.

Overview

The paper introduces OPRO, a framework allowing LLMs to serve as optimizers by generating new solutions based on natural language prompts of previously evaluated solutions.
The framework is tested on classical optimization problems like linear regression and the Traveling Salesman Problem, showing competitive performance in small-scale settings and notable improvements in prompt-based optimization for LLM tasks.
The paper discusses theoretical and practical implications, such as the adaptability of LLMs to various tasks and the potential reduction in reliance on expert human prompt engineering, and lays out future directions to enhance the framework's robustness and scalability.

LLMs as Optimizers

The paper "LLMs as Optimizers" by Chengrun Yang et al. introduces the concept of treating LLMs as effective optimization tools. This idea deviates from traditional optimization algorithms, particularly for derivative-free scenarios where gradient information is unavailable or infeasible to compute. The proposed framework, termed Optimization by PROmpting (OPRO), leverages the natural language capabilities of LLMs to iteratively improve solutions based on prior optimization states provided through prompting.

Main Contributions

Optimization Framework (OPRO):

The framework allows LLMs to serve as optimizers by describing the optimization task in natural language.
During each optimization step, the LLM generates a new set of solutions based on a prompt containing previously evaluated solutions and their scores.
The newly generated solutions are then evaluated and incorporated into the prompt for subsequent steps.
This iterative process continues until the optimization converges or a predefined number of steps is reached.

Applications in Mathematical Optimization:

The paper examines two classical optimization problems: linear regression and the Traveling Salesman Problem (TSP).
Results indicate that LLMs can effectively navigate optimization landscapes and sometimes perform on par with heuristic methods in small-scale settings.

Prompt Optimization:

A distinctive application of OPRO is in optimizing prompts for LLMs themselves, aiming to improve the accuracy of tasks such as natural language processing.
The paper demonstrates that optimized prompts can significantly outperform human-designed prompts, achieving up to 8% improvement on GSM8K and 50% on Big-Bench Hard tasks.

Discussion of Results:

The optimized instructions for various tasks reveal the LLM's ability to adapt to different styles and tasks effectively.
Empirical evaluations show that instructions generated via OPRO consistently improve performance until convergence.
The paper also examines the transferability of optimized prompts, showing notable generalization across different datasets within the same domain.

Implications and Future Directions

The concept of using LLMs as optimizers opens several theoretical and practical avenues. On the theoretical front, this approach challenges traditional optimization paradigms by incorporating natural language understanding into the optimization loop. Practically, it provides a flexible and powerful toolset for tasks where formal mathematical representations are cumbersome or impractical.

Theoretical Implications:

The ability of LLMs to leverage natural language descriptions for optimization suggests that similar techniques could be applied to other areas of combinatorial optimization and even broader AI challenges.
OPRO's iterative process, which mimics evolutionary algorithms without explicit mutation and crossover operations, underscores the emergent capabilities of LLMs to implicitly learn optimization heuristics from prompt structures.

Practical Implications:

Prompt optimization has immediate utility in improving the performance of LLMs across various NLP tasks without requiring extensive domain-specific engineering.
The framework's adaptability to new tasks and instructions decreases the dependency on expert human prompt engineering, potentially democratizing access to advanced optimization techniques.

Future Work:

Addressing the limitations related to the LLM context window size, especially for larger problem instances in mathematical optimization.
Enhancing the robustness of the optimization process to reduce sensitivity to initial conditions and to better balance exploration and exploitation.
Further exploration into leveraging richer feedback mechanisms beyond accuracy, such as error types and failure modes, could provide additional benefits in guiding the optimization process more effectively.

Conclusion

"LLMs as Optimizers" significantly contributes to the understanding of how LLMs can be harnessed for optimization tasks traditionally outside the purview of natural language processing. The OPRO framework not only demonstrates the versatility and power of prompt-based optimization but also sets the stage for future research in integrating natural language capabilities with optimization and other decision-making processes. By showcasing both theoretical advancements and practical applications, the paper paves the way for innovative uses of LLMs in AI and optimization.