Can large language models explore in-context?

Exploring the Limits of Exploration: How LLMs Fare in Multi-Armed Bandit Environments

Introduction

The capacity for exploration underpins effective decision-making in complex environments. This study scrutinizes the inherent abilities of contemporary LLMs to engage in exploration, crucial for reinforcement learning (RL) and sequential decision making. By deploying LLMs as agents within multi-armed bandit (MAB) settings—without any training adjustments—the investigation uniquely positions LLMs in scenarios that demand exploration for successful navigation and learning.

Experimental Design

Given the emerging relevance of in-context learning, this study introduces a systematic examination of LLMs' exploration capabilities via simple yet foundational RL problems: multi-armed bandits. This choice is motivated by the simpleness and analytical tractability of MAB problems, which isolate the exploration-exploitation dilemma fundamental to decision making.

The research employs three LLMs: Gpt-3.5, GPT-4, and Llama2, leveraging various prompt designs to enact the MAB scenario and gather responses. These models are exposed to a set of specifically designed prompts that detail the bandit environment and query for next actions, giving rise to different experimental configurations based on the prompt nuances.

The exploration behaviors of these LLMs are probed across multiple settings:

• Environment Complexity: Easy and hard instances of MAB are chosen based on the number of arms and reward distribution complexities.
• Temperature Settings: Zero and one temperature settings in LLM prompts aim to distinguish between intrinsic exploration and externally injected randomness.
• Prompt Variations: Ranging from basic to advanced prompts, this study encompasses different scenarios, framings, summarization levels, and prompting for chain-of-thought reasoning.

Results and Findings

Across numerous experimental runs, a single configurational success emerged: GPT-4, complemented by specific prompt attributes that suggested exploration, summarized interaction history, and enforced chain-of-thought reasoning. This configuration exhibited robust exploratory behavior, effectively identifying and exploiting the most rewarding actions in the stipulated bandit environment.

Contrastingly, the majority of configurations demonstrated significant exploration deficiencies, manifesting either through an undue focus on exploiting immediate rewards (akin to a greedy strategy) or through an almost uniform, undiscriminating choice distribution across all actions, indicative of a failure to learn from past interactions.

Specifically, configurations not employing summarized interaction histories or lacking in prompt attributes that explicitly incited exploration were prone to these failures. Interestingly, the exploration success with GPT-4 also highlighted the nuanced but critical role of prompt design in eliciting more sophisticated behaviors from LLMs.

Implications and Future Directions

This investigation underlines the necessity of non-trivial prompt engineering or potential algorithmic interventions to unlock and elevate the decision-making capacities of LLMs in settings that demand robust exploration strategies. Thefindings prompt several lines of inquiry and development:

• Further Prompt Exploration: Expanding the diversity and depth of prompts may uncover more nuanced aspects of LLM capabilities.
• Algorithmic Interventions: Fine-tuning or custom training paradigms might be essential for cultivating sophisticated exploration behaviors in more complex RL environments.
• Methodological Advances: Developing methodologies for cost-effective, large-scale evaluations of LLM behaviors in decision-making contexts is paramount.

Conclusion

While a singular configuration demonstrated the potential for LLMs to engage in strategic exploration within a controlled environment, the overarching evidence points to a generalized struggle among LLMs to autonomously navigate the exploration-exploitation trade-off without explicit guidance. This study, while focusing on the elemental RL challenge of multi-armed bandits, lays foundational insights for the development of LLMs as more adept decision-making agents intackling broader and more complex decision-making tasks.