Supervising strong learners by amplifying weak experts (1810.08575v1)

Abstract: Many real world learning tasks involve complex or hard-to-specify objectives, and using an easier-to-specify proxy can lead to poor performance or misaligned behavior. One solution is to have humans provide a training signal by demonstrating or judging performance, but this approach fails if the task is too complicated for a human to directly evaluate. We propose Iterated Amplification, an alternative training strategy which progressively builds up a training signal for difficult problems by combining solutions to easier subproblems. Iterated Amplification is closely related to Expert Iteration (Anthony et al., 2017; Silver et al., 2017), except that it uses no external reward function. We present results in algorithmic environments, showing that Iterated Amplification can efficiently learn complex behaviors.

• The paper demonstrates that Iterated Amplification enables learning in scenarios where direct evaluation is infeasible by decomposing tasks into manageable subproblems.
• It employs a composite system where human experts collaborate with multiple model instances to iteratively refine outputs.
• Experimental results highlight that the method scales to complex tasks, offering a robust alternative to traditional supervised learning.

Iterated Amplification: Supervising Learning via Amplified Expertise

The paper presented explores the concept of Iterated Amplification as a novel methodology to supervise machine learning models, particularly in contexts where target objectives are complex or difficult to evaluate directly. This method offers a shift from traditional practices by leveraging the decomposition of tasks into subproblems that are more easily managed, enabling learning in settings where external reward functions or human evaluations are impractical.

Methodology Overview

Iterated Amplification differs from standard forms of learning which rely heavily on either algorithmic model evaluations (e.g., winning a game) or supervised signals from human demonstrations or preferences. The authors address scenarios that exceed simple evaluations, where a single evaluator (human or algorithm) cannot feasibly oversee and judge the model's output comprehensively. By using Iterated Amplification, the process combines outputs from multiple simpler tasks into a coherent signal that guides the learning process. This method relies on an idea similar to Expert Iteration but operates without predefined external rewards.

The process involves several design choices:

1. Task Selection: Opt to train the agent on question-answering tasks that are adequately representative of the larger goal.
2. Composition Framework: Use a composite system, termed Amplify^H{X}, where a human expert collaborates with multiple instances of the learning model to iteratively solve and refine task outputs.
3. Model Learning: Implement supervised learning, where the model learns to predict the composite system's behavior.

Initially acting randomly, the model relies heavily on human expertise but progressively shifts towards a more autonomous role as the iterations refine the agent’s capabilities.

Experimental Approach

To validate their methodology, the authors tested Iterated Amplification with algorithmic problems that allowed for easy comparison to supervised learning approaches. The results highlighted that even complex tasks could be efficiently learned in this framework, offering an alternative to direct supervised learning, especially when such direct supervision is infeasible. Tasks were progressively scaled in complexity, demonstrating stability and improvement in agent capabilities over time.

Implications and Future Directions

The paradigm of Iterated Amplification provides a robust framework for approaching real-world problems where training data is difficult to label or reward structures are not externally definable. This could have significant implications for domains such as policy-making, economic modeling, or extensive system management, which are currently beyond the reach of simple algorithmic or direct human-evaluation approaches.

The paper lays the groundwork for further exploration into removing simplifications used in their experiments—most notably employing human decomposition of realistic tasks and scaling the model to more significant real-world applications. The authors project that such frameworks might proliferate the capacity of ML techniques, avoiding the pitfalls of misaligned proxy objectives prevalent in current AI-driven ecosystems.

In conclusion, Iterated Amplification proposed in this research provides a cornerstone for future developments in AI, particularly the domain of complex problem-solving without direct, scalable human assessment.