Learning Human-like Representations to Enable Learning Human Values (2312.14106v3)

Abstract: How can we build AI systems that can learn any set of individual human values both quickly and safely, avoiding causing harm or violating societal standards for acceptable behavior during the learning process? We explore the effects of representational alignment between humans and AI agents on learning human values. Making AI systems learn human-like representations of the world has many known benefits, including improving generalization, robustness to domain shifts, and few-shot learning performance. We demonstrate that this kind of representational alignment can also support safely learning and exploring human values in the context of personalization. We begin with a theoretical prediction, show that it applies to learning human morality judgments, then show that our results generalize to ten different aspects of human values -- including ethics, honesty, and fairness -- training AI agents on each set of values in a multi-armed bandit setting, where rewards reflect human value judgments over the chosen action. Using a set of textual action descriptions, we collect value judgments from humans, as well as similarity judgments from both humans and multiple LLMs, and demonstrate that representational alignment enables both safe exploration and improved generalization when learning human values.

• The paper demonstrates that AI systems with human-like internal representations better align with ethical norms compared to misaligned models.
• It employs reinforcement learning with support vector and kernel regression models in a multi-armed bandit setting to evaluate performance across ethical decision benchmarks.
• Results show that even partial representational alignment improves both reward maximization and ethical action selection over traditional baselines such as Thompson sampling.

Introduction to Value Alignment in AI

The growing power and autonomy of machine learning models necessitate ensuring their alignment with human values and societal norms, to mitigate harm and adhere to acceptable behavior. This topic has been historically challenging within the field of AI research, with several approaches proving insufficient. The focus of academic inquiry is shifting toward the relationship between machines' internal representations of the world and their ability to learn and adhere to human values—a concept known as representational alignment. In essence, the research probes whether AI adopting human-like worldviews can lead to better understanding and implementation of human values.

Representational Alignment and its Importance

Representational alignment involves the concordance of internal worldviews between humans and AI models. A significant amount of research establishes that AI systems with human-like representations exhibit better performance in tasks involving few-shot learning, robustness to changes, and generalization. Crucially, such alignment may assist AI systems in gaining trust since humans can better understand decisions made by these models, paving the way for broader deployment in sensitive, human-centric applications. This paper postulates that representational alignment is an essential, though not exhaustive, step toward achieving value alignment.

Ethics in Value Alignment

The ethical dimension of value alignment becomes particularly relevant in reinforcement learning contexts. Agents in these scenarios are given autonomy, raising the potential for decisions that could deviate from human values. This research utilizes a reinforcement learning model in which an agent undertakes actions characterized by various morality scores. By examining the link between representational alignment and the agent's capability to choose ethically sound actions, the paper provides empirical evidence that suggests agents with higher representational alignment perform better in ethical decision-making tasks.

Methodology and Results

The paper involved training agents using support vector regression and kernel regression models within a multi-armed bandit setting. Morality scores, simulating ethical valuations, were assigned to the agent's actions. To ascertain the impact of representational misalignment, the agents were subjected to differing levels of alignment degradation, affecting their internal worldviews. The researchers observed a clear correlation: as representational alignment diminished, performance across several benchmarks—including reward maximization and taking ethical actions—also decreased. Notably, even partially aligned agents surpassed a traditional Thompson sampling baseline, underscoring the advantages of representational alignment.

Implications and Future Work

The relationship between representational and value alignment represents a critical component of developing more secure and value-consistent AI systems. This paper's findings indicate that greater representational alignment can support AI in making decisions that are more ethically sound. Future research directions could involve the translation of these empirical observations into formal mathematical models and assessing the implications for more complex AI systems. The ultimate goal is a collaborative advancement in AI development that reliably upholds human values.