Abstract
Is there a canonical way to think of agency beyond reward maximisation? In this paper, we show that any type of behaviour complying with physically sound assumptions about how macroscopic biological agents interact with the world canonically integrates exploration and exploitation in the sense of minimising risk and ambiguity about states of the world. This description, known as active inference, refines the free energy principle, a popular descriptive framework for action and perception originating in neuroscience. Active inference provides a normative Bayesian framework to simulate and model agency that is widely used in behavioural neuroscience, reinforcement learning (RL) and robotics. The usefulness of active inference for RL is three-fold. \emph{a}) Active inference provides a principled solution to the exploration-exploitation dilemma that usefully simulates biological agency. \emph{b}) It provides an explainable recipe to simulate behaviour, whence behaviour follows as an explainable mixture of exploration and exploitation under a generative world model, and all differences in behaviour are explicit in differences in world model. \emph{c}) This framework is universal in the sense that it is theoretically possible to rewrite any RL algorithm conforming to the descriptive assumptions of active inference as an active inference algorithm. Thus, active inference can be used as a tool to uncover and compare the commitments and assumptions of more specific models of agency.
Overview
-
The paper presents a framework for modeling decision-making and actions that influence the environment, known as agency.
-
Active inference is proposed as a normative Bayesian approach that surpasses traditional RL by integrating exploratory and exploitative behaviors.
-
The paper describes the theoretical foundations of active inference in terms of physical principles and classical mechanics.
-
Active inference offers robust solutions, universal applicability to RL algorithms, and transparent decision-making for safety-critical domains.
-
The authors discuss future research directions, including studying human generative models and integrating probabilistic models with deep neural networks.
Overview of Active Inference and Agency
In the paper titled "Active Inference as a Model of Agency," a comprehensive framework for understanding and modeling agency is presented. Agency, within this context, refers to the ability of an entity—be it biological or artificial—to make decisions and take actions that influence the environment. The authors propose active inference as a normative Bayesian approach that naturally integrates both exploratory and exploitative behaviors by minimizing risk and ambiguity concerning external states.
Agency Beyond Reward Maximization
Active inference is highlighted as an advance over traditional reinforcement learning (RL) models, which primarily focus on reward maximization. The authors argue that active inference, by minimizing expected free energy, can address the exploration-exploitation dilemma without resorting to ad hoc methods for inducing exploratory behavior. This paradigm shift is significant as it aligns with the natural behavior observed in biological systems, providing a more realistic and principled methodology for simulating intelligent behavior while offering explainability through explicit generative world models.
Theoretical Foundations in Physics
The foundation of active inference lies in well-established physical principles, as the paper elucidates by discussing the behavior of macroscopic biological systems subject to classical mechanics, known as precise agents. A precise agent responds deterministically to environmental stimuli given the information available up until the current moment. The mathematical formalization of this interaction delineates the nature of decision-making and agency as optimization problems, governed by an expected free energy functional—a criterion that balances objective probabilities of states with subjective preferences or utilities.
Advantages and Applications
Active inference presents several advantages over traditional models of decision-making in living organisms and artificial systems. Firstly, it emphasizes the value of understanding preferences and predictions in a unified framework, leading to robust exploration-exploration solutions. Secondly, the universality of the active inference framework allows RL algorithms to be recast as active inference algorithms, fostering comparisons and advancing theory development in agent-based modeling. Finally, by explicitly encoding predictions and preferences, active inference lays down a blueprint for transparent and justifiable algorithmic decision-making, proving highly applicable to safety-critical domains.
Path Forward
The authors conclude by reflecting on the potential of active inference and pose several open questions important for future research. One area of intrigue is identifying the generative models humans use to represent their environments and the computational mechanisms through which these representations evolve from childhood to adulthood. The integration of hierarchical probabilistic models and deep neural architectures is suggested as a promising path forward, with the goal of understanding and eventually replicating the development of human intelligence in artificial agents.
Create an account to read this summary for free: