Reframing the Expected Free Energy: Four Formulations and a Unification (2402.14460v1)

Abstract: Active inference is a leading theory of perception, learning and decision making, which can be applied to neuroscience, robotics, psychology, and machine learning. Active inference is based on the expected free energy, which is mostly justified by the intuitive plausibility of its formulations, e.g., the risk plus ambiguity and information gain / pragmatic value formulations. This paper seek to formalize the problem of deriving these formulations from a single root expected free energy definition, i.e., the unification problem. Then, we study two settings, each one having its own root expected free energy definition. In the first setting, no justification for the expected free energy has been proposed to date, but all the formulations can be recovered from it. However, in this setting, the agent cannot have arbitrary prior preferences over observations. Indeed, only a limited class of prior preferences over observations is compatible with the likelihood mapping of the generative model. In the second setting, a justification of the root expected free energy definition is known, but this setting only accounts for two formulations, i.e., the risk over states plus ambiguity and entropy plus expected energy formulations.

• The paper unifies four formulations of Expected Free Energy by deriving them from a singular foundational definition.
• It employs a generative model with variational inference to balance observational accuracy with efficient policy exploration.
• It highlights limitations in prior preferences and likelihood mappings, paving the way for more robust theoretical frameworks.

Reframing the Expected Free Energy: Four Formulations and a Unification

Introduction

The concept of active inference is rooted in the modeling of perception, decision-making, and learning, offering applications that extend into fields like neuroscience, robotics, and machine learning. At the core of active inference is the Expected Free Energy (EFE), which comprises various formulations that are largely derived for their intuitive appeal. The unification problem as discussed in this paper addresses the challenge of deriving these formulations from a singular foundational definition of EFE.

Generative Model and Variational Framework

Active inference employs a generative model encapsulating hidden states, observations, and actions across time, adhering to a Partially Observable Markov Decision Process (POMDP) structure. To infer hidden states from observations, a variational inference approach is used, replacing the intractable true posterior with a more computationally feasible variational distribution. The agent minimizes the Variational Free Energy (VFE) to achieve this approximation, balancing accuracy and complexity.

Planning with Expected Free Energy

The decision-making component in active inference leverages EFE to evaluate and choose policies. EFE quantifies the trade-off between exploration and exploitation, aiming to maximize information gain alongside pragmatic value from the agent's prior preferences. The computational complexity of this process necessitates efficient policy exploration strategies such as Monte-Carlo tree search or sophisticated inference.

Unification Problem and EFE Formulations

The unification problem seeks to derive four known EFE formulations: Risk Over States and Ambiguity, Risk Over Observations and Ambiguity, Information Gain and Pragmatic Value, and Expected Energy and Entropy, from a unified definition. By establishing commonality through forecast and target distributions in theoretical frameworks, the paper aims to trace these formulations back to a foundational EFE description.

Analysis of EFE Formulations

• Risk Over Observations Plus Ambiguity: The paper posits Risk Over Observations plus Ambiguity as a potential root EFE definition. It explores the equivalences and derivations between different formulations, revealing that while this formulation allows recovery of all four EFE expressions, it lacks a formal theoretical justification.
• Risk Over States Plus Ambiguity: This formulation emerges as an upper bound of EFE and is currently the only one with a theoretical underpinning. However, it only supports two of the four EFE expressions.

The derivations rely critically on assumptions about distributions and marginalization, illustrating limitations like the restricted class of valid prior preferences over observations and potential inconsistency issues.

Limitations and Challenges

One prominent limitation is the restriction on arbitrary prior preferences due to compatibility issues with likelihood mappings. Additionally, there is a lack of justification for the Risk Over Observations plus Ambiguity formulation, raising questions about the theoretical integrity of the unified EFE derivation problem.

Conclusion

The attempt to unify varying formulations of EFE presents significant insights but also reveals critical constraints in theoretical assumptions and justifications. Future research should focus on developing a robust theoretical framework that encompasses all EFE formulations while resolving inconsistencies between prior preferences and likelihood mappings. Further exploration is required to bridge the theoretical discussions with practical implementations, particularly in the context of deep active inference applications.