A dynamical clipping approach with task feedback for Proximal Policy Optimization (2312.07624v3)

Abstract: Proximal Policy Optimization (PPO) has been broadly applied to robotics learning, showcasing stable training performance. However, the fixed clipping bound setting may limit the performance of PPO. Specifically, there is no theoretical proof that the optimal clipping bound remains consistent throughout the entire training process. Meanwhile, previous researches suggest that a fixed clipping bound restricts the policy's ability to explore. Therefore, many past studies have aimed to dynamically adjust the PPO clipping bound to enhance PPO's performance. However, the objective of these approaches are not directly aligned with the objective of reinforcement learning (RL) tasks, which is to maximize the cumulative Return. Unlike previous clipping approaches, we propose a bi-level proximal policy optimization objective that can dynamically adjust the clipping bound to better reflect the preference (maximizing Return) of these RL tasks. Based on this bi-level proximal policy optimization paradigm, we introduce a new algorithm named Preference based Proximal Policy Optimization (Pb-PPO). Pb-PPO utilizes a multi-armed bandit approach to refelect RL preference, recommending the clipping bound for PPO that can maximizes the current Return. Therefore, Pb-PPO results in greater stability and improved performance compared to PPO with a fixed clipping bound. We test Pb-PPO on locomotion benchmarks across multiple environments, including Gym-Mujoco and legged-gym. Additionally, we validate Pb-PPO on customized navigation tasks. Meanwhile, we conducted comparisons with PPO using various fixed clipping bounds and various of clipping approaches. The experimental results indicate that Pb-PPO demonstrates superior training performance compared to PPO and its variants. Our codebase has been released at : https://github.com/stevezhangzA/pb_ppo

• The paper introduces a dynamic clipping mechanism for PPO using task feedback and multi-armed bandit methods to optimize clipping bounds.
• It demonstrates enhanced sample efficiency and training stability, outperforming fixed-bound PPO on various reinforcement learning benchmarks.
• The bi-level optimization framework of Pb-PPO successfully balances exploration and exploitation, suggesting potential extensions to include human feedback.

A Dynamical Clipping Approach with Task Feedback for Proximal Policy Optimization

Abstract

The paper introduces a novel approach to enhancing Proximal Policy Optimization (PPO) by introducing a dynamic clipping mechanism that adjusts the clipping bound during the training process based on task feedback. This new algorithm, named Preference-based Proximal Policy Optimization (Pb-PPO), leverages a multi-armed bandit algorithm to suggest optimal clipping bounds, improving training efficiency and performance. Pb-PPO demonstrates superior stability and outcomes in various reinforcement learning tasks compared to traditional fixed-bound PPO.

Introduction

Proximal Policy Optimization (PPO) is a widely adopted reinforcement learning algorithm known for its stability and efficiency in training policies. However, the default fixed clipping bound used in PPO imposes limitations on exploration and policy update stability. The paper addresses this issue by introducing a bi-level optimization framework in PPO that considers dynamic adjustment of the clipping bound aligned with the task's preference for maximizing cumulative returns.

Dynamic Clipping Mechanism

The core improvement proposed is the dynamic adjustment of PPO's clipping bound using a multi-armed bandit framework. The algorithm dynamically samples clipping bounds by estimating their expected returns in each epoch and selects the one with the highest Upper Confidence Bound (UCB) value. This dynamic sampling is designed to improve the exploration-exploitation balance and optimize the policy effectively during training.

Figure 1: Presentation of Pb-PPO. Pb-PPO comprises two optimization objectives: 1) Enhancing the Upper Confidence Bound (UCB) value to approximate the true value estimation while maintaining a balance between exploration and exploitation in the sampling of optimal clipping bounds. 2) Implementing Proximal Policy Optimization to maximize policy performance.

Algorithmic Formulation

Pb-PPO operates through a bi-level optimization process. The primary objectives are:

1. Policy Optimization: Maximize the expected return using Proximal Policy Optimization constrained by the dynamically sampled clipping bound.
2. Clipping Bound Selection: Utilize UCB theory to update the expected returns and uncertainties of candidate clipping bounds, ensuring effective exploration and exploitation during sampling.

The approach is validated on various locomotion tasks, demonstrating significant improvements in stability and performance compared to PPO using fixed clipping bounds.

Results and Discussion

The experimental results highlight Pb-PPO's superior performance across multiple RL benchmarks such as Gym-mujoco. Pb-PPO consistently outperformed traditional PPO with fixed clipping bounds and other on-policy algorithms like TRPO and DDPG.

Figure 2: Pb-PPO (task feedback) on locomotion tasks. Each solid curve in these figures represents the average experimental results across multiple seeds, and the shadowed area corresponds to the minimum and maximum returns.

Performance Comparison

Pb-PPO achieved higher sample efficiency and more stable training curves, leading to better overall training performance. The dynamic adjustment of clipping bounds aligns closely with task feedback, thus reflecting task or human preferences effectively in the training process.

Scalability and Future Directions

The paper suggests that Pb-PPO can be extended to incorporate human feedback, showcasing its versatility beyond RL tasks. Future work could explore scaling Pb-PPO in more complex domains and incorporating additional feedback mechanisms to fine-tune policy optimization further.

Conclusion

Pb-PPO introduces a significant enhancement to PPO by incorporating a dynamic clipping mechanism through task feedback and multi-armed bandit theory. This approach not only resolves limitations of fixed clipping bounds but also achieves improved training efficiency and stability. Future advancements can explore broader applications and further improvements in automated machine learning settings.