BET: Explaining Deep Reinforcement Learning through The Error-Prone Decisions (2401.07263v1)
Abstract: Despite the impressive capabilities of Deep Reinforcement Learning (DRL) agents in many challenging scenarios, their black-box decision-making process significantly limits their deployment in safety-sensitive domains. Several previous self-interpretable works focus on revealing the critical states of the agent's decision. However, they cannot pinpoint the error-prone states. To address this issue, we propose a novel self-interpretable structure, named Backbone Extract Tree (BET), to better explain the agent's behavior by identify the error-prone states. At a high level, BET hypothesizes that states in which the agent consistently executes uniform decisions exhibit a reduced propensity for errors. To effectively model this phenomenon, BET expresses these states within neighborhoods, each defined by a curated set of representative states. Therefore, states positioned at a greater distance from these representative benchmarks are more prone to error. We evaluate BET in various popular RL environments and show its superiority over existing self-interpretable models in terms of explanation fidelity. Furthermore, we demonstrate a use case for providing explanations for the agents in StarCraft II, a sophisticated multi-agent cooperative game. To the best of our knowledge, we are the first to explain such a complex scenarios using a fully transparent structure.
- Decision tree c4. 5 algorithm for tuition aid grant program classification (case study: Department of information system, universitas teknokrat indonesia). Jurnal Ilmiah Edutic: Pendidikan dan Informatika, 7(1):40–50, 2020.
- Verifiable reinforcement learning via policy extraction. Advances in Neural Information Processing Systems (NeurIPS), 31, 2018.
- Look where you look! saliency-guided q-networks for generalization in visual reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 35:30693–30706, 2022.
- Openai gym. arXiv preprint arXiv:1606.01540, 2016.
- Safe learning in robotics: From learning-based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 5:411–444, 2022.
- Classification based on decision tree algorithm for machine learning. Journal of Applied Science and Technology Trends, 2(01):20–28, 2021.
- Kevin Chen. Deep reinforcement learning for flappy bird. CS 229 Machine-Learning Final Projects, 2015.
- A reduction from reinforcement learning to no-regret online learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), pages 3514–3524. PMLR, 2020.
- Statemask: Explaining deep reinforcement learning through state mask. In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS), 2023.
- Distilling deep reinforcement learning policies in soft decision trees. In Proceedings of the 28-th International Joint Conference on Artificial Intelligence (IJCAI), pages 1–6, 2019.
- Evolving interpretable decision trees for reinforcement learning. Artificial Intelligence, page 104057, 2023.
- A theoretical analysis of deep q-learning. In Learning for dynamics and control, pages 486–489. PMLR, 2020.
- Decision tree-based diagnosis of coronary artery disease: Cart model. Computer methods and programs in biomedicine, 192:105400, 2020.
- Edge: Explaining deep reinforcement learning policies. Advances in Neural Information Processing Systems (NeurIPS), 34:12222–12236, 2021.
- Local and global explanations of agent behavior: Integrating strategy summaries with saliency maps. Artificial Intelligence, 301:103571, 2021.
- Explaining by imitating: Understanding decisions by interpretable policy learning. The International Conference on Learning Representations (ICLR), 2021.
- How to train your robot with deep reinforcement learning: lessons we have learned. The International Journal of Robotics Research, 40(4-5):698–721, 2021.
- A unified game-theoretic approach to multiagent reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
- Juewu-mc: Playing minecraft with sample-efficient hierarchical reinforcement learning. Proceedings of the 31-th International Joint Conference on Artificial Intelligence (IJCAI), 2022.
- Zachary C Lipton. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue, 16(3):31–57, 2018.
- Effective interpretable policy distillation via critical experience point identification. IEEE Intelligent Systems, 2023.
- Towards improving decision tree induction by combining split evaluation measures. Knowledge-Based Systems, 277:110832, 2023.
- Christoph Molnar. Interpretable machine learning. Lulu. com, 2020.
- Bridging the gap between value and policy based reinforcement learning. Advances in neural information processing systems, 30, 2017.
- Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
- Monotonic value function factorisation for deep multi-agent reinforcement learning. The Journal of Machine Learning Research, 21(1):7234–7284, 2020.
- The starcraft multi-agent challenge. CoRR, abs/1902.04043, 2019.
- Reinforcement learning algorithms: A brief survey. Expert Systems with Applications, page 120495, 2023.
- A gradient boosting decision tree based gps signal reception classification algorithm. Applied Soft Computing, 86:105942, 2020.
- Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350–354, 2019.
- Lei Xiong and Ye Yao. Study on an adaptive thermal comfort model with k-nearest-neighbors (knn) algorithm. Building and Environment, 202:108026, 2021.
Collections
Sign up for free to add this paper to one or more collections.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
