Verifiable Boosted Tree Ensembles (2402.14988v1)
Abstract: Verifiable learning advocates for training machine learning models amenable to efficient security verification. Prior research demonstrated that specific classes of decision tree ensembles -- called large-spread ensembles -- allow for robustness verification in polynomial time against any norm-based attacker. This study expands prior work on verifiable learning from basic ensemble methods (i.e., hard majority voting) to advanced boosted tree ensembles, such as those trained using XGBoost or LightGBM. Our formal results indicate that robustness verification is achievable in polynomial time when considering attackers based on the $L_\infty$-norm, but remains NP-hard for other norm-based attackers. Nevertheless, we present a pseudo-polynomial time algorithm to verify robustness against attackers based on the $L_p$-norm for any $p \in \mathbb{N} \cup {0}$, which in practice grants excellent performance. Our experimental evaluation shows that large-spread boosted ensembles are accurate enough for practical adoption, while being amenable to efficient security verification.
- Provably robust boosted decision stumps and trees against adversarial attacks. In NeurIPS, 2019.
- Measuring neural net robustness with constraints. In NeurIPS, 2016.
- Evasion attacks against machine learning at test time. In ECML PKDD, 2013.
- Classification and Regression Trees. Wadsworth, 1984.
- Beyond robustness: Resilience verification of tree-based classifiers. Comput. Secur., 121, 2022.
- Verifiable learning for robust tree ensembles. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, CCS 2023, Copenhagen, CA, USA, November 26-30, 2023, 2023.
- Certifying decision trees against evasion attacks by program analysis. In ESORICS, 2020.
- Treant: training evasion-aware decision trees. Data Min. Knowl. Discov., 34(5):1390–1420, 2020.
- Robust decision trees against adversarial examples. In ICML, 2019.
- Robustness verification of tree-based models. In NeurIPS, 2019.
- Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785–794, 2016.
- Cost-aware robust tree ensembles for security applications. In USENIX Security Symposium, 2021.
- Learning security classifiers with verified global robustness properties. In ACM CCS, 2021.
- Adversarial exemples: A survey and experimental evaluation of practical attacks on machine learning for windows malware detection. ACM Trans. Priv. Secur., 24(4):27:1–27:31, 2021.
- Verifying tree ensembles by reasoning about potential instances. In SDM, 2021.
- Output range analysis for deep feedforward neural networks. In NFM, 2018.
- Verifying robustness of gradient boosted models. In AAAI, 2019.
- A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. Syst. Sci., 55(1):119–139, 1997.
- Jerome H Friedman. Greedy function approximation: a gradient boosting machine. Annals of statistics, pages 1189–1232, 2001.
- Computers and Intractability; A Guide to the Theory of NP-Completeness. W. H. Freeman & Co., USA, 1990.
- Verification of neural networks: Enhancing scalability through pruning. In ECAI, 2020.
- Fast provably robust decision trees and boosting. In ICML, 2022.
- Safety verification of deep neural networks. In CAV, 2017.
- Efficient exact verification of binarized neural networks. In NeurIPS, 2020.
- Evasion and hardening of tree ensemble classifiers. In ICML, 2016.
- Reluplex: An efficient SMT solver for verifying deep neural networks. In CAV, 2017.
- The marabou framework for verification and analysis of deep neural networks. In CAV, 2019.
- Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30, 2017.
- Globally-robust neural networks. In ICML, 2021.
- An approach to reachability analysis for feed-forward relu neural networks. CoRR, abs/1706.07351, 2017.
- Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
- Knapsack problems. Handbook of Combinatorial Optimization: Volume1–3, pages 299–428, 1998.
- Abstract interpretation of decision tree ensemble classifiers. In AAAI, 2020.
- Genetic adversarial training of decision trees. In GECCO, 2021.
- Formal verification of a decision-tree ensemble model and detection of its violation ranges. IEICE Trans. Inf. Syst., 103-D(2):363–378, 2020.
- Intriguing properties of neural networks. In ICLR, 2014.
- Evaluating robustness of neural networks with mixed integer programming. In ICLR, 2019.
- Formal verification of input-output mappings of tree ensembles. Sci. Comput. Program., 194:102450, 2020.
- Efficient training of robust decision trees against adversarial examples. In ICML, 2021.
- Adversarially robust decision tree relabeling. In ECML PKDD, 2022.
- Robust optimal classification trees against adversarial examples. In AAAI, 2022.
- On lp-norm robustness of ensemble decision stumps and trees. In ICML, 2020.
- Training for faster adversarial robustness verification via inducing relu stability. In ICLR, 2019.
- On the certified robustness for ensemble models and beyond. In ICLR, 2022.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
