A Generalized Variable Importance Metric and Estimator for Black Box Machine Learning Models (2212.09931v3)
Abstract: In this paper we define a population parameter, ``Generalized Variable Importance Metric (GVIM)'', to measure importance of predictors for black box machine learning methods, where the importance is not represented by model-based parameter. GVIM is defined for each input variable, using the true conditional expectation function, and it measures the variable's importance in affecting a continuous or a binary response. We extend previously published results to show that the defined GVIM can be represented as a function of the Conditional Average Treatment Effect (CATE) for any kind of a predictor, which gives it a causal interpretation and further justification as an alternative to classical measures of significance that are only available in simple parametric models. Extensive set of simulations using realistically complex relationships between covariates and outcomes and number of regression techniques of varying degree of complexity show the performance of our proposed estimator of the GVIM.
- Molnar, C. Interpretable machine learning (Lulu. com, 2020).
- Breiman, L. Random forests. Machine learning 45, 5–32 (2001).
- ” Why should i trust you?” Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, 1135–1144 (2016).
- A unified approach to interpreting model predictions. In Proceedings of the 31st international conference on neural information processing systems, 4768–4777 (2017).
- Predicting suicidal thoughts and behavior among adolescents using the risk and protective factor framework: A large-scale machine learning approach. Plos one 16, e0258535 (2021).
- Correlation and variable importance in random forests. Statistics and Computing 27, 659–678 (2017).
- All Models are Wrong, but Many are Useful: Learning a Variable’s Importance by Studying an Entire Class of Prediction Models Simultaneously. J. Mach. Learn. Res. 20, 1–81 (2019a).
- Unrestricted permutation forces extrapolation: variable importance requires at least one more model, or there is no free variable importance. Statistics and Computing 31, 1–16 (2021).
- Bias in random forest variable importance measures: Illustrations, sources and a solution. BMC bioinformatics 8, 1–21 (2007).
- Conditional variable importance for random forests. BMC bioinformatics 9, 1–11 (2008).
- Kaneko, H. Cross-validated permutation feature importance considering correlation between features. Analytical Science Advances 3, 278–287 (2022).
- Variable importance and prediction methods for longitudinal problems with missing variables. PloS one 10 (2015).
- Targeted learning: causal inference for observational and experimental data (Springer Science & Business Media, 2011).
- Analyzing bagging. Annals of statistics 30, 927–961 (2002).
- Quasi-regression for visualization and interpretation of black box functions (2002).
- Causal interpretations of black-box models. Journal of Business & Economic Statistics 1–10 (2019).
- Targeted learning in data science (Springer, 2018).
- An application of collaborative targeted maximum likelihood estimation in causal inference and genomics. The International Journal of Biostatistics 6 (2010).
- Bootstrapping and sample splitting for high-dimensional, assumption-lean inference. The Annals of Statistics 47, 3438–3469 (2019).
- Machine learning for comprehensive forecasting of Alzheimer’s Disease progression. Scientific reports 9, 1–14 (2019b).
- Grouped variable importance with random forests and application to multiple functional data analysis. Computational Statistics & Data Analysis 90, 15–35 (2015).
- Feature Importance: A Closer Look at Shapley Values and LOCO. arXiv preprint arXiv:2303.05981 (2023).
- Improvements on cross-validation: the 632+ bootstrap method. Journal of the American Statistical Association 92, 548–560 (1997).
- An introduction to the bootstrap. Monographs on statistics and applied probability 57, 1–436 (1993).
- Breiman, L. Bagging predictors. Machine learning 24, 123–140 (1996).
- Friedman, J. H. Multivariate adaptive regression splines. The annals of statistics 19, 1–67 (1991).
- Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, 785–794 (2016).