OpenFE: Automated Feature Generation with Expert-level Performance (2211.12507v3)

Abstract: The goal of automated feature generation is to liberate machine learning experts from the laborious task of manual feature generation, which is crucial for improving the learning performance of tabular data. The major challenge in automated feature generation is to efficiently and accurately identify effective features from a vast pool of candidate features. In this paper, we present OpenFE, an automated feature generation tool that provides competitive results against machine learning experts. OpenFE achieves high efficiency and accuracy with two components: 1) a novel feature boosting method for accurately evaluating the incremental performance of candidate features and 2) a two-stage pruning algorithm that performs feature pruning in a coarse-to-fine manner. Extensive experiments on ten benchmark datasets show that OpenFE outperforms existing baseline methods by a large margin. We further evaluate OpenFE in two Kaggle competitions with thousands of data science teams participating. In the two competitions, features generated by OpenFE with a simple baseline model can beat 99.3% and 99.6% data science teams respectively. In addition to the empirical results, we provide a theoretical perspective to show that feature generation can be beneficial in a simple yet representative setting. The code is available at https://github.com/ZhangTP1996/OpenFE.

• The paper introduces FeatureBoost, which estimates incremental performance gains without full model retraining, optimizing feature evaluation.
• It employs a two-stage pruning algorithm to efficiently narrow the candidate feature set based on standalone and interaction-based effectiveness.
• Empirical results across benchmarks, including Kaggle competitions, show that OpenFE enables simple models to outperform seasoned data science teams.

Overview of "OpenFE: Automated Feature Generation with Expert-level Performance"

The paper introduces OpenFE, a tool for automated feature generation that aims to alleviate the intensive labor involved in manual feature creation for tabular data. Recognizing the pivotal role of feature quality in enhancing machine learning performance, OpenFE employs sophisticated mechanisms to efficiently and accurately discern beneficial features from an extensive array of possibilities.

Core Contributions

OpenFE encapsulates two innovative techniques addressing the challenges of automated feature generation: a feature boosting method and a two-stage pruning algorithm. These methodologies enable the identification of potential features without exhausting computational resources.

1. FeatureBoost: Inspired by gradient boosting, this algorithm estimates the incremental performance of new features without the need for model retraining, significantly optimizing the evaluation process. It performs incremental training based on predictions rendered by the base feature set, enhancing efficiency.
2. Two-Stage Pruning Algorithm: Comprising successive featurewise pruning and feature importance attribution, this technique reduces the candidate feature space by assessing each feature's standalone effectiveness before refining the selection with interaction-based evaluations.

Empirical and Theoretical Validation

OpenFE’s effectiveness is measured across ten benchmark datasets and two Kaggle competitions. Empirical results showcase its superiority, where OpenFE surpasses existing baseline methods by notable margins. Particularly in Kaggle competitions, OpenFE's generated features enable simple models to outperform the vast majority of data science teams—exceeding even those of winning human experts. These results are accompanied by a theoretical framework demonstrating the statistical advantage of feature generation, modeled in a transductive learning setup. The theoretical model proves the feasibility of achieving near-zero test loss through feature generation, a feat unattainable with base features alone.

Implications and Future Directions

The contributions of OpenFE pivotally improve automated machine learning workflows. The paper provides a scalable solution for feature generation, which is critical as datasets continue to expand in size and complexity. It reaffirms the pertinence of feature generation despite advances in deep learning, which were once thought to diminish its necessity for tabular data.

Future research may delve into extending OpenFE's functionalities to accommodate time-series data or optimizing for computational constraints when handling exceedingly large datasets. Additionally, further explorations could address algorithm enhancements for high-order feature generation, ensuring a balance between interpretability and model performance improvements.

Conclusion

OpenFE stands out as a robust tool in the repertoire of automated machine learning, offering significant advancements in feature generation with practicality and theoretical grounding. Its competitive performance not only against baseline methods but also seasoned data science teams underscores its potential as a staple in automated feature engineering endeavors.