SATzilla: Portfolio-based Algorithm Selection for SAT (1111.2249v1)

Abstract: It has been widely observed that there is no single "dominant" SAT solver; instead, different solvers perform best on different instances. Rather than following the traditional approach of choosing the best solver for a given class of instances, we advocate making this decision online on a per-instance basis. Building on previous work, we describe SATzilla, an automated approach for constructing per-instance algorithm portfolios for SAT that use so-called empirical hardness models to choose among their constituent solvers. This approach takes as input a distribution of problem instances and a set of component solvers, and constructs a portfolio optimizing a given objective function (such as mean runtime, percent of instances solved, or score in a competition). The excellent performance of SATzilla was independently verified in the 2007 SAT Competition, where our SATzilla07 solvers won three gold, one silver and one bronze medal. In this article, we go well beyond SATzilla07 by making the portfolio construction scalable and completely automated, and improving it by integrating local search solvers as candidate solvers, by predicting performance score instead of runtime, and by using hierarchical hardness models that take into account different types of SAT instances. We demonstrate the effectiveness of these new techniques in extensive experimental results on data sets including instances from the most recent SAT competition.

SATzilla: Portfolio-based Algorithm Selection for SAT - An Expert Overview

The authors present "SATzilla: Portfolio-based Algorithm Selection for SAT," a comprehensive and methodologically rigorous approach to improving SAT solver performance by optimizing solver selection on a per-instance basis. This paper provides a detailed exploration of the SATzilla framework and demonstrates its effectiveness through both theoretical development and empirical validation.

Motivation

Solving propositional satisfiability problems (SAT) efficiently is a cornerstone in the field of artificial intelligence and computer science. Despite the plethora of SAT solvers developed over the years, no single solver consistently outperforms others across all instances. SATzilla addresses this challenge by employing a portfolio-based approach, wherein multiple solvers are utilized, and the most suitable one is selected dynamically for each instance.

Methodology

The core innovation of SATzilla is its use of empirical hardness models to predict the performance of various SAT solvers based on instance-specific features. The process involves several stages:

1. Instance Characterization: Problem instances are characterized using a set of features that are cheap to compute yet strongly indicative of instance hardness.
2. Empirical Hardness Modeling: The authors employ ridge regression to build empirical hardness models that predict the runtime of each solver on a given instance.
3. Portfolio Construction: SATzilla constructs solver portfolios by selecting the best subset of solvers and determining effective pre-solvers.
4. Predictive Selection: During runtime, SATzilla uses the empirical hardness models to predict which solver will perform best on a given instance, factoring in feature computation time and pre-solving strategies.

Empirical Validation

The paper details extensive experiments, leveraging data from multiple SAT competitions to evaluate SATzilla's efficacy. SATzilla's performance was validated against several metrics:

1. Average Runtime: SATzilla consistently demonstrated lower average runtimes compared to individual solvers.
2. Instance Coverage: The portfolio approach significantly increased the percentage of instances solved within a given cutoff time.
3. Performance Scores: SATzilla achieved high performance scores, particularly in the SAT 2007 competition, where it outperformed most other solvers.

Contributions and Novel Techniques

This work advances the field through several key contributions:

1. Automated Portfolio Construction: The authors introduce an automated technique for pre-solver selection, optimizing the choice of pre-solvers and their cutoff times.
2. Incorporation of Local Search Solvers: For the first time, the framework integrates local search solvers, which are adept at solving certain types of instances, thus enhancing the portfolio's robustness.
3. Hierarchical Hardness Models: The paper expands on hierarchical models, which predict instance categories (satisfiable/unsatisfiable) and use tailored hardness models for these categories. This hierarchical approach improves prediction accuracy and solver performance.
4. Prediction of Performance Scores: SATzilla's method extends beyond mere runtime prediction to optimize complex performance scores, showing the approach's adaptability to various evaluation metrics.

Implications and Future Directions

The implications of SATzilla extend beyond the field of SAT solving. The methodology is generalizable to other combinatorial optimization problems where solver performance varies significantly across instances. Future developments could explore:

1. Enhanced Feature Engineering: Investigating richer and more impactful features could improve prediction accuracy further.
2. Dynamic Solver Integration: Implementing mechanisms to integrate new solvers dynamically as they become available.
3. Cross-domain Applications: Extending the portfolio approach to other NP-complete problems, such as constraint programming and integer programming.

Conclusion

SATzilla represents a sophisticated and highly effective approach to algorithm selection for SAT problems. By leveraging empirical hardness models, automated portfolio construction, and dynamic solver selection, the framework sets a new standard in the field. The paper shows that SATzilla not only achieves superior performance in controlled benchmarks but also demonstrates robustness and scalability, making it a valuable tool in advancing SAT solver technologies.