Computational Tradeoffs of Optimization-Based Bound Tightening in ReLU Networks (2312.16699v2)
Abstract: The use of Mixed-Integer Linear Programming (MILP) models to represent neural networks with Rectified Linear Unit (ReLU) activations has become increasingly widespread in the last decade. This has enabled the use of MILP technology to test-or stress-their behavior, to adversarially improve their training, and to embed them in optimization models leveraging their predictive power. Many of these MILP models rely on activation bounds. That is, bounds on the input values of each neuron. In this work, we explore the tradeoff between the tightness of these bounds and the computational effort of solving the resulting MILP models. We provide guidelines for implementing these models based on the impact of network structure, regularization, and rounding.
- Funahashi, K.I.: On the approximate realization of continuous mappings by neural networks. Neural Networks (1989)
- Gurobi: Gurobi Machine Learning. https://github.com/Gurobi/gurobi-machinelearning (2023), accessed: 2023-12-03
- Yarotsky, D.: Error bounds for approximations with deep ReLU networks. Neural Networks (2017)
Collections
Sign up for free to add this paper to one or more collections.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
