Model selection for contextual bandits

Abstract: We introduce the problem of model selection for contextual bandits, where a learner must adapt to the complexity of the optimal policy while balancing exploration and exploitation. Our main result is a new model selection guarantee for linear contextual bandits. We work in the stochastic realizable setting with a sequence of nested linear policy classes of dimension $d_1 &lt; d_2 &lt; \ldots$, where the $m<sup>\star$-th class contains the optimal policy, and we design an algorithm that achieves $\tilde{O}(T<sup>{2/3}d<sup>{1/3}_{m<sup>\star})$ regret with no prior knowledge of the optimal dimension $d_{m<sup>\star}$. The algorithm also achieves regret $$\tilde{O}(T^{3/4} + \sqrt{Td_{m<sup>\star}})$$, which is optimal for $d_{m<sup>{\star}}\geq{}\sqrt{T}$. This is the first model selection result for contextual bandits with non-vacuous regret for all values of $d_{m<sup>\star}$, and to the best of our knowledge is the first positive result of this type for any online learning setting with partial information. The core of the algorithm is a new estimator for the gap in the best loss achievable by two linear policy classes, which we show admits a convergence rate faster than the rate required to learn the parameters for either class.