Optimistic Rates for Learning with a Smooth Loss

We establish an excess risk bound of O(H Rn² + Rn \sqrt{H L}) for empirical risk minimization with an H-smooth loss function and a hypothesis class with Rademacher complexity R_n, where L is the best risk achievable by the hypothesis class. For typical hypothesis classes where R_n = \sqrt{R/n}, this translates to a learning rate of O(RH/n) in the separable (L=0) case and O(RH/n + \sqrt{L^ RH/n}) more generally. We also provide similar guarantees for online and stochastic convex optimization with a smooth non-negative objective.