Papers
Topics
Authors
Recent
Search
2000 character limit reached

Active Linear Regression for ℓp\ell_p Norms and Beyond

Published 9 Nov 2021 in cs.LG, cs.DS, and stat.ML | (2111.04888v4)

Abstract: We study active sampling algorithms for linear regression, which aim to query only a few entries of a target vector b∈Rnb\in\mathbb R^n and output a near minimizer to min⁡x∈Rd∥Ax−b∥\min_{x\in\mathbb R^d} \|Ax-b\|, for a design matrix A∈Rn×dA\in\mathbb R^{n \times d} and loss ∥⋅∥\|\cdot\|. For pp norm regression for any 0<p<∞0<p<\infty, we give an algorithm based on Lewis weight sampling outputting a (1+ϵ)(1+\epsilon)-approximate solution using just O~(d/ϵ2)\tilde O(d/\epsilon^2) queries to bb for p∈(0,1)p\in(0,1), O~(d/ϵ)\tilde{O}(d/\epsilon) queries for $1<p<2$, and O~(dp/2/ϵp)\tilde{O}(d^{p/2}/\epsilon^p) queries for 2<p<∞2<p<\infty. For $0<p<2$, our bounds are optimal up to log factors, settling the query complexity for this range. For 2<p<∞2<p<\infty, our dependence on dd is optimal, while our dependence on ϵ\epsilon is off by at most ϵ\epsilon, up to log factors. Our result resolves an open question of [CD21], who gave near optimal bounds for the $1$ norm, but required d2/ϵ2d^2/\epsilon^2 samples for ℓp\ell_p regression with $1<p<2$, and gave no bounds for 2<p<∞2<p<\infty or $0<p<1$. We also give the first total sensitivity bound of O(dmax⁡{1,p/2}log⁡2n)O(d^{\max\{1,p/2\}}\log^2n) for loss functions of degree pp polynomial growth, improving a result of [TMF20]. By combining this with our techniques for ℓp\ell_p regression, we obtain an active regression algorithm making O~(d1+max⁡{1,p/2}/poly(ϵ))\tilde O(d^{1+\max\{1,p/2\}}/\mathrm{poly}(\epsilon)) queries for such loss functions, including the Tukey and Huber losses, answering another question of [CD21]. For the Huber loss, we further improve our bound to O~(d4−22/poly(ϵ))\tilde O(d^{4-2\sqrt2}/\mathrm{poly}(\epsilon)) samples. Our sensitivity bounds also have many applications, including Orlicz norm subspace embeddings, robust subspace approximation, and dimension reduction for smoothed pp-norms. Finally, our active sampling results give the first sublinear time algorithms for Kronecker product regression under every pp norm.

Citations (7)

Summary

No one has generated a summary of this paper yet.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Continue Learning

We haven't generated follow-up questions for this paper yet.