Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
166 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Statistical Query Lower Bounds for Learning Truncated Gaussians (2403.02300v1)

Published 4 Mar 2024 in cs.DS, cs.LG, math.ST, stat.ML, and stat.TH

Abstract: We study the problem of estimating the mean of an identity covariance Gaussian in the truncated setting, in the regime when the truncation set comes from a low-complexity family $\mathcal{C}$ of sets. Specifically, for a fixed but unknown truncation set $S \subseteq \mathbb{R}d$, we are given access to samples from the distribution $\mathcal{N}(\boldsymbol{ \mu}, \mathbf{ I})$ truncated to the set $S$. The goal is to estimate $\boldsymbol\mu$ within accuracy $\epsilon>0$ in $\ell_2$-norm. Our main result is a Statistical Query (SQ) lower bound suggesting a super-polynomial information-computation gap for this task. In more detail, we show that the complexity of any SQ algorithm for this problem is $d{\mathrm{poly}(1/\epsilon)}$, even when the class $\mathcal{C}$ is simple so that $\mathrm{poly}(d/\epsilon)$ samples information-theoretically suffice. Concretely, our SQ lower bound applies when $\mathcal{C}$ is a union of a bounded number of rectangles whose VC dimension and Gaussian surface are small. As a corollary of our construction, it also follows that the complexity of the previously known algorithm for this task is qualitatively best possible.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. Statistical query algorithms and low degree tests are almost equivalent. In Conference on Learning Theory, pages 774–774. PMLR, 2021.
  2. N. Balakrishnan and C. Erhard. The art of progressive censoring. Statistics for industry and technology, 2014.
  3. D. Bernoulli. Essai d’une nouvelle analyse de la mortalité causée par la petite vérole, et des avantages de l’inoculation pour la prévenir. pages 1–14, 1760.
  4. What makes a good fisherman? linear regression under self-selection bias. In Symposium on Theory of Computing, pages 1699–1712, 2023.
  5. Learning (very) simple generative models is hard. In NeurIPS, 2022.
  6. A. C. Cohen. Truncated and censored samples: theory and applications. CRC press, 1991.
  7. Efficient statistics, in high dimensions, from truncated samples. In Foundations of Computer Science (FOCS), pages 639–649, 2018.
  8. Computationally and statistically efficient truncated regression. In Conference on Learning Theory, pages 955–960. PMLR, 2019.
  9. I. Diakonikolas and D. Kane. Near-optimal statistical query hardness of learning halfspaces with massart noise. In Conference on Learning Theory, volume 178 of Proceedings of Machine Learning Research, pages 4258–4282. PMLR, 2022. Full version available at https://arxiv.org/abs/2012.09720.
  10. I. Diakonikolas and D. M. Kane. Algorithmic high-dimensional robust statistics. Cambridge university press, 2023.
  11. Learning general halfspaces with general massart noise under the gaussian distribution. In STOC ’22: 54th Annual ACM SIGACT Symposium on Theory of Computing, pages 874–885, 2022. Full version available at https://arxiv.org/abs/2108.08767.
  12. Algorithms and SQ lower bounds for PAC learning one-hidden-layer relu networks. In Conference on Learning Theory, COLT 2020, volume 125 of Proceedings of Machine Learning Research, pages 1514–1539. PMLR, 2020.
  13. Statistical query lower bounds for list-decodable linear regression. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, pages 3191–3204, 2021.
  14. The optimality of polynomial regression for agnostic learning under gaussian marginals in the sq model. In Conference on Learning Theory, pages 1552–1584. PMLR, 2021.
  15. SQ lower bounds for learning mixtures of separated and bounded covariance gaussians. In The Thirty Sixth Annual Conference on Learning Theory, COLT 2023, volume 195 of Proceedings of Machine Learning Research, pages 2319–2349. PMLR, 2023.
  16. Sq lower bounds for non-gaussian component analysis with weaker assumptions. In NeurIPS, volume 36, pages 4199–4212, 2023.
  17. Statistical query lower bounds for robust estimation of high-dimensional gaussians and gaussian mixtures. In 58th IEEE Annual Symposium on Foundations of Computer Science, FOCS 2017, pages 73–84, 2017. Full version at http://arxiv.org/abs/1611.03473.
  18. List-decodable robust mean estimation and learning mixtures of spherical gaussians. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, STOC 2018, pages 1047–1060, 2018. Full version available at https://arxiv.org/abs/1711.07211.
  19. Efficient algorithms and lower bounds for robust linear regression. In Proceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2019, pages 2745–2754, 2019.
  20. SQ lower bounds for learning mixtures of linear classifiers. CoRR, abs/2310.11876, 2023. Conference version in NeurIPS 2023.
  21. A statistical taylor theorem and extrapolation of truncated densities. In Conference on Learning Theory, pages 1395–1398. PMLR, 2021.
  22. Near-optimal SQ lower bounds for agnostically learning halfspaces and relus under gaussian marginals. In on Neural Information Processing Systems, 2020.
  23. Truncated linear regression in high dimensions. Advances in Neural Information Processing Systems, 33:10338–10347, 2020.
  24. Efficient truncated linear regression with unknown noise variance. Advances in Neural Information Processing Systems, 34:1952–1963, 2021.
  25. Statistical algorithms and a lower bound for detecting planted cliques. In Proceedings of STOC’13, pages 655–664, 2013. Full version in Journal of the ACM, 2017.
  26. R. A. Fisher. Properties and applications of hh functions. Mathematical tables, 1(815-852):2, 1931.
  27. Efficient parameter estimation of truncated boolean product distributions. In Conference on Learning Theory, pages 1586–1600. PMLR, 2020.
  28. F. Galton. An examination into the registered speeds of american trotting horses, with remarks on their value as hereditary data. Proceedings of the Royal Society of London, 62(379-387):310–315, 1898.
  29. Statistical-query lower bounds via functional gradients. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, 2020.
  30. S. B. Hopkins. Statistical inference and the sum of squares method. PhD thesis, Cornell University, 2018.
  31. A theoretical and practical framework for regression and classification from truncated samples. In International Conference on Artificial Intelligence and Statistics, pages 4463–4473. PMLR, 2020.
  32. D. M. Kane. The gaussian surface area and noise sensitivity of degree-d polynomial threshold functions. Computational Complexity, 20(2):389–412, 2011.
  33. M. J. Kearns. Efficient noise-tolerant learning from statistical queries. Journal of the ACM, 45(6):983–1006, 1998.
  34. Learning geometric concepts via Gaussian surface area. In Proc. 49th IEEE Symposium on Foundations of Computer Science (FOCS), pages 541–550, 2008.
  35. Efficient truncated statistics with unknown truncation. In 2019 IEEE 60th Annual Symposium on Foundations of Computer Science (FOCS), pages 1578–1595. IEEE, 2019.
  36. Notes on computational hardness of hypothesis testing: Predictions using the low-degree likelihood ratio. In ISAAC Congress (International Society for Analysis, its Applications and Computation), pages 1–50. Springer, 2022.
  37. A. Lee. Table of the gaussian" tail" functions; when the" tail" is larger than the body. Biometrika, 10(2/3):208–214, 1914.
  38. Learning exponential families from truncated samples. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  39. J. Neeman. Testing surface area with arbitrary accuracy. In Symposium on Theory of Computing, STOC 2014, 2014, pages 393–397. ACM, 2014.
  40. K. Pearson. On the systematic fitting of curves to observations and measurements: part ii. Biometrika, 2(1):1–23, 1902.
  41. K. Pearson and A. Lee. On the generalised probable error in multiple normal correlation. Biometrika, 6(1):59–68, 1908.
  42. O. Plevrakis. Learning from censored and dependent data: The case of linear dynamics. In Conference on Learning Theory, pages 3771–3787. PMLR, 2021.
  43. H. Schneider. Truncated and censored samples from normal populations. Marcel Dekker, Inc., 1986.
  44. G. Szegö. Orthogonal Polynomials. Number τ𝜏\tauitalic_τ. 23 in American Mathematical Society colloquium publications. American Mathematical Society, 1967.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now