Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Adaptive posterior concentration rates for sparse high-dimensional linear regression with random design and unknown error variance (2405.19016v1)

Published 29 May 2024 in math.ST, stat.ML, and stat.TH

Abstract: This paper investigates sparse high-dimensional linear regression, particularly examining the properties of the posterior under conditions of random design and unknown error variance. We provide consistency results for the posterior and analyze its concentration rates, demonstrating adaptiveness to the unknown sparsity level of the regression coefficient vector. Furthermore, we extend our investigation to establish concentration outcomes for parameter estimation using specific distance measures. These findings are in line with recent discoveries in frequentist studies. Additionally, by employing techniques to address model misspecification through a fractional posterior, we broaden our analysis through oracle inequalities to encompass the critical aspect of model misspecification for the regular posterior. Our novel findings are demonstrated using two different types of sparsity priors: a shrinkage prior and a spike-and-slab prior.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (60)
  1. High-dimensional classification by sparse logistic regression. IEEE Transactions on Information Theory, 65(5):3068–3079.
  2. On optimality of bayesian testimation in the normal means problem. Annals of Statistics, 35(5):2261.
  3. Concentration of tempered posteriors and of their variational approximations. The Annals of Statistics, 48(3):1475–1497.
  4. Spike-and-slab group lassos for grouped regression and sparse generalized additive models. Journal of the American Statistical Association, 117(537):184–197.
  5. Bayesian inference in high-dimensional models. arXiv preprint arXiv:2101.04491.
  6. Empirical bayes oracle uncertainty quantification for regression. Annals of Statistics, 48(6):3113–3137.
  7. Slope meets lasso: improved oracle bounds and optimality. The Annals of Statistics, 46(6B):3603–3642.
  8. Bayesian fractional posteriors. Annals of Statistics, 47(1):39–66.
  9. A general framework for updating belief distributions. Journal of the Royal Statistical Society Series B: Statistical Methodology, 78(5):1103–1130.
  10. Statistics for high-dimensional data: Methods, theory and applications. Springer Series in Statistics. Springer, Heidelberg.
  11. The horseshoe estimator for sparse signals. Biometrika, 97(2):465–480.
  12. Empirical Bayes analysis of spike and slab posterior distributions. Electronic Journal of Statistics, 12:3953–4001.
  13. Bayesian linear regression with sparse priors. Annals of Statistics, 43(5):1986–2018.
  14. Needles and straw in a haystack: Posterior concentration for possibly sparse sequences. The Annals of Statistics, 40(4):2069–2101.
  15. Bayesian sparse multiple regression for simultaneous rank reduction and variable selection. Biometrika, 107(1):205–221.
  16. Consistency of variational bayes inference for estimation and model selection in mixtures. Electronic Journal of Statistics, 12:2995–3035.
  17. Mirror averaging with sparsity priors. Bernoulli, 18(3):914–944.
  18. Sparse regression learning by aggregation and langevin monte-carlo. Journal of Computer and System Sciences, 78(5):1423–1443.
  19. Derumigny, A. (2018). Improved bounds for square-root lasso and square-root slope. Electronic Journal of Statistics, 12:741–766.
  20. High-dimensional properties for empirical priors in linear regression with unknown error variance. Statistical Papers, 65(1):237–262.
  21. A general framework for bayes structured linear models. Annals of Statistics, 48(5):2848–2878.
  22. Variable selection via gibbs sampling. Journal of the American Statistical Association, 88(423):881–889.
  23. Giraud, C. (2022). Introduction to high-dimensional statistics, volume 168 of Monographs on Statistics and Applied Probability. CRC Press, Boca Raton, FL, second edition.
  24. Inconsistency of Bayesian inference for misspecified linear models, and a proposal for repairing it. Bayesian Analysis, 12(4):1069–1103.
  25. Approximate bayesian inference based on expected evaluation. Bayesian Analysis, pages 1–22.
  26. The elements of statistical learning: Data mining, inference, and prediction. Springer Series in Statistics. Springer, New York, second edition.
  27. Model misspecification, Bayesian versus credibility estimation, and Gibbs posteriors. Scandinavian Actuarial Journal, 2020(7):634–649.
  28. Posterior contraction in sparse generalized linear models. Biometrika, 108(2):367–379.
  29. Unified bayesian theory of sparse linear regression with nuisance parameters. Electronic Journal of Statistics, 15(1):3040–3111.
  30. General bayesian loss function selection and the use of improper models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(5):1640–1665.
  31. Jiang, W. (2007). Bayesian variable selection for high dimensional generalized linear models: Convergence rates of the fitted densities. The Annals of Statistics, 35(4):1487–1511.
  32. Needles and straw in haystacks: Empirical Bayes estimates of possibly sparse sequences. Ann. Statist., 32(1):1594–1649.
  33. An optimization-centric view on bayes’ rule: Reviewing and generalizing variational inference. Journal of Machine Learning Research, 23(132):1–109.
  34. Semiparametric inference using fractional posteriors. The Journal of Machine Learning Research, 24(1):18619–18679.
  35. Bayesian subset modeling for high-dimensional generalized linear models. Journal of the American Statistical Association, 108(502):589–606.
  36. Bayesian inference in high-dimensional linear models using an empirical correlation-adaptive prior. Statistica Sinica, 31(4):2051–2072.
  37. General bayesian updating and the loss-likelihood bootstrap. Biometrika, 106(2):465–478.
  38. Mai, T. T. (2023). A reduced-rank approach to predicting multiple binary responses through machine learning. Statistics and Computing, 33(6):136.
  39. Mai, T. T. (2024). High-dimensional sparse classification using exponential weighting with empirical hinge loss. Statistica Neerlandica, (accepted):1–28.
  40. Pseudo-Bayesian quantum tomography with rank-adaptation. Journal of Statistical Planning and Inference, 184:62–76.
  41. Empirical bayes posterior concentration in sparse high-dimensional linear models. Bernoulli, 23(3):1822–1847.
  42. Empirical priors for prediction in sparse high-dimensional linear regression. Journal of Machine Learning Research, 21(144):1–30.
  43. Robust generalised bayesian inference for intractable likelihoods. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(3):997–1022.
  44. On the robustness to misspecification of α𝛼\alphaitalic_α-posteriors and their variational approximations. Journal of Machine Learning Research, 23(147):1–51.
  45. Bayesian variable selection in linear regression. Journal of the american statistical association, 83(404):1023–1032.
  46. Bayesian linear regression for multivariate responses under group sparsity. Bernoulli, 26(3):2353–2382.
  47. Posterior contraction in sparse bayesian factor models for massive covariance matrices. The Annals of Statistics, 42(3):1102–1130.
  48. Variational bayes for high-dimensional linear regression with sparse priors. Journal of the American Statistical Association, 117(539):1270–1281.
  49. Spike and slab variational bayes for high dimensional logistic regression. Advances in Neural Information Processing Systems, 33:14423–14434.
  50. Rivoirard, V. (2006). Nonlinear estimation over weak besov spaces and minimax bayes method. Bernoulli, 12(4):609–632.
  51. EMVS: The EM approach to Bayesian variable selection. Journal of the American Statistical Association, 109(506):828–846.
  52. The Spike-and-Slab LASSO. Journal of the American Statistical Association, 113(521):431–444.
  53. Seeger, M. W. (2008). Bayesian inference and optimal design for the sparse linear model. Journal of Machine Learning Research, 9:759–813.
  54. Calibrating general posterior credible regions. Biometrika, 106(2):479–486.
  55. Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1):267–288.
  56. Rényi divergence and kullback-leibler divergence. IEEE Transactions on Information Theory, 60(7):3797–3820.
  57. Wainwright, M. J. (2019). High-dimensional statistics: A non-asymptotic viewpoint, volume 48 of Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge.
  58. Adaptive bayesian nonparametric regression using a kernel mixture of polynomials with application to partial linear models. Bayesian Analysis, 15(1):159–186.
  59. α𝛼\alphaitalic_α-variational inference with statistical guarantees. Annals of Statistics, 48(2):886–905.
  60. Adaptation of the tuning parameter in general bayesian inference with robust divergence. Statistics and Computing, 33(2):39.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now