Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Density Uncertainty Layers for Reliable Uncertainty Estimation (2306.12497v2)

Published 21 Jun 2023 in cs.LG and stat.ML

Abstract: Assessing the predictive uncertainty of deep neural networks is crucial for safety-related applications of deep learning. Although Bayesian deep learning offers a principled framework for estimating model uncertainty, the common approaches that approximate the parameter posterior often fail to deliver reliable estimates of predictive uncertainty. In this paper, we propose a novel criterion for reliable predictive uncertainty: a model's predictive variance should be grounded in the empirical density of the input. That is, the model should produce higher uncertainty for inputs that are improbable in the training data and lower uncertainty for inputs that are more probable. To operationalize this criterion, we develop the density uncertainty layer, a stochastic neural network architecture that satisfies the density uncertain criterion by design. We study density uncertainty layers on the UCI and CIFAR-10/100 uncertainty benchmarks. Compared to existing approaches, density uncertainty layers provide more reliable uncertainty estimates and robust out-of-distribution detection performance.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (42)
  1. Christopher M. Bishop. Pattern recognition and machine learning. Springer-Verlag New York, Inc., Secaucus, NJ, 2006.
  2. Weight uncertainty in neural network. In ICML, 2015.
  3. Stochastic gradient hamiltonian monte carlo. In ICML, 2014.
  4. UCI machine learning repository, 2017. URL http://archive.ics.uci.edu/ml.
  5. Efficient and scalable bayesian neural nets with rank-1 factors. In ICML, 2020.
  6. Pathologies of factorised gaussian and mc dropout posteriors in bayesian neural networks. In NeurIPS 4th Workshop on Bayesian Deep Learning, 2019a.
  7. ’in-between’uncertainty in bayesian neural networks. In ICML Uncertainty and Robustness in Deep Learning Workshop, 2019b.
  8. On the expressiveness of approximate inference in bayesian neural networks. In NeurIPS, 2020.
  9. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In ICML, 2016.
  10. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359–378, 2007.
  11. Alex Graves. Practical variational inference for neural networks. In NeurIPS, 2011.
  12. On calibration of modern neural networks. In ICML, 2017.
  13. Deep residual learning for image recognition. In CVPR, 2016.
  14. Auto-encoding variational bayes. In ICLR, 2014.
  15. Variational dropout and the local reparameterization trick. In NeurIPS, 2015.
  16. Learning multiple layers of features from tiny images. Technical report, Computer Science Department, University of Toronto, 2009.
  17. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. In NeurIPS, 2020.
  18. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2017.
  19. Structured and efficient variational deep learning with matrix gaussian posteriors. In ICML, 2016.
  20. A simple baseline for bayesian uncertainty in deep learning. 2019.
  21. Spectral normalization for generative adversarial networks. 2018.
  22. Obtaining well calibrated probabilities using bayesian binning. In AAAI, 2015.
  23. Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
  24. Reading digits in natural images with unsupervised feature learning. In NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
  25. Benchmarking the neural linear model for regression. In AABI, 2019.
  26. The promises and pitfalls of deep kernel learning. In UAI, 2021.
  27. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. In NeurIPS, 2019.
  28. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019.
  29. Stochastic backpropagation and approximate inference in deep generative models. In ICML, 2014.
  30. A scalable laplace approximation for neural networks. In ICLR, 2018.
  31. Sparse uncertainty representation in deep learning with inducing weights. In NeurIPS, 2021.
  32. Sparse gaussian processes using pseudo-inputs. In NeurIPS, 2005.
  33. Scalable bayesian optimization using deep neural networks. In ICML, 2015.
  34. Functional variational bayesian neural networks. In ICLR, 2019.
  35. Michalis Titsias. Variational learning of inducing variables in sparse gaussian processes. In AISTATS, 2009.
  36. Overpruning in variational bayesian neural networks. In NeurIPS Approximate Bayesian Inference Workshop, 2017.
  37. On feature collapse and deep kernel learning for single forward pass uncertainty. arXiv preprint arXiv:2102.11409, 2021.
  38. Conditional image generation with pixelcnn decoders. In NeurIPS, 2016.
  39. Fast dropout training. In ICML, 2013.
  40. Bayesian learning via stochastic gradient langevin dynamics. In ICML, 2011.
  41. Deep kernel learning. In AISTATS, 2016.
  42. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets