Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

On the Asymptotic Mean Square Error Optimality of Diffusion Models (2403.02957v4)

Published 5 Mar 2024 in cs.LG and stat.ML

Abstract: Diffusion models (DMs) as generative priors have recently shown great potential for denoising tasks but lack theoretical understanding with respect to their mean square error (MSE) optimality. This paper proposes a novel denoising strategy inspired by the structure of the MSE-optimal conditional mean estimator (CME). The resulting DM-based denoiser can be conveniently employed using a pre-trained DM, being particularly fast by truncating reverse diffusion steps and not requiring stochastic re-sampling. We present a comprehensive (non-)asymptotic optimality analysis of the proposed diffusion-based denoiser, demonstrating polynomial-time convergence to the CME under mild conditions. Our analysis also derives a novel Lipschitz constant that depends solely on the DM's hyperparameters. Further, we offer a new perspective on DMs, showing that they inherently combine an asymptotically optimal denoiser with a powerful generator, modifiable by switching re-sampling in the reverse process on or off. The theoretical findings are thoroughly validated with experiments based on various benchmark datasets

Definition Search Book Streamline Icon: https://streamlinehq.com
References (52)
  1. MIMO channel estimation using score-based generative models. IEEE Transactions on Wireless Communications, 22(6):3698–3713, 2023.
  2. The perception-distortion tradeoff. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  6228–6237, 2018.
  3. Callahan, J. J. Advanced Calculus: A Geometric View. Springer New York, NY, 2010.
  4. Probability Theory: Independence, Interchangeability, Martingales. Springer NY, 3rd edition, 1997.
  5. Come-closer-diffuse-faster: Accelerating conditional diffusion models for inverse problems through stochastic contraction. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.  12403–12412, 2022.
  6. Inversion by direct iteration: An alternative to denoising diffusion for image restoration, 2023. arXiv: 2303.11435.
  7. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2):295–307, 2016.
  8. A general derivative identity for the conditional mean estimator in Gaussian noise and some applications. In IEEE International Symposium on Information Theory (ISIT), pp.  1183–1188, 2020.
  9. Image denoising: The deep learning revolution and beyond—a survey paper. SIAM Journal on Imaging Sciences, 16(3):1594–1654, 2023.
  10. DiracDiffusion: Denoising and incremental reconstruction with assured data-consistency, 2023. arXiv: 2303.14353.
  11. Feller, W. On the theory of stochastic processes, with particular reference to applications. In Proceedings of the [First] Berkeley Symposium on Mathematical Statistics and Probability, volume 1, pp.  403–433. 1949.
  12. On the mean square error optimal estimator in one-bit quantized systems. IEEE Transactions on Signal Processing, 71:1968–1980, 2023.
  13. Applied Biomedical Engineering. IntechOpen, Rijeka, Aug 2011.
  14. Regularisation of neural networks by enforcing Lipschitz continuity. Machine Learning, 110:393 – 416, 2018.
  15. Iterative α𝛼\alphaitalic_α-(de)blending: A minimalist deterministic diffusion model. In ACM SIGGRAPH Conference Proceedings, 2023.
  16. Denoising diffusion probabilistic models. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pp.  6840–6851, 2020.
  17. Wavedm: Wavelet-based diffusion models for image restoration, 2023. arXiv: 2305.13819.
  18. Denoising diffusion medical models, 2023. arXiv: 2304.09383.
  19. Robust compressed sensing using generative models. In Advances in Neural Information Processing Systems, volume 33, pp.  713–727, 2020.
  20. Stochastic solutions for linear inverse problems using the prior implicit in a denoiser. In Advances in Neural Information Processing Systems, volume 34, pp.  13242–13254, 2021.
  21. Bayesian denoising: From MAP to MMSE using consistent cycle spinning. IEEE Signal Processing Letters, 20(3):249–252, 2013.
  22. Variational diffusion models. In Advances in Neural Information Processing Systems, volume 34, pp.  21696–21707, 2021.
  23. An asymptotically MSE-optimal estimator based on Gaussian mixture models. IEEE Transactions on Signal Processing, 70:4109–4123, 2022.
  24. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021.
  25. MNIST handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist, 2, 2010.
  26. Loizou, P. C. Speech Enhancement: Theory and Practice. CRC Press, Inc., USA, 2nd edition, 2013.
  27. Diffusion model based posterior sampling for noisy linear inverse problems, 2023. arXiv: 2211.12343.
  28. Approximation by finite mixtures of continuous density functions that vanish at infinity. Cogent Mathematics & Statistics,, 7(1):1750861, 2020.
  29. Improved denoising diffusion probabilistic models. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pp.  8162–8171, 2021.
  30. MMSE based MAP estimation for image denoising. Optics & Laser Technology, 57:252–264, 2014.
  31. Deep learning techniques for inverse problems in imaging. IEEE Journal on Selected Areas in Information Theory, 1(1):39–56, 2020.
  32. Librispeech: An ASR corpus based on public domain audio books. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.  5206–5210, 2015.
  33. Pedregosa, F. et al. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011.
  34. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention, pp.  234–241. Springer, 2015.
  35. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4713–4726, 2023.
  36. Statistical Signal Processing: Detection, Estimation, and Time Series Analysis. Pearson, 1991.
  37. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pp.  2256–2265, 2015.
  38. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021a.
  39. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, volume 32, 2019.
  40. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021b.
  41. Consistency models, 2023. arXiv: 2303.01469.
  42. Revisiting denoising diffusion probabilistic models for speech enhancement: Condition collapse, efficiency and refinement. Proceedings of the AAAI Conference on Artificial Intelligence, 37(11):13627–13635, Jun. 2023.
  43. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
  44. Speech enhancement with score-based generative models in the complex STFT domain. In Proc. Interspeech, pp.  2928–2932, 2022.
  45. Deblurring via stochastic refinement. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.  16272–16282, 2022.
  46. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms, 2017. arXiv: 1708.07747.
  47. Diffusion model for generative image denoising, 2023. arXiv: 2302.02398.
  48. Compressive sensing by learning a Gaussian mixture model from measurements. IEEE Transactions on Image Processing, 24(1):106–119, 2015.
  49. Yang, Z. et al. Eliminating Lipschitz singularities in diffusion models, 2023. arXiv: 2306.11251.
  50. Wide residual networks, 2017. arXiv: 1605.07146.
  51. Loss functions for image restoration with neural networks. IEEE Transactions on Computational Imaging, 3(1):47–57, 2017.
  52. Denoising diffusion models for plug-and-play image restoration. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp.  1219–1229, 2023.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets