Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
124 tokens/sec
GPT-4o
8 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Conformal Loss-Controlling Prediction (2301.02424v2)

Published 6 Jan 2023 in cs.LG

Abstract: Conformal prediction is a learning framework controlling prediction coverage of prediction sets, which can be built on any learning algorithm for point prediction. This work proposes a learning framework named conformal loss-controlling prediction, which extends conformal prediction to the situation where the value of a loss function needs to be controlled. Different from existing works about risk-controlling prediction sets and conformal risk control with the purpose of controlling the expected values of loss functions, the proposed approach in this paper focuses on the loss for any test object, which is an extension of conformal prediction from miscoverage loss to some general loss. The controlling guarantee is proved under the assumption of exchangeability of data in finite-sample cases and the framework is tested empirically for classification with a class-varying loss and statistical postprocessing of numerical weather forecasting applications, which are introduced as point-wise classification and point-wise regression problems. All theoretical analysis and experimental results confirm the effectiveness of our loss-controlling approach.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. C. Li, G. Tang, X. Xue, A. Saeed, and X. Hu, “Short-term wind speed interval prediction based on ensemble gru model,” IEEE Transactions on Sustainable Energy, vol. 11, no. 3, pp. 1370–1380, 2019.
  2. P. Wang, P. Wang, D. Wang, and B. Xue, “A conformal regressor with random forests for tropical cyclone intensity estimation,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–14, 2021.
  3. G. Morales and J. W. Sheppard, “Dual accuracy-quality-driven neural network for prediction interval generation,” IEEE Transactions on Neural Networks and Learning Systems, 2023.
  4. J. Lu, J. Ding, C. Liu, and T. Chai, “Hierarchical-bayesian-based sparse stochastic configuration networks for construction of prediction intervals,” IEEE Transactions on Neural Networks and Learning Systems, vol. 33, no. 8, pp. 3560–3571, 2021.
  5. V. Gómez-Verdejo, E. Parrado-Hernández, and M. Martínez-Ramón, “Adaptive sparse gaussian process,” IEEE Transactions on Neural Networks and Learning Systems, 2023.
  6. T. Melluish, C. Saunders, I. Nouretdinov, and V. Vovk, “Comparing the bayes and typicalness frameworks,” in European Conference on Machine Learning.   Springer, 2001, pp. 360–371.
  7. H. Papadopoulos, “Guaranteed coverage prediction intervals with gaussian process regression,” arXiv preprint arXiv:2310.15641, 2023.
  8. Z. Zhang, Y. Dong, and W.-C. Hong, “Long short-term memory-based twin support vector regression for probabilistic load forecasting,” IEEE Transactions on Neural Networks and Learning Systems, 2023.
  9. R. Koenker and G. Bassett Jr, “Regression quantiles,” Econometrica: Journal of the Econometric Society, pp. 33–50, 1978.
  10. A. N. Angelopoulos and S. Bates, “A gentle introduction to conformal prediction and distribution-free uncertainty quantification,” arXiv preprint arXiv:2107.07511, 2021.
  11. M. Fontana, G. Zeni, and S. Vantini, “Conformal prediction: a unified review of theory and new challenges,” Bernoulli, vol. 29, no. 1, pp. 1–23, 2023.
  12. H. Wang, X. Liu, I. Nouretdinov, and Z. Luo, “A comparison of three implementations of multi-label conformal prediction,” in International Symposium on Statistical Learning and Data Sciences.   Springer, 2015, pp. 241–250.
  13. S. Messoudi, S. Destercke, and S. Rousseau, “Copula-based conformal prediction for multi-target regression,” Pattern Recognition, vol. 120, p. 108101, 2021.
  14. J. Lei, A. Rinaldo, and L. Wasserman, “A conformal prediction approach to explore functional data,” Annals of Mathematics and Artificial Intelligence, vol. 74, no. 1, pp. 29–43, 2015.
  15. J. Diquigiovanni, M. Fontana, and S. Vantini, “Conformal prediction bands for multivariate functional data,” Journal of Multivariate Analysis, vol. 189, p. 104879, 2022.
  16. A. Fisch, T. Schuster, T. Jaakkola, and R. Barzilay, “Few-shot conformal prediction with auxiliary tasks,” in International Conference on Machine Learning.   PMLR, 2021, pp. 3329–3339.
  17. R. J. Tibshirani, R. Foygel Barber, E. Candes, and A. Ramdas, “Conformal prediction under covariate shift,” Advances in Neural Information Processing Systems, vol. 32, 2019.
  18. R. F. Barber, E. J. Candes, A. Ramdas, and R. J. Tibshirani, “Conformal prediction beyond exchangeability,” arXiv preprint arXiv:2202.13415, 2022.
  19. V. Jensen, F. M. Bianchi, and S. N. Anfinsen, “Ensemble conformalized quantile regression for probabilistic time series forecasting,” IEEE Transactions on Neural Networks and Learning Systems, 2022.
  20. M. Zaffran, O. Féron, Y. Goude, J. Josse, and A. Dieuleveut, “Adaptive conformal predictions for time series,” in International Conference on Machine Learning.   PMLR, 2022, pp. 25 834–25 866.
  21. S. Bates, A. Angelopoulos, L. Lei, J. Malik, and M. Jordan, “Distribution-free, risk-controlling prediction sets,” Journal of the ACM (JACM), vol. 68, no. 6, pp. 1–34, 2021.
  22. A. N. Angelopoulos, S. Bates, A. Fisch, L. Lei, and T. Schuster, “Conformal risk control,” arXiv preprint arXiv:2208.02814, 2022.
  23. A. Fisch, T. Schuster, T. Jaakkola, and R. Barzilay, “Conformal prediction sets with limited false positives,” arXiv preprint arXiv:2202.07650, 2022.
  24. M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poczos, R. R. Salakhutdinov, and A. J. Smola, “Deep sets,” Advances in neural information processing systems, vol. 30, 2017.
  25. H. Papadopoulos, “Inductive conformal prediction: Theory and application to neural networks,” in Tools in artificial intelligence.   IntechOpen, 2008.
  26. J. Lei, M. G’Sell, A. Rinaldo, R. J. Tibshirani, and L. Wasserman, “Distribution-free predictive inference for regression,” Journal of the American Statistical Association, vol. 113, no. 523, pp. 1094–1111, 2018.
  27. A. Ortega, P. Frossard, J. Kovačević, J. M. Moura, and P. Vandergheynst, “Graph signal processing: Overview, challenges, and applications,” Proceedings of the IEEE, vol. 106, no. 5, pp. 808–828, 2018.
  28. Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and S. Y. Philip, “A comprehensive survey on graph neural networks,” IEEE transactions on neural networks and learning systems, vol. 32, no. 1, pp. 4–24, 2020.
  29. C. Gupta, A. K. Kuchibhotla, and A. Ramdas, “Nested conformal prediction and quantile out-of-bag ensemble methods,” Pattern Recognition, vol. 127, p. 108496, 2022.
  30. A. N. Angelopoulos, S. Bates, E. J. Candès, M. I. Jordan, and L. Lei, “Learn then test: Calibrating predictive algorithms to achieve risk control,” arXiv preprint arXiv:2110.01052, 2021.
  31. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,” Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011.
  32. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” in Advances in Neural Information Processing Systems 32.   Curran Associates, Inc., 2019, pp. 8024–8035. [Online]. Available: http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
  33. A. Asuncion and D. Newman, “Uci machine learning repository,” 2007.
  34. J. Platt, “Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods,” Advances in Large Margin Classifiers, vol. 10, no. 3, pp. 61–74, 1999.
  35. J. Schmidhuber, “Deep learning in neural networks: An overview,” Neural Networks, vol. 61, pp. 85–117, 2015.
  36. L. Breiman, “Random forests,” Machine Learning, vol. 45, no. 1, pp. 5–32, 2001.
  37. S. Vannitsem, J. B. Bremnes, J. Demaeyer, G. R. Evans, J. Flowerdew, S. Hemri, S. Lerch, N. Roberts, S. Theis, A. Atencia et al., “Statistical postprocessing for weather forecasts: Review, challenges, and avenues in a big data world,” Bulletin of the American Meteorological Society, vol. 102, no. 3, pp. E681–E699, 2021.
  38. T. Palmer, “The ecmwf ensemble prediction system: Looking back (more than) 25 years and projecting forward 25 years,” Quarterly Journal of the Royal Meteorological Society, vol. 145, pp. 12–24, 2019.
  39. National Centers for Environmental Prediction, National Weather Service, NOAA, U.S. Department of Commerce, Japan Meteorological Agency, Japan, Met Office, Ministry of Defence, United Kingdom, China Meteorological Administration, China, Meteorological Service of Canada, Environment Canada, Korea Meteorological Administration, Republic of Korea, Meteo-France, France, European Centre for Medium-Range Weather Forecasts, and Bureau of Meteorology, Australia, “Thorpex interactive grand global ensemble (tigge) model tropical cyclone track data,” Boulder CO, 2008. [Online]. Available: https://doi.org/10.5065/D6GH9GSZ
  40. H. Hersbach, B. Bell, P. Berrisford, S. Hirahara, A. Horányi, J. Muñoz-Sabater, J. Nicolas, C. Peubey, R. Radu, D. Schepers et al., “The era5 global reanalysis,” Quarterly Journal of the Royal Meteorological Society, vol. 146, no. 730, pp. 1999–2049, 2020.
  41. Z. Li, F. Liu, W. Yang, S. Peng, and J. Zhou, “A survey of convolutional neural networks: Analysis, applications, and prospects,” IEEE Transactions on Neural Networks and Learning Systems, 2021.
  42. O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention.   Springer, 2015, pp. 234–241.
  43. P. Grönquist, C. Yao, T. Ben-Nun, N. Dryden, P. Dueben, S. Li, and T. Hoefler, “Deep learning for post-processing ensemble weather forecasts,” Philosophical Transactions of the Royal Society A, vol. 379, no. 2194, p. 20200092, 2021.
  44. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  45. A. N. Angelopoulos, A. P. Kohli, S. Bates, M. Jordan, J. Malik, T. Alshaabi, S. Upadhyayula, and Y. Romano, “Image-to-image regression with distribution-free uncertainty quantification and applications in imaging,” in International Conference on Machine Learning.   PMLR, 2022, pp. 717–730.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now