Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
124 tokens/sec
GPT-4o
8 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Consistency of semi-supervised learning, stochastic tug-of-war games, and the p-Laplacian (2401.07463v2)

Published 15 Jan 2024 in math.ST, cs.LG, cs.NA, math.AP, math.NA, math.PR, and stat.TH

Abstract: In this paper we give a broad overview of the intersection of partial differential equations (PDEs) and graph-based semi-supervised learning. The overview is focused on a large body of recent work on PDE continuum limits of graph-based learning, which have been used to prove well-posedness of semi-supervised learning algorithms in the large data limit. We highlight some interesting research directions revolving around consistency of graph-based semi-supervised learning, and present some new results on the consistency of $p$-Laplacian semi-supervised learning using the stochastic tug-of-war game interpretation of the $p$-Laplacian. We also present the results of some numerical experiments that illustrate our results and suggest directions for future work.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (124)
  1. M. Alamgir and U. V. Luxburg. Phase transition in the family of p-resistances. In Advances in Neural Information Processing Systems, pages 379–387, 2011.
  2. R. K. Ando and T. Zhang. Learning on graph with Laplacian regularization. In Advances in neural information processing systems, pages 25–32, 2007.
  3. S. Armstrong and C. Smart. A finite difference approach to the infinity Laplace equation and tug-of-war games. Transactions of the American Mathematical Society, 364(2):595–636, 2012.
  4. A tour of the theory of absolutely minimizing functions. Bulletin of the American mathematical society, 41(4):439–505, 2004.
  5. Gradient and Lipschitz estimates for tug-of-war type games. SIAM Journal on Mathematical Analysis, 53(2):1295–1319, 2021.
  6. A. Azad. Learning Label Initialization for Time-Dependent Harmonic Extension. arXiv preprint arXiv:2205.01358, 2022.
  7. Optimal control and viscosity solutions of Hamilton-Jacobi-Bellman equations, volume 12. Springer, 1997.
  8. Regularization and semi-supervised learning on large graphs. In Learning Theory: 17th Annual Conference on Learning Theory, COLT 2004, Banff, Canada, July 1-4, 2004. Proceedings 17, pages 624–638. Springer, 2004.
  9. M. Belkin and P. Niyogi. Using manifold stucture for partially labeled classification. Advances in neural information processing systems, 15, 2002.
  10. M. Belkin and P. Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation, 15(6):1373–1396, 2003.
  11. M. Belkin and P. Niyogi. Semi-supervised learning on Riemannian manifolds. Machine learning, 56:209–239, 2004.
  12. Label Propagation and Quadratic Criterion, pages 193–216. MIT Press, semi-supervised learning edition, January 2006.
  13. Concentration Inequalities: A Nonasymptotic Theory of Independence. Univ. Press, 2013.
  14. Simplified energy landscape for modularity using total variation. SIAM Journal on Applied Mathematics, 78(5):2439–2464, 2018.
  15. N. Bridle and X. Zhu. p-voltages: Laplacian regularization for semi-supervised learning on high-dimensional data. In Eleventh Workshop on Mining and Learning with Graphs (MLG2013), 2013.
  16. Utilizing Contrastive Learning for Graph-Based Active Learning of SAR Data. SPIE Defense and Commercial Sensing: Algorithms for Synthetic Aperture Radar Imagery XXX, 2023.
  17. Uniform Convergence Rates for Lipschitz Learning on Graphs. IMA Journal of Numerical Analysis, 2022.
  18. Ratio convergence rates for Euclidean first-passage percolation: Applications to the graph infinity Laplacian. To appear in Annals of Applied Probability, 2023.
  19. J. Calder. The game theoretic p-Laplacian and semi-supervised learning with few labels. Nonlinearity, 32(1), 2018.
  20. J. Calder. Consistency of Lipschitz learning with infinite unlabeled data and finite labeled data. SIAM Journal on Mathematics of Data Science, 1(4):780–812, 2019.
  21. Graph-Based semi-supervised learning with Poisson equations. In preparation, 2023.
  22. Poisson Learning: Graph based semi-supervised learning at very low label rates. Proceedings of the 37th International Conference on Machine Learning, PMLR, 119:1306–1316, 2020.
  23. J. Calder and M. Ettehad. Hamilton-Jacobi equations on graphs with applications to semi-supervised learning and data depth. Journal of Machine Learning Research, 23(318):1–62, 2022.
  24. J. Calder and N. García Trillos. Improved spectral convergence rates for graph Laplacians on ε𝜀\varepsilonitalic_ε-graphs and k-NN graphs. Applied and Computational Harmonic Analysis, 60:123–175, 2022.
  25. Lipschitz regularity of graph Laplacians on random data clouds. SIAM Journal on Mathematical Analysis, 54(1):1169–1222, 2022.
  26. J. Calder and D. Slepčev. Properly-weighted graph Laplacian for semi-supervised learning. Applied Mathematics and Optimization, 82:1111–1159, 2020.
  27. Rates of convergence for Laplacian semi-supervised learning with low labeling rates. Research in Mathematical Sciences special issue on PDE methods for machine learning, 10(10), 2023.
  28. Semi-supervised learning. MIT, 2006.
  29. Novel Batch Active Learning Approach and Its Application on the Synthetic Aperture Radar Datasets. SPIE Defense and Commercial Sensing: Algorithms for Synthetic Aperture Radar Imagery XXX (Best Student Paper), 2023.
  30. A mixed problem for the infinity Laplacian via tug-of-war games. Calculus of Variations and Partial Differential Equations, 34(3):307–320, 2009.
  31. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  32. R. R. Coifman and S. Lafon. Diffusion maps. Applied and computational harmonic analysis, 21(1):5–30, 2006.
  33. On the equivalence of decoupled graph convolution network and label propagation. In Proceedings of the Web Conference 2021, pages 3651–3662, 2021.
  34. D. L. Donoho and C. Grimes. Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data. Proceedings of the National Academy of Sciences, 100(10):5591–5596, 2003.
  35. Large data and zero noise limits of graph-based semi-supervised learning algorithms. Applied and Computational Harmonic Analysis, 49(2):655–697, 2020.
  36. Asymptotic behavior of ℓpsubscriptℓ𝑝\ell_{p}roman_ℓ start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT-based Laplacian regularization in semi-supervised learning. In Conference on Learning Theory, pages 879–906, 2016.
  37. On the game p𝑝pitalic_p-Laplacian on weighted graphs with applications in image processing and data clustering. European Journal of Applied Mathematics, 28(6):922–948, 2017.
  38. Nonlocal PDEs on graphs: From tug-of-war games to unified interpolation on images and point clouds. Journal of Mathematical Imaging and Vision, 57(3):381–401, 2017.
  39. On the p𝑝pitalic_p-Laplacian and ∞\infty∞-Laplacian on graphs with applications in image and data processing. SIAM Journal on Imaging Sciences, 8(4):2412–2451, 2015.
  40. Tug of War games and PDEs on graphs with applications in image and high dimensional data processing. Scientific Reports, 13(1):6045, 2023.
  41. Deep semi-supervised label propagation for SAR image classification. SPIE Defense and Commercial Sensing: Algorithms for Synthetic Aperture Radar Imagery XXX, 2023.
  42. L. Evans. Partial Differential Equations (Graduate Studies in Mathematics, V. 19) GSM/19. American Mathematical Society, June 1998.
  43. L. C. Evans. A new proof of local C1, α𝛼\alphaitalic_α regularity for solutions of certain degenerate elliptic pde. Journal of Differential Equations, 45(3):356–373, 1982.
  44. Analysis and algorithms for Lp-based semi-supervised learning on graphs. Applied and Computational Harmonic Analysis, 60:77–122, 2022.
  45. Multiclass data segmentation using diffuse interface methods on graphs. IEEE transactions on pattern analysis and machine intelligence, 36(8):1600–1613, 2014.
  46. Error estimates for spectral convergence of the graph Laplacian on random geometric graphs toward the Laplace–Beltrami operator. Foundations of Computational Mathematics, 20(4):827–887, 2020.
  47. N. García Trillos and D. Slepčev. Continuum limit of total variation on point clouds. Archive for rational mechanics and analysis, 220:193–241, 2016.
  48. Predict then Propagate: Graph Neural Networks meet Personalized PageRank. In International Conference on Learning Representations, 2018.
  49. Deep learning. MIT press, 2016.
  50. Nonlocal p𝑝pitalic_p-Laplacian Variational problems on graphs. arXiv:1810.12817, 2018.
  51. Semisupervised alignment of manifolds. In International Workshop on Artificial Intelligence and Statistics, pages 120–127. PMLR, 2005.
  52. J. Han. Time-dependent tug-of-war games and normalized parabolic p-Laplace equations. Nonlinear Analysis, 214:112542, 2022.
  53. Manifold-ranking based image retrieval. In Proceedings of the 12th annual ACM international conference on Multimedia, pages 9–16. ACM, 2004.
  54. Generalized manifold-ranking-based image retrieval. IEEE Transactions on image processing, 15(10):3170–3177, 2006.
  55. Graph Laplacians and their convergence on random neighborhood graphs. Journal of Machine Learning Research, 8(6), 2007.
  56. From Graphs to Manifolds-Weak and Strong Pointwise Consistency of Graph Laplacians. In COLT, volume 3559, pages 470–485. Springer, 2005.
  57. Spectral analysis of weighted Laplacians arising in data clustering. Applied and Computational Harmonic Analysis, 56:189–249, 2022.
  58. A method based on total variation for network modularity optimization using the MBO scheme. SIAM Journal on Applied Mathematics, 73(6):2224–2246, 2013.
  59. Combining label propagation and simple models out-performs graph neural networks. arXiv preprint arXiv:2010.13993, 2020.
  60. Auction dynamics: A volume constrained MBO scheme. Journal of Computational Physics, 354:288–310, 2018.
  61. M. Ji and J. Han. A Variance Minimization Criterion to Active Learning on Graphs. In Artificial Intelligence and Statistics, pages 556–564, Mar. 2012.
  62. Semi-supervised learning via sparse label propagation. arXiv preprint arXiv:1612.01414, 2016.
  63. D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
  64. R. Kohn and S. Serfaty. A deterministic-control-based approach motion by curvature. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences, 59(3):344–407, 2006.
  65. Learning multiple layers of features from tiny images. University of Toronto, 2009.
  66. Algorithms for Lipschitz learning on graphs. In Conference on Learning Theory, pages 1190–1223, 2015.
  67. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
  68. Graph-based semi-supervised learning with multi-modality propagation for large-scale image datasets. Journal of visual communication and image representation, 24(3):295–302, 2013.
  69. G. Leoni. A first course in Sobolev spaces. American Mathematical Soc., 2017.
  70. M. Lewicka. Random Tug of War games for the p-Laplacian: 1<p<∞1𝑝1<p<\infty1 < italic_p < ∞. arXiv preprint arXiv:1810.03413, 2018.
  71. M. Lewicka. A course on Tug-of-War games with random noise. Springer, 2020.
  72. M. Lewicka. Non-local Tug-of-War with noise for the geometric fractional p-Laplacian. Adv. Differential Equations, 27(1-2):31–76, 2022.
  73. M. Lewicka and J. J. Manfredi. Game theoretical methods in PDEs. Bollettino dell’Unione Matematica Italiana, 7(3):211–216, 2014.
  74. M. Lewicka and J. J. Manfredi. The obstacle problem for the p-laplacian via optimal stopping of tug-of-war games. Probability Theory and Related Fields, 167:349–378, 2017.
  75. M. Lewicka and Y. Peres. The Robin mean value equation I: A random walk approach to the third boundary value problem. Potential Analysis, pages 1–32, 2022.
  76. M. Lewicka and Y. Peres. The Robin mean value equation II: asymptotic Hölder regularity. Potential Analysis, pages 1–35, 2022.
  77. P. Lindqvist. Notes on the stationary p-Laplace equation. Springer, 2019.
  78. Harnack’s inequality for p-harmonic functions via stochastic games. Communications in Partial Differential Equations, 38(11):1985–2003, 2013.
  79. U. v. Luxburg and O. Bousquet. Distance-based classification with Lipschitz functions. Journal of Machine Learning Research, 5(Jun):669–695, 2004.
  80. ΣΣ\Sigmaroman_Σ-Optimality for Active Learning on Gaussian Random Fields. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, pages 2751–2759. Curran Associates, Inc., 2013.
  81. X. Mai. A random matrix analysis and improvement of semi-supervised learning for large dimensional data. Journal of Machine Learning Research, 19(79):1–27, 2018.
  82. X. Mai and R. Couillet. Random matrix-inspired improved semi-supervised learning on graphs. In International Conference on Machine Learning, 2018.
  83. X. Mai and R. Couillet. Consistent semi-supervised graph regularization for high dimensional data. The Journal of Machine Learning Research, 22(1):4181–4228, 2021.
  84. An asymptotic mean value characterization for p-harmonic functions. Proceedings of the American Mathematical Society, 138(3):881–889, 2010.
  85. Nonlinear elliptic partial differential equations and p-harmonic functions on graphs. Differential Integral Equations, 28(1-2):79–102, 2015.
  86. Dynamic programming principle for tug-of-war games with noise. ESAIM: Control, Optimisation and Calculus of Variations, 18(1):81–90, 2012.
  87. On the definition and properties of p𝑝pitalic_p-harmonious functions. Annali della Scuola Normale Superiore di Pisa-Classe di Scienze, 11(2):215–241, 2012.
  88. A semi-supervised heat kernel pagerank MBO algorithm for data classification. Communications in Mathematical Sciences, 16(5):1241–1265, 2018.
  89. Diffuse interface methods for multiclass segmentation of high-dimensional data. Applied Mathematics Letters, 33:29–34, 2014.
  90. An MBO scheme on graphs for classification and image processing. SIAM Journal on Imaging Sciences, 6(4):1903–1930, 2013.
  91. Motion of multiple junctions: A level set approach. Journal of computational physics, 112(2):334–363, 1994.
  92. Graph-based active learning for semi-supervised classification of SAR data. SPIE Defense and Commercial Sensing: Algorithms for Synthetic Aperture Radar Imagery XXIX, 12095, 2022.
  93. K. Miller and A. L. Bertozzi. Model-change active learning in graph-based semi-supervised learning. arXiv preprint arXiv:2110.07739, 2021.
  94. K. Miller and J. Calder. Poisson Reweighted Laplacian Uncertainty Sampling for Graph-based Active Learning. To appear in SIAM Journal on Mathematics of Data Science, 2023.
  95. J. M. Murphy and M. Maggioni. Unsupervised Clustering and Active Learning of Hyperspectral Images With Nonlinear Diffusion. IEEE Transactions on Geoscience and Remote Sensing, 57(3):1829–1845, Mar. 2019.
  96. Semi-supervised learning with the graph Laplacian: The limit of infinite unlabelled data. Advances in neural information processing systems, 22:1330–1338, 2009.
  97. A. Oberman. A convergent difference scheme for the infinity Laplacian: construction of absolutely minimizing Lipschitz extensions. Mathematics of computation, 74(251):1217–1230, 2005.
  98. A. M. Oberman. Finite difference methods for the infinity Laplace and p-Laplace equations. Journal of Computational and Applied Mathematics, 254:65–80, 2013.
  99. M. Parviainen. Notes on tug-of-war games and the p-Laplace equation. SpringerBriefs on PDEs and Data Science, 2024.
  100. Biased tug-of-war, the biased infinity Laplacian, and comparison with exponential cones. Calculus of Variations and Partial Differential Equations, 38(3-4):541–564, 2010.
  101. Tug-of-war and the infinity Laplacian. Journal of the American Mathematical Society, 22(1):167–210, 2009.
  102. Tug-of-war with noise: A game-theoretic view of the p𝑝pitalic_p-Laplacian. Duke Mathematical Journal, 145(1):91–120, 2008.
  103. Uncertainty quantification for semi-supervised multi-class classification in image processing and ego-motion analysis of body-worn videos. Image Processing: Algorithms and Systems, 2019.
  104. Laplacenet: A hybrid graph-energy neural network for deep semisupervised classification. IEEE Transactions on Neural Networks and Learning Systems, 2022.
  105. B. Settles. Active learning literature survey. University of Wisconsin-Madison Department of Computer Sciences, 2009.
  106. Weighted nonlocal Laplacian on interpolation from sparse data. Journal of Scientific Computing, 73(2-3):1164–1177, 2017.
  107. D. Slepcev and M. Thorpe. Analysis of p-laplacian regularization in semisupervised learning. SIAM Journal on Mathematical Analysis, 51(3):2085–2120, 2019.
  108. Geometric structure of graph Laplacian embeddings. The Journal of Machine Learning Research, 22(1):2934–2988, 2021.
  109. U. von Luxburg. A tutorial on spectral clustering. Statistics and Computing, 17(4):395–416, Dec. 2007.
  110. Dynamic label propagation for semi-supervised multi-class multi-label classification. In Proceedings of the IEEE international conference on computer vision, pages 425–432, 2013.
  111. Multi-manifold ranking: Using multiple features for better image retrieval. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 449–460. Springer, 2013.
  112. A. Weihs and M. Thorpe. Consistency of Fractional Graph-Laplacian Regularization in Semi-Supervised Learning with Finite Labels. arXiv preprint arXiv:2303.07818, 2023.
  113. D. Williams. Probability with martingales. Cambridge university press, 1991.
  114. Efficient manifold ranking for image retrieval. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, pages 525–534. ACM, 2011.
  115. Saliency detection via graph-based manifold ranking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3166–3173, 2013.
  116. A survey on deep semi-supervised learning. IEEE Transactions on Knowledge and Data Engineering, 2022.
  117. Learning with local and global consistency. In Advances in neural information processing systems, pages 321–328, 2004.
  118. Learning from labeled and unlabeled data on a directed graph. In Proceedings of the 22nd international conference on Machine learning, pages 1036–1043. ACM, 2005.
  119. D. Zhou and B. Schölkopf. Learning from labeled and unlabeled data using random walks. In Joint Pattern Recognition Symposium, pages 237–244. Springer, 2004.
  120. D. Zhou and B. Schölkopf. Regularization on discrete spaces. In Joint Pattern Recognition Symposium, pages 361–368. Springer, 2005.
  121. Ranking on data manifolds. In Advances in neural information processing systems, pages 169–176, 2004.
  122. X. Zhou and M. Belkin. Semi-supervised learning by higher order regularization. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 892–900. JMLR Workshop and Conference Proceedings, 2011.
  123. Semi-supervised learning using Gaussian fields and harmonic functions. In Proceedings of the 20th International conference on Machine learning (ICML-03), pages 912–919, 2003.
  124. Combining Active Learning and Semi-Supervised Learning Using Gaussian Fields and Harmonic Functions. In International Conference on Machine Learning (ICML) 2003 workshop on The Continuum from Labeled to Unlabeled Data in Machine Learning and Data Mining, pages 58–65, 2003.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now