Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Uniform $\mathcal{C}^k$ Approximation of $G$-Invariant and Antisymmetric Functions, Embedding Dimensions, and Polynomial Representations (2403.01339v1)

Published 2 Mar 2024 in cs.LG and math.RT

Abstract: For any subgroup $G$ of the symmetric group $\mathcal{S}_n$ on $n$ symbols, we present results for the uniform $\mathcal{C}k$ approximation of $G$-invariant functions by $G$-invariant polynomials. For the case of totally symmetric functions ($G = \mathcal{S}_n$), we show that this gives rise to the sum-decomposition Deep Sets ansatz of Zaheer et al. (2018), where both the inner and outer functions can be chosen to be smooth, and moreover, the inner function can be chosen to be independent of the target function being approximated. In particular, we show that the embedding dimension required is independent of the regularity of the target function, the accuracy of the desired approximation, as well as $k$. Next, we show that a similar procedure allows us to obtain a uniform $\mathcal{C}k$ approximation of antisymmetric functions as a sum of $K$ terms, where each term is a product of a smooth totally symmetric function and a smooth antisymmetric homogeneous polynomial of degree at most $\binom{n}{2}$. We also provide upper and lower bounds on $K$ and show that $K$ is independent of the regularity of the target function, the desired approximation accuracy, and $k$.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (73)
  1. Anti-symmetric barron functions and their approximation with sums of determinants, 2023.
  2. Contextual deep learning-based audio-visual switching for speech enhancement in real-world environments. Information Fusion, 59:163–170, 2020. ISSN 1566-2535. doi: https://doi.org/10.1016/j.inffus.2019.08.008. URL https://www.sciencedirect.com/science/article/pii/S1566253518306018.
  3. Review of deep learning: concepts, cnn architectures, challenges, applications, future directions. Journal of Big Data, 8(1):53, Mar 2021. ISSN 2196-1115. doi: 10.1186/s40537-021-00444-8. URL https://doi.org/10.1186/s40537-021-00444-8.
  4. Graph neural network: A comprehensive review on non-euclidean space. IEEE Access, 9:60588–60606, 2021. doi: 10.1109/ACCESS.2021.3071274.
  5. Introduction to commutative algebra. Addison-Wesley Publishing Co., Reading, Mass.-London-Don Mills, Ont., 1969.
  6. A.R. Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information Theory, 39(3):930–945, 1993. doi: 10.1109/18.256500.
  7. François Bergeron. Algebraic combinatorics and coinvariant spaces. CMS Treatises in Mathematics. Canadian Mathematical Society, Ottawa, ON; A K Peters, Ltd., Wellesley, MA, 2009. ISBN 978-1-56881-324-0. doi: 10.1201/b10583. URL https://doi.org/10.1201/b10583.
  8. Emmanuel Briand. When is the algebra of multisymmetric polynomials generated by the elementary multisymmetric polynomials? Beiträge Algebra Geom., 45(2):353–368, 2004. ISSN 0138-4821.
  9. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478, 2021.
  10. Augustin Louis Cauchy. Œuvres complètes. Series 2. Volume 1. Cambridge Library Collection. Cambridge University Press, Cambridge, 1905. ISBN 978-1-108-00290-5. Reprint of the 1905 original.
  11. Lawrence Cayton. Algorithms for manifold learning. Technical Report CS2008-0923, UCSD, 2004.
  12. Representation theorem for multivariable totally symmetric functions, 2023.
  13. Exact and efficient representation of totally anti-symmetric functions, 2023.
  14. Fermionic neural-network states for ab-initio electronic structure. Nature Communications, 11(1), May 2020. ISSN 2041-1723. doi: 10.1038/s41467-020-15724-9. URL http://dx.doi.org/10.1038/s41467-020-15724-9.
  15. George V. Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 2:303–314, 1989. URL https://api.semanticscholar.org/CorpusID:3958369.
  16. Li Deng and Yang Liu. A Joint Introduction to Natural Language Processing and to Deep Learning. Springer Singapore, Singapore, 2018. ISBN 978-981-10-5209-5. doi: 10.1007/978-981-10-5209-5. URL https://doi.org/10.1007/978-981-10-5209-5.
  17. Recent advances in deep learning for speech research at microsoft. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 8604–8608, 2013. doi: 10.1109/ICASSP.2013.6639345.
  18. The role of permutation invariance in linear mode connectivity of neural networks, 2022.
  19. Exploiting spectro-temporal locality in deep learning based acoustic event detection. EURASIP Journal on Audio, Speech, and Music Processing, 2015(1):26, Sep 2015. ISSN 1687-4722. doi: 10.1186/s13636-015-0069-2. URL https://doi.org/10.1186/s13636-015-0069-2.
  20. Dropout vs. batch normalization: an empirical study of their impact to deep learning. Multimedia Tools and Applications, 79(19):12777–12815, May 2020. ISSN 1573-7721. doi: 10.1007/s11042-019-08453-9. URL https://doi.org/10.1007/s11042-019-08453-9.
  21. A. M. Garsia and M. Haiman. A remarkable q,t𝑞𝑡q,titalic_q , italic_t-Catalan sequence and q𝑞qitalic_q-Lagrange inversion. J. Algebraic Combin., 5(3):191–244, 1996. ISSN 0925-9899,1572-9192. doi: 10.1023/A:1022476211638. URL https://doi.org/10.1023/A:1022476211638.
  22. Multi-digit number recognition from street view imagery using deep convolutional neural networks. In ICLR2014, 2014.
  23. A combinatorial formula for the character of the diagonal coinvariants, 2004.
  24. Mark Haiman. Combinatorics, symmetric functions, and Hilbert schemes. In Current developments in mathematics, 2002, pages 39–111. Int. Press, Somerville, MA, 2003. ISBN 1-57146-102-7.
  25. Mark D. Haiman. Conjectures on the quotient ring by diagonal invariants. Journal of Algebraic Combinatorics, 3:17–76, 1994. URL https://api.semanticscholar.org/CorpusID:16526954.
  26. Solving many-electron schrödinger equation using deep neural networks. Journal of Computational Physics, 399:108929, December 2019. ISSN 0021-9991. doi: 10.1016/j.jcp.2019.108929. URL http://dx.doi.org/10.1016/j.jcp.2019.108929.
  27. Universal approximation of symmetric and anti-symmetric functions, 2022.
  28. Deep-neural-network solution of the electronic schrödinger equation. Nature Chemistry, 12(10):891–897, September 2020. ISSN 1755-4349. doi: 10.1038/s41557-020-0544-y. URL http://dx.doi.org/10.1038/s41557-020-0544-y.
  29. Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural Networks, 4(2):251–257, 1991. ISSN 0893-6080. doi: https://doi.org/10.1016/0893-6080(91)90009-T. URL https://www.sciencedirect.com/science/article/pii/089360809190009T.
  30. Marcus Hutter. On representing (anti)symmetric functions, 2020.
  31. Stefanie Jegelka. Theory of graph neural networks: Representation and learning, 2022.
  32. Richard Kane. Reflection groups and invariant theory, volume 5 of CMS Books in Mathematics/Ouvrages de Mathématiques de la SMC. Springer-Verlag, New York, 2001. ISBN 0-387-98979-X. doi: 10.1007/978-1-4757-3542-0. URL https://doi.org/10.1007/978-1-4757-3542-0.
  33. Speech recognition with deep learning. Journal of Physics: Conference Series, 1854(1):012047, apr 2021. doi: 10.1088/1742-6596/1854/1/012047. URL https://dx.doi.org/10.1088/1742-6596/1854/1/012047.
  34. Symmetric and antisymmetric kernels for machine learning problems in quantum physics and chemistry. Machine Learning: Science and Technology, 2(4):045016, August 2021. ISSN 2632-2153. doi: 10.1088/2632-2153/ac14ad. URL http://dx.doi.org/10.1088/2632-2153/ac14ad.
  35. Imagenet classification with deep convolutional neural networks. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL https://proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf.
  36. P. Laird and R. Saul. Automated feature extraction for supervised learning. In Proceedings of the First IEEE Conference on Evolutionary Computation. IEEE World Congress on Computational Intelligence, pages 674–679 vol.2, 1994. doi: 10.1109/ICEC.1994.349977.
  37. An introduction to deep learning in natural language processing: Models, techniques, and tools. Neurocomputing, 470:443–456, 2022. ISSN 0925-2312. doi: https://doi.org/10.1016/j.neucom.2021.05.103. URL https://www.sciencedirect.com/science/article/pii/S0925231221010997.
  38. A survey of convolutional neural networks: Analysis, applications, and prospects. IEEE Transactions on Neural Networks and Learning Systems, 33(12):6999–7019, 2022. doi: 10.1109/TNNLS.2021.3084827.
  39. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.
  40. Di Luo and Bryan K. Clark. Backflow transformations via neural networks for quantum many-body wave functions. Physical Review Letters, 122(22), June 2019. ISSN 1079-7114. doi: 10.1103/physrevlett.122.226401. URL http://dx.doi.org/10.1103/PhysRevLett.122.226401.
  41. Hideyuki Matsumura. Commutative ring theory, volume 8 of Cambridge Studies in Advanced Mathematics. Cambridge University Press, Cambridge, second edition, 1989. ISBN 0-521-36764-6. Translated from the Japanese by M. Reid.
  42. Set Aggregation Network as a Trainable Pooling Layer, page 419–431. Springer International Publishing, 2019. ISBN 9783030367114. doi: 10.1007/978-3-030-36711-4˙35. URL http://dx.doi.org/10.1007/978-3-030-36711-4_35.
  43. On the gröbner bases of some symmetric systems and their application to coding theory. Journal of Symbolic Computation, 35(2):177–194, 2003. ISSN 0747-7171. doi: https://doi.org/10.1016/S0747-7171(02)00131-1. URL https://www.sciencedirect.com/science/article/pii/S0747717102001311.
  44. Deep learning for multigrade brain tumor classification in smart healthcare systems: A prospective survey. IEEE Transactions on Neural Networks and Learning Systems, 32(2):507–522, 2021. doi: 10.1109/TNNLS.2020.2995800.
  45. Janossy pooling: Learning deep permutation-invariant functions for variable-size inputs, 2019.
  46. Leopoldo Nachbin. Sur les algebres denses de fonctions différentiables sur une variété. Comptes Rendus de l’Académie des Sciences de Paris, 228:1549–1551, 1949.
  47. A survey of the usages of deep learning for natural language processing. IEEE Transactions on Neural Networks and Learning Systems, 32(2):604–624, 2021. doi: 10.1109/TNNLS.2020.2979670.
  48. Ab initio solution of the many-electron schrödinger equation with deep neural networks. Physical Review Research, 2(3), September 2020. ISSN 2643-1564. doi: 10.1103/physrevresearch.2.033429. URL http://dx.doi.org/10.1103/PhysRevResearch.2.033429.
  49. An extension of nachbin’s theorem to differentiable functions on banach spaces with the approximation property. Arkiv för Matematik, 14:251–258, 1976. URL https://api.semanticscholar.org/CorpusID:120704067.
  50. Pointnet: Deep learning on point sets for 3d classification and segmentation, 2017a.
  51. Pointnet++: Deep hierarchical feature learning on point sets in a metric space, 2017b.
  52. Multi-transformer: A new neural network-based architecture for forecasting s&p volatility. Mathematics, 9(15), 2021. ISSN 2227-7390. doi: 10.3390/math9151794. URL https://www.mdpi.com/2227-7390/9/15/1794.
  53. Bruce E. Sagan. The symmetric group, volume 203 of Graduate Texts in Mathematics. Springer-Verlag, New York, second edition, 2001. ISBN 0-387-95067-2. doi: 10.1007/978-1-4757-6804-6. URL https://doi.org/10.1007/978-1-4757-6804-6. Representations, combinatorial algorithms, and symmetric functions.
  54. Transformer-based neural network for answer selection in question answering. IEEE Access, 7:26146–26156, 2019. doi: 10.1109/ACCESS.2019.2900753.
  55. Mastering the game of go without human knowledge. Nature, 550(7676):354–359, Oct 2017. ISSN 1476-4687. doi: 10.1038/nature24270. URL https://doi.org/10.1038/nature24270.
  56. On deep set learning and the choice of aggregations. In Igor V. Tetko, Věra Kůrková, Pavel Karpov, and Fabian Theis, editors, Artificial Neural Networks and Machine Learning – ICANN 2019: Theoretical Neural Computation, pages 444–457, Cham, 2019. Springer International Publishing. ISBN 978-3-030-30487-4.
  57. Convolutional neural networks for radiologic images: A radiologist’s guide. Radiology, 290(3):590–606, 2019. doi: 10.1148/radiol.2018180547. URL https://doi.org/10.1148/radiol.2018180547. PMID: 30694159.
  58. Phases of two-dimensional spinless lattice fermions with first-quantized deep neural-network quantum states. Physical Review B, 102(20), November 2020. ISSN 2469-9969. doi: 10.1103/physrevb.102.205122. URL http://dx.doi.org/10.1103/PhysRevB.102.205122.
  59. Evolutionary programming based deep learning feature selection and network construction for visual data classification. Information Systems Frontiers, 22(5):1053–1066, Oct 2020. ISSN 1572-9419. doi: 10.1007/s10796-020-10023-6. URL https://doi.org/10.1007/s10796-020-10023-6.
  60. Listening while speaking: Speech chain by deep learning. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 301–308, 2017. doi: 10.1109/ASRU.2017.8268950.
  61. Universal approximation of functions on sets, 2021.
  62. Nolan Wallach. The representation of GL(k) on the alternants of minimal degree for the diagonal action of Sn on k copies of the permutation representation. 09 2021. URL https://www.researchgate.net/publication/354389796_The_representation_of_GLk_on_the_alternants_of_minimal_degree_for_the_diagonal_action_of_S_n_on_k_copies_of_the_the_permutation_representation.
  63. Nolan R. Wallach. Geometric invariant theory. Universitext. Springer, Cham, 2017. ISBN 978-3-319-65905-3; 978-3-319-65907-7. doi: 10.1007/978-3-319-65907-7. URL https://doi.org/10.1007/978-3-319-65907-7. Over the real and complex numbers.
  64. Deep learning based autonomous vehicle super resolution doa estimation for safety driving. IEEE Transactions on Intelligent Transportation Systems, 22(7):4301–4315, 2021. doi: 10.1109/TITS.2020.3009223.
  65. Understanding and scheduling weight decay, 2022. URL https://openreview.net/forum?id=J7V_4aauV6B.
  66. How powerful are graph neural networks?, 2019.
  67. Deep sets, 2018.
  68. Deep convolutional neural networks to predict cardiovascular risk from computed tomography. Nature Communications, 12(1):715, Jan 2021. ISSN 2041-1723. doi: 10.1038/s41467-021-20966-2. URL https://doi.org/10.1038/s41467-021-20966-2.
  69. Point-x: A spatial-locality-aware architecture for energy-efficient graph-based point-cloud deep learning. In MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO ’21, page 1078–1090, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450385572. doi: 10.1145/3466752.3480081. URL https://doi.org/10.1145/3466752.3480081.
  70. Deep potential molecular dynamics: A scalable model with the accuracy of quantum mechanics. Physical Review Letters, 120(14), April 2018a. ISSN 1079-7114. doi: 10.1103/physrevlett.120.143001. URL http://dx.doi.org/10.1103/PhysRevLett.120.143001.
  71. End-to-end symmetry preserving inter-atomic potential energy model for finite and extended systems, 2018b.
  72. Deep learning for autonomous vehicle and pedestrian interaction safety. Safety Science, 145:105479, 2022. ISSN 0925-7535. doi: https://doi.org/10.1016/j.ssci.2021.105479. URL https://www.sciencedirect.com/science/article/pii/S0925753521003222.
  73. Towards antisymmetric neural ansatz separation, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Soumya Ganguly (5 papers)
  2. Khoa Tran (14 papers)
  3. Rahul Sarkar (11 papers)

Summary

We haven't generated a summary for this paper yet.