The universal approximation theorem for complex-valued neural networks

Abstract: We generalize the classical universal approximation theorem for neural networks to the case of complex-valued neural networks. Precisely, we consider feedforward networks with a complex activation function $σ: \mathbb{C} \to \mathbb{C}$ in which each neuron performs the operation $$\mathbb{C}^N \to \mathbb{C}, z \mapsto σ(b + w^T z)$$ with weights $w \in \mathbb{C}^N$ and a bias $b \in \mathbb{C}$, and with $σ$ applied componentwise. We completely characterize those activation functions $σ$ for which the associated complex networks have the universal approximation property, meaning that they can uniformly approximate any continuous function on any compact subset of $\mathbb{C}^d$ arbitrarily well. Unlike the classical case of real networks, the set of "good activation functions" which give rise to networks with the universal approximation property differs significantly depending on whether one considers deep networks or shallow networks: For deep networks with at least two hidden layers, the universal approximation property holds as long as $σ$ is neither a polynomial, a holomorphic function, or an antiholomorphic function. Shallow networks, on the other hand, are universal if and only if the real part or the imaginary part of $σ$ is not a polyharmonic function.

• The paper demonstrates that complex-valued neural networks can universally approximate continuous functions when using non-polynomial, non-holomorphic activation functions.
• It establishes distinct criteria for shallow and deep architectures, highlighting a broader design space for activation functions in the complex domain.
• The findings provide a theoretical framework paving the way for practical CVNN implementations in applications like signal processing and MRI imaging.

Universal Approximation Theorem for Complex-Valued Neural Networks

Complex-valued neural networks (CVNNs) offer a promising direction for expanding the applicability of neural architectures, particularly in domains where inputs naturally involve complex numbers, such as in signal processing and MRI imaging. The paper "The universal approximation theorem for complex-valued neural networks" (2012.03351) addresses a generalization of the universal approximation theorem to complex-valued networks, detailing the conditions under which a complex neural network can approximate any continuous function within specified bounds.

Characterization of Activation Functions

The distinguishing feature of CVNNs, where both weights and activation functions extend over the complex plane, introduces unique challenges and opportunities compared to their real-valued counterparts. This work rigorously defines which complex activation functions enable networks to achieve universal approximation. For deep networks with at least two hidden layers, the paper identifies conditions under which they universally approximate any continuous function on $C^d$:

• Deep Networks: The universal approximation property holds as long as the activation function is neither a polynomial nor purely holomorphic or antiholomorphic. Activation functions that avoid these restrictions can enable a network to densely cover the space of continuous functions, achieving practical universality.
• Shallow Networks: These only achieve universality if neither the real nor the imaginary part of the activation function is polyharmonic. This reflects a stricter constraint on approximation ability at lower network depths.

Insights and Implications

The implications of these results are profound for theoretical and practical advancements in CVNNs:

• Variation in Activation Function Properties: The characteristics required vary distinctly between shallow and deep networks, underscoring the increased flexibility of deeper architectures. This insight informs architectural decisions when developing complex-valued models for specific applications.
• Complex vs. Real-Valued Domains: The findings underscore unique properties in the complex domain, revealing that broader classes of activation functions are permissible compared to real-valued networks. This expands the potential architecture design space and supports increased model complexity without sacrificing approximation capabilities.
• Applications in Complex Domains: Effectiveness in domains such as quantum computing and advanced imaging techniques could be significantly influenced by the appropriate selection and design of complex activation functions, enhancing computational efficiency and accuracy.

Future Directions

This theoretical foundation invites several avenues for further exploration:

• Practical Implementation: Developing efficient training algorithms for CVNNs leveraging universally approximating activation functions.
• Exploring Holomorphic Functions: Investigating the boundary conditions where nearly holomorphic functions might offer approximation advantages without fully satisfying universality, possibly bridging gaps for specific applications.
• Cross-Domain Network Structures: Incorporating cross-domain architectures where complex and real-valued components interact seamlessly, enhancing model versatility and performance.

Conclusion

By extending the universal approximation theorem to complex-valued networks, this research deepens our understanding of neural approximators in multidimensional complex domains. The results offer practical guidelines for designing networks that leverage the richness of the complex plane, broadening the horizon for future innovations in neural network design and application.