2000 character limit reached
Comprehensive Survey of Complex-Valued Neural Networks: Insights into Backpropagation and Activation Functions (2407.19258v1)
Published 27 Jul 2024 in cs.LG
Abstract: Artificial neural networks (ANNs), particularly those employing deep learning models, have found widespread application in fields such as computer vision, signal processing, and wireless communications, where complex numbers are crucial. Despite the prevailing use of real-number implementations in current ANN frameworks, there is a growing interest in developing ANNs that utilize complex numbers. This paper presents a comprehensive survey of recent advancements in complex-valued neural networks (CVNNs), focusing on their activation functions (AFs) and learning algorithms. We delve into the extension of the backpropagation algorithm to the complex domain, which enables the training of neural networks with complex-valued inputs, weights, AFs, and outputs. This survey considers three complex backpropagation algorithms: the complex derivative approach, the partial derivatives approach, and algorithms incorporating the Cauchy-Riemann equations. A significant challenge in CVNN design is the identification of suitable nonlinear Complex Valued Activation Functions (CVAFs), due to the conflict between boundedness and differentiability over the entire complex plane as stated by Liouville theorem. We examine both fully complex AFs, which strive for boundedness and differentiability, and split AFs, which offer a practical compromise despite not preserving analyticity. This review provides an in-depth analysis of various CVAFs essential for constructing effective CVNNs. Moreover, this survey not only offers a comprehensive overview of the current state of CVNNs but also contributes to ongoing research and development by introducing a new set of CVAFs (fully complex, split and complex amplitude-phase AFs).
Sponsored by Paperpile, the PDF & BibTeX manager trusted by top AI labs.
Get 30 days freePaper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a single, concrete list of what remains missing, uncertain, or unexplored in the paper, framed so future researchers can act on them:
- Lack of standardized empirical benchmarks: No comprehensive, reproducible comparisons of CVNNs vs. RVNNs across diverse tasks (e.g., OFDM channel estimation, MRI reconstruction, radar/sonar classification, speech enhancement) with common datasets, unified training protocols, and metrics that explicitly measure amplitude- and phase-fidelity.
- Unclear computational trade-offs: No systematic analysis of training/inference time, memory footprint, and energy cost for CVNNs (complex multiplications, complex convolutions, complex BN) vs. RVNN baselines; no guidance on where CVNNs become computationally advantageous.
- Convergence and stability of complex backpropagation: The three CBP variants (complex derivative, partial derivatives, CR-based) lack formal convergence guarantees under realistic settings (non-holomorphic AFs, stochastic optimization, nonconvex losses), including conditions on step sizes, Lipschitz constants, and noise.
- Gradients for real-valued losses with complex parameters: Insufficient rigorous treatment of how Wirtinger derivatives interact with real-valued cost functions; need formal equivalence proofs between (i) Wirtinger calculus, (ii) split real–imaginary gradients, and (iii) implementation-level autodiff in major frameworks.
- Optimization algorithms in the complex domain: No validated complex-valued versions of Adam, RMSProp, AdaBound, and adaptive gradient clipping tailored to complex parameters (bias correction, moment estimates on complex vectors, handling correlations between real and imaginary parts).
- Complex weight initialization: Missing principled initialization strategies for complex weights (e.g., distributions over magnitude/phase, unit-modulus constraints, polar vs. Cartesian parameterizations) and their empirical impact on training speed, stability, and generalization.
- Regularization methods for CVNNs: No systematic paper of L2/L1 norms for complex weights (Cartesian vs. polar forms), phase- or amplitude-specific penalties, complex dropout, spectral regularization, or CR-consistency penalties for non-holomorphic layers.
- Complex normalization layers: BatchNorm/LayerNorm/GroupNorm in complex domain are under-specified; need definitions and empirical evaluation for complex mean/variance, full 2×2 covariance normalization over real and imaginary parts, and stability effects.
- Expressivity and universal approximation: Absent theoretical results that characterize the function classes approximable by CVNNs with specific CVAF families (fully complex, split, amplitude–phase), including how analyticity constraints and Liouville’s theorem shape approximation power.
- Activation function properties: The newly introduced CVAFs lack proofs and measurements of key properties (boundedness regimes, differentiability, Lipschitz continuity, monotonicity in modulus/phase, CR-equation compliance, gradient variance) and their downstream impact on optimization dynamics.
- Training with non-analytic AFs: For split AFs and other non-holomorphic choices, the paper does not resolve how gradient definitions affect stationary points, bias in learning dynamics, or error propagation—nor provide diagnostics or remedies (e.g., CR-penalty terms).
- Phase preservation metrics: No task-agnostic metrics and evaluation protocols that quantify phase distortion across layers (e.g., mean phase error, circular correlation, amplitude–phase decomposition losses) and relate them to downstream performance.
- When to prefer fully complex vs. split architectures: No ablation framework or decision criteria that link data characteristics (phase structure, circular statistics, SNR) to architectural choices (fully complex CVAFs vs. split AFs vs. amplitude–phase AFs).
- Deep architecture scalability: Limited discussion of training stability for very deep CVNNs (residual/skip connections, attention/transformers, normalization stacks) and whether common deep-learning heuristics transfer to complex domain.
- Complex convolution and pooling: No detailed treatment of complex-valued convolution kernels (padding, stride, circular vs. linear convolution), complex pooling strategies (magnitude/phase pooling, quaternion-like alternatives), or their spectral implications.
- Loss design for complex outputs: Missing practical guidance on losses beyond complex MSE (e.g., amplitude–phase decoupled losses, circular distances for phases, complex-valued cross-entropy), and how loss choice affects phase fidelity and convergence.
- Robustness and adversarial behavior: No analysis of CVNN robustness to complex-domain noise, phase jitter, adversarial perturbations on amplitude/phase, or distribution shifts specific to complex-valued data.
- Interpretability in complex domain: Absent tools and methodologies to interpret learned complex filters (e.g., phase-rotation profiles, frequency-domain transfer functions, amplitude–phase saliency) and to connect them with domain physics.
- Hybrid RVNN–CVNN designs: The idea of hybridizing RVNN and CVNN components is mentioned but not formalized—no architectural blueprints, training protocols, or ablations showing when and how hybridization helps.
- Framework-level support and reproducibility: No reference implementations in PyTorch/TensorFlow with correct Wirtinger gradients and complex operations; a need for open-source tooling, unit tests, and examples to ensure correctness and adoption.
- Empirical validation of new CVAFs: The introduced fully complex, split, and amplitude–phase CVAFs lack head-to-head benchmarking across tasks, sensitivity analyses (hyperparameters, scaling factors), and failure mode characterizations (e.g., saturation, exploding modulus).
- Practical guidance for phase-learning limitations: For fully complex AFs that “preserve phase” but cannot learn phase variations, there is no proposed workaround (e.g., learnable phase offsets, phase-normalization layers, phase-residual modules) or empirical assessment of the limitation in real tasks.
- Handling singularities and branch cuts: No discussion of how AFs or layers that involve functions like log or inverse trig manage branch cuts, multi-valuedness, and essential singularities—and their effects on gradient stability and numerical robustness.
- Second-order and curvature-aware methods: Absent development of complex-domain Hessians, Gauss–Newton/LM variants, or natural-gradient approaches tailored to complex parameters and their geometry.
- Bayesian CVNNs and uncertainty: No exploration of probabilistic modeling (complex priors, posterior inference, uncertainty in amplitude/phase) for CVNNs, despite relevance in signal processing and communications.
- Evaluation under hardware constraints: Claims about future hardware (e.g., quantum, specialized processors) are not substantiated with current accelerator support or kernel-level optimizations; need concrete pathways and benchmarks for complex arithmetic on GPUs/TPUs.
Practical Applications
Immediate Applications
The following items translate the survey’s findings on complex-valued neural networks (CVNNs), complex backpropagation (CBP), Wirtinger calculus, and complex-valued activation functions (CVAFs) into practical, deployable use cases.
- Phase-aware speech enhancement and dereverberation (sector: healthcare, software, consumer electronics)
- Description: Deploy CVNNs that operate directly on complex STFTs to jointly learn amplitude and phase for noise suppression, dereverberation, and source separation in hearing aids, teleconferencing, and smartphones.
- Potential tools/products/workflows: PyTorch/TensorFlow CVNN layers with Wirtinger-based autograd; split AFs (Split-Tanh, Split-Parametric Sigmoid) or amplitude–phase CVAFs (e.g., APSF, CAP-ELU, CAP-Softplus) for mask estimation; complex-valued loss functions (e.g., phase-sensitive SI-SDR); on-device inference pipelines.
- Assumptions/dependencies: Mature complex autograd and CBP implementations; datasets with reliable phase annotations or consistent STFT conventions; real-time constraints require efficient complex ops on mobile DSP/NPUs.
- Wireless communications baseband processing (sector: communications, software-defined radio)
- Description: Replace RVNN blocks in SDR stacks with CVNNs for IQ imbalance correction, carrier frequency/phase tracking, channel equalization, MIMO detection, and QAM classification where phase rotation and amplitude attenuation are central.
- Potential tools/products/workflows: CVNN-enabled demodulators/equalizers using CAP-Swish, CAP-ArcTanS, CAP-ES; CBP with Wirtinger derivatives; model compression for FPGA/ASIC inference; integration with GNU Radio, srsRAN, or proprietary 5G stacks.
- Assumptions/dependencies: Availability of complex-domain training data (pilot sequences, IQ captures); latency budgets in baseband; hardware support for complex MACs; standardized metrics that reward phase-accurate decoding.
- Radar/sonar clutter suppression and target detection (sector: defense, automotive, robotics, maritime)
- Description: Apply CVNNs on complex range–Doppler maps and phase-coherent returns to improve clutter rejection, target classification, and phase-coherent integration in radar/sonar.
- Potential tools/products/workflows: Complex CNNs with amplitude–phase CVAFs for feature extraction; CBP pipelines; domain-coloring diagnostics for failure analysis and interpretability.
- Assumptions/dependencies: Access to labeled complex datasets; integration with legacy DSP blocks; compute budgets for embedded platforms in ADAS and UAVs.
- MRI and ultrasound reconstruction (sector: healthcare imaging)
- Description: Use CVNNs to reconstruct images directly from complex k-space (MRI) or beamformed RF data (ultrasound), leveraging phase-aware learning to reduce artifacts and enhance resolution.
- Potential tools/products/workflows: Complex U-Nets equipped with CAP-ELU, CAP-Softplus; CBP training for holomorphic components; clinical validation workflows with uncertainty quantification.
- Assumptions/dependencies: Regulatory approval pathways; robust generalization across scanners/protocols; alignment with safety and QA procedures.
- Phasor measurement unit (PMU) analytics for smart grids (sector: energy)
- Description: Employ CVNNs on grid synchrophasors (complex voltage/current) for anomaly detection, state estimation, and oscillation damping recommendations.
- Potential tools/products/workflows: CAP-type CVAFs for phase-stable features; streaming inference gateways; integration with SCADA and EMS dashboards.
- Assumptions/dependencies: Reliable PMU deployments; operator acceptance; cybersecurity constraints; interpretable alerts.
- Fourier-domain image restoration (sector: software, industrial inspection)
- Description: Implement CVNNs for frequency-domain deblurring/sharpening that preserves phase, enabling better restoration of periodic patterns in manufacturing QA and scientific imaging.
- Potential tools/products/workflows: Complex-domain CNN modules; split AFs with per-channel phase constraints; workflow bridges between spatial and frequency domains.
- Assumptions/dependencies: Standardized preprocessing/normalization of FFTs; task-specific metrics that account for phase fidelity; integration into existing image processing pipelines.
- Radar/sonar SLAM and beamforming for robotics (sector: robotics)
- Description: Improve SLAM and beamforming with CVNNs that use amplitude–phase activations to handle coherent sensor arrays, benefiting navigation in low-visibility environments.
- Potential tools/products/workflows: CVNN modules in ROS; complex beamformer neural layers; phase-aware cost functions in mapping.
- Assumptions/dependencies: Robust synchronization/calibration of sensor arrays; computational budgets on edge platforms.
- Academic instruction and visualization (sector: education)
- Description: Use domain-coloring and complex plots to teach CVNNs, holomorphy, Cauchy–Riemann conditions, and Wirtinger calculus; prototype CVAF behavior and CBP gradients.
- Potential tools/products/workflows: Mathematica notebooks (ComplexPlot/ComplexPlot3D) for AF and function visualization; Jupyter-based demos of CBP variants; curriculum modules on CVNN mathematics.
- Assumptions/dependencies: Access to tooling (Mathematica or open-source equivalents); instructor familiarity with complex analysis.
- Developer tooling for CVNNs (sector: software)
- Description: Provide CVNN libraries with CBP support (complex derivative approach, partial derivatives, Cauchy–Riemann-informed methods), CVAF layers (FC-Swish, FC-Mish, CAP variants), and complex BatchNorm equivalents.
- Potential tools/products/workflows: PyTorch/TensorFlow addons for Wirtinger gradients; layer catalogs for split and fully complex AFs; gradient checkers and stability diagnostics.
- Assumptions/dependencies: Community maintenance; test suites against analytical derivatives; guidance on analyticity vs boundedness trade-offs per Liouville’s theorem.
Long-Term Applications
These items rely on further research, scaling, hardware development, or standardization before widespread deployment.
- 6G/Next-gen baseband with native complex AI (sector: communications)
- Description: End-to-end CVNN blocks in the PHY/MAC for joint channel estimation, beam management, waveform design, and interference cancellation leveraging phase-rotation modeling.
- Potential tools/products/workflows: CVNN-optimized ASICs with complex MACs; co-design toolchains coupling RF, DSP, and AI; standardized datasets and KPIs for phase-sensitive tasks.
- Assumptions/dependencies: Hardware acceleration for complex ops; standardization in 3GPP/ETSI; provable reliability and low-latency guarantees.
- Quantum control and error mitigation (sector: quantum computing)
- Description: Utilize CVNNs to model and control complex-valued quantum states and unitary dynamics for gate synthesis and noise mitigation.
- Potential tools/products/workflows: CVNN-assisted pulse shaping; amplitude–phase AFs tuned to unitary constraints; hybrid classical–quantum training loops.
- Assumptions/dependencies: Rigorous theory bridging CVNNs with quantum control; datasets from real quantum hardware; guarantees respecting unitarity and physical constraints.
- Optical/photonics neuromorphic computing with complex arithmetic (sector: hardware, energy)
- Description: Implement CVNN primitives (complex multiplication, phase rotation) in analog photonic circuits to exploit natural amplitude–phase behaviors for energy-efficient AI.
- Potential tools/products/workflows: Photonic accelerators embedding CAP-type AFs; integrated optical interconnects for complex computation; toolchains for mapping CVNN graphs to photonic hardware.
- Assumptions/dependencies: Fabrication maturity; programmability and stability; calibration protocols for phase and amplitude control.
- Holomorphic CVAF design and training theory (sector: academia)
- Description: Develop CVAFs that balance differentiability with practical boundedness via amplitude normalization and phase-aware scaling; formalize generalization and stability analyses for CBP.
- Potential tools/products/workflows: Libraries of provably stable AFs (e.g., amplitude-normalized, phase-preserving); training curricula and open benchmarks for CVNN stability.
- Assumptions/dependencies: New theory mitigating Liouville constraints; community consensus on evaluation standards; reproducible baselines.
- Hybrid RVNN–CVNN architectures (sector: software, cross-industry)
- Description: Architectures that route wave-like signals (RF, acoustics, optics) through CVNN branches while non-coherent features are processed by RVNN, improving efficiency and accuracy.
- Potential tools/products/workflows: Model compilers that partition pipelines; AutoML for modality-aware branching; interpretable fusion layers managing amplitude–phase features.
- Assumptions/dependencies: Tooling to seamlessly mix complex and real layers; training schedules for heterogeneous gradients; deployment frameworks handling mixed-precision complex types.
- Standards and policy for phase-aware AI systems (sector: policy, healthcare, communications)
- Description: Create guidelines and certification for phase-sensitive AI (e.g., medical imaging, spectrum sensing) to ensure safety, interoperability, and auditability.
- Potential tools/products/workflows: Regulatory test suites emphasizing phase fidelity; documentation standards for complex-data pipelines; procurement policies favoring phase-aware models where applicable.
- Assumptions/dependencies: Cross-stakeholder engagement (vendors, regulators, clinicians); validated metrics for clinical efficacy and RF performance; privacy/security frameworks for complex signals.
- Consumer devices with complex-native AI (sector: consumer electronics)
- Description: Next-gen smartphones, AR/VR headsets, and home routers with CVNNs for RF self-calibration, multipath mitigation, and phase-aware audio processing.
- Potential tools/products/workflows: On-device complex accelerators; phase-coherent Wi-Fi optimization; hearing assistance features with CVNN noise suppression.
- Assumptions/dependencies: Silicon support for complex ops; robust energy profiles; user-facing benefits that justify BOM changes.
- Grid-wide complex analytics and control (sector: energy)
- Description: CVNN-driven state estimation and protection across regional grids using synchrophasors for real-time resilient operations and fault isolation.
- Potential tools/products/workflows: Digital twin platforms with complex state variables; closed-loop controllers informed by CVNN forecasts; interoperability with utility standards.
- Assumptions/dependencies: Data-sharing agreements; high-fidelity sensors; assurance frameworks for safety-critical ML.
- Methodological ecosystems for CVNNs (sector: software, academia)
- Description: Full-stack support for CVNNs—data types, optimizers, visualization, debugging, and reproducibility—parallel to mature RVNN ecosystems.
- Potential tools/products/workflows: Complex-aware optimizers; complex BatchNorm/LayerNorm; domain-coloring-based explainability; open datasets with complex labels (IQ, k-space).
- Assumptions/dependencies: Broad community adoption; funding for maintenance; clear demonstration of advantages over RVNN baselines.
Notes on feasibility across applications:
- CVNN compute overhead remains higher than RVNN; hardware acceleration and efficient complex kernels are key.
- CBP variants and Wirtinger calculus lower theoretical barriers but require robust implementations and numerical stability checks.
- Liouville’s theorem constrains fully analytic AFs; split and amplitude–phase CVAFs are practical compromises with empirically strong performance.
- Success depends on sector-specific datasets that preserve phase and amplitude information, along with metrics that reward phase fidelity (not just magnitude-only accuracy).
Collections
Sign up for free to add this paper to one or more collections.