Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse of Dimensionality in Approximation on Hölder Class

Abstract: In this paper, we construct neural networks with ReLU, sine and $2^x$ as activation functions. For general continuous $f$ defined on $[0,1]<sup>d$ with continuity modulus $\omega_f(\cdot)$, we construct ReLU-sine-$2^x$ networks that enjoy an approximation rate $$\mathcal{O}(\omega_f(\sqrt{d})\cdot2<sup>{-M}+\omega_{f}\left(\frac{\sqrt{d}}{N}\right))$$, where $M,N\in \mathbb{N}<sup>{+}$ denote the hyperparameters related to widths of the networks. As a consequence, we can construct ReLU-sine-$2^x$ network with the depth $5$ and width $\max\left{\left\lceil2d^{{3/2}\left(\frac{3\mu}{\epsilon}\right)^{{1/{\alpha}}\right\rceil,2\left\lceil\log_2\frac{3\mu}} d<sup>{\alpha/2}}{2\epsilon}\right\rceil+2\right}$ that approximates $$f\in \mathcal{H}<em>{\mu}<sup>{\alpha}([0,1]<sup>d)$$ within a given tolerance $\epsilon &gt;0$ measured in $L<sup>p$ norm $p\in[1,\infty)$, where $\mathcal{H}</em>{\mu}<sup>{\alpha}([0,1]<sup>d)$ denotes the H\"older continuous function class defined on $[0,1]<sup>d$ with order $\alpha \in (0,1]$ and constant $\mu &gt; 0$. Therefore, the ReLU-sine-$2^x$ networks overcome the curse of dimensionality on $\mathcal{H}_{\mu}<sup>{\alpha}([0,1]<sup>d)$. In addition to its supper expressive power, functions implemented by ReLU-sine-$2^x$ networks are (generalized) differentiable, enabling us to apply SGD to train.