Deep Networks with Stochastic Depth (1603.09382v3)

Abstract: Very deep convolutional networks with hundreds of layers have led to significant reductions in error on competitive benchmarks. Although the unmatched expressiveness of the many layers can be highly desirable at test time, training very deep networks comes with its own set of challenges. The gradients can vanish, the forward flow often diminishes, and the training time can be painfully slow. To address these problems, we propose stochastic depth, a training procedure that enables the seemingly contradictory setup to train short networks and use deep networks at test time. We start with very deep networks but during training, for each mini-batch, randomly drop a subset of layers and bypass them with the identity function. This simple approach complements the recent success of residual networks. It reduces training time substantially and improves the test error significantly on almost all data sets that we used for evaluation. With stochastic depth we can increase the depth of residual networks even beyond 1200 layers and still yield meaningful improvements in test error (4.91% on CIFAR-10).

• The paper introduces stochastic depth to mitigate vanishing gradients by randomly skipping residual blocks during training.
• It demonstrates that a linearly decaying survival probability significantly reduces test errors on benchmarks like CIFAR-10 and CIFAR-100.
• The method enhances gradient flow and reduces training time while using full network depth during testing for optimal performance.

Deep Networks with Stochastic Depth: An Analysis

The development of Deep Convolutional Neural Networks (DCNNs) has seen a trend toward increasingly deeper architectures, as evidenced by models such as VGG, GoogLeNet, and more recently, ResNet. While deeper networks have been theoretically and empirically proven to be more expressive, they also introduce significant practical challenges, such as vanishing gradients, diminishing feature reuse, and prohibitive training times.

The paper "Deep Networks with Stochastic Depth" by Gao Huang et al. addresses these challenges by introducing a novel training paradigm called stochastic depth. The core idea revolves around dynamically altering the network depth during training to enhance gradient flow and reduce training time, while maintaining the network's full depth during testing to leverage its representational capacity.

Methodology

Stochastic Depth Mechanism

Stochastic depth operates by randomly dropping entire layers (specifically, residual blocks) during training. For each mini-batch, a subset of layers is bypassed using identity shortcuts. The survival probability of each layer, denoted as $p_\ell$, can either be uniform across all layers or decay linearly from the first to the last layer. The authors find that a linearly decaying $p_\ell$ yields better empirical results, with $p_L = 0.5$ performing robustly across experiments.

Formally, if $H_\ell$ represents the output of the $\ell^{th}$ layer and $f_\ell(\cdot)$ denotes the layer's transformation function, the forward propagation rule is modified to:

$$H_\ell = \text{ReLU} \left( b_\ell f_\ell(H_{\ell-1}) + H_{\ell-1} \right),$$

where $b_\ell$ is a Bernoulli random variable with parameter $p_\ell$. This framework ensures that during each iteration, only a fraction of the network is active, effectively reducing the depth and mitigating the vanishing gradient problem.

During testing, all layers are utilized, but the output of each layer $f_\ell$ is scaled by its associated survival probability $p_\ell$:

$$H_\ell^\text{Test} = \text{ReLU} \left( p_\ell f_\ell(H_{\ell-1}^\text{Test}) + H_{\ell-1}^\text{Test} \right).$$

Experimental Results

The efficacy of the proposed method is demonstrated across several benchmark datasets: CIFAR-10, CIFAR-100, SVHN, and ImageNet. Some noteworthy results include:

• CIFAR-10: With standard data augmentation, stochastic depth reduces the test error from 6.41% (constant depth ResNet) to 5.25%. Notably, a 1202-layer ResNet with stochastic depth further reduces the error to 4.91%.
• CIFAR-100: A significant reduction in test error from 27.76% to 24.98% was achieved.
• SVHN: The test error improved from 1.80% to 1.75%.
• ImageNet: Despite an increase in complexity, stochastic depth facilitates a reduction in validation error from 21.78% to 21.98% within the same training schedule, highlighting computational efficiency gains.

The results underscore the method's capability to allow deeper architectures to be effectively trained, thereby harnessing their representational power without suffering from training inefficiencies.

Theoretical Insights and Practical Implications

Gradient Magnitude

Empirical analysis reveals that stochastic depth maintains stronger gradient magnitudes, particularly in early layers, during the entire training process. This characteristic is attributed to the shorter paths for gradient flow introduced by bypassing some layers, thus preventing gradient vanishing, especially after epochs when the learning rate is decayed.

Robustness to Hyper-parameters

Stochastic depth demonstrates robustness to the choice of survival probability $p_L$. The method performs consistently well across a range of $p_L$ values, with the linear decay rule proving particularly effective. This stability alleviates the need for extensive hyper-parameter tuning, making the methodology accessible and scalable.

Future Directions

The success of stochastic depth in training extremely deep networks opens new avenues for developing even deeper and more sophisticated architectures. Future research may explore:

• Adaptive Survival Probabilities: Dynamic adjustment of $p_\ell$ based on layer-wise training dynamics may further enhance performance.
• Extensions Beyond Residual Networks: Applying stochastic depth to other architectures, including those beyond residual frameworks, to examine its generalizability.
• Combined Regularization Techniques: Integrating stochastic depth with other regularization strategies like Dropout and Batch Normalization to investigate synergistic effects.

In conclusion, the introduction of stochastic depth provides a practical and theoretically sound approach to mitigate the challenges associated with training very deep networks. Its ability to facilitate deeper architectures while improving training efficiency and model accuracy renders it a valuable addition to the deep learning toolkit.