Which technique is used to address vanishing gradients in modern network architectures?

The technique utilized to effectively address the vanishing gradients problem in modern network architectures is through the use of both residual connections and skip connections. These techniques are fundamentally related and serve to facilitate the flow of gradients during backpropagation, making it easier for the network to learn.

Residual connections are designed to allow the input to bypass one or more layers and be added back later in the computation process. By allowing this shortcut, the gradient can flow directly across layers, mitigating the diminishing effect usually seen with deep networks where the gradients may decay to zero before reaching the initial layers.

Skip connections perform a similar function by linking non-adjacent layers together, also providing alternate paths for the gradient. Both types of connections help in preserving information across layers, ensuring that earlier layers continue to receive adequate gradient signals even as the depth of the network increases.

This architectural design has become particularly prevalent in deep learning frameworks such as ResNet, where deep architectures can be trained more effectively without the hindrances posed by vanishing gradients. Layer normalization, while useful for stabilizing learning and improving convergence rates, does not specifically target the gradient flow issue associated with deep architectures, distinguishing it from the more direct impact of residual and skip connections on mitigating vanishing gradients.