What technique adjusts learning rates based on the norms of layer weights?

Layer-wise Adaptive Rate Scaling (LARS) is designed to adjust learning rates based on the norms of layer weights, making it particularly effective for training deep neural networks. This technique scales the learning rate for each layer according to the weight update of that layer, promoting better convergence and efficiency.

The rationale behind LARS is that different layers in a neural network may have parameters with varying magnitudes. By adaptively scaling the learning rates, LARS ensures that updates are more appropriately sized relative to the weight norms of each layer. This can help mitigate issues like exploding gradients and encourage faster convergence by preventing any single layer from disproportionately affecting the training process.

In contrast, while gradient clipping is useful for managing the extreme values of gradients, it does not adaptively adjust the learning rates based on weight norms. Adaptive Learning Rate Optimization refers to methods that adapt the learning rates over time but do not specifically use the norms of the layers as LARS does. A Learning Rate Scheduler modifies the learning rate at predefined intervals, rather than on a per-layer basis based on weight norms.