What technique is used to reduce memory usage during training a model?

Gradient checkpointing is an effective technique used to reduce memory usage during the training of deep learning models. In standard training processes, intermediate activations from each layer must be stored in memory for backpropagation, which can consume a significant amount of memory, especially in large models or when using high-resolution input data.

With gradient checkpointing, only a subset of these intermediate activations is saved during the forward pass, while the others are recomputed as needed during the backward pass. This approach allows for a notable decrease in memory requirements because it alleviates the burden of having to store all activations at once. While it introduces some additional computational overhead due to the repeated calculations of the omitted activations, the trade-off is typically worthwhile, particularly for large models training on limited hardware resources.

Synchronous and asynchronous updates pertain to how gradient updates are communicated and applied across parallel training processes but do not directly address memory usage. Cross-entropy loss is a commonly used loss function for classification tasks that measures the difference between predicted and actual probability distributions, but it is also unrelated to memory management strategies during model training.