Which training approach involves updating model weights sequentially after processing each batch?

The training approach that involves updating model weights sequentially after processing each batch is known as synchronous updates. In this method, each batch of data is processed, and once the loss is calculated from the model's predictions, the gradients are computed and the model's weights are updated in a synchronized fashion. This means that all updates happen consistently across the system, ensuring that all workers or processes are in sync and are using the same model weights for the next batch of data.

The key aspect of synchronous updates is that they ensure that the learning process is stable and consistent, as every worker will wait for the others to finish processing their respective batches before moving on to the next iteration. This can lead to potentially slower training times due to waiting for all updates to synchronize, but it greatly helps prevent issues related to inconsistency in model training.

In contrast, other methods like asynchronous updates process batches independently and allow for model weights to be updated immediately after a batch is processed, regardless of whether other workers have completed their updates. This can introduce variability and instability in the training process, which can lead to erratic convergence behavior. Gradient checkpointing, on the other hand, is a technique used to save memory during training by only storing certain activations and recomputing others