Which training approach is less efficient due to the computational overhead it introduces?

Synchronous updates are considered less efficient primarily due to the computational overhead involved in coordinating updates from all participating nodes before proceeding with the next round of model training. In this approach, each worker must wait for all other workers to complete their computations and share their gradients before any updates to the model can occur. This waiting time can lead to increased latency, particularly in large, distributed training scenarios where the communication overhead becomes significant.

In contrast, other training approaches, like asynchronous updates, allow individual workers to update the model independently, which can lead to faster overall training times since there is no need for synchronization at every step. Gradient checkpointing and the objective function primarily relate to memory management and optimization, respectively. They do not inherently introduce the same level of coordination overhead that synchronous updates incur. Thus, the synchronous update mechanism's requirement for complete collaboration at each update step makes it less efficient in terms of computational resources.