Which training method might involve higher computational overhead due to its sequential nature?

Synchronous updates involve processes that wait for all parts of a distributed system to finish before proceeding to the next step. This means that every worker in a distributed training environment must complete their computations and share their gradients before the model weights are updated. Because of this sequential dependency, synchronous updates can lead to idle time for some workers while they wait for others to finish, thereby increasing computational overhead.

In contrast, asynchronous updates allow individual workers to update the model independently, which can optimize the utilization of resources and reduce computation time overall. Gradient checkpointing is a technique used to trade off memory usage for computation time by saving intermediate results, but it does not inherently involve higher computational overhead due to sequence dependencies. The objective function generally refers to the metric being optimized and does not pertain directly to the nature of training updates. Thus, synchronous updates are noted for their requirement of coordination and synchronization, leading to increased computational expense.