What approach allows the weights to be updated in parallel, improving training speed?

The approach that allows the weights to be updated in parallel, thereby enhancing training speed, is asynchronous updates. In asynchronous updates, multiple processes can compute gradients and update the model weights simultaneously without waiting for each other. This allows for more efficient use of computational resources, particularly in distributed settings or when using GPUs, as it reduces idle time where nodes might otherwise be waiting for others to complete their calculations before proceeding.

In contrast, synchronous updates require all nodes to compute their gradients and only then update the weights collectively, which can lead to bottlenecks and delays, particularly as the number of nodes increases. While penalization mechanisms and the cross-entropy loss pertain to how models learn and minimize error during training, they do not directly address the parallel updating of weights that characterize asynchronous updates.