What is the primary goal of Leaky ReLU activation function?

The primary goal of the Leaky ReLU activation function is to avoid dead neurons. This is crucial in neural networks since dead neurons, or neurons that output zero for all inputs, can inhibit the learning process. Leaky ReLU addresses this issue by allowing a small, non-zero, constant gradient when the input is negative. This prevents neurons from becoming inactive and ensures they can still contribute to learning, even when dealing with negative values in the input space.

Other options, while relevant to neural networks, do not capture the fundamental aim of the Leaky ReLU function. For instance, while Leaky ReLU can help improve training speed indirectly by maintaining neuron activations, its primary purpose is not explicitly about training speed enhancement. Similarly, though it returns outputs that can be negative, its role is not to ensure that outputs are always positive, as that is characteristic of other activation functions like ReLU or Softmax. Lastly, weight initialization concerns relate to how weights are set prior to training, which is separate from the function of activation functions like Leaky ReLU.