Without nonlinear activations, stacked linear layers would collapse into a single linear transformation. The activation's shape affects gradient flow, sparsity, smoothness, and computational cost.
An activation function applies a nonlinear transformation to a neural network's intermediate values.
Without nonlinear activations, stacked linear layers would collapse into a single linear transformation. The activation's shape affects gradient flow, sparsity, smoothness, and computational cost.