Early gradients can be unstable before activations and optimizer statistics settle. A warmup schedule reduces the chance that large initial steps damage useful parameter structure.
Learning rate warmup begins training with small updates and gradually raises the learning rate to its planned level.
Early gradients can be unstable before activations and optimizer statistics settle. A warmup schedule reduces the chance that large initial steps damage useful parameter structure.