It may track moving averages, rescale coordinates, add momentum, or apply weight decay rather than using raw gradients directly. Its settings interact with the learning rate, batch size, and loss landscape.
An optimizer turns computed gradients into parameter updates during model training.
It may track moving averages, rescale coordinates, add momentum, or apply weight decay rather than using raw gradients directly. Its settings interact with the learning rate, batch size, and loss landscape.