Each microbatch runs forward and backward passes, but the optimizer waits until the configured number has been processed. This approximates a larger effective batch when device memory cannot hold it at once.
Gradient accumulation combines gradients from several smaller batches before applying an optimizer update.
Each microbatch runs forward and backward passes, but the optimizer waits until the configured number has been processed. This approximates a larger effective batch when device memory cannot hold it at once.