Data-parallel training commonly sums or averages gradients with ring, tree, or topology-aware algorithms. Its communication cost often limits scaling when computation per worker becomes small.
All-reduce is a collective communication operation that combines values across workers and returns the result to every worker.
Data-parallel training commonly sums or averages gradients with ring, tree, or topology-aware algorithms. Its communication cost often limits scaling when computation per worker becomes small.