parallel training at scale. Existing parameter synchronization protocols cannot effectively
leverage available network resources in the face of ever increasing hardware heterogeneity.
To address this issue, we propose Blink, a collective communication library that dynamically
generates optimal communication primitives by packing spanning trees. We propose
techniques to minimize the number of trees generated and extend Blink to leverage …