Just one byte (per gradient): A note on low-bandwidth decentralized language model finetuning using shared randomness
Language model training in distributed settings is limited by the communication cost of
gradient exchanges. In this short note, we extend recent work from Malladi et al.(2023) …
gradient exchanges. In this short note, we extend recent work from Malladi et al.(2023) …
Just One Byte (per gradient): A Note on Low-Bandwidth Decentralized Language Model Finetuning Using Shared Randomness
E Zelikman, Q Huang, P Liang, N Haber… - arXiv e …, 2023 - ui.adsabs.harvard.edu
Abstract Language model training in distributed settings is limited by the communication cost
of gradient exchanges. In this short note, we extend recent work from Malladi et al.(2023) …
of gradient exchanges. In this short note, we extend recent work from Malladi et al.(2023) …