S Rajbhandari,
J Rasley,
O Ruwase… - … , Networking,
Storage …, 2020 - ieeexplore.ieee.org
… These efforts are complimentary and can work together with ZeRO. In fact, activation memory
reduction in ZeRO-R works in parallel with activation checkpointing. 2) CPU Offload: [18], […