Efficient training of large language models on distributed infrastructures: a survey
Large Language Models (LLMs) like GPT and LLaMA are revolutionizing the AI industry with
their sophisticated capabilities. Training these models requires vast GPU clusters and …
their sophisticated capabilities. Training these models requires vast GPU clusters and …
Hybridflow: A flexible and efficient rlhf framework
Reinforcement Learning from Human Feedback (RLHF) is widely used in Large Language
Model (LLM) alignment. Traditional RL can be modeled as a dataflow, where each node …
Model (LLM) alignment. Traditional RL can be modeled as a dataflow, where each node …
AntDT: A Self-Adaptive Distributed Training Framework for Leader and Straggler Nodes
Many distributed training techniques like Parameter Server and AllReduce have been
proposed to take advantage of the increasingly large data and rich features. However …
proposed to take advantage of the increasingly large data and rich features. However …
Understanding and Alleviating Memory Consumption in RLHF for LLMs
Fine-tuning with Reinforcement Learning with Human Feedback (RLHF) is essential for
aligning large language models (LLMs). However, RLHF often encounters significant …
aligning large language models (LLMs). However, RLHF often encounters significant …
[PDF][PDF] Dynamic Fast Device Placement Strategies for Real-Time Resource Allocation
H Zhang, Z Chen, XLY Liu, J Wu, L Wang - researchgate.net
Large-scale distributed systems are increasingly reliant on efficient resource allocation to
meet the demands of real-time applications. However, the challenges of maintaining low …
meet the demands of real-time applications. However, the challenges of maintaining low …
[PDF][PDF] Leveraging Machine Learning Techniques for Efficient Fast Device Placement
S Volkov, J Wang, D Ivanov, A Petrov, J Smith, D Zhao - researchgate.net
Modern device placement poses significant challenges due to the increasing complexity of
environments and user demands. Our study introduces a method that leverages advanced …
environments and user demands. Our study introduces a method that leverages advanced …
[引用][C] Adaptive Placement Algorithms for Energy-Efficient ML Graph Execution
Y Shen, M Wang, Y Liu, M Johnson, J Smith, E Brown