Search Results for author:
Found 4 papers, 0 papers with code
Sparse Matrix in Large Language Model Fine-tuning
In this work, we introduce a method for selecting sparse sub-matrices that aim to minimize the performance gap between PEFT vs. full fine-tuning (FT) while also reducing both fine-tuning computational cost and memory cost.
Adjacent Leader Decentralized Stochastic Gradient Descent
This work focuses on the decentralized deep learning optimization framework.
Accelerating Parallel Stochastic Gradient Descent via Non-blocking Mini-batches
In this paper, we propose the (de)centralized Non-blocking SGD (Non-blocking SGD) which can address the straggler problem in a heterogeneous environment.
RCD-SGD: Resource-Constrained Distributed SGD in Heterogeneous Environment via Submodular Partitioning
The convergence of SGD based distributed training algorithms is tied to the data distribution across workers.
