ADMM Training Algorithms for Residual Networks: Convergence, Complexity and Parallel Training

no code implementations • 23 Oct 2023 • Jintao Xu, Yifei Li, Wenxun Xing

Convergence of the proximal point version is proven based on a Kurdyka-Lojasiewicz (KL) property analysis framework, and we can ensure a locally R-linear or sublinear convergence rate depending on the different ranges of the Kurdyka-Lojasiewicz (KL) exponent, in which a necessary auxiliary function is constructed to realize our goal.

Convergence Rates of Training Deep Neural Networks via Alternating Minimization Methods

no code implementations • 30 Aug 2022 • Jintao Xu, Chenglong Bao, Wenxun Xing

Training deep neural networks (DNNs) is an important and challenging optimization problem in machine learning due to its non-convexity and non-separable structure.