no code implementations • 12 Feb 2024 • Ping Wu, Heyan Huang, Zhengyang Liu
Specifically, we introduce a family of Follow the Delayed Regularized Leader algorithms for feedback with full information on the loss function, a family of Delayed Mirror Descent algorithms for feedback with gradient information on the loss function and a family of Simplified Delayed Mirror Descent algorithms for feedback with the value information of the loss function's gradients at corresponding decision points.
1 code implementation • 12 Jun 2020 • Xunpeng Huang, Runxin Xu, Hao Zhou, Zhe Wang, Zhengyang Liu, Lei LI
Due to its simplicity and outstanding ability to generalize, stochastic gradient descent (SGD) is still the most widely used optimization method despite its slow convergence.
no code implementations • 10 Feb 2020 • Xunpeng Huang, Xianfeng Liang, Zhengyang Liu, Yitan Li, Linyun Yu, Yue Yu, Lei LI
SPAN computes the inverse of the Hessian matrix via low-rank approximation and stochastic Hessian-vector products.
no code implementations • 25 Sep 2019 • Xunpeng Huang, Zhengyang Liu, Zhe Wang, Yue Yu, Lei LI
To the best of our knowledge, Acutum is the first adaptive gradient method without second moments.