Maximal Initial Learning Rates in Deep ReLU Networks

no code implementations • 14 Dec 2022 • Gaurav Iyer, Boris Hanin, David Rolnick

Training a neural network requires choosing a suitable learning rate, which involves a trade-off between speed and effectiveness of convergence.