Learning rates for stochastic gradient descent with nonconvex objectives

Research output: Contribution to journalArticlepeer-review

Authors

Colleges, School and Institutes

Abstract

Stochastic gradient descent (SGD) has become the method of choice for training highly complex and nonconvex models since it can not only recover good solutions to minimize training errors but also generalize well. Computational and statistical properties are separately studied to understand the behavior of SGD in the literature. However, there is a lacking study to jointly consider the computational and statistical properties in a nonconvex learning setting. In this paper, we develop novel learning rates of SGD for nonconvex learning by presenting high-probability bounds for both computational and statistical errors. We show that the complexity of SGD iterates grows in a controllable manner with respect to the iteration number, which sheds insights on how an implicit regularization can be achieved by tuning the number of passes to balance the computational and statistical errors. As a byproduct, we also slightly refine the existing studies on the uniform convergence of gradients by showing its connection to Rademacher chaos complexities.

Details

Original languageEnglish
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
Early online date23 Mar 2021
Publication statusE-pub ahead of print - 23 Mar 2021

Keywords

  • Stochastic Gradient Descent, Learning Rates, onconvex Optimization, arly Stopping