Learning rates for stochastic gradient descent with nonconvex objectives

Yunwen Lei, Ke Tang

Research output: Contribution to journalArticlepeer-review

249 Downloads (Pure)


Stochastic gradient descent (SGD) has become the method of choice for training highly complex and nonconvex models since it can not only recover good solutions to minimize training errors but also generalize well. Computational and statistical properties are separately studied to understand the behavior of SGD in the literature. However, there is a lacking study to jointly consider the computational and statistical properties in a nonconvex learning setting. In this paper, we develop novel learning rates of SGD for nonconvex learning by presenting high-probability bounds for both computational and statistical errors. We show that the complexity of SGD iterates grows in a controllable manner with respect to the iteration number, which sheds insights on how an implicit regularization can be achieved by tuning the number of passes to balance the computational and statistical errors. As a byproduct, we also slightly refine the existing studies on the uniform convergence of gradients by showing its connection to Rademacher chaos complexities.
Original languageEnglish
Pages (from-to)4505-4511
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
Issue number12
Early online date23 Mar 2021
Publication statusE-pub ahead of print - 23 Mar 2021


  • Learning Rates
  • Stochastic Gradient Descent
  • arly Stopping
  • onconvex Optimization


Dive into the research topics of 'Learning rates for stochastic gradient descent with nonconvex objectives'. Together they form a unique fingerprint.

Cite this