Academy of Mathematics and Systems Science, CAS
Colloquia & Seminars

Speaker:	Dr. Yuhua Zhu, Stanford University
Inviter:	刘歆副研究员
Title:	Towards the theoretical understanding of large batch training in stochastic gradient descent
Time & Venue:	2019.6.24 8:30-9:30 Z301
Abstract:	Stochastic gradient descent (SGD) is almost ubiquitously used for training non-convex optimization tasks. Recently, a hypothesis proposed by Keskar et al. that large batch methods tend to converge to sharp minimizers has received increasing attention by researchers. We theoretically justify this hypothesis by providing new properties of SGD in both finite-time and asymptotic regime. In particular, we give an explicit escaping time of SGD from a local minimum in the finite-time regime and prove that SGD tends to converge to flatter minima in the asymptotic regime (although may take exponential time to converge) regardless of the batch size. We also find that SGD with a larger learning rate to batch size ratio tends to converge to a flat minimum faster, however, its generalization performance could be worse than the SGD with a smaller learning rate to batch size ratio. We include experiments to corroborate these theoretical findings.