Effective dynamics and critical scaling for Stochastic Gradient Descent in high dimensions

vendredi 13 mai 2022, 14h00 - 15h00

Salle du conseil, espace Turing

Joint work with Reza Gheissari (UC Berkeley) and Aukosh Jagannath (Waterloo)

SGD is a workhorse for optimization and thus for statistics and machine learning, and it is well understood in low dimensions. But understanding its behavior in very high dimensions is not yet a simple task. We study here the limiting effective dynamics of some summary statistics for SGD in high dimensions, and find interesting and new regimes, i.e. not the expected one given by the usual wisdom, i.e. the population gradient flow. We find that a new corrector term is needed and that the phase portrait of these dynamics is quite complex and substantially different from what would be predicted using the classical low-dimensional approach, including for simple tasks.