Sparse Stochastic Optimization: From Dual Averaging to Variance Reduction
Lin Xiao, Research Scientist at Facebook AI Research (FAIR)
In many optimization problems arising from signal processing and machine learning, sparsity is often a desired property, providing simplicity and robustness of the solution. An effective way to obtain sparse solutions is to add a sparsity-inducing regularization to the objective, and then solve the regularized problem using a proximal-gradient method. However, direct extensions of this approach to the stochastic optimization setting become ineffective in obtaining sparse solutions, due to the diminishing step sizes required for stochastic gradient type of methods to converge. In this talk, we explain how the regularized dual-averaging method can overcome such difficulty and enable large-scale sparse stochastic optimization. In addition, we present recent progress in stochastic gradient methods with variance reduction, which hold promise for sparse optimization in both convex and non-convex settings.
About the speaker
Lin Xiao is a Research Scientist at Facebook AI Research (FAIR) in Seattle, Washington US. He received PhD in Aeronautics and Astronautics from Stanford University, and was a postdoctoral fellow at California Institute of Technology. Before joining Facebook, he spent 14 years as a Researcher at Microsoft Research. He was a winner of the Young Researcher competition at the first International Conference on Continuous Optimization (2004) for his work on fastest mixing Markov chains, and the Test-of-Time Award at NeurIPS 2019 for his work on the regularized dual averaging method for sparse online optimization. His current research interests include optimization algorithms for machine learning, reinforcement learning, and parallel and distributed computing.
Register for this event
About the seminar series
The Jean Golding Institute has teamed up with the Heilbronn Institute for Mathematical Research to showcase the latest research in Data Science - methodology with roots in Mathematics and Computer Science with important applied implications.
Our seminar series features a range of internationally regarded high-profile speakers on topics that will be relevant to a broad audience.