← List all talks ...

Learning by Pruning and the Hunt for Lottery Tickets

Dimitris Papailiopoulos – Assistant Professor, University of Wisconsin-Madison

Fri, 4-Feb-2022 / 2:00pm / https://stanford.zoom.us/j/92716427348

Abstract

A recent work by Ramanujan et al. (2020) provides empirical evidence that sufficiently overparameterized, random neural networks contain untrained subnetworks that achieve state-of-the-art accuracy across several predictive tasks. Follow-up theoretical work establishes a version of this phenomenon when the random network to be pruned is a large polynomial factor wider than a target one. We offer an exponential improvement and show that logarithmically overparameterized, random neural networks can be pruned to approximate any target network. We further show that the amplitude of the random weights does not matter, and one can find almost anything within mildly overparameterized random binary networks. I will conclude with discussing how the above hint on a path towards resolving a largely open problem: finding sparse trainable networks at initialization, aka lottery tickets.

Bio

Dimitris Papailiopoulos is an Assistant Professor of Electrical and Computer Engineering at the University of Wisconsin-Madison. His research interests span machine learning, information theory, and ML systems, with a current focus on efficient large-scale training algorithms. Between 2014 and 2016, Dimitris was a postdoctoral researcher at UC Berkeley. He earned his Ph.D. in ECE from UT Austin in 2014, under the supervision of Alex Dimakis. He received his ECE Diploma and M.Sc. degree from the Technical University of Crete, in Greece. Dimitris is a recipient of the NSF CAREER Award (2019), three Sony Faculty Innovation Awards (2019, 2020, 2021), a joint IEEE ComSoc/ITSoc Best Paper Award (2020), an IEEE Signal Processing Society, Young Author Best Paper Award (2015), the Vilas Associate Award (2021), the IEEE Education Society Mac Van Valkenburg Early Career Teaching Award (2021), the Emil Steiger Distinguished Teaching Award (2021), and the Benjamin Smith Reynolds Award for Excellence in Teaching (2019). In 2018, he co-founded MLSys, a new conference that targets research at the intersection of machine learning and systems.