Mathematics Colloquium: Pratik Patil

Pratik Patil from the University of California, Berkeley will be the guest speaker at our upcoming colloquium held on Tuesday, January 21 at 3:30 p.m. in Carnegie 122. We hope to see you there!

Title: Facets of regularization in overparameterized machine learning

Abstract: Modern machine learning often operates in an overparameterized regime in which the number of parameters exceeds the number of observations. In this regime, models can exhibit surprising generalization behaviors: (1) Models can achieve zero training error yet still generalize well (benign overfitting); furthermore, in some cases, adding and tuning explicit regularization can favor no regularization at all (obligatory overfitting). (2) The generalization error can vary non-monotonically with the model or sample size (double/multiple descent). These behaviors challenge classical notions of overfitting and the role of explicit regularization.

In this talk, I will present theoretical and methodological results related to these behaviors, focusing on the concrete case of ridge regularization. First, I will identify conditions under which the optimal ridge penalty is zero (or even negative) and show that standard techniques such as leave-one-out and generalized cross-validation, when analytically continued, remain uniformly consistent for the generalization error and thus yield the optimal penalty, whether positive, negative, or zero. Second, I will introduce a general framework to mitigate double/multiple descent in the sample size based on subsampling and ensembling and show its intriguing connection to ridge regularization. As an implication of this connection, I will show that the generalization error of optimally tuned ridge regression is monotonic in the sample size (under mild data assumptions) and mitigates double/multiple descent. Key to both parts is the role of implicit regularization, either self-induced by the overparameterized data or externally induced by subsampling and ensembling.

The talk will feature joint work with the following collaborators (in surname-alphabetical order): Jin-Hong Du, Arun Kumar Kuchibhotla, Alessandro Rinaldo, Ryan Tibshirani, Yuting Wei. The corresponding papers (in talk-chronological order) are: optimal ridge landscape (https://pratikpatil.io/papers/ridge-ood.pdf), ridge cross-validation (https://pratikpatil.io/papers/functionals-combined.pdf), risk monotonization (https://pratikpatil.io/papers/risk-monotonization.pdf), and ridge equivalences (https://pratikpatil.io/papers/generalized-equivalences.pdf).