Fundamental limits of overparametrized shallow neural networks for supervised learning

We carry out an information-theoretical analysis of a two-layer neural network trained from input–output pairs generated by a teacher network with matching architecture, in overparametrized regimes. Our results come in the form of bounds relating i) the mutual information between training data and network weights, and ii) the Bayes-optimal generalization error, to the same quantities but for a simpler (generalized) linear model for which explicit expressions are rigorously known. Our bounds, which are expressed in terms of the number of training samples, input dimension and number of hidden units, yield fundamental performance limits for learning a target function with the form of a shallow neural network when limited data is available. The proof relies on rigorous tools from spin glasses and is guided by “Gaussian equivalence principles” lying at the core of numerous recent analyses of neural networks. Our results are information-theoretic (i.e. are not specific to any learning algorithm) and, importantly, cover a setting where all the student network parameters are trained.