Year: 2022
Author: Hongkang Yang
Journal of Machine Learning, Vol. 1 (2022), Iss. 4 : pp. 373–431
Abstract
The modeling of probability distributions, specifically generative modeling and density estimation, has become an immensely popular subject in recent years by virtue of its outstanding performance on sophisticated data such as images and texts. Nevertheless, a theoretical understanding of its success is still incomplete. One mystery is the paradox between memorization and generalization: In theory, the model is trained to be exactly the same as the empirical distribution of the finite samples, whereas in practice, the trained model can generate new samples or estimate the likelihood of unseen samples. Likewise, the overwhelming diversity of distribution learning models calls for a unified perspective on this subject. This paper provides a mathematical framework such that all the well-known models can be derived based on simple principles. To demonstrate its efficacy, we present a survey of our results on the approximation error, training error and generalization error of these models, which can all be established based on this framework. In particular, the aforementioned paradox is resolved by proving that these models enjoy implicit regularization during training, so that the generalization error at early-stopping avoids the curse of dimensionality. Furthermore, we provide some new results on landscape analysis and the mode collapse phenomenon.
Journal Article Details
Publisher Name: Global Science Press
Language: English
DOI: https://doi.org/10.4208/jml.221202
Journal of Machine Learning, Vol. 1 (2022), Iss. 4 : pp. 373–431
Published online: 2022-01
AMS Subject Headings:
Copyright: COPYRIGHT: © Global Science Press
Pages: 59
Keywords: Generative modeling Density estimation Generalization error Memorization Implicit regularization.
Author Details
-
Tessellating the Latent Space for Non-Adversarial Generative Auto-Encoders
Gai, Kuo
Zhang, Shihua
IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 46 (2024), Iss. 2 P.780
https://doi.org/10.1109/TPAMI.2023.3325282 [Citations: 0]