generative models

generates o/p but many o/p valid
if we know p(x) we can sample x ~ p and also estimate the likelihood of o/p
p(x) does not have to be known explicitly!
conditional generation - p(x|c)
diff b/w generative and discriminative models
- discriminative - p(y|x) → only 1 true label y for each input x (many-to-one)
- conditional generative - p(x|c) → many valid outputs x for a condition c - (one-to-many or many-to-many)
formulating generative modeling

implicit vs explicit representations
- explicit - directly learn the probability density function using max likelihood estimation method ex- VAEs, Autoregressive GPT etc.
- implicit - bypass the need to calc exact probability numbers, the dist is learned within the model’s weights, instead focuses on generation ex- GANs
Types of Generative Models:
- Autoregressive Models
- Latent Variable Models
- Flow-based Models
- Energy-based Models

main objectives of generative model is to max -ve log likelihood
flow-based models - does not need a latent space, takes an intial dist and turns it into output dist.
latent variable models - sample a latent variable z and then conditionally generates the output based on the prior z.
directly allows to learn the energy function via NN, the exponential function ensures positivity and partition function ensures normalization.
Problem with partition function? → computing it is intractable!
how to avoid the partition function? gradient of the log p(x) is the score function of p wrt to x
we can optimize the score (which is independent of the partition function) by minimizing the Fisher divergence which has the same optimum as the max log likelihood
Denoising Score matching

Links:

202609022007