deep generative models

Deep generative model tries to learn the true dist from a set of samples from the true dist
true dist - ground truth aka P_data(x) (we do not have the true dist as it is vexy complex!)
predicted dist - the dist learned by the model based on the samples collected from the true dist aka P_phi(x)
Goal - make the predicted dist as close as possible to the true dist! P_phi(x) ~ P_data(x)
DGMs take as an input a large collection of real-world examples drawn from an unknown and complex dist and outputs a trained neural network that parametrizes an approx dist
there are 2 main goals of DGM - realistic generation and controllable generation

lets say we have a sample of cat images and we want to train our DGM such that it can learn the underlying dist of cats, if the predicted dist is close the original one, we can sample an arbitrary number of new data points using the sampling methods!

the params phi are learnt by minimizing the discrepancy between p_data(x) and p_phi(x)
how do we measure the difference between 2 probability dist? → KL divergence
KL divergence compares a true dist P and a model Q
It is 0 when P = Q and grows as Q becomes worse especially when P(x) is large and Q(x) is very wrong
KL divergence looks at every possible outcome, checks how much the model’s prob differs from reality, weighs it by how often that outcome actually happens, and sums everything up.
The final number tells how how far the model dist is from the original dist

to model a complex data dist, we can parametrize the prob density func using a nn with params phi, creating a model we denote as P_phi, for P_phi to be valid:

non-negativity → can be eliminate by taking exponential fn or square
enforcing normalization:
- the crucial part to understand is that the number by which we divide the score to normalize it is also called as a normalizing constant or the partition function
- for most problems, the partition function is intractable and impossible to calculate, this intractability is a central problem that motivates the development of many different families of deep generative models
Prominent DGMs:
1. Energy Based Models
- For example imagine that we collect exam scores of 1000 students in a course
- we want to convert this into an energy graph so scores which have a higher density should get a lower energy and scores which have a lower density should get a higher energy. The graph will look as follows:
- now we convert this into probability which looks as follows: