The Fastest Way to Diffusion Model Theory - I

When I first learned about diffusion models, I was introduced to them as a type of variational autoencoder (VAE) applied to a series of quantities $x_{0}, \dots, x_{T}$ . Deriving the forward and reverse processes required lengthy derivations spanning multiple pages, dense with priors, posteriors, Bayesian theorems, and mathematical intricacies. Later, I encountered the stochastic differential equation (SDE) perspective, which frames diffusion models through Fokker-Planck and Kolmogorov backward equations—concepts no simpler to grasp than the VAE approach.

Instead, this blog series aims to deliver a concise, self-contained, and rigorous introduction to diffusion models. While some concepts may be challenging, I believe this approach offers the fastest and most straightforward pathway to understanding diffusion model theory. We will focus exclusively on fundamental SDE principles and calculus to intuitively derive the core theory, uncovering its intrinsic structure without relying on advanced machinery.

In this section, we cover the basics of Stochastic Differential Equations (SDEs), focusing on two fundamental concepts:

Brownian noise ( $d W$ ): The core random process driving SDE dynamics
Langevin Dynamics: The basic SDE to generate samples from a probability distribution.

Prerequisites: Calculus, particularly series expansions and vector calculus (gradients, Laplacians).

Diffusion Process#

The Diffusion Process forms the mathematical foundation of diffusion models, describing a system’s evolution through deterministic drift and stochastic noise. Here we consider a diffusion process of the following form of stochastic differential equation (SDE):

d x_{t} = μ (x_{t}, t) d t + σ (x_{t}, t) d W_{t},

where the drift term $μ (x_{t}, t) d t$ governs deterministic motion, while $d W_{t}$ adds Brownian noise.

The Brownian noise $d W$ is a key characteristic of SDEs, capturing their stochastic nature. It represents a series of infinitesimal Gaussian noise. A good way to understand it is through a formal definition:

d W_{t} = d t n \to \infty lim i = 1 \sum n \frac{1}{n} ϵ_{i},

where $ϵ_{i}$ are independent standard Gaussian noises with mean $0$ and identity covariance matrix $I$ . The limit in this definition shows that $d W$ is not just a single Gaussian random variable with mean $0$ , but rather the cumulative effect of infinitely many independent Gaussian increments. Such cumulation allows us to compute the covariance of $d W$ as vector product:

d W_{t} \cdot d W_{t}^{T} = C o v (d W_{t}, d W_{t}) = I d t,

where $I$ is the identity matrix.

NOTE
When no quadratic terms of $d W_{t}$ are involved, $d W_{t}$ can often be roughly treated as $d t ϵ$ , where $ϵ \sim N (0, 1)$ is a standard Gaussian random variable.

The Brownian noise $d W_{t}$ scales as $d t$ , which fundamentally alters the rules of calculus for SDEs. A change of variable in ordinary calculus has $d s = \frac{d s}{d t} d t$ , but for Brownian noise it is $d W_{s} = \frac{d s}{d t} d W_{t}$ . Moreover, the differentiation of a function is $df (t, x_{t}) = \partial_{t} fd t + \nabla_{x} f \cdot d x_{t}$ in ordinary calculus, but for SDE, it follows the Itô’s lemma:

df (t, x_{t}) = \partial_{t} fd t + \nabla_{x} f \cdot d x_{t} + stochastic effect \frac{σ ^{2}}{2} \nabla_{x}^{2} f d t .

This is derived by differentiating $f$ using the chain rule with the help of the SDE and covariance of $d W_{t}$ , while keeping all terms up to order $d t$ (note that $d W$ scales as $d t$ ). The emergence of the second-order Laplacian term $\nabla_{x}^{2} f$ is the key distinction from ordinary calculus. We will later use this lemma to analyze the evolution of the distribution of $x_{t}$ .

Langevin Dynamics#

Langevin Dynamics is a special diffusion process that aims to generate samples from a distribution $p (x)$ . It is defined as:

d x_{t} = s (x_{t}) d t + 2 d W_{t},

where $s (x) = \nabla_{x} lo g p (x)$ is the score function.

This dynamics is often used as a Monte Carlo sampler to draw samples from $p (x)$ , since $p (x)$ is its stationary distribution—the distribution that $x_{t}$ converges to as $t \to \infty$ , regardless of the initial distribution of $x_{0}$ . More precisely, this means that if an ensemble of particles at positions ${x_{t}^{(1)}, x_{t}^{(2)}, \dots, x_{t}^{(N)}}$ evolves according to the given SDE, and their initial positions ${x_{0}^{(1)}, x_{0}^{(2)}, \dots, x_{0}^{(N)}}$ follow a distribution $p (x)$ , then their positions will continue to be distributed according to $p (x)$ at all future times $t > 0$ .

To verify stationarity, we will show that after evolution from time $0$ to $Δ t$ , the distribution of $x_{Δ t}$ is still $p (x)$ . Consider a test function $f$ and initial positions $x_{0} \sim p (x)$ , stationary can be assessed by tracking the change in the expectation $E_{x_{0} \sim p (x)} [f (x_{Δ t})]$ . Using $(It \overset{o}{ˆ} ’s lemma)$ and note that $E_{x} [d W] = 0$ for any distribution of $x$ , we compute:

E_{x_{0} \sim p (x)} [f (x_{Δ t}) - f (x_{0})] \approx Δ t \int p (x) (\nabla_{x} f \cdot s + \nabla_{x}^{2} f) d x (to the first order) = Δ t \int f (x) (- \nabla_{x} \cdot (p s) + \nabla_{x}^{2} p) d x (integration by parts) = Δ t \int f (x) \nabla_{x} \cdot (- p s + \nabla_{x} p) d x = 0,

where $0$ is obtained by substituting $s = \nabla_{x} lo g p$ . Because $E_{x_{0} \sim p (x)} [f (x_{Δ t}) - f (x_{0})] = 0$ for any test function $f$ , this means the distribution of $x_{Δ t}$ must have been kept the same as $x_{0}$ .

Alternative form of the Langevin Dynamics:#

Recall that the term $d W_{t}$ in Langevin dynamics scales as $d t$ . We can reformulate the Langevin dynamics by substituting $d t$ with $\frac{1}{2} d t$ , resulting in the alternative form of the Langevin Dynamics:

d x_{t} = \frac{1}{2} s (x_{t}) d t + d W_{t} .

IMPORTANT
Langevin Dynamics as ‘Identity’#
The stationary of $p (x)$ is very important: The Langevin dynamics for $p (x)$ acts as an “identity” operation on the distribution, transforming samples from $p (x)$ into new samples from the same distribution. This property enables what I believe is the simplest way to derive the forward and backward diffusion processes in diffusion models.

Langevin Dynamics as Monte Carlo Sampler#

Langevin dynamics can be used to generate samples from a distribution $p (x)$ , given its score function $s$ . But its success hinges on two critical factors. First, the method is highly sensitive to initialization - a poorly chosen $x_{0}$ may trap the sampling process in local likelihood maxima, failing to explore the full distribution. Second, inaccuracies in the score estimation, particularly near $x_{0}$ , can prevent convergence altogether. These limitations led to the development of diffusion models, which eliminate the difficulty in choosing $x_{0}$ : all samples are generated by gradually denoising pure Gaussian noise.

What is Next#

In the next section, we will use Langevin dynamics as a stepping stone to derive the forward and backward diffusion processes. We will examine their mathematical formulation and how they form a dual pair—each reversing the other’s evolution.

Stay tuned for the next installment!

Discussion#

If you have questions, suggestions, or ideas to share, please visit the discussion post.