Training DDPM with the Denoising Objective#

Numerical Implementation of the Forward Process#

To numerically simulate the forward diffusion process, we divide the time range $[0,T]$ into intervals of length ${\beta }_i$, where $i=1,\dots ,n$. We denote the intermediate times as $t_i={\sum }_{j=0}^{i-1}{\beta }_j$.

The vanilla discretization of the $\text{Forward Process}$ is given by:

where we approximate $d{\mathbf{W}}_t$ as $\sqrt{dt}{\mathbf{ϵ}}_i$, ${\mathbf{ϵ}}_i$ is standard Gaussian random variable. (refer to the previous section).

A more subtle but equivalent implementation is the variance-preserving (VP) form ¹:

The variance-preserving form adds small Gaussian noise to an image step by step, eventually turning it into ${\mathbf{x}}_n\sim \mathcal{N}(0,I)$. When $\beta$ is small, it is equivalent to the vanilla discretization, since $\sqrt{1-{\beta }_{i-1}}\approx 1-\frac{1}{2}{\beta }_{i-1}$. Its key benefit is maintaining unit variance: if ${\mathbf{x}}_0$ starts with unit variance, ${\mathbf{x}}_i$ keeps it too. This is because $\text{Var}({\mathbf{x}}_i)=(1-{\beta }_{i-1})\text{Var}({\mathbf{x}}_{i-1})+{\beta }_{i-1}\text{Var}({\mathbf{ϵ}}_{i-1})$.

WARNING

Note that our interpretation of $\beta$ differs from that in ¹, treating $\beta$ as a varying time-step size to solve the autonomous SDE (1.5 OU process noise) instead of coefficients of a time-dependent SDE. Our interpretation greatly simplifies future analysis, but it holds only if every ${\beta }_i$ is sufficiently small.

Instead of expressing the iterative relationship between ${\mathbf{x}}_i$ and ${\mathbf{x}}_{i-1}$, we can directly represent the dependency of ${\mathbf{x}}_i$ on ${\mathbf{x}}_0$ using the following forward relation:

where ${\mathbf{x}}_0$ are clean images from dataset, ${\bar{\alpha }}_i={\prod }_{j=0}^{i-1}(1-{\beta }_j)$ denotes the contamination weight, and ${\overline{\mathbf{ϵ}}}_i$ represents standard Gaussian noise that accumulates noises from ${\mathbf{ϵ}}_0$ to ${\mathbf{ϵ}}_i$.

TIP

An useful property we shall exploit later is that for infinitesimal time steps $\beta$, the contamination weight ${\bar{\alpha }}_i$ is the exponential of the diffusion time $t_i={\sum }_{j=0}^{i-1}{\beta }_j$

$$\underset{{\max }_j{\beta }_j\to 0}{\lim }{\bar{\alpha }}_i\to e^{-t_i}.$$

Numerical Implementation of the Backward Process#

The backward diffusion process is used to sample from the DDPM by removing the noise of an image step by step. It is the time reversed version of the OU process, starting at $x_{0^{\prime }}\sim \mathcal{N}(\mathbf{x}|0,I)$, using the reverse of the OU process (1.5 reverse diffusion process).

The vanilla discretization of the $\text{Backward Process}$ is given by:

where $k=1,\dots ,n$ represents the backward time step, and ${\mathbf{x}}_k$ is the image at the $k$th step with time $t_k^{\prime }={\sum }_{j=0}^{k-1}{\beta }_{n-1-j}=T-t_{n-k}$.

A more common discretization ¹ is:

This formulation is equivalent to the vanilla discretization when ${\beta }_i$ is small. The score function $\mathbf{s}$ is typically modeled by a neural network trained using a denoising objective.

Training the Score Function#

Training the score function requires a training objective. We will show that the score function could be trained with a denoising objective.

DDPM is trained to removes the noise ${\overline{\mathbf{ϵ}}}_i$ from ${\mathbf{x}}_i$ in the forward diffusion process, by training a denoising neural network ${\mathbf{ϵ}}_{\theta }(\mathbf{x},t_i)$ to predict and remove the noise ${\overline{\mathbf{ϵ}}}_i$. This means that DDPM minimizes the denoising objective ²:

where ${\overline{\mathbf{ϵ}}}_i$ is determined from ${\mathbf{x}}_i$ according to the $\text{discrete forward diffusion}$ process.

Now we show that ${\mathbf{ϵ}}_{\theta }$ trained with the above objective is proportional to the score function $\mathbf{s}$. There are two important properties regarding the relationship between the noise ${\overline{\mathbf{ϵ}}}_i$ and the score function $\mathbf{s}$:

1. Gaussian Distribution of ${\mathbf{x}}_i$:
   According to the $\text{discrete forward diffusion}$, the distribution of ${\mathbf{x}}_i$ given ${\mathbf{x}}_0$ is a Gaussian distribution, expressed as:

   $$p({\mathbf{x}}_i|{\mathbf{x}}_0)=\mathcal{N}({\mathbf{x}}_i|\sqrt{{\bar{\alpha }}_i}{\mathbf{x}}_0,(1-{\bar{\alpha }}_i)I).$$

2. Proportionality of Noise to Score Function:
   The noise ${\overline{\mathbf{ϵ}}}_i$ is directly proportional to a score function, given by:

   $${\overline{\mathbf{ϵ}}}_i=-\sqrt{1-{\bar{\alpha }}_i}\mathbf{s}({\mathbf{x}}_i|{\mathbf{x}}_0,t_i),$$

   where $\mathbf{s}({\mathbf{x}}_i|{\mathbf{x}}_0,t_i)={\nabla }_{{\mathbf{x}}_i}\log p({\mathbf{x}}_i|{\mathbf{x}}_0)$ represents the score of the conditional probability density $p({\mathbf{x}}_i|{\mathbf{x}}_0)$ at ${\mathbf{x}}_i$.

These properties indicate that the noise ${\overline{\mathbf{ϵ}}}_i$ is directly related to a conditional score function, which connects to the score function $\mathbf{s}(\mathbf{x},t)$ through the above equations.

Now we are very close to our target. The conditional score function $\mathbf{s}({\mathbf{x}}_i|{\mathbf{x}}_0,t_i)$ is connected to the score function $\mathbf{s}(\mathbf{x},t)$ through the following equation:

where $f$ is an arbitrary function, $p_{t_i}({\mathbf{x}}_i)=\int p({\mathbf{x}}_i|{\mathbf{x}}_0)p_0({\mathbf{x}}_0)d{\mathbf{x}}_0$, and $\mathbf{s}(\mathbf{x},t)={\nabla }_{\mathbf{x}}\log p_t(\mathbf{x})$ is the score function of the probability density of ${\mathbf{x}}_t$.

Substituting the $\text{score-noise relationship}$ into the $\text{denoising objective}$, expanding the squares, and utilizing the above equation, and drop terms that are irrelavant with parameter $\theta$, we can show that optimizing the denoising objective is equivalent to optimize a denoising score matching objective:

This objectives says that the denoising neural network ${\mathbf{ϵ}}_{\theta }(\mathbf{x},t_i)$ is trained to approximate a scaled score function $\mathbf{ϵ}(\mathbf{x},t_i)$ ³

With the help of this relation between ${\mathbf{ϵ}}_{\theta }$ and score function, we could rewrite the backward process as

Summary:#

• We’ve covered all aspects of DDPM theory. To implement it:
  • Select a suitable dataset as the empirical distribution $p_0(\mathbf{x})$ and sample data points as ${\mathbf{x}}_0$.
  • Apply the $\text{discrete forward diffusion}$ to add noise to the data.
  • Record the noise ${\overline{\mathbf{ϵ}}}_i$ used during this process.
  • Train a denoising neural network ${\mathbf{ϵ}}_{\theta }$ using the noise recorded and the $\text{denoising objective}$.
  • Generate new samples using the $\text{discrete sampling process}$.

What is Next#

In the next section, we will discuss an alternative version of the backward diffusion process: ordinary differential equation (ODE) based backward sampling. This approach serves as the foundation for several modern architectures, such as rectified flow diffusion models.

Stay tuned for the next installment!

Discussion#

If you have questions, suggestions, or ideas to share, please visit the discussion post.

Cite this blog#

This blog is a reformulation of the appendix of the following paper.

1
@misc{zheng2025lanpainttrainingfreediffusioninpainting,
2
      title={LanPaint: Training-Free Diffusion Inpainting with Asymptotically Exact and Fast Conditional Sampling},
3
      author={Candi Zheng and Yuan Lan and Yang Wang},
4
      year={2025},
5
      eprint={2502.03491},
6
      archivePrefix={arXiv},
7
      primaryClass={eess.IV},
8
      url={https://arxiv.org/abs/2502.03491},
9
}