Recap
Section I introduced Langevin Dynamics for sampling from $p (x)$ :
$d x_{t} = s (x_{t}) d t + 2 d W_{t} or d x_{t} = \frac{1}{2} s (x_{t}) d t + d W_{t}$
where $s (x) = \nabla_{x} lo g p (x)$ is the score function, and $d W_{t} \sim d t N (0, I)$ .
Section II defined DDPM’s processes:

Forward Process ( $t \in [0, T]$ ):
$d x_{t} = - \frac{1}{2} x_{t} d t + d W_{t}$
The discrete version adds noise through:
$x_{i} = \overset{α}{ˉ}_{i} x_{0} + 1 - \overset{α}{ˉ}_{i} \overset{ˉ}{ϵ}_{i}, 1 \leq i \leq n$
where $\overset{ˉ}{ϵ}_{i} \sim N (0, I)$ is the noise at step $i$ , and $\overset{α}{ˉ}_{i}$ controls the noise schedule.

Backward Process ( $t^{'} = T - t$ ):
$d x_{t^{'}} = (\frac{1}{2} x_{t^{'}} + s (x_{t^{'}}, T - t^{'})) d t^{'} + d W_{t^{'}}$
where $s (x, t) = \nabla_{x} lo g p_{t} (x)$ .

Section III covered the denoising objective:
$ϵ_{θ} min \frac{1}{n} i = 1 \sum n E_{x_{0} \sim p_{0} x_{i} \sim p (x_{i} ∣ x_{0})} ∥ \overset{ˉ}{ϵ}_{i} - ϵ_{θ} (x_{i}, t_{i}) ∥_{2}^{2}$
which trains $ϵ_{θ}$ to predict $\overset{ˉ}{ϵ}_{i}$ (approximating $- 1 - \overset{α}{ˉ}_{i} s (x, t_{i})$ ).

ODE and Flow-Based Diffusion Model#

The ODE Based Backward Diffusion Process#

The backward diffusion process $Backward Process$ is not the only reverse process for the forward process $Forward Process$ . We can derive a deterministic ordinary differential equation (ODE) as an alternative, removing the stochastic term $d W$ in the backward process.

To obtain this ODE reverse process, consider the Langevin dynamics with a rescaled time ( $d τ \to \frac{1}{2} d τ$ ):

d x_{τ} = \frac{1}{2} s (x_{τ}, t) d τ + d W_{τ}, = Forward - \frac{1}{2} x_{τ} d τ + d W_{τ} + Backward \frac{1}{2} x_{t} d τ + \frac{1}{2} s (x_{τ}, t) d τ,

Following the same logic used to derive the backward diffusion process in Section II, we could read from this splitting the backward ODE (known as the probability flow ODE ¹):

d x_{t^{'}} = (\frac{1}{2} x_{t^{'}} + \frac{1}{2} s (x, T - t^{'})) d t^{'},

where $t^{'} \in [0, T]$ is backward time, and $s (x, t) = \nabla_{x_{t}} lo g p_{t} (x)$ is the score function of the density of $x_{t}$ in the forward process. This ODE maintains the same forward-backward duality as the SDE reverse process $Backward Process$ .

Since the ODE is deterministic, it enables faster sampling than the SDE version. Established ODE solvers—such as higher-order methods and exponential integrators—can further reduce computational steps while maintaining accuracy.

Variance Perserving, Variance Exploding, and Rectified Flow#

With the ODE based backward process, we can discuss three common formulations of ODE based diffusion models: variance-preserving (VP), variance-exploding (VE), and rectified flow (RF). We demonstrate their mathematical equivalence and show how they can be transformed into one another.

To simplify notation, we now use continuous time $t$ and its corresponding state $x_{t}$ , rather than discrete notations like $t_{i}$ and $x_{i}$ .

Variance Preserving (VP)#

The ‘variance-preserving’ formulation is very similar to what we have introduced in the previous section, just replacing the SDE backward process to the ODE version.

The forward diffusion process in continuous time $t$ is:

x_{t} = \overset{α}{ˉ}_{t} x_{0} + 1 - \overset{α}{ˉ}_{t} \overset{ˉ}{ϵ}_{t},

where $\overset{α}{ˉ}_{t} = e^{- t}$ and $\overset{ˉ}{ϵ}_{t} \sim N (0, I)$ . It is the same as $discrete backward process$ introduced in the previous section.

Forward and Backward Processes#

The forward and backward processes in VP notation are consistent with the previously introduced DDPM model, with the only difference being the substitution of the SDE backward process with the ODE version.

Forward SDE ( $Forward Process$ ):

d x_{t} = - \frac{1}{2} x_{t} d t + d W_{t}

Backward ODE ( $Probability flow ODE$ ):

d x_{t^{'}} = \frac{1}{2} (x_{t^{'}} + s (x_{t^{'}}, T - t^{'})) d t^{'},

where $t^{'} \in [0, T]$ is reversed time, and the score function $s (x, t) = \nabla_{x} lo g p_{t} (x)$ is learned via the denoising objective.

Discrete Forward and Backward Processes#

We also list the discrete version of Forward and Backward Processes, which is used in trainining and inference practice.

Discrete Forward SDE: Suppose we discretize the time $t \in [0, T]$ as ${t_{0} = 0, \dots, t_{i}, \dots, t_{n} = T}$ and denote $β_{i} = t_{i + 1} - t_{i}$ , $x_{i} = x_{t_{i}}$ , then

x_{i} = \overset{α}{ˉ}_{t} x_{0} + 1 - \overset{α}{ˉ}_{t} \overset{ˉ}{ϵ}_{i}, i = 1, \dots, n

Discrete Backward ODE: Correspondingly, we discretize the backward diffusion time as $t_{k}^{'} = T - t_{n - k}$ and denote $x_{k} = x_{t_{k}^{'}}$ . This discretizes the backward diffusion process ¹ into:

x_{k} = (2 - 1 - β_{n - k}) x_{k - 1} + \frac{1}{2} s (x_{k - 1}, t_{n - k + 1}) β_{n - k}, k = 1, \dots, n

To see how this matches the continuous version, note that $1 - β_{n - k} \approx 1 - \frac{1}{2} β_{n - k}$ .

Score Matching Objective#

While we previously trained the denoising network $ϵ_{θ}$ using the $denoising objective$ , we can alternatively model the score function $s_{θ}$ directly. This yields the equivalent score matching objective:

L_{score} (s_{θ}) = E_{t \sim U [0, 1]} E_{x_{0} \sim p_{0} (x)} E_{\overset{ˉ}{ϵ}_{t} \sim N (0, I)} \frac{ϵ ˉ _{t}}{1 - α ˉ _{t}} + s_{θ} (x_{t}, t)_{2}^{2},

where $x_{t}$ follows the forward process. This represents an equivalent but reweighted version of the original denoising objective.

Variance Exploding (VE)#

The variance exploding formulation provides an alternative to variance preserving. Define:

σ = \frac{1 - α ˉ _{t}}{α ˉ _{t}}; σ^{'} = \frac{1 - α ˉ _{T - t^{'}}}{α ˉ _{T - t^{'}}}; z_{σ} = \frac{x _{t}}{α ˉ _{t}}; z_{σ^{'}} = \frac{x _{t^{'}}}{α ˉ _{T - t^{'}}}; ϵ (z_{σ}, σ) = - 1 - \overset{α}{ˉ}_{t} s (x_{t}, t),

Substituting the definitions above and rewriting the $VP forward$ process in VE notation yields:

z_{σ} = z_{0} + σ \overset{ˉ}{ϵ}_{σ},

where $z_{0} = x_{0}$ is the clean image corrupted by the standard Gaussian noise $\overset{ˉ}{ϵ}_{σ}$ .

Forward and Backward Processes#

Substituting the definitions from $VE notations$ and rewriting both the $VP forward SDE$ and $VP backward ODE$ yields:

Forward SDE:

d z_{σ} = 2 σ d W_{σ}, σ \in [0, \frac{1 - α ˉ _{T}}{α ˉ _{T}}]

Backward ODE:

d z_{σ^{'}} = ϵ (z_{σ^{'}}, σ^{'}) d σ^{'}, σ^{'} \in [\frac{1 - α ˉ _{T}}{α ˉ _{T}}, 0]

The advantage of the VE notation lies in its simpler backward ODE compared to the VP notation. In practice, directly discretizing the $VE backward ODE$ using an Euler solver tends to yield greater accuracy than the $VP backward ODE$ , which includes an additional $\frac{1}{2} x$ term that can introduce numerical errors. However, a notable disadvantage of the VE notation is that $σ$ can become quite large at time $T$ , potentially leading to numerical instability.

Discrete Forward and Backward Processes#

We also list the discrete version of Forward and Backward Processes, which is used in trainining and inference practice.

Discrete Forward SDE: Suppose we discretize the $σ \in [0, \frac{1 - α ˉ _{T}}{α ˉ _{T}}]$ as ${σ_{0} = 0, \dots, σ_{i}, \dots, σ_{n} = \frac{1 - α ˉ _{T}}{α ˉ _{T}}}$ and $z_{i} = z_{σ_{i}}$ , then

z_{i} = z_{0} + σ_{i} \overset{ˉ}{ϵ}_{σ_{i}}, i = 1, \dots, n

where $\overset{ˉ}{ϵ}_{σ_{i}}$ is a standard Gaussian noise.

Discrete Backward ODE: Correspondingly, define $σ_{k}^{'} = σ_{n - k}$ and denote $z_{k} = z_{σ_{k}^{'}}$ . This discretizes the backward diffusion process into:

z_{k} = z_{k - 1} + ϵ (z_{k - 1}, σ_{k - 1}^{'}) (σ_{k}^{'} - σ_{k - 1}^{'}), k = 1, \dots, n

Denoising Objective#

To directly model $ϵ_{θ} (z, σ)$ , we adapt the $denoising objective$ to VE coordinates by replacing $x_{t}$ with $z_{σ}$ :

L_{d e n o i se} (ϵ_{θ}) = E_{σ \sim U [0, σ_{ma x}]} E_{z_{0} \sim p_{0} (x)} E_{\overset{ˉ}{ϵ}_{σ} \sim N (0, I)} ∥ \overset{ˉ}{ϵ}_{σ} - ϵ_{θ} (z_{σ}, σ) ∥_{2}^{2},

where $σ_{ma x} = (1 - \overset{α}{ˉ}_{T}) / \overset{α}{ˉ}_{T}$ and $z_{σ}$ follows the VE forward process. This preserves the denoising objective’s structure while operating in VE space.

Rectified Flow (RF)#

While often presented as a distinct framework from DDPMs, rectified flows are mathematically equivalent ² to DDPMs. We now provide a much simpler proof via the following transformations:

s = \frac{σ}{1 + σ}; s^{'} = \frac{σ ^{'}}{1 + σ ^{'}}; r_{s} = \frac{z _{σ}}{1 + σ}; r_{s^{'}} = \frac{z _{σ^{'}}}{1 + σ ^{'}}; v (r_{s}, s) = \frac{ϵ ( z _{σ} , σ ) - r _{s}}{1 - s}

Rewriting the $VE forward$ process in $RF notation$ yields:

r_{s} = (1 - s) r_{0} + s \overset{ˉ}{ϵ}_{s},

This process linearly interpolates between clean data ( $r_{0}$ ) and standard Gaussian noise $\overset{ˉ}{ϵ}_{s}$ . While many sources refer to $\overset{ˉ}{ϵ}_{s}$ as $r_{1}$ since both represent Gaussian noise, this notation can be misleading. Such notation suggests that $r_{s}$ is a deterministic interpolation between $r_{0}$ and $r_{1}$ . But in fact $r_{s}$ is a combination of clean data $r_{0}$ and random Gaussian noise, similar to the formulation in VP and VE notation.

Forward and Backward Processes#

The forward and backward process of rectified flow model could be derived from the $VE forward SDE$ and $VE backward ODE$ by substituting the $RF notations$ .

Forward SDE:

d r_{s} = - \frac{r _{s}}{1 - s} d s + \frac{2 s}{1 - s} d W_{s}, s \in [0, 1]

Backward ODE:

d r_{s^{'}} = v (r_{s^{'}}, s^{'}) d s^{'}, s^{'} \in [1, 0]

The advantage of the rectified flow notation is its simple backward ODE, which eliminates the diverging behavior of $σ$ at time $T$ found in the VE notation, ensuring that $s$ remains within a finite range of $[0, 1]$ .

Discrete Forward and Backward Processes#

We also list the discrete version of Forward and Backward Processes, which is used in trainining and inference practice.

Discrete Forward SDE: Suppose we discretize the $s \in [0, 1]$ as ${s_{0} = 0, \dots, s_{i}, \dots, s_{n} = 1}$ and $r_{i} = r_{s_{i}}$ , then ³

r_{i} = (1 - s_{i}) r_{0} + s \overset{ˉ}{ϵ}_{s_{i}}, i = 1, \dots, n

where $\overset{ˉ}{ϵ}_{s_{i}}$ is a standard Gaussian noise.

Discrete Backward ODE: Correspondingly, define $s_{k}^{'} = s_{n - k}$ and denote $r_{k} = r_{s_{k}^{'}}$ . This discretizes the backward diffusion process into:

r_{k} = r_{k - 1} + v (r_{k - 1}, s_{k - 1}^{'}) (s_{k}^{'} - s_{k - 1}^{'}), k = 1, \dots, n

Flow Matching Objective#

To directly model $v_{θ} (r_{s^{'}}, s^{'})$ , we transform the $VE denoising objective$ by substituting $\overset{ˉ}{ϵ}_{σ}$ and $ϵ_{θ}$ with $\overset{ˉ}{ϵ}_{s}$ and $v_{θ}$ , respectively. This transformation utilizes the $VE forward$ and $RF forward$ processes and the definitions from $RF notation$ , while also neglecting a constant scaling factor of $(1 - s)$ . As a result, we obtain the flow matching objective:

L_{f l o w} (v_{θ}) = E_{s \sim U [0, 1]} E_{r_{0} \sim p_{0} (x)} E_{\overset{ˉ}{ϵ}_{s} \sim N (0, I)} ∥ \overset{ˉ}{ϵ}_{s} - r_{0} - v_{θ} (r_{s}, s) ∥_{2}^{2},

where $r_{s}$ follows the RF forward process. This represents a re-weighted equivalent of the denoising objective, interpreted in the flow matching framework where $\overset{ˉ}{ϵ}$ corresponds to the endpoint $r_{1}$ and $v_{θ}$ models the velocity field transporting $r_{0}$ to $r_{1}$ .

IMPORTANT
The equivalence between VP, VE, and RF notation#
The three notations (VP, VE, and RF) are mutually transformable through the mappings defined above. Models trained with score matching, denoising, and flow matching objectives can be converted into other notations. This implies that samplers or guidance designed for one notation can be easily transformed and adapted to the others.

What is Next#

Now that we have covered the major theories of diffusion models, including DDPMs, ODE-based diffusion models, and flow models, it is important to note that these models are primarily unconditional. In the next section, we will explore how diffusion models can be utilized to model conditional distributions.

Stay tuned for the next installment!

Discussion#

If you have questions, suggestions, or ideas to share, please visit the discussion post.

Cite this blog#

This blog is a reformulation of the appendix of the following paper.

1
@misc{zheng2025lanpainttrainingfreediffusioninpainting,
2
      title={LanPaint: Training-Free Diffusion Inpainting with Asymptotically Exact and Fast Conditional Sampling},
3
      author={Candi Zheng and Yuan Lan and Yang Wang},
4
      year={2025},
5
      eprint={2502.03491},
6
      archivePrefix={arXiv},
7
      primaryClass={eess.IV},
8
      url={https://arxiv.org/abs/2502.03491},
9
}

Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2020). Score-Based Generative Modeling through Stochastic Differential Equations. arXiv preprint arXiv:2011.13456. ↩ ↩²
Gao, R., Hoogeboom, E., Heek, J., De Bortoli, V., Murphy, K. P., & Salimans, T. (2025). Diffusion Models and Gaussian Flow Matching: Two Sides of the Same Coin. The Fourth Blogpost Track at ICLR 2025. https://openreview.net/forum?id=C8Yyg9wy0s ↩
Liu, X., Gong, C., & Liu, Q. (2022). Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. ArXiv, abs/2209.03003. ↩

ODE and Flow-Based Diffusion Model#

The ODE Based Backward Diffusion Process#

Variance Perserving, Variance Exploding, and Rectified Flow#

Variance Preserving (VP)#

Forward and Backward Processes#

Discrete Forward and Backward Processes#

Score Matching Objective#

Variance Exploding (VE)#

Forward and Backward Processes#

Discrete Forward and Backward Processes#

Denoising Objective#

Rectified Flow (RF)#

Forward and Backward Processes#

Discrete Forward and Backward Processes#

Flow Matching Objective#

The equivalence between VP, VE, and RF notation#

What is Next#

Discussion#

Cite this blog#

Footnotes#