Characteristic Guidance: Non-linear Correction for Diffusion Model at Large Guidance Scale

Candi Zheng*^, Yuan Lan*,
The Hong Kong University of Science and Technology
*Indicates Equal Contribution ^Corresponding to czhengac@connect.ust.hk
Paper (ICML 2024) Stable Diffusion WebUI Extension arXiv

WebUI Extension for Stable Diffusion

We are excited to share our publicly available extension, the Characteristic Guidance Web UI, which provides large CFG (Cassifier-Free Guidance) scale correction for the Stable Diffusion web UI (AUTOMATIC1111). This tool is an application of the methods and theories presented in our paper, offering improved control in sample generation and compatibility with existing sampling methods. For researchers and practitioners interested in exploring our new sampling approach, the extension and its installation instructions can be found on its Github page.

Preliminary Support for ControlNet

MY ALT TEXT

1girl, kimono
Steps: 40, Sampler: DPM++ 2M Karras, CFG scale: 10, Seed: 0, Size: 512x512, Model hash: 1a189f0be6, Model: v1-5-pruned, VAE hash: 735e4c3a44, VAE: vae-ft-mse-840000-ema-pruned.safetensors,
Characteristic Guidance:
Regularization Strength: 1, Regularization Range Over Time: 1, Max Num. Characteristic Iteration: 30, Num. Basis for Correction: 1, Reuse Correction of Previous Iteration: 0, Log 10 Tolerance for Iteration Convergence: -4, Iteration Step Size: 1, Regularization Annealing Speed: 0.4, Regularization Annealing Strength: 0.5, AA Iteration Memory Size: 2,
ControlNet 0: "Module: openpose, Model: control_v11p_sd15_openpose [cab727d4], Weight: 1, Resize Mode: Crop and Resize, Low Vram: False, Processor Res: 512, Guidance Start: 0, Guidance End: 1, Pixel Perfect: False, Save Detected Map: True",
Version: v1.6.1

We are excited to announce that the characteristic guidance WebUI extension now preliminarily supports ControlNet. This new integration offering users the ability to leverage ControlNet's robust control capabilities within the familiar webUI environment. Please note that as this is an initial integration, some features may still be in development, and users might encounter limitations or issues. We are committed to continuously updating and improving this integration and look forward to the community's input and support in this journey. Stay tuned for more updates, and welcome to contribute to our extension development!

How characteristic guidance works

MY ALT TEXT

In comparing classifier-free guidance with characteristic guidance, we examine their behavior during the sampling process at time t. Classifier-free guidance applies a linear combination of positive and negative prompts at a given point along the sampling trajectory. This approach, however, does not adhere to the non-linear dynamics of the Fokker-Planck equation of scores. On the other hand, characteristic guidance leverages the characteristic line of the Fokker-Planck equation of scores, effectively bring the linear combination back to the initial time (t=0). At this point, the Fokker-Planck equation of scores permits linear combinations, thus ensuring that characteristic guidance adheres to the equation’s constraints.

Abstract

Popular guidance for denoising diffusion probabilistic model (DDPM) linearly combines distinct conditional models together to provide enhanced control over samples. However, this approach overlooks nonlinear effects that become significant when guidance scale is large. To address this issue, we propose characteristic guidance, a novel method that provides non-linear correction for classifier-free guided DDPMs. Such correction forces the guided DDPMs to respect the Fokker-Planck equation of their underlying diffusion process, in a way that is first-principle, training-free, derivative-free, and compatible with existing sampling methods. Experiments show that characteristic guidance is robust to various applications, offers enhanced control over sample generation, suppresses color and exposure issues even for latent space sampling, and can handle physics problems such as the phase transitions of magnets.

BibTeX

@misc{zheng2023characteristic,
      title={Characteristic Guidance: Non-linear Correction for DDPM at Large Guidance Scale}, 
      author={Candi Zheng and Yuan Lan},
      year={2023},
      eprint={2312.07586},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}