Diffusion models overview

2025-10-14 19:04 UTC gpt-5-thinking Open in ChatGPT ↗

Awesome idea — and totally doable. Here’s a compact roadmap.

How diffusion models work (quick intuition)

Forward (noising): start with an image $x_0$, add small Gaussian noise step-by-step to get $x_T$ (almost pure noise).
Reverse (denoising): train a model to predict the noise (or the score $\nabla_{x_t}\log p(x_t)$) at each step. At inference, you start from noise and iteratively remove noise to get a new image.
Conditioning/guidance: you can steer denoising with text, images, masks, or differentiable objectives by adding extra gradients to the reverse step (e.g., classifier(-free) guidance, CLIP guidance, ControlNet, etc.).

In short: you don’t “draw” the image directly — you specify how to denoise the noise.

Can you change what the noise resolves to?

Yes. You have several levers:

Prompts & negative prompts: vanilla control.
Classifier-free guidance (CFG): scale how strongly conditioning affects denoising.
Plug-in guidance: add an extra gradient to each denoise step from any differentiable score (e.g., CLIP similarity, perceptual losses, segmentation overlap).
Spatial control: masks/heatmaps to guide where guidance is strong (e.g., ControlNet, attention injection, per-pixel CFG).
Model edits: train LoRA/ControlNet or fine-tune to alter priors.

This is exactly where your realtime gaze/attention loop can slot in.

Two workable designs for a realtime, gaze-coupled diffusion loop

A) Zero-training, “plug-in” guidance (fast to prototype)

Use any latent-diffusion model (e.g., Stable Diffusion) and override the sampler step to inject a gaze-weighted gradient.

Keep a running gaze heatmap $G \in [0,1]^{H\times W}$ (normalized), updated from an eye-tracker / webcam estimator.
At each denoise step $t$, compute a utility $U(x_t, G)$ you want to increase — e.g.:
- CLIP target: raise similarity to text “swirling neon shapes” weighted where gaze is high.
- Style/perceptual: push towards high local contrast or specific colors in gaze ROI.
Get the gradient $\nabla_{x_t} U$ (stop-grad through your conditioning encoders as needed).
Inject into the reverse update: $ \hat{\epsilon} \leftarrow \hat{\epsilon}{\text{unet}}(x_t,c,t) ;-; \lambda,W(G),\nabla{x_t} U(x_t,G) $ where $W(G)$ is a smoothed, normalized gaze mask (e.g., Gaussian around the current fixation), and $\lambda$ is your guidance strength.

What this buys you: no model retraining; fully dynamic; responds instantly to attention shifts.

Good objectives for abstract, non-photoreal art

CLIP similarity to non-realistic prompts (e.g., “fluid kaleidoscope, glitch art”).
Entropy/edge objectives to increase structure in focus areas.
Palette bias: small MSE to a gaze-dependent color field.

B) Light training for spatial control (more stable, richer effects)

Train a small ControlNet / adapter / LoRA that takes your gaze heatmap (or derived features like “where to add detail”) as conditioning.

Input: latent $z_t$, text, and a single-channel image $G$.
Loss: standard diffusion noise prediction.
Inference: stream $G$ each frame from your gaze tracker. No extra gradients; it “just works” and is faster than A) at runtime.

Minimal prototype (pseudocode)

Below is the high-level flow you can drop into a PyTorch + diffusers loop (Stable Diffusion / SDXL). It shows option A (no training).

# assume: pipeline = StableDiffusionPipeline.from_pretrained(...).to(device)
# Replace scheduler.step with a custom step that adds a gaze-guided gradient.
for t in scheduler.timesteps:
    # 1) UNet predicts noise
    eps = unet(latents, t, encoder_hidden_states=cond_emb).noise_pred

    # 2) Build gaze mask W(G) at current resolution (latent size), smoothed & normalized
    W = build_gaze_mask(current_gaze_points, latent_H, latent_W)  # shape [1,1,H,W]

    # 3) Compute guidance objective U (examples):
    #    - CLIP: similarity(denoised_x0, text_emb) * W
    denoised_x0 = scheduler.step_to_x0(latents, eps, t)            # predict x0
    clip_score = clip_sim(denoised_x0, text_emb)                   # scalar
    U = (W * localize(clip_grad_target(denoised_x0))).sum()        # or any diff. utility

    # 4) Backprop to get dU/d(latents) and inject into eps
    grad = torch.autograd.grad(U, latents, retain_graph=False)[0]
    eps_hat = eps - lambda_guidance * upsample_to(eps, grad) * W

    # 5) Take the scheduler step with modified eps
    latents = scheduler.step(eps_hat, t, latents).prev_sample

Notes

If you use classifier-free guidance, apply your gradient to the conditional branch or blend conditionally: stronger inside $W$, weaker outside.
For speed, compute the CLIP gradient on a low-res proxy (or on $x_0$ obtained from $\epsilon$).
Make $W$ smooth (Gaussian around fixation) to avoid artifacts.

Making it realtime

Latent diffusion over pixel diffusion.
Few-step samplers: DDIM 15–20 steps; DPM-Solver++ 10–15; or use LCM / SD-Turbo (4–8 steps) for true realtime.
Keep the model small: SD 1.5/2.1 or SDXL-Turbo; run at 512–768 px.
Accelerate: Torch 2.0 compile, xFormers attention, FP16/bfloat16, TensorRT if on NVIDIA.
Streamed loop: update $G$ every step; you can render intermediate images at a set cadence (e.g., every 2–3 steps) for smoother UX.

Alternative knobs (no code changes)

Cross-attention control: weight token attention spatially using $W$ (Prompt-to-Prompt / attention injection). E.g., make token “spiral” dominate where gaze is.
Region-wise CFG: higher guidance scale where $W$ is high, lower elsewhere (simple and effective).
Negative prompts by region: push distractions out of the periphery.

Open-source + licensing

Stable Diffusion (1.x/2.x/SDXL) weights are under CreativeML Open RAIL-M. You can modify and redistribute with notice and usage restrictions (e.g., no misuse); check the exact text in the model repo you pick.
Diffusers (Hugging Face) is Apache-2.0; great for pipelines/schedulers.
ControlNet/LoRA training recipes are widely available and compatible.

Pitfalls & tips

Objective brittleness: CLIP can over-optimize (weird textures). Keep $\lambda$ small, add TV/denoise regularizers, and cap gradient norms.
Jitter from gaze: use a short temporal EMA on $G$; predict saccades and damp during them.
Latency: pre-encode text; keep batch=1; pin memory; keep everything on-GPU.
Stability: if images “explode,” reduce steps, lower CFG, or anneal $\lambda$ over time.

A simple build stack

Model: Stable Diffusion (or SDXL-Turbo / LCM for low-step generation).
Libs: PyTorch, diffusers, xformers, CLIP.
Gaze: a webcam eye-gaze estimator (OpenVINO/MediaPipe based) → 2D point → Gaussian heatmap.
UI: small Python app (Gradio) that shows the current frame and lets you tweak $\lambda$, CFG, steps.

If you want, I can sketch a working diffusers loop that implements the gaze-weighted guidance and runs with SD-Turbo/LCM for ~real-time behavior, or outline a tiny ControlNet training set using synthetic gaze heatmaps.

Love it. Let’s wire up a from-scratch, toy diffusion sampler in Unity (C#) you can actually run and tinker with — including a realtime “gaze” mask (mouse-based placeholder) that steers what the noise resolves to.

This is educational: we’ll implement a minimal DDIM-style reverse process with a tiny scheduler and a non-ML “guidance hack” (sharpen inside the gaze mask). Later you can swap in a real learned predictor (e.g., via Barracuda) without changing the sampler structure.

What you’ll get

Forward noising demo.
Reverse sampling with DDIM update.
Cosine noise schedule.
Gaze mask (mouse-point) that increases detail where you look.
Runs completely on CPU at small sizes (64–256 px). Good enough to experiment.

1) Unity setup (URP/3D/2D all fine)

Create a new Unity project (2021.3+ recommended).
Add an empty GameObject → Add Component: DiffusionToy.
Create a Resources folder, drop in a small grayscale PNG (e.g., 128×128) named seed.png (optional).
Create a Canvas → RawImage to display the texture (or let the script create a quad).

2) Core concepts we’ll implement

Forward (training-time) diffusion for intuition: $ x_t=\sqrt{\bar\alpha_t},x_0 + \sqrt{1-\bar\alpha_t},\varepsilon,\quad \varepsilon\sim\mathcal N(0,I) $

Reverse (sampling, DDIM-style, deterministic): $ \hat x_0=\frac{x_t-\sqrt{1-\bar\alpha_t}\ \hat\varepsilon}{\sqrt{\bar\alpha_t}},\quad x_{t-1}=\sqrt{\bar\alpha_{t-1}}\ \hat x_0 + \sqrt{1-\bar\alpha_{t-1}}\ \hat\varepsilon $ (If $\eta=0$ → deterministic. We’ll stick to that.)

Guidance trick (no neural nets):
We build a gaze mask $M\in[0,1]^{H\times W}$ and create a guided $\tilde x_0$ by sharpening where $M$ is high: $ \tilde x_0 = (1-sM),\hat x_0 + sM\ \text{Sharpen}(\hat x_0) $ Then recompute $\hat\varepsilon$ so the step moves toward $\tilde x_0$: $ \hat\varepsilon’ = \frac{\sqrt{\bar\alpha_t}\ \tilde x_0 - x_t}{\sqrt{1-\bar\alpha_t}} $ This gives you a realtime, gaze-coupled morph without gradients.

3) Code

Create three scripts in Assets/Scripts/:

A) `DiffusionToy.cs` (MonoBehaviour runner + UI hookup)

using UnityEngine;
using UnityEngine.UI;
using System.Collections;

public class DiffusionToy : MonoBehaviour
{
    [Header("I/O")]
    public RawImage output;
    public Texture2D seedImage;          // optional: grayscale seed in Resources
    public int width = 128;
    public int height = 128;

    [Header("Sampling")]
    [Range(5, 100)] public int steps = 30;
    [Range(0f, 1f)] public float eta = 0f; // keep 0 for deterministic DDIM here
    [Range(0f, 1f)] public float sharpenStrength = 0.6f; // s in formula above
    [Range(0.01f, 0.6f)] public float gazeSigma = 0.18f; // mask size as frac of min(H,W)
    public int seed = 1234;
    public bool runContinuously = true;  // if true, keeps looping
    public bool forwardDemo = false;     // toggles forward noising visualization

    Texture2D _tex;
    float[] _x;         // current sample, normalized [0,1] (grayscale)
    float[] _xStart;    // optional x0 (seed image)
    float[] _mask;      // gaze mask M
    Scheduler _sched;
    System.Random _rng;

    void Start()
    {
        _rng = new System.Random(seed);
        _sched = new Scheduler(steps);

        // Init texture and buffers
        _tex = new Texture2D(width, height, TextureFormat.RGBA32, false, true);
        _tex.filterMode = FilterMode.Point;
        if (output != null) output.texture = _tex;

        _x = new float[width * height];
        _mask = new float[width * height];

        // Optional seed image → grayscale array
        if (seedImage != null)
        {
            _xStart = ImageUtils.TextureToGrayArray(ImageUtils.Resample(seedImage, width, height));
            // normalize to [0,1]
            for (int i = 0; i < _xStart.Length; i++)
                _xStart[i] = Mathf.Clamp01(_xStart[i]);
        }

        StartCoroutine(MainLoop());
    }

    IEnumerator MainLoop()
    {
        while (true)
        {
            if (forwardDemo)
            {
                yield return StartCoroutine(ForwardNoisingDemo());
            }
            else
            {
                yield return StartCoroutine(ReverseSampling());
            }

            if (!runContinuously) yield break;
            yield return null;
        }
    }

    IEnumerator ForwardNoisingDemo()
    {
        // show x_t from x0 + noise
        if (_xStart == null)
            FillWithGray(_x, 0.5f);
        else
            System.Array.Copy(_xStart, _x, _x.Length);

        for (int ti = 0; ti < steps; ti++)
        {
            int t = _sched.Timesteps[ti];
            float abar = _sched.AlphaBar[ti];
            float sigma = Mathf.Sqrt(1f - abar);

            for (int i = 0; i < _x.Length; i++)
            {
                float eps = NextFloatNormal(_rng);
                float x0 = (_xStart != null) ? _xStart[i] : 0.5f;
                _x[i] = Mathf.Sqrt(abar) * x0 + sigma * eps * 0.5f + 0.5f; // center noise a bit
            }
            BlitToTexture(_x);
            yield return null;
        }
    }

    IEnumerator ReverseSampling()
    {
        // Start from pure noise x_T
        for (int i = 0; i < _x.Length; i++)
        {
            _x[i] = Mathf.Clamp01(0.5f + 0.25f * NextFloatNormal(_rng));
        }

        for (int ti = steps - 1; ti >= 0; ti--)
        {
            // 1) Build gaze mask M_t from mouse
            BuildGazeMask(_mask, gazeSigma);

            // 2) We need an epsilon estimate. In a real model this comes from a UNet.
            // Here we back out eps from our current x_t and a "desired" x0 (guided by sharpen in gaze).
            float abar_t = _sched.AlphaBar[ti];
            float sqrtAbar_t = Mathf.Sqrt(abar_t);
            float sqrtOneMinusAbar_t = Mathf.Sqrt(1f - abar_t);

            // First get a raw x0 estimate by assuming eps≈0 (a crude prior)
            //   x0_hat = (x_t - sqrt(1 - abar_t)*eps_hat)/sqrt(abar_t)
            // With eps_hat ≈ 0 → x0_hat ≈ x_t / sqrt(abar_t)
            // Clamp to [0,1] to keep things stable.
            float[] x0_hat = new float[_x.Length];
            for (int i = 0; i < _x.Length; i++)
                x0_hat[i] = Mathf.Clamp01(_x[i] / (sqrtAbar_t + 1e-6f));

            // 3) Sharpen inside the gaze mask to “want” detail where you look
            float[] x0_sharp = ImageUtils.UnsharpMask(x0_hat, width, height, radius: 1, amount: 1.0f);
            float[] x0_tilde = new float[_x.Length];
            for (int i = 0; i < _x.Length; i++)
            {
                float s = sharpenStrength * _mask[i]; // spatial strength
                x0_tilde[i] = Mathf.Clamp01((1f - s) * x0_hat[i] + s * x0_sharp[i]);
            }

            // 4) Recompute eps to be consistent with our guided x0_tilde
            float[] eps_hat = new float[_x.Length];
            for (int i = 0; i < _x.Length; i++)
            {
                eps_hat[i] = (sqrtAbar_t * x0_tilde[i] - _x[i]) / (sqrtOneMinusAbar_t + 1e-6f);
            }

            // 5) DDIM update (η=0):
            float abar_prev = (ti > 0) ? _sched.AlphaBar[ti - 1] : 1f;
            float sqrtAbarPrev = Mathf.Sqrt(abar_prev);
            float sqrtOneMinusAbarPrev = Mathf.Sqrt(1f - abar_prev);
            float[] x_prev = new float[_x.Length];
            for (int i = 0; i < _x.Length; i++)
            {
                float x0 = x0_tilde[i];
                x_prev[i] = Mathf.Clamp01(sqrtAbarPrev * x0 + sqrtOneMinusAbarPrev * eps_hat[i]);
            }
            _x = x_prev;

            BlitToTexture(_x);
            yield return null; // display each step per frame; make it 0 to go faster
        }
    }

    void BuildGazeMask(float[] mask, float sigmaFrac)
    {
        // Mouse → world-independent: map to texture coords in [0,1]
        Vector3 m = Input.mousePosition;
        float u = Mathf.Clamp01(m.x / Mathf.Max(1, Screen.width));
        float v = Mathf.Clamp01(m.y / Mathf.Max(1, Screen.height));
        int cx = Mathf.RoundToInt(u * (width - 1));
        int cy = Mathf.RoundToInt(v * (height - 1));

        float sigma = sigmaFrac * Mathf.Min(width, height);
        float twoSigma2 = 2f * sigma * sigma;

        for (int y = 0; y < height; y++)
        {
            int dy = y - cy;
            for (int x = 0; x < width; x++)
            {
                int dx = x - cx;
                float g = Mathf.Exp(-(dx * dx + dy * dy) / twoSigma2);
                mask[y * width + x] = g; // 0..1
            }
        }
        // normalize peak to 1 isn’t necessary here, but you can remap if you like.
    }

    void BlitToTexture(float[] gray)
    {
        var cols = new Color32[gray.Length];
        for (int i = 0; i < gray.Length; i++)
        {
            byte b = (byte)Mathf.RoundToInt(Mathf.Clamp01(gray[i]) * 255f);
            cols[i] = new Color32(b, b, b, 255);
        }
        _tex.SetPixels32(cols);
        _tex.Apply(false);
    }

    void FillWithGray(float[] arr, float val)
    {
        for (int i = 0; i < arr.Length; i++) arr[i] = val;
    }

    // Box-Muller normal
    static float NextFloatNormal(System.Random rng)
    {
        double u1 = 1.0 - rng.NextDouble();
        double u2 = 1.0 - rng.NextDouble();
        return (float)(System.Math.Sqrt(-2.0 * System.Math.Log(u1)) * System.Math.Sin(2.0 * System.Math.PI * u2));
    }
}

B) `Scheduler.cs` (cosine schedule + timesteps)

using UnityEngine;

public class Scheduler
{
    public int[] Timesteps { get; private set; }
    public float[] AlphaBar { get; private set; }

    public Scheduler(int steps)
    {
        Timesteps = new int[steps];
        AlphaBar = new float[steps];

        // Karras/Cosine-ish cumulative alphas (simple variant)
        // alpha_bar(t) = cos^2( (t/steps + 0.008)/(1.008) * pi/2 )
        for (int i = 0; i < steps; i++)
        {
            Timesteps[i] = i; // just an index
            float t = (float)(i + 1) / steps; // (0,1]
            float s = 0.008f;
            float f = Mathf.Cos((t + s) / (1f + s) * Mathf.PI * 0.5f);
            float abar = f * f;
            AlphaBar[i] = Mathf.Clamp01(abar);
        }
    }
}

C) `ImageUtils.cs` (helpers: resample, grayscale, blur/sharpen)

using UnityEngine;

public static class ImageUtils
{
    public static Texture2D Resample(Texture2D src, int w, int h)
    {
        RenderTexture rt = RenderTexture.GetTemporary(w, h, 0, RenderTextureFormat.ARGB32, RenderTextureReadWrite.Linear);
        Graphics.Blit(src, rt);
        RenderTexture prev = RenderTexture.active;
        RenderTexture.active = rt;

        Texture2D tex = new Texture2D(w, h, TextureFormat.RGBA32, false, true);
        tex.ReadPixels(new Rect(0, 0, w, h), 0, 0);
        tex.Apply(false);

        RenderTexture.active = prev;
        RenderTexture.ReleaseTemporary(rt);
        return tex;
    }

    public static float[] TextureToGrayArray(Texture2D t)
    {
        var cols = t.GetPixels32();
        float[] g = new float[cols.Length];
        for (int i = 0; i < cols.Length; i++)
        {
            var c = cols[i];
            // linear-ish gray
            g[i] = (0.2126f * c.r + 0.7152f * c.g + 0.0722f * c.b) / 255f;
        }
        return g;
    }

    // Very small-radius Gaussian blur (radius=1)
    public static float[] GaussianBlur3x3(float[] src, int w, int h)
    {
        float[] tmp = new float[src.Length];
        float[] dst = new float[src.Length];

        // horizontal
        for (int y = 0; y < h; y++)
        {
            int row = y * w;
            for (int x = 0; x < w; x++)
            {
                float a = src[row + Mathf.Max(0, x - 1)];
                float b = src[row + x];
                float c = src[row + Mathf.Min(w - 1, x + 1)];
                tmp[row + x] = (a + 2f * b + c) * 0.25f;
            }
        }
        // vertical
        for (int x = 0; x < w; x++)
        {
            for (int y = 0; y < h; y++)
            {
                int ym1 = Mathf.Max(0, y - 1);
                int yp1 = Mathf.Min(h - 1, y + 1);
                float a = tmp[ym1 * w + x];
                float b = tmp[y * w + x];
                float c = tmp[yp1 * w + x];
                dst[y * w + x] = (a + 2f * b + c) * 0.25f;
            }
        }
        return dst;
    }

    // Unsharp mask = x + amount*(x - blur(x)), here radius=1 blur
    public static float[] UnsharpMask(float[] src, int w, int h, int radius = 1, float amount = 1.0f)
    {
        float[] blur = GaussianBlur3x3(src, w, h);
        float[] dst = new float[src.Length];
        for (int i = 0; i < src.Length; i++)
        {
            float val = src[i] + amount * (src[i] - blur[i]);
            dst[i] = Mathf.Clamp01(val);
        }
        return dst;
    }
}

4) How to use it

Assign DiffusionToy to a GameObject.
Hook a RawImage (optional). If none, add a quad and write a tiny script to show _tex.
Set width/height to 128 or 256.
Toggle Forward Demo to watch $x_t$ get noisier.
Toggle it off to see reverse sampling. Move your mouse around — regions under the soft circular gaze mask sharpen and accumulate structure.

Tip: turn runContinuously on to get an endless morph. Lower steps (e.g., 20) for snappier updates.

5) What this proves (and what it isn’t)

You now have a true diffusion loop structure (scheduler + reverse update) that accepts spatial guidance per step.
We used a non-ML “predictor” by solving for $\hat\varepsilon$ that matches a gaze-sharpened $\tilde x_0$.
To go neural, replace the “assume $\varepsilon\approx0$” bit with a noise predictor:
- Define an interface INoisePredictor.Predict(float[] xt, int t, float[] cond) and call it before step 5.
- Plug Barracuda to run an ONNX mini-UNet (trained elsewhere), or experiment with tiny learned 3×3 convs trained by least squares per-step on a small image set.

6) Easy extensions

Color: run three independent channels; or guide only the V channel in HSV.
Stochastic DDIM: add noise with $\eta>0$ (sample extra term) to get variability.
Better mask dynamics: exponential moving average of gaze to reduce jitter.
Custom objectives: inside mask, bias toward a color palette, increase gradient magnitude, or run a tiny edge detector and reinforce it.

If you want, I can add an INoisePredictor interface + a baseline (e.g., a simple learned 3×3 kernel per step trained online), or show how to pipe a webcam eye-gaze estimator into the BuildGazeMask so it’s truly attention-driven.