Open models for uncanny AI

2025-12-12 09:35 UTC gpt-5-2-thinking Open in ChatGPT ↗

Yep — you can get very close to that “DALL-E 2 era” uncanny vibe, but there’s a catch: DALL-E 2 itself wasn’t open, and most “AI video” from that period (2021–early 2022) was either research demos or closed systems. The good news is that open models from ~2019–2022 (plus intentionally “undertrained”/low-guidance settings in later open models) can recreate the same weirdness.

Here are the best open options that match that time’s look:

1) The closest “early uncanny” open generators (images → animate)

VQGAN+CLIP (2021)
This is the classic “dreamy, melty, wrong textures, unsettling semantics” look. It’s not a diffusion model; it’s an older CLIP-guided optimization approach, which is why it feels so uncanny.

You can generate frames with VQGAN+CLIP and then animate them (zoom/pan, optical flow, or frame interpolation).

BigGAN + CLIP guidance (2019–2021-ish workflows)
Also very “wrong but confident” in a similar way, though often more “poster-like” than DALL-E 2.

StyleGAN2 / StyleGAN3 (2019–2021)
Great for uncanny faces/people when you push it out of distribution (latent walks, mixing, truncation weirdness). Works well for “uncanny video” via latent interpolation.

2) Early open diffusion models that can feel “DALL-E 2-ish”

Stable Diffusion 1.4 / 1.5 (2022)
Not as early as DALL-E 2, but if you:

lower steps,
use weaker/odd prompts,
reduce CFG guidance,
use older samplers, you can get that “almost right, but cursed” look very reliably.

Older latent diffusion repos / checkpoints (2021–2022 research) Some early Latent Diffusion Model checkpoints exist in open research codebases (often less polished than SD).

3) “AI video” (open) that resembles that era

If you want actual generative video models, open ones from that exact period are limited, but you can recreate the vibe with:

Frame-by-frame generation (VQGAN+CLIP or SD1.x) + consistent seed tricks
Optical-flow guided “video diffusion” notebooks (more recent, but can be tuned to look early/uncanny)
Latent walks (StyleGAN) exported as video

How to make it look more like early 2022 even with newer tools

Even if you use later open models, you can “time-travel” the aesthetics by intentionally degrading coherence:

Use lower resolution (256–512), then upscale poorly or minimally
Keep low step counts (e.g., 10–25)
Use low/medium CFG (too high looks modern/crisp)
Prompt with underspecified or contradictory phrases
Add noise / JPEG artifacts / film grain
Avoid modern “video-consistency” tools; let it drift

Quick practical recommendation (if you just want results fast)

VQGAN+CLIP for frames (uncanny by default) → animate with simple motion or interpolation
Or Stable Diffusion 1.4/1.5 with low steps + modest CFG → generate 12–24 keyframes → interpolate to 8–12 fps

If you tell me what vibe you mean (e.g., “melting objects,” “almost-human faces,” “wrong text,” “surreal rooms,” etc.) and what hardware you have (CPU only / consumer GPU / cloud), I can point you to the most suitable open repo + settings to get that exact early-uncanny look.

Totally doable — for “bodies melting into each other / cursed walking / impossible poses” the easiest way is early-ish open text-to-image + animation tricks that don’t try too hard to keep consistency (that’s what makes it feel like 2021–2022).

Best “early uncanny” open stacks for this exact vibe

1) Disco Diffusion (CLIP-guided diffusion, 2021–2022 vibe)

Produces that dreamlike, semi-coherent anatomy where limbs fuse and motion feels wrong.
Also has built-in “3D animation” / motion modes in the notebook era.
Start here if you want the most authentic “early AI video” feel. citeturn0search0turn0search3

2) VQGAN+CLIP (the melty, wrong-body classic)

Very good at “people merging / bodies turning into blobs” because it’s optimizing images toward CLIP rather than “understanding” anatomy.
There are variants explicitly geared toward zooming / keyframes, which can be turned into video easily. citeturn0search1turn0search15turn0search7

3) Stable Diffusion 1.4 / 1.5 + Deforum (2022-style animation)

SD1.x can be pushed into “uncanny bodies” if you deliberately lower coherence (low steps, mid/low CFG, higher denoise).
Deforum is a popular open animation pipeline for SD that stitches frames into video and encourages that drifting, glitchy motion. citeturn0search5turn0search2turn0search10turn0search18

Settings to force “weird walking / melting people” (works especially well in SD1.x + Deforum)

Try these knobs (they’re basically “make it 2022 again”):

Resolution: 512×512 (or even 448×448)
Steps: 12–25
CFG: 4–7 (too high looks “modern” and overly crisp)
Denoise strength (img2img / Deforum): 0.55–0.8 (higher = more melting / identity loss)
Add motion: camera translate/rotate + a bit of noise each frame (Deforum), or just generate keyframes and interpolate

Prompting tips (since you don’t care about faces):

Put the body weirdness in the positive prompt:
“contorted full body, twisted limbs, unnatural gait, body horror, limbs fused, people merging, liquid anatomy, incorrect anatomy”
Put “faces” in negative prompt:
“face, portrait, detailed face, eyes, teeth”
Also add: “full body, wide shot” to stop it from zooming into heads.

Quick “choose your weapon”

If you want maximum cursed melting: VQGAN+CLIP citeturn0search1turn0search15
If you want authentic early AI-video drift: Disco Diffusion citeturn0search3turn0search0
If you want fast + controllable + lots of motion: SD1.4/1.5 + Deforum citeturn0search5turn0search10

If you tell me what you’ll run on (Colab vs local, and GPU model if local), I’ll give you a concrete “copy these settings” recipe for one of the three (including a sample Deforum JSON / prompt template for weird walking + melting crowds).

You can set up SD 1.4 + Deforum in AUTOMATIC1111, and you can involve TensorRT — but one important reality check:

Deforum is a frame-sequence renderer, not a true “real-time” video synth. You can get fast (seconds per frame → sub-second per frame on high-end GPUs), but “interactive live video at 24–60fps” isn’t what Deforum was designed for.
If you truly need real-time-ish, I’ll also point you to the more “live pipeline” option at the end (ComfyUI + TensorRT).

That said, here’s the cleanest setup path.

Option A (most straightforward): AUTOMATIC1111 + Deforum + NVIDIA TensorRT extension

1) Install AUTOMATIC1111

Install/run AUTOMATIC1111 Stable Diffusion WebUI normally.

2) Add Deforum extension

Clone into your WebUI extensions folder and restart WebUI:

Deforum (official webui extension): citeturn0search0

3) Put SD 1.4 checkpoint in place

Put your sd-v1-4 .ckpt / .safetensors in: stable-diffusion-webui/models/Stable-diffusion/

(You can run SD 1.4 in A1111 just like 1.5; architecture is the same class of model.)

4) Add TensorRT acceleration (recommended: NVIDIA’s extension)

Install NVIDIA’s TensorRT extension for A1111: citeturn0search1turn0search22turn0search9
Then generate TensorRT “engines” (optimized builds) for the resolutions you’ll use.

Key TensorRT gotchas (important):

You generally need separate engines per resolution (and sometimes per batch size / setting). So pick a small set like 512×512 and maybe 768×432 (16:9-ish) and stick to them.
Some features can be limited depending on the TensorRT path you use (e.g., certain conditioning add-ons). The old A1111 TensorRT repo also documents some limitations. citeturn0search20

5) Deforum settings for your “weird walking / melting bodies” look

In Deforum:

Use 512×512 (or 512×288 / 576×320 if you’re doing widescreen)
Steps: 12–20
CFG: 4–7
Strength (img2img / animation): 0.6–0.8 (higher = more “melting into each other”)
Add a bit of camera motion (translate/rotate) + noise to encourage drift.

Prompt starter (faces not needed):

Positive: full body, wide shot, contorted pose, twisted limbs, unnatural gait, people melting together, limbs fused, surreal movement, body horror, incorrect anatomy
Negative: face, portrait, close-up, detailed face, eyes, teeth

About “real-time”: what’s realistic with Deforum + TensorRT

With TensorRT you can often roughly double performance in A1111 workflows (varies by GPU/model/settings). citeturn0search22turn0search9
But Deforum still has overhead (frame management, warping, saving, etc.). You can aim for:

near-real-time preview at low res / low steps
then a final render pass at better settings

If you truly mean live generation responding instantly (VJ-style), Deforum is usually not the best core.

Option B (better for “real-time” pipelines): ComfyUI + TensorRT

ComfyUI has an official TensorRT node pack that supports SD 1.5/2.1/SDXL/etc. citeturn0search2turn0search5
Even though it lists SD 1.5 explicitly, SD 1.4 is the same family; many people just run 1.5 for compatibility and then push it “older/weirder” via settings.

Why ComfyUI helps for “real-time-ish”:

You can build a graph that accepts changing prompts per frame, or a stream of seeds/time, and you can keep everything on-GPU with fewer UI-side bottlenecks.

I need 3 quick details to give you the exact “do this, paste that” install steps

OS: Windows or Linux?
GPU model + VRAM: (e.g., RTX 3060 12GB, 4090 24GB)
Your “real-time” target: fps or seconds/frame at what resolution?

Meanwhile, if you want to proceed immediately without waiting: do Option A (A1111 + Deforum + NVIDIA TensorRT extension) and restrict yourself to one resolution (512×512) so engine-building stays sane.

Great questions — these two things are historical artifacts of how Stable Diffusion evolved, so the confusion is totally reasonable.

1. What is AUTOMATIC1111?

AUTOMATIC1111 (often called A1111) is the de-facto open-source Stable Diffusion UI/runtime from 2022.

Think of it as:

A Python application that loads Stable Diffusion models and exposes them through a web interface + plugin system

It is not a model, it is:

a runner for SD models (1.4, 1.5, 2.x, etc.)
a sampler / scheduler controller
a place where extensions (like Deforum) hook in
a bridge between “raw PyTorch code” and “usable workflows”

Why everyone used it (and still does)

In the SD 1.4 / 1.5 era:

Raw SD repos were research-grade
AUTOMATIC1111 turned SD into something usable without writing code
It standardized:
- prompts
- CFG / steps
- img2img
- batch generation
- extensions

Historically:
If someone says “Stable Diffusion workflow from 2022”, they almost always mean SD 1.x + AUTOMATIC1111.

What it is not

Not required by Stable Diffusion
Not fast by default
Not designed for real-time video

It’s just the most common host for SD models.

2. Why would you even need Deforum?

Short answer:

Stable Diffusion makes images. Deforum turns image generation into motion.

What SD does by itself

Out of the box, SD:

generates single images
or does img2img (one image → one image)

There is no concept of time or frames.

What Deforum adds

Deforum is an extension that:

generates a sequence of frames
feeds the previous frame back into SD (img2img)
applies camera transforms (translate, rotate, zoom)
injects noise over time
manages seeds, strength, cadence, schedules

This is what creates:

drifting motion
melting bodies
“people walking wrong”
the early AI video look

Without Deforum, you’d have to:

manually generate frames
write your own frame loop
handle motion math yourself

Deforum is basically:

“A hacked-together temporal loop around SD”

Which is exactly why it looks so uncanny.

Why Deforum is especially good for your use case

You want:

bodies merging
unstable anatomy
motion that almost makes sense
no face consistency

Deforum excels because:

It does not enforce identity consistency
It lets anatomy drift frame-to-frame
It encourages temporal instability
It predates modern “coherent video diffusion”

Modern video models try to prevent exactly what you want.

Do you have to use Deforum?

No — but here’s the tradeoff:

Option	What you get	What you lose
No Deforum	Full control, simpler pipeline	You must write your own animation loop
Deforum	Proven “2022 uncanny video” look	Not real-time, heavy
ComfyUI graph	Faster, modular, RT-friendly	Less historically authentic
Custom Python loop	Maximum control	More engineering

If your goal is authentic early-AI uncanniness, Deforum is historically correct.

If your goal is live performance, Deforum is a stepping stone — not the final architecture.

Mental model (important)

Think of the stack like this:

Stable Diffusion 1.4   ← the model
AUTOMATIC1111         ← the engine / host
Deforum               ← temporal hack on top
TensorRT              ← speed optimization layer

If you want, next I can:

Draw you a minimal real-time-ish pipeline (no Deforum, but same aesthetic)
Give you a Deforum preset specifically tuned for “melting walkers”
Help you decide when to ditch A1111/Deforum and move to ComfyUI or a custom loop

Just tell me which direction you want to go.

To get “Deforum vibes” (warped motion + recursive img2img drift + melting bodies) at real-time speeds, you basically have to change two things:

Stop doing full multi-step diffusion per frame (Deforum’s core bottleneck)
Treat it as a streaming video-to-video loop, not “render 600 frames offline”

The modern recipe is:

(A) a fast diffusion variant (LCM / Turbo / Lightning) + (B) a streaming pipeline (StreamDiffusion) + (C) optional TensorRT + (D) optional frame interpolation

Here are the best practical paths.

Path 1: StreamDiffusion (the “real-time Deforum replacement” idea)

StreamDiffusion is explicitly designed for real-time/streaming diffusion generation (interactive throughput rather than offline rendering). citeturn0search8turn0search16

What it gives you that matches Deforum’s look:

You can feed the previous output / an input stream continuously
You get that temporal drift / morphing / melting feel
It’s meant for “live” use cases rather than “render frames to disk”

If you want Deforum-style weird walking, you do it by:

driving the input with a moving latent / noise schedule
keeping prompts broad (“full body, contorted pose, unnatural gait…”)
letting the stream “forget” and re-form (controlled instability)

Best when: you mean “real-time-ish” as in interactive visuals (VJ, live camera transform, game-like responsiveness).

Path 2: LCM / LCM-LoRA (reduce steps to ~2–4) + TensorRT

Deforum’s killer is step count. Latent Consistency Models (LCM) and LCM-LoRA are designed to make SD generate in a few steps instead of dozens, and NVIDIA specifically discusses big speedups when paired with TensorRT. citeturn0search0turn0search6

How it maps to Deforum:

You still do a frame loop (like Deforum), but each frame is 2–4 steps, not 20–50.
The look becomes more “alive” and unstable (often a plus for your uncanny body-meld goal).

Best when: you want to keep a Deforum-like “previous frame → img2img” concept, but actually make it fast.

Path 3: ComfyUI + TensorRT nodes (fastest practical “graph” setup)

If you want performance + modularity, ComfyUI_TensorRT is the most direct “plug TensorRT into diffusion workflows” path right now. citeturn0search1turn0search30

Why this matters for “real-time Deforum”:

You can build a graph that replicates Deforum’s core loop:
- previous frame → warp (optional) → img2img → output → feedback
Keep it entirely GPU-resident where possible
Add TensorRT-accelerated helpers like pose estimation or frame interpolation (there are TensorRT nodes for things like RIFE interpolation). citeturn0search3

Best when: you want a “production-ish” live pipeline and don’t care about being historically faithful to A1111/Deforum.

What “real-time” usually means in practice

Most people hit “live-feeling” results by combining:

Diffusion at 5–15 fps (depending on GPU, res, steps, model)
Frame interpolation (RIFE) to display at 30–60 fps citeturn0search3
Lower resolution (e.g., 512 on diffusion, upscale later if needed)

This is very common in live AV setups: generate fewer “true” frames and interpolate for smoothness.

Where TensorRT fits (and where it doesn’t)

TensorRT can help a lot, but it doesn’t magically turn 20-step SD into 60fps.

TensorRT helps most when:

you fix resolution(s) and batch shapes
you use few-step models (LCM/Turbo/etc.) so the pipeline isn’t dominated by repeated denoising NVIDIA explicitly frames “real-time” gains around fast variants + TensorRT. citeturn0search0turn0search17

The blunt truth (so you don’t waste time)

If you try to make Deforum itself real-time, you’ll fight its design:

lots of Python overhead
offline assumptions
multi-step per frame

If you instead build a streaming vid2vid diffusion loop, you’ll get:

the same “melting walkers” aesthetic
much better latency

A concrete “do this” recommendation for your goal

If you want the closest “Deforum feel” but live:

ComfyUI + ComfyUI_TensorRT citeturn0search1
Use SD1.5 + LCM-LoRA (SD1.4 is fine aesthetically, but SD1.5 tends to have better tooling compatibility) citeturn0search0turn0search6
Build a loop: Prev frame → (optional warp) → img2img (strength 0.6–0.8) → output → feedback
Add RIFE TensorRT interpolation to present at 30–60fps citeturn0search3

If you want a ready-made “streaming” concept instead of building a loop graph, start with StreamDiffusion. citeturn0search8turn0search16

If you tell me your GPU model + VRAM and your target (e.g., 512px at 10fps, or “needs to look like 30fps output”), I’ll give you a specific pipeline choice (StreamDiffusion vs ComfyUI loop), plus the exact speed-oriented settings (steps / model type / interpolation strategy) to hit it.

Perfect choice. I’ll give you a clean, reproducible ComfyUI install path that sets you up for:

SD 1.x
LCM (few-step diffusion)
TensorRT acceleration
A Deforum-like realtime feedback loop

I’ll write this assuming Linux or Windows + NVIDIA GPU. I’ll clearly mark OS-specific steps.

Overview: what you’re installing (so it makes sense)

You’ll end up with this stack:

ComfyUI
 ├─ Stable Diffusion 1.x checkpoint
 ├─ LCM-LoRA (2–4 step diffusion)
 ├─ ComfyUI Manager (for nodes)
 ├─ ComfyUI-TensorRT (speed)
 └─ (optional) RIFE interpolation (smooth output)

This is already enough to recreate Deforum’s behavior in a live loop.

0. Prerequisites (don’t skip)

Hardware

NVIDIA GPU (RTX 20xx+ strongly recommended)
8 GB VRAM minimum (12–24 GB ideal)

Software

Python 3.10.x (important)
CUDA 11.8 or 12.x
Git

Check GPU + CUDA:

nvidia-smi

1. Install ComfyUI (base)

Clone

git clone https://github.com/comfyanonymous/ComfyUI.git
cd ComfyUI

Create virtual environment (recommended)

python -m venv venv

Activate:

Linux

source venv/bin/activate

Windows

venv\Scripts\activate

Install dependencies

pip install -r requirements.txt

Test run

python main.py

Open browser → http://127.0.0.1:8188

If ComfyUI loads, stop it (Ctrl+C) and continue.

2. Install ComfyUI Manager (essential)

This lets you install almost everything else inside the UI.

cd custom_nodes
git clone https://github.com/ltdrdata/ComfyUI-Manager.git
cd ..

Restart ComfyUI:

python main.py

In the UI, you should now see “Manager” in the menu.

3. Install Stable Diffusion 1.x model

You can use SD 1.4 or 1.5.
(1.5 is more compatible; you can still make it “look old.”)

Place checkpoint here:

ComfyUI/models/checkpoints/

Example filename:

sd-v1-5.safetensors

Restart ComfyUI once it’s in place.

4. Install LCM (this is what makes it fast)

Install via Manager

In ComfyUI:

Manager → Install Models
Search for LCM
Install LCM-LoRA for SD1.x

Or manually:

ComfyUI/models/loras/
  └─ lcm-lora-sdv1-5.safetensors

Why this matters

LCM lets you run 2–4 steps per frame instead of 20–50.
This is the key to realtime diffusion.

5. Install ComfyUI-TensorRT (speed layer)

This is the most sensitive part.

Install node

cd custom_nodes
git clone https://github.com/comfyanonymous/ComfyUI_TensorRT.git
cd ..

Install TensorRT Python bindings

Make sure CUDA is installed first.

pip install nvidia-tensorrt
pip install polygraphy onnx onnxruntime-gpu

⚠️ TensorRT works best with fixed resolutions.
Decide now: 512×512 (recommended).

Restart ComfyUI.

You should now see TensorRT nodes available.

6. (Optional but recommended) RIFE interpolation

This makes output look 30–60fps even if diffusion is 8–12fps.

Via Manager:

Manager → Install Custom Nodes
Search RIFE
Install a RIFE node pack (several exist)

Restart ComfyUI.

7. Folder sanity check (important)

You should now have:

ComfyUI/
├─ models/
│  ├─ checkpoints/   ← SD 1.x model
│  ├─ loras/         ← LCM LoRA
│  └─ vae/
├─ custom_nodes/
│  ├─ ComfyUI-Manager
│  └─ ComfyUI_TensorRT

8. What you can now do (conceptually)

You are now ready to build a Deforum-like realtime loop:

Previous frame
   ↓
(img2img, denoise 0.6–0.8)
   ↓
LCM (2–4 steps)
   ↓
TensorRT acceleration
   ↓
(optional RIFE interpolation)
   ↓
Display / feedback into next frame

This is functionally Deforum — but live.

9. Very important performance rules (read this)

To stay realtime:

Resolution: 512×512
Steps: 2–4 (LCM)
CFG: 1.5–3 (LCM prefers low CFG)
Denoise: 0.6–0.8 (melting bodies live here)
Fixed resolution for TensorRT engines

If you break these, speed collapses.

10. Next steps (tell me what you want next)

I can now:

Give you a ready-to-load ComfyUI workflow JSON that already loops frames
Help you build TensorRT engines properly (one-time step)
Tune prompts specifically for weird walking / melting crowds
Show how to hook this into live camera / OSC / MIDI / VJ tools

👉 Tell me:

OS
GPU model + VRAM
Target: “interactive” vs “locked 30fps output”

I’ll tailor the next step exactly to your machine.

Good question — this matters more than people think, especially for the “early-era uncanny body” look you’re after.

Below is a safe, reliable way to get the right kind of Stable Diffusion model for SD 1.4 / 1.5–style results, without accidentally grabbing a modern, over-trained aesthetic model.

TL;DR (quick answer)

For your use case:

SD 1.4 or 1.5 base models
From Hugging Face or official mirrors
NOT custom “photoreal / anime / realism” checkpoints

Put the file in:

ComfyUI/models/checkpoints/

Best sources (trusted + historically accurate)

1. Hugging Face (primary source)

This is the canonical place.

SD 1.4 (closest to DALL-E 2 era) Search on Hugging Face for:

CompVis stable-diffusion-v1-4

File name usually:

sd-v1-4.ckpt   or   sd-v1-4.safetensors

SD 1.5 (more compatible, still old-looking) Search:

runwayml stable-diffusion-v1-5

File:

sd-v1-5.safetensors

💡 Recommendation:
Use SD 1.5 for tooling compatibility, then degrade it via settings (low steps, low CFG, high denoise) to get the uncanny look.

2. What NOT to download (important)

Avoid:

“Realistic Vision”
“DreamShaper”
“Deliberate”
“Anything vX”
“Juggernaut”
“SDXL”
Anything trained for faces, portraits, or photorealism

These models actively fight the body-melting behavior you want.

Why base models are better for weird bodies

Base SD 1.x models:

have poor anatomical priors
don’t strongly enforce pose correctness
drift frame-to-frame beautifully
melt under img2img recursion

Fine-tuned models:

over-correct anatomy
lock identity
resist melting
look “too modern”

safetensors vs ckpt (which to choose)

Choose .safetensors if available:

safer
faster to load
same output quality

ComfyUI supports both.

Optional but recommended: VAE choice

If your outputs look washed out or gray:

Use the original SD 1.x VAE (often bundled)
Or the common vae-ft-mse-840000 VAE

Place VAEs in:

ComfyUI/models/vae/

Sanity check in ComfyUI

Once installed:

Launch ComfyUI
Load Checkpoint node
You should see:
- sd-v1-4 or sd-v1-5
Generate a simple image:
```
full body person, strange pose, surreal
```

If bodies already look “wrong,” you picked the right model 🙂

My strong recommendation for you

Use:

SD 1.5 base
LCM-LoRA
Low steps / low CFG / high denoise
Recursive feedback loop

You’ll get:

walking blobs
limb fusion
bodies becoming architecture
very “2021-2022 AI video” energy

If you want, next I can:

Give you exact ComfyUI node values for SD 1.5 + LCM
Provide a download checklist (model + LoRA + VAE)
Help you intentionally break anatomy even more (prompts + noise schedules)

Just say which one you want.

Great, this is an important step — and luckily LCM-LoRA is simple once you know where it plugs in.

Below is the exact, correct way to add and use LCM-LoRA in ComfyUI, plus the gotchas that trip people up.

What LCM-LoRA actually is (quick clarity)

LCM-LoRA is not a model replacement.
It is a LoRA add-on that modifies how the base SD 1.x model denoises, allowing:

2–4 steps instead of 20–50
Much faster generation
Slightly unstable / “alive” outputs (great for your use case)

So you need both:

an SD 1.x checkpoint AND
the LCM-LoRA loaded on top

Step 1: Get the correct LCM-LoRA file

You need an LCM-LoRA for SD 1.x (not SDXL).

Typical filename looks like:

lcm-lora-sdv1-5.safetensors

(Works for SD 1.4 and 1.5 — naming just reflects training base.)

Step 2: Put it in the correct folder

Place the file here:

ComfyUI/models/loras/

Final path example:

ComfyUI/models/loras/lcm-lora-sdv1-5.safetensors

👉 Restart ComfyUI after copying the file.

Step 3: Load it in your ComfyUI graph

You need two nodes for this:

1️⃣ Load Checkpoint

Load your base model:
- sd-v1-4 or
- sd-v1-5

2️⃣ Load LoRA

Add a Load LoRA node and set:

lora_name: lcm-lora-sdv1-5
strength_model: 1.0
strength_clip: 1.0

⚠️ Do NOT lower these initially — LCM expects full strength.

Step 4: Wire the nodes correctly (this matters)

Your graph must flow like this:

Load Checkpoint
      ↓
    Load LoRA
      ↓
  (to sampler)

Specifically:

Connect MODEL → Load LoRA
Connect CLIP → Load LoRA
Use the LoRA outputs for sampling

If you bypass this, LCM will silently not work.

Step 5: Use the RIGHT sampler settings (critical)

LCM will not work properly with standard SD settings.

Required settings:

Steps: 2–4 (start with 4)
CFG: 1.5–3.0 (start with 2.0)
Sampler:
- LCM (if available), or
- Euler a / Euler (works fine)
Scheduler: LCM or Normal

If you run:

20 steps
CFG 7+ you’re basically undoing LCM.

Step 6: Denoise strength (for Deforum-like melting)

When using img2img / recursive feedback:

Denoise: 0.6 – 0.8
- 0.6 = motion continuity
- 0.8 = heavy melting / identity loss

This is where bodies start merging and walking wrong.

Step 7: How to confirm LCM-LoRA is actually active

You should notice all of these:

✅ Images appear in 1–2 seconds
✅ 2–4 steps already look “complete”
✅ Higher steps do NOT improve quality much
✅ Motion feels more unstable / alive

If it still takes a long time or looks undercooked:

LoRA probably isn’t loaded
CFG is too high
Steps are too many

Common mistakes (avoid these)

❌ Putting the file in checkpoints/
❌ Forgetting to restart ComfyUI
❌ Using CFG 7–12
❌ Using SDXL LCM by accident
❌ Not wiring MODEL+CLIP through Load LoRA
❌ Expecting LCM to look “clean” (it won’t — that’s good)

Minimal “known-good” starter config

Use this once to confirm everything works:

Model: SD 1.5 base
LoRA: LCM-LoRA @ 1.0 / 1.0
Steps: 4
CFG: 2.0
Sampler: Euler a
Resolution: 512×512

Prompt:

full body person, strange pose, twisted limbs, surreal movement

If this works and is fast — you’re set.

Next logical step

I can now:

Give you a ready-made ComfyUI workflow JSON with:
- recursive feedback loop
- img2img melting
- realtime-friendly structure
Tune LCM specifically for walking / merging bodies
Help you add TensorRT engines on top safely

Tell me which one you want next.

You want the official LCM-LoRA for SD 1.x (not SDXL).

Here’s exactly where to get it and how to know it’s the correct one.

🔹 Where to download LCM-LoRA (SD 1.x)

1️⃣ Official Hugging Face release (recommended)

Go to Hugging Face and search for:

latent-consistency lcm-lora-sdv1-5

You’re looking for:

Repository name:

latent-consistency/lcm-lora-sdv1-5

File name should look like:

pytorch_lora_weights.safetensors

or sometimes:

lcm-lora-sdv1-5.safetensors

That’s the correct one for SD 1.4 / 1.5.

🔹 Important: Make sure it is NOT

❌ lcm-lora-sdxl
❌ Anything mentioning SDXL
❌ Anything mentioning “Lightning”
❌ Random reuploads from unknown users

You want the one from the Latent Consistency team.

🔹 Where to put it

Place the downloaded .safetensors file here:

ComfyUI/models/loras/

Example:

ComfyUI/models/loras/lcm-lora-sdv1-5.safetensors

Restart ComfyUI afterward.

🔹 Quick sanity check

In ComfyUI:

Add a Load LoRA node
Check the dropdown list
You should see:
```
lcm-lora-sdv1-5
```

If it appears, you’re good.

🔹 Why this one specifically?

This LoRA was trained to modify:

SD 1.4
SD 1.5

It enables:

2–4 step sampling
Low CFG (~2)
Fast generation
Slight instability (perfect for your melting-body videos)

🔹 After installing, use these settings

Steps: 4
CFG: 2
Sampler: Euler a
Resolution: 512x512
LoRA strength: 1.0 / 1.0

If you want, I can next:

Show you how to confirm it’s actually running in LCM mode
Give you a minimal ComfyUI graph layout
Or help you wire it into a realtime feedback loop

Just tell me which step you’re at.

If you’re using LCM-LoRA (the speed trick), don’t treat sampler/scheduler as “creative choice” first — some combos just behave better at low steps.

Here are the best picks for minimalistic B&W uncanny.

Best default (LCM-LoRA)

Sampler: euler_a
Scheduler: normal
Steps: 4 (try 3–6)
CFG: 1.6–2.4

Why: euler_a is “loose” and drifty at low steps → great for uncanny bodies and melting. It also tends to simplify shapes rather than over-polish.

If you have an explicit LCM sampler/scheduler option

(Some ComfyUI installs/nodes expose these)

Sampler: lcm
Scheduler: lcm (or normal if lcm isn’t there)
Steps: 4
CFG: 1.5–2.0

Why: it’s the most “intended” pairing for LCM, often stable at very low steps.

If you are NOT using LCM-LoRA (slower, but cleaner minimalism)

Sampler: dpmpp_2m
Scheduler: karras
Steps: 12–20
CFG: 3–6

Why: DPM++ 2M + Karras is very good at clean, graphic shapes. It’s less melty than Euler a, more “designed.”

Getting “minimalistic B&W” reliably

Sampler choice helps, but you’ll get the look by combining 3 things:

1) Prompt it into monochrome + simple forms

Add to positive:

black and white, monochrome, high contrast, minimalism, silhouette, ink wash, lino cut, xerox, posterized

Add to negative:

color, vibrant, detailed texture, skin pores, photorealistic, glossy

2) Force a B&W output in the graph (most reliable)

After VAE Decode, add an image grayscale / desaturate node (or “Color” node set saturation to 0), then optionally a threshold/posterize node for harsh minimalism.

3) Encourage uncanny anatomy without faces

Positive:

full body, wide shot, contorted pose, unnatural gait, limbs fused, people merging, body horror

Negative:

face, portrait, close up, eyes, teeth

Quick presets (copy these)

“Uncanny minimal B&W melt” (fast)

euler_a + normal
Steps 4
CFG 2.0
(If img2img feedback loop) Denoise 0.65–0.8
Grayscale + slight posterize after decode

“Graphic poster uncanny” (less melty)

dpmpp_2m + karras
Steps 16
CFG 4
Grayscale + threshold

If you tell me whether you’re running text2img only or a feedback/img2img loop, I’ll give you the exact best pairing + node settings for that case (because the “best” sampler changes a bit once you start feeding frames back in).

“Real-time” for SD1.x in a Deforum-like feedback loop boils down to how many denoising steps you can afford per second at a fixed resolution, plus a bunch of overhead (VAE decode/encode, warping, UI/display, etc.).

Here’s what it would take.

What “real-time” actually means here

True real-time: 24–60 new diffusion frames per second (rare, hard)
Perceptual real-time (common in VJ setups): 6–15 diffusion fps + frame interpolation to 30–60fps
Interactive: latency under ~150ms (also hard with full diffusion per frame)

For your “uncanny melting walkers” look, the perceptual real-time route is usually the sweet spot.

The minimum recipe to even be in the realtime conversation

1) Few-step sampling (mandatory)

Use LCM-LoRA (or another “few-step” variant).

Steps: 2–4
CFG: ~1.5–2.5
Sampler: euler_a (or lcm if available) Anything like 15–30 steps will immediately push you out of real-time.

2) Fixed resolution (mandatory for TensorRT)

Pick ONE size and stick to it:

512×512 (best starting point)
Or a fixed widescreen like 768×432 / 640×360 Changing resolution kills optimization and forces re-building TRT engines.

3) Keep the VAE cheap

VAE encode/decode is non-trivial overhead.

Prefer fp16 everywhere
Avoid fancy post chains (lots of image filters) in the same graph if you’re chasing fps

4) TensorRT (strongly recommended)

TensorRT can speed up the U-Net (the heavy part) a lot when shapes are fixed.

Export/compile engines for your chosen resolution
Keep batch size = 1 unless you’re doing multi-stream

5) Keep the loop simple

The “Deforum feel” loop is basically:

previous frame → (optional warp) → img2img → output → feedback

Avoid:

ControlNet stacks
multiple LoRAs
high-res fix
upscalers inside the live loop

Those are “offline render” luxuries.

Hardware expectations (rough, but practical)

To get interactive / near-real-time diffusion (5–15 fps at 512)

You typically need:

RTX 3080 / 3090 / 4070 Ti / 4080 class or better
12GB+ VRAM helps a lot for stability and avoiding swapping

To push toward true real-time diffusion (20+ fps at 512)

You’re generally in:

RTX 4090 territory
aggressive optimization (TensorRT, minimal graph, no extras)

If you’re on laptop GPUs / 8GB VRAM

You can still do it, but you’ll likely need:

512→384 resolution
2–3 steps
and rely more on interpolation to look smooth

(Exact fps varies wildly by GPU, driver, TRT engine quality, and graph complexity—so think of these as “what class of hardware,” not a promise.)

The standard “looks realtime” trick

Generate at 8–12 diffusion fps, then:

RIFE interpolation to 30–60 fps for display

This keeps the weird morphing (which you want) while making motion feel fluid.

What will stop you from realtime, even with a fast GPU

Too many steps (biggest killer)
High resolution (768+ quickly gets expensive)
ControlNet / pose / depth in the live loop
Switching resolutions dynamically
Running in a heavy UI capture + preview pipeline (display can become the bottleneck)

A concrete “realtime target” config (good baseline)

For minimalistic B&W uncanny:

Model: SD1.5 base (or 1.4)
LCM-LoRA strength: 1.0 / 1.0
Resolution: 512×512
Steps: 4
CFG: 2.0
Sampler/Scheduler: euler_a + normal
Img2img denoise (feedback strength): 0.65–0.8
Post: grayscale + (optional) mild threshold/posterize

Then add RIFE for smooth display.

The honest bottom line

To run “this model” in realtime you need few-step sampling + fixed resolution + GPU acceleration (ideally TensorRT), and you’ll usually present it as diffusion frames + interpolation rather than trying to brute-force 30–60 fully-diffused frames per second.

If you tell me your GPU model + VRAM, I’ll translate that into a realistic preset (resolution / steps / TRT worth it or not / whether interpolation is necessary) without any guesswork.

You’re basically trying to do a 3-stage compile:

.safetensors checkpoint → PyTorch modules (UNet, VAE, CLIP)
PyTorch → .onnx export (usually only the UNet, optionally VAE+CLIP)
ONNX → TensorRT engine (.engine / .plan)

There are two sane ways to do it:

Recommended (ComfyUI-native): let ComfyUI_TensorRT do the ONNX export + TensorRT build for you. citeturn0search0turn0search8
Manual (more control): export via Hugging Face Optimum/Diffusers then run trtexec. citeturn0search1turn0search3turn0search10

Below are both, with the exact “where the .safetensors goes” answer.

Option A (recommended): ComfyUI_TensorRT does safetensors → ONNX → TensorRT for you

1) Put your `.safetensors` checkpoint where ComfyUI expects it

ComfyUI/models/checkpoints/your_sd15_or_sd14.safetensors

Note: ComfyUI_TensorRT officially lists SD 1.5 support (and more). In practice most people use SD 1.5 base for compatibility. citeturn0search0

2) Install ComfyUI_TensorRT

Follow its README requirements and install steps. citeturn0search0

3) Use the provided “Build Engine” workflows

ComfyUI_TensorRT docs list example workflows like:

Build TensorRT Engine SD 1.5 Static
Build TensorRT Engine SD 1.5 Dynamic citeturn0search8

In ComfyUI:

Load the “Build Engine” workflow
Select your checkpoint
Choose Static if you want max speed (one resolution)
Choose Dynamic if you need a range (slower + more VRAM)

This path automatically:

loads the checkpoint
exports the needed parts to ONNX
builds the TRT engine(s)

That’s the fastest route and avoids a ton of shape/IO mistakes.

Key rule: pick and stick to one resolution (e.g., 512×512) for static engines.

Option B (manual): `.safetensors` → ONNX → TensorRT with CLI tools

Stage 1: `.safetensors` → ONNX

A .safetensors SD checkpoint is not directly “convertible” to ONNX; you must first load it into a pipeline and export.

Method B1: Optimum (Diffusers ONNX export CLI)
Hugging Face Optimum supports exporting Diffusers models to ONNX via optimum-cli. citeturn0search1
This is easiest when your model is already in Diffusers format (a folder with unet/, vae/, etc.). If you only have a single .safetensors, you’ll typically convert it to Diffusers format first (tools exist, but it’s fiddly).

Method B2: NVIDIA/Diffusers TensorRT scripts (often the smoothest “manual” route)
NVIDIA and Diffusers provide scripts that do: export ONNX + build TRT engines in one flow for SD pipelines. citeturn0search3turn0search5

Stage 2: ONNX → TensorRT (`trtexec`)

Once you have an ONNX (commonly the UNet), you compile it:

trtexec --onnx=unet.onnx --saveEngine=unet_fp16.engine --fp16 --workspace=8192

trtexec is the standard TensorRT builder tool. citeturn0search10

For diffusion UNet, shapes matter (latent H/W, token length, batch). That’s why the ComfyUI/A1111 builders ask for min/opt/max shapes, and why “dynamic engines” cost VRAM/perf. citeturn0search15

What you should do for your setup (fastest + least pain)

If your goal is ComfyUI realtime-ish SD1.x, do this:

Use SD 1.5 base .safetensors in models/checkpoints/
Install ComfyUI_TensorRT
Run the Build TensorRT Engine SD 1.5 Static workflow at 512×512
Then swap your graph to the TRT loader/model nodes

This is exactly what the ComfyUI_TensorRT project and its node docs are designed for. citeturn0search0turn0search8

Two “gotchas” that will save you hours

TensorRT engines are GPU + driver + TensorRT-version sensitive. Moving to another machine often means rebuilding. (Common in TRT workflows; NVIDIA forums discuss rebuild needs.) citeturn0search18
Static engines are fastest. Dynamic engines support ranges (like 512–768) but cost performance and VRAM. citeturn0search15

If you tell me Windows or Linux, your GPU model, and what resolution you’re targeting (likely 512×512), I can give you the exact ComfyUI_TensorRT “engine build” settings to choose (static vs dynamic, fp16 vs int8) and where the engine file will be written on disk.

Good — this is the right question.
TensorRT sounds like magic until you understand what it actually does.

What TensorRT Actually Is

TensorRT is NVIDIA’s inference compiler.

Not a model.
Not a training framework.
Not a different neural network.

It is a graph optimizer + kernel compiler that:

Takes a frozen neural network (ONNX)
Rewrites the computation graph
Fuses operations
Chooses hardware-specific GPU kernels
Optionally lowers precision (FP16 / INT8)
Builds a highly optimized engine for your exact GPU

The output is a compiled binary .engine file that runs directly on your GPU with minimal overhead.

Why It’s Faster

1️⃣ Kernel Fusion

Normal PyTorch execution:

Conv → Add → Activation → Norm → ...

Each is a separate GPU kernel launch.

TensorRT fuses many of these into a single fused kernel, meaning:

fewer memory writes
fewer GPU launches
better cache usage

Memory bandwidth is a huge bottleneck in diffusion — fusing reduces that.

2️⃣ Static Shape Optimization

PyTorch is dynamic:

accepts many shapes
keeps flexibility
adds overhead

TensorRT is static:

you say “512×512”
it optimizes exactly for that shape
no runtime shape logic
no graph rebuilding

Static assumptions = speed.

3️⃣ Lower Precision (FP16 / INT8)

Most SD models don’t need full FP32 precision.

TensorRT can:

run in FP16 (huge speedup)
even INT8 with calibration

Lower precision:

doubles throughput
reduces memory pressure
increases tensor core utilization

Diffusion UNets benefit massively from this.

4️⃣ Better GPU Scheduling

TensorRT understands:

tensor core usage
memory layout
layer reordering
kernel selection

It picks kernels specifically tuned for:

your GPU architecture (Ampere, Ada, etc.)
your CUDA version
your exact shape

PyTorch runs more generic kernels.

Why Isn’t It Default Everywhere?

This is the important part.

Because TensorRT trades flexibility for speed.

Here’s what that means:

❌ 1. Static Resolution Requirement

If you build for 512×512:

It runs blazing fast at 512.

Change to 640×640?

It breaks.
You must rebuild.

Most users constantly change:

resolution
batch size
controlnets
LoRAs
models

TensorRT hates that.

❌ 2. GPU + Driver Lock-In

TensorRT engines are:

GPU architecture specific
Driver version sensitive
CUDA version sensitive
TensorRT version sensitive

Move the engine to another machine?

Often rebuild required.

That’s terrible for distributing general-purpose apps.

❌ 3. Build Time & Complexity

Building engines:

can take minutes
requires CUDA + TensorRT installed
sometimes fails due to shape mismatches
requires understanding dynamic ranges

Most users don’t want that.

❌ 4. Reduced Feature Compatibility

Certain things complicate TensorRT:

dynamic ControlNet stacks
changing LoRA combinations
unusual custom nodes
some schedulers

Standard PyTorch:

everything just works.

TensorRT:

everything must be predictable.

❌ 5. Precision Tradeoffs

FP16 is usually fine.

INT8 can:

slightly degrade quality
introduce subtle artifacts
require calibration datasets

Many tool creators don’t want “why does my image look slightly different” bug reports.

So Why Use It?

Because in your case:

You want:

fixed resolution
fixed model
fixed loop
realtime-ish feedback

That is exactly the environment TensorRT thrives in.

You’re building a performance pipeline, not a model experimentation sandbox.

Real Numbers (rough intuition)

On a 4090:

Mode	512x512 SD1.5	4 steps
PyTorch FP16	~8–12 fps
TensorRT FP16	~15–25 fps

(Not guarantees — just illustrating magnitude.)

The more steps you run, the more TRT helps.

Why Diffusion Benefits So Much

Diffusion = UNet called repeatedly.

Even at 4 steps:

4 full UNet passes
each UNet is massive
many convolutions

TensorRT optimizes the UNet heavily.

So the speedup compounds.

Big Picture

TensorRT is:

A compiler that sacrifices flexibility to squeeze every ounce of performance from a specific model at a specific resolution on a specific GPU.

That’s why it’s not default.

Most users:

constantly experiment
change models
tweak settings
mix LoRAs
use different resolutions

They need flexibility more than speed.

You are trying to build a live system.

Different priorities.

Important Reality Check

TensorRT alone does not make SD “real-time”.

It makes it much faster.

To hit real-time you still need:

LCM / few-step sampling
small fixed resolution
minimal graph
maybe interpolation

TRT is an amplifier, not magic.

If you tell me your GPU, I can give you a realistic answer like:

“Yes, with LCM + TRT at 512 you can hit ~X fps”

“You’ll need interpolation to look smooth”

And we can design accordingly.

Nice — once you have the .engine, using it in ComfyUI is mostly about swapping the UNet “MODEL” path to go through a TensorRT Loader node, while still keeping CLIP + VAE from your normal checkpoint.

0) The one “why isn’t it showing up?” fix

If you built the engine during the same ComfyUI session, it often won’t appear in the TensorRT Loader dropdown until you refresh the UI (browser refresh / F5) or restart ComfyUI. citeturn0search0

1) Where the engine is expected to live

Depending on how you built it, engines are typically stored either:

in a TensorRT models folder under ComfyUI/models/ (common convention), or
sometimes in ComfyUI’s output folder (people have hit this when a tensorrt folder wasn’t set up). citeturn0search7

You don’t have to manually move it if your Loader can see it — but if the dropdown is empty, that’s the first thing to check.

2) The “correct” way to load it in a workflow

Add the TensorRT Loader node

In ComfyUI, add:

TensorRT Loader (from ComfyUI_TensorRT)

Then:

Select your engine from the unet_name dropdown (or similarly named field). citeturn0search0turn0search11

This node outputs a MODEL that represents the TensorRT-compiled UNet.

3) How to wire it (this is the key)

You generally keep using your normal checkpoint loader for CLIP + VAE, but replace the UNet model that goes into sampling.

Typical wiring:

Load Checkpoint
- Use it for:
  - CLIP → text encoding nodes
  - VAE → decode/encode nodes
TensorRT Loader
- Use it for:
  - MODEL → your sampler (KSampler / KSampler Advanced)

So the sampler gets:

MODEL: from TensorRT Loader
Positive/Negative conditioning: from CLIP/text encode (normal)
Latents: normal path
VAE decode: normal path

This matches the official guidance: connect the TensorRT loader’s MODEL to the sampler, while using CLIP/VAE from the original checkpoint. citeturn0search11turn0search5

4) Common gotchas (so it actually runs)

Engine not listed

Refresh browser (F5) / restart ComfyUI. citeturn0search0
Confirm the engine is in the folder ComfyUI_TensorRT is scanning (or wherever your build workflow saved it).

Resolution mismatch

If you built a static engine for (say) 512×512, then you must run:

width/height that match what you compiled (or a compatible latent shape range if it’s a dynamic engine).

LoRA / ControlNet compatibility

ComfyUI_TensorRT has had limitations around LoRAs/ControlNets depending on node path/version; the README explicitly notes compatibility limitations (and that improvements are planned). citeturn0search0turn0search3
(There are community workarounds in some setups, but start simple: base model + TRT UNet first.)

5) Quick “does it work?” test

Use a super simple text2img graph:

fixed resolution (the one you built)
no ControlNet
no extra LoRAs
standard KSampler

If it runs fast and doesn’t error, your loader + wiring is correct.

If you tell me ONE thing, I can be precise

Which engine did you build:

Static 512×512 (most likely), or Dynamic?

And what’s your ComfyUI graph using for sampling:

KSampler or KSampler Advanced?

If you answer those, I’ll describe the exact node names/ports to connect for your specific graph layout (and the 2–3 settings that must match your engine).

Perfect — this is a very common issue and it’s not your fault.

What’s happening:

The TensorRT Loader node does NOT scan output/tensorrt
It scans a specific models directory.

When you build via the workflow, it often defaults to saving in:

ComfyUI/output/tensorrt/

But the loader usually looks in:

ComfyUI/models/tensorrt/

That’s why the dropdown is empty.

✅ Fix (2 minutes)

1️⃣ Create the correct folder (if it doesn’t exist)

Inside your ComfyUI root:

ComfyUI/models/tensorrt/

If it doesn’t exist, create it.

2️⃣ Move the engine file

Move your .engine file from:

ComfyUI/output/tensorrt/

to:

ComfyUI/models/tensorrt/

Example:

ComfyUI/models/tensorrt/sd15_512_static.engine

3️⃣ Restart ComfyUI

Stop the server completely:

Ctrl+C

Then:

python main.py

Now open the UI again.

4️⃣ Refresh browser (important)

Hit:

F5

The dropdown should now populate.

🔍 If it’s STILL empty

Check:

Is the file extension actually .engine?
Did it get renamed to .engine.engine accidentally?
Are you running the same ComfyUI install where you moved the file?

You can also confirm by printing the directory:

ls ComfyUI/models/tensorrt/

You should see your engine file there.

⚠️ One more important thing

Since you built a static engine, make sure:

Resolution matches EXACTLY what you built (likely 512×512)
Batch size = 1
No dynamic resizing

Otherwise it will load but fail at runtime.

If you want, tell me:

The engine filename
The resolution you compiled for

And I’ll tell you exactly what sampler/latent size settings must match it.

i found it but when i run the workflow:

got prompt model weight dtype torch.float16, manual cast: None model_type EPS Using pytorch attention in VAE Using pytorch attention in VAE VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16 CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16 Requested to load SD1ClipModel loaded completely; 9650.80 MB usable, 235.84 MB loaded, full load: True model_type EPS Requested to load BaseModel loaded completely; 3210.11 MB usable, 0.00 MB loaded, full load: True 0%| | 0/10 [00:00<?, ?it/s] !!! Exception during processing !!! cannot access local variable ‘curr_split_batch’ where it is not associated with a value Traceback (most recent call last): File “C:\Users\GeorgeAdamon\AppData\Local\Programs\ComfyUI\resources\ComfyUI\execution.py”, line 530, in execute output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\GeorgeAdamon\AppData\Local\Programs\ComfyUI\resources\ComfyUI\execution.py”, line 334, in get_output_data return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\GeorgeAdamon\AppData\Local\Programs\ComfyUI\resources\ComfyUI\execution.py”, line 308, in _async_map_node_over_list await process_inputs(input_dict, i) File “C:\Users\GeorgeAdamon\AppData\Local\Programs\ComfyUI\resources\ComfyUI\execution.py”, line 296, in process_inputs result = f(**inputs) ^^^^^^^^^^^ File “C:\Users\GeorgeAdamon\AppData\Local\Programs\ComfyUI\resources\ComfyUI\nodes.py”, line 1590, in sample return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\GeorgeAdamon\AppData\Local\Programs\ComfyUI\resources\ComfyUI\nodes.py”, line 1555, in common_ksampler samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\GeorgeAdamon\AppData\Local\Programs\ComfyUI\resources\ComfyUI\comfy\sample.py”, line 66, in sample samples = sampler.sample(noise, positive, negative, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\GeorgeAdamon\AppData\Local\Programs\ComfyUI\resources\ComfyUI\comfy\samplers.py”, line 1177, in sample return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\GeorgeAdamon\AppData\Local\Programs\ComfyUI\resources\ComfyUI\comfy\samplers.py”, line 1067, in sample return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\GeorgeAdamon\AppData\Local\Programs\ComfyUI\resources\ComfyUI\comfy\samplers.py”, line 1049, in sample output = executor.execute(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed, latent_shapes=latent_shapes) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\GeorgeAdamon\AppData\Local\Programs\ComfyUI\resources\ComfyUI\comfy\patcher_extension.py”, line 112, in execute return self.original(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\GeorgeAdamon\AppData\Local\Programs\ComfyUI\resources\ComfyUI\comfy\samplers.py”, line 993, in outer_sample output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed, latent_shapes=latent_shapes) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\GeorgeAdamon\AppData\Local\Programs\ComfyUI\resources\ComfyUI\comfy\samplers.py”, line 979, in inner_sample samples = executor.execute(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\GeorgeAdamon\AppData\Local\Programs\ComfyUI\resources\ComfyUI\comfy\patcher_extension.py”, line 112, in execute return self.original(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\GeorgeAdamon\AppData\Local\Programs\ComfyUI\resources\ComfyUI\comfy\samplers.py”, line 751, in sample samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\GeorgeAdamon\Documents\ComfyUI.venv\Lib\site-packages\torch\utils_contextlib.py”, line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\GeorgeAdamon\AppData\Local\Programs\ComfyUI\resources\ComfyUI\comfy\k_diffusion\sampling.py”, line 1496, in sample_gradient_estimation denoised = model(x, sigmas[i] * s_in, **extra_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\GeorgeAdamon\AppData\Local\Programs\ComfyUI\resources\ComfyUI\comfy\samplers.py”, line 400, in call out = self.inner_model(x, sigma, model_options=model_options, seed=seed) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\GeorgeAdamon\AppData\Local\Programs\ComfyUI\resources\ComfyUI\comfy\samplers.py”, line 952, in call return self.outer_predict_noise(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\GeorgeAdamon\AppData\Local\Programs\ComfyUI\resources\ComfyUI\comfy\samplers.py”, line 959, in outer_predict_noise ).execute(x, timestep, model_options, seed) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\GeorgeAdamon\AppData\Local\Programs\ComfyUI\resources\ComfyUI\comfy\patcher_extension.py”, line 112, in execute return self.original(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\GeorgeAdamon\AppData\Local\Programs\ComfyUI\resources\ComfyUI\comfy\samplers.py”, line 962, in predict_noise return sampling_function(self.inner_model, x, timestep, self.conds.get(“negative”, None), self.conds.get(“positive”, None), self.cfg, model_options=model_options, seed=seed) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\GeorgeAdamon\AppData\Local\Programs\ComfyUI\resources\ComfyUI\comfy\samplers.py”, line 380, in sampling_function out = calc_cond_batch(model, conds, x, timestep, model_options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\GeorgeAdamon\AppData\Local\Programs\ComfyUI\resources\ComfyUI\comfy\samplers.py”, line 205, in calc_cond_batch return _calc_cond_batch_outer(model, conds, x_in, timestep, model_options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\GeorgeAdamon\AppData\Local\Programs\ComfyUI\resources\ComfyUI\comfy\samplers.py”, line 213, in _calc_cond_batch_outer return executor.execute(model, conds, x_in, timestep, model_options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\GeorgeAdamon\AppData\Local\Programs\ComfyUI\resources\ComfyUI\comfy\patcher_extension.py”, line 112, in execute return self.original(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\GeorgeAdamon\AppData\Local\Programs\ComfyUI\resources\ComfyUI\comfy\samplers.py”, line 325, in calc_cond_batch output = model.apply_model(input_x, timestep, **c).chunk(batch_chunks) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\GeorgeAdamon\AppData\Local\Programs\ComfyUI\resources\ComfyUI\comfy\model_base.py”, line 167, in apply_model return comfy.patcher_extension.WrapperExecutor.new_class_executor( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\GeorgeAdamon\AppData\Local\Programs\ComfyUI\resources\ComfyUI\comfy\patcher_extension.py”, line 112, in execute return self.original(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\GeorgeAdamon\AppData\Local\Programs\ComfyUI\resources\ComfyUI\comfy\model_base.py”, line 209, in _apply_model model_output = self.diffusion_model(xc, t, context=context, control=control, transformer_options=transformer_options, **extra_conds) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “C:\Users\GeorgeAdamon\Documents\ComfyUI\custom_nodes\comfyui_tensorrt\tensorrt_loader.py”, line 73, in call self.set_bindings_shape(model_inputs, curr_split_batch) ^^^^^^^^^^^^^^^^ UnboundLocalError: cannot access local variable ‘curr_split_batch’ where it is not associated with a value

That traceback is a known ComfyUI_TensorRT failure mode: the TensorRT loader hits a code path where it expects to be splitting the batch for conditioning, but the variable it uses (curr_split_batch) never gets set — so it crashes with UnboundLocalError. This has been reported against ComfyUI_TensorRT’s loader. citeturn0search0

In practice it usually happens when your sampler path / batching path isn’t one the TensorRT loader supports (or when other patches like LoRA/ControlNet/advanced sampling change how batches are formed).

Here’s how to get you unstuck fast.

1) Use a “supported” sampler path (avoid SDE / gradient-estimation / exotic samplers)

Your stack shows it’s going through sample_gradient_estimation (and similar reports go through sample_dpmpp_sde*). Those are exactly the kinds of paths that tend to break TRT loader batching. citeturn1view0

Fix: in your KSampler, set:

sampler: euler or euler_a
scheduler: normal (or karras if you must, but start with normal)
batch size: 1
disable anything that does “gradient estimation”, “SDE”, “GPU SDE”, etc.

This alone often stops the crash.

2) Remove LoRAs and ControlNets when using the TRT `.engine` (important)

ComfyUI_TensorRT’s own README lists LoRAs and ControlNets as not compatible with TensorRT engines (yet). citeturn0search1

So if your workflow has LCM-LoRA applied, that’s a problem for two reasons:

The TRT engine was built from the base UNet weights
A LoRA modifies weights at runtime — the TensorRT engine can’t “patch” itself that way

Fix: temporarily run base SD only with the TRT engine (no LoRA nodes) to confirm TRT works.

If you want LCM + TRT

You have two workable approaches:

A) Merge (“bake”) the LCM-LoRA into a checkpoint, then build the TRT engine from that merged checkpoint.
(That produces a TRT engine whose weights already include LCM.)

B) Don’t use the static TRT loader; instead use a TRT conversion workflow/node that converts after LoRA application.
(Some community setups do this, but the official loader path still calls out LoRA limits. citeturn0search1turn0search9)

If you’re going for reliability right now: do A.

3) Update ComfyUI + ComfyUI_TensorRT (you might be on a buggy revision)

Even if the core limitation remains, updating often reduces “random” loader crashes.

Update ComfyUI_TensorRT (Windows)

From your ComfyUI folder:

cd custom_nodes\ComfyUI_TensorRT
git pull

Update ComfyUI itself (where you cloned it):

cd ..\..
git pull

Restart ComfyUI.

(If your ComfyUI is the “portable” bundle and not a git checkout, update via whatever updater that bundle provides.)

4) Quick “known-good” TRT test workflow settings

To validate your .engine works:

Resolution: exactly what you built (ex: 512×512 if static)
Sampler: euler_a
Scheduler: normal
Steps: 10 (just for the test)
CFG: 5
No LoRAs
No ControlNet
No fancy sampler nodes

If that runs, then TRT is fine and your crash was workflow/sampler/patch related.

If you tell me these 2 things, I’ll give you the exact next move

Which sampler + scheduler were you using when it crashed? (screenshot or just names)
Is LCM-LoRA currently in the graph when you use the TRT model?

Based on that I’ll either:

point you to the exact sampler combo that works with TRT in your setup, or
walk you through merging LCM-LoRA into a checkpoint and rebuilding the engine (the “correct” way to get realtime-ish LCM + TRT).

Good — the screenshot helps a lot.

I can already see the problem.

In your TensorRT Loader node:

model_type: sdxl_base   ❌

But your checkpoint is:

v1-5-pruned  (SD 1.5)   ✅

You built an SD1.5 static engine, but the loader is set to SDXL mode.

That mismatch will absolutely cause internal shape/batching errors like the curr_split_batch crash you’re seeing.

✅ Fix This First

In the TensorRT Loader node, change:

model_type: sdxl_base

to:

sd15

(or whatever the SD1.x option is in your dropdown — sometimes labeled sd15, sd1, or sd_base)

Then:

Save workflow
Restart ComfyUI completely
Run again

🔴 Second Issue I See

Your KSampler shows:

scheduler: simple
steps: 10
cfg: 10

For a TensorRT static engine, especially SD1.x:

Change this to:

sampler: euler
scheduler: normal
steps: 8–12
cfg: 5–7

Avoid:

simple
karras
SDE
any gradient-estimation variants

Start simple.

🔵 Also Important

Your graph is correct structurally:

✔ CLIP from checkpoint
✔ VAE from checkpoint
✔ MODEL from TensorRT loader → KSampler

That part is good.

Why the Error Happened

When you selected sdxl_base, the loader:

expected SDXL UNet shapes
expected SDXL conditioning format
expected SDXL context size

But you gave it an SD1.5 engine.

So internally, batching logic didn’t match expected tensor structure → crash.

After You Fix model_type

If it still crashes:

Tell me:

Exact engine filename
Resolution you compiled (512x512?)
Whether you built STATIC or DYNAMIC

But 90% chance this is just the sdxl_base mismatch.

Change that first and report back.

Got it. If changing model_type didn’t fix it, then this error is almost certainly coming from the TensorRT custom node code you’re using, not from your graph wiring.

Two big clues in your traceback:

It’s calling ...custom_nodes\comfyui_tensorrt\tensorrt_loader.py
The crash is curr_split_batch uninitialized inside that loader

That’s a bug/compatibility issue in that loader implementation (often happens when ComfyUI core changes its batching/conditioning behavior and the TRT node hasn’t caught up).

Do this in order (fastest path to “it works”)

1) Make sure you’re using the official ComfyUI_TensorRT node (and only one TRT node)

Right now your folder name is comfyui_tensorrt (lowercase). That’s often an older fork or an older install.

Fix:

Close ComfyUI.
Go to:
```
ComfyUI/custom_nodes/
```
Delete or rename the folder:
```
comfyui_tensorrt
```
Install the official node into a fresh folder:
```
ComfyUI_TensorRT
```
(If you already have it too, keep only one TensorRT node pack installed.)

Then restart ComfyUI and try again.

If you keep two TRT node packs installed, ComfyUI can load the wrong one and you’ll keep seeing this exact error.

2) Update ComfyUI + the TRT node to matching recent commits

This bug shows up a lot when:

ComfyUI is newer
TRT loader is older (or vice versa)

On Windows (PowerShell/CMD), from your ComfyUI folder:

cd custom_nodes\ComfyUI_TensorRT
git pull
cd ..\..
git pull

Restart ComfyUI.

3) Run a “no surprises” test graph (to rule out batch splitting paths)

Make sure these are true:

Empty Latent Image (or whatever creates latents): batch_size = 1
KSampler
- sampler: euler or euler_a
- scheduler: normal
- steps: 10
- cfg: 5 (not 10)
No LoRAs, no ControlNet, no special samplers

Why: your earlier log shows it went into sample_gradient_estimation which typically happens with certain sampler/scheduler combos. TRT loaders frequently don’t support those code paths well.

4) If you want LCM + TensorRT: you must bake it

Important reality: a static TensorRT engine can’t “apply” LoRAs at runtime the way PyTorch does.

So if you intend to use LCM-LoRA and TRT:

merge/bake the LoRA into the checkpoint
then build the TRT engine from that merged checkpoint

Otherwise you’ll get either crashes or “it runs but ignores LoRA” depending on node behavior.

Quick yes/no that tells me exactly what’s wrong

In your ComfyUI console, at startup, do you see it loading two TensorRT custom node folders? (anything mentioning both comfyui_tensorrt and ComfyUI_TensorRT)

If yes → delete the older one and you’re basically done.

If no → tell me the exact folder name under custom_nodes that contains tensorrt_loader.py, and I’ll tell you whether it’s the official node or a fork and what to swap.

Below is a working “skeleton project” (uv-based) that:

loads your SD 1.x checkpoint (.safetensors) for CLIP + VAE (PyTorch)
loads your TensorRT .engine for the UNet (fast part)
runs generation in a tight loop
displays frames live (OpenCV), no saving

This is the same architecture ComfyUI uses conceptually, but with way less overhead and no graph UI.

Reality check: TensorRT engines aren’t standardized like “one file fits all.” Binding names and dtypes can differ based on how you exported/built. The code below auto-inspects bindings and tries to match the common SD1.x UNet signatures. If your engine uses different binding names, you’ll only need to tweak a small mapping section.

1) Create the uv project

mkdir trt_sd_live
cd trt_sd_live
uv init --python 3.10

Add dependencies (you may already have TensorRT installed via NVIDIA; uv/pip may not install it cleanly on Windows):

uv add numpy opencv-python diffusers transformers safetensors accelerate

Install PyTorch separately (pick the right CUDA build from pytorch.org). Example (CUDA 12.1):

uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121

For TensorRT + CUDA Python:

uv add cuda-python

If import tensorrt as trt fails, you need the NVIDIA TensorRT Python package installed the way you installed it for ComfyUI (often via NVIDIA installers / pip install tensorrt in that environment). Use the same Python/venv for this project.

2) Project layout

Create this:

trt_sd_live/
  main.py
  assets/
    sd15.safetensors              # your checkpoint
    unet_fp16.engine              # your TRT engine

3) main.py (live loop generator)

This assumes SD1.5/SD1.4 style UNet engine with inputs like:

latent sample: [1,4,64,64] for 512x512

timestep: scalar or [1]

encoder_hidden_states: [1,77,768]

It also assumes your engine was built static at 512x512 (latent 64x64).

import os
import time
import numpy as np
import cv2

import torch
from diffusers import StableDiffusionPipeline, EulerDiscreteScheduler

# TensorRT + CUDA
import tensorrt as trt
from cuda import cudart

# ----------------------------
# Utils: CUDA buffer helpers
# ----------------------------
def cuda_check(status):
    if status[0] != cudart.cudaError_t.cudaSuccess:
        raise RuntimeError(f"CUDA error: {status[0]} ({status[1] if len(status)>1 else ''})")
    return status

def malloc_device(nbytes: int):
    status, ptr = cudart.cudaMalloc(nbytes)
    cuda_check((status,))
    return ptr

def memcpy_h2d(dst_ptr, src_np: np.ndarray):
    nbytes = src_np.nbytes
    status = cudart.cudaMemcpy(dst_ptr, src_np.ctypes.data, nbytes, cudart.cudaMemcpyKind.cudaMemcpyHostToDevice)
    cuda_check((status,))

def memcpy_d2h(dst_np: np.ndarray, src_ptr):
    nbytes = dst_np.nbytes
    status = cudart.cudaMemcpy(dst_np.ctypes.data, src_ptr, nbytes, cudart.cudaMemcpyKind.cudaMemcpyDeviceToHost)
    cuda_check((status,))

def make_stream():
    status, stream = cudart.cudaStreamCreate()
    cuda_check((status,))
    return stream

def stream_sync(stream):
    status = cudart.cudaStreamSynchronize(stream)
    cuda_check((status,))

# ----------------------------
# TensorRT UNet Runner
# ----------------------------
class TRTUNet:
    def __init__(self, engine_path: str, verbose: bool = True):
        self.logger = trt.Logger(trt.Logger.WARNING)
        self.runtime = trt.Runtime(self.logger)

        with open(engine_path, "rb") as f:
            engine_bytes = f.read()

        self.engine = self.runtime.deserialize_cuda_engine(engine_bytes)
        if self.engine is None:
            raise RuntimeError("Failed to deserialize TensorRT engine.")

        self.context = self.engine.create_execution_context()
        if self.context is None:
            raise RuntimeError("Failed to create TensorRT execution context.")

        self.stream = make_stream()

        # Inspect bindings
        self.bindings = []
        self.binding_names = []
        self.binding_is_input = []
        self.binding_dtypes = []
        self.binding_shapes = []

        for i in range(self.engine.num_io_tensors):
            name = self.engine.get_tensor_name(i)
            self.binding_names.append(name)
            self.binding_is_input.append(self.engine.get_tensor_mode(name) == trt.TensorIOMode.INPUT)
            self.binding_dtypes.append(self.engine.get_tensor_dtype(name))
            self.binding_shapes.append(tuple(self.engine.get_tensor_shape(name)))

        if verbose:
            print("=== TensorRT Engine IO ===")
            for n, is_in, dt, sh in zip(self.binding_names, self.binding_is_input, self.binding_dtypes, self.binding_shapes):
                print(("IN " if is_in else "OUT"), n, dt, sh)

        # Heuristic mapping (adjust here if needed)
        # Common names seen: "sample", "latent", "x", "timestep", "timesteps", "encoder_hidden_states", "context", "out_sample"
        self.name_sample = self._find_name(["sample", "latent", "x"])
        self.name_timestep = self._find_name(["timestep", "timesteps", "t"])
        self.name_cond = self._find_name(["encoder_hidden_states", "context", "cond", "text_embeddings"])
        self.name_out = self._find_output_name(["out", "noise_pred", "output", "out_sample"])

        if not (self.name_sample and self.name_timestep and self.name_cond and self.name_out):
            raise RuntimeError(
                "Could not auto-map engine bindings. "
                "Edit the mapping section in TRTUNet.__init__ to match your engine IO names."
            )

        # Allocate device buffers for static shapes
        self.dev_ptrs = {}
        self.host_out = None

        self._allocate_static_buffers()

    def _find_name(self, candidates):
        for c in candidates:
            for n in self.binding_names:
                if c.lower() in n.lower() and self.engine.get_tensor_mode(n) == trt.TensorIOMode.INPUT:
                    return n
        return None

    def _find_output_name(self, candidates):
        for c in candidates:
            for n in self.binding_names:
                if c.lower() in n.lower() and self.engine.get_tensor_mode(n) == trt.TensorIOMode.OUTPUT:
                    return n
        # fallback: first output
        for n in self.binding_names:
            if self.engine.get_tensor_mode(n) == trt.TensorIOMode.OUTPUT:
                return n
        return None

    def _np_dtype(self, trt_dtype):
        if trt_dtype == trt.float16:
            return np.float16
        if trt_dtype == trt.float32:
            return np.float32
        if trt_dtype == trt.int32:
            return np.int32
        if trt_dtype == trt.int64:
            return np.int64
        raise RuntimeError(f"Unhandled TRT dtype: {trt_dtype}")

    def _allocate_static_buffers(self):
        # Set shapes (static engine should already be concrete)
        # Still set them on context to be safe:
        for n in self.binding_names:
            sh = tuple(self.engine.get_tensor_shape(n))
            self.context.set_input_shape(n, sh) if self.engine.get_tensor_mode(n) == trt.TensorIOMode.INPUT else None

        # Allocate device buffers
        for n in self.binding_names:
            sh = tuple(self.context.get_tensor_shape(n))
            dt = self.engine.get_tensor_dtype(n)
            np_dt = self._np_dtype(dt)
            nbytes = int(np.prod(sh)) * np.dtype(np_dt).itemsize
            self.dev_ptrs[n] = malloc_device(nbytes)

        # Host output buffer
        out_shape = tuple(self.context.get_tensor_shape(self.name_out))
        out_dt = self._np_dtype(self.engine.get_tensor_dtype(self.name_out))
        self.host_out = np.empty(out_shape, dtype=out_dt)

        # Register tensor addresses
        for n in self.binding_names:
            self.context.set_tensor_address(n, int(self.dev_ptrs[n]))

    def __call__(self, sample: np.ndarray, timestep: np.ndarray, encoder_hidden_states: np.ndarray) -> np.ndarray:
        """
        sample: (1,4,64,64) float16
        timestep: (1,) int64 or int32 or float32 (depends on engine)
        encoder_hidden_states: (1,77,768) float16
        returns: noise_pred same shape as sample
        """
        # Copy inputs
        memcpy_h2d(self.dev_ptrs[self.name_sample], sample)
        memcpy_h2d(self.dev_ptrs[self.name_timestep], timestep)
        memcpy_h2d(self.dev_ptrs[self.name_cond], encoder_hidden_states)

        # Execute
        ok = self.context.execute_async_v3(self.stream)
        if not ok:
            raise RuntimeError("TensorRT execute_async_v3 returned False.")

        # Copy output
        memcpy_d2h(self.host_out, self.dev_ptrs[self.name_out])
        stream_sync(self.stream)
        return self.host_out

# ----------------------------
# Main generation loop
# ----------------------------
def main():
    # Paths
    ckpt_path = os.path.join("assets", "sd15.safetensors")
    engine_path = os.path.join("assets", "unet_fp16.engine")

    # Load SD pipeline (we only use tokenizer, text encoder, VAE, scheduler config)
    pipe = StableDiffusionPipeline.from_single_file(
        ckpt_path,
        torch_dtype=torch.float16,
        safety_checker=None,
        requires_safety_checker=False,
    ).to("cuda")

    # Use a simple Euler scheduler (fast + “uncanny” at low steps)
    pipe.scheduler = EulerDiscreteScheduler.from_config(pipe.scheduler.config)

    # Load TensorRT UNet
    unet_trt = TRTUNet(engine_path, verbose=True)

    # Prompt
    prompt = "minimalist black and white, high contrast, full body, contorted pose, unnatural gait, people melting together, silhouette"
    negative = "face, portrait, close up, color, text, watermark, logo"

    # Encode prompt once (for speed)
    with torch.no_grad():
        text_in = pipe.tokenizer(
            [prompt],
            padding="max_length",
            max_length=pipe.tokenizer.model_max_length,
            truncation=True,
            return_tensors="pt",
        ).to("cuda")
        text_embeddings = pipe.text_encoder(text_in.input_ids)[0].half()

        neg_in = pipe.tokenizer(
            [negative],
            padding="max_length",
            max_length=pipe.tokenizer.model_max_length,
            truncation=True,
            return_tensors="pt",
        ).to("cuda")
        neg_embeddings = pipe.text_encoder(neg_in.input_ids)[0].half()

        # Classifier-free guidance: concat [uncond; cond]
        # Engine may or may not be built for batch=2.
        # Most TRT SD engines are built batch=1. To keep it robust,
        # we do *no CFG* here in TRT and keep CFG=1-ish behavior.
        #
        # If your engine supports batch=2, you can implement CFG properly.
        cond_np = text_embeddings.detach().cpu().numpy().astype(np.float16)

    # Generation settings
    W, H = 512, 512
    latent_w, latent_h = W // 8, H // 8
    steps = 6  # for live feel; with LCM you'd use 2-4
    cfg = 1.0  # see note above
    fps_smooth = True

    # Precompute timesteps
    pipe.scheduler.set_timesteps(steps, device="cuda")
    timesteps = pipe.scheduler.timesteps

    # OpenCV window
    cv2.namedWindow("trt_sd_live", cv2.WINDOW_NORMAL)

    frame = 0
    last_time = time.time()
    while True:
        frame += 1

        # Random latent
        latents = torch.randn((1, 4, latent_h, latent_w), device="cuda", dtype=torch.float16)

        # Denoising loop
        with torch.no_grad():
            for t in timesteps:
                # Scale input for scheduler
                latent_model_input = pipe.scheduler.scale_model_input(latents, t)

                # Prepare TRT inputs
                sample_np = latent_model_input.detach().cpu().numpy().astype(np.float16)

                # timestep dtype can vary by engine; try int64 by default
                # If your engine expects int32/float, change here.
                t_np = np.array([int(t.item())], dtype=np.int64)

                # UNet noise prediction via TensorRT
                noise_pred = unet_trt(sample_np, t_np, cond_np)
                noise_pred_torch = torch.from_numpy(noise_pred).to("cuda", dtype=torch.float16)

                # Step
                latents = pipe.scheduler.step(noise_pred_torch, t, latents).prev_sample

            # Decode latents -> image
            imgs = pipe.vae.decode(latents / 0.18215).sample
            imgs = (imgs / 2 + 0.5).clamp(0, 1)

        # Convert to displayable BGR uint8
        img = imgs[0].permute(1, 2, 0).detach().cpu().numpy()
        img = (img * 255).astype(np.uint8)
        img = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)

        # Force monochrome (optional, makes “minimal b&w” consistent)
        gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
        img = cv2.cvtColor(gray, cv2.COLOR_GRAY2BGR)

        # Show
        cv2.imshow("trt_sd_live", img)

        # FPS display
        now = time.time()
        dt = now - last_time
        last_time = now
        fps = 1.0 / max(dt, 1e-6)
        cv2.setWindowTitle("trt_sd_live", f"trt_sd_live  |  fps: {fps:.1f}  |  steps: {steps}")

        # Exit on ESC
        key = cv2.waitKey(1) & 0xFF
        if key == 27:
            break

    cv2.destroyAllWindows()

if __name__ == "__main__":
    main()

Run it:

uv run python main.py

4) Very important notes (so you don’t get stuck)

A) This code currently runs CFG ≈ 1.0

True CFG requires running UNet twice per step (uncond + cond), or batching them together. Many static TRT engines are built for batch=1, so batching [2,4,64,64] will fail.

If your engine was built with batch=2 support, tell me and I’ll show you the 6-line change to do real CFG at almost no extra overhead.

B) Timestep dtype might not match your engine

Some TensorRT UNet exports expect:

int64 timestep
int32 timestep
or float32 timestep

If you get a binding dtype mismatch error, change:

t_np = np.array([int(t.item())], dtype=np.int64)

to np.int32 or np.float32.

C) If your engine binding names differ

You’ll see them printed at startup:

=== TensorRT Engine IO ===
IN  sample ...
IN  timestep ...
IN  encoder_hidden_states ...
OUT out_sample ...

If they’re different, tweak the mapping candidates in TRTUNet.__init__.

5) How to get “high fps / uncanny / minimal b&w”

For speed:

steps = 4–6 (or 2–4 with LCM baked into the engine)
keep 512×512 fixed (static engine)
keep post-processing light (we only grayscale)

For more “Deforum melt” feel:

instead of random latents each frame, feed the previous latents back in with a little noise (I can add that loop).
or do an img2img-style latent blend between frames.

One key question (so I can tailor it perfectly)

When you built your .engine, was it compiled for:

batch = 1 only, or
batch = 2 (for CFG batching), or even dynamic batch?

If you tell me that + your compiled resolution, I’ll:

enable proper CFG (if possible),
switch the loop to a temporal feedback mode (Deforum-like),
and push the highest stable FPS your engine can handle.

Good — this screenshot explains everything.

You built the engine with:

batch_size_opt = 4   ❗
height = 512
width = 512

That is the root cause.

🔥 The Real Problem

Your TensorRT engine was compiled expecting:

batch = 4

But your KSampler + workflow is running with:

batch = 1

Static engines are strict about batch size.

When the loader tries to split conditioning internally (that curr_split_batch variable), the mismatch explodes and you get:

UnboundLocalError: curr_split_batch

It’s not really about that variable — it’s a batch shape mismatch.

✅ The Correct Fix

Rebuild the engine with:

batch_size_opt = 1

Leave everything else:

height_opt = 512
width_opt = 512
context_opt = 1

Then:

Delete the old .engine
Build again
Move it into models/tensorrt/
Restart ComfyUI
Run with batch=1 in KSampler

That will fix the crash.

💡 Why batch=4 was wrong

The TensorRT builder is NOT building a “maximum batch” engine.

It is building a static-shape optimized engine.

If you tell it:

batch_size_opt = 4

It assumes:

Input shape = (4, 4, 64, 64)
Conditioning = (4, 77, 768)

But your graph sends:

(1, 4, 64, 64)

Mismatch → loader logic breaks.

🧠 When Would You Use batch=4?

Only if:

You are manually feeding 4 images at once
Or you are batching CFG inside the engine (advanced)
Or you are doing 4 parallel streams

For live generation:

Always use batch_size_opt = 1

⚡ Performance Note

Batch=4 does NOT make single-image generation 4x faster.

It just allows processing 4 images simultaneously.

For realtime video-style loop:

Batch 1
Few steps
Fixed resolution
TRT static

That’s the optimal setup.

🎯 After You Rebuild

Once it works, we can:

Enable proper CFG batching (if you want)
Or switch to LCM baked engine for 2–4 step realtime
Or convert your loop into a true temporal feedback generator

Rebuild with batch=1 and tell me what happens.

You can integrate it into TouchDesigner in two common ways people actually use in live setups:

External Python process generates frames → TouchDesigner ingests as a video stream (most stable + fastest)
Run generation inside TouchDesigner’s Python (possible, but usually painful because TD’s Python env + CUDA/TensorRT dependencies)

Given you want high FPS + TensorRT, do #1. TouchDesigner stays “the renderer/patcher,” your generator stays “the GPU inference server.”

Option 1 (recommended): Run your TensorRT SD loop as a local “frame server” and pull frames into TD

A. Send frames over Spout (Windows) / Syphon (macOS)

This is the most “TouchDesigner-native” live VJ way.

Pipeline

Your Python app renders frames on GPU
Publishes them to Spout
TD receives them with a Spout In TOP

How

In Python, use a Spout sender library (commonly: pyspout / SpoutGL wrappers).
In TD: add Spout In TOP → you’ll see your sender name.

Pros

Very low latency
Designed for real-time video
No networking required

Cons

Windows-centric (Spout); macOS needs Syphon equivalent
You’ll likely do an RGB texture upload each frame (still fast)

B. Send frames over NDI (cross-platform)

People do this a lot for installations and multi-machine.

Pipeline

Python generates frames
Encodes/sends NDI stream
TD receives with NDI In TOP

Pros

Very easy to route across machines
TD integration is straightforward

Cons

More latency than Spout
Compression/encoding cost

C. Stream frames over a local socket (fast + simple, but more coding)

Python sends raw frames over ZeroMQ / UDP / WebSocket
TouchDesigner Python DAT receives + writes into a TOP (e.g., via Script TOP)

Pros

Portable, no special video tech
Easy to attach metadata (seed, prompt params, fps)

Cons

More TouchDesigner-side scripting
Raw frames are big; you want shared memory or compression

Option 2: Put the model inside TouchDesigner (usually not worth it)

TouchDesigner can run Python, but:

Its Python package situation is fragile
CUDA/TensorRT wheels, PyTorch, etc. are hard to keep stable inside TD’s environment
Updates break easily

People have done it, but most serious realtime setups avoid it and run inference externally.

What most people actually do (the common “pro” pattern)

External generator + Spout into TD

TD handles compositing, feedback, post effects, projection mapping
Python/TensorRT handles only inference at a fixed res
You can also send control data back (OSC/MIDI)

Best control method: OSC

TouchDesigner is great at OSC.

So you can do:

TD sends OSC messages to Python:
- prompt string
- CFG / steps
- denoise strength
- seed / noise amount
Python updates generation live and streams frames back

This is the clean “instrument” setup.

Concrete setup recipe (Windows, Spout + OSC)

1) Python side

Add Spout sender
Add OSC server (e.g. python-osc)
In your loop:
- generate frame
- publish via Spout
- read OSC updates non-blocking

2) TouchDesigner side

Spout In TOP → get the frames
OSC Out CHOP to send controls
OSC In CHOP (optional) to receive generator stats (fps, seed)

What I need from you to give exact steps + code

OS (Windows/macOS/Linux)
Are you using Spout, Syphon, or NDI already in other projects?
What resolution do you want to send to TD? (512? 1024?)

If you say “Windows + Spout,” I’ll paste a ready-to-run Python script that:

loads your TensorRT engine + checkpoint
outputs frames as a Spout sender
accepts OSC controls from TD
…and I’ll tell you exactly which TOP/CHOP nodes to drop in TouchDesigner and how to wire them.

Great question — both StreamDiffusion and Depth Anything for TouchDesigner look “magical” from TD, but architecturally they’re actually very straightforward.

They both follow the same pattern:

TouchDesigner = UI / routing / compositing
Python backend = heavy model inference (GPU)
Frames move between them via Spout / shared memory / CUDA

Let’s break them down.

1️⃣ How StreamDiffusion works (conceptually)

StreamDiffusion is not “a new model.”
It’s a low-latency inference loop built on top of diffusion models.

Normal diffusion:

For each frame:
    Run UNet 20–50 times

StreamDiffusion:

Keep latent state warm
Reuse noise scheduling
Minimize per-frame overhead
Reduce steps (LCM / Turbo)
Avoid rebuilding graphs

What makes it “streaming”

Instead of:

Generate image from scratch every time

It does:

previous latent → small perturbation → few denoise steps → output

That’s very Deforum-like, but optimized for:

low step count
minimal Python overhead
pre-allocated tensors
no graph rebuild per frame

In TouchDesigner context

Most StreamDiffusion TD integrations do:

TD camera or TOP → sent to Python backend
Python backend runs:
- vid2vid diffusion (LCM / Turbo)
- few steps
Output frame sent back via:
- Spout (Windows)
- Syphon (macOS)
- NDI
- shared memory

TouchDesigner is not running diffusion.

It is just:

sending frames
receiving frames
displaying them

2️⃣ How Depth Anything for TouchDesigner works

Depth Anything is a lightweight depth estimation model.

Architecture is simpler:

Input frame → depth network → depth map

No iterative denoising loop.

In TD integrations:

TD sends TOP frame to Python backend
Python runs depth model (usually ONNX or PyTorch)
Sends grayscale depth map back to TD
TD uses it for:
- displacement
- masks
- compositing
- particle systems

Depth models are fast because:

single forward pass
no 20-step loops
smaller networks

That’s why Depth Anything often feels “real-time” even without TensorRT.

3️⃣ Why these feel smooth inside TouchDesigner

Because they avoid:

❌ ComfyUI
❌ Heavy Python graph overhead
❌ Rebuilding diffusion pipeline per frame
❌ CPU↔GPU thrashing

They:

✔ Keep tensors on GPU
✔ Preload everything once
✔ Run a tight loop
✔ Use low-step schedulers
✔ Only move final RGB frame to TD

4️⃣ The Real Secret: Latent Recycling

For real-time diffusion in TD, the trick is:

Instead of:

latent = random()

Use:

latent = previous_latent + small_noise

Then run 2–4 denoise steps.

That creates:

motion continuity
melting
weird body morphing
no full reset cost

This is how you hit interactive speeds.

5️⃣ Why StreamDiffusion feels faster than ComfyUI

ComfyUI does:

graph evaluation
node routing
safety checks
memory swaps
dynamic shape handling
Python object overhead

StreamDiffusion does:

fixed loop
fixed shapes
direct tensor ops
minimal abstraction

Even without TensorRT, that difference is noticeable.

With TensorRT + few steps, it becomes dramatic.

6️⃣ How your TensorRT project fits into this

What we built earlier:

TensorRT UNet
PyTorch VAE + CLIP
manual denoise loop
OpenCV display

That is essentially a simplified StreamDiffusion backend.

To integrate into TouchDesigner, you’d:

Replace:

cv2.imshow()

With:

Spout send

And optionally:

accept input frames from TD
use them as img2img latent conditioning

7️⃣ Typical TD real-time architecture

Here’s what many advanced installations use:

TouchDesigner
   ↓ (Spout Out TOP)
Python backend (StreamDiffusion / TRT)
   ↓ (Spout In TOP)
TouchDesigner
   ↓
Effects / mapping / projection

Optional control:

TD → OSC → Python

8️⃣ Why not just run everything inside TD?

Because:

TD’s Python environment is fragile
CUDA / TensorRT / torch version mismatches break easily
Updating TD can break inference
External backend = cleaner separation

Most serious projects isolate inference.

9️⃣ Summary

StreamDiffusion in TD =

Few-step diffusion
Latent recycling
Preallocated tensors
GPU-only loop
Frame transport via Spout/NDI

Depth Anything in TD =

Single forward pass model
Frame in → depth out
Much lighter

If you want, I can now:

Show you how to modify your TensorRT loop to behave exactly like StreamDiffusion (latent recycling + small noise injection)
Or show you a full TouchDesigner + Spout integration example
Or design a hybrid where depth controls diffusion in real time

Which direction are you aiming for:
🔹 Live camera stylization
🔹 Generative melting walkers
🔹 Depth-driven diffusion
🔹 Interactive installation instrument?