Real time Stable Diffusion in TouchDesigner

Рет қаралды 5,796

Күн бұрын

Video Tutorial on TouchDiffusion: Interactive Real-Time Generation using StreamDiffusion and TensorRT in TouchDesigner
TouchDiffusion: github.com/olegchomp/TouchDif...
If you'd like, you can treat me to a coffee: boosty.to/vjschool
Socials:
/ olegchomp
/ oleg__chomp
00:00 - Intro
00:09 - TouchDiffusion features
00:42 - Beginning of installation
00:50 - Python install
01:08 - Git install
01:18 - Cuda Toolkit install
01:38 - TouchDiffusion install
01:55 - where python
02:13 - webui install
03:10 - Fix for error in pop up window
03:27 - Model downloading & engine preparation
04:45 - SD Turbo engine
05:07 - TouchDiffusion settings
05:46 - Usage example
07:13 - Parameters explanation

Пікірлер: 57

@burakuville 27 күн бұрын

amazing job explaining the entire process...

@alessandrosaracino5463 Ай бұрын

this is great! lean and clean, thank you!

@marlus Ай бұрын

Thanks for this great solution! Amazing performance on 4070 12GB! SD Turbo and 1 sample step. I will test making other engine now. 3 sample steps were slow on 4070.

@Remvisuals Ай бұрын

instnaly fast and so nice to be in 512, congrats on this one, cant wait for more features!

@VJSCHOOL Ай бұрын

Custom resolutions already available, but keep in mind, that it will affect FPS.

@Remvisuals Ай бұрын

@@VJSCHOOL I tried building a model with different res and more steps, but it wouldn't finish... Super impressed with the speed, it's still super jittery and "stabley", but that's just animdif in general I geuss, excited to try out control net!!

@cayetanojozami4067 Ай бұрын

Amazing

@detailsstudio Ай бұрын

Thanks bro

@orkuncihan 11 сағат бұрын

Thank you ! I made it work but it just creates the image only when i press load engine button and its a stable image. How can i make it live =?

@user-ik6mr1jn7n 20 күн бұрын

Nevertheless I got finished engine preparation and selected the file which webui includes, it doesnt work.. I tried to press load engine and refresh engine several times, but it dosent work.. how to resolve this :(

@dleon9781 27 күн бұрын

hi ,i have followed your tutorial and every step went well .But in TouchDesigner,the log shows :cannot access local variable 'pipe' where it is not associated with a value.I don't know how did it happened

@VJSCHOOL 24 күн бұрын

Create issue on GitHub or Discord with full log

@prictor 24 күн бұрын

hi, i followed every step and there are no erros, but in touchdesigner when i hit pulse on "load engine" nothing happens, can anybody help me?

@VJSCHOOL 24 күн бұрын

Create issue on GitHub or Discord and share full log.

@mahmoud0800 Күн бұрын

thats amazing but i got an ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (tensorrt) please help!

@mahmoud0800 Күн бұрын

"No module named 'tensorrt'"

@TheDciel1 29 күн бұрын

Git is installed. Creating .venv directory... The filename, directory name, or volume label syntax is incorrect. Failed to create virtual environment. What is the problem?

@ogunkayikci7751 28 күн бұрын

chech where the python with "where python" command and copy paste in to webui.bat like he said

@placebo_yue Ай бұрын

does it have to be python 311? i have 310 and i don't wanna break all my other installations to try this...

@VJSCHOOL 29 күн бұрын

1) everything installed in venv, so it not affecting main Python env. 2) if you want to try with Python 3.10, then provide path to it in webui & use TouchDesigner version before 2023

@user-tz5ub1yq6o Ай бұрын

Very interesting, but when I clicked load engine in touchdesigner, I got an error like "cannot access local variable 'pipe' where it is not associated with a value". What is the problem?

@VJSCHOOL Ай бұрын

That’s mean either model or engine can’t be found

@yanyu813 18 күн бұрын

Have you solved it?

@VJSCHOOL 17 күн бұрын

@@yanyu813 download portable version on Github and try with it.

@N92601 7 күн бұрын

amazing fast speeds would you know why using my null of the render tops of my td projects as input isnt working? im getting flashing white and black with some cat images but noise top works normally, denoising value dosent change this would we have to repeat the webui model proceses with higher steps seems like it could be that or a different problem if so how should i do that thanks!

@N92601 7 күн бұрын

seems the same problem with a default moveifileintop

@N92601 Күн бұрын

I figured out I had to reinitalize pulse touchddiffusion folder and possibly clear log too I did both then reload engine

@alexdoho Ай бұрын

What's the difference between this and streamdiffusion?

@VJSCHOOL Ай бұрын

Explained in first 3 min of video It’s near x2 faster

@developmentmotion4765 27 күн бұрын

amazing... this is realy faster, can we put lora and vae to the folder? what lora and vae for the best creation

@developmentmotion4765 27 күн бұрын

and why icant change the seed ?

@VJSCHOOL 24 күн бұрын

Tiny VAE already baked in for best performance. Loras not supported yet.

@VJSCHOOL 24 күн бұрын

Seed can be changed in TouchDesigner with seed parameter

@leonidkrykhtin Ай бұрын

How to increase CFG to 7-10 instead of 1? The quality of generations is very low at 1.

@VJSCHOOL Ай бұрын

You should try to increase sampling steps (batch size)

@SwoleKitchen 24 күн бұрын

I really need a computer to do this. is there a way I could offload that GPU onto Google Collab?

@VJSCHOOL 24 күн бұрын

TouchDiffusion component required TouchDesigner, so it can’t run in Google Collab. You can try with original StreamDiffusion repo.

@SwoleKitchen 24 күн бұрын

@@VJSCHOOL - Sigh guess i just need better hardware.

@SwoleKitchen 23 күн бұрын

@@VJSCHOOLsweet I’ll try that

@user-eo3zn9kj9h Ай бұрын

I don't understand how it's posssible to have multiple frames per second with 20 steps. Why don't we use TensorRT on normal UI's like Automatic or Comfy ? Sorry in advance for having such a vague question but i find this quite amazing haha.

@VJSCHOOL Ай бұрын

Real-time can be achieved with Turbo models and Acceleration Lora, on low steps like 1-4 with TensorRT. TouchDesigner allow to copy data from and to GPU, that decrease latency.

@PeterBowlesify 19 күн бұрын

Cool! Can it work with controlnet?

@VJSCHOOL 10 күн бұрын

No support for now

@th3curator 16 күн бұрын

I am getting an error: name 'StreamDiffusionWrapper' is not defined. Any ideas?

@carloslavina9779 14 күн бұрын

same over here :( did you solve it?

@VJSCHOOL 10 күн бұрын

Try portable version

@user-iz7zt6md4b 14 күн бұрын

Hi! Does it also work with Mac?

@VJSCHOOL 10 күн бұрын

Only NVIDIA GPU

@kafidov Ай бұрын

As I understand it, this miracle cannot be run under Mac?

@VJSCHOOL Ай бұрын

Only PC & Windows

@SwoleKitchen 4 күн бұрын

Can you do this with AMD Ryzen Somehow?

@VJSCHOOL 3 күн бұрын

For now there is no way

@SwoleKitchen 3 күн бұрын

@@VJSCHOOL yeah I got it to run stable diffusion but it was quite slow, and then I couldn’t figure out how to put it in touch designer, I’ll just bite the bullet return this computer and get one with an nvidia videographics card