Real-time diffusion in TouchDesigner - StreamdiffusionTD Setup + Install + Settings

Рет қаралды 53,606

Күн бұрын

Important ! Read description for Installation Tips ! This uses TouchDesigner 2023 and will not work in TouchDesigner 2022.
Currently only working using windows + nvidia graphics card.
StreamDiffusionTD TOX available here - / dotsimulate
StreamDiffusion github - github.com/cumulo-autumn/Stre...
BEFORE Installing - make sure to have:
Python 3.10 (make sure Python is added to Path in your system's environment variables)
CUDA 11.8 or CUDA 12.1
NDI-SDK - ndi.video/download-ndi-sdk/
GIT (check add to PATH when asked) - git-scm.com/download/win
If GIT is already installed and not in PATH - www.delftstack.com/howto/git/...
00:00 Introduction to StreamDiffusionTD
00:48 System Requirements for StreamDiffusionTD
01:12 Setting up operator in network
01:37 Downloading StreamDiffusion repo
02:10 Installing venv (first install only)
03:04 Setting Model + parameters before Stream
03:30 Step Schedule part 1
03:46 Similar Image Filter
04:00 Starting the Stream / Cmd window
04:41 Disable / Enable to see Stream bug
05:10 Seed
05:47 Prompts
06:16 Setting up Prompt Box
07:10 Negative Prompts do NOT work
07:22 Step Schedule part 2
08:16 Guidance Scale / Delta
08:47 Experiment !
09:16 Lora loader page
09:33 Callbacks
10:03 Stopping the Stream
10:09 Setting OSC Ports / Stream name
10:18 Visible Window (ON is default)
10:32 Say hey on discord !
11:00 Thanks !!

Пікірлер: 187

@elekktronaut 4 ай бұрын

absolutely mindblowing!! getting 6fps on my laptop 3070, thank you so much ❤

@mecherodriguez5114 3 ай бұрын

me too

@TouchDesignerOfficial 4 ай бұрын

Amazing work and great tutorial. A small tip for your callbacks DAT, on the Common page of parameters if you switch 'Content Language' to Python then the DAT will use python syntax highlighting.

@FunctionStore 4 ай бұрын

Dotsimulate InSession when?

@blankensmithing 4 ай бұрын

Yessssss 100 %@@FunctionStore

@dotsimulate 4 ай бұрын

❤Thank you !! I will make sure to set that when callbacks are created.

@engard 4 ай бұрын

❤

@gleix_ 4 ай бұрын

Lyell you've got a voice ready for public access. ALSO this looks amazing! For people who know TD and already have a competent workflow, but are lacking in knowledge of SD, tools like this immediately open the door and make it more accessible than it has any right to be. I think seeing SD generating images in real-time, within TD in a format I can immediately wrap my head around to a better degree, is a pull I haven't had with SD until now. Awesome work!

@blankensmithing 4 ай бұрын

Yesssss Lyell! Amazing work and tutorial!! Would love to see you create more tutorials in the future!!

@ChristianBrinkmannLive 4 ай бұрын

Yeees finally! 😍 You‘re the best for putting this out! And the tutorial is well made made too 🖤

@cristianpenas 2 ай бұрын

This is super inspiring, it gives a clear visual image of the fundamentals of diffusion models

@MarkDrMindsetChavez 3 ай бұрын

Whoa! Now this is what I needed! Thanks for the great work and generosity.

@jaysonwall434 3 ай бұрын

Whoa! That was cool. Thanks for the breakdown. Downloading stream diffusion now. :)

@datsmabowl 3 ай бұрын

Incredible work and very accessible tools. Thanks so much!

@SilentMediaLab 4 ай бұрын

nice nice! super nice! thank you for your work, it's amazing the opportunity to create and work with these tools. cheers

@RetropunkAI 5 күн бұрын

This is EXACTLY what I needed this weekend. Thank you so much for this.

@xavierg_music 4 ай бұрын

Thank you so much for sharing this!! ❤ My dreams just came true..

@LANworldArchive 4 ай бұрын

That's amazing gonna try if for sure!!! Thank u for sharing this 🙏🌀

@Jonny11299 Ай бұрын

Guys this is frickin insane. Amazing work!

@antialias8588 4 ай бұрын

Amazing. Thank you for this !

@manu.vision 4 ай бұрын

Proud of you Lyell! ❤

@gregderivative2647 4 ай бұрын

fabulous tool design and great explanation in the video.

@menguzar 3 ай бұрын

Great tutorial and great asset man, thanks! What's the OP that uyou use for the prompt list? is that a table dat or something else?

@brvoigt 4 ай бұрын

Nice use of those new sequential parameters! 🔥

@dotsimulate 4 ай бұрын

Thanks ! They are quite handy. definitely a great addition for 2023.

@FunctionStore 4 ай бұрын

he's done it! I know what I'll be doing tomorrow...

@nicsnutzius6305 3 ай бұрын

Thanks a lot for this amazing work on StreamDiffusion! Regarding the NDI source name issue: I had similiar issues with the refresh on another project and realized that using 'Bind' instead of 'Reference' fixes the issue.

@dotsimulate 3 ай бұрын

Will investigate. Thank you.

@Adellovisuals 4 ай бұрын

Thanks a lot for this! Running at 8 FPS with 4070

@kinkysoybean 4 ай бұрын

thank you for this amazing tutorial!

@kiksu1 4 ай бұрын

Fantastic work!

@freivonstil 3 ай бұрын

Thank you so much for sharing! I'm a big fan even after the first minute already! Sub

@_geom 3 ай бұрын

Wow thanks for sharing

@nothingrhymeswithferg3744 4 ай бұрын

this is actually just the craziest thing i've ever seen

@InOneArt 15 күн бұрын

And now. Imagine an AI chatbot, when communicating with which you are accompanied by a visual series that displays your adventures in real time. Impressive

@iloveallvideos 3 ай бұрын

AMAZING!!! FROM THE CREATUVES AROUND THE WORLD, THANK YOU

@folders_apries 3 ай бұрын

amazing stuff!

@osamajamil4804 2 ай бұрын

Amazing work. A quick question, can we connect this diffusion model in touch designer to depth camera, so the changes are done according to human motion?

@unveil7762 4 ай бұрын

You my new best friend!!! ❤❤❤❤

@lee_sung_studio 3 ай бұрын

Thank you.

@user-jp1wx6sy2w Ай бұрын

you are best!!!! thanks!!!!

@mecherodriguez5114 3 ай бұрын

beautiful work, great tutorial, I'm running on average at 6 FPS . If I wanted to ask how to load new MODEL IDs?

@santiago15608 3 ай бұрын

Thanks for the video! Is it a must to download the Python v 3.10.9 + cuda 11.8?

@diogomarques8 4 ай бұрын

Amazing! I would really like a toturial (or get in touch with) about the fireball clip. Looks impressive, and i'm working on a project with similar aesthetic, but im having trouble on generating the refrence image part before the SD tox.

@teodoricoteodorico 4 ай бұрын

Seems from his video he is using hand tracking (possibly to move a circle and then generate from there) i'm currently experimenting with mediapipe live segmentation tool as a starting generation point

@amiiineh 4 күн бұрын

This is pretty cool! any advice on getting it running on a mac?

@ChyaBra 2 күн бұрын

Super excited to try this! The tutorial starts out with the Stream Diffusion operator on the screen but I'm not sure how to get that operator. Do I install the repo somewhere and point to the directory in the parameters box?

@alphapyro360 4 ай бұрын

amazing!

@jakoblacour 4 ай бұрын

Really great! Got 1.5 fps with my GTX 1070 😅

@teodoricoteodorico 4 ай бұрын

same setup, confirming drop to 0.7/0.8 with live image segmentation running parallel

@Kirmm 3 ай бұрын

Awesome stuff! That 4090 really helps.

@nyangau 4 ай бұрын

awesome!!!

@user-sv3rq3qp4w 2 ай бұрын

Hey! Thank you very much! Just became a patreon. I'm really excited. Does this work with python 3.12? or only 3.10.. I struggled to find the install file for 3.10 version.. had to change it directly on the url.

@zpacetree 29 күн бұрын

did you ever find out if it works on 3.12?

@ANGELDISENY 9 күн бұрын

@@zpacetree in case it helps you: I have tried all versions and it only worked to me with 3.10

@moritzmaas2011 2 ай бұрын

looks amazing. is there another way without an nvidia card?

@leono5122 Ай бұрын

Thank you for the video! Is an image-to-image generation also possible? For example, a jpg file as input and stream diffusion creates similar looking images in real time?

@louisfievet9341 19 күн бұрын

Heyyy Mate ! Great tut and infos !! Thanks for sharing ! Does this work on Mac ?

@dotsimulate 18 күн бұрын

As of a few days ago, yes ! On Apple silicon macs and definitely is slower than nvidia graphics cards though.

@jennnital 3 ай бұрын

Hi there! i've previously gotten streamdiffusion to work locally with conda instead of venv - whats the best way to mod this TOP? :) or should i still use the venv process?

@DJFortune1502 24 күн бұрын

Amazing! Thank you for sharing. Would this work on a Mac? Specifically MBP M2 Max 64 GB

@rec-trick 4 ай бұрын

amazing

@rudolf_II 3 ай бұрын

since i work on a mac pro (win 11 is installed) with a vega 56, i question is will the amd GPU gets a support for this?

@v.rtx_void 4 ай бұрын

Awesome as usual, thank you so much for this!! Just 2 questions: Will it work with 2022 Touchdesigner Versions? And do you, by any chance, also have a Gumroad page for purchasing, instead of Patreon?

@gregderivative2647 4 ай бұрын

nom it uses Custom Sequential Parameters (see wiki, opens a lot of doors!), which is in 2023.Official only.

@v.rtx_void 4 ай бұрын

thanks, when I ayked it was not in the description yet

@antoniaangel6384 4 ай бұрын

Hey! I tried and it seems not to work with 2022 TD.

@ithamardimarco9082 2 ай бұрын

Amazing 🔥🔥🔥 does it run offline?

@swadedenimproduction Ай бұрын

Incredible! Can this run on A Mac Studio with an M2 Max/ultra?

@nicotritschler675 Ай бұрын

Thank you so much for this! I may be lost right now but how do I write the path to a local safetensor model? I can't seam to get it working

@teodoricoteodorico 4 ай бұрын

simply wonderful, any insight on how to extract the live fps from streamdiffusion in order to set a time comp?

@dotsimulate 3 ай бұрын

Ive got CHOPs exposed as out2 for both fps and framecount (frame total since stream began)

@anjiraj2899 2 ай бұрын

@dotsimulate bro where can i find or install the StreamdiffusionTD operator new to touch designer

@infernall_blessfree_waghun8377 2 ай бұрын

Hey :) would it be possible to do a turial of this tipic on Mac :) kind regards

@borisdesgeyl666 2 ай бұрын

Great! How do I get/create the Container nvidia_smi in TD?

@mrspazecrafte 2 ай бұрын

Wen Mac version? :) is there compatibility issues?

@eventservice8511 3 ай бұрын

How did you make a photoshop generation in examples on github?

@jasmine8182 Ай бұрын

Is there any way to integrate your own LoRa/safetensor into this ??

@user-sd1vh5ds6t 2 ай бұрын

Hello, it's cool work. Does your plugin support tensorrt acceleration?

@dotsimulate 2 ай бұрын

Yep! I’ve got an improved installation process with a separate step for tensorrt.

@yizhu1391 3 ай бұрын

thanks for the amazing work! but I have some issues when tochdesigner try to open powershell, the powershell will crash. Anyone knows the reason or solution?

@tiwonku 4 ай бұрын

i think im in love..

@elekktronaut 4 ай бұрын

❤

@MrPlasmo 4 ай бұрын

great tutorial and will try this out. Is there a way to easily save the output generations to mp4 file or other format?

@dotsimulate 4 ай бұрын

Definitely ! moviefileout TOP (Touchdesigner can do everything)

@hoole3796 Ай бұрын

very cool video. I have a problem for the installation I have an error for cuda python How can i resolve ?

@thiagojosebraganca8704 11 күн бұрын

hello. I was enchanted by the possibilities of stream diffusion. I'm a complete newbie and I don't know how to do the installations prior to the tutorial. How can I access this operator? Do I need to subscribe to your Patreon to have access or would I be able to use this tutorial? tanks

@georgiaharrison1572 Ай бұрын

Hey when I start stream with the pulse and my cmd comes up, it get to the end of the code but doesn't get past 'preparing stream...' Any idea why not? I've left it for over an hour and still the same. Have a good computer and high vram nvida card. Really appreciate any help - thanks!!!

@marshalldestro Ай бұрын

wait, so the only way to do this is to subscribe to the patreon? i went through the trouble of installing all the stuff only to realize the TD base featured in this clip doesnt exist in TD :(

@santobosco5008 4 ай бұрын

Thanks for this! I am currently unable to run it, after the Intializing NDI streaming etc, it runs into: D:\StreamDiffusion\StreamDiffusion\venv\lib\site-packages\diffusers\configuration_utils.py:135: FutureWarning: Accessing config attribute requires_safety_checker directly via 'StableDiffusionPipeline' object attribute is deprecated. Please access 'requires_safety_checker' over 'StableDiffusionPipeline's config object instead, e.g. 'scheduler.config.requires_safety_checker'. deprecate("direct config name access", "1.0.0", deprecation_message, standard_warn=False) Process Process-4: and goes on and on..what should i do? thankssss🙏🙏🙏🙏🙏🙏

@arthurquet2014 24 күн бұрын

Thanks a lot ! But Lora models don't work, how do you setup this ?

@jasmine8182 Ай бұрын

how do u set up for the first time? as in even getting the stream diffusion window up on touchdesigner?

@dvdgoldberg 2 ай бұрын

great work !! would it be possible to use IPA adapters ?

@dotsimulate 2 ай бұрын

I absolutely want to add this.

@tianlianyang 13 күн бұрын

Can you tell me how to use controlnet? Are there any tutorials? Thank you very much

@yashasv.doodles 4 ай бұрын

Truly awesome. Just became a patron on Patreon haha. I'm having a slight trouble once i start stream. After the Triton error thing , i get a message saying : OSError: [WinError 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted Fetching 12 files: 0%| | 0/12 [00:00

@dotsimulate 3 ай бұрын

close all other instances of the operator in TD and also close all old cmd windows. Seems like you are running into the osc server not being able to start up.

@yashasv.doodles 3 ай бұрын

My stream keeps turning on and off ( starts and stops) and i'm getting like 0.96 fps whenever it comes on for a fraction of a second. Can you guide me as to how i can maintain my FPS and keep the stream active ? Thank you! I'm so close to this, can feeeeel it

@yashasv.doodles 3 ай бұрын

How can i maintain a steady fps with no intermittent stops on stream?

@dotsimulate 3 ай бұрын

@@yashasv.doodles might be a limitation of your graphics card. To stop the start/stop stream, there is a timer inside that loops/checks for active stream. Called osctimer. If you change the length of that timer to like a few seconds, the stream should stay on once on. You can toggle the stream active toggle if it gets stuck in the on position as well.

@yashasv.doodles 3 ай бұрын

Okay start/stop issue is resolved. But i don't see any latent space / morphing take place on my null to the tox. Could you guide on how to connect hugging face model path/id to TD please!

@nstDesign 2 ай бұрын

Hi! @dotsimulate, I am having issues with it, i try to install venv and all requirements and everything appears as an error ERROR: Exception: Traceback (most recent call last): File "C:\TestRepostreamDiffusion\StreamDiffusion\venv\Lib\site-packages\pip\_internal\cli\base_command.py", line 160, in exc_logging_wrapper status = run_func(*args) ^^^^^^^^^^^^^^^ Not sure what to do... i don't really understand it.

@excido7107 3 ай бұрын

I know this might be a no brainer but I've tried to get a couple SDXL models working, like FenrisXL and DreamshaperXL turbo and they dont work, do I need to be using the Stabilityai huggingface models? or can I use my own comfyUI?

@dotsimulate 3 ай бұрын

sd-turbo is a 2.1 based model. It is not sdxl-turbo. Models supported are 1.5 based, because of the 1.5 LCM lora and also sd-turbo. You can use local model from safetensors file if you use the full file path instead of a hugging face id.

@excido7107 3 ай бұрын

@@dotsimulate Awesome I got 1.5 models going, starting up the stream is way faster now. I have a 4090 and can get pretty fast generations with SDXL especially utilising the turbo Loras and such, is there a way to do it with StreanDuffusion Tox or do I need to try connecting it to ComfyTD? :)

@sir_bass_tian 8 күн бұрын

hi there, im having issues. just upgraded my patreon account and downloaded the file but when i open it in TD the parameters box in the set up tab is empty. i already installed all that is needed or so i believe. can anyone please help me

@gnanny Ай бұрын

How do you load the operator, its not showing in the tab menu. im very new to td.

@ysy69 4 ай бұрын

Wow, this is amazing. Do I gain access to this Operator as your Start SD Patreon ?

@dotsimulate 4 ай бұрын

Yes ! thank you !

@ysy69 4 ай бұрын

@@dotsimulateI will join.

@dr0mp507 3 ай бұрын

any chance to use control net? or masks on the inputs to control some shapes in the generation?

@dotsimulate 3 ай бұрын

Can definitely give it a sense of the shape with input image. Not quite fast enough to run with controlnet at this point for realtime. But definitely possible non realtime with ComfyUI.

@dr0mp507 2 ай бұрын

Comfy ui is in itself a whole process, I saw in their demos that they were using photoshop as an input for the generation, drawing a base shape and sent it to generation.

@johnmcglynn7174 3 ай бұрын

Would this work on a Windows VM on a mac?

@unijascha3005 20 күн бұрын

i am super unexperienced so how do I add the operator into touchdesginer network?

@gabrielecella 2 ай бұрын

wow

@gabrielecella 2 ай бұрын

wht is for you the best way to record the stream? thx in advance

@R0n7n Күн бұрын

You start the Tutorial by adding the StreamDiffusionTD Operator in TD, however I don't have this in my TD installation what do I do?

@jeonsewon.__ 3 ай бұрын

Hi I want to ask you one thing about this awesome tutorial I bought your patreon and I did what you taught me but the fps came out around 0.272 and the stream kept turning off and on and off. It doesn't work properly over and over again. What's wrong with this?

@dotsimulate 3 ай бұрын

The fps is likely due to computer hardware speeds Easy fix but for the start/stop. Send me a message on discord/Patreon and I will explain.

@nathangalan7778 Ай бұрын

when all the setup is great how do i import streamdiffusion in touchdesigner

@diegodereisfuchs Ай бұрын

Isthere any way of implementing it with an AMD Graphics Card?

@drew9555 2 ай бұрын

is anything out of pocket aside from becoming a patreon?

@glev6015 Ай бұрын

How I can duble the resolution?

@M4Pxls 3 ай бұрын

Looks amazing! Is it possible to make it portable with anaconda venv instead of installing directly on system?

@dotsimulate 3 ай бұрын

It is using venv so it isn't installing to system python. A few people are using it with conda env as well, as the script can be run from cmd line. I plan to add option to support launch with conda in future.

@M4Pxls 3 ай бұрын

@@dotsimulate sounds great!

@liammurphy366 3 ай бұрын

Does this work on mac?

@grimmwerks 2 ай бұрын

Is there a branch that would work with m1 MacBook?

@pelol8819 Ай бұрын

Hi, im unable to get trough the start streaming step 'cause cmd gives me this message as soon as it opens: Traceback (most recent call last): File "C:\Users amir\Desktop\TestInstall\streamdiffusionTD\main_ndi.py", line 20, in from utils.viewer import receive_images ModuleNotFoundError: No module named 'utils' and then it closes immediately, I cant get no output. I think i've followed every step accordingly, even adding git to path, and installing the model with all of its features in a folder (in my desktop in first instance to try it out). I would love to try this awesome feature!!!

@FilipTrichkov Ай бұрын

Same thing for me, I get to start streaming step but I get the No module named: tkinter error. I really would like to know how to get past this error please!!