ComfyUI: Stable Video Diffusion (Workflow Tutorial)

  Рет қаралды 33,121

ControlAltAI

ControlAltAI

Күн бұрын

This is a comprehensive workflow tutorial on using Stable Video Diffusion in Comfy UI. Stable Video Weighted Models have officially been released by Stabality AI and support up to 25 frames per second of video generation. While it might seem that the motion generation is random, it is not. In the tutorial, I showcase 6 unique workflow examples explaining each and every setting regarding the new video nodes and how to gain control over the motion animation.
------------------------
Relevant Links:
JSON File (KZfaq Membership): www.youtube.com/@controlaltai...
Comfy UI Manager GitHub: github.com/ltdrdata/ComfyUI-M...
SVD XT Model: huggingface.co/stabilityai/st...
SVD Model: huggingface.co/stabilityai/st...
FFmpeg: ffmpeg.org/
ComfyUI 101 Tutorial: • ComfyUI for Stable Dif...
------------------------
TimeStamps:
0:00 Intro.
1:02 Requirements.
4:29 Candle Workflow.
14:58 Hand Waving Workflow.
18:12 Portrait Workflow.
29:46 Cycling Workflow.
31:34 Motobike Workflow.
33:39 Boat Workflow.
43:00 Text2Image2Video Workflow.

Пікірлер: 113
@controlaltai
@controlaltai 5 ай бұрын
For Stable Video Diffusion XT 1.1, these settings work great: Motion Bucket - 127 Min CFG 2 Target CFG 2.5 Augmentation (0 - 0.10) - mostly 0 Steps 40 Sampler Euler Ancestral Scheduler SGM_uniform
@Adrian_Marmy
@Adrian_Marmy 4 ай бұрын
This must be one of the best tutorials I've ever seen. Not too slow, straight to the point and still very clear. 👌
@gabrielmoro3d
@gabrielmoro3d 8 ай бұрын
This is beautiful. Thank you so much for explaining everything in detail!
@joshualloyd6694
@joshualloyd6694 8 ай бұрын
Love this. You have made my last week or so (very new to this) so amazing! I can not thank you enough! Please don't stop! We need you...
@controlaltai
@controlaltai 8 ай бұрын
Will do. Thank you!!
@jeffscott3257
@jeffscott3257 8 ай бұрын
Great video! Detailed explanation of the nodes, adjustments and workflows helped a lot.
@controlaltai
@controlaltai 8 ай бұрын
Great to hear! Thank You.
@dariayudina8463
@dariayudina8463 Ай бұрын
this is the most useful video about comfy ui animation that i've already seen. thank you!
@ryanstark1869
@ryanstark1869 8 ай бұрын
Great job! So clear description, it's very helpful. Thanks for your work.
@controlaltai
@controlaltai 8 ай бұрын
Glad it was helpful! And thanks for watching.
@jagsdesign
@jagsdesign 6 ай бұрын
super cool explanation and methods and very methodical approach to a creative journey
@1manfactory
@1manfactory Ай бұрын
Thank you so much for providing this VERY helpful tutorial.
@tcmbravo2336
@tcmbravo2336 4 ай бұрын
Excellent tutorial so far in internet. Keep going
@JonathanWiese-bu3pk
@JonathanWiese-bu3pk 7 ай бұрын
Incredible good guide, thank you!!
@PirateOnYoutube
@PirateOnYoutube 7 ай бұрын
This is GOLD ! , I have to say , U re special ! Thanks for step by step 🍻
@controlaltai
@controlaltai 7 ай бұрын
Thank you!!
@frederikmaj4643
@frederikmaj4643 7 ай бұрын
Impressive. Thanks alot.
@danilsi6431
@danilsi6431 8 ай бұрын
🐌The second one! As always a fascinating guide with detailed explanations. Let's practice🧑‍🔬. Thank you very much for your video and the channel in general.
@controlaltai
@controlaltai 8 ай бұрын
Thank you! I am glad the video helped! If you have any questions during practice, feel free to ask. Happy to help, if I can.
@ZhiyuXiao
@ZhiyuXiao 4 ай бұрын
so helpful! thank you so much
@FeyaElena
@FeyaElena 8 ай бұрын
The first one! thanks for the video!
@controlaltai
@controlaltai 8 ай бұрын
Thanks for watching! Appreciate the support.
@M4Pxls
@M4Pxls 7 ай бұрын
Brilliant, thanks!
@controlaltai
@controlaltai 7 ай бұрын
😊 Thank you!
@tomaszwota1465
@tomaszwota1465 6 ай бұрын
A tip: You can get rid of manually setting the resolution in SVD_img2vid_Conditioning by passing the cropped image through "Generation Resolution From Image" which takes an image and spits out width and height -- connect those to the width and height converted into inputs in SVD img2vid Conditioning and voila, you can push different resolution images into it and it will automatically work fine.
@Yanus3D
@Yanus3D 6 ай бұрын
Good stuff!
@oracleofwater
@oracleofwater 5 ай бұрын
Memberships, neat. That's a good way to make sure you never see any of my money.
@user-bi9wf4xs5l
@user-bi9wf4xs5l 7 ай бұрын
Thank you very much!Your teaching quality is very high! If we use the LCM model and VAE, will it be faster?
@controlaltai
@controlaltai 7 ай бұрын
Thank you!! There is no lcm model for stable video diffusion. Stability ai has only svd xt and svd models available. One does 25 frames and other 14 frames. Lcm Lora might work with animate diff, I doubt for svd atm.
@perspectalog
@perspectalog 5 ай бұрын
Amazing as always Have you tried to build a SVD inpaint workflow?
@controlaltai
@controlaltai 5 ай бұрын
Thank you!! I was experimenting with some other things, but that's a good idea. Will give it a shot. Just a tip: If you have 24gb vram you can up the resolution to full hd 1920x1080 and SVD can handle it. Currently there should be one or two upcoming workflows which include motion control and temporal ControlNet.Will see if inpainting works somehow. Also SVD 1.1 has been released. You have to download the model from hugging face. This model is specifically trains for motion bucket 127, 6fps, 25 frames, standard resolution. You should get consistent results from it.
@revidcuteo
@revidcuteo 2 ай бұрын
What you convey is really helpful, I'll subscribe for that. I have a small question for you: how can we increase the duration of an image after processing? For example, if I have an image that lasts 2 seconds after processing, how can I extend it to 4 seconds or more?
@controlaltai
@controlaltai 2 ай бұрын
Reduce the frame rate of the output or add frame interpolation in between. Techniques are shown here kzfaq.info/get/bejne/qMyYaLijlrvZoIk.htmlsi=jOtvZhmzxeLM5NIU
@saffyk1
@saffyk1 5 ай бұрын
Hey amazing video very educational!! Do you have plans to make one for animatediff sparse ctrl? I have tried both and sparse ctrl is not bad at all, it cuts down the quality a little bit but i still cant figure out how to control the motion
@controlaltai
@controlaltai 5 ай бұрын
Hi, thank you! I will put animate diff / sparse Ctrl in the Todo list. Haven't played around with it yet.
@saffyk1
@saffyk1 5 ай бұрын
@@controlaltaithat sounds great will be looking forward to it :) Thanks
@nitinburli7814
@nitinburli7814 8 ай бұрын
Hi! Thanks for the tutorial. I have a few questions, in SVD_img2vid_Conditioning node, what is the difference between video_frames and fps? And how does it relate to frame_rate in the Final Video Output node?
@controlaltai
@controlaltai 8 ай бұрын
Hi, video frames is the total frames for generation. FPS is how many frames per second it will generate. Assume say we have no restriction on video frames. In such a case say I need exactly 1 min of video. So 1 min of video can be in the following ways Sluggish: 1800 Frames but 30 fps = 60 seconds. Calculation goes as 1800 ÷ 30 = 60 seconds. Now I want it smooth, so I need to get it at 60 fps. That means 1800 Frames 60 fps = 30 seconds To get 1 min frames have to be 3600. That's 3600 ÷ 60 = 60 seconds. The reason to not play around this is because the model is trained for 25 frames only. So if you select 25 fps it would be a 1 second video. Hence default is kept at 6. You can then make a 2 to 5 second video playing around with fps value. But it will make the video more sluggish or smoother. Total frames are too less to increase or decrease fps. For the output after it has generated the frames to make it slightly smoother you say put 10 fps. It will take the generated video and run it slightly faster. So say you took 3600 frames generated at 60 fps which made a 1 min video. Now this is generated. The video combine node can make it run at the same speed of 60 fps, slower at 30 fps or faster at 120 fps without regeneration. Like playing a video of KZfaq at 1x speed, 0.5x speed and 2x speed.
@nitinburli7814
@nitinburli7814 8 ай бұрын
@@controlaltai Thanks!
@tvanime6747
@tvanime6747 7 ай бұрын
30 fps ? 9 second de video ?@@controlaltai
@controlaltai
@controlaltai 7 ай бұрын
@@tvanime6747 I just gave an example. Svd only supports 2 to 5 second videos, max 25 frames only. Default 6 fps will give you approx 3 seconds.
@manolomaru
@manolomaru 3 ай бұрын
Suuuuper and tremendous video! ...Simply wow ✨✨
@sidejike438
@sidejike438 3 ай бұрын
@controlaltai how did you bring out the search option used for adding the nodes
@controlaltai
@controlaltai 3 ай бұрын
Double click the left mouse.....that brings up search.
@kikoking5009
@kikoking5009 Ай бұрын
Good Tutorial. Somehow I can't find that exact image Resize node after installing it from manager. I don't see the same settings in the node. Do you know an alternative node
@controlaltai
@controlaltai Ай бұрын
Try searching for "imageresize" and "image resize". One of them is the correct one.
@Littlepielittle
@Littlepielittle 8 ай бұрын
Thank you! Do you know the steps to install ffmpeg on colab?
@controlaltai
@controlaltai 8 ай бұрын
Hi, thank you for the support. I Never used google collab so I cannot test it. Try this Please: ! add-apt-repository -y ppa:savoury1/ffmpeg4 ! apt-get -qq install -y ffmpeg
@rezahasny9036
@rezahasny9036 5 ай бұрын
hello there, I have tried following the tutorial you provided for animating portrait images, but when I attempted to generate it, the resulting image was very different from what I expected. Suddenly, the image became blurry and also changed into various colors. I have followed all the settings in your tutorial. Please help me with this case.
@controlaltai
@controlaltai 5 ай бұрын
Hi, what is your motion id, fps and augmentation value in svd conditioning. Also what is your min cfg and the final cfg in ksmapler.
@bluesatin6307
@bluesatin6307 4 ай бұрын
I'm trying to complete blinking a character in the workflow, but every time I try, the camera just pans side to side and ignores the image with the eyes closed... It seems that something is wrong in my attempts... >.< The resolution is 9:16 instead of 16:9, I don't know if that's one of the consequences.
@controlaltai
@controlaltai 4 ай бұрын
Just try with 16 is to 9, cause for me it worked for that. I could not get it to work for 9:16. Use low motion.
@user-ns2qi1qu7b
@user-ns2qi1qu7b 7 ай бұрын
How did you make the picture of closing your eyes? I'd appreciate it if you could answer!!!
@controlaltai
@controlaltai 7 ай бұрын
That can be simply done using the same seed and inpaint. You can use standard in painting, ip adapter or a detailer. All would work.
@user-ns2qi1qu7b
@user-ns2qi1qu7b 7 ай бұрын
thx~!!@@controlaltai
@RamonRodgers
@RamonRodgers 4 ай бұрын
Thank you. I learned so much! But pythongosssss custom node wont load. I can't load "Show Text". Anyone else have this problem?
@controlaltai
@controlaltai 4 ай бұрын
Explain by what you mean can't load show text. Does it mean the nodes shows by its empty? If that's the case then its normal since you have to run queue prompt to get the info in the node.
@RamonRodgers
@RamonRodgers 4 ай бұрын
​@@controlaltai At 36:38 is shows launching "ShowText", but this node is not in the list even though I installed and I loaded "Pythongosssss". I tried to making just the "ShowTest" my own custom node and removed the snake. That worked, and now I have "ShowText". But now when I run "Queue Prompt" i get a new error. "ERROR:root:Failed to validate prompt for output 6:... ERROR:root:* SVD_img2vid_Conditioning 3:...ERROR:root: - Required input is missing: init_image...ERROR:root:Output will be ignored
@controlaltai
@controlaltai 4 ай бұрын
Check the workflow again and check the nodes connections. Something is wrong with the node connection. The error show SVD conditioning node is not receiving an image input.
@Catapumblamblam
@Catapumblamblam 6 ай бұрын
I installed the vhs node, but I can't find it and when I right click, it's not the same window, in mine there is no "search", there is only a list of nodes and groups to choose from.
@controlaltai
@controlaltai 6 ай бұрын
Search comes from double left click. Try that and search for vhs.
@Catapumblamblam
@Catapumblamblam 6 ай бұрын
@@controlaltai perfect, but VHS module is not present (but correctly installed, when I search it in the manager, it say to unistal or search for update)
@controlaltai
@controlaltai 6 ай бұрын
Check if it shows up here. Right click - add nodes - Video Helper Suit, if it doesn't close everything and restart, still nothin then go to comfy manager - uninstall - restart - install again - restart. Restart means close browser and command prompt.
@Ai_mayyit
@Ai_mayyit 2 ай бұрын
Error occurred when executing VHS_LoadVideoPath: module 'cv2' has no attribute 'VideoCapture'
@controlaltai
@controlaltai 2 ай бұрын
Cannot diagnose from just this error. Elaborte what are you doing and which node is giving the error.
@RamonGuthrie
@RamonGuthrie 7 ай бұрын
is "Image Resize" a default ComfyUI node or is it a custom node?
@controlaltai
@controlaltai 7 ай бұрын
A custom node, not the default one. Geth it via the comfy manager: palant - Image Resize for ComfyUI. You can then add node - image - "Image Resize" right at the bottom. Or Just search for "ImageResize" without space.
@user-rk3wy7bz8h
@user-rk3wy7bz8h Ай бұрын
Is this methode still good? Do you think its helful for someone who never worked with Videos in comfyui, like me.. because i dont know how to start learning videos. Appreciate
@controlaltai
@controlaltai Ай бұрын
Yeah nothing is changed in SVD models, except you now get a even better mod SVD XT 1.1. Basically with SVD you don't have any control. If you want to do proper video animation you need to look at other tutorials related to "Animate Difff and IP Adapter". None of these are for people who never worked in Comfy. As you need to understand the basic know how in Comfy UI like, what is SD, ControlNet, Custom Nodes, How to go about in ComfyUI, shortcuts etc. First check out basic tutorials about comfy then come to these tutorials.
@Arewethereyet69
@Arewethereyet69 7 ай бұрын
can you manipulate the direction you want the movement to be?
@controlaltai
@controlaltai 7 ай бұрын
Not yet. Stability AI will release 3 official Motion LoRA for horizontal, vertical and zoom.
@Arewethereyet69
@Arewethereyet69 7 ай бұрын
@@controlaltaithanks
@Arewethereyet69
@Arewethereyet69 7 ай бұрын
@@controlaltai also anyway to do this as a batch?
@controlaltai
@controlaltai 7 ай бұрын
Yes, in K-Sampler change "control after generate" to randomize. Then Below queue prompt tick mark "Extra". Change batch count to 3. This should generate 3 random videos effects form the same image and setting. You will get in consistent results.
@saffyk1
@saffyk1 5 ай бұрын
do these settings also work for xt1.1?
@controlaltai
@controlaltai 5 ай бұрын
No, 1.1 has a different setting sweet spot; the basic principle explained is the same, though.
@controlaltai
@controlaltai 5 ай бұрын
Try these for 1.1 and consistent animations: Motion Bucket - 127 Min CFG 2 Target CFG 2.5 Augmentation (0 - 0.10) - mostly 0 Steps 40 Sampler Euler Ancestral Scheduler SGM_uniform
@lazwardi
@lazwardi 8 ай бұрын
why my Image Resize node is not as complex as in the video. No ratio, resize_mode and some other. Edit: never mind, I forgot to reload the webui.
@controlaltai
@controlaltai 8 ай бұрын
There are two. If you installed the custom node, you will find one as "Image Resize" and one as just "ImageResize". Try both to see which is the correct one.
@RamonGuthrie
@RamonGuthrie 7 ай бұрын
@@controlaltai is this a default ComfyUI node or is it a custom node?
@controlaltai
@controlaltai 7 ай бұрын
A custom node, not the default one. Geth it via the comfy manager: palant - Image Resize for ComfyUI. You can then add node - image - "Image Resize" right at the bottom. Or Just search for "ImageResize" without space.
@moongaming4743
@moongaming4743 5 ай бұрын
Can I extend the duration? maybe 5 seconds?
@controlaltai
@controlaltai 5 ай бұрын
The only way to extend or reduce time is playing around with the fps in the video output node. The model is designed for 25 frames at 6 fps only. Once you generate these 25 frames. How fast or slow you want can be via the fps in video combine node. You can add frame interpolation to smoothen it out .
@Comic_Book_Creator
@Comic_Book_Creator 7 ай бұрын
ok I CREATE IT, BUT my resize image has no RATIO option, why?
@controlaltai
@controlaltai 7 ай бұрын
Install this custom node form Comfy manager (Image resize by Palant), then when searching in add node, search for "ImageResize" without space. The one with space is the default one.
@Comic_Book_Creator
@Comic_Book_Creator 7 ай бұрын
thanks @@controlaltai
@Stick3x
@Stick3x 8 ай бұрын
Not working on Silicon mac.
@controlaltai
@controlaltai 8 ай бұрын
Yeh I check. It's unfortunate. Another user reported this. The problem is with Torch, they dev says they plan to add support but even in torch 2.0 there was no support added. It will eventually come.
@SergeySokolov-rh7hu
@SergeySokolov-rh7hu 7 ай бұрын
@@controlaltai is it working now?
@controlaltai
@controlaltai 6 ай бұрын
@@SergeySokolov-rh7hu No way for me to test. I don't have a mac. I checked online, I could not find anything new regarding the same.
@Comic_Book_Creator
@Comic_Book_Creator 7 ай бұрын
please can you share the workflow?
@controlaltai
@controlaltai 7 ай бұрын
It’s already shared for Channel Members. Please check description on how to join.
@Comic_Book_Creator
@Comic_Book_Creator 7 ай бұрын
@@controlaltai thanks, I build it from your video
@blender_wiki
@blender_wiki 8 ай бұрын
SDV is damn bad or maybe is just a joke?. Fun for kids under 6. If you want good img to video use animatediff in confyUI.
@controlaltai
@controlaltai 8 ай бұрын
Agree, I had animated Diff on the bucket list, was about to do a tutorial on that, and this came out. Will get to animated diff in some time.
@danilsi6431
@danilsi6431 8 ай бұрын
A journey of a thousand miles starts with a single step, now a toy for kids under 6, and in a couple years a work routine for adults.
@Jennercreative
@Jennercreative 7 ай бұрын
Quien lo use le saca el provecho, yo lo uso para documentales y mi cliente esta satisfecho
@arifkuyucu
@arifkuyucu 5 ай бұрын
I wonder if I expect too much from Ai technology. In fact, such simple animations have been possible for a very long time with simple online .gif maker softwares without ai. It is enough to know a little bit of Photoshop and animation logic. Ai should be able to give more for this much effort! It seems much more logical to spend a few dollars every month and turn to paid software until you understand and solve the details of this crazy business!
@controlaltai
@controlaltai 5 ай бұрын
Its getting there. Give it some time. I totally get what you are saying. Since I am aware of what's upcoming (in pipeline), I can tell you things are progressing way beyond than expected. Problem is current hardware (consumer grade). We are like 1-2 years aways to make our own short 5 minute story driven animation, using nothing but single images.
@arifkuyucu
@arifkuyucu 5 ай бұрын
@@controlaltai I hope it will be faster :)
@jonw377
@jonw377 6 ай бұрын
While this might be a good tutorial it's not very friendly for someone just starting out with ComfyAI. First, she never explained how to do do a search for nodes. Fortunately I was able to google that and find out. Then around 22:45 she goes into warp-speed not explaining anything and next thing you know she has added a good 3 nodes, one of them just saying "Image" that I have no idea how she got. I tried to go back and stop the video, but it lasts for maybe one frame, so I have no clue how she did that, and how that specific node affects anything. Like I said, maybe for the more seasoned user this was a good tutorial, but for someone just starting it was a bit too frustrating.
@controlaltai
@controlaltai 6 ай бұрын
Hi, 22:45 Explanation: I have 2 images (eyes open, eyes closed). I am trying to create an image sequence. For that I use a node called "Batch Image." The output from this node will combine the input images. So with just one batch image node I get 2 images (eyes open, eyes closed). I duplicated this node and daisy chained it in order to get 4 images in the sequence of eyes open, closed, open, closed. You then connect this output to SVD Conditioning. How does this Node Affect Output? SVD conditioning is predefined and does not have a text conditioning input. Instead of giving 1 image I try and give the SVD input multiple images in sequence to influence the animation output. Unfortunately, it is not possible to include the basics of comfy in every video as it is very vast. This tutorial does require some basic ComfyUI know how. Have a look at this tutorial to understand the basics: kzfaq.info/get/bejne/o7uhraRmqZbPaXk.html If you have any questions, feel free to ask. Happy to answer and help out.
@jonw377
@jonw377 6 ай бұрын
@controlaltai That part I understood. WhereI got lost was the mouse jiujitsu you did right after you changed the image and got that one node that just says "Image" on it. Not really sure what you did to get that node, and what it actually does. I suspect it's like an "extension cord" like deal, but not sure. Sorry if I sound kind of thick here, but like I said, I am still very new to ConfyUI.
@controlaltai
@controlaltai 6 ай бұрын
No problem. That's completely ok. I understand. Can you please tell me the exact time frames +-5 seconds. So I can check and give an explanation.
@controlaltai
@controlaltai 6 ай бұрын
Okay I understood. That is called a re-route. Extension. You drag out from any node and in the middle of nowhere stop, leave the mouse it will give a reroute option. It's like an extension to connect longer noodles in the comfy graph. It's only function is extension for easy connection. The reason for fast forwarding was that's basic comfy thing. Important things are slowed down. But if you are new to comfy it's understandable. Feel free to ask any questions. The point of making these are you getting to use it and learn. Happy to help answer any further questions.
@jonw377
@jonw377 6 ай бұрын
@@controlaltai Thanks for your explanation and you patience :)
@Velanteg
@Velanteg 7 ай бұрын
Useless, you using lot of nods not linked in requirements. You using custom image resize nod without any info what it from.
@controlaltai
@controlaltai 7 ай бұрын
It’s already mentioned in the video. Check here 3:07
@Velanteg
@Velanteg 7 ай бұрын
@@controlaltai Anyway it seems it requre 4090, even 512x512 consumes whole 12 Gb VAM.
Stable Video Diffusion
9:04
XpucT
Рет қаралды 67 М.
Finger Heart - Fancy Refill (Inside Out Animation)
00:30
FASH
Рет қаралды 28 МЛН
Llegó al techo 😱
00:37
Juan De Dios Pantoja
Рет қаралды 59 МЛН
Ouch.. 🤕
00:30
Celine & Michiel
Рет қаралды 23 МЛН
AI Video Tools Are Exploding. These Are the Best
23:13
Futurepedia
Рет қаралды 50 М.
ip adapter Version 2 + automask [comfyui workflow tutorial]
9:58
Archilives | Ai | Ue5
Рет қаралды 8 М.
ControlNet Union for SDXL - one Model for everything
4:45
Olivio Sarikas
Рет қаралды 40 М.
Stable Video Diffusion  - RELEASED! - Local Install Guide
7:28
Olivio Sarikas
Рет қаралды 123 М.
ComfyUI: Scaling-UP Image Restoration, SUPIR (Workflow Tutorial)
46:57
The TRUE Resolution of FILM
12:31
ShyStudios
Рет қаралды 22 М.
New Image2Video. Stable Video Diffusion 1.1 Tutorial.
10:51
Sebastian Kamph
Рет қаралды 57 М.
Fooocus Update 2.5.0 - Enhance Feature, Automatic Masking and more!
27:15
The poor girl couldn't take care of the cat 😢😔
0:38
Ben Meryem
Рет қаралды 11 МЛН
Jumping off balcony pulls her tooth! 🫣🦷
1:00
Justin Flom
Рет қаралды 26 МЛН
ToRung short film: 🐶puppy is hungry🥹
0:32
ToRung
Рет қаралды 26 МЛН
Son ❤️ #shorts by Leisi Show
0:41
Leisi Show
Рет қаралды 2,8 МЛН
 tattoo designs  #tubigontattooartist #nctdream #straykids #txt
0:17
Hp Shorts video
Рет қаралды 26 МЛН