RTX Background Removal in TouchDesigner

RTX Background Removal in TouchDesigner - Tutorial

Рет қаралды 15,564

3 жыл бұрын

Get access to 200+ hours of TouchDesigner video training, a private Facebook group where Elburz and Matthew Ragan answer all your questions, and twice-monthly group coaching/mastermind calls here: iihq.tv/Trial
If you’re a TouchDesigner Beginner, check out our TouchDesigner Tutorial Series! We teach you all the basics of how to use TouchDesigner, including mouse controls, hotkeys, parameters, the operator families, and more: interactiveimmersive.io/touch...
The Nvidia RTX GPUs have tons of amazing and easy to use features for your interactive and immersive art. One of the most used features is the ability to quickly and easily to a high quality background removal using almost any camera without the need of any fancy lighting, equipment, or green screen. In this video, we'll cover setting up RTX's Background Removal natively inside of TouchDesigner.

Пікірлер: 48

@atltvhead2840 3 жыл бұрын

Nice!!! The RTX background removal is pretty great. I've been testing out a few different ai tools to for segmentation. Such a fun area!

@TheInteractiveImmersiveHQ 3 жыл бұрын

It's awesome how far this stuff has come in such a short amount of time! It's really exciting and so flexible that you can use it basically anywhere without much extra gear or environment setup.

@berkvarol1417 23 күн бұрын

you just saved our project! thank youuu

@TheInteractiveImmersiveHQ 22 күн бұрын

Glad it helped! Thanks for watching :)

@mateuskenis8504 3 жыл бұрын

Very good Tutorial love it

@TheInteractiveImmersiveHQ 3 жыл бұрын

Thanks! Glad you enjoyed it :)

@mavro360 3 жыл бұрын

Nice Erza!! Can u teach how to use touchengine to control ue4's actors paramet in realtime??. 🙏🙏🙏 Always grateful for share your knowledge w the comunity...

@TheInteractiveImmersiveHQ 3 жыл бұрын

For sure! It's on our to-do list :)

@MatijaErceg 3 жыл бұрын

I'd be curious to know if the 3 frame lag is different when using the Nvidia Broadcast virtual video-in device, which also offers an alpha channel

@TheInteractiveImmersiveHQ 3 жыл бұрын

Great question! I'll give it a test and see. I assume at least 1 frame of lag will still be present in Nvidia Broadcast App, but where it falls in the 1-3 range would be interesting to see.

@jiakuaibu 3 жыл бұрын

thx for sharing...i am a td newbie , so which version of touchdesigner has nvdia background top？i cant find it.....

@TheInteractiveImmersiveHQ 2 жыл бұрын

You'll have to use one of the newest stable builds. I believe it was released only within the last few months. Which version are you on now?

@holoments Жыл бұрын

AMAZING! I am trying it but doesn´t work well with 5 webcams at the same time in my installation. Can we change the pretrained model to another with a better performance in stand up human body segmentation? I was trying to change it by myself but i can´t.

@TheInteractiveImmersiveHQ Жыл бұрын

Thanks! The default model it uses is best suited for a single person sitting in front of the camera, so that might be why you're running into issues. You should be able to use the Model Folder parameter to point to a different folder with compatible model files (see wiki page: wiki.derivative.ca/Nvidia_Background_TOP).

@holoments Жыл бұрын

@@TheInteractiveImmersiveHQ wiki web page is not active. Can you tell us just a little bit more about best compatible models formats and some links to download them? I reallyapretiate sooo much you quick answer. There´s not so much easy info about this so... THANKS!

@AndreaSorrentiZancle Жыл бұрын

wow! very nice! is there a way to get the same result in an MacOS? Nvidia background is supported just in Windows

@TheInteractiveImmersiveHQ Жыл бұрын

Yes, although you’ll need to work with external software for it to work. OBS (free) has a plugin that emulates the Zoom-style background removal (obsproject.com/forum/resources/background-removal-portrait-segmentation-virtual-green-screen.1260/), which you could apply to your webcam feed and then route the video into TouchDesigner. There are also paid options like XSplit VCam (www.xsplit.com/vcam) as well.

@AndreaSorrentiZancle Жыл бұрын

@@TheInteractiveImmersiveHQ thank yoooouuuu

@jjplano 3 ай бұрын

nice tutorial! it works on my machine, but the nvidia background dialog box doesn't show a GPU option to select. does it has something to do with using the non commercial version? I checked that TD uses the RTX GPU and not the Radeon that's also present :/

@TheInteractiveImmersiveHQ 3 ай бұрын

I can confirm that it does work in the non-commercial version, which RTX card do you have? The Nvidia Background TOP requires a 20, 30 or 40 series Nvidia RTX card to operate. It's also worth updating to the most recent version of TouchDesigner if you haven't already.

@jjplano 3 ай бұрын

@@TheInteractiveImmersiveHQ thanks for answering! I have the last version of TD, and an RTX 3060 card. are you on the TD help forum? I've made a post over there with a screen capture, but no one answered yet =(

@JBroadway 3 ай бұрын

Here before 100k subs

@TheInteractiveImmersiveHQ 2 ай бұрын

Yesss! 🙌

@moonisshah2127 3 ай бұрын

would it work with live webcam or camera feed?

@TheInteractiveImmersiveHQ 3 ай бұрын

Yes! Elburz is using the RTX Background Removal effect on a live webcam feed in the video :)

@DrakeRubicon 2 жыл бұрын

This might be somewhat out there, but would this also work with NVIDIA Tesla or Ampere Cards?

@TheInteractiveImmersiveHQ 2 жыл бұрын

Good question! I had a lot of trouble finding a definite answer to this. Closest I found was that the RTX features work on Ampere architecture, but I'm not sure if they'll work on the server side A100 cards that you're thinking about. I didn't find much for Tesla either. Do you have one that you're trying to use or looking at cloud solutions for hosting something like this?

@DrakeRubicon 2 жыл бұрын

@@TheInteractiveImmersiveHQ No, I guess we'll just have to try in the cloud. We currently have a project that performs few/one-shot-based training on videoframes. On the RTX we got onsite the training is just a notch too slow, so we were looking into cloud-solutions. The learning benefits a lot from good masks, where the background removal would come in. So I though, if we have to use a cloud VM, why not put TD into it, when it brings additional benefits, like good masks. - Also thx for the fast answer!

@TheInteractiveImmersiveHQ 2 жыл бұрын

@@DrakeRubicon It could work just make sure to have some patience. A lot of the time the cloud systems with those virtualized GPUs don't come with the best drivers and packages in Windows, so you'll likely have to install a bunch of things to get it working. Paperspace is also a good option if you need Cloud infrastructure but want to continue to use RTX cards in an easy to use setup.

@Kriti-fk8vk 12 күн бұрын

How do I put effects on just the body instead of putting a background behind?

@TheInteractiveImmersiveHQ 10 күн бұрын

Once the background has been removed, you can add a Threshold TOP afterwards, which will create a solid fill where the person is in the image. Then you can use a Composite TOP to add texture/effects. Hope that helps!

@7cd664 Жыл бұрын

"Error: failed to load feature: There was a problem deserializing the inference runtime engine" What can I do?

@TheInteractiveImmersiveHQ Жыл бұрын

What RTX graphics card do you have? If you have an RTX4xxx series card, this operator is unfortunately not supported yet. This error will pop up when you have an unsupported card.

@7cd664 Жыл бұрын

@@TheInteractiveImmersiveHQ 4080. OK thank you

@danbuckland250 Жыл бұрын

This doesn't work with the 40 series Nvidia cards

@TheInteractiveImmersiveHQ Жыл бұрын

Unfortunately yes, the Nvidia Background TOP doesn’t currently support 40 series cards :(

@RusticRaver 3 жыл бұрын

Anything new? this seems exact same as kinect again 3 years back!

@TheInteractiveImmersiveHQ 3 жыл бұрын

It might look like that at the surface but it has MANY benefits. The keying is a lot better, tighter, and less noisy than the Kinect player index/depth map. It also works in many more environments that a Kinect would have trouble with like outdoors, environments with existing IR lighting, and it won't trip out if you're wearing clothes that happen to reflect IR (very common issue with fashionable jackets at events haha). Finally it doesn't need any special hardware, so you don't have to worry about buying kinects, dealing with the trouble and cost of extending their signal, etc. All of that becomes so much easier and more cost effective when you can use any regular RGB camera like a generic logitech webcam.

@RusticRaver 3 жыл бұрын

@@TheInteractiveImmersiveHQ Nice one thx for reply, I'm think who actually owns and RTX for now? i'm sure in long run will benefits you just explaned. I'll be sad to see the kinect go but cool simplification of process

@RusticRaver 3 жыл бұрын

@@TheInteractiveImmersiveHQ You don't think motion capture with sensors will always beat what the electronic eye can see and deduct. hardware over software sort off. Going off topic here

@TheInteractiveImmersiveHQ 3 жыл бұрын

@@RusticRaver I think a lot of people are starting to get RTX cards. RTX 2060 came out at the beginning of 2019 so it's almost 2 years old now, meaning most people who bought a computer in the last 1.5 years are likely on an RTX card and basically anyone buying a system now is getting a capable RTX card. It's sad to see kinect go, but it's already kind of gone as far as I'm concerned. Kinect 2 is officially discontinued, and Kinect Azure I'm still not 100% sold on it being a full replacement, so at this point you're either using a new camera that's still only just a dev kit (azure), a camera that's been discontinued and you have to buy OEM adapters for (kinect 2), or you can use the RTX stuff, which is current and supported and being rolled out as moderately production ready. So I see it as the lesser of 3 evils haha

@RusticRaver 3 жыл бұрын

@@TheInteractiveImmersiveHQ I see , very interesting, I'm after a motion capture sort of body glove that would allow me interact in VR world like in real life. Best fun in Touchdesigner so far was kinect + vr interacting with particles, if we could have that control over particles minus the glsl headfuck, excuse my french, waw, so close to making small games in node fun way rather than c++ madness.

@zainabsaleemm Жыл бұрын

It won't work for me

@TheInteractiveImmersiveHQ Жыл бұрын

Hi! Could you describe the issues that you're running into? One thing to note about this particular effect is that it uses the Face Track CHOP, which needs a computer running Windows and an Nvidia RTX graphics card for it to work. If you don't have an RTX graphics card or are running Mac OS, you'll end up with an error saying something like "This node is not supported".

@auroracrowleystudio Жыл бұрын

@@TheInteractiveImmersiveHQ Exactly what I got (Mac m1) Any work around for M1 to remove black / make transparent background? Thanks!