Nvidia only: you want your card in P8 mode as soon as you don't use it You don't want it to remain at P0 A script can do this easily and run the command nvidia-smi -pm 1 Remain to determine the conditions If gpu in P0 and no app or vm using the GPU is started run script

you are a true hero making these videos!

I have a work around with some optimisation scripts I use that lower & force the GPU into a low power state : 1st script runs on 1st boot of array : nvidia-smi --persistence-mode=1 2nd script that forces the lower power state when not in use (sometimes there can be lingering inactive processes ie. plex for no reason), run every 10 mins or so : gpupstate=$(nvidia-smi --query-gpu="pstate" --format=csv,noheader); gpupid=$(nvidia-smi --query-compute-apps="pid" --format=csv,noheader); if [ "$gpupstate" == "P0" ] && [ -z "$gpupid" ]; then fuser -kv /dev/nvidia*; fi; With these 2 scripts you can fix all the power usage issues you get from an idle nvidia GPU.

this is fascinating for home lab nerds like me. Thank you for the content & analysis

Great info. And this is not only about power. This will help to keep disks cooler as well if in the same box/case.

Great video Ed! always look forward to the new uploads :) Thanks for always sharing them with us!💖👌👍😎JP

Very interesting, Never would have thought...

This is some great testing!!

Thank you for the Video. I activated the autostart to my gaming VM. Because as you describe, my 2070S did 10-15W less when passtrhough to the Gamging VM and just let the VM running.

I've found that as long as you touch the GPU at least once - say, with Plex - then it will idle after you stop Plex. It's almost like the driver power management isn't kicking in until the GPU is used at least once. No need to run local GUI mode - I suspect we could write a script that makes the GPU do something (anything) to kick the driver into gear and then terminate that process.

Whenever he asks SpaceInvader: "Sounds interesting?" Me: glued to my desk nodding head yes

Thanks for the video! I actually have ditched unraid because of this exact issue. I have 2 GPUs for 3D rendering and had a very power hungry rig when shutting down the VM. Maybe I will have another look in a while.

Nice to see an AMD GPU working properly in unraid. I had so many issues I scrapped my Mac VM and just built a hackintosh. I may circle back to that idea.

Thanks I've been running the best way along without knowing it running my print server with a Quadro all the time, and I am glad I only pay $.11 kwh not $.25 kwh.

I observed the same thing ever since i started to use unraid back in 2019 but in addition to that there is some more: If you started up a Win10 VM for example with an Nvidia GPU the driver allows then to throttle back the frequency of the GPU in idle to have the power saving effect (you can prevent that from happening with the Nvidia settings in windows). However - when you add a second display to the GPU or an HDMI dummy plug (which can be useful for headless remote desktop access via anydesk for example) then you won't save any power - since with two displays attached the GPU driver does not seem to allow throttling back the frequency anymore.

Very interesting 👍

Thanks for the review. Can you do a video showing the hardware of your UNRAID servers? The PCIE-to-PCI adapter was a surprise for me.

Interesting info! Unrelated but I spotted you have pfSense and "Ultramagnus" as tabs/links on your Unraid server menu bar. Could you elaborate on what those are?

Awesome video! How about how to self-host a website in unraid video?

Interesting video! What if you use NVidia GPU only for transcoding in dockers? Should use then unraid in GUI mode or not?

i have 2, 290x. half the power draw when running a vm with passthrough.

I've been looking for an effective way to monitor and graph power draw from the wall (wanna save on power bills) and the SONOFF POWR2 smart switch is EXACTLY what I have been looking for. Are there good alternatives/competitors? Can you log Tasmota metrics to a Grafana database?

I have 2 Gaming RIG VM in 1 tower. My unraid draws below 100w when 2 gpu passthrough and idle but when i shutdown 1 VM it draws 120w that is why i usually turn on my other VM idle instead of turning it off. Btw i'm using APC Battery with powerchute software to read KW.

In unraid version 6.12.3, i can start easily an Ubuntu VM with a passed through NVIDIA GPU but a windows VM still doesn't want to start. I have a very similar server with an AMD 3900x cpu and a RTX2060, but unfortunately only in PCIe 2.0 x 2 mode and I am measuring also around 120W idle with the loaded drivers. If I don't pass the GPU through then GPU stats shows only 7W.

If you put your graphics card in the primary slot in stead of the secundary, it's becomes PCIE 3.0 x16 in stead of PCIE 3.0 x8. That's why you have a bigger power draw, but also a little bit more performance...

Let's say I wanting to get a razer core X egpu and connect it to thunderbolt on a labtop that has i7 10th Gen with 2080 super Max q now I want to revert to the egpu and use a 3080 or 3090 card would egpu use less power and make the labtop cooler avoiding usage from the Max q or will it increase the electric bill?

thank you for this video. I was unaware it was possible to start os gui mode with an AMD card and later pass it to a vm without crashing. It works perfect. Is there a way to "recover" unraid gui mode once the VM is shutdown, without restarting the server? simply shutting it down does not bring the gui back. Thank you again

Can you boot in GUI mode with 2 GPUs, have 1 for transcoding and use the second for passthrough?

How did you get the UPS information to appear in the bottom bar of the dashboard??

Can you please share details on the ESP flash interface that you used?

realy? right on time just woke up and i was looking at the Unraid ryzen. that has quadro p4000 (passed to debian) and gt1030 doing nothing. and watched the W consumption(me no like) thinking to swap them.

Have you tried how the whole thing behaves when you start Unraid in standard mode but connected with a HDMI dummy plug? Thereby the monitor output is addressed, maybe the power draw is regulated correctly then.

Really interesting. I'm a newbie with Unraid (youve taught me everything I needed to know!) and GPUs will be my next iteration! But 17p per KW??? 14p is the average nowadays, if you have a smart meter you can also sometimes sign up for a time of use tariff (sometimes only available to EV users though) that goes to 5.5p per KW from 00:30 to 04:30 and still flat rates to 14p at other times.

What you're seeing is that the driver is in P0 mode rather than P8 when you haven't loaded a window system. I get around this by using a script on array start to set persistence mode for my 1660S and 3070: #set persistence mode for both cards nvidia-smi -pm 1 Additionally I have a libvert hook setup when a VM stops to reset the mode for the card I pass through to the VM, otherwise the persistence mode is reset and stays at P0.

interesting, my server is the opposite and pulls less power when idle. ryzen 3700x, 1050ti, RX460. 70w idle and around 140w with 2 gaming VMs

I think a good video idea would be to find out how much power different spec unraid servers use and how much it would cost per year to run, like if I used 1 vm and a few dockers, compared to running 4 vms and 5-10 dockers.

Huh. Actually I noticed that after my laptop’s motherboard died and I started using vm in unraid the server started to draw less power a day (around 2.2KW dropped to about 1.8KW). I have a second P400 for plex, I’ll test power draw in gui mode. Thanks for the video :*

you can look the nvidia P state and put it in P2 state (idle). I’m pretty sure that’s what the driver do in windows.

Now I wonder what ultramangus and firefly is (also poor unraid logo, can't get up to the invader in 2021 ether 😔)

I wonder what the power consumption would be if the GPU is stubbed and not used in a VM or unRAID. Would it be just as low or lower than when the VM is using it.

How are you installing the Nvidia Driver when it is no longer available in the apps store?

👍👍

Just thought I'd add my two cents here. With the Nvidia plugin installed I can't use any of the cards it detects with VMs or I get his error... kernel: NVRM: Attempting to remove minor device 1 with non-zero usage count! The only way I could get my cards working with the VMs was to VFIO them off leaving only one GPU assigned to Emby for transcoding. The concern here was that without a VM running what power state would the cards be in? As they're no longer detected by the Nvidia drivers there was no way of telling (that I know of) but I do have HX1200i PSU which I can monitor power draw within Unraid. So with the two spare cards under VFIO and assigned to VMs the power draw was around 184 watts, with both VMs running the average went up to 194w which is what I expected. Shutting down both VMs resulted in a power draw of 184w again. So it seems that if the cards a reserved under VFIO they will naturally go to an idle power state if the VMs are off. This is what I wanted. In conclusion it appears the high power draw issue seems to be with cards detected with the Nvidia plugin. If nvidia-persistenced isn't set right then they won't throttle back to P8 (idle state). Cards under VFIO seem to throttle correctly as they should. EDIT: The drivers do indeed capture the cards but only if persistence mode is activated. This will enable the cards to toggle their respective power states HOWEVER it means if you try and use them with VMs you'll get a hardware lock. According the Ich777 (the maker of the plugin) it is possible to enable persistence mode per card in the CLI nvidia-smi -i -pm ENABLED Enabled persistence mode for GPU . All done. The should be the HW ID I think (something like 0000:01:00.0). I discovered by cards 'stubbed' by VFIO would indeed power throttle correctly even if the VM was in use or not however according to Ich777 this is very much a card specific behaviour and not all will behave the same as it depends on the cards BIOS and how the manufacture set the thing up. See post here for more info. forums.unraid.net/topic/98978-plugin-nvidia-driver/?do=findComment&comment=962490 In conclusion i'll keep my transcoding card assigned to the Nvidia plugin and the VM cards stubbed in VFIO this way everything power throttles correctly. It does mean my docker options are limited to one card without unbinding from VFIO and restarting but I only use one for emby transcoding anyway.

Has this been improved with 6.9? I saw something about drivers?

👍

Pretty amazing you were able to secure a RTX 3080

Was this problem fixed or is it still in the newest unraid version?

This is really interesting. In a real application, I do wonder if dedicated CPU/RAM will end up offsetting these power wins. Something to test for sure!

My server idles at ~40 Watts and I thought THAT was much for running 24/7...

How did you change your login screen for unraid?

Not the samething, but I decided to down clock my CPU to save on electricity. I turned my old gaming/editing PC into an Unraid server, it's the i9-10980XE 3.0GHZ with a max boost of 4 5GHz with 94GB ram. I downclocked the CPU to 2.5GHZ and turned off XMP. At some point I need to buy a watt meter so test the difference.

You should get a display port or hdmi loop back adapter and retest. They simulate a display being connected and I have used them in other virtualization situations to help with performance

Woooooo

Is there any feedback from limetech about this?

Big sur with unsupported 3080 ????? How its working??????

Abandoned a server build because of this. Every time I stoped my Mac vm, my Vega 56 would draw a lot more power and fans would go to 100%. Maybe need to revisit it with this knowlage. Anyone who knows if the unraid team try and do something about this? instead of us having to do these kinds of workarounds?

Sadly you missed to test with an HDMI dongle to fake a connected monitor. Maybe this helps to save power after shutting down a VM?!

I basically can't use my GPU because the UnRAID Nvidia driver is deprecated. Are there any alternatives.

My UNRAID server draws 200W when Windows VM is at idle.

For nvidia why not just run this command after every aray-startup "nvidia-smi -pm 1" that should put the gpu in power-idle

Space invader please figure me out something. Single GPU with multiple simultaneous VM’s.

"migrating everything to rancher" video?

The latest Nvidia drivers along with 6.9.1 always puts the cards in P8 mode consistently now.

nvidia-smi -pm 1

I save even more power by not being able to buy 3080.

I thought this was common knowledge. I my server i start a win10 vm at start and close it after 3 minutes, just to stop the gpu fans and activate power savings (AMD).