Looking at Swap on Proxmox: Configuration, zswap, zram, swappiness and more.

  Рет қаралды 3,127

ElectronicsWizardry

ElectronicsWizardry

Күн бұрын

In this video I run a few experiments with swap configurations. I also pull from my experience of Proxmox use to see what might be a better configuration for your use case. I run tests on different swap drives, zswap, and zram.
Let me know about other ideas for Proxmox videos in the comments below
00:00 Intro
00:53 What is swap
01:28 Performance Impact of using swap
04:48 Swap on host or guest
05:55 Swappiness
06:43 ZFS ARC
07:28 Testing swappiness and OOM killer
09:02 Real world experiences
10:44 Conclusion

Пікірлер: 35
@lifefromscratch2818
@lifefromscratch2818 26 күн бұрын
Love your videos. You deep dive on technical details that no one else seems to.
@maxg4958
@maxg4958 26 күн бұрын
As sys admins we often do not have the time to play with settings and experiment; hence, why we often end up with over-spec-ed machines just to be sure to be sure. Thank you for your work and analysis, and more so sharing it. Thanks you.
@parl-88
@parl-88 26 күн бұрын
Loving all your PVE and PBS videos! Great Detail, great information. Thanks so much.
@chromerims
@chromerims 25 күн бұрын
👍 One of the best videos I've seen in a while anywhere. Amazing work.
@danwilhelm7214
@danwilhelm7214 26 күн бұрын
Excellent analysis! I was surprised by the results.
@ericneo2
@ericneo2 25 күн бұрын
Very interesting, especially your findings on swap set to 0 or 1. Thanks for the video!
@SB-qm5wg
@SB-qm5wg 25 күн бұрын
Wow, I've never heard of zram. Thank you.
@jothain
@jothain 26 күн бұрын
It was definitely interesting vid. Nice to get some info about how things affect.
@LampJustin
@LampJustin 25 күн бұрын
Also try to starve the VM of RAM as much as possible to be able to use the disk cache on the hypervisor. This will help performance a lot as the cache is shared between all the VMs
@LampJustin
@LampJustin 25 күн бұрын
Some experience from work: Don't ever overcommit, not good you'll have a bad time and if you have to do it, use KSM, the lowest swapiness and try to have as much of the same OSes running in the same version as possible. Then you might be able to do 1.1x -1.3x overcommit safely. But if you don't have to disable swap and never think of it again. It will help you a lot especially on slower drives when live migrating...
@nazar5549
@nazar5549 24 күн бұрын
I think mongo won't start in lxc without overcommit sysctl
@shephusted2714
@shephusted2714 25 күн бұрын
I think the item you sort of glossed over was i/o and implementation of the pci-e based ssd - those units from intel are absolute units and can significantly boost performance thereby mitigating having to go to swap in the first place, also caching can be a big performance boost either with zfs or bcache or a ram drive, also just using bigger arrays/raid10 and stuff like that could end up helping - if you did another video and looked at effects of disk subsytems and the ultimate setup for max perf that would be helpful and put some more metrics on the board - swap is just one component
@ElectronicsWizardry
@ElectronicsWizardry 22 күн бұрын
Yea probably skimmed the 900p being nearly the best swap device out there. I also had a quick graph with a 850 evo as well, and while it was worse, it wasn't that much worse. The effect of ram on caching is interesting and I have some ideas for scripts to test different caching systems, but its hard to test caching is all about trying to predict user workloads, and it can be hard to get consistent numbers that are applicable to the real world.
@kingneutron1
@kingneutron1 22 күн бұрын
Thanks!
@LuisYomba
@LuisYomba 26 күн бұрын
08:11 ..good to know. Thanks
@viruslab1
@viruslab1 25 күн бұрын
cool
@heyah349
@heyah349 26 күн бұрын
🤝
@MrEtoel
@MrEtoel 25 күн бұрын
I have never been able to understand swap in Proxmox. I have a cluster of 3 identical Intel NUCs with 64GB RAM and and they are all below 50% mem usage, but still one node uses 30% swap for some reason - even when it is reporting only using 27% RAM and no VMs are reporting more than 70% mem usage either. Swapiness is 60 on all. No ZFS. Very insightful video - as always.
@mentalplayground
@mentalplayground 26 күн бұрын
KSM what are your hints to use it. I have Ryzen 3900, 48GB of ram with KSM active from 50% of ram usage and it seems to be going fantastic job. Please share your view on the topic.
@ElectronicsWizardry
@ElectronicsWizardry 25 күн бұрын
Thanks for the idea. I should look more into KSM sharing and when/how it works.
@Ray88G
@Ray88G 6 күн бұрын
Would it be possible to set up zRam for installing operating systems on it. . I read in proxmox forum that it's possible but I don't understand how to make it work
@ElectronicsWizardry
@ElectronicsWizardry 5 күн бұрын
Like as a boot drive? Sure you can use that for the boot drive for the VM with a bit of configuration. In general RAM drives aren't worth it for VM boot, but I can make a video looking into it if your interested.
@Andy-fd5fg
@Andy-fd5fg 25 күн бұрын
Its not just on Proxmox this doesn't get talked about.... the subject isn't really talked about in all Linux based systems... and probably not even in other systems either. All to often you see "just set it at the same amount or double you real RAM".
@settlece
@settlece 26 күн бұрын
swappiness
@salat
@salat 26 күн бұрын
8:11 As it's the kernel that does OOM kills - it's not even guaranteed that the Proxmox web service survives, right? I think your zRAM tests are flawed measuring ffmpegs speed only - try swapping windows/programs in a Windows VM and see how the latency feels..
@ElectronicsWizardry
@ElectronicsWizardry 26 күн бұрын
As far as the Proxmox services getting killed by OOM, It looks like linux has a score that the OOM uses to determine what to kill first, and from a quick look at a system the vms would get killed before all the proxmox services from my quick look. I agree my zram tests were flawed, but the issue is snappyness is hard to measure well. Let me know if you know a good way to measure snappyness and I'll give it a run, but since I couldn't really think of a better test I stuck with the ffmpeg encode. I'm sure zram has its uses as its the default on many distros.
@autohmae
@autohmae 26 күн бұрын
8:20 it's not proxmox which kills it, it's the Linux kernel.
@autohmae
@autohmae 26 күн бұрын
11:42 I would not recommend a low swappiness setting and waiting for swap being used, keep it higher, because you are reducing performance because you loose things like file system cache.
@ElectronicsWizardry
@ElectronicsWizardry 26 күн бұрын
My mindset was that a low swappyness will be best if you normally have enough RAM, and want to make sure there aren't performance hits if a part of a VM gets swapped out and then needed later on. The swap would then keep the OOM killer from taking out VMs, and be an indicator if more RAM is needed. If you plan on using swap, then a higher swappiness value would be better as its proactively swapping out unused ram instead of waiting for ram usage to be high. This really depends on the usecase of the system and VMs. I often seen Proxmox servers where RAM usage isn't that high, so my thought was avoid performance hits by having parts of VMs in swap and the smaller disk cache shoudn't hurt performance that much, but again really depends on the use case. Hope this explains some of my thoughts in the video.
@autohmae
@autohmae 26 күн бұрын
@@ElectronicsWizardry "If you plan on using swap, then a lower swappiness value would be better" OK, agree. I guess I should have thought about the title of the video again. 🙂
@ElectronicsWizardry
@ElectronicsWizardry 26 күн бұрын
Oops that should have been higher value to proactively swap out ram. I edited the earlier comment.
@ewenchan1239
@ewenchan1239 22 күн бұрын
So...the thing with swap REALLY depends on what you're doing. On my main Proxmox server at home (which has 256 GB of RAM), I think that I am running something like 17 LXC containers (with almost all of them being Ubuntu), 1 WinXP VM, 1 Win7 VM, 2 Win10 VMs, and 2 Ubuntu VMs. It uses somewhere between 150-180 GB of RAM nominally. That SHOULD leave me with ~56 GB of RAM free, right? And yet, despite this, my Proxmox server WILL swap out before using the remaining ~56 GB of RAM. Always. Without fail. swappiness is set to 500, which means that it SHOULDN'T swap out to disk unless it ABSOLUTELY has to. Between ZFS ARC and virtio-fs, some combination of things is causing the system where it will NEVER be allowed to use the remaining 56 GB of free (and available) RAM. Moral of the story: RAM, for VMs and CTs ended up being the most important hardware choice when picking a virtualisation server.
@ElectronicsWizardry
@ElectronicsWizardry 22 күн бұрын
I'd check that 500 swappiness value. I think swappiness should max out at 200. And larger values typically mean swap will be used more. I'm curious how big you have your ARC set to.
@ewenchan1239
@ewenchan1239 22 күн бұрын
@@ElectronicsWizardry *edit* I stand corrected. I just checked my system (couldn't do it earlier whilst I was at work). Swappiness is set to 10 on my main Proxmox "do it all" server. It was vm.vfs_cache_pressure that I have set to 500. "vfs_cache_pressure This percentage value controls the tendency of the kernel to reclaim the memory which is used for caching of directory and inode objects. At the default value of vfs_cache_pressure=100 the kernel will attempt to reclaim dentries and inodes at a “fair” rate with respect to pagecache and swapcache reclaim. Decreasing vfs_cache_pressure causes the kernel to prefer to retain dentry and inode caches. When vfs_cache_pressure=0, the kernel will never reclaim dentries and inodes due to memory pressure and this can easily lead to out-of-memory conditions. Increasing vfs_cache_pressure beyond 100 causes the kernel to prefer to reclaim dentries and inodes. Increasing vfs_cache_pressure significantly beyond 100 may have negative performance impact. Reclaim code needs to take various locks to find freeable directory and inode objects. With vfs_cache_pressure=1000, it will look for ten times more freeable objects than there are." (Source: www.kernel.org/doc/html/v5.4/admin-guide/sysctl/vm.html) Either way, even with a vm.swappiness=10, my system still swaps out like crazy despite the fact that I have 56 GB of RAM free.
Trying out 40GBe, Does It Make Sense In A Homelab?
10:03
ElectronicsWizardry
Рет қаралды 4,1 М.
Running a NAS on Proxmox, Different Methods and What to Know
18:47
ElectronicsWizardry
Рет қаралды 28 М.
Викторина от МАМЫ 🆘 | WICSUR #shorts
00:58
Бискас
Рет қаралды 4,3 МЛН
ПРОВЕРИЛ АРБУЗЫ #shorts
00:34
Паша Осадчий
Рет қаралды 7 МЛН
Proxmox Backup Server Saves You Money And Time!
24:48
Jim's Garage
Рет қаралды 47 М.
Monitor your Proxmox Cluster with Grafana & InfluxDB!
17:10
SelfHostedToast
Рет қаралды 345
Protect Your Home-Lab using Proxmox SDN and Firewall
38:59
LincPlus LincStation N1 Review: All SSD NAS with unRAID
18:25
ElectronicsWizardry
Рет қаралды 1,5 М.
This homelab setup is my favorite one yet.
21:30
Dreams of Autonomy
Рет қаралды 131 М.
A SERIOUS Home Server That's Affordable
18:59
Hardware Haven
Рет қаралды 132 М.
What to do with a Degraded ZFS Pool
13:28
Craft Computing
Рет қаралды 36 М.
So if Hardware RAID is dead... then what?
29:00
Level1Techs
Рет қаралды 144 М.
Сколько реально стоит ПК Величайшего?
0:37
Что делать если в телефон попала вода?
0:17
Лена Тропоцел
Рет қаралды 3,2 МЛН
8 Товаров с Алиэкспресс, о которых ты мог и не знать!
49:47
РасПаковка ДваПаковка
Рет қаралды 171 М.