How To Get the Most From Your TrueNAS Scale: Arc Memory Cache Tuning

Рет қаралды 31,096

Күн бұрын

lawrence.video/truenas
ZFS Arc Value Location
/sys/module/zfs/parameters/zfs_arc_max
Bytes converter
whatsabyte.com/P1/byteconvert...
TrueNAS Scale Half Used Memory Issue
ixsystems.atlassian.net/brows...
Deeper dive into Linux Memory Management
blogs.oracle.com/linux/post/l...
Connecting With Us
---------------------------------------------------
+ Hire Us For A Project: lawrencesystems.com/hire-us/
+ Tom Twitter 🐦 / tomlawrencetech
+ Our Web Site www.lawrencesystems.com/
+ Our Forums forums.lawrencesystems.com/
+ Instagram / lawrencesystems
+ Facebook / lawrencesystems
+ GitHub github.com/lawrencesystems/
+ Discord / discord
Lawrence Systems Shirts and Swag
---------------------------------------------------
►👕 lawrence.video/swag/
AFFILIATES & REFERRAL LINKS
---------------------------------------------------
Amazon Affiliate Store
🛒 www.amazon.com/shop/lawrences...
UniFi Affiliate Link
🛒 store.ui.com?a_aid=LTS
All Of Our Affiliates that help us out and can get you discounts!
🛒 lawrencesystems.com/partners-...
Gear we use on Kit
🛒 kit.co/lawrencesystems
Use OfferCode LTSERVICES to get 10% off your order at
🛒 www.techsupplydirect.com?aff=2
Digital Ocean Offer Code
🛒 m.do.co/c/85de8d181725
HostiFi UniFi Cloud Hosting Service
🛒 hostifi.net/?via=lawrencesystems
Protect you privacy with a VPN from Private Internet Access
🛒 www.privateinternetaccess.com...
Patreon
💰 / lawrencesystems
⏱️Time Stamps ⏱️
00:00 TrueNAS Scale Setting ZFS Arc Cache Memory
02:00 TrueNAS Scale Memory Usage
02:48 How To Set zfs_arc_max
#truenas #storage #zfs

Пікірлер: 114

@TechnoTim 6 ай бұрын

Thank you Tom! Great video!

@danknemez 10 ай бұрын

Thank you! Now I can just point people to a very well put together and comprehensive video when this topic comes up - especially because you very clearly outlined the potential issues you may run into!

@travisaugustine7264 10 ай бұрын

I can't believe this is something that isn't exposed in the GUI, especially considering ixsystems touts the application and virtualization abilities of Scale.

@johndroyson7921 10 ай бұрын

Scale is a bit less mature than core still.

@rawa9891 9 ай бұрын

Perfect! Just changed over from Truenas Core yesterday, and your tip helped greatly! Thanks! 😊

@peteryuen121292 10 ай бұрын

Thanks, Tom The memory shortage was a pain for me Now I am feeling way chillier about it Love the guild you provided Awesome

@xazrael 8 ай бұрын

I feel like I truly lucked out venturing forth in July and August of 2023 to initiate and build our TrueNAS home video and photo archives server like I did. Videos like THIS in PARTICULAR helped me shave the one thing that annoyed me about how much oomph I was getting out of what I invested in particularly. Thank you so very much.

@mhos5730 10 ай бұрын

Thanks Tom for posting this. While memory for my potato NAS is cheap, I'd like to get the most bang for my buck.

@Yuriel1981 10 ай бұрын

Thanks Tom! You really are one of the best Scale resources for the community. Love the hair btw. And hope you are still healing well.

@stanislavtodorov8705 4 ай бұрын

Indeed, Tom is my go to tutor for Truenas Scale and PfSense.

@MaxKulik 4 ай бұрын

Thanks for going over this! It was super helpful!

@gugajedi 10 ай бұрын

Thanks Tom! You are the best! Fixed here for me, awesome, now I can run some VM's again...

@jorgelemaire4506 9 ай бұрын

Incredible, I was trying to find a way to limit the memory consumption, thank you Tom

@terrylyn 5 ай бұрын

I was wondering why ARC filled only half of my memory, glad I found this explanation. However I think this setting should be accessible from TrueNAS GUI.

@zlatko87 10 ай бұрын

Some Core features we take for granted... tnx for info and solution 👍👍

@TeflonBilly426 10 ай бұрын

Thanks for this video, I just recently upgraded my home NAS to 512GB (prices have really dropped on 64GB DIMMS recently) and I also moved to TrueNAS Scale.

@mr_jarble 10 ай бұрын

Same boat as you just jumped to scale with my server and was only using 256GB of arc

@DragonReborn100 10 ай бұрын

Another great guide, Thanks mate! Like to use all i have purchased, as in RAM. Dont use a load of apps, just a few, so left about 6Gb free for other things. Think thats sensible.

@TinyWorkshop 10 ай бұрын

thanks finally a simple explanation :)

@LAWRENCESYSTEMS 10 ай бұрын

Glad it helped!

@VORTEXJOGOS 8 ай бұрын

THANK YOU SO MUCH!

@porcuzburator8778 6 ай бұрын

Thank you!

@SPLPuroOne 10 ай бұрын

Worked really well, I don’t know why Truenas defaults using full of ram for ZFS caching

@mr_jarble 10 ай бұрын

Thankyou so much

@sanderdelft 10 ай бұрын

Cool. Wasn't aware this was even a thing. I would be interested in a follow-up on the scale versus core as of 2023. Lots of videos date back from launch date and a lot has changed since. What would be the recommendation for a variety of standard use cases today?

@LAWRENCESYSTEMS 10 ай бұрын

It's become more stable and more tuned in terms of performance, maybe it needs a new video once the new version of scale come out soonish.

@sanderdelft 10 ай бұрын

@LAWRENCESYSTEMS thanks for the consideration. I would eat it up. Even as a short.

@nixxblikka 10 ай бұрын

@@LAWRENCESYSTEMSI like the effort and quality you put in your videos, would wait until February when the new version is actually stable

@Franchyze923 6 ай бұрын

thanks!

@ZiggyDaZigster Ай бұрын

I'm confused mine uses all but 20 gigs all the time, and I never set anything up? I'm on TrueNAS Scale Dragonfish-24.04.0 - Currently 377.8GiB total available (ECC) - Free: 21.3 GiB Was something added recently to auto expand past 50% of the memory capacity?

@king1fspades 4 ай бұрын

Can anyone advise if this is still a problem in the latest Cobia build? TrueNAS-SCALE-23.10.1.3 I've heard they fixed some issues with ARC metadata caching recently and didn't know if this was also addressed.

@NickyNiclas 10 ай бұрын

The way it seems to work currently is when I start a VM it just falls back to the 50% limit no matter what. I've been testing this for over 6 months with 90/128G and have never ended up in a crash scenario.

@marcogenovesi8570 10 ай бұрын

There is probably some logic in Scale that writes that value when you start the VM, this does not happen in Proxmox or other generic Linux system. The value you set sticks.

@-szega 9 ай бұрын

1:10 I don't really agree with this assessment, because ARC memory is not swappable on FreeBSD (it's Wired, and it shouldn't be swapped out), but this also gives it an ugly tendency to push text pages out to swap. That's kinda okay if swap is on dedicated SSDs, but TrueNAS CORE will put the swap - by default - on the main pool devices. So by default you can easily end up with CORE aggressively swapping to the pool devices, and then system responsiveness goes down the drain as swap I/O competes with pool I/O. I was wondering on my CORE setup why I experienced these sudden, dramatic slowdowns of share performance, and well, it was ARC Wiring up all the memory.

@mattvisaggio 10 ай бұрын

After I did this to expand the arc, it seems like the memory in the main dashboard is nearly completely free and not used in ZFS any more. Any ideas? I returned to 0 and removed the init script, but it still shows almost no cache used when before it was about 115 gb or so. However, the system functioned fine before and after I made this change. The only thing impacted is/was the dashboard in truenas showing almost no memory being used for zfs.

@jttech44 7 ай бұрын

When you reboot, ARC is purged, so, you'll have to use it for awhile in order to fill the ARC again.

@DavidAshwell 10 ай бұрын

Would 45 Drives's Rocky Linux / Houston system also be capped at 50% by default? This is a general Linux behavior with ZFS, correct?

@marcogenovesi8570 10 ай бұрын

yes this is default ZFS on linux behaviour. On a more generic Linux system (and Proxmox) what he did is done by creating/editing in "/etc/modprobe.d/zfs.conf" file with "options zfs zfs_arc_max={your_number}" and then rebuilding initrd images so this config is loaded on boot.

@ChicagoMed2000 10 ай бұрын

As shown in the video it appears you need root access to do this. For those of us with root disabled, sudo doesn't appear to work. Is there a workaround?

@ChicagoMed2000 10 ай бұрын

To answer my own question in case anyone has the same issue: found on another forum, use sudo sh -c. Tom please correct me if I'm wrong here but this seems to have worked for me. Thank you!

@donwald3436 5 ай бұрын

lol my freenas locked up until I manually capped the arc, it was in some boot param bullshit flash area too.

@yassinezara9641 10 ай бұрын

Thanks Tom for this tips, I'm wondering 🤔 which value I set ? I have 64GB.

@LAWRENCESYSTEMS 10 ай бұрын

Set it as much as you want to use for cache while allowing enough for services

@ShOookYx 5 ай бұрын

may i know even how to open the comand windows ?

@urzalukaskubicek9690 5 ай бұрын

I had problem with persmission: zsh: permission denied: /sys/module/zfs/parameters/zfs_arc_max and had to use this workaround: sudo sh -c 'echo NEW_ARC_SIZE > /sys/module/zfs/parameters/zfs_arc_max'

@nadpro16 3 ай бұрын

I am logged in as admin, and even if I use sudo I keep getting zsh: permission denied

@nadpro16 10 күн бұрын

for anyone else finding this issue, this is how you have to put in the command (note: this is for 100gb of ram) sudo sh -c 'echo 107374182400 >> /sys/module/zfs/parameters/zfs_arc_max'

@bokami3445 Ай бұрын

Question; I have a 40TB Raidz2 on a system with 32GB ram. After setting the zfs_arc_max to 27GB (only samba on this system), running a scrub. I left it for a few hours running, when I checked on it's progress, the arc cache had consumed almost a 32GB (less samba). If I had left it alone, I'm sure oom would have kicked in. Fortunately I was able to lower the zfs_arc_max value and complete the scrub. The question is why would the scrub not honor the zfs_arc_max value?

@LAWRENCESYSTEMS Ай бұрын

Not sure, but this video will soon be sunset since Dragonfish changes all of this.

@bokami3445 Ай бұрын

@@LAWRENCESYSTEMS :-) Something to look forward to. Thanks

@zhenhaochen8170 Ай бұрын

@@LAWRENCESYSTEMS After upgrading to Dragonfish, memory cache would use up to 50G of 64G in my system, which could be troublesome at times because there is not enough memory left for my VM as other services use about 6G and there is only 3-5G left. When I try to start up a VM, there will be a warning about memory overcommitment. If I proceed anyway, the VM would start without issue and the memory cache would be reduced automatically by the system. Is that something I should be worried about?

@LAWRENCESYSTEMS Ай бұрын

@@zhenhaochen8170 It only uses the memory that is free so start those services before the memory is used and then you don't have to worry about it.

@NecromancerGarage 10 ай бұрын

Could you do a video on tuning TrueNAS on an all flash system to use the speed efficiently?

@LAWRENCESYSTEMS 10 ай бұрын

Nothing to tune besides that setting.

@JohnnyLai 10 ай бұрын

Can you make more video regard s1, huntress and ninjaone?

@neosmith80 10 ай бұрын

so, how would this be done in core?

@LAWRENCESYSTEMS 10 ай бұрын

As I said in the video, not needed in FreeBSD/TrueNAS Core.

@neosmith80 10 ай бұрын

@@LAWRENCESYSTEMS sorry, i missed that part. But also, good to know! :)

@yusky03 7 ай бұрын

While I have 128GB of RAM I have 46GB used in services. So at least I know now that if I deploy some VMs with only 17GB of RAM free the cache isn't going to automatically scale down like in Core.

@ryanmalone2681 25 күн бұрын

So is this solved now? I just installed TN in Proxmox and also a baremetal install, and both had 500G of RAM and TN consumed >95% RAM and also gave it back when I was starting other VMs where I allocated 64G and 128G RAM.

@LAWRENCESYSTEMS 25 күн бұрын

Yes, it's solved in the latest version.

@ryanmalone2681 25 күн бұрын

@@LAWRENCESYSTEMS Sweet dude. Totally sweet.

@paoloposo 7 ай бұрын

That value converter website is pretty weird. I noticed TrueNAS shows RAM utilization in GiB (base 2), not GB (base 10). Your manual calculation using the factor 2^30 is in line with that and yields the same result the website gives. However, the website calls 1024 bytes a kB instead of a KiB. On the same page they state that "kilobyte" has different meanings depending on whether you are in a binary or a decimal system. That's plain wrong. A kB is ALWAYS equal to 10^3 bytes. It's just that many people tend to confuse the binary and the decimal prefixes. They should have explained that instead of further contributing to this misunderstanding. Also the conversion table on that page claims that a kB is 100^1 B, but that's probably just a typo. I'm sure (I hope!) they meant to say 1000^1.

@WillFuI 5 ай бұрын

I wish we could like have options to have like 10-90% in 5% increments. I would be fine with that solution

@DangoNetwork 10 ай бұрын

I want to know how to fix share permission. Only owner and onwer group seem to be working.

@LAWRENCESYSTEMS 10 ай бұрын

kzfaq.info/get/bejne/a59-erGLldGzYGg.html

@justisstudding2101 3 ай бұрын

Fixed: Unfortunately this is still relevant in Dragonfish 24.04-RC.1 Almost 3/4 of my memory was locked into ZFS Arc Cache. Giving me no room. Thanks. I have 32gb of Ram in my System what would you recommend for a ZFS Arc Cache? I have it set now for 4gb.

@justisstudding2101 3 ай бұрын

When I did the "cat /sys/module/zfs/parameters/zfs_arc_max" I didn't have a 0. I had a locked in amount of 30 something Gb.

@justisstudding2101 3 ай бұрын

After doing: sudo sh -c 'echo 4294967296 > /sys/module/zfs/parameters/zfs_arc_max' I entered Admin Pass - Hit enter Checked ZFS Cache Size sudo sh -c 'cat /sys/module/zfs/parameters/zfs_arc_max' 4294967296 Went to System Settings - Advanced - I removed the Init script command - rebooted Checked ZFS Cache sudo sh -c 'cat /sys/module/zfs/parameters/zfs_arc_max' I entered Admin Pass - Hit Enter - Returned 0. Went to Dashboard - Memory, and it's no longer locked in an insane 3/4 ZFS Cache memory.

@christopherjackson2157 10 ай бұрын

Yes thank you this is so annoying

@benardmensah7688 10 ай бұрын

So what of tunables we had in core?

@LAWRENCESYSTEMS 10 ай бұрын

What about them?

@marcogenovesi8570 10 ай бұрын

not needed in Core, only in Scale

@zparihar 10 ай бұрын

Setting swap is a good way from preventing VM crashing. But should be low swappiness

@LAWRENCESYSTEMS 10 ай бұрын

I doubt it would be persistent between upgrades which is why you set it in the UI.

@kwinzman 10 ай бұрын

If I want the NAS to primarily work as a NAS I am probably happy if the ARC uses 50% of my memory, even on a 256GB RAM server. Right?

@LAWRENCESYSTEMS 10 ай бұрын

I set it to use AS MUCH AS POSSIBLE as that is a big speed boost.

@sophware 10 ай бұрын

I use mine as a NAS-only system--no VMs, no containers. My thought is the opposite of yours @kwinzman. If the only value of the RAM is for NAS, why not use almost all the RAM for NAS (including ARC). No need to have half of your RAM just sitting there doing nothing, reserved for something you're explicitly not interested in. Now, my question for myself is: why am I not just using CORE instead of SCALE, since I don't need or want the unRAID-like stuff SCALE offers and the NAS-specific stuff SCALE offers isn't mature and/ or I don't have the hardware for it? Time to watch the new video on SCALE vs CORE!

@l3l4ckR4sp3rry2 10 ай бұрын

So Tom says there is a risk of crashing when using up too much RAM with services when arc_max is set manually. Does this also apply to the default setting? I thought scales arc would behave more like Tom said about core - use free space and give it back when services need it.

@marcogenovesi8570 10 ай бұрын

the actual issue here is that on Linux the ZFS module is often too slow to let go of the memory when there is memory pressure, so this triggers an OOM that will start to kill OS processes. So yeah you can have that issue on default settings too if your application need more than half the RAM. But it's the middle point between safety and performance so that's why it's a default. If you choose a manual limit then it's down on you to choose wisely and know how much RAM the applications on the NAS require.

@l3l4ckR4sp3rry2 10 ай бұрын

@@marcogenovesi8570 Thanks for clarification! Hopefully they will find a soultion for this. Better sooner than later.

@AfsanehAbbasnezhadianghashghae 10 ай бұрын

I applied it to my system and it started using it immediately, so you may get away without a restart if you have work to do

@marcogenovesi8570 10 ай бұрын

yes the command from console will change the setting immediately but it's not kept on reboot. To run it automatically on reboot you have to add the startup script as shown in the video

@minifig404 10 ай бұрын

I'm surprised there isn't a "tunables" spot in the UI like there is with TrueNAS Core. Ah well.

@minifig404 10 ай бұрын

Oh, there's a spot for sysctl variables (System -> Advanced, left column), but this isn't a sysctl var, is it?

@LAWRENCESYSTEMS 10 ай бұрын

I don't think so.

@littlenewton6 10 ай бұрын

No. I think the "When" of this command should be PreInit rather than PostInit.

@LAWRENCESYSTEMS 10 ай бұрын

Why?

@Prophes0r 10 ай бұрын

The value is live. You can run the command in the terminal and the new value will take effect immediately. As long as it gets run before your ARC hits the default 50% high-water, it will be fine.

@marcogenovesi8570 10 ай бұрын

does not matter. ZFS module is loaded before Preinit anyway and you can do that command at any time you want

@marchagen3893 5 ай бұрын

Will be patched in 24.04 (Dragonfish) ! See the Version Notes

@LAWRENCESYSTEMS 5 ай бұрын

Yup

@peterworsley 10 ай бұрын

What about how to get different IP addresses from DHCP like trueas non scale

@gh8447 10 ай бұрын

Do you mean TrueNAS Core?

@peterworsley 10 ай бұрын

@@gh8447 I have used scale but all apps used the same IP address

@imadecoy. 10 ай бұрын

If you want it from DHCP I'm pretty sure that's on the router side. That's how I did it anyway, static mapping in pfsense

@Asdf-jy5ts 2 ай бұрын

TrueNAS Scale is WEIRD. It's neither for home use nor for enterprise use. They really should pick a lane.

@timskiduh2416 10 ай бұрын

the only think i dont understant why he put 2gb on web site and not 8gb if truenas have 8gb memory. i put 32gb because i have 32gb memory but im not sure if i do good im french and not all good understanding

@mr_jarble 10 ай бұрын

You want to leave room for applications setting it to the max of your system will not work out well. Drop the number down into the 20's.

@timskiduh2416 10 ай бұрын

@@mr_jarble ok thanks for answer

@zparihar 10 ай бұрын

Tom, on Debian systems you can set the etc/modprobe.d/zfs.conf. Is that persistent and TrueNAS scale?

@vladkarpenko2649 10 ай бұрын

In Proxmox (uses Linux kernel so it should be the same) there is a hint in documentation if you need to set zfs_arc_size permanently you can set it in "/etc/modprobe.d/zfs.conf" file with "options zfs zfs_arc_max={your_nubmer}" string. Isn't it the case for Truenas aswell?

@LAWRENCESYSTEMS 10 ай бұрын

Because TrueNAS scale is built more like an appliance you should always use the UI for updating settings so it will survive updates.

@Prophes0r 10 ай бұрын

@@LAWRENCESYSTEMS TrueNAS also keeps the values in a different spot. TrueNAS - /sys/module/zfs/parameters/zfs_arc_max

@marcogenovesi8570 10 ай бұрын

@@Prophes0r that file is available on all linux systems running zfs too. It's two ways of controlling the value. One with the modprobe command on start of the module or online by writing a number in bytes in that file

@Prophes0r 10 ай бұрын

@@marcogenovesi8570Hrm... I remember when I was first trying to change these values I looked at the official OpenZFS documentation and couldn't figure out why the settings weren't taking. It took a few days to realize that TrueNAS was storing values in a different place. I could have sworn it was these values...