But, he did ... Didn't he? He used Crystal Disk Mark at 11:29. That's very useful in having a basic idea of how the drives / storage device will perform.

I would be interested in the performance boost in his TrueNAS machine. ...also why use Proxmox and a TrueNAS VM vs TrueNAS Scale on bare metal, which has KVM for any other VMs?

Yes exactly, what is the performance improvement ?, my guess is there is no improvement. If you use ZFS in a synchronous more safe mode of writing data, which is a slower but more secure way of writing data, only in that case will a ZFS log cache increase performance. But it will always be less fast than using the less secure way of writing data to your ZFS pools. Lawrence mentions this in some of his tutorials.

@@bananafartmanmd8775 any motherboard that supports PCIe bifurcation will work (splits the 16x slot into 4x4pcie lanes there is zero logic in this card it's just directly wired to 4 lanes of the slot) nothing to do with Intel vRoc

@UC7InT0ngXR50dhyDDtX0oJg it's a China product so they say what ever they want on the label, this is purely a simple direct to 4x4pcie Lane card only requirement is bifurcation support for the bios to split it into 4x4 lanes (it will work with vRoc but not a requirement) this card has no smarts and it's not an "IT mode" card as he stated in the video, IT mode would be HBA card or has a bridge chip of some sort

Yeah I need to read more into ZFS for sure.

I also think that 512GB slog device is insanely large. I think L2ARC might provide a benefit as only 64 Gb of RAM are available over a +24TB pool. But not sure, would be cool to see some tests !

@@KifKroker SLOG is only good for certain scenarios like NFS shares and SLOG SSD need power protection on them and be able to take massive write endurance, as in Optane, desktops SSDS are shit for SLOG.

Update the BIOS on your drives it may reduce that problem in the future.

hahaha you never did chia plotting on nvme? then you really gonna complain ;) 60% health left per nvme after +300TB writes.

Likely 0, because unless your ARC (ram) is full, it wont use L2ARC at all.

as a general rule of thumb: always max out the ram of a truenas system before you start adding cache layers. btw: this video is not very good (if ever) researched and has some details wrong or explains them in a not suffficent way. truenas caching is a very deep topic and you will not see much benefits (or even loose performance) if you "just add some nvme ssds" to a system, with the possibility of severely hurting data integrity on top.

Yes your motherboard needs to support 4x4x4x4 bifurcation.

@@RaidOwl Sorry I missed that part in your video. Great video ! It just gets me when you say PCI instead of PCIe haha. I remember PCI 100/133 mhz slots they rocked ! White slots for the win !

For most people that’s perfectly fine.

They always do that, still, lots of people run their NAS virtualized and experience no problems with them whatsoever.

@@BoraHorzaGobuchul Since I did the same and there is no issue so far. The only stability issue was related a power cable. Also I am looking for a process to convert raid5 to raid6 (of course this is called raid z1 and raid z2 if I am correct) in zfs by adding an additional hdd. So far I read this is not possible...

@@zhanko73 it isn't, afaik. There's talk of done new expansion functionality in the works but it's been a long time and still no news there. The only option I know of is creating a new vdev and migrating data to it

Surprising because I’ve seen pretty high speeds in Windows. I know Windows file transfer is single threaded, but I don’t think that’s the bottleneck here. Does it have something to do with the striping? Either way I’ll give it a shot on one of my Ubuntu VMs, thanks!

@@RaidOwl I mean maybe you are right but tell me the results also lines talked about it in the honey badger review you can see it maybe you will find a solution

For sure, I’m curious now. I didn’t really expect amazing results out of a $25 card. I’ll let you know what I find!

@@RaidOwl thanks

Get them all squeaky clean

Yeah it’s not optimal but it sounds cool. It’s certainly dependent on each persons use case.

2 KB of RAM...oh my

Yeah I was close to buying the ASUS one too, but had to try out the AliExpress version first haha.

I don't think that's necessary ...

If your running TrueNAS more ram is always #1 before you go with an L2ARC (cache drive)

Hey, thanks for the super informative post. I am not super well versed in storage tech but I've been learning. I assumed that an l2arc cache was only necessary if you're running out of RAM space for ZFS. I have read so many posts about why you should/shouldn't have an slog/l2arc cache and there never seems to be a general consensus. Again, thanks for your informative post; I appreciate it!

​@@RaidOwl You are very welcome. I too have been reading forum up and down, in regards to ZFS, and it can be quite difficult to wrap your head around, although, simple ones you grasp it. The issue at times, seems to be that the information gets picked up and paid forward, without fully understanding it( again, how would one know, that they fully understand it ). I am somewhat getting confident in how the ZFS system works. It is true, that ram is the first cache arc. but it will only hold reasent data in memory, and it is fast as hell, but as you copy more data to the NAS, the cache in memory gets discarded, so when you need it at a later time, it no longer exists in cache. That's where the L2Cache kicks in. It retains a copy of reasent written and accessed data, so a 1TB l2arc, would give you NVME timing and speed, of the latest 1TB data that has been accessed, and it gets a new timestamp every time you access it. So a game you play on a daily or weekly basis, will always load from the l2Arc. Also, l2arc is now persistent, so unlike the main ram arc, l2arc cache is available immediately after reboot, and does not need to build up again. The slog is only for writing log info, for synced transfers, which slows down the transfer, as it has to write that to disk during "copy on write sync". Sync is not allowed to write this log to memory, as this would mark the data as "dirty" (data that is not yet written to and logged to disk) this is a security measure, to ensure you dont get inflight data corruption. For home use, turning on sync entirely will give you more performance, slog or no-slog. No sync, does not put your data in danger, only inflight data, which will get lost, just like when pulling the plug on your computer, it will be just fine, but you may have lost the last minute change. Metadata VDEV, is the new cache, this will really speed up the NAS, as what mostly slows down a NAS, is small files, databases and indexing. So splitting the data into classes: small files and big files, putting the small on SSD, and the big files on HDD is quite clever, as the small files, usually only take up 0.1 - 0.5 percent of the total storage, though they are many and slow for HDDs to transfer. Large files tranfer quite well on the HDDs, especially in a raid. Hope that shed some light on it. ( I base my conclusion on study and thorough testing )

Would you say that an l2arc is unnecessary if you have adequate RAM space? For example, if my RAM fills up and some files are loaded into the l2arc, would those files be hindered once the RAM space clears back up (since they are 'stuck' on l2arc instead of being able to be reloaded into RAM)? This sounds like a strange case that may come up with transferring a large amount of medium-ish sized files back and forth (maybe editing phots/video). The Metadta VDEV sounds cool and I'll definitely look into that on my main TrueNAS system.

@@RaidOwl No. It is a 1-2-3 priority. If it is not available in Ram, it will look to the l2arc, and then to the raid. I have 64gb Ram for my ZFS, and my arc gets hit from time to time, but mostly in loading samples for music production. For my storage of general data, it does next to nothing for me. Yes, the Metadata vdev, is closer to what most people want from a cache, while still retaining the ZFS file system features. Also, the Metadata vdev will render slog somewhat obsolete, as the reason that sync slows down the the write, is that it need to wrote a lot of small asynchronous data to disk. And this will go directly to SSD automatically. However, it will not replace the l2arc, as that is ready to use cache, of both big and small files, and it does not need to wait for the disks to spin up, or seek within big files. however, with meta vdev and sufficient ram, you will not feel much of a difference, and it will work for the whole pool, regardless of reboot, and amount of ram. Just be sure to make it safe. At least 3 mirrors. and in your case, make the last one a hot spare. You could put it like this. Arc(ram) is for workspace, it is for the currently active cache, and it is absolutely essential for VM's. l2arc, is for loading/copying/accessing older data. Metadata, will be a constant cache for small files, which are a nightmare for HDD's

Okay yeah I'm in the same boat there with 64 gigs of RAM. I think ill throw a Metadata vdev in there and monitor the changes in different use cases.

PCIe*

@@RaidOwl Why on earth did you refer to it as PCI throughout the video and in the title? Are you not aware that PCI exists and is meaningfully distinct?

​@@rileybaker8294 Thanks for watching!

PCIe. Not much consumer stuff these days is PCI.

Unless you have a threadripper, you'll have a loss

Depending on what you’re doing, sure

Yeah these cheap AliExpress cards are definitely hit or miss.

I tried something similar with the Asus on my X570, but it didn’t work because you need bifurcation and because the Ryzen platform is really limited in lanes. I’m working on an epyc naples file server, and trying again - hopefully will have better results.

These are cache drives not storage drives. If you are watching a video about TrueNAS I am assuming you know the pros/cons vs HDDs and SSDs.

Yeah that makes sense lol

For dev stuff

I know…it’s sad

Cool 👍

I pour the milk before the cereal too

@@RaidOwl You are going to far. :)

Lame :/

I remember 95

Yeah I can’t believe it went up so much.

很好！

I agree, dude sucks

Yeah dude sucks