Sorry about the mistake by saying 5.25" drives! While researching and testing, I was trying to figure out how many drives I could fit in the Corsair's 5.25" bays and somehow that got into my script. 🤦‍♂ In the spirit of mixing things up, let me know what you've mixed up before!

I, also, hate using 5.25" hard drives. Such a pain ;)

Hey Tim, I’m hard of hearing, but I just want to say thank you for your time and effort into adding subtitle😊

Your public library might have a 3D printer if you don't want to purchase one. I use my library's printer often.

For an ultimate all-in-one homelab server, a hypervisor whiteout even thinking. One solution (nearly) fits all.

Great video. Some of your inflections remind me of LGR. I was chuckling.

5 1/4 DRIVES?? They're 3.5 inch!

Also, thanks for the info on the Sliger cases. You just made me spend more money, they look great and are made in the USA for a reasonable price. My Threadripper platform is getting a new home. :)

Yes! Thanks for the change in content

Awesome stuff, thanks for sharing!

I’ve run for years an unRaid server which had a w10 VM with GPU and NVME passthrough that I used for playing games and the rest of the system was used for docker stuff: Plex, arr suite, homeassistant and a large etc. Just make sure you have a Renesas chip based USB PCIe card passed to the VM so you can plug and unplug peripherals without freezing the VM.

15:18 When racking servers by myself, I've found that there's usually holes in both the sliding part of the rail in the rack, and the stationary part of the rail (the portion that attaches to the rack itself). Every rail is different, so it always takes some experimentation, but I've found that I can put a spare screw/toothpick/pointy-thing through both holes so that sliding part of the rail doesn't push back while I get things lined up. I do this for both sides but sticking out different amounts so I can line things up one side at a time. Just be sure that the holes you pick in the rail can be reached from the front of the rack.

I moved to a single giant server build a while back from my own giant rack with dell power edge servers. But it was to much power usage. For most use cases in a single server build. I found unraid to be the best base OS for me. 5 years later. Still rock solid and never had any major issues. Kind of on autopilot and it just works.

Proxmox for the win! No reason why, I just love it.

Don’t go with the EVO 870’s, I made the same mistake (They are consumer drives and wear out quickly!) I replaced all of them with the Samsung SM883’s (ZFS pool)

Same here. 😁👍 Taking a play from my corporate sys admin world. Separate storage and compute boxes. Going to build a TrueNAS Scale box as the central storage for the entire homelab. Then use the 2nd unRAID lic to build a fresh compute server. Both will have 10Gbe until i can swap for fiber.

Thanks Tim.

Hey Tim, great build can't wait to see how that works out, Always looking for projects like this. I ended up with the iStarUSA D-410-DE36 case that allows for 36 drives a while back for the hotswap trays.Paired it with NORCO RPC-4224 4U with uses 24 3.5in drives. Purpose was to run Flash NAS and backup to spinning disk NAS. If you end up making 3d printed drive holders for this project, I would love see that adventure.

I went bear metal on my home lab server for a while, because it could do everything I wanted. However, things changed and now I’ve reinstalled everything on proxmox. The overhead is low and you have the flexibility anything in the future. So I would just install a hyper visor of you’re choosing.

4:40: x16 will only run as x8 and x8 runs as x4 if you are using a feature that shares the same lane, x16 and NVMe Gen4 slot for example. It's important to know your mobo limitations like how many PCIe and which features shares the same lanes.

Remember folks the ultimate homelab server for you is the first one that gets you into the hobby! You can always upgrade use what you have and upgrades over time are part of the hobby you don't need to jump to the highest end!

Proxmox or xcpng are what I'd go with. It's nice to not have projects competing for ports or service configs.

When you were talking about the power supply with those giant connectors you've never seen before, you really proved to me how old I actually am, so thank for that. LOL. Looking forward to seeing how this turns out.

"I want a machine that does everything" 1 minute later "I already have a NAS" :P To say something more productive: I prefer buying used enterprise drives because their TBW rating is vastly higher. Even compared to high end consumer SSDs it is often between 3 to 9 times higher. And it gets so much worse if you compare against QLC drives. Considering things like write amplification and RAID you can run through a lot more written data than you expect. That is probably a trade-off you made intentionally, considering you have a separate NAS that can serve as a backup target, but it is something to keep in mind for others attempting to replicate it.

This describes EXACTLY what I want to do (in the intro anyways). Perfect workout watch.

So tired of them artificially holding back pcie lanes and also holding back on bifurcation of pcie lanes on consumer cpu's and motherboards. For the 2.5" drives- print out a bar that goes across the top of the drives and long enough to touch both sides of the case. There would be small ridges printed into the bar at each drive location to keep them in place. That way you don't block air flow. And you could easily lift off if you need access to drives.

200 Watt idle. Where I live (Netherlands) that's about 500€/year.

im tempted to threadripper my next all in one homelab build. but this is a good and cheaper alternative I feel

I'm excited for the software part! What would you say are the disadvantages of proxmox in a build like this? Doesn't that give you more flexibility?

I rackmounted my PC a week ago. The chassis was less than (the equivalent of) £100, including rails, and fits my 3 chunky radiators in too. My office is so much cooler and incredibly quiet now.

Kewl setup, especiallt delivery of ssds is funny 😂 they weren’t stolen

I can’t remember landing the plane in Top Gun on the NES. Well done ! 🎉 As for the OS, I would vote for Proxmox or a regular distribution like Debian, Ubuntu or a RHEL clone. You could use Cockpit if you want to spin up VMs on them.

@5:04 I don't know cpu lane calculation is as simple as you explain. you should probably check motherboard info, how many lanes are used for internals like lan, usb, ...

Tim, great video! Have you considered exploring Proxmox on this server and demonstrating GPU passthrough? It would be valuable to see which remote software is optimal for accessing VMs, and perhaps even conduct a gaming test. Looking forward to your future content! Greetings from Bosnia :)

Probably too late to the conversation but recommend sticking with Proxmox but using SRIOV to pass through part of the GPU to multiple machines

Great video and awesome build Tim. Thank you.

Sliger offers watercooling options for the Threadripper/Epyc line of CPUs in that chassis. I'm not aware of any constraints that would stop you from using random AIOs as long as they aren't too thick for your GPU clearances. There is a build someone did in that case with a thick 360MM cooler and a 3090ti.

Thanks, Tim. I thought I was over getting traumatized by the carrier landing in Top Gun lol

I don't think this is the ideal project to try Unraid on, but you really need to try it if you haven't. I always have at least a couple of servers in my rack and love changing things out, but the Unraid box is always there

Nice video!

With 'only' PCIE gen 3.0, you can hardly call it the 'Ultimate' HomeLab server 2024.

love the video. just setup some ai local following network chuck video it works awesome. cant wait for your take

omg the top gun carrier lmao. I've never once landed it as a kid.

16:55 do it! Unleash that hardware, sir!

The Silverstone RM52 is fairly roomy, rackable, and supports dual 360 rads.

Interesting. I am literally building a AMD Threadripper 7960x using a Sliger 4170i case with the ASetek 836SA AIO cooler. Just waiting on the case to be delivered to start the build.

100% add this as a worker node to your K8s cluster! No need to have a different management layer! K8S ALL THE THINGS!!! 🤘

I had the same idea a few months ago (all encompassing build with a lot of PCI), and I ended up going with a barebones Dell Precision T5820. I threw in an Intel Xeon W-2140B and some ECC RAM. Don't sleep on repurposing used workstation hardware!

Ooh look at that sliger case! Very pretty! PCI lanes is why I’m still using x299 systems :/

what patchcables are you using? they look fancy

Hi Tim, Quick question What are you using for your NAS currently?

If you didn't have the hardware already, I would've recommend with a Threadripper system because it offers more PCIe lanes. (That's the direction that I'm heading down, except that I'll likely end up with something like an 8U chassis and then using PCIe risers/extensions so that it won't cover the rest of the slots. My other option will be a Supermicro 4U or 5U GPU server, but those are limited to dual-slot-wide GPUs only, which means that my 3090s will be blocking some of the other slots.)

I'm just now noticing the shirt. I like it, haha!

Strap a fan to that hba. It needs airflow. Look up the cfm needed for it and youll see why, or touch the heatsink after running under load for a while

There are companies that offer 3D printing services; if you are not going to be printing stuff regularly, could be a good idead to look into one of those services. Great video, love your content.

Love my Sliger Cases. Have the CX4712 for my NAS and CX4150a for my desktop. I will probably buy a CX4200a for a gpu upgrade though. 3090FTW BARLEY fits with a low-profile radiator. It just fits a credit card between them.

I'm about to rebuild my server and I'm actually really curious what OS you're gonna run because it may influence my decision. I really didn't enjoy truenas because setting up apps wasn't intuitive at all. Curious about unraid but not thrilled about the price. Proxmox seems logical but I wanna be able to add more storage later on (Ideally) into the same pool. Cool build! Will be looking forward to the part 2!

Pretty case :-)

5.25" hard drives?? Sign me up!!

I would go with Unraid for a build like this as it's easy to use and have a lot of features and if you really want to tinker, it's running on linux and have a terminal for any custom tinkering :)

Question, is that power supply able to handle the power spikes 3090s are infamous for? Genuine question

getting a long pcie riser cable and making full use of the CPU you have seems like the best move imo

Try Unraid! I love it and is very user friendly

Try out Unraid. It's imho the best OS for a single system. file shares, VMs, passthrough, containers and ZFS support

Lol that saved-by-the-bell timeout

There are lots of makerspace’s that have 3D printers. Some public libraries as well.

If you do find a way to convert those 3.5" bays to 5.25", Wendell has shown some interesting enclosures that adapt 5.25" bay to 2.5" or nvme flash backplanes

Custom watercooling would give you back some more lanes. But i am maybe a bit to much in love with wc 😂

Built a homlab server as well - ran Unraid for a while which was in the end a patchwork of tools for functions that a server system should already have integrated. And running from a USB stick is not something I want to rely on. So I decided to run Proxmox which is the most flexible system in my eyes. It's lightweight, can run VMs or LXCs, passthrough hardware, it's reliable and definitely the more professional choice.

15:25 - Interstellar - Docking Scene LLama3: Endurance Rotation is 37,64RPM. It's not possible Techno Tim: No. It's neccessary

Use Harvester!! I loved the video where you showed the solution.

You mentioned this will not be taking over your NAS role. What about your firewall? I've seen some folks virtualize pfSense or vyos, but I don't think I would be comfortable hosting a firewall/ids/vpn on the same machine in case the virtualization solution has a security issue.

Yea try incus! Changed my homelab life up! Run it on Debian

This is the road I've been going down

Xeon. ZFS. Consumer SSDs (granted they are dirty RZATs). 10gig Ethernet. AST2500. Anythihg but the RTX 4000 SFF ADA. I see great moments of joy and power efficiency in your future.

Great video Tim! Thanks for sharing it with us! 💖👍😎JP

So where did we end up on power usage?

So apparently we are on the same tech wavelength and I just now noticed it, lol. I too just built the a few servers to do all the things. I ended up going with an Epyc system for all 128 of its PCI lanes dumped into a HL15. Another machine made use of the SilverStone RM41-506 4U chassis. I needed the 5.25" bays for the tape drive and an Icy Dock 4x 2.5" ssd hot swap cage, but the GPU just fits :)

Can I please have the stl files for the ssd cage you showed 9:07, or atleast guide me to the closest model for personal use

The last PCIe slot on that motherboard is served by the PCH and it’s a 4x, so it doesn’t “count” for the CPU lanes total. ;)

I'm contemplating similar build, and here are my observations: for proper local AI selfhosting, having Mac Studio with 64GB unified memory is MUCH more suitable than RTX 3090 which is limited to "only" 24GB vram, that said nVidia gpus utilizing CUDA are often faster than Apple's neural engine, all reasonably-priced Xeons are limited to PCIe 3.0 which is not futureproof at all, and already bit of a bottleneck for current gpus, any strong gpu takes lot of space and covers most PCIe slots on the motherboard, so it's rather difficult to decide on PCIe expansion layout, preferably gpu on "lowest" x16 slot, which then needs bigger than ATX case, that said some local AI tools can utilize multiple gpus or even multiple computers, so there's a delicate (cost/efficiency) balance between running let's say 4 gpus in one rig, 4 computers connected via 100Gbe together, latest Windows Server or plain Windows Pro, with WSL (Subsystem for Linux) seems best OS for widest range of local AI applications - arguably better manageable via Proxmox but with undesired performance loss, to sum it up, one all-in-one server doesn't seem that efficient, depending on what to run on it

Tiny subtitling correction at 2:05. I’m pretty sure you just left a gap in speech, rather than starting a new sentence. Generally though, your subtitles are very good. Thank you for taking the time to do them.

What about your server in the DC?

I'm curious how your SSD pool works out. I have a Truenas Scale setup with 6 2TB Samsung EVO's and I've struggled to get decent performance. Literally it's less than a single disk. I tried Stiped 2 x (3 raidz1), 3 mirrored vdevs and raidz1 across 5 of them. Hopefully you can do a follow up video on your SSD pool setup and final performance numbers.

Switch to a fiber SFP+ transceiver and it will run much cooler. The RJ45 copper ones run really hot and use a lot of power.

You mention the AMD in the script that has more PCI express lanes but in the video showed 20,16 then said "same number" but actually the Ryzen series have Native PCIe Lanes (Total/Usable) 28 , 24.

Consider the throughput you expect to need on your NIC and on your HBA, consider the max PCIe Gen your NIC and HBA support, you might be able to get away with using them in slots that are connected through the Board chipset instead of directly to the CPU (thats one of the ways these systems get so many pci-e slots/lanes other than switch chips) for example, if your 2x10Gbe NIC supports PCIe gen 3.0 4-lane, thats 3.938 GB before overhead is taken out. but if you are only using 1x 10Gbe connection, that NIC could be put into a slot that is shared with others on the motherboard chipset which might reduce the max throughput slightly over all. keeping your CPU PCIe lanes available for the more latency sensitive or higher throughput devices like the gpu or NVMe storage options. For your HBA, if you are planning to use 15x Sata SSD's, then your max sequential throughput would be about 8.25 GBytes/s, if your HBA supports PCIe Gen4 8-lane then you have nearly double the PCIe bandwith (15.754 GB/s) vs what your SSD's are capable of supplying so putting it in a PCIe 3.0 8x slot or a PCIe 4.0 4x slot would have minimal impact to your usability, especially if you were to use a hardware based mirror instead of ZFS mirror (done on the raid HBA processor). This is not a risk as you can still run smartctl and get direct access through modern HBA's and Raid cards to see the individual drive health for a LUN. EDIT: looks like that HBA is possibly a LSI 9305-16i which is PCIe 3.0 8x, so moving it to a PCIe 3.0 4x slot will cut your available bandwidth in 1/2 vs the SSDs max sequential throughput. so I would put the NIC in the bottom most slot (4x lanes in 8x slot) instead of between your GPU and HBA. For temps you have lost of headway, most HBA's and NICs for servers are able to operate safely up to 80C for ten plus years.

How about Nutanix for your Hypervisor?

Do a Harvester install, opensource kubernetes based IAC for containers and VMs with immutable nodes.

@technotim what cpu cooler is that?

Definitely suggest going the kubernetes route. There will be some configuration like installing the Nvidia GPU Operator, but it'll make good content ;)

You should look at AMD threadripper. They have R-DDR5 support and 128 pci-e lanes! Intel and AMD mainstream chips will stay at 20/24 lanes of pci-e and the motherboards are removing pci-e slots in favor of m.2 slots. a gaming rig only needs a main slot of a GPU and the rest goes through the chipset.

i would go baremetal and install cockpit and cockpit-machines for the VMs with lxc img

Great job, can’t wait for part two. Try unraid

I think you might like 3d printing!

Supermicro manuals will tell you how the pcie slots are wired, even if you don't use a slot that's say wired for x4 buts its in a x8, you don't magically move those lanes to somewhere else, they are stuck there. I'm willing to bet two of those x8 are actually x4, I haven't looked at the manual for that board but that's more likely. Nevermind I just saw in the diagram the last slot is wired x4, and those 16s are just x8. So your gpu will most likely get a small bottleneck.

So why are u preffering 48 pcie 3.0 lanes over 20-24 pcie 5.0 lanes?

Hey Tim, if you don't need that beautiful rack mountable case, I'd gladly take it for a new home server build :) I can pick it up at any time and throw in a six pack of your favorite Surlys too :D Cheers from the twin cities

I have a all in one server that is also my workstation. Using UnRaid with GPU and USB pass through.

Proxmox or Ubuntu please!