I haven’t finished the video yet, but 2 minutes in all I can think is “oh, that’s gonna be loud.”

4 node proxmox server with ceph FS across the SSD-s. Really with that hardware this should work great!

Would love to see some GPUs in those PCI-E slots to demonstrate pass-through and VGPU support in the various platforms.

Clustering Proxmox with CEPH is tha bomb

Nice work on these videos man! That build montage was great! The music was a good choice. I have scalable CPU's in my homelab but this is a CRAZY build!

It sounds like you have reservations about Hyper-V, but Hyper-V does have an interesting advantage when using a quad node system like you have. As far as I'm aware VMware, Proxmox, and Nutanix all need a minimum of 5 nodes if you want the cluster to be able to take any two storage failures (Such as a failed disk, a failed host, or even a host being rebooted). VMware for example will make 3 copies of the VM and place them on 3 different hosts and then a "witness" component on 2 more hosts (A small files used to vote for control of the VM). This way if the cluster is split and say 3 nodes can talk to each other, but not the other 2, they can use the fact they own 3 of 5 parts of each VM to know they can safely bring up the VM's and the other two nodes are not elsewhere doing the same. It probably sounds a bit complicated in a 5-node setup, but in a larger setup when a split cluster could have whole VM's it each half, it allows each half to operate independently starting only the VM's they have majority control over. Hyper-V handles this a bit differently by electing a master node based on a cluster vote and that node decides who can run what. If the master node fails, the remaining 3 of 4 nodes will vote to elect a new one. The tricky bit is the 2nd failure. If 2 nodes fail, the remaining two nodes have no way to know if failed nodes are actually offline or if they just lost communication with them. In this situation 2 of 4 nodes can't obtain a majority vote to run the cluster, as that would allow a 2-2 split cluster to each start a copy of all the VM's. To avoid this, we need a 5th vote. Rather than a 5th node, which is kind of a pain when running with a quad node chassis, Hyper-V can use an external witness. VMware has an external witness as well, but in their case the witness is a whole VM that has to be ran off cluster. Hyper-V just needs an SMB2 compliant share the hosts can read and write to. In face the requirements are so low that in Microsoft's own documentation, they advise even a flash drive shares out from a SOHO router is enough. No need to worry about router reboots as well, as this share is only needed when there are not more than 50% of the cluster hosts available. I'm not 100% sure about Proxmox/Ceph yet, but both VMware and Nutanix are limited to one host fault at 4 nodes. This is likely fine for a homelab, but I run clusters at work and home and it's something to keep in mind.

Awesome video thank you for sharing and all the work

Nice Job Rich. I can’t wait to see Proxmox clustering with that bad boy!

A real nice thing about having 4 identical builds in one system, a 2u in this case, is you can do true A:B testing of different setups. I know you are thing VMs right now, but showing how fast/complicated a Truenas VS Unraid setup is one idea, and then testing default speeds of network shares on same said systems. As one simple example. Going with the VM theme, have one as HyperV, Turenas scale, XCPNG, and ESXI all running at the same time with all running the same VM loads and see what cpu usage is on different hpervisres and how responsive the VMs are. I know at work we just moved all of our VMs from an NFS share on 10K rpm SAS drives to SAS SSD's and they are so much more snappy, some hardware otherwise, and I know this is spinning rust vs ssd, but wonder if same sort of testing could be done for esxi vs everything else.

I'm interested in seeing a video showcasing Netdata, where agents are installed on servers and communicate with a central dashboard. It would be great to see the agents' communication with the dashboard and how they can be turned off from their own web interface. Kinda like how Zabbix works

Okay, before i insert the comment: I would like to see a massive Kubernetes cluster on those 4 Nodes! The amount of containers would be absurd. But you can stack them in Namespaces, the auto-heal capabilities lets you easy pull out the nodes live when the Kubernetes cluster pulls up the containers elsewhere. And Containers are the god king to effective use every last of resources on servers on a massive scale. And the core concepts of Kubernetes are fantastic made, because it’s declarative: Say via YAML what you want and let the automations take care of the rest. Would be nice to have such a big server, so much more power and ressources, hosted on prem. But the energy costs 😅 that lower my excitement, not gonna lie. Back to my initial comment: I‘m at 8:56 and what surprises me the most is the price for the cpus, i was sitting here like DOE?! The Chassis is a very good deal, the RAM, yeah ECC that always costs a ton of money but the upgrade capabilities means you won‘t get out of ram in the next 50 years. But what gives me a headache are the wd blues, i had issues with the HDD SATA ones of those, until i grabbed some more cash and when to the WD Red SATA HDD, and what i can say: My initial stupidity was thinking „well, I exchange them in my NAS when they going down“ but FFS they don‘t go down. On the current level of degeneration they will survive me and my dad 😅 i went lately one my server trio after the NVMe SSD begun to fail also to a WD RED NVMe SSD, currently on Amazon for 87 Bucks each and a PCIe to M.2 Card for 17 Bucks each. Will test it out on my first server, to put them in RAID 1 as the boot disk for my proxmox instance after migration the existing installation of the failing NVMe SSD to the RAID Array. When they degrade as slow as the SATA HDDs then if will have a lot of time with them 😊 and i will begin to switch out the main NVMe SSD on the other 2 servers to the same setup. And add an RAID 5 for data to each server on the long run. Currently my HA is degraded but still functional, that was the initial reason to buy 3 servers for HA Subscribed and Like already for a fellow server geek like me 😇 But one constructive tip: Please reduce the bass on the „building music“ that hurts a bit for headphone users

I am building a second R730 I got a couple years ago but never put together as a NAS. I am almost wondering if I should take my 4 node cluster and set it up with a combination of Proxmox running TrueNAS on each and combining the storage using Ceph. Then using the R730 for a backup only unit to stay powered off except when needed? I get two NVMe drives per node and four SATAs. I already have eight 4tb sata SSDs that were going to go into the R730 but could easily slap two in each node. Then buy four 8TB NVMes and slap them in. That way expansion would be available by getting eight more 4tb satas and four more 8TB NVMes. I am wondering the speed vs just having a dedicated R730. I would just load spinning rust into the R730 and use a 3.5 in JBOD for backup purposes.

Hi, thank you for this great video. I would like to see how you cluster these nodes, for fail, or for enhanced function.

That beast has afterburners!

One thing I think that would be interesting is setting up Open Stack. It's complex enough that it should warrant a couple of videos getting it setup. Another thing you can do is setup a huge virtual network with this new server. Something with eve-ng or gns3 could stress even this machine if the network is big enough.

Just got a similar setup on ebay that runs on Epycs. I got an awesome deal with CPUs and RAM. Seems really cool and I am going to use the 4th node for a working Win11 RDP system.

After your video series for the hypervisors, I'd love some to compare the full management stack of VMware alternatives (vCenter vs... x) and their potential administration overhead. Have a good time with this beast! 😉

I'd be interested in seeing the clustering/failover/live migration with xcpng, currently using vmware but looking at xcp-ng. Also passing through hardware such as gpu with xcpng with these nodes. What is the power draw like running the 4 nodes?

Having gone through this on a very similar Supermicro trying, Hyper-V, XCP-ng and PVE … I recommend this config for max performance: PVE with 192GB zfs boot mirror on the two NVME. then zpool with 2xz1 vdevs (3x mirror vdevs would be a bit faster but less storage). Use the 10Gb nics for replication/migration and backup. Add PBS to one of the PVE nodes is also fun to do. With the remaining space on the nvme drives you can play with testing a mirrored SLOG, small mirror vdev, special vdev or l2arc. All fun stuff. You could also do a ceph configuration and it would be usable for sure, but not max performance. If you had 40 or 100 Gbps nics then Ceph can really give some great performance.

Holy crap, those CPUs are dirt cheap now!! I need some!! 😅

Craft computing has done a couple videos with blade/multi node servers so you are not alone.

how about looking into some of the sdn features of proxmox?

So whats the draw to xeon golds, compared to say 7000 series epycs which can overtake in core capacity very easily and have gobs of pci-e lanes. I'm newish to the homelab game but you can get a 64 core epyc 7000 series, board and the same ram and still spend less than the price of that entire chassis (or two for something like 2500). I agree that the xeon scalables cpus are very frustrating to navigate compared to the previous generations. On a one to one basis in a different chassis xeons are crazy cheap compared to epycs. Gotta say, I love supermicros ipmi compared to idrac 6 and 7 series. The desktop app is really cool too.

10:20 the nvme daughter board has the two plastic hold downs for tool less install. You can see them on the right side of the pcb.

Nutanix ! 🎉 you can even do that with Terraform 😅 not sure for CE though

I installed a couple hundred of these nodes in Nutanix trim - the rails weren't you're standard ball bearing sliders instead having a little shelf the bottom of the node sat on with retention when the node was fully inserted. Haven't been able to find a picture of what I'm looking for. I think what you will find when you get to testing Nutanix CE is that 128GB of memory might be a little light when all SSD as I think the minimum CVM memory for all SSD was 32-48GB which won't leave a lot of space for your workload. But in that configuration you will have some very crazy IOPS/disk performance. Also would recommend balance-slb or balance-tcp network bonds instead of physically separating vm vs storage networks.

cluster PLEX. would love to see the scalability. just not sure that those cpus have Intel QS

Welcome to the blade family! There's (probably less than) dozens of us! I won't ask you to check out Hyper-V, but I will ask you to check out it's bigger buffer brother, Azure Stack HCI. Can't beat a 60-day free trial

I was thinking Hyper-V failover cluster... Come to the dark side!

I wonder how much power this monster draws?

Getting wrong rails from ebay sellers is super common. I have several boxes full of useless rails I've gotten. Only twice have I gotten a server that either didn't have rails or the wrong rails, and the box-o-rails delivered. I don't know what is with SuperMicro, but their rails kits seem to be hyper specific to single models.

It seems good, do you know if is possible to install Nutanix CE?

I used to run fat stacks of Xeons in my lab, but I've been replacing them with single socket AMD Epyc systems, and in some cases mini PCs, because newer CPU architectures are dramatically faster and more power efficient than old Xeons, and having only one socket means far fewer SMP/NUMA speedbumps. And the noise... just one of my old Xeon Supermicros, even with the optimized fan setting, is louder than my other 5 nodes and all the switches combined. I don't wanna go back to the jet engines.

It would really be great if you tried out Harvester HCI from SUSE. I am a huge of this open source project and this kind of sizing is perfect as a Harvester use case.

I haven’t seen it mentioned yet but what about Openstack?

Run a Minecraft Server on each node!

I'd love to see a proxmox cluster with Ceph and a bunch of VDI VM's running off of it with gpu passthrough. Would make 16 or even 32 very high performance workstations if you can slice up the GPU's enough. Proxmox templates would help build all of those instances.

Since you have a big enough cluster you should try rancher hci with longhorn

I would dearly love more 3647 kit in my homelab, and this machine isn't it... but the boards are NOT inexpensive by a long way, unlike the chips.

next videos on nutanix and prox+ceph!!! this was a great video!

Everything was impressive apart from the 2133mhz RAM. $27? What a deal!

Video + Video Description + Link has conflicting info, "SuperMicro 2029BT-HNC0R" vs "Supermicro 2029BT-JNC04" vs "2029BT-HNC04" :-?

nice work! Now build an "absurd" home AI server 😀

I’d want to see XCP-ng, but specifically with their newly officially supported XOSTOR. Their equivalent to vSAN.

Hyper-V! GO GO!

why is it staring at me 11:40

XCP_NG and Nutanix CE

U crazy ;) I was considering similar old server HW, but maybe I just enjoy your suffering while buying new regular HW instead.

Incus and Kubernetes clustering. Incus is the relatively new player on the block in containerization BUT has a solid foundation.

Openshit and other HCI offerings might be interesting too

maybe cloudstack apache?

"Don't offer $50 below asking." Dude, you asking $500, i'm going to offer $350, you counter with what your bottom dollar is. If you get offended, hey, i'll take my money elsewhere. If you select the option to accept offers, you better be prepared for that.

What is the difference between this "4 node server" and the old blade server technology that is hardly being still used? Don't you actually have here a blade enclosure with 4 server blades, that instead of calling it 'blade technology' you call it a '4 node server'?

power consumption??? the only issue you missed covering.

Azure Stack HCI

Hyper V, hyper V

Your WAF quota might require consolidation and selling older gear to recover some $ if she sees this😂

It may not help but you should tell your wife you got a bargain. I bought a Dell T640 18-Bay new in 2016 and it was $3,500. It came with one single 1TB HDD, a 10-core Xeon Silver 4114, and 32GB of DDR4 PC2666.

you are killing me with only having 10gbe of bandwidth between them. if you spent this much already, get 4 100gbe cards and a mikrotik CRS504-4XQ-IN...or at least 4 more 10/25gbe cards and cross connect everything for storage and/or replication.

Unraid or proxmox

First!

HarvesterHCI