It’s ok that you had the original idea to steal my idea, I stole the idea too 😂

"Cause I can, and I am a nerd." Words to live by.

The way to get your workloads to prefer a node is to create HA groups for each node. Set the node priority for each group to be the same for all the non preferred nodes, and increase that value by one or two for the preferred node. Workloads will migrate to the node that is up with the highest priority first, and then prefer nodes of equal priority with the most amount of free memory and lowest CPU usage

Built a cluster with 3x Minisforum EliteMini HM90 with 32GB RAM (max. is 64GB) + 512GB NVMe + 1TB SSD each - plenty of cluster power with (3x9=) 27 Watts idle. The mini PCs also have two NICs so I'm running a separate Cluster Network for running the synchronization of the nodes as a best practice tip from Proxmox says: "Storage communication should never be on the same network as corosync!" Actually didn't want to run out of CPU power even when operating the one or another Windows OS on the cluster. Anyway: Proxmox rocks! Hope to see more content on your channel - which is one of my favorites. 👍🏻

As others have commented, Ceph needs more memory but it should also have it's own isolated (preferably) network, the faster the better. It will constantly be copying changed blocks on one node to the other two nodes to stay in sync. You should use that second NIC on each node for this.

This is such a cool project. Hadn't taken this board seriously until you posted this but now I keep thinking about it as I'm trying to adjust my new southern, basement-less lifestyle where I'm baking my wife in her office with my homelab servers.

"Because I can, and I'm a nerd." - RaidOwl That's exactly my reasoning for most of my Home Lab projects.

Very informative and clear. I have now a better understanding about how a Cluster and Ceph work in Proxmox. Thanks

"Or if you want an actual good video ..." lol, that took me by surprise - good to have a sense of humour!

Thank you for making this video! Proxmox cluster was the very next thing on my lab to do list!

Great video! I ultimately want to get to this point as well and possibly include plex in a supermicro 1u cluster with a p400 in each.

Only 10 watts and so much functionality is definitely a winner.

I'm surprised back-plane pcie interconnects aren't a thing with SBCs like this, would make for far better clusters.

Thank you, another rabbit hole for me to go in.

Ayee, that map screenshot shows my house. What a small world. I need to set something like this up, but w/ more power than the ZimaBoards. I know all the options, but it's hard balancing performance vs cost vs efficiency. Thanks for the vid!

I'm going to build my proxmox server with my old 3rd gen quad core laptop. Thanks for inspiring me

I've done the same exact thing on 4 different small form factor hardware. I'm running Open Media Vault which file sharing and docker, portainer support running all kinds of cool stuff. The must have was a UNIFI controller in docker. Having that in a HA cluster on battery backup is AMAZING!. It's very cool. I'll have to give the CEPH a try though. This looks like it save me some power. I'm using an NSF cluster for HA storage on 3 other mini PCs.

Such a great video. I’m doing the exact same thing with intel nuc. I use ceph at work and I despise it, but seeing how simple it is inside of proxmox has convinced me to try it at home.

I absolutely love your easy way of explaining. Subscribed ;)

Now that is a super handy cluster. You can feed it with power from a small array of solar panels and boom. You could even host your own satellite with such a setup 😂 And since proxmox is just, plain debian, anything could be done with it.

I have a similar setup, but use the Lenovo mini-pcs with I5's and 16gb RAM, SSD + NVME, a great setup. Expandable to 32gb on each node, things work great. Just need some fast shared storage now.

Awesome - gave me some ideas! Some quick questions.. 1) Do all the SSD / HDDs used in the Ceph nodes have to be the same size? 2) If I started off with (say) 500Gb drives and wanted to upgrade to (say) 2Tb drives, could I do this one disk at a time and avoid data loss, or would I have to back everything up, upgrade all the disks, and restore? 3) In your example, you had three 1Tb disks. How much storage do you end up with on the cluster? 4) Can you add more Ceph nodes once it's up and running? 5) If you can add more nodes, will it start with fewer than 3 nodes? (I need to spread the cost of disk purchases out!) TIA

Running Proxmox and a VM just to run Kubernetes in HA seems like an over kill. You are running HA container management on top of HA hypervisor. Why not just install an OS that supports cephfs and install Kubernetes on top. Just need to be sure that your etcd cluster spans all three nodes and each node is schedulable for workloads. One node goes down and Kubernetes can move the workload without having to first move Kubernetes and then the workload. Not as much fun but less hassle in the long run.

My system not being HA is exactly my problem. I have set up services along with pihole on it, but I don't want to put e.g. pfsense into it and have it work as a router, because if this single device dies, I don't even have access to the internet. Thanks for this video as it shows how to make the server part HA. I wonder if there is an easy way to have a router highly available. My current router worked for years without any trouble, but I am using the pihole as a DNS, because I wanted an intranet domain. Still: if it breaks, I have to manually adjust things to at least have internet back on.

Fascinating. Would definitely like for more videos to be posted on this SBC.

I experimented with a qdevice approach: main server, small x86 where critical services start if main server is down and and old pi to keep quorum. It worked fine, but the way I did it, storage wasn't redundant (it was all on my NAS) and when I wanted to stop using HA I had to mess around with proxmox files on disk directly.

another good option are Dell Wyse thinclients. they are low power, x86, more powerful, and about 3X cheaper than the zima board

The memory is a bottleneck here. CEPH needs a good chunk of memory. I found 16GB was not enough for a 500gb CEPH pool. With a vm or two.

Very interesting project. I see that they are selling the new ZimaBlade in a 3 unit cluster configuration. Hopefully, we can get slightly better performance out of the new ZimaBlades

I run a similar setup (but on NUCs instead of Zima boards) - one thing I've noticed is that LXC containers migrate between nodes in a few seconds, whereas my VMs take about a minute to migrate.

Crazy good. Exactly what i was looking for

This is amazing, thanks for sharing this architecture

My suggestion would be using low power cluster as backup.. Maybe consider using a setup like this: Cluster A (High Power) - Node A1, Node A2 Cluster B (Lower Power) - Node B1, Node B2 HA - Cluster A and Cluster B (maybe cross wired) for efficient and more practical usecase.. This will also make sense in regards to performance per watt.

Love it!

Man proxmox is so good!!

Best practices to disable PG Autoscale. It has killed a lot go ceph clusters.

Great informative video! Gives me something to think about with my intel NUC's

Building a HA Cluster with single Point of Failure (Switch) is exactly my kind of Humor👍

I'm thinking about doing the same with 3-5 R86S. The nice part being the dual 10gbe ports. 1 port for client traffic, and 1 port for the ceph network. also there are 2.5gbe ports left over for management or something else. Or I could use the two 10gbe ports for a ring network for ceph, and the 2.5gbe ports for client. The ring network is interesting because I could avoid buying a 10gbe switch just for ceph

NICE buddy !! this is awesome !!

Oh wow! That's pretty much my "wet dream" right there. Nicely done. I'm really jealous.

I am currently using truenas scale for my vms and containers but this is tempting. I have two questions: 1. What is the performance penalty compared to single node? Is it noticeable for Web services? 2. How does upgrading proxmox to newer version works with cluster like this? You just shut down node A, do upgrade, reconnect it to cluster with older versions? Then next node and so on?

Funny your video popped up on my phone this morning. I spent my entire weekend trying to figure out how to auto migrate a VM back to its original node. Apparently, it can be done with hook scripts.

Proxmox seems pretty heavy for something like this. I'd like to see something like this with updated software. Possibly Fedora CoreOS to get auto OS updates with rollback, with k3s and Rancher dashboard, or just go full first-party with Docker Swarm with ___ dashboard (I wish there was a first-party one), replicated masters in case the master node goes down, but also sharing storage for HA. Double drives (NVMe or SATA) on the Zimaboards for redundancy to handle a single drive outage without losing full HA support for services (can drive mirroring be done with clustered storage within the same host in place of RAID?). I was thinking something like this with Cockpit to manage the system hardware and OS and Rancher to handle the apps. It's too bad that Cockpit doesn't have a Kubernetes extension. They used to, but it can't be found anymore. Probably because they're supporting Podman already. There's nothing in Cockpit (AFAIK) that creates or manages clusters unless you go over to the Red Hat corporate side of things and start looking at OpenShift, and even the community version OKD is too much for a Zimaboard. Minishift is also dead.

This looks awsome. For testing like you said its fine. Now i consider one or two of these for pfSense

@Raid Owl please please make in-depth proxmox tutorial I am struggling to understand the LVM stuff I use EX4, and for instance, if something happens to the system and I plugged that hard disk into a working system, how do I access the files that were inside that proxmox that are like LVM? I don't know. There's just no really good in-depth proximox tutorial. There's a few but a really good one I think a lot of people would really like that

Presumably this would be a good way to run a HA HomeAssistant home automation system? to my knowledge, although folk use Proxmox to host HomeAssistant (I do), no one has posted a video on how to run one in a HA way.

great video, thanks for making it. have my engagement!

The ZimaBoard could have been straight up God tier if it had built-in NVME support. Installing a PCI-E add-in card doesn't count.

Cool setup! I want to make my own mini cluster and this inspired me! Maybe Beelink EQ12(I dont know whether it can be sold in your untry, but it is availble in China and Singapore) is an alternative to ZimaBoard if you want more compute power hahahaha. it has the new Intel N100(which may be 2~3X faster than N3450, 4 cores) , and the no memory no ssd version cost about $150 in China.

Bitwarden would be nice on a Cluster.

"...it sits about 10Watts total.." What hardware/software do you use to monitor these SBC's? Fantastic info. Time to give Proxmox a try (been using UNRAID for years now).

The best reason for everything is "because i can"!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! LOVE IT!

I want a setup like this, due to draw power.

Great video!!!!!! Love it

I can't stop but feeling like ceph is a bit overkill for a cluster in this scale and size and with the given hardware. Instead I would look towards zfs and the use replication between the nodes for the vm's that needs HA. This would also minimize the complexity of the setup a lot and get rid of things like split brain issues which can happen with ceph 😊

I plan to go K8s native, no proxmox. Using something cheap as Control plane, and this zima boards as workers. Let's see how it goes.,

I really want to like the Zima board, but the pcie slot sticking out bugs me. Especially with such a nice looking case/heatsink design

you know what would be cool? For you to do some sort of collab video with techno tim/ idk maybe some sort of unraid to unraid server replication guide??? maybe a wireguard tutorial ?? Doesnt have to be something groundbreaking or complex, just the idea of getting you two on a video with some friendly bantering and dark sarcastic humor on your side, (to see if you get tim to break out into LoL's) could be of fun to watch

I'm suffering from cluster envy

Cool. I dig it. 🔥

You could buy for that money a much better system. For an example, for 120-150 Dollar, u could get a refurbed lenovo thinkcentre m700 with core i5 like 6400t or 6500t, 8 GB RAM and 256GB SSDs, and it takes only 15-25 watt with a much better perfomance rate. You''ll get also a clean case and so on.

I was recently actually looking for a tutorial to do this lol

If you plan to have k8s on zima in vm then why to use proxmox in the first? Only reason I can think of is to use other OS then Linux but resources will still limit you. Ceph can be installed inside k8s or next to it. Besides ceph is not only block. You can have s3/Swift object storage or cephfs POSIX compatible server.

I am currently looking at a cluster build with intel nuc, but this set up would be cool to tip my toes in clusters and such.

You can tell it's monks from the holiness in their design.

Hi there, is it possible to daisy chain zimaboards since all of them have 2 ethernet ports or is it neccessary to connect all of them to a switch?

Does sound like a fun thing to do!!

Can you please let me know as a beginner, is it a good idea to run out of the box casaos on Zimaboard or do I need to set up Proxmox and boot it from a USB Please help me understand, as this is my first mini home server. I would like to learn and build on top of it.

Can you run pfSense on a Proxmox cluster? Having a highly available router/firewall seems like a neat solution

Would this be a good design for a micro super computer in a small pelican case? Something better than a standard laptop.

I have a question... how those zima boards works in 24[7 schedule? and what doing on if your zima1 (for example) with ssd has been down? how you connect ssd as ha nas?

Don’t use rancher for small resources, just use openlens which runs on your desktop machine and connects to your k3s

What if that single switch fails? It's not high avail then is it

hard to call something HA when it's all running on the same power supply, same switch, etc... here's a video idea: HA-ception - 3 instances of proxmox running in another instance of proxmox. (And then for the next video, HA your HA)

I started to set up an HA ProxMox cluster, and got stuck at the "Fencing" part. Is fencing optional, or does it just seem more intimidating than it is?

is this pwerful enough to run and train open source LLMs?

Hey bro. I heard you want to build an HA proxmox setup to monitor your other HA proxmox setup.

I am trying to do something similar with three Gigabyte Brix machines, but I'm running into issues because creating an OSD seems to require an empty disk (ie, it won't accept the disk where the os is). Did you partition the zima drive somehow before install?

If using a single NAS isn't HA because its failure would down the whole system, then how can using a single switch be HA? Both appear to be single points of failure.

Great job - done by the italien munks ;)

Good idea❤❤❤❤❤❤

I installed proxmox on my Zimmaboard on emmc too. Question is how long will it last until it dies. There’s maybe a reason emmc not being supported

I'm lost on how you are able to setup HA mode when the storage is local on each device. I have a similar setup and I want to do that. Can anyone point me in the right direction with clear instructions? Or explain in detail for me?

With my budget I'll have to settle for "Somewhat available"

Very interesting, that is exactly what I am planning to set up for home automation (NodeRed, Grafana, ...). Right now I am hanging with one topic. (How) is it possible to share data between VMs or Container? I think, Proxmox is running on its own (the internal) disk. The VMs, and containers are on the external SSD. When I am setting up a container, that container gets his own virtual hdd assigned that is placed on the external SSD / the Ceph disk. Is it possible / how to have a folder / diskarea, partition, ... lets call it "shared folder" where different container can read and write data. Later on there is maybe a container with a simple NAS software solution or just a SMB share, that gives me acces to that "shared folder" via LAN so I can backup that data from time to time. I would be very happy, if someone can help me out how to do that. Thanks a lot

Question: does proxmox have a feature to migrate vm's between nodes if the resource usage (ram/cpu) Is very high?

Thanks !!

With the pci express port, can you not attach a video card for plex?

You are using a single network interface on each Zima Boards for regular networking as well as cluster networking? That never worked for me, time and again, the cluster would incorrectly detect a network issue and the cluster would go bizzare forcing me to reboot the entire thing!

Does the shared storage have any redundancy built in? or just creates 1 pool essentially?

Very very impressive,could you do it with R86S-U4 which are Intel N6005 CPU with 32GB ram+3*2.5G+2*10.0G SFP+ port? That was so expected!!

How do I run one VM across all three nodes in my cluster to take advantage of all cores?

I'm not a fan of running a system with a PF of around or below 0.5 as there's a lot of wasted power being burned...

Zima boar support mikrotik router os or not sir 😅😅

I am about to do that, I am going to use a Vima board as the NAS. What are you using for storage?

Only thing about this is installing it on emmc... not idea.

i though this was an arm powered SBC. automatically assumed it was using Pimox

Why not using harvester instead of proxmox ?

Nice video