Thank you for pointing this out. Strange thing - from my testing every time CEPH takes a lead for transfer time between node compared to NFS.

I am glad all worked out for you. Just please note, USB storage does work but Read/Write will kill USB storage using faster compared to using a HDD/SSD storage.

Good Luck. At first i was the same ... scared to even touch that option. Once i tried to setup couple times, turns out it is not that hard to setup, it is just the order you need to to things and once all setup just leave it. it does almost everything by it self.

Hi, Thank you for your comment. I have started to write up script for Proxmox Cluster setup "From 0 to Hero" It is taking some time but one thing is for sure - it will happen.

I have 3 cheap N100 mini PCs. I added 256GB SATA SSDs to install Proxmox OS, and then I'm using the 500GB NVMEs that came installed with the mini PC to be the Ceph pool drives since they should be faster than SATA SSD. All important VMs & LXC containers will be installed in the Ceph pool with HA, and when I'm simply testing, I can use the local storage left over from the Proxmox install on the SATA SSD.

You can 100% use it for production. Difference between FREE repository and PAID is that PAID repository packages getting validated by Proxmox team before available for update. With PAID version if something goes wrong with your Proxmox host, Proxmox support team will help you out to get things sorted where FREE version you are on your own to find solution / fix. So far i have helped to deploy 5+ Proxmox setups where they being used in "production" environment.

@@MRPtech thanks so much for the straight answer.

Hi, I don't use them and as i don't have enough understanding about them - i am not in a position to teach over. Here is a paragraph from CEPH Documentation: A write-ahead log (WAL) device (identified as block.wal in the data directory) can be used to separate out BlueStore’s internal journal or write-ahead log. Using a WAL device is advantageous only if the WAL device is faster than the primary device (for example, if the WAL device is an SSD and the primary device is an HDD). A DB device (identified as block.db in the data directory) can be used to store BlueStore’s internal metadata. BlueStore (or more precisely, the embedded RocksDB) will put as much metadata as it can on the DB device in order to improve performance. If the DB device becomes full, metadata will spill back onto the primary device (where it would have been located in the absence of the DB device). Again, it is advantageous to provision a DB device only if it is faster than the primary device. How PGs getting calculated: Total PGs = (OSDs x 100) / Pool Size If i calculate my production cluster using this formula i get PGs = 100. PGs in my Production CEPH fluctuates between 50 and 120 which average of 90. More info here: docs.ceph.com/en/latest/rados/operations/placement-groups/ edit: forgot to mention, during night when server does almost nothing, PGs drops to 30 - 40

@@MRPtech Thanks for the info. I think according to the formula your PGs would be 3 (OSDs) x 100 /2 (Tb) = 150. How the 100 came up as a result?

The documentation says „faster drive than…“ does that include a ramdrive? Or must it not be non-volatile? A ramdrive would be the absolute fastest and significantly increase log, db and metadata performance

For you to have HA (High Availability) where is one node goes offline, VM will automatically start on another node - you need 3 Nodes minimum. Nodes count should be odd number (3,5,7,...) (more info : pve.proxmox.com/wiki/Cluster_Manager ) With two nodes you can migrate VM from one node to another manually, Auto VM migration can't be done without Proxmox HA.

Thank you for a comment ! Couple thing i want to comment back if i may, PiMox - i don't think Arm version of Proxmox has support for CEPH. I could be wrong, but last time when i tried to get 3x RPi 4s to get in to cluster, everything worked so-so-ok until i tried to get CEPH working. When you have want to remove any node from proxmox and same time all its connection to CEPH, you need to do in very specific order otherwise all your cluster full fall apart as house of cards. I know that because i messed up quite a few times while trying to get one node off cluster list. I am making a list of steps you need to take to get this process done without (hopefully) any problems and get video done about it. My main proxmox home lab cluster is 3x Beelink Mini S12Pro with N100 CPU. Each has 512GB NVME, 2TB SSD, 32GB Ram. When you setting up CEPH, you can't configure CEPH to work with one node. It needs to be minimum of 3 nodes. If one Node had 3 drives, you can set all 3 drives as CEPH OSDs and make replication work that way but this brings another problem, what if that node goes offline, CEPH will go offline too and all this hard work to make CEPH work is pointless.

@@MRPtech Yep, I see the madness in my logic. I think I will take the other two 1TB Disks and build an OMV server somewhere else in the house! Like you, I learned the hard way. Don't mess with CEPH.. It works, and well, but if you look at it wrong, it bites you!

At the start i did too had only one mini PC. my setup back then was: mini PC with 1x NVME 512GB for Proxmox HOST and 1x 2TB SSD for keeping all VMs / LCSs. Backups where stored on same 2TB SSD and once backups where done sync to outside location. If i loose 2TB SSD - that is all my VM/LCX are gone + all backups are gone still i have option to restore all from outside location. For you i would recommend same setup. Depending on how many VMs you planning to run - increase backup frequency to run maybe twice a day, i had mine once a night due to amount of VMs i had going. Backup used to take 4 hours and running twice a day wasn't an option for me. If you mini PC allows to have 3 drives - 1 for proxmox, drive 2 and 3 are mirrored ZFS.

@@MRPtech thanks, in your first setup that second drive (2TB SSD) was mounted as a ZFS?

At the start i had it mounted as simple drive inside /mnt/2tb_drive, but later i changed it to ZFS single drive when i learned one or things about ZFS. Using ZFS even with single drive was great thing to have as i was able to do snapshots which by the way saved me from headache more then once.

I've got an external NVMe on that node that is ext4. I created a mount point for it, a directory in DataCenter, and then created a 500GB virtual disk on it in a VM. That seems to work fine. It was mostly a proof of concept with Proxmox Backup Server so I could repurpose it easily enough as the ZFS disk for my Proxmox HA proof of concept if that's a better option

With ZFS as your VM storage, migrating VM or LXC won't migrate data. You need to do Replication to make sure that all the nodes has up to date information. Do you have a backup of that VM? If you do, Destroy VM and delete VM disk. Once all that is created you can restore from backup

@@MRPtech Hello and thank you so much for the reply!!! and no that’s the issue I do NOT have a back up for that VM (it’s on my list to do) and it’s still on the host it migrated to without the data drive the data Drive is still intact on the original host tho. I’ve been hesitant to do anything because I don’t want to lose it… The ZFS storage pools we’re already originally created when Proxmox was originally installed onto those nodes not sure if that has something to do with it or not. Because in your video, I noticed you created ZFS storage onto the three nodes, and I figured that was already done in my instance. Another interesting thing is I did try to migrate it back to the original host and not only the error of the missing volume but also because it had a USB attachment to the original so it says that it could not be migrated, but in fact, it’s there. And I wanted to mention that my plan is to create either NSF on My Synology and or Ceph storage. I do have an additional drive on each of those hosts. Plus, I want to install the proxmox back up server not sure the method of that yet tho.

I was going to use ZFS since I'm simply running a home lab. Nightly sync for ZFS would be more than adequate. I don't care if I loose 1 day of data. However, since a home lab is for learning, I've decided to use Ceph.

What do you mean "already made Radi1 while installing"? You installed Proxmox OS inside Raid?

Once Node is OFF. What Proxmox HA page is showing? Quorum OK or not?

@@MRPtech HEALTH_WARN mon pve32 is low on available space; 1/3 mons down, quorum pve31,pve32; Degraded data redundancy: 41800/125400 objects degraded (33.333%), 33 pgs degraded, 33 pgs undersized. Each node is ceph monitor as well as ceph manager.

I have all 3 nodes set as Monitor and Master. and i have 2 WordPress websites hosted. If website runs on Node 1 and Cloudflare Tunnel Runs on let's say Node 2 or 3 or even 1, once Node 1 dies, all stuff gets migrated automatically and website up and running again.

@@MRPtech are you showing this configuration on details on one of your videos because I have lack of knowledge how it works on the networking part since when you open the port 443 or 80 is for a specific IP on a specific node this part is not clear

You have port forwarded done for the site? I have cloudflare tunnel pointing to my website. This way i don't need to port forward anything inside my router. Cloudflare Tunnel : kzfaq.info/get/bejne/jt9znaaepsW2Yqs.html&pp=ygUabXJwIHRlY2ggY2xvdWRmbGFyZSB0dW5uZWw%3D Let's say your website local ip is 192.1.2.3:80/443 So you need to port forward 80 and 443 to point to 192.1.2.3 If website VM hosted inside node 1 and node 1 goes offline, VM will auto-migrate (if Proxmox HA setup propertly) to another node. VM Local IP won't change and your port forward rules will still apply.

If node goes offline, Home Assistant VM gets migrated ... but ... i have 2 USB devices linked to HAOS VM. Zigbee and Z-Wave dongles. I been looking for work around for that, thinking about USB-over-IP setup where both dongles will be shared by 4th device and HAOS will pick usb devices over network.

Thanks my case. I have two Ethernet Zigbee coordinator. In this case , I don’t have any device physically connected to the VM or machine. Another question? It a mandatory to configure a network vlan only for the ceph? Could I use the same network net that I use for all?

With my proxmox cluster everything is linked via one connection as each node has 1 lan port because each node has only 1 single LAN port. If each node had 2 i would definitely configure 2nd port to work only for CEPH. Is it mandatory to split CEPH data traffic from everything else - i don't think so. If you pushing a lot of data in and out of CEPH - data transfer speed is a priority. For me, for home lab - would be nice to have best possible data transfer but it is not at the top of my list - i just want CEPH to work. To add to all of this: when my proxmox cluster doing backup of VMs and sends backup data to Synology NAS. Data transfer is going via that one single LAN port. If same time i am creating VM or LXC container and virtual disk is stored inside CEPH - i do notice that this process takes longer compared to the time when LAN port is not busy. CEPH > OSB list and check "Apply/Commit Latency (ms)" showing how long it takes for CEPH to sort data in/out traffic. If LAN port is not busy - latency usually sits around double digits, but when LAN port is doing something - that latency goes to 200 - 300ms.

Thanks a lot for your helpful information. Appreciated

I confirm taht CEPH affects my network, I must be mandatory create a Vlans otherwise more of my devices drop the network, bacuase my wifi master mesh is in the same network than CEPH and also in the same switch. ZFS is working as CEPH, or need less network capabilities?

Yes. CEPH is redundant storage. Where RAID storage located on one node, CEPH is spread across multiple nodes.

At least 10 Gbps dedicated network.

10 Gbps is good to have but i think 2.5 is a good starting spot, for home lab user at least. Once i got everything set, even 1 Gbps was doing ok to move stuff around. Where it fails is when i try to do a lot of things at the same time (backup VM, Restore VM, Migrate VM) all at the same time.

Yes you can make VM to be relicated to more than one Node at the same time.

@@MRPtech what I means is i want to have a failover storage just incase my first nfs storage restarted or somehow fail to boot for various reason. My pfsense is installed on truenas drive but i want to failover to nfs setup on raspberry pi just incase truenas fail to boot is that possible?

All 3 nodes (OSDs) are link between using 1 Gbps which is same link between Cluster and NAS. If my intention in this video would be performance between storage options, you comment would be more then valid, my intention in this video was - Setting up HS Cluster and NFS does a job if you want to have HA cluster. As i mention in the video NFS and CEPH performance from my testing is almost the same, but NFS has "single point of failure" and that is why, and i quote my own video "CEPH is best option for you"

doesn't Ceph allow locally stored data to be accessed locally? admittedly some of the data may require a network link but you could probably configure it so that your primary compute node for a given service would be almost 100% assured of having local data (the benefit of which is only realized if assumption that local data doesn't need network IO).

Hi, To me these errors showing that there is an issue with connection between nodes or drives inside CEPH failing to sync. When similar thing happened to me i lost 2TB SSD. Drive still works but its performances is doomed. All 3 drives needs to be in sync, when migration happens, a lot of data being pushed between nodes and drives, while all this happening - proxmox relocating VM drive. if data connection between nodes is not speedy enough or drives can't read/write data fast enough - CEPH will struggle to keep up with all block changes. What it is showing now?

@@MRPtech yep I guess your correct, york's OSD Drive are smaller that the other 2. This could be very well the problem. thx for your response.

OSD drives recommended to be same size, from my testing and for home lab is not mandatory. You can have 2 OSB drives being 4TB and one 3TB is fine. but when you have 2 x 2TB drives and one just measly 512GB - this could lead to a disaster.