Thanks, will be covering the config in the next video (once I've figured it all out 😂)

Thanks, you too!

Thanks, I'll retest. Might be complicated by using a WAS-110

I could confirm it works. I use a vlan for my isp to fw

Mini Forums should have a AMD edition on the way for next month

@@ulrikboesen that's good to know

you dont even need vlans, just connect all minis to a switch + isp and whenever one mini fails, opnsense will get restarted and connect with either pppoe or based on MAC dhcp, good thing is, even if using virtIO networking, you set MAC inside opnsense without having to set it physically on the port

Good to know. I haven't seen that yet, I'm getting 2.5Gb on speed test.

It is working well for me, my only concern is longevity of the device but time will tell. It's a great little machine.

@@Jims-Garage Thank you for your answer! I'll think about it more, I'm currently testing my DIY server Specs below: CPU: AMD Ryzen 9 5900X, MOBO: Gigabyte MC12-LE0, RAM: 4x 32GB DDR4 ECC UDIMM, 2 port 10Gbps SFP+ NIC, 2x 256GB SSD in the mirror on Proxmox and 4x 1TB NVME in the adapter connected to PCIE x16. This mobo supports bifurcation. The server is in the IPC 2U-2404L housing So far I'm very satisfied. Now I'm wondering whether to build two more such servers or buy three MS-01.

⁠​⁠@@oSAendI’m curious about the power draw on your DIY server as I like the BOM you came up with. How does it compare to the MS-01? I particularly like that it has ECC RAM and you can bifurcate the x16 slot to get the most flexibility / performance out of it. As you say it’s really food for thought 🤔

from a power draw perspective alone, this is going to be a huge upgrade, especially with the i5 version. and you don't even sacrifice management, you still have IPMI etc. IMO this box is a perfect homelabber system.

Thanks, good idea 💡

Thanks, I'll add it

Thanks, that's what I'm thinking of doing as well. Hopefully cover it in the future.

All depends what you're after. This is a great all in one, but the 13500t likely is more expandable and easier to work with.

@@Jims-Garage Absolutely, these machines are great for the size. More expandable is also going to mean bigger. I currently do similar to your setup but using micro pc's (10th gen cpu) but always limited by the network which they only have one with these machines you get four plus expansion. I think these machines are just in between the dell & HP micro pc's and a full size pc. My only other concern would be life span of the units and replacement options for when they do fail, I suspect in 5 years if one fails you won't find them around and your back to hashing it out again :) Fair play for taking the jump not sure I would, well not until I watch your videos anyway :)

I'm also a bit torn here, I'm trying to get as close to a "do it all" server for VMs, router/firewall, NAS, all in a somewhat low-power and small package (sure I could just go with a cheap Epyc chip + mobo from China on ebay, but that's a bit much). The Ar900i ITX from minisforum is also appealing, it's got four m.2 slots and a 16x PCIe slot, after doing some digging I've found someone on reddit had success with a M.2 to 16x riser cable on this board, and was able to use an LSI HBA on this adapted slot to add hard drives to it (keeping the onboard 16x slot free), he also made a 3d printed bracket to fit it in a Jonsbro N3 case on the opposite side of the board from the onboard 16x slot to hold this LSI card. I also found another video confirming the Synology E10M20-T1 works on this board, this adds a 10g base-t port and two additional m.2 slots. So as long as I'm fine without a dedicated GPU, this would be quite the beast of a machine that has everything else in an ITX package. I'm leaning back to the MS-01 though because of costs, having a bunch of HDD storage would be nice, but I don't really need it, and PCIe 3.0 U.2 drives are pretty affordable for a good chunk of reliable storage. I'm thinking of going two Intel Optane 1600x 128gb m.2 in a stripe, not a lot of capacity but enough for VM OS volumes for what I'm doing, and the limitation of the 3rd slot at PCI 3.0 x2 doesn't hurt this drive. Then in the future if I really want to add HDDs, I could hook up a USB 3.2 jbod enclosure. And now I still have the 16x free to add a better GPU or whatever else.

@@coinholio470 it's always a difficult decision but both are a good choice. Pick one and be happy IMO. As for m.2 to PCIe, I've been using one for a year in my main PC. I record all of my videos with an elgato card that is attached to my m.2 via PCIe adapter. So far it's been perfect.

@@coinholio470 If you need storage you could always hook up some network attached storage instead or usb (more stable) I think the key thing with these systems is power usage vs network interfaces, you just can't find anything else in the same size, power usage with the same nics (I have tried). My only concern is how long they will last, I expect if they fail it will be an entire new system and this is were the price may seem not so good. I'm still on the fence with purchasing and fair play to Jim for taking the plunge.

Would be great if you could buy the board and fit your own cooling

@@Jims-Garage yes totally. Wouldn’t mind trying to fit 3 in a rack shelf or custom chassis either, that would be an interesting project.

Good tip! I was thinking the same!

Thanks 👍

Thanks, still have a lot to learn.

Thanks for the info! Sadly I have a dual chip tpu and I cannot get it to show using lspci. Do you have single?

​@@Jims-Garage I have the single indeed, because my slot has only one pci-e lane connected, as most wifi slots have. Some have no pci-e connected at all... Only CNVI / CNViov2. If your slot has only one lane connected, at least one TPU should show up... You could try it in an other slot with a M-key to A/E key adapter?

Thanks, it's running well on the MS-01 atm

Hopefully soon? I was on 80/20 for years...

Thanks, I used to do something similar. Little trickier with hardware passthrough and the was-110 ont SFP stick but I'm going to keep experimenting.

@Jims-Garage I believe you can power off a guest and migrate. The same hardware pass through then applies I believe providing you have configured.

Haha, thanks. I'll be covering my vLANs soon

Thanks for the sub!

Thank you. I actually ordered one earlier this afternoon!

@@Jims-Garagewell done :)

It's a cable matters thunderbolt 4 certified cable. Yes, I'll cover configuration in the next video.

Thanks, appreciate the warning. Some experience but lots to learn. Was going to try with 1TB nvme with 550TBW - see how that works out. I've some similar with longhorn for a few years and imagine it's similar wear and tear.

@@Jims-Garage The problem is not really TBW (for good consumer drives) but rather the poor write speeds over time/and both with heat. Also, you really should use a drive with PLP power loss protection. Consumer drives really are just a poor choice for Ceph.

KZfaq ate a reply I had in here, but it was along the lines of considering an M.2 NVMe carrier board (with a switch chip as the MS-01 doesn't appear to have bifurcation support, unless that has changed recently) -- won't be able to recommend the Supermicro AOC-SLG3-2M2 because of this but there are other options out there if you need more NVMe storage. Do keep in mind that the switch chip on these carrier boards will be the limiting factor so look for something beefy that carries all the needed lanes for each NVMe SSD. Finally, backing up what Justin has added -- you really want to seek out NVMe SSDs with PLP -- these are usually designated as M.2 22110 and are all mostly enterprise SSDs as they need that additional room for capacitors~

@@avluis86 thank, I'll check those drives out (I had these because UGreen sent me them).

You absolutely need SSDs with PLP or your CEPH cluster will be painfully slow.

Coming soon! I'll link to the documentation I used and show the process and some testing. It's really impressive.

So far, so good!

@@Jims-Garage can I ask where you bought them please? Was it amazon or minisforum direct?

@@kevinhughes9801 2 from minis forum website, the third from Amazon UK (be sure to click the discount check box for £150 off)

@@Jims-Garage great thank you

Awesome, thank you!

I agree. Put a quad NIC on it and it's a great option. I did consider it for a while.

@@Jims-Garage I have four BD790is clustered like this with a Miktrotik 10G switch (used the PCIe slot for a 10G card) plus two 2TB NVMes per machine. runs like a top. and 4 nodes makes Ceph extra happy, lol.

@@0xKruzr damn, that's some serious horsepower!

Yea, that's the plan. However, I do iGPU Passthrough which means they won't fail over. Others will benefit from it though, and the data will be secure.

A good idea but sadly those ports are only 1Gb

@@Jims-Garage SFP instead of SFP+?

My internet is 2Gb not 1Gb, I'd lose half of my speed.

Nice, that sounds like a great setup

@@Jims-Garage I'm also trying to go low power now so am messing with a lot of arm stuff, have you seen the turing pi2 and the RK1s? I have 4 of those with 32Gb RAM each and did something funky with the M.2 slots. Instead of being conventional I got some m2 to 10Gb adapters and stuck them in so each of the RK1 has a 10Gb going to the flashstor too. Working well, just got proxmox running on them al running in a 2U rackmount.

@@DigisDen those look cool! I like the idea of them, I just find arm lacking in compatibility for many of the things I need. Hopefully that'll change in the near future.

Thanks, Henry. That's very generous.

All 3 machines have been running k3s for a few days, I haven't had a single issue. I wonder if you have a bad unit?

@@Jims-Garage that was my first thought, but I saw a lot of posts with people having the same issue! no crashes in 1 week tho, im hoping im good

Thanks, it kind of is. My fibre goes into a switch and converts it to 2.5Gbe rj45. It's because all the IO is stupidly upside on the MS-01 and my was-110 doesn't fit due its heatsink.

Thanks. I'll discuss some of my vLANs in the next one. There's no sdn, the thunderbolt 4 is used as a network adapter

I'll hopefully be able to feedback on performance in the coming weeks. It will definitely hit the drives harder but I estimate around 5 years for my nvmes. I'm happy with that.

@@Jims-Garage it was 3 or so years for me when I used QLC drives so that is probably just about right.

Thanks - took a while this one!

@@Jims-Garage I was not expecting you to upgrade everything on your network!

Right now I'm focusing on migration and stability, as such migrating existing k3s. I will likely go RKE2, but am definitely going to look into Talos. Want to limit the number of unknown variables at present.

My ISP doesn't have modem mode currently. I'm having to bypass their router with a was-110. I plug it into a switch and convert it to rj45 to save an SFP port on the ms-01. It could go straight into the ms-01 though (if the heat sink wasn't so large!)

@@Jims-Garage cool, didn't know about the was-110. i will investigate. thanks

I have it working without issue (no sr-iov though). Follow my GPU passthrough guide, those instructions still worked for me on 24.04 server.

@@Jims-Garage Great. Do you mean you currently have full iGPU Xe passthrough working with a Ubuntu VM on an MS-01? I track a few forums and I have not seen anyone post anything positive yet. Which guide do you mean?

That was the plan... Unfortunately the thunberbolt 4 wasn't stable on the UGreen NAS, you cannot use both ports simultaneously from my experience.

@@Jims-Garage Well the good news is I can just use the 10GbE connection for now as i'm not limited there. Curious if you planed on using another OS on the Ugreen NAS. I will probably go TrueNAS Scale. I have to see if TrueNAS Scale as a CSI driver.

4! Very nice 👍 sounds like you have a similar setup.

@@Jims-Garage yep. I put the opnsense on dedicated hardware though. virtualizing the router sketches me out. I did it with a 12th gen intel but if i were to do it now I would probably investigate the aliexpress lower power stuff with 10 gig and then do two of them in high availability just for the router.

Check this out gist.github.com/scyto/67fdc9a517faefa68f730f82d7fa3570

Thanks. If it helps I have a dedicated NAS outside of this cluster. I don't intend to change that.

Hi, as shown in the video and description links. Cable matters thunderbolt 4 cable, just a cable. Speed is a little over 25Gb (around 2.6GB/s)

@@Jims-Garage Were you able to fix the IPv4 issue with TB networking? I have similar issue.

Nice, sounds like a cool setup

I'm on kernel 6.8 which I believe has an improved scheduler. So far I haven't had to manually intervene and everything seems fine. I will dig into core utilisation once I'm a little more settled.

Latest kernel tends to just handle it, you can also pin cores if you need. Have to remember it's basically a laptop in SFF.

Will look into it. I'm currently using longhorn but looking into ceph.

Longhorn can be extremely buggy. I have lost databases because of it. When everything works, it works beautifully, but every few months, something fails to start. The biggest issue, in my opinion, is the io speed. If you have a demanding work load or a large db, everything comes to a crawl. This is with nvme storage backing across 4 nodes

@@hansaya yes, I've witnessed many of the things you mentioned. Thankfully I don't have anything that is too heavy on IO but if that changes I might need a different solution. I'm keen to go ceph for a single solution but need to figure it all out first.

@@Jims-Garage One thing saved my bacon many times is using a reliable backup solution. Learned that from my longhorn saga :(. I use Velero and I highly recommend it. Very easy to use as well

@@hansaya thanks, will take a look. Currently using PBS, rclone and Google drive

So far no issues at all. I simply upgraded to the latest version with kernel 6.8. I'll report back any problems I face.

Thanks, John. That's good to know, I haven't yet dabbled with the vPro feature yet. This will be useful

Thanks, I'm reading up on it now.

If you use something like Proxmox, it can handle it as well (when 1 node goes down, it'll just move the VM given you use something like Ceph or another way of "shared storage"). The issue would still be that his WAN goes out but the UDM Pro has that exact same issue (unless you have say, redundant WAN links).

The whole cluster (all 3 combined).

@@Jims-Garage thanks! Thats about 20w lower than my current single hypervisor.

I've had it running for a week without an issue but that on a single unit running latest kernel 6.8

@@Jims-Garage So you have the OpnSense running as a VM - I use it plain vanilla.

@@Jims-Garage Kernel 6.8 indicates you are using OpnSense under Proxmox. Is it true that you use the NIC in bridged / virtualized mode or do you have it passed through to the VM? My reason for asking is that I found a "reset on TX hang" part in the igc driver - but that is present only in Linux, not in FreeBSD.

That's great to hear. Do you have a link to a guide you used in case of any caveats, or is it the standard approach?

@@Jims-Garage There is some caveats atm : you must pin kernel to last 6.5.X version because 6.8 not compatible with the i915 dkms

@@gtarrare thanks, I just read up on that. I'm already running 6.8...

@@Jims-Garage HW accelation working in a plex LXC and Jellyfin in VM (Ubuntu 22)

@@Jims-Garage I tried to left a comment yesterday but YT removed it... I pointed that out as well. Don't run kernels newer then 6.6

You should be able to fit at least 8 drives to an HBA and probably 16. However, without full ECC I won't be doing it.

@@Jims-Garage for homelabs this is the kind of thing where DDR5 for consumers comes in handy. For the lab I'm overall not too worried about ECC. I'd never go without it in an enterprise environment though. Sometimes I miss the hardware raid controllers with their own redundant power. Then I remember how annoying it was when they died.

Nothing yet but potentially. I don't really see a need for it at the moment.

@@Jims-Garage Likewise, although the baked in IPAM could be interesting.

@@kc9nyy would love to learn how to use that myself, personally. (also 73 DE KC2KOA!)

Thanks, this is similar to how I used to have it. However, using the WAS-110 SFP stick has complicated matters. I'm keen to keep trying though.

Yes, if you had a shelf it would just fit a standard 19" (about 21")

@@Jims-Garage might be time to retire my 3 Dell r620s

@@JohnWeland they've done their duties!

Glad you enjoyed it

It's because the networking stack within TB4 caps out at ~20Gb. Data transfer should reach ~40GB. The TB4 used is fully certified in the MS-01 for full speed.

Yes, unfortunately links are not permitted. It's a KZfaq anti-spam thing.

It's fine, you only need 1 kidney and lung 😉

It's £90 / month

@@Jims-Garage WOW that's incredible especially for the UK!

NPCs count right?

The speed test in the video is the best I've done so far, maybe?

Will do, but you can skip ahead and read here: gist.github.com/scyto/67fdc9a517faefa68f730f82d7fa3570

In an ideal world I'd use enterprise kit but for now they're fine. Most workloads I have are small and bursts

I could but it doesn't really solve anything given I'm using a WAS-110. I might be wrong though.

It would, but 25Gb is faster than 10Gb.

I bought these myself, no sponsor unfortunately. I appreciate these are expensive and not suitable for everyone (you don't need 3!), but I see this is a long term investment where I should reover the money in energy saving (not cheap in the UK).

@@Jims-Garage I know, I live here as well. My complete Home LAB is around 150W 24/7 - that includes 3 servers, few Raspberries etc.

What's wrong with a bicycle ;)

@@Jims-Garage Good point, well made :D

Perhaps, but it's about the same price as a high end gaming machine and for me it's far more useful.

@@Jims-Garage the benefit in self-training, setting situations like this up, is well worth it imo.

Did similar setup with R740XD. So not that much overkill. Overkill is nice.

@@TantissTheEmperor nice! That's some horsepower! 💪