ONIE is love, ONIE is life. Commodity switches are the future.

People are afraid of 5g, and here you are handling 100g without any protective head gear!

I love how cheap network hardware can get. You can pick up 40GB Mellanox switches under 200 and the NICs under 100. Great fun!

Did I just watch a kid in a candy store video?

I love this content, keep it comming!

Good news, looks like connection speed is slightly beating Moore's Law. Based on the speed difference between the two you'd expect the old modem to have been made in 1964

They don't make em beep-boop like they used to though.

I am starting to understand why I watch all these videos. They are just outside of my capacity to understand, but I just feel smarter listening to Wendell (I really hope I spelled your name correctly) talk about things!

Heads up, 400G is not 4x 100G, its 8x 50G. The switch to 400G changed the physical port from a QSFP28 (rated for 4x 25g) to QSFP-DD (rated for 8x 50G). Along with this physical change, the signal modulation was also changed from NRZ Modulation to PAM4 Modulation. You can buy a 400G optic that will combine 2x channels per lane for 4x 100G transport/breakout. But the originating channels are still 8x 50G

do I have to pay royalties for reusing the Mt. Stupid quote? That one deserves a T-shirt for sure.

nice video, good editing, great content. thanks

From a Network Engineer point of view I would change the 100G port distribution, 2x100G for the inter-switch link and 1x100G for the Storage per switch, why? usually on this type of deployment MLAG is used and if that single 100G inter-switch goes down it will wreak havoc where the best scenario will be that a whole switch will shutdown all of its ports

This channel is so cool.

great scott! I have found my plateau! so, a great world 4 software processing!

400G is only available on QSFP-DD ports (which is actually 8x50G lanes, not 4x100). AFAIK, 100Gb single lane SERDES aren't commonly available, yet.

I'm looking forward to replacing my 1GbE switch in the next year or so, but for now I'm fine with a single 10Gb link in my home network between my server & desktop. The two machines are only a couple meters apart so I got a pair of X520-DA1 cards from China (knockoffs, not genuine Intel) and connected those with a cheap DAC cable. A proper switch will be a nicer setup, although considering my main storage is spinning rust in raid6, I may not be in a rush to go beyond 10G. The switch I want is something like: 4 x SFP+, 8 x 2.5GbE (4 w/ PoE). I've been leaning toward Mikrotik but I'll consider Mellanox too.

Wendell, long time viewer since teksyndicate days and still a noob here. I am trying to dabble in making a clustered SAN lab but my head is spinning understanding Mellanox through iSCI, iSER, RDMA, Infiniband, iWarp. Would it be too much to create a video explaining the lay of the land?

I don't even have 10gb cards to play with and Wendel has 100gb cards to play with :O

I'm glad you mentioned how slow the cloud can be, comparatively, even if it's on great hardware. Also cloud equipment is shared, with bandwidth split between customers. One thing I have not tried is the "dedicated" cloud hardware options, when you pay extra for bare metal equipment. I wonder if you can also get dedicated links between all the hardware, since different machines host CPU, drives, etc. Back when I did cloud things, it just chugged (compared to a high-speed desktop) due to the shared resource model. I remember several blog posts about "cpu steal" where another tenant could steal cpu time and your VM would be slow. There were also several blog posts comparing speed between cloud providers, some with better Iops, etc. Maybe this is largely solved now, I've been out of this space for a few years.

The problem with Mellanox 40 and 100Gbps NIC is actually availability. I got some quotes recently, and it is 2 months waiting list, plus need to sign no-resale agreement.

Just a small question... @ 6:07 kzfaq.info/get/bejne/oqeba5x6yZOZgok.html it was supposed to be 12x SFP28 which is 12x 25GbE and it got mixed up ?

These numbers all sound incredible for a homelaber like me, but can imagine busy servers needing it. Love the videos :)

so cool... iplay around with some 10gbit X3 still not easy to use all the bandwith on a standard NAS, even with ssd.

I'm old enough to remember my dad being elated to get 2400 baud running on his hobby system. Sometimes. It was hard to tell on the amber monitor.

Ahm, about the acoustic coupler is it possible you were talking about 300 baud, not bit/s? Happy new year.

Looking forward to that openshift video

I understand that at the ends of the DAC cable there is an EEPROM and it says who I am. Question - does the NIC card ports reading the eeprom compare it to the eeprom on the other end of the cable? I connected 2 DAC cables through a docking board (QSFP28 QSFP28) and can't get a connection between NIC cards I don't understand what's wrong.

I'm curious if the 100G/40G cards behave like a single 10GB/s transfer or if they behave like a link aggregation of 4 * 25G/10G? I'm looking for the fastest access for a single desktop to a single server for single file copies and access. So it's really 25gbe or 40gbe that are the two specs that are interesting and in budget.

Did that open shift video ever come out? (mentioned toward end of video)

Used these cards for some time; most Mellanoxs are a good card for their custom SoC stack and ToE with DDPK, not sure why would you use them for run-of-the-mill setup!

there is a way to have a good bus connection between (4x NVIDIA 3080 TI) and (8x AMD 6700XT) using this technology to use in AI training ?

9:00 Link is in fact not in the description ;)

what are those beautiful panel screens in the background open to firefox?

Which server chassis model is shown in this video?

So I have 3 servers, 2 are data storage servers exceeding 1250MB/s read/write, and the last server is for running VMs and pfSense. The VM server has a dual 100GbE NIC, with one connection to each server...any reason a switch would be better?

I'm holding out for 1Tbps

I came across this random piece of hardware while browsing the web and decided to youtube it. Asking myself, who is this relevant to? I am now in a whole new rabbit hole. So cool even tho I don't understand much yet.

This looks awesome! Anyone got a recommendation for a 24 port 1Gb switch with at least 4x 10GB ports that won’t break the bank?

LAN party over acoustic modem anyone?

Is that a 300 baud modem from the 80s or from the 70s? :)

Wasn't the modem CPS and not bits?

More TrueNAS SCALE video please. I am baning my head on 40Gb ConnectX-3 without RDMA.

Linus is drooling right now.

Hey Wendel, I take offense to that tomato fruit comment in that image. Tomatoes are totally fruit, and why is that useful to use as a classification? They ripen, even after detached to the plant, you can use them to make new tomato plants, it's awesome to learn about biology (you would not want to take wrong or the genitalia and leg of a person so why a plant?)

If i look around, that switch cost around 3000-7000€ piece, depending if new or refurbished. All this onie switches are expensive as hell.

But what about cx-6 cards?

100g fiber isn't ethernet? It is unless you are talking about fiber channel, just like 10g, 40g or 25g SFP ports are just ethernet.

Can someone please educate me. 25gb are hundreds of dollars online. But eBay has a bunch of "40gb" cards for under $100. I see a bunch of postings for Mellanox MCX314A-BCCT for cheap. Is there some sort of catch?

Dell was pushing 25Gb minimum connectivity four years ago. Now they're working toward having 100Gb as their new minimum standard. Given how their storage arrays are designed to function, with clusters of storage nodes aggregating high-performance NVMe flash RAID arrays, the array itself should be more than capable of providing multiple 100+Gbps data streams, sustained. I mean, the nodes in the clusters themselves are interconnected using no fewer than a pair of 200Gb infiniband connectors each, if not more. With the introduction of NVMe packetization for network transport protocols, the floodgates have really opened for high-speed networking well in excess of 10Gb.

I have no idea what you are saying. I am a Desktop PC builder. This enterprise kit is so far beyond me. I used to connect to the MF via a 300 baud acoustic modem. Those were the days. I would love a series on the components of an enterprise solution. But keep it at a level so Krista's dog can understand it. Cheers

We called them acoustic couplers

Wendell, review the NVIDIA CONNECTX-7 that's 400Gb/s!!! That should be fun xD

small g, we don't wanna look like we're showing off

This makes my 5 port 10/100 BaseT unmanaged switch feel inadequate. :(

The connectx-4 is obsolete with esxi8

Heads up. Mellanox RDMA is not standards compliant. It doesn't work with any other vendors RoCE. It breaks the key tenant of Ethernet... standards, interoperability, and open.

I thought "100g" was 100 gramms. To warn the users how heavy the card is ;-)

And here I sit. Thought those two fibers dangling under my ceiling running 10Gig were fast...

Cries in 1Gbps networking with 10Mbps internet (on good days)

ok completely unrelated but that BGM is basically if the mother series had to do "The Lick" in band class

I think I'm halfway between Mount Stupid and the dip that is after Mount Stupid.

nvidia bought Mellanox March 11, 2019

All my Linux and BSD ISOs distributed to my close friends and family quickly in my network of fun?

I thought he was talking about some drug, 100g doses

QSFP112 Please.

I would love to get that Dell switch...but not for $3000 off ebay....oh man scalpers are even in enterprise stuff...

It blows me away sometimes how persistent work has been using 95% 1Gbit.

Wendel your fired !!! Joking

I’m so glad Wendell stopped correcting himself “mellanox I mean nvidia”

We only need 333,333,333 or so of those phone shoes to be equal. parrel is the future

Til, 2.5x higher bandwidth is orders of magnitude. :-p

Thank God I am an idiot and I can just think I am right all the time. Pretty sure the DBA I have been working with thinks I am.

I hope I found this video before. I was fighting with my dell switches to get some of the ports from 25 Gbits to 10 Gbits and I wasn't able until a friend told me about the 4 ports groups...

Dammnit, stop calling it linIx, its LinUx, there is a God damned U there in the end not and i. Happy New year btw.

Who stold my first modem? Mom, hang up the phone, you kicked me off CompuServ... Little did I know then how far we would come. Now I understand why my dad was so amazed with personal computers like the C64. It was miles beyond a typewriter. Of course, there are very few places that have a 100Gb connection to the world, but like everything else, eventually someone will ask you why you're still stuck on slow af 100Gb to your house when its barely fast enough to surf the metaverse and certainly not in full 5 sense mode. Please bury me before we have metaverse fart jokes that you can smell!

7:26

Lol I can barely make use of 10gig as-is

About chips shortage... 😅😂😂😂😂

It's still almost impossible for single server to deliver more than 10 Gbps application layer throughput. Over 10 Gbps is over kill for application server such as web server, database, etc.

this video would have saved me a day if I had it 5 months ago ;)

One hundred billion vs 300. Needless to say, these aren't the spartans you're looking for...

My first mofdem was a 300 bit per second one... I swear I could type faster . Jajaja

Nobody wants to write an operating system? Then why does routerOS, HP Comware, etc. exist? Microsoft has an allegedly FOSS switch software named Sonic but MSFT software is always a trap. These Dell switches are cool but Nvidia now owns Cumulus and will not be supporting any Broadcom-based chipsets. These Dell switches are Broadcom chip devices.

In 40 years time someone will be making a video holding up a 100gbit card and laughing at it while they show their 1 petabit NIC. In 400 years time there will be no concept of data speeds as we know it. They will have NICs hardwired into their brains.

You watch, with all the r/selfhosting going on everywhere. Pretty soon you're going to see server racks and blade servers sold to end users for their IOT and micro services running at home. People will start buying up solar and house batteries for powering these beasts (And Texans for other reasons) Sys-admin skills will become part of the school curriculum, and the humble home desktop PC disappears as every house becomes it's own VPS, perhaps with redundant backup to family in other cities..

Drools ~~~~~~~~~~~ wags wags wags wags wags wags wags wags wags

Heh I remember b4 300 baud... and when 300 came out it was blistering!