A video of a data center power failure sent me here

7:03 "Sorry was that 'yes' or 'not yet'?" "Nawts... Yest" "OK just gonna do the thing"

I see youtube algorithm has chosen this channel

The sheer volume and sound is scary enough. But if you also image the amount of energy, enough to vaporize you in a bright flash, is being transferred and switched, its terrifying.

It is good to see that some companies do a proper full test of the whole system as i have found that some computer bosses will not allow this full changeover but only to run the generators for a short amount of time that does not do them any good without a load and does not give any indication that the whole system will work as designed when a mains power failure occurs,Foolhardy!

This is a channel I never knew I needed in my life.

Very good that you test the generators under full load, diesel loves to be run hard.

3:47 Subtitles say: "James you're sleeping" "Yes" 😂😂

I am just about to get a small UPS for a FreeNAS Server - Now if feels like a joke 😂

Hearing a Gen-Set under load ... the most "Beautiful Sound on the Planet" .. during a prolonged power outage. Cheers ........... thanks for sharing this

Not sure how your large UPS systems work in the UK but here in the USA, UPS systems of this size and class are always online. We call it double conversion where the utility (mains) AC is rectified and float charges the battery bank. The inverter is always feeding the load. As the rectifier and battery are DC, they are simply spliced together. There is no switch involved. The rectified mains DC is normally about 10 volts above the battery string voltage to keep the batteries in charge mode. Of course there is a static and maintenance bypass switch on each UPS but normally the UPS output is never interrupted during a grid failure. Also at this size, we normally run at 480 volts which is a US standard distribution voltage, that is stepped down to 120/208 volts for the loads.

Just found this channel yesterday and it has quickly become one of my favorite channels. Keep up the good work and the videos coming!

My inner "What if it goes wrong? scenario" and "intermittent beeping noises" nerds are going crazy for this video

Nice facility - very clean.

Guy on radio: no, yes! You should agree on clearly speaking. Look to aviation or military.

Who came from the youtube algorithm?

Please please please please make these more often :D I watched one video and I HAD to watch those others too! I absolutely LOVE these videos! Thanks for the entertaiment!

That's really great to get info straight from datacenter guys like you, keep up the good work !

Great to see this stuff. Thanks for showing us this testing.

Awwww those are some cute little generators!

The transfer switch we had was a 3 pump up then turn the knob. Them BAM! It made me jump too like you feel the amps. Then the generator diesel pumped out black smoke for a few seconds while picking up the load. Zero automation by the way.

Cool to see this being done. I work in a DC as a hall engineer. Changing Server motherboards etc. So i hear the tests happening but we never get to see whats being done.

Thank you, this is absolutely golden information

Very informative ! Glad I found this channel.

The dude on the radio sounds like a imperial droid

Thank you very much for fantastik video👍👍👍Greate job!

A friend of 20 years works at Services Australia, Centrelink, Medicare, Many more Data Centre.. he blew my mind when he told me about a Direct Fuel line that runs from a Petrol Service Station close by, Directly to there Generators..... ( He said the system must never go down )

Love your videos guys. Keep up the good work!

Really cool video guys. Thank you!

great work guys!! scaring during CB operation happens to me everytime too :)

Thanks for sharing this .. Cheers

I felt like George Russell was giving me the tour

Man, that's a lot of power out of that tiny engine. Ours is .9 megawatt and is easily three times that size. It's the one part of my job that I truly loathe - monthly engine runs. It is incredibly foolish to stand in front of those ATS panels when the transfers takes place. I flip the switch and take off. I can get out of the room into a safe spot before the transfer takes place. Hopefully this guy won't end up in some arc flash accident video later on down the road. it scares me just watching him in this video.

Just found the channel and love it. What a great idea shows transparency within the industry, I work for NTT Global EMCA at my site with have 40 Gen-set at 1250kw total load. all of our network is via PLC so we can export back to the grid. keep it up guys

Great Insight Video!

linus tech tips data centre edition

Great video , very well explained ⚡️

So, can those guys say they drive a V10 at work?

I have recently carried out a series of load-bank tests on mission-critical sets from 40kVA to 2000 kVA. The customer used to run the sets up once a month to make sure they started, but never ran on full load. A lot of them were over-specified, resulting in carbon build-up, sticky valves, clogged silencers etc. My job was to run the sets up to 100% of plate rating for 4 hours, to make sure that they do not fail under operating conditions. After the usual checks and inspection, staged increases of loading, (5 minutes warm-up, 5 minutes 10%, 10 minutes 25%, 10 minutes 50%, then the balance of 4 hours at 100%), were implemented. One set would only run to 80% without overheating, another had a failed turbo-charger bearing, another had a radiator leak that only showed on full load, etc, etc. Once the test is complete, a 10 minute no-load cool-down period is done. Once the customer was aware of the issues, and they were dealt with, their decision was to load-bank the sets bi-weekly. (It's a great job for a winter's day; parking 20 feet in front of a 400kW fan-heater certainly keeps the van warm!)

Would like to see a video on the type of, and how you test you're fire suppression system(s). Love these video's as they give us more insight on the daily workings in a DC, and see that the people working there enjoy their jobs.

love those castell locks

I work in a DC myself as an engineer and it was cool to see the differences and similarities between my site and yours. Wish we had radios though!

Nice job, just a quick clarification,why ATS -6 gen takes too long to be up. usually it's less than 12-15 sec. but here it took more than 30 sec .

i know this may be a bit of a secret topic, but i would love to see a video about some of the security of the DC, both network and physical, monitoring and detection, prevention and response

Ash, simply superb 😎

extremely interesting video guys, keep it up 😉

you should've said "bravo Six, going dark" on the radio, the other guy sounded like makarov on the radio. You getting scared gets me all the time 🤣🤣🤣🤣

Gracias por el video ¡¡ saludos desde Bolivia

these sites are amazing. I take care of a couple of unmanned ILA's for a national data line and the system there is entirely automatic, we do a test every 6 months, even though i've requested a minimum 3 months due to the nature of these sites on the network.... BUT its absolutely terrifying hearing the ATS click, especially since mine is a 3 way between mains, all open and gen. the click is powerful enough to shake the hut and you can feel the transfer through the earth.

Love it

Thanks this was interesting

Nice setup guys

Thanks very instructive videos

Great to see this being done properly! Far too many places run the generators unloaded as a test.. :/

5:26 yes that contact made sure you knew it was kicking in.

Ayyy, Edgar’s Water, the same water dispenser fairy that we have at work!

That's a nice PHP weathermapper graph you have there :)

Interesting to see this being done this is something I get involved in . Just wondering what if the electronics don’t work are you able to start and sync the generators by hand and switch the system by hand ?

I like your DCIM tool

@ 5:27...get used to it man...LOL!!!

Good good good video!

You can tell this dude is on the verge of a nervous breakdown.

you guys are the bomb,

thank guys

The longest wait of your life when doing mains fail, 30 seconds waiting for generator contactor /ACB to close, a million possibilities go through your mind 😂😂

Hello really good video

At my company we use a manual diesel generator which means we have to manually swap all distribution boards one by one to the gen which is fun lol, jealous at how simple and automated a data centre can do it which makes sense when you cant afford any downtime whatsoever

Great video guys !!! I'm very interested what DCIM you using, thanks in advance :)

I feel like slamming a door would give this dude ptsd

2:00 i think it run at night by itself and return back to not get cought, smart energy working.

i love how this all comes across as a walk in the park during the test. I've been on site during a power failure waiting for a generator to kick in before the UPS dies and i can tell you its one of the most stressful times in an IT guys life....

Love it! why do you need this for such a long time?

Do you inform you "grid" about that test? I wonder how they compensate the loss of the load in the grid when you're doing the tests. thx

I swear I thought I heard him say "not yet" and then you say okay, let's turn the key. lol

Regarding the implemetantations of the generators, was there a handover, regarding what procedures to follow when it comes to testing, periodic checks, and switching over from the mains to the generators?

I have a question, if you guys are still watching comments. Why put the UPS outside of the load for normal operation conditions? What's the benefit of doing so? It almost seems like it'd be much more expensive to do it that way, but I'm sure there's some benefit to it. I'm more used to mains/gen > UPS > load (with a automatic switch between the mains and the generators in case of mains failure), since the company I work for uses the UPS not only as a immediate backup but also to clean up the power going into the server racks.

Those big ATS auto breakers scare the $#@^& out of me too! every time!!

Pff small gensets and breakers, on my previous work they have breakers of 3kV/150A on the small blowers they make a BANG if they switch. The gen sets are automated synced with the grid, 1 genset is 3kV/2000A. This is nice to :)

a great morning at the power plant

Running Cacti for monitoring I see :) Good olde reliable cacti

i need this for my off grid cabin data server

at 12:21 why did you guys blur out the stats of the gensets doesn't make much senseI wouldn't think to know what RPM and temperature and frequency is sensitive information

I like your videos, I am just wondering if 10v1600 series is enough to get the energy for the hole data center. They have only 700kva and for example 20v4000 series has 4000kva output. Maybe it makes sense to replace all small once to 2 bigger once :)

Hi, can you show a video of the power distribution, Switchgear, UPS, PDU and connection to the racks.

Good video; you think it’s loud when it works properly; I was doing a test one time and a a switch jammed; so loud, and you could feel the thud; I jumped off the ground; I had to abort the test that time.

Over by the ups was the vent on top for battry off gass or for heat offput

Interesting video. Thank you! What are the green dongles in the background at 12:00+?

This may look simple but testing all these generators, ATSs, UPSs, and alarm setpoints WITH IT LOAD is very ballsy. Well done.

Maaagic!

How many gensets do you have?? I saw 3 of course, but do I understand you have 3 more?? I have run outdoor concerts on 10 Mw but this is very nice indeed, ty for the info

1600 series MTU 😎✌🏻

good jobs

*stares in procedure use & adherence and place keeping*

Lookup what arc flash is. You should have protection gear. Good reason it makes you jump.

Good video - I used to do monthly genset testing on several large data centres (From 3 to 12 Meg). Once had the mains actually fail just before I conducted the test - saved me some work. 👍😁. Gensets on the 3MVA setup were V12 50 litre engines, the 12 meg site had 6 x 2MVA - 78 litre, V20’s. 👍

Great

Dang old switchover scares

MTU is best Engine...and 500kva is small enough for small datacenter...but in our dc we use demand of 32MW per day...and 2.25MVA D.G set N+1 15+ D.Gs its a HT alternator which means 11kv alternator we are running a large amount of IT load so there are 1mva ups like N+1 30+...any way good video...

WebNX should watch this

All fun and games until you have a massive arc fault and all that shit blows into pieces, but you already know that. Have a nice day.