Wow. Two takeaways from this video. The first is as ever, that your content and your commitment to your international viewers is through the roof. Thank you for the English content Roman. It's one thing to do a de-lidding video in two languages but this in unreal. The second is this place! Amazing attention to detail in every single thing. Just fascinating. I have also learned that perhaps my hardware is not as top-tier as I thought /s

OK, hands down this is the most awesome data centre tour I've ever found on KZfaq. My mind was blown multiple times: the non-crossing infrastructure, spill gate for battery acid and the reduced oxygen level.

Dr Oetker has an IT business?!! They really do absolutely everything, not just pizzas.

This is fantastic! Wow, I'm blown away by that site. I work in a datacenter myself, or at least I thought it was before I saw this.. Lol. This is a completely different level than what I'm used to. Really looking forwards to part 2. :) Keep it up Roman, this was an amazing video.

it's so fascinating how many different technologies (and of course therefore technological experts) have to work together perfectly to make an operation like this work so flawlessly. Awesome video!

Thank you so much for everyone who made this project possible. I've been waiting for a data center walkthrough for the last 5 1/2 years that had this much in-depth knowledge.

you've outdone yourself on this video. thoroughly enjoyed the tour, very interesting all of the aspect to make a stable room for the servers to run smoothly. thank you for the very unique chance to see behind the scenes of a data-center. danke Roman. ;)

Congrats on getting in there to see and touch all those things and thank you for sharing it with us Roman. Seeing the datacenter up close was awesome, seeing your excitement as you described it all, even better.

That was a great tour, amazing to see into a modern datacenter.

Stunning video Roman. Totally amazed at the complexity of the facility.

Thanks for the tour. Love the details you took care about. Can't wait for the continuation.

I like shows like "How It's Made", but this video is even deeper than that! Major datacenters are a huge engineering achievement.

Its just amazing that you let us watch how this industry works, ty Roman for releasing this for the international community

This is one of the best videos I've seen. SO freaking cool they let you have that level of access. What an incredible place!

Thanks for the tour, it reminds me of all the equipment I used to work on in my career.

Roman: Thank You again for the peek behind the curtain! Fascinating! And Mahalo to the crew at the data center as well! Just excellent in every way!

Top notch content. Extremely interesting. Thank you for this glimpse into cutting edge enterprise stuff.

These videos are amazing Roman, great work. Thanks for taking the time to share all this.

Amazing video. I love these tours you're doing Roman.

Really enjoyed this. Thank you so much, and I hope you gave the owners a huge thank you, getting this level of access is pretty much unheard of.

Most fascinating info about datacenter! Just amazing video! Mad props to Roman for doing this in eng and de!!

I would like to thank you truly and sincerly for this video.. I watched it from start to finish with full attention. I am soon starting a new Sales job at a DC company. Quite excited about it because It's going to be my first expirience in the DC industry. This video is very educational and helped me so much understand the nature of this business. Again, thank you so much for all the hard effort you have put to help people like me. I appreciate it and I wish you more success! Looking forward for part 2.

I like how everything there is clean and organized

WOW! Amazing how cleanly laid out everything is laid out!

That fan spin up 😮 That SSD storage 😅 LTTs $1m unboxing looks quaint all of a sudden.

Always a pleasure to look inside a datacenter. I had the chance to look inside a very small one in the basement of an IT service company in Cologne with my CompSci class back in school. We had a bit of extra luck, because the operator decided to reschedule the test run of one diesel generator (by one or two days) so we could witness it live in the gen room. Sadly, no blade servers, POWER or z-Series there, just standard 19inch Intel x86 servers, Ethernet, and fiber access. Eagerly awaiting part 2.

Awesome video. I miss my data center days working for EMC and this brought back lots of memories.

Cool video man, appreciate tat you took the time to run through the infrastructure side before hitting the tech. Most people don't realize how much engineering goes into housing these things. I work in the hyperscale arena, but those smaller enterprise/hosting sites will always have a special place in my heart!

i loved watching this vid - so amazing

What an amazing video! Very very cool! Thanks Roman.

Thats out of this world, even when compared to other data centres that i have seen tours of. Best. Content. Ever.

another amazing video.. nice to see this kind of insight perspectives!

Data center without wearing ear plugs? Fun times...

Amazing video!! So grateful for the gents who allowed you to tour and record the facility 🙌🏾🙏🏾

This facility is insane. Very cool content Roman.

Thank you so much for going through this whole datacenter, this is so interesting

Wow what a high end facility. These guys know what they are doing, and to think this started in the food industry

Love that kind of content. Please try to find more companies that will showcase thier hard work.

This video is great. I have been sharing it to my colleagues that need to go to Power Plant sites, you have "mini data rooms" over there and you have the same cooling, fire detection and energy backup systems. Thank you for the detailed tour!

This is a great video. I was surprise impressed with all the technology behind to run data center. And the way you explain everything was fantastic. Thank you

just WOW . thanks for taking us in this great tour.i appreciate the efoort you had put in this video. really enjoyed the video. sehr informativ auch

Data centers are always fun to go through. Some things I have come across which could make for better, more efficient data centers than the one you went through, and I am sure you have thought about some of this with the videos you have made, but some of it may be new to you: 1. 380V DC standard for power delivery. This is +=190V nominal, 300V to 425V range to the servers. What this standard does is after the transformer step down to 480V AC, do one conversion to 380V DC nominal and have those batteries hooked up in series and parallel to equal 380V DC nominal. From here the power goes over power rails more efficiently, as DC travels over the wires more efficiently than AC, straight to the servers. The server power supplies have basically half of the hardware in them as they skip the AC/DC conversion, which takes a lot of power electronics to efficiently and cleanly convert AC to DC, instead going straight from 380V DC to 12V DC. This is far more efficient and uses far less hardware than traditional AC power delivery to the server. Something like 30% space savings in data centers that do this (and I spell out data center long to not confuse with direct current) while getting a big boost in efficiency. You mention the efficiency of data centers at the end of the video, which is an important metric and this is a way to raise the actual efficiency and even save on costs when this tech is done at scale. 2. Considering how much Germany relies on renewable energy, something else that I think should be done, especially when doing a 380V DC standard data center build, is to swap out the batteries in the battery room with LFP batteries and build for at least 4 hours of storage. As you may have noticed, the space used in the battery room is not that efficient and they built around the notion of lead acid batteries spilling, which you will also see in old telco buildings where they built everything to run on 48V DC, which takes some crazy big bus bars to do at that voltage level. The idea of having these hours of storage is you can balance against the green energy grid and may even stop taking power from the grid for a while when the electricity prices / demand is the highest. At least when there is variable pricing involved / you make a deal with the power company to help balance the grid, such a setup can save / make a data center a lot of money as you will already have the conversion hardware and the big power use case, just add more batteries to your battery room. At this those diesel generators don't need to run as much during a power outage and can have a much longer grace period to warm up, saving on electricity to keep them warm when you have a battery system with hours of storage; you just make sure to have a certain minimum storage to give the generators more time to warm up. LFP batteries in the data center is also a good thing as modern LFP batteries will last for decades. Also by the end of the year (2022), Germany will have a large LFP battery manufacturing facility in operation run by CATL, one of the biggest names in LFP battery manufacturing. So you will likely make the batteries used in the data center in Germany. 3. Getting to where your expertise comes in, liquid cooling the high powered components in servers with a negative pressure liquid cooling loop. A number of data centers do this, especially for supercomputers, and it is extremely efficient and ironically uses a lot less water than the system you showed. The reason for this is air is a very poor carrier of heat, so you have to cool the air to a certain low temperature, much lower than the temperature of the components you are cooling, or else the server hardware in the racks will get too hot because the delta-T (change in temperature) with air is high. With direct liquid to the hot running components using water blocks, you can run the coolant at much higher temperatures as the delta-T is much lower. These much higher temperatures get into a hot summer day where it is say 37C outside is cool enough to not need any extra cooling measures such as evaporative cooling nor air conditioners. Some data centers get down into a PUE of 1.1 where this one you specify as 1.35 and the bar is 2.0. So yeah, efficiency can be better, granted this one is pretty good, granted they use a lot of water, which gets to be a problem in some places where there is not enough water to go around. This problem is getting worse with global climate change, so this thinking about evaporative cooling has to be changed unfortunately. 4. A number of data centers are moving to back of rack radiators. 5. Also in your wheelhouse, use of liquid metal thermal transfer compound and high W/mK thermal pads. The idea being the more efficiently you transfer the heat to the heat sinks with a smaller delta-T (change in temperature) between the die and heat sink, the less you have to work to keep the die at or below its max target temperature. Data centers are built around keeping the components down to a certain target temperature at max load and when you throw in all of the inefficiencies of low end thermal transfer compounds, IHS's (Internal Heat Spreaders), air cooling, and heat buildup as you go through long, high powered servers, you end up spending a lot of energy and often water to reach that target inlet temperature. Also those super noisy server fans use a tonne of energy to spin that fast and get into significant air friction heating, so if you carry most of the heat away with high density water blocks where you don't have to work as hard to move the more heat dense liquid coolant around, you can use much slower, more efficient fans for the remaining lower powered air cooled components. Anything you can do to allow the target temperature to be higher reduces your PUE and/or water consumption. 6. Shifting gears a little, use of ZFS RAIDZ in the data center. While I have used ZFS RAIDZ level 2 on Solaris in the data center primarily on mechanical drives with SSD caching drives, ZFS under Linux and FreeBSD has gotten a lot better in recent years and supports TRIM on SSDs. RAID controllers do not support TRIM. If you have ever done SSD RAID arrays, those SSDs take a beating when used with hardware RAID controllers, especially as hardware RAID does not support TRIM and in general uses the SSDs in a very write intensive fashion. ZFS is setup a lot more intelligently in terms of how much writing you do or I should say it is one of its optimizations at a slight cost elsewhere (space usage), and TRIM support is icing on the cake greatly reducing write amplification. I would venture to say that ZFS is a more reliable and flexible storage system than hardware RAID based on my experience with it in the data center and my usage of it under Linux and FreeBSD. The thing is where a data center may go for super expensive 10 DWDP (Drive Writes Per Day) SSDs when using hardware RAID controllers, they may find they can get the exact same job down with much less costly 1 DWDP drives when using ZFS RAIDZ. I mean the improvement you will see with ZFS RAIDZ level 2 over RAID 6 is huge. As a bit of a side note on this RAID level, with RAID 1 mirrors sometimes the mirror fails before you can rebuild it, causing data lose and RAID 10 just amplifies this problem by adding more mirrors to an array. In a data center there are enough drives to where you will see this happen, it is a guarantee when you are dealing with this many drives. RAID 5 also amplifies this problem as you add more drives to the array. RAID 6 is a lot better at not losing your data to random physical drive failures as you still have redundancy with a single drive failure, so the occasional second drive failure and just sector loss doesn't kill you, granted in a data center as soon as a drive starts losing sectors, you replace it right away. (At least any good admin would.) RAIDZ level 2 is basically ZFS's version of this, except a lot better in terms of data integrity and recovery capabilities. Standard practice where I worked is to have 8 drive vdevs and just add more vdevs to a zpool when increasing storage. In other words you have an 8 drive array with 2 of the drives for redundancy and then you just add more of these arrays into a single logical 'drive' / storage pool to get to the desired storage size. If you have ever dealt with hardware RAID enough, you start finding there are ways to lose data in a more traditional overall setup where there should be ways to make it better and not lose that data. ZFS RAIDZ level 2 is a great answer to these issues. There is a lot to explain here, but this is something you can read about and then this already long post doesn't need to get a lot longer and you can see why ZFS RAIDZ with direct access to the drives is just better. It is also cheaper as you don't have to spend money on all of these fancy RAID controllers, instead just need simple HBAs (Host Bus Adapters) to access the drives. 7. I was a bit surprised with all of that fiber you saw, there weren't any specialized high speed fiber connections such as InfiniBand. (Maybe you saw InfiniBand, but just didn't know what it was?) I suppose this is a thing when you go to a bank's data center, they are going to be a bit more conservative in how they setup things than say a scientific supercomputing center and their hardware suppliers are going to be a bit more traditional in their offerings, so you just won't see some of these ideas to make the data center even more efficient than the setup you saw. The most radical stuff tends to happen with hyperscale data centers. It is just these are even harder to get into as the operators tend to be a bit more secretive on how they do things.

Really cool- thanks for showing us!!!

Super interesting and detailed tour! Thanks :)

This is amazing! Can't believe I'm just now finding this channel. You deserve a few hundred thousand more subscribers IMO

Wow thank you! This was amazing to see such an immense complex piece of kit. Absolute privilege so see and understand through this video

Thank you making the video. I didn't expect to see mainframes there

What a tour, great work. thank you.

truly facinating would love to see more dc tours

Great tour! Thank you

This was absolutely fascinating! My hats off to the people who build this.

As an electrician doing a lot of commercial renovations and new construction in San Francisco, I've seen a lot of much smaller server rooms, but this is next level! Very impressed with the electrical work (power in particular) , btw. Very clean and neat. Thanks for the absolutely impressive tour! And thanks to the company for giving you such access.

Awesome work, definitely put together with a high availability mentality. Thank you and keep it up.

That's some amazing setup. Appreciate the detailed video, this is the type of stuff you just can't get access to unless you're in high-end hosting or if you work for a place like that typically. The redundancy in every aspect, mildly out of my $ range, but very, very impressive how they managed to do this with the power efficiency stated.

That was the cleanest, best laid out, DCIM, SCIF, I have ever seen. Great Content :)

Very cool to see inside this DC. The company I work for have equipment in two datacentres and I am part of the DC team so get to visit often and I'm impressed with some of the stuff they have going on here that is not seen in a regular DC such as the oxygen reduction stuff, I didn't know that was a thing either!

This was incredible. Thank you!

Mind blowing stuff. Thanks guys.

this video was very interesting. even though I know nothing about data centers, you explained everything in a way I can understand. thank you

Thank you very much for this video tour! You're correct in that's it was incredible how the scale of computing and storage capacity is run in this datacenter! I hope you keep these videos coming and cheers to you!

Have 14 years Data Center Experience working as an operating engineer taking care of equipment. Great video and explanation of everything. Really enjoy seeing different Data Centers.

I just love the attention to detail. If I had stayed in the industry after 2001, that is the kind of place that I would have wanted to be working in. Thanks so much for the tour. Loved it. Oh and Dr Oetker make the best pizzas, we have them here in the Netherlands too. The last section with the backup tape drives, I would also want a further backup of in a separate city. Back in the day I argued for data to be backed up from Manchester, Cardiff, Glasgow and London in each of the other centres. My bosses thought I was being OTT.

Very Enlightening. Thank you :)

Nice and educative, thank you and thank you to the IT data center dudes.

Wow, awesome content, thanks!!!

Wow i am impressed at how professional and advanced everything is inside this data center.

this was great! We have been using ibm power for decades. We have both Z and power. Rotating out power 8. Right now i am implementing 2x 1080s and will be migrating from 980s to the 1080s. Running IBMi, AIX, and RHEL.

very cool, thanks for the tour!

Fantastic video, really enjoyed it!🤯

That was absolutely amazing to see! I never knew how advanced everything was.

Thank you for producing these in both English and German!

Excellent video, thanks very much for making it

every time you step out of the datacenter it suddenly feels so peaceful

Okay this is awesome. I absolutely love this kind of content, the engineering challenges and super cool tech in high end datacentres is just so cool

nicely efficient building , awesome video.

I always appreciate the effort you put into publishing your videos on both your main and your EN channel, but this video goes above and beyond. Doing the full tour once must have been an ordeal to organize, but convincing them to let you go over everything twice really takes your content that extra mile.

So Cool, So Clean, and very organized DC...

this is amazing content! This DC is very impressive compared to some I have seen in Australia.

Great Video! I could only wish that the DC's I've been in were that well equipped and organised.

Awesome tour, impressive the amount of information shared! Did not even imagine such complex and effective redundant cooling exist. German infrastructure tech at it’s peak not to speak about the servers side! It’s at the level of banks/telecom data centers terms of redundancy/safety. I wonder if they have geographic redundancies as well eg synced across country. Definitely I will look differently at the I Dr Oetker pudding in supermarket after this :)

The gas suppression and aspirating systems was great to see for me. I work in that part of the fire industry here in Australia, and was interested in how ( and what brand) the systems are used overseas . I have set up similar systems for data centres here.

Amazing tour and lessons thank you!

I don't usually comment on videos, but this was amazing. Thank you for this great content. Seeing those IBM Power E's got me hyped.

It amazed me how little the infrastructure in a new data centre has changed from one I worked in over 20 years ago in Australia.

This is the coolest video you have made to date. Nothing else come close to this imho. Wow, just wow. Thank you for this. To se you geek out like that was awesome. I bet this was one of the coolest thing you have done in a while Roman? Love from Norway.

This is super cool. Im a fan of this sort of content 👌👍

Love this! Now we're getting into some crazy hardware.

Cool video. IBM is such a weird and fascinating technology

Just one word to say "Awesome" 👍✌😁 Thank you for this nice go through.

Absolutely fascinating !

I used to do maintenance testing on Data Center Power Systems. We had been using bigger and bigger batteries and had similar issues with the hazardous properties of them. Now, A 1.2 MWatt Diesel Generator (Caterpillar,, Cummins, Generac....like the one you showed) provides a 2nd power source which when used with a transfer switch allows the use of smaller and smaller batteries which provide power only long enough to start the generator. After the loss of Mains Power, with the Generator started, the transfer Switch switches to the Generator. The Data Center Power is derived from those UPS systems like the ones you showed. whose Inverter is always running off the batteries which themselves are charged either from the Mains or Generator.

Great video man. Thank you.

Comprehensive video, well done

Interesting video. Even more interesting is the timing. Linus just did a tour of IBM in NY and was able to take apart a mainframe which looked similar to the power 10 box. Is IBM on a PR campaign at the moment?

This was great, thankyou!

This is great insight into datacenter infrastructure.

Really interesting to get an insight into a massive server farm, thanks.

awesome video! Super efficient DC!

Amazing video!

Amazing piece of engineering, there's a single flaw in the entire datacenter, the Emergency Stop sign on the OxyReduct is in english 😄 Great video!