How Bad Leap Day Math Took Down Microsoft

Рет қаралды 174,694

Күн бұрын

A look into one of the largest leap day bugs in history, as well as how Microsoft Azure's compute platform works (well, worked - it has been over 10 years. Though the fundamentals likely remain the same).
Sources:
azure.microsoft.com/en-us/blo...
Chapters:
0:00 Intro
0:43 Cloud Stuff
3:41 Azure VM Stuff
5:10 The Incident
7:59 Mitigation
10:23 Aftermath
Corrections:
-
Music:
- Ubiquitous by Diamond Ortiz
- Jane Street by Track Tribe
- Blackout by LEMMiNO ( • LEMMiNO - Blackout (BGM) )
- Cipher by LEMMiNO ( • LEMMiNO - Cipher (BGM) )
- Funk Game Loop by Kevin Macleod

Пікірлер: 440

@hadipawar2539 Ай бұрын

"instructed to do it 3 times cause 2 isn't enought and 4 is too many" I am dying at this

@theborg6024 Ай бұрын

well i do have it on good authority that 3 is the number of the counting

@rixxan Ай бұрын

@@theborg6024 And I have heard that 5 is Right Out.

@Bleenderhead Ай бұрын

Two is not enough, excepting that thou then proceed to three.

@satunnainenkatselija4478 Ай бұрын

Next up: Microsoft removes leap day from calendar because it was too complex. This comes after Microsoft demanded leap second be removed because it's too complex.

@enderger5308 Ай бұрын

@@rixxanand that three is the third number

@abebuckingham8198 Ай бұрын

"Double it and give it to the next person" - Automatic Service Healing, on bugs

@StromMakeVid Ай бұрын

🤣🤣🤣Underrated Comment

@MasonBitByte Ай бұрын

Azure AD, also known as Microsoft Identity, also known as Entra ID, also known as sadness

@influx__ Ай бұрын

no idea why they switched from azure to entra... Let's pick an even MORE arbitrary word

@JITSoftware Ай бұрын

Then they hit you with the "NOTE: Microsoft Entra ID is the new name for Azure AD. No action is required from you." On every single Microsoft documentation page

@yufgyug3735 Ай бұрын

fuck ad or entra, or whatever its called

@dalar2 Ай бұрын

Spent the last 6 weeks studying and revising for a specific Azure Certification... and at the last minute they updated the exam material to reference Microsoft Entra instead of Azure AD.... FUCK MY LIFE.

@boomknuffelaar Ай бұрын

I just went through madness with their deprecated library passport-azure-ad, the npm listing for this package doesn't even mention that it's deprecated.

@MidnightMidas Ай бұрын

A libaba cloud, oracle cloud, IBM cloud, google cloud. Its golden

@orangejjay Ай бұрын

I prefer Nimbus cloud but will settle for Cumulonimbus from time to time.

@jeanlasalle2351 Ай бұрын

I feel blue balled

@donatocapitella Ай бұрын

😂😂😂😂

@Twisted_Code 9 күн бұрын

Yeah, especially with an orange smile on screen as he getting to... Google cloud. I mean that's obviously Google's logo, we don't know any other big name IaaS providers.

@huy1k995 Ай бұрын

Y2K wished it would be like this.

@Weissenschenkel Ай бұрын

Wait until Y2K38 and all legacy crap running on 32 bits...

@ShrirajGPethe Ай бұрын

And on wider space

@1234567qwerification Ай бұрын

It was already a problem for software dealing with near future, as 'now + 20 years' 🤷🏼‍♂️

@Twisted_Code 9 күн бұрын

@@Weissenschenkel XKCD 2697 feels relevant here.

@Twisted_Code 9 күн бұрын

@@1234567qwerification Good point.

@Lars16 Ай бұрын

Love that the datacenter in Australia was upside down. Nice touch

@magentamonster Ай бұрын

Perhaps the one in South America should be upside down too. Given that the upside down Australia joke is due to Australia being in the Southern Hemisphere. But the joke says "Australia" rather than "Southern Hemisphere" to be funnier.

@oliverer3 18 күн бұрын

@@magentamonsterisn't that more because south-up maps are much more common in Australia?

@magentamonster 16 күн бұрын

@@oliverer3 No, because south-up maps are rare, even in Australia. Almost all the maps we use are north-up. Apparently south-up maps are used as souvenirs, but that's because of the joke.

@oliverer3 16 күн бұрын

@@magentamonster Huh, the more you know. I appreciate the lesson. :)

@InspectorGadget923 Ай бұрын

10:20 I love that you rolled the date over to 2/30.

@Anonymous-df8it Ай бұрын

MDY is like putting the tens place before the ones place before the hundreds place

@magentamonster Ай бұрын

@@Anonymous-df8it Irrelevant. MD is just as much a part of YMD as MDY. And YMD is the best. DMY may be in order, but DMYhmsp (11 May, 2024 at 12:42:18 PM) isn't, and neither is hmspDMY (12:42:18 PM on 11 May, 2024) for that matter. To be consistent, you'd have to use something like smHDMY (18:42:12 on 11 May 2024). No one uses this truly little endian time format, nor do they use smhpDMY (18:42:12 PM on 11 May, 2024). Also, as users of a left-to-right script, we use big endian numbers, making big endian the only truly consistent time order for us.

@Anonymous-df8it Ай бұрын

@@magentamonster We should also get rid of months, hours, minutes, and seconds, and just represent every point in time as year-day, where the day is the number of (fractional) days that have passed since midnight on New Year's Day Also, the original video used MDY and not YMD, so your point is moot

@Pixiuchu Ай бұрын

@@magentamonster YMD is in fact based! Seeing 2024/12/31 pleases me.

@andycivil 17 күн бұрын

@@Pixiuchu This is why it's the International Standard (ISO 8601). It pleases most people.

@catcatcatcatcatcatcatcatcatca Ай бұрын

Imagine waiting 75 minutes for a VM initialisation. Why would it take 25 minutes? Is there an intern hand-delivering the public key across the facility?

@thewhitefalcon8539 Ай бұрын

They usually spin up in minutes but they don't actually promise that.

@kv4648 Ай бұрын

The intern had time to wander off, have a nap, do a lap and have a snack too

@hubertnnn Ай бұрын

Its the cloud. They are waiting for the correct weather.

@Caphalem Ай бұрын

@@hubertnnn Meanwhile preparing the air balloon to go up there

@blikthepro972 Ай бұрын

you aren't a real tech company if you haven't assigned a stupid job to an unpaid intern

@ShadowSlayer1441 Ай бұрын

It's a good day when Kevin Fang uploads.

@kjyu Ай бұрын

It definitely wasn't a good day for those involved!

@eco_craft Ай бұрын

I thought it was pronounced Azure

@johndoe4290 Ай бұрын

Nah you are wrong, its pronounced Azure

@Dr-Zed Ай бұрын

I'm pretty sure you're both wrong, it's definetely called Azure.

@mme725 Ай бұрын

Classic mistake, it's Azure

@aze4308 Ай бұрын

no its azure duh

@Bajo85 Ай бұрын

I'm hearing a Nordic accent... Are you Swedish?

@ShrirajHegde Ай бұрын

0:50 skipping AWS was a nice gag 😂

@JCel Ай бұрын

Twice even 😂 That was a real push and pull there lol

@31redorange08 Ай бұрын

What is AWS?

@williamdrum9899 Ай бұрын

Amazon Web Services

@uSkizzik Ай бұрын

@@31redorange08 Amazon's primary business - Amazon Web Services.

@FugaceFugite Ай бұрын

@@31redorange08 an AWP with a typo, probably

@MHX11 Ай бұрын

I'm in love with your visualizations, they're eye candy

@amyisreallybored Ай бұрын

the AWS teasing at the beginning had me feeling the square hole trauma all over again

@C.I... Ай бұрын

The Zune also had a similar bug. I believe the solution was simply to wait until it was no longer the day in question.

@YoshiAsk Ай бұрын

Zune mentioned, raahhhhh!

@spaghetto181 Ай бұрын

microsoft

@renakunisaki 20 күн бұрын

As did the PlayStation 3, though that manifested on Dec 31, when the system couldn't comprehend that it was the 366th day of the year.

@spaghetto181 20 күн бұрын

@@renakunisaki true... and it was a disaster(plus it happened in the same timeframe playstation network got hacked severely)

@electric7487 3 күн бұрын

@@spaghetto181 megahard

@justicefool3942 Ай бұрын

0:50-0:55 The lengths you went to avoid saying AWS is commendable.

@baconerie Ай бұрын

why is it always the dates

@jonathangawrych5195 Ай бұрын

Just wait for Y2K38. The Epochalypse will do this to tons of outdated, unmaintained, embedded systems, or just faulty code worldwide.

@WoolyCow Ай бұрын

i love that its called the epochalypse lol

@jan.tichavsky Ай бұрын

We already saw effects of outdated unmaintained software and embedded systems when the GPS epoch rolled over. Nobody expected it would work for more than 20 years.

@Nadia1989 Ай бұрын

Airports, for sure. Some of them run on XP

@mfaizsyahmi Ай бұрын

@@Nadia1989 apparently the entire airline industry's booking system runs on win3.11 or something.

@notyourfox Ай бұрын

2106 will also be that, but won't really matter at that point ...legends say after Jan 19, 2038; 3:14:07 it will be Jan 01, 1970; 00:00:00 again...

@aeghohloechu5022 Ай бұрын

i like the fact that all the high availability/disaster recovery stuff inevitably ends up making the situation into something way worse than if we had just let it fail and tell customers to go take a break

@nicholascopsey4807 Ай бұрын

I’ll tell you why it took 5 hours to fix the bug, they spent 4 hours and 50 minutes in meetings strategizing about how the engineers would identify the bug and the procedure to test any changes that would go out.

@samuvisser Ай бұрын

Absolutely. Also, these systems are massive and from experience i know sometimes u can know what the bug is based on observed behavior but it still take hours to identify the code that causes the bug because there are so darn many systems talking with each other all in their own git repo

@XxZeldaxXXxLinkxX Ай бұрын

In incident response this is actually crucial. The last thing you want is to do something wrong and cause even more issues. Measure twice, cut once applies wholeheartedly here

@aeghohloechu5022 Ай бұрын

@@XxZeldaxXXxLinkxXthey fucked up 7 servers anyway though so uh

@XxZeldaxXXxLinkxX Ай бұрын

@@aeghohloechu5022 OK and? If you have actually managed production servers you would understand why incident response takes time. If you haven't, maybe one day you'll get it when you footgun yourself while haphazardly trying to hotfix a production system.

@andyvirus2300 Ай бұрын

@@XxZeldaxXXxLinkxXwell not everywhere, I’m sshing and viming my way into fixing prod every other week. Losing time in useless meetings isn’t the way most of the time.

@worgenzwithm14z Ай бұрын

Everytime a coworker suggests building our own date library

@mangodude-nq6su Ай бұрын

Bet it's the same guy who makes livecoding problems

@grumpy989 19 күн бұрын

Force them to watch the tom scott video on timezones for 12 hours straight

@pdlbackup Ай бұрын

It was fixed the next day??? They might as well have done nothing and it would've fixed itself!

@redyau_ Ай бұрын

Well no, as whole clusters were HI by then. But you have a point 😅

@cskiller86 Ай бұрын

I was going to comment the same thing. Since they found the problem so late in the day, and the fix was ultimately deployed in March, why not reboot everything without the fix on March 1st? It would have been less downtime probably. And, after that, they had 4 years at their disposal to create and test the fix.

@ribstogo12 Ай бұрын

They might have known that, but imagine what their bosses face would have looked like if they had just sat on their hands while support calls kept rolling in. Not a good look.

@xelspeth Ай бұрын

And then 4 years later it all happens again

@tbuk8350 Ай бұрын

Well, they still had to restore all the clusters. A couple VMs were corrupted because of the constant shifting, and a couple clusters were all stuck in the HI state.

@TheHadrian54 Ай бұрын

Surprise PHP facts! PHP DateTime implicitly "fixes" impossible dates except instead of going to the last day of the month, it goes to the next month and adds the number of days missing. For example, 02-31 becomes 03-02. This means that adding 1 month to the following series of dates: 01-28 01-29 01-30 01-31 02-01 02-02 Will result in this: 02-28 02-29 03-01 03-02 03-01 03-02 Isn't that awesome? 😊

@williamdrum9899 Ай бұрын

Trying to wrap my head around how Microsoft couldn't think of this

@TheHadrian54 Ай бұрын

@@williamdrum9899 With the solution that PHP went with you're a lot less likely to cause a catastrophic failure but you still run into some issues just different ones 🤷‍♂ At the end of the day the issue with dates is that our brain takes the way they work for granted when they're actually really complex

@HenryLoenwind Ай бұрын

Every good date library does that. But that requires people to actually use a date library and not do things "by hand".

@grumpy989 19 күн бұрын

Rare PHP W

@seifenspender Ай бұрын

Insane that this first happened only 12 years ago. I had to double check why this didn't occur earlier and made the realization that azure really is only 14 years old. Crazy.

@shkron Ай бұрын

Man, I just want to tell you that your videos are amazing, and every time a new one comes out, it is like the happiest day of my life

@Phroggster Ай бұрын

These are so freaking good! I particularly loved the timeline at 10:19, as that is such a Microsoft thing: resolving an issue on February 30th.

@MSPaintOfficial Ай бұрын

nice joke, there is no February 30th

Ай бұрын

Some months ago I was manually typing DAX formulas in Powerquerry and one consisted on subtracting one year. It only took me 5 minutes to realise "but what about leap years". How did Microsoft not think about this? Sadly, by the time a leap year comes, everyone at work will have forgotten my Excel.

@subanark Ай бұрын

Powerquery is a language like DAX, PowerQuery can be used inside Excel and PowerBi, DAX can be used in PowerBi, but not Excel. Microsoft has a lot of engineers and there are a lot of places this could fail. We have leap year awareness notices, training and detection to mitigate something like this in the future.

@FinlayDaG33k Ай бұрын

Man, if only there was a numeric standard that didn't really care about whether the date actually exists or not as long as the number isn't higher than the other number. But a'las, we'll have to wait for Apple to invent it in 5 years or so.

@Dr-Zed Ай бұрын

*laughs in unix time*

@Aura_Mancer Ай бұрын

I assume this is a joke for the unix timestamp right. Because yknow, there's that

@Weissenschenkel Ай бұрын

@@Aura_Mancer true, but Y2K38 is coming. Everything in 32-bit will be kicked back to 1970-01-01 00:00.

@Aura_Mancer Ай бұрын

@@Weissenschenkel Most unix timestamp things use 64bit nowadays. So it is going to be a non-issue, if people have foresight. Which some will not, which will make for Kevin Fang videos. Win win if you ask me

@twentylush Ай бұрын

microsoft doing everything in their power to not use unix timestamp, even when it means using 3 different epochs in their kernel

@willpeterson3943 Ай бұрын

Can't wait for all the bugs in 2100, which is NOT a leap year

@guy7329 Ай бұрын

why wait so long? We'll have tons of problems in 2038 when 32 bit system clocks just wrap back to 19xx or something.

@iamfinkyuk Ай бұрын

I was working at Microsoft around 2008-9 and, whilst on a training course in Stockholm (or Prague, I forget which), the whole of Azure went down during day 1 or 2 of the course which resulted in a few of us jokingly saying to each other "did you just break azure?". It later transpired (publicly) that the entire platform went down because of an SSL certificate expiry that cascaded across the entire cloud infrastructure. Some time later, I asked the NOC to get a copy of the transcript of what happened and how it was handled and these guys were REALLY good. Abject professionals the whole way. The reason there are large time delays between "finding a fix" and "making it live" is the huge volume of testing needed for approval.

@riddixdan5572 Ай бұрын

Love your videos. Very educational and entertaining. Keep em coming

@BerlingoQC Ай бұрын

There is not enough of your video , can't wait for the next

@megamasterbloc Ай бұрын

negative leap seconds are gonna be fun to watch

@Nico-qq7xl 29 күн бұрын

such a good explenations keep up the good work boss!

@Berdes1 23 күн бұрын

11:00 "0 UTC happens at the same time everywhere". I don't know how widespread this practice is, but I know of some large services that have a couple of instances running with a clock configured 24 hours and/or 7 days ahead of time to catch those kind of bugs.

@AndersonPEM Ай бұрын

HONEY! STOP EVERYTHING! KEVIN FANG DROPPED A VIDEO! GRAB THE POPCORN!

@Xavier-xb7is Ай бұрын

Best tech channel by a tremendously large margin. Can't get enough of these.

@jamescollier3 Ай бұрын

As a materials engineer, I know the words "Automated service healing," is not created by men who work past 5pm

@Fay7666 Ай бұрын

That is, until they work past multiple 5pms in a row.

@rabik_dev Ай бұрын

You need to post more videos Kevin

@3rdalbum Ай бұрын

Whenever I watch a Kevin Fang video, I know my next piece of amateur hacky software at the office will be better designed and less vulnerable. And that's good for everyone in my directorate. EDIT: My software won't break on leap year, but every February it delays archiving a few days of support tickets until the following month, lol. EDIT 2: I'm impressed at the systems Microsoft had to try to heal its service automatically and migrate VMs onto other servers when there's a suspected hardware problem. Shame it blew up in their faces this time.

@boomknuffelaar Ай бұрын

Wouldn't a delayed archive make you MORE vulnerable? If the February bug required a rollback you'd lose more data.

@3rdalbum Ай бұрын

@@boomknuffelaar Archiving is just to "get this old resolved ticket out of my hair", it's not a backup. The data is all stored in SharePoint lists and as such it's all backed up and versioned automatically in the cloud, regardless of whether it's in the archive or the main list.

@francescourdih Ай бұрын

10:20: the graphics says it’s the 30 of February **azure cluster**: wanna see me going down again?

@cirkulx Ай бұрын

feb. 31: patch again 💀

@francescourdih Ай бұрын

@@cirkulx I wouldn’t blame the developers for forgetting the 30th of February

@Anonymous-df8it Ай бұрын

@@francescourdih That was actually a real date in Sweden at one time

@jayfraxtea Ай бұрын

The way you pronounce Azure remembers me of the good old days in 2012, when almost every Microsoft marketing employee pronounced it differently. My favourite back then were some German Microsoft representatives who pronounced it like [aˈʒuːɐ̯] ... as it would be a blend of a Polish-German word.

@aboxinspace Ай бұрын

I work in a company that has folks from USA, India, Brazil, Mexico... saying the word "Azure" is a language bomb 😂 Almost leads to an argument every time, then everyone just says "Microsoft Cloud"

@jayfraxtea Ай бұрын

@@aboxinspace, speaking about a language bomb ... better don't use Azure Cognitive Services Translator Service, now re-named as Azure AI Translator, to translate sentences that contain the word "Azure" 😜

@AraniWendinah Ай бұрын

Let's go Kevin upload their videos again. Grab snacks!

@Froschkoenig751 Ай бұрын

Your humor and animations are the greatest!

@richardfarrer5616 Ай бұрын

Been there, done that (just on a much smaller scale). The product I work on reads in messages where some dates are fully represented, and some come in with just day and month specified. We know those are on or before certain other dates so we can calculate the correct year, but leap days regularly broke this. Just to add to the fun, the values have to be passed around as dates before we complete the validations. Since we know the dates will only be within a year or so of today, there is a marvellous bit of code which gives a dummy year of 1968 for these values prior to validation. Why 1968? 1. It's long enough ago that no real date will be for that year. 2. It's a leap year. 3. It's the year the developer was born. And, yes, I do happen to be 56 as it happens.😀 Oh, and then there's the code which adds one day by adding 24 * 60 * 60 seconds to a date - which works unless a day has 23 or 25 hours, i.e. daylight saving or local equivalent.

@hubertnnn Ай бұрын

It reminds me of a payment platform we used that accepted integers that could be both in dollars and in cents and decided which one it is based on the amount. Someone rewritten a library that had an overloaded method, that accepted integer in cents or float in dollars into language where all numbers are stored as a float.

@Akaterial Ай бұрын

I love your videos. You make theses dry and boring subjects entertaining.

@NicosLeben Ай бұрын

10:19 Nice touch with the 2/30.

@SegNode Ай бұрын

Great video, I was chuckling the whole way through lol

@alejandroalzatesanchez Ай бұрын

Timezones are lovely!

@gamerk316 29 күн бұрын

To be fair: Any software engineer who's had to work with time zone/leap day/year/second(!) logic knows that every time format we have *sucks*. The only acceptable solution is to do everything in UTC and convert back to the users desired time zone after the fact.

@xnehaxixh Ай бұрын

Bro wake up, Kevin Fang uploaded a new video!

@earthling_parth Ай бұрын

Finally, a new video! Banger as usual 😆 Rhat restart logic being 3 times was hilarious

@2k18banvalaki5 Ай бұрын

The VM was like “Double it and give it to the next cluster”

@ItsVingtdeux Ай бұрын

how to make a kevin fang video: 1. add stock footage 2. represent programs with amogus characters 3. overuse that one explosion sound effect

@tgz39j4ndywmm7 Ай бұрын

That VR headset analogy was very good. As a VR headset i approve of this

@magic_pink_horse Ай бұрын

The stock explosions are still my favorite ❤

@eddydude100 Ай бұрын

Another great video!

@NithinJune Ай бұрын

great vid as always

@caduhidalgo4996 Ай бұрын

Feels good to have a new Kevin Fang video! 🙏 Thanks for the great upload!

@dom1310df Ай бұрын

Great code review picking up on that error ahead of time.

@hadipawar2539 Ай бұрын

welcome back Kevin!

@MasanaAnta Ай бұрын

love how you always find new ways to present information!

@nessitro Ай бұрын

the missile blast got me rofl. keep em' coming :D

@shubhamsawant1551 Ай бұрын

the funniest part in 2018 in My Diploma in computer Engineering i wondered why we write code to print dates today i understand specially i understand why we calculate leap month and all

@donchaput8278 Ай бұрын

"Because 2 isn't enough and 4 is too many" @6:15 -Five is right out. Once the number three, being the third number, be reached......

@harsha1306 Ай бұрын

I love that you included a Feb 30th

@nathanr136 Ай бұрын

You need a patreon, greatly explained videos I always enjoy watching and would like to contribute :)

@TickUwU Ай бұрын

Now my brain is HI

@nexdemise4182 Ай бұрын

This year a leap day bug in Sothos's software knocked my workplace offline. Not sure what happened exactly there, I experienced it as issues with AWS Secrets Manager (can't pull the secret down, not sure if you could manually pull the values out, I think I tried and that failed too), there were probably other issues as well but that's how it manifested to me since that's generally the first error most applications I work on will throw. My guess it could also be something about authentication, certificates, etc. I didn't really dig because if it's on the other end then not like I can fix it. So I just kicked back and relaxed, I'm a developer, it's the operations' problem. It worked the next day but like that'd happen anyways.

@_tsu_ Ай бұрын

The man himself is BACK!

@frag0638 Ай бұрын

Not using epoch timestamps?

@davefellows Ай бұрын

I remember very well when this happened, it wasn't a good day for cloud computing. Amazing how far things have come since then.

@skythra2895 Ай бұрын

Fantastic delivery as always

@wardrich Ай бұрын

0:06 I've always found it weird how people fumble over the word "azure". It's a shade of blue. There was also a once popular torrent client named Azureus (named after the dart frog. It's now called Vuze). Anyway, every one of those pronunciations said in that section were wrong too 😂. It's like.. a-zhur where the zh is like an "sh" but not quite lol

@hamburgerfatso Ай бұрын

Holy shit i just searched to see if you had any videos recently and there's one 15 minutes ago

@DiamondLegends Ай бұрын

That VR headset analogy was perfect

@Andrew90046zero Ай бұрын

I saw what you did with the upside down servers for Australia xP

@ACoupleStoners 9 сағат бұрын

You should do a video on the hacking of the pharmacy's in the US a few months ago.

@MonochromeWench Ай бұрын

This is the sort of bug I would expect from a beginner programmer not the experienced ones that Microsoft would have working on Azure but i guess anyone can do it and that would be why Microsoft is updating their C++ compiler to detect some leap year bug.

@mrmarkom Ай бұрын

Once they figured the source of problem, they could have just wait for a day to pass. Next day the bug would not manifest, and they would have 4 years to deploy fix. Recovery lasted until tomorrow anyway :)

@Twisted_Code 9 күн бұрын

I laughed so hard when you showed Amazon's smile at 0:52 and said Google cloud. I hope your editor is well-paid (whether that's you or someone you hired, obv)

@AnindoSarker Ай бұрын

The irony of my laptop crashing exactly at 5:38 is too surreal. Crashed twice

@armandoromero5661 Ай бұрын

Go UTC already Microsoft, and use Posix Date and Time functions to get future dates. Windows and its grampa VMS used local time, and were awful during DST, we just shut down the servers during those times

@BenMclean007 Ай бұрын

There are actually companies that currently provide rentable computing power in space. So not literally in the cloud, but literally above the cloud

@psicommander Ай бұрын

Does the timeline really end on 2/30/2014? :D Hope they could create certificates on that date, though

@Anonymous-df8it Ай бұрын

Yes, it ended on the second of Octovigintember

@nax47 Ай бұрын

I love this videos.

@jamescollier3 Ай бұрын

great video

@Manabender 21 күн бұрын

10:20 Can we talk about how you have "February 30th" on the timeline? Brilliant.

@codeman99-dev Ай бұрын

Oh my goodness! I kid you not... I received an in-video ad for Azure right as Kevin is explained the VM crashes (roughly 6:25).

@ThompYT Ай бұрын

They should've just added 31536000 to unix time

@elliot20201 28 күн бұрын

I may be out of the loop but I totally thought the -zure was stressed in azure

@Caphalem Ай бұрын

This is... easily the most entertaining developer channel on KZfaq. As a mainly BE oriented developer, I'm dying xD

@m1m1c_ Ай бұрын

MY GLORIOUS KING RETURNS

@NithinJune Ай бұрын

how long before you think Primeogen reacts to this vid

@shubhamsawant1551 Ай бұрын

Pls Make video on 2022 Microsoft port wan port change Issue I still remember teams And all office products are down

@NithinJune Ай бұрын

supposed this channel hasn’t hit the algorithm yet

@chuckfarley7642 Ай бұрын

Hard to believe this was over 12 years ago. I remember it like it was yesterday. I was working for Microsoft at the time and my service got impacted by this. It was a long 12 hours. There was no laughing at that rookie mistake!