What runs ChatGPT? Inside Microsoft's AI supercomputer

What runs ChatGPT? Inside Microsoft's AI supercomputer | Featuring Mark Russinovich

Рет қаралды 691,356

Күн бұрын

Get an inside look at the AI supercomputer infrastructure built to run ChatGPT and other large language models, and see how to leverage it for your workloads in Azure, at any scale.
Go behind the scenes:
-For how we collaborated with NVIDIA to deliver purpose-built AI infrastructure with NVIDIA GPUs
-How Project Forge checkpointing works to restore job states if a long training job fails or needs to be migrated
-How we used LoRA fine-tuning to update a fraction of the base model for more training throughput and smaller checkpoints
-How UK-based company, Wayve, is using Azure's AI supercomputer infrastructure for self-driving cars
-And how Confidential Computing works with Azure AI to combine datasets without sharing personally identifiable information for secure multiparty collaborations.
Mark Russinovich, Azure CTO, joins Jeremy Chapman to break it down.
► QUICK LINKS:
00:00 - Introduction
01:15 - AI innovation building specialized hardware and software
04:22 - Optimizing hardware
05:40 - Improved throughput
06:17 - Project Forge
08:01 - Project Forge checkpointing demo
10:02 - LoRA fine tuning
11:29 - Use AI supercomputer infrastructure for your workloads
12:34 - How Wayve is leveraging AI supercomputer infrastructure
13:47 - How Confidential Computing works with Azure AI
15:21 - Wrap up
► Link References:
Leverage Azure AI capabilities for yourself at aka.ms/AzureAIInfrastructure
► Unfamiliar with Microsoft Mechanics?
As Microsoft's official video series for IT, you can watch and share valuable content and demos of current and upcoming tech from the people who build it at Microsoft.
• Subscribe to our KZfaq: / microsoftmechanicsseries
• Talk with other IT Pros, join us on the Microsoft Tech Community: techcommunity.microsoft.com/t...
• Watch or listen from anywhere, subscribe to our podcast: microsoftmechanics.libsyn.com...
► Keep getting this insider knowledge, join us on social:
• Follow us on Twitter: / msftmechanics
• Share knowledge on LinkedIn: / microsoft-mechanics
• Enjoy us on Instagram: / msftmechanics
• Loosen up with us on TikTok: / msftmechanics
#Azure #OpenAI #Supercomputer #LLM

Пікірлер: 304

@Pazfora Жыл бұрын

This was a great Microsoft ad.

@acdsp Жыл бұрын

Definitely they’ve earned this right.

@wilsonbotlero2363 Жыл бұрын

Well, duh! What else did you expect?

@Papers40 Жыл бұрын

😂

@vylbird8014 Жыл бұрын

@Pathipati Sai Well, I wouldn't go that far. I'd say they are /less/ evil than they used to be, but that's really not a difficult achievement.

@chrisrogers1092 Жыл бұрын

Great Nvidia ad as well

@Ionizap Жыл бұрын

A fascinating look at the big picture, I've always wondered about that. This vid shows it . I'm an Apple Guy at home but I am so impressed with what MS has done in the last few years. I run MS Edge and Bing Ai on my desktop at home and at work. The brilliance of these people is just beyond belief.

@romlfox Ай бұрын

ربزؤؤمظميمؤمؤمؤخ

@AlexanderMorou Жыл бұрын

Holy crap. The number of iterations in the technology stack that they've done in the background is massive. To make these tools usable, scalable, distributed, and likely things I don't even comprehend since it's out of my domain, is massive. I can't imagine how much it costs to use this stack, probably an enterprise level offering.

@flwi Жыл бұрын

Impressive to see how Microsoft has transformed.

@kitebeachinnbeachinn2888 Жыл бұрын

The new CEO is a vision-er much better than Steve Developer, develope, developer... MS should come back with windows phone.

@jamesjonnes Жыл бұрын

Much better than Meta, at least.

@VadimBolshakov Жыл бұрын

how?

@harriehausenman8623 11 ай бұрын

Finally some actual insight into the system! Thank you so much for clarifying LoRA usage, clustering and your "AIOps" apporach 🙏 Good to see someone so knowledgeable talk about it!

@shabsZA 11 ай бұрын

Love hearing about the hardware that exists behind the scenes - thanks for sharing

@ankushalgudkar1646 Жыл бұрын

Absolutely fascinating look into the BTS of these incredibly powerful AI tools now. I was always guessing that the easier something looks and feels for the consumer, ALOT of manpower, cost, time, resources would have gone into making that a reality. Massive respect to Microsoft for the transparency and releasing these videos on how they're building their AI capabilities. Incredible time to be alive!

@SirIsaacMewtonIII Жыл бұрын

microsoft is just killing it in this area. seriously impressive engineering. i wonder if they'll ever release an ai agent for pc that will fix os-level issues (eventlog issues, give human-readable advice based on eventlog errors, etc)

@Nightspyz1 Жыл бұрын

Microsoft Copilot?

@Ozymandias1 Жыл бұрын

No more low level techs.

@professional7583 9 ай бұрын

NEVER 😂

@animaze86 8 ай бұрын

It's called 'Clippy'

@absbi0000 6 ай бұрын

Windows Copilot is already a thing. Things are moving fast.

@glitchedpixelscriticaldamage Жыл бұрын

after listening to these guys.... i feel that my work is useless and only a few people over the world get to do really impressive stuff.

@aeromotive2 Жыл бұрын

what do you do

@HelloThere-xs8ss Жыл бұрын

Kinda yeah. People benefit in all industries from a small group of capable and driven people

@andrewmayorga6649 Жыл бұрын

Comparison is the killer of joy!

@sbrunner69 Жыл бұрын

Same. Chin up, your work is good too.

@therealb888 Жыл бұрын

@@monad_tcp This is unfortunately true because everyone wants the easiest path to make the most money.

@naipaulojar7761 Жыл бұрын

A very good session showing what is under the covers with Microsoft's AI computing. Microsoft is going a good job with the confidential compute which is ultra important for business who want to use AI for both the business & their customers.

@4.0.4 Жыл бұрын

It's impressive how GPUs have become such a powerhouse. CUDA was a bright idea way back when. Wish AMD competed better here.

@theskeletonboi Жыл бұрын

Don't forget Tensor Cores

@Speedboycentral Жыл бұрын

george hotz is going to change that - AMD will be a powerhouse soon

@soraaoixxthebluesky 11 ай бұрын

@@SpeedboycentralI personally admire George but this is not a simple problems. Even he himself gave up on AMD hardware after trying to run the 7900XTX to utilize his own AI framework, Tinygrad. But in AMD has already build a translation layer for ROCm to run CUDA on AMD hardware. How much of a performance penalty? We don’t know.

@ITube4RealFun Жыл бұрын

The legendary Mark Russinovich.

@AndersKeisHansen Жыл бұрын

Epic video, thank you :)

@ShareReachCommunity Жыл бұрын

This was so nice. Cant wait to learn more about Azure.

@TomWhi Жыл бұрын

I love it when Mark is on the show, he's an absolute tech heart throb!

@TheB1nary Жыл бұрын

Just said that to my wife. The look I got! 🤣😂

@TomWhi Жыл бұрын

@@TheB1nary rookie mistake, I quite quickly found out my wife is very judgmental about my tech-celeb crushes. For example I definitely don't talk about John Savill in front of her any more! 😂

@TheB1nary Жыл бұрын

@@TomWhi Duly noted -- I often watch the man-beast John Savill and brag about him! 🤣

@TomWhi Жыл бұрын

@@TheB1nary it's hard not to! The brain on that guy...!

@ArnaudMEURET Жыл бұрын

He is legend ! 😊 #FluffYouSony

@danypell2517 Жыл бұрын

So cool. Keep it up!

@wildfire6866 Жыл бұрын

Very impressive!

@cowboyuniverse7258 Жыл бұрын

I’ve never seen comp specs as this sexy. It’s freakin insane.

@aldosansan2335 11 ай бұрын

Russinovich Is one of my biggest inspiration in IT!

@ndolson Жыл бұрын

chat gpt와 대화할때, 대화 한줄에 얼마의 전력과 냉각수를 사용할까요? 물 50ml정도를 냉각수로 사용하나요? 궁금하네요 How much power and coolant do you use for a line of conversation when you talk to a chat gpt? Do you use about 50ml of water as cooling water? I'm wondering.

@letsworksimple Жыл бұрын

Less than a bitcoin lol

@tombranson9341 Жыл бұрын

Mark is the Master of the Microsoft Universe!!!! He has got more genius in his pinkie that I got in my entire body.

@mantrax314 7 ай бұрын

Just came to my youtube this video... it is very impressive, the infrastructure.

@Shahawir Жыл бұрын

The fat tree topology still used in every data center, brilliant and simple ideas live very long

@maithriashokan 6 ай бұрын

It was fun and informative listening to the CTO talk about Azure infrastructure. His vocabulary is amazing and the jargons he uses is so smooth. It shows his tenure!

@ivan_i_am 5 ай бұрын

Mark is the Azure, fascinating amount of knowledge came from this guy in short period of time. Go systernals! 😊

@seebradrun 6 ай бұрын

Excellent presentation

@jeffzhang769 Жыл бұрын

Great video!

@KazenoniKakuremi Жыл бұрын

I don't think people how insane MSFT execution of their Ai strategy is - I have never ever seen a company execute a strategy so cohesively as a unit - not even small business - let alone a behemoth like MSFT To get every single department of MSFT to work collaboratively to embed Ai into every product is unheard of ---

@AtYourService-fh5fu 11 ай бұрын

Great insight into a great technology!

@harriehausenman8623 11 ай бұрын

BTW, there's your "moat" right there: MS is in the perfect position in the middle of the triangle between hardware (nvidia), foundation systems (OpenAI) and the customer (Azure)!

@p.c.336 Жыл бұрын

I didn't imagine Mark Russinovich like this, when I used his tiny but very powerful tools several decades ago 😁 Bginfo, filemon etc..

@dibu28 Жыл бұрын

Same😂

@harriehausenman8623 11 ай бұрын

@@dibu28 procexp64.exe FTW! 🥳

@TerragonDE Жыл бұрын

Very interesting insight, thank you! Microsoft is back :-)

@CarletonTorpin Жыл бұрын

15:00 - Confidential GPU seems like a great idea; individual models can retain their IP while simultaneously contributing to a larger brain.

@fmcmarques Жыл бұрын

I remember Mark Russinovich from the old days, from sysinternals tools

@aliveandwellinisrael2507 6 ай бұрын

lol yep

@CMDRScotty Жыл бұрын

Its intersting to see how supercomputers have changed over time. Supercomputer before the Cray 1 in 1976 were the size of warehouses. Cray built small supercomputers then starting in the 90s they started getting bigger again and now they are the size of warehouses again.

@vylbird8014 Жыл бұрын

The Crays were from that brief time when supercomputers were custom-engineered architecture. Every supercomputer in recent decades is made by connecting off-the-shelf hardware together. The engineering challenge is in making all that hardware operate coherently without excessive overhead and able to tolerate the inevitable hardware faults. You take a pile of hardware accelerators and stuff them in a server, a stack of those servers in a rack, a row of those in a room, wire it all together with the fastest ethernet or infiniband you can afford, then hire some crazy-good computer science specialists to make them all work together, and some pretty-good HVAC and power engineers to stop the thing from melting itsself or the local substation.

@Mr.Denmark Жыл бұрын

Keep it up, thanks microsoft.

@SereneDiaries Жыл бұрын

Impressive!

@huytranvan2754 7 ай бұрын

Can't believe I spent nearly 16 minutes watching a MS Azure advert... :) So interesting

@udirt Жыл бұрын

holy... I had no idea Mark would be quite young still - he was already god-level windows guru when friends and I started our careers in the late 90s - he must've broken through the technological ceiling at 25 or something!

@staffanlundberg Жыл бұрын

I am a fitness freak and my first thought when I saw him was like...ok, this somewhat elderly guy is EFFICIENT due to his physical condition. He actually looks like another fitness freak ....which makes me wonder...I always thought efficient nerds were pale and skinny as they work too much, but this guy must be semi-retired to be in that shape ? If not, then I don´t understand how he does it !

@brownianmotion6319 Жыл бұрын

Rumour has it that he keeps a self portrait in the loft.

@harriehausenman8623 11 ай бұрын

@@brownianmotion6319 🤣That would be quite "gray" !

@raffriff42 6 ай бұрын

@@staffanlundberg He only works part time (sold Winternals to MS long ago) and can afford a good gym with pro trainer(s). Good for him, I say.

@tombyrne6433 11 ай бұрын

Awesome stuff appricaite all the effor Microsoft puts in to build the future!

@timburtell8718 6 ай бұрын

I like to imagine a time traveling 1700s pilgrim watching this video and trying to make sense of anything said here.

@Popdad Жыл бұрын

Loved it

@bright5801 Жыл бұрын

he knows his stuff.

@tuapuikia Жыл бұрын

9:45 I'm glad criu is now used in mainstream

@WilliamAshleyOnline 11 ай бұрын

What I am wondering is if they have built in industrial process to use the heat produced for some industrial process as well as thermocoupling to capture energy from the differential? Is the heat built into the model to process like dehydration, or chemical process?

@coopmanfamily Жыл бұрын

Do you build power plants by these servers?

@kaizen9554 Жыл бұрын

At this point Microsoft is killing it. On services: Azure, productivity suite, search engine, and also with edge browser being the best browser on the market; I do everything on edge- reading pdf, it reads text now, fast browsing. Microsoft is making a big comeback

@optimusctts Жыл бұрын

Quite the work

@emaayan Жыл бұрын

1:37 did they actually called it MEGATRON?

@You_Name_It Жыл бұрын

Yup they did

@MSFTMechanics Жыл бұрын

Indeed. From the blog, "the largest and the most powerful monolithic transformer language model trained to date, with 530 billion parameters" www.microsoft.com/en-us/research/blog/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model/

@canalgeek42 Жыл бұрын

LOL :D

@MasayaShida Жыл бұрын

Uh ohh we need Optimus.

@MSFTMechanics Жыл бұрын

@@MasayaShida...more than meets the eye

@thuokagiri5550 11 ай бұрын

Brilliant

@markarca6360 Жыл бұрын

From the man who brought us Sysinternals!

@RockoPY 11 ай бұрын

Omg amazing video

@grey8_ Жыл бұрын

13:42 "... and that's where we excel." Me: "Word."

@8eck Жыл бұрын

Great and skillful people are pushing big companies. Microsoft understood its weakness and now self-healing step by step.

@ItsWesSmithYo 5 ай бұрын

Watching the free market at work is an amazing thing 🤙🏽😎🖤🐓

@portlyoldman 11 ай бұрын

Sounds wonderfully like a talk about the Turbo Encabulator 🤣

@randomsam83 Жыл бұрын

Thanks for Sysinternals

@nickspang Жыл бұрын

If I saw this without knowing that it’s successfully in production, I’d be think to myself “yeah it’s a lot of hype, but does it really work?” My perception of Microsoft has now changed.

@meepk633 Жыл бұрын

What is in production?

@ApostolisKourtis 6 ай бұрын

Yea but can it run Crysis?

@KlausRosenberg-et2xv Жыл бұрын

Why have I never heard of a Microsoft's Megatron Turing? It is three times bigger than gpt-3 and can do a variety of stuff, even use natural language...

@NotSure416 6 ай бұрын

What is the reliance of ChatGPT on phase detractors and magneto reluctance?

@zy2870 5 ай бұрын

it's great to see how MS and NVIDIA are working together to build the AI infra. Though wondering how good is it compared to Amazon and GCP?

@MSFTMechanics 5 ай бұрын

Azure's support for NVIDIA Quantum InfiniBand and NVLink is a differentiator here and it's something Azure has had a head start on vs. other vendors who currently offer it.

@sergeymatpoc Жыл бұрын

that's an amazing story. Now I feel jealous =). Why can't I work on the most sophisticated hardware for the most cutting edge technologies, and just providing services to our Biz customers (whoever they may be) =). (P.S. technically, I work in Microsoft though, but still - our sub-org is way far behind)

@cricketer1987 11 ай бұрын

Very impressive. Makes me regret selling Microsoft stock last year - they are just so ahead of the game when it comes to AI. Fortunately though it is still driving my S&P 500 and NASDAQ 100 ETF gains for this year so still benefitting from the excellent innovations from Microsoft and OpenAI.

@lassmirandadennsiewillja3943 11 ай бұрын

Its now the ENTRA CTO:-)

@NK-iw6rq Жыл бұрын

I love Chat GPT!

@TonyFarley-gi2cv 6 ай бұрын

So how come your linear and at the options of Wi-Fi won't align together

@Maisonier Жыл бұрын

So ... tell us ... are you using megatron-turing nlg 530b for your own enterprise decisions?

@mpp9796 11 ай бұрын

Does it requires a reboot?

@devrim-oguz Жыл бұрын

10:08 it is a bad idea to name it as LoRa since the name is used for radio communication modules. I guess the guy invented the name never "googled" it.

@Lou44000 24 күн бұрын

At 2:00 the question was « it is super ressource intensive to run and expensive » But you did not answer the question. In a context of global warming and in an energy constrained world it would have been interesting to have an answer.

@SanjeevKrSingh-xi2zr 11 ай бұрын

That's good 👍🏻

@monad_tcp Жыл бұрын

8:59 so this is why GPU access is so expensive in all the other products, you just hog the GPU, damn, I want this.

@ChuckNorris-lf6vo Жыл бұрын

Sold.

@letsworksimple Жыл бұрын

Chuck Norris once opened Windows and ChatGPT was invented by the operating system to respond 😂

@srikark3532 Жыл бұрын

Super!

@kevinmarti2099 7 ай бұрын

Thats a lot of compute. 😯

@adriancoanda9227 6 ай бұрын

well adapt that AI for gaming generating and imagine that every person will be able to create their own personalized games or software's verry nice

@javerikr Жыл бұрын

How can I prove that I am the only one using this application, and it is a real person? But my question is, why is it a real person?

@vibdib 11 ай бұрын

How do you input 530 billion parameters

@christerry1773 10 ай бұрын

So is chatGPT essentially located in one location? Where he described the server stacks and gpu’s.

@MSFTMechanics 10 ай бұрын

There are multiple instances of GPT-4 running concurrently in multiple locations to run ChatGPT and other GPT-based services.

@nangld Жыл бұрын

Will MS finish the Project Milo now?

@deflekt 5 ай бұрын

what about STABILITY AI , weredo they fit into all of this ??

@88heiling Жыл бұрын

And plebs believe Microsoft is just using OpenAI’s AI when GPT-4 solely exists because of Microsoft’s supercomputer.

@DanKaschel Жыл бұрын

Does everything built on azure belong to Microsoft?

@onetruekeeper 3 ай бұрын

It would be great if it could make a customized videogame just for me at no charge.

@eyesoffloraandfauna8728 Жыл бұрын

3:40 why they use GPUs and without gpu is it possible ? Apple like technology used

@oldtwinsna8347 10 ай бұрын

Running exclusively on CPUs would be exponentially slower for the same amount of money spent on hardware. GPUs are purposely built for parallel and vector processing, which is what a lot of AI needs. CPUs, meanwhile, are built for general instruction processing.

@jeremiahmullikin Жыл бұрын

Is there any reason why GPT-4 can't handle calculating simple problems like power factors or the height of the meniscus in a capillary tube without making mistakes?

@bendito999 Жыл бұрын

when combined with Wolfram to help it do the math, it can do better. Gpt itself is bad at being exact with numbers, being good with the numbers isn't one of the things it is trained in, that's not the 'game' GPT itself is playing. GPT itself is playing more of a 'if this is written so far, what do you think should be written next' game, that it is pretty good at

@kevinmcfarlane2752 Жыл бұрын

In my playing around with the Chat AIs I’ve found that they are both better than you expect and worse than you expect. There are some surprisingly simple things they still can’t do. But there is also an art to formulating your prompts too. So you can sometimes correct their answers with a little prodding.

@VulcanOnWheels Жыл бұрын

12:20 That keyboard looks as if the video is mirrored, but the text does not look mirrored. What's going on here?

@marie-chantalcote3157 Жыл бұрын

She have a left hand keyboard

@therealb888 Жыл бұрын

Are we going to see the latest gen nvidia H100s being used here? How much of a difference would it make if nvidia open sourced their drivers?

@SlCKB0Y-sb1kg Жыл бұрын

Lol. Never going to happen!

@therealb888 Жыл бұрын

@@SlCKB0Y-sb1kg I haven't followed the news lately, but why? Last time I checked the out cry was to halt LLM research before these H100s are deployed. Why say, never gonna happen?

@SlCKB0Y-sb1kg Жыл бұрын

@@therealb888 Sorry, I should have given context. Various individuals and organisations related to Linux have been extremely vocal with trying to get Nvidia to open their drivers for literally decades. Nvidia has always been extremely hostile to open source and consider their drivers to be an integral part of their intellectual property.

@therealb888 Жыл бұрын

@@SlCKB0Y-sb1kg I know right, even Linus flipped nvidia 😂. I hope there would be leak of all their firmware, drivers, etc A change in senior management?, even better. But now with AI the demand for open sourcing should be from deep pockets. Even microsoft sold windows source code to governments. It's an unfortunate misery that AI research relies on nvidia cuda & it's closed source stack while AMD is going out of it's way to opensource. Time has proven that in software, opensource wins.

@SlCKB0Y-sb1kg Жыл бұрын

@@therealb888 Yea, I love it! kzfaq.info/get/bejne/n7-HrbCn0dCWgoU.html . Realistically though, I think the fact that the whole AI industry is so dependent on Nvidia will become really problematic in the near future.

@SaurabhOKumar Жыл бұрын

What are the energy optimization techniques you use to save the power for all these forms information processing?🤠🤔What does your electric meter display read now-a-days? 🧐🤫🤑🤯 Can you portray energy consumption in modules/sections and represent it in a graphical workflow for the whole picture? Not trying to be sarcastic, just asking. Okay🤗👍

@scosminv Жыл бұрын

Chat GPT is the new Crypto :D

@chrisw8105 Жыл бұрын

Here from MisoDope

@Krishnendulaha 7 ай бұрын

If Microsoft now launches Surface devices with its own in house AI chip with Windows 12 full with AI features its going to change the personal computers... I hope