Microsoft CTO Kevin Scott on How Far Scaling Laws Will Extend

Microsoft CTO Kevin Scott on How Far Scaling Laws Will Extend | Training Data

Рет қаралды 36,661

Күн бұрын

The current LLM era is the result of scaling the size of models in successive waves (and the compute to train them). It is also the result of better-than-Moore’s-Law price vs performance ratios in each new generation of Nvidia GPUs. The largest platform companies are continuing to invest in scaling as the prime driver of AI innovation.
Are they right, or will marginal returns level off soon, leaving hyperscalers with too much hardware and too few customer use cases? To find out, we talk to Microsoft CTO Kevin Scott who has led their AI strategy for the past seven years. Scott describes himself as a “short-term pessimist, long-term optimist” and he sees the scaling trend as durable for the industry and critical for the establishment of Microsoft’s AI platform.
Scott believes there will be a shift across the compute ecosystem from training to inference as the frontier models continue to improve, serving wider and more reliable use cases. He also discusses the coming business models for training data, and even what ad units might look like for autonomous agents.
Hosted by: Pat Grady and Bill Coughran, Sequoia Capital
00:00 - Introduction
01:20 - Kevin's backstory
06:56 - The role of PhDs in AI engineering
09:56 - Microsoft's AI strategy
12:40 - Highlights and lowlights
16:28 - Accelerating investments
18:38 - The OpenAI partnership
22:46 - Soon inference will dwarf training
27:56 - Will the demand/supply balance change?
30:51 - Business models for data
36:54 - The value function
39:58 - Copilots
44:47 - The 98/2 rule
49:34 - Solving zero-sum games
57:13 - Lightning round

Пікірлер: 102

@shrek22 Ай бұрын

Kevin is incredible in this interview. i appreciate him being so open. thank you!!!!

@eirikgg Ай бұрын

Great interview. Happy to be a partner to Microsoft with this guy as CTO

@SiyaMaliChannel Ай бұрын

29:15 the frontier model analogy versus inference as an equivalent of building out your own phone and OS to build an app.😂

@wonseoklee80 Ай бұрын

This is the most honest explanation for whole AI situation at the moment: better frontier models are coming. They might not be AGI, but still can have a huge impact on the industry - cheaper, better reasoning, less hallucination and so on. Anyone who architects the product assuming the frontier model will be stuck at today's capabilities, won't be in a good situation.

@squamish4244 27 күн бұрын

People went from being blown away at GPT-4 to assuming AI will be stuck at it in the space of literally one year. Even Claude 3.5, which is better, didn't do much to change that perception, because it wasn't astoundingly better. It is the strangest and least justified assumption to make about a rapidly advancing technology in living memory. It's how we are wired as humans, that we are almost impossible to impress, but even so, rationally, it is nonsensical. I don't care if it takes one, two or even three more years for the next major leap in frontier models - if GPT-5 is to GPT-4 what GPT-4 was to GPT-3, it will be a staggering improvement, and we will be entering territory that if it isn't AGI, will be so capable as to render the term meaningless. AGI is such a vague term with so many definitions now that the term is already functionally meaningless - yeah yeah, AGI or not, what can this model DO? AlphaFold 2 has been cited over 20,000 times in papers since it became open-source exactly three years ago, turning medicine on its head. But human trials by their very nature take time. And now AlphaFold 3, which is substantially simpler and better, is out, and who knows what it will do, along with all the other AI models being integrated into medicine right now. But because we don't already have the cure for cancer or whatever, and these are medical models and not sexy chatbots, nobody notices outside of those of us who take a particular interest in such things.

@AIForHumansShow Ай бұрын

every time we hear Kevin Scott get interviewed the more we like him

@ToonamiAftermath Ай бұрын

49:17 Gary Marcus is fuming

@squamish4244 Ай бұрын

Gary Marcus is always fuming. It's like Gary Marcus has a Gary Marcus agenda to push as a Gary Marcus cognitive scientist. If Gary Marcus could shut the *fck up for one week about how much AI sucks, I would be impressed - but Gary Marcus is incapable of doing that. Gary Marcus! Gary. Marcus.

@jabowery Ай бұрын

Speaking of Solomonoff, it would be really interesting to find out the details of why Microsoft didn't underwrite the Hutter Prize back in 2006.

@BrayanVZ-of8zl Ай бұрын

This channel is a go-to for information. It's the best!

@tahir2443 Ай бұрын

so incredibly based

@ShowRisk Ай бұрын

this feels such a genuine interview

@ianglenn2821 Ай бұрын

this is very good but watch at 2.5x speed

@squamish4244 12 күн бұрын

Not all of us are hooked on the crazy fast talk of a lot of these interviews.

@GNARGNARHEAD Ай бұрын

slow start but that really got going 😎

@shinkurt Ай бұрын

Great interview

@squamish4244 Ай бұрын

This tells me GPT-4 was the top of the last sigmoid curve, and it will take about a year from now to scale the next generation of models for release. And no, he is not some hack, he DOES know what he is talking about. Check out his Wiki bio.

@nuclearenergy919 Ай бұрын

A year is also very fast if you take into account the change in capabilities of gpt 3 vs gpt 4, gpt 5 would be crazy. And If that be the case I will start believing in this theory of scaling laws

@squamish4244 Ай бұрын

@@nuclearenergy919 Indeed. If it be the case. But really, twice as long as that would still be insanely fast too, again considering the jump between 3 and 4. Even if scaling stopped dead tomorrow, what we already have is enough to massively change society as it is integrated into all kinds of systems over the next decade, and we know that the models are just going to keep getting better in that timeframe anyway, whether it be due to scaling or other methods. There's lots of gas in the tank, scaling is just one tool. Interview was grounded and hype-free. Dude is based af, as others here have said.

@JazevoAudiosurf Ай бұрын

just let this man talk, nothing else to do

@420_gunna Ай бұрын

35:50 dark

@profit_sales_arch Ай бұрын

Is there a Spotify podcast channel of the same?

@sequoiacapital Ай бұрын

Training Data is available wherever you listen to podcasts!

@Tubernameu123 Ай бұрын

co piolt is too hyper filtered. microsoft is a corporation that does not care about out privacy but hes a really neat guy. Lets not forget with their openai partnership and all the sketch behind openai since day one..... then the nsa on the board thats scary. still hes a neat guy.

@chriswiles87 Ай бұрын

dumb

@niederrheiner8468 8 күн бұрын

Bill Gates has a different opinion. He does not expect much improvement with GTP 5...

@squamish4244 Күн бұрын

Either way, we’ll know whether scaling LLMs has exhausted itself or hit diminishing returns in no more than a few years. None of these companies can hype forever. But they’re working on other approaches if scaling fails, and there is no lack of potential in those. These companies didn’t get where they are by being idiotic or investing in one approach. If the LLM bubble bursts, AI will bounce back quickly.

@niederrheiner8468 Күн бұрын

@@squamish4244 There will be other approaches. But why shall OpenAI or Microsoft be better in those "other approaches" than anybody else?

@squamish4244 Күн бұрын

@@niederrheiner8468 I didn't say Microsoft specifically. I just said "none of these companies", by which I meant big tech in general. They are not hedging everything on LLMs and scaling. AlphaFold isn't generative AI, and it's pretty f*cking amazing. Of course, that's DeepMind. But all of these half-dozen or so companies would have to be loaded with morons to not be investing heavily in architectures that aren't generative AI.

@ParnianMotamedi Ай бұрын

is the sequoi digital currency? it listed in pancake swap? Please answer me soon i'm in hurry

@bola986 Ай бұрын

1.5x

@ishaankoratkar9242 Ай бұрын

holy based

@zoner__ Ай бұрын

He could retire now and pursue life 2 but if he's concerned about impact stay. My son works there and feels like a cog at times. He feels like he wants to fix things right but of course there is always pressure to just get it done.

@captain_crunk Ай бұрын

I was at LinkedIn during the msft acquisition, and stayed another 5 years after that. Imho Microsoft is amongst the very best tech companies to work for. Feeling like a cog is obviously par for the course in capitalism across the board, but in my experience it's less prevalent at Microsoft than other places. YMMV, but I prefer being a cog to being a nail, which is fairly common at many startups :)

@umuti5ik Ай бұрын

Getting it done and in the hands of customers quickly is the way to do it right. :)

@Ikbeneengeit Ай бұрын

FYI your wall is broken

@Copa20777 Ай бұрын

I remember him from the whale presentation with Sam Altman.. 🎉🎉🎉 4rmZambia 🇿🇲 and thanks Microsoft🎉🎉🎉

@catshaman7098 Ай бұрын

Not Scott gu?

@dadsonworldwide3238 Ай бұрын

The problem I have above all else is rooted even before 80 years into the transitor age. The whole nation is banking on this, and most arr uneducated to how to triangulate free will inertia in a frame of reference. It has dangerous loopholes to fear for future generations and how the world beliefs won't fit or worse. It puts so much on this means of production beast of burden textualism methodology objectivism = shining capital on the hill metamorphosis detoured by 1900s structuralism just to end back up where our classheritage was. 100s of millions of Americans know more than everything physicalism ease of access that everything doesn't start in greece. In many ways the puritan pilgrim movement is just now picking backup fighting for 3 lines of measure pragmatic common sense objectivism but it's surrounded by and through very messed up irrational informational models and maths to use .

@LevelofClarity Ай бұрын

Can you elaborate?

@dadsonworldwide3238 Ай бұрын

@@LevelofClarityeveryone is equally broke and America has closed its private sector individual liberal power of Entrepreneurship until the more recent decade. in short 1945s Smith_mundt act, everything physicalism deterministic simplicity while pushing infinite sums of complexity upon all we held dear . Flipped our 3 line of measure = truest True known standard into a 3 body problem ,censorship of all things pragmatic common sense objectivism proper.. We granted liberal power to the state & higher education in a call to arms that has since been followed by one excuse after another, rebuild Europe, tranfer wealth to and for liberating all common sense marginalized groups leaving only criminals or extremists. Industrialized all new nations now crying that it's killing the planet after we successfully get eveyone on their feet. Lol

@dadsonworldwide3238 Ай бұрын

It Is a more nostalgic ancestral one as well,picking up where the puritan & pilgrim left off. Since we sold the soul of the nation, after getting so rudly interrupted, it is a more ancestral quest for eqaulibrium, Technically the shining capital on the hill ( metaphor) is metamorphosis In reverse textualism methodology objectivism = technological advancements in concert with free flow of information education.

@dadsonworldwide3238 Ай бұрын

@LevelofClarity we've not only held back and censored our psychology we've been disciplining and attacking our mind thru sanctions, fines,suspensions,monopolization, chasing material dope or physically carving up our bodu tissue to try and fix problems in our lives. We really need this streamlining to help free parents up and allow us to better incentives the family cells pay them back for all that been charged upon passing down countless generations of burden

@GggatsGggss Ай бұрын

From $37K to $45K that's the minimum range of profit return every week I thinks it's not a bad one for me, now I have enough to pay bills and take care of my family.

@VssgGsggsd Ай бұрын

Excuse me for real?,how is that possible I have struggling financially, how was that possible?

@GggatsGggss Ай бұрын

Thanks to Elizabeth Marie Hawley.

@GggatsGggss Ай бұрын

Though I started with as low as $15,000 actually because it was my first time and it was successful, She's is a great personality in the states

@Shanefrancis-vk8ks Ай бұрын

YES!!! That's exactly her name (Elizabeth Maria Hawley) so many people have recommended highly about her and am just starting with her 😊 from Brisbane Australia🇦🇺

@FavourMamahdhe Ай бұрын

I have heard a lot of wonderful things about Elizabeth Maria Hawley on the news but didn't believe it until now. I'm definitely trying her out