The 10,000x Yolo Researcher Metagame

No video

The 10,000x Yolo Researcher Metagame - with Yi Tay of Reka

Рет қаралды 2,538

Ай бұрын

Pod: www.latent.spa...
It’s easy to get de-sensitized to new models topping leaderboards every other week - however, the top of the LMsys leaderboard has typically been the exclusive domain of very large, very very well funded model labs like OpenAI, Anthropic, Google, and Meta. OpenAI had about 600 people at the time of GPT-4, and Google Gemini had 950 co-authors. This is why Reka Core made waves in May - not only debuting at #7 on the leaderboard, but doing so with all-new GPU infrastructure and 20 employees with no more than 5 people on pre-training and a relatively puny $60m in funding.

Пікірлер: 6

@LatentSpaceTV Ай бұрын

Audio pod and show notes: www.latent.space/p/yitay

@prasannad5719 Ай бұрын

keep em coming 🔥

@soupbob5813 Ай бұрын

[00:00:00] Intro [00:01:57] Yi Tay Intro [00:03:02] Path into LLMs [00:09:41] Google Brain: PaLM, UL2, DSI, Emergent Abilities [00:11:54] PaLM 2 [00:15:27] Emergent Abilities [00:18:26] Quoc Le [00:24:16] Marketing Research: How to Start from Zero with No Reach [00:27:34] What's needed to be a successful AI Researcher? [00:30:31] Reka Origin [00:33:24] Starting Reka Infra [00:35:04] Why not to use TPUs outside Google [00:36:29] Chaotic vs Stable Infra [00:38:04] Risk Sharing of Bad Nodes [00:41:05] Checkpointing and Orchestration [00:43:39] Reka Flash/Core/Edge [00:46:59] Recruiting the team [00:47:22] Noam Architecture - Swiglu, GQA, RMSnorm, ROPE [00:52:26] Encoder-decoder vs Decoder-only [00:55:52] LLM Trends - Llama 3 and Phi 3 Glowup [00:57:46] LLM Trends - Benchmarks and Evals [01:03:25] LLM Trends - Early vs Late Fusion Multimodality [01:07:22] LLM Trends - Scaling Laws [01:09:41] LLM Trends - Long Context vs RAG [01:12:31] Long Context vs Finetuning [01:14:14] If emergence is real, when does Efficiency work? [01:17:41] MoEs and Upcycling [01:20:47] The Efficiency Misnomer - Efficiency != Speed [01:25:05] Open Source vs Closed Models [01:28:08] Personal Productivity [01:33:19] Singapore vs US Academic Scene [01:37:42] Building Silicon Valley outside Silicon Valley [01:40:29] TechInAsia Meetup

@swyxTV Ай бұрын

that is the audio timestamps, but the video has different editing therefore i didnt put it there

@NA-sd8bw Ай бұрын

I am a beginner in the LLM world , but I have 5 years experience in ml and data science. Can you a get a and learn from this amazing guy in his start up ? I can work for free for him.