Speaker bio: Lily Cheng's expertise lies at the intersection of corporate strategy, governance and advanced technologies, including artificial intelligence for publicly-traded companies. She serves own the board of companies in aviation, travel, luxury retail, professional services and data centres. Her role as an independent non-executive director spans over businesses that trade over US$20 Billion of annual revenue in aggregate. Lily holds a Bachelor of Arts degree in Engineering, a Master of Engineering degree with distinction from the University of Cambridge and holds a graduate certificate in Artificial Intelligence from Stanford University.
Video summary:
00:00:00 - Intro - four key differences between AI today and the "narrow AI" phase
00:00:23 - four key differences between AI today and the "narrow AI" phase
00:01:03 - Single Domain to General - What is the MMLU Benchmark?
00:02:27 - Supervised to Unsupervised Learning - example of ImageNet dataset
00:03:04 - Transfer learning
00:03:27 - Exponential Growth of Neural Network sizes - GPT4 at 1.8 Trillion parameters
00:03:46 - Large Language Model Training Pipeline: Unsupervised Pre-training, Supervised Finetuning, Reinforcement Learning with Human Feedback (RLHF)
00:04:53 - Common Crawl Dataset as massive unstructured dataset for unsupervised learning
00:05:10 - Intuition for how unsupervised learning works - self-supervision, masking
00:06:00 - Analogies between how humans learn and how machines learn
00:06:19 - Reinforcement Learning example with Google Deepmind's AlphaGo Zero - synthetic data
00:06:41 - Embodied AI, AGI
00:07:03 - Fixed Input Scheme to Natural Language - example of AlexNet input format
00:07:30 - OpenAI / Gemini LLM context window size
00:07:51 - Example from Google Gemini paper - multimodal interleaved input
00:08:10 - What is Zero-shot, few-shot prompting?
00:08:55 - Everyone is now a programmer - Jensen Huang, CEO of Nvidia
00:09:06 - Reductive to Generative - Sora example