How to Build an LLM from Scratch

How to Build an LLM from Scratch | An Overview

Рет қаралды 173,408

Күн бұрын

Need help with AI? Book a call: calendly.com/shawhintalebi
This is the 6th video in a series on using large language models (LLMs) in practice. Here, I review key aspects of developing a foundation LLM based on the development of models such as GPT-3, Llama, Falcon, and beyond.
Series Playlist: • Large Language Models ...
📰 Read more: towardsdatascience.com/how-to...
More Resources
[1] BloombergGPT: arxiv.org/pdf/2303.17564.pdf
[2] Llama 2: ai.meta.com/research/publicat...
[3] LLM Energy Costs: www.statista.com/statistics/1...
[4] arXiv:2005.14165 [cs.CL]
[5] Falcon 180b Blog: huggingface.co/blog/falcon-180b
[6] arXiv:2101.00027 [cs.CL]
[7] Alpaca Repo: github.com/gururise/AlpacaDat...
[8] arXiv:2303.18223 [cs.CL]
[9] arXiv:2112.11446 [cs.CL]
[10] arXiv:1508.07909 [cs.CL]
[11] SentencePience: github.com/google/sentencepie...
[12] Tokenizers Doc: huggingface.co/docs/tokenizer...
[13] arXiv:1706.03762 [cs.CL]
[14] Andrej Karpathy Lecture: • Let's build GPT: from ...
[15] Hugging Face NLP Course: huggingface.co/learn/nlp-cour...
[16] arXiv:1810.04805 [cs.CL]
[17] arXiv:1910.13461 [cs.CL]
[18] arXiv:1603.05027 [cs.CV]
[19] arXiv:1607.06450 [stat.ML]
[20] arXiv:1803.02155 [cs.CL]
[21] arXiv:2203.15556 [cs.CL]
[22] Trained with Mixed Precision Nvidia: docs.nvidia.com/deeplearning/...
[23] DeepSpeed Doc: www.deepspeed.ai/training/
[24] paperswithcode.com/method/wei...
[25] towardsdatascience.com/what-i...
[26] arXiv:2001.08361 [cs.LG]
[27] arXiv:1803.05457 [cs.AI]
[28] arXiv:1905.07830 [cs.CL]
[29] arXiv:2009.03300 [cs.CY]
[30] arXiv:2109.07958 [cs.CL]
[31] huggingface.co/blog/evaluatin...
[32] www.cs.toronto.edu/~hinton/ab...
--
Homepage: shawhintalebi.com/
Socials
/ shawhin
/ shawhintalebi
/ shawhint
/ shawhintalebi
The Data Entrepreneurs
🎥 KZfaq: / @thedataentrepreneurs
👉 Discord: / discord
📰 Medium: / the-data
📅 Events: lu.ma/tde
🗞️ Newsletter: the-data-entrepreneurs.ck.pag...
Support ❤️
www.buymeacoffee.com/shawhint
Intro - 0:00
How much does it cost? - 1:30
4 Key Steps - 3:55
Step 1: Data Curation - 4:19
1.1: Data Sources - 5:31
1.2: Data Diversity - 7:45
1.3: Data Preparation - 9:06
Step 2: Model Architecture (Transformers) - 13:17
2.1: 3 Types of Transformers - 15:13
2.2: Other Design Choices - 18:27
2.3: How big do I make it? - 22:45
Step 3: Training at Scale - 24:20
3.1: Training Stability - 26:52
3.2: Hyperparameters - 28:06
Step 4: Evaluation - 29:14
4.1: Multiple-choice Tasks - 30:22
4.2: Open-ended Tasks - 32:59
What's next? - 34:31

Пікірлер: 193

@ShawhinTalebi 7 ай бұрын

[Correction at 15:00]: words on vertical axis are backward. It should go "I hit ball with baseball bat" from top to bottom not bottom to top. 🔧Fine-tuning: kzfaq.info/get/bejne/m6lme8dizazan6c.html 🤖Build a Custom AI Assistant: kzfaq.info/get/bejne/arhxqa2lmMy7n30.html 👉Series playlist: kzfaq.info/sun/PLz-ep5RbHosU2hnz5ejezwaYpdMutMVB0 📰 Read more: towardsdatascience.com/how-to-build-an-llm-from-scratch-8c477768f1f9?sk=18c351c5cae9ac89df682dd14736a9f3 -- More Resources [1] BloombergGPT: arxiv.org/pdf/2303.17564.pdf [2] Llama 2: ai.meta.com/research/publications/llama-2-open-foundation-and-fine-tuned-chat-models/ [3] LLM Energy Costs: www.statista.com/statistics/1384401/energy-use-when-training-llm-models/ [4] arXiv:2005.14165 [cs.CL] [5] Falcon 180b Blog: huggingface.co/blog/falcon-180b [6] arXiv:2101.00027 [cs.CL] [7] Alpaca Repo: github.com/gururise/AlpacaDataCleaned [8] arXiv:2303.18223 [cs.CL] [9] arXiv:2112.11446 [cs.CL] [10] arXiv:1508.07909 [cs.CL] [11] SentencePience: github.com/google/sentencepiece/tree/master [12] Tokenizers Doc: huggingface.co/docs/tokenizers/quicktour [13] arXiv:1706.03762 [cs.CL] [14] Andrej Karpathy Lecture: kzfaq.info/get/bejne/oamTa6meqsiVno0.html [15] Hugging Face NLP Course: huggingface.co/learn/nlp-course/chapter1/7?fw=pt [16] arXiv:1810.04805 [cs.CL] [17] arXiv:1910.13461 [cs.CL] [18] arXiv:1603.05027 [cs.CV] [19] arXiv:1607.06450 [stat.ML] [20] arXiv:1803.02155 [cs.CL] [21] arXiv:2203.15556 [cs.CL] [22] Trained with Mixed Precision Nvidia: docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html [23] DeepSpeed Doc: www.deepspeed.ai/training/ [24] paperswithcode.com/method/weight-decay [25] towardsdatascience.com/what-is-gradient-clipping-b8e815cdfb48 [26] arXiv:2001.08361 [cs.LG] [27] arXiv:1803.05457 [cs.AI] [28] arXiv:1905.07830 [cs.CL] [29] arXiv:2009.03300 [cs.CY] [30] arXiv:2109.07958 [cs.CL] [31] huggingface.co/blog/evaluating-mmlu-leaderboard [32] www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf

@amortalbeing 6 ай бұрын

thanks a lot for the refs , Shahin Jan ❤ keep up the great job 👍

@seanwilner 4 ай бұрын

This is a about as perfect a coverage of this topic as I could imagine. I'm a researcher with a PhD in NLP who trains LLMs from scratch for a living and often find myself in need of communicating the process in a way that's digestible to a broad audience without back and forth question answering, so I'm thrilled to have found your piece! As an aside, I think the token order on the y-axis of the attention mask for decoders on slide 10 is reversed

@ShawhinTalebi 4 ай бұрын

Thanks Sean! It's always a challenge to convey technical information in a way that both the researcher and general audience can get value from. So your approval means a lot :) Thanks for pointing the out. The blog article has a corrected version: medium.com/towards-data-science/how-to-build-an-llm-from-scratch-8c477768f1f9?sk=18c351c5cae9ac89df682dd14736a9f3

@AritraDutta-tz4je 10 күн бұрын

Sir can you tell me how are you training your llms?

@racunars 7 ай бұрын

All the series on using large language models (LLMs) are really very helpful. This 6th article, really helps me to understand in a nutshell the transformer architecture. Thank you. 👏

@ShawhinTalebi 7 ай бұрын

Glad it was helpful!

@shilpyjain6147 Ай бұрын

Hey Shaw - Thank you for coming up with this extensive video on building LLM from Scratch, it certainly gives a fair idea on, how some of the existing LLMs were created !

@GBangalore 4 ай бұрын

Thank you so much for putting these videos together and this one in particular. This is such a broad and complex topic and you have managed to make it as thorough as possible in 30ish minute😮 timeframe which I thought was almost impossible.

@ShawhinTalebi 4 ай бұрын

My pleasure, glad it was informative yet concise :)

@dauntlessRx 2 ай бұрын

This is literally the perfect explanation for this topic. Thank you so much.

@asha328 Ай бұрын

One of the best videos explaining the process and cost to build LLM🎉.

@tehreemsyed8621 4 күн бұрын

This is such a fantastic video on building LLMs from scratch. I'll watch it repeatedly to implement it for a time-series use case. Thank you so much!!

@bradstudio 4 ай бұрын

This was a very thorough introduction to LLMs and answered many questions I had. Thank you.

@ShawhinTalebi 3 ай бұрын

Great to hear, glad it was helpful :)

@ethanchong1026 5 ай бұрын

Thanks for putting together this short video. I enjoy learning this subject from you.

@ShawhinTalebi 5 ай бұрын

Thanks Ethan, glad you enjoyed it!

@sinan325 6 ай бұрын

I am not a programmer or now anything about programming or LLMs but I find this topic fascinating. Thank you for your videos and sharing your knowledge.

@ShawhinTalebi 6 ай бұрын

Happy to help! I hope they were valuable.

@Hello_kitty_34892 4 ай бұрын

Your voice is relaxing.. I love that you don't speak super fast like most tech bros... And you seem relaxed about the content rather than having this "in a rush" energy. def would watch you explain most things LLM and AI! Thanks for the content.

@ShawhinTalebi 4 ай бұрын

Thanks for the feedback. More AI/LLM content to come!

@goldholder8131 3 ай бұрын

This is the most comprehensive and well rounded presentation I've ever seen in my life, topic aside. xD Bravo, good Sir.

@ShawhinTalebi 3 ай бұрын

Thanks so much! Glad you liked it :)

@theunconventionalenglishman Ай бұрын

This is excellent - thanks for putting this together and taking the time to explain things so clearly!

@chrstfer2452 5 ай бұрын

That was simply incredible, how the heck does it have under 5k views. Literal in-script citations, not even cards but vocal mentions!! Holy shit im gonna share this channel with all my LLM enamored buddies

@ShawhinTalebi 5 ай бұрын

Thanks, I'm glad it was helpful. You're referrals are greatly appreciated 😁

@barclayiversen376 Ай бұрын

Pretty rare that I actually sit through an entire 30+ minute video on youtube. Well done.

@LudovicCarceles Ай бұрын

"Garbage in, garbage out" is also applicable to our brain. Your videos are certainly high quality inputs.

@ShawhinTalebi Ай бұрын

Glad it was helpful :)

@lihanou Ай бұрын

clicked with low expectation, but wow what a gem. Great clarity with just the right amount of depth for beginners and intermediate learners.

@ShawhinTalebi Ай бұрын

Glad it was helpful :)

@aldotanca9430 5 ай бұрын

Thoroughly researched and referenced, clear explanations inclusive of examples. I will watch it again to take notes. Thanks so much!

@ShawhinTalebi 5 ай бұрын

Great to hear! Feel free to reach out with any questions or suggestions for future content :)

@aldotanca9430 5 ай бұрын

Thanks! I would have plenty of questions actually, but they are probably a bit too specific to make for a generally relevant video. I am exploring options for a few non-profit projects related to musical education and research. They need to integrate large bodies of text and produce precise referencing to what comes from where, so I was naively toying with the idea to perhaps produce a base model partially trained on the actual text in question. Which, I understood from the video, is a non-starter. So I will look into fine-tuning, RAG and prompt engineering. I suspect I will spend quite a lot of time watching your convent, given you covered quite a lot. I also learned quite a bit more from this specific video. Right now I am studying the basics, including a bit of the math involved, and it is a bit slow going, so I am quite grateful :)

@ShawhinTalebi 5 ай бұрын

@@aldotanca9430 That sounds like a really cool use case (I've been a musician for over 14 years)! If you want to chat about more specific questions feel free to set up some office hours: calendly.com/shawhintalebi/office-hours

@aldotanca9430 5 ай бұрын

@@ShawhinTalebi That's very generous of you! I will book a slot, would love to chat, I think it would help me immensely to rule out blind alleys and at least get a well informed idea of what is feasible to attempt. I did notice the congas, piano and Hanon lurking in the background, so I suspected the topic will be interesting to you. It is about historical research, but it is also very applicable and creative for improvvisation. Perhaps I can compile a very short list of interesting resources, in case you want to check it out at some point for musical reasons :)

@qicao7769 4 ай бұрын

Best and most efficient video about the basic of LLM!!!! I think I have saved 10h for reading. Thanks!

@ShawhinTalebi 4 ай бұрын

Love to hear it! Glad it helped

@mujeebrahman5282 3 ай бұрын

I am typing this after watching half of the video as I am already amazed with the clarity of explanation. exceptional.

@ShawhinTalebi 3 ай бұрын

Thanks, hope the 2nd half didn't disappoint!

@joedigiovanni8758 2 ай бұрын

Great job demystifying what is happening under the hood of these LLMs

@ShawhinTalebi 2 ай бұрын

Happy to help!

@ares106 7 ай бұрын

thank you, this is infinitely more enjoyable for me than reading a paper.

@ShawhinTalebi 7 ай бұрын

😂😂 I’m glad you liked it!

@fab_spaceinvaders 6 ай бұрын

second this, keep the good work flowing all around 🎉 🙏

@EigenA 2 ай бұрын

Great channel, 3rd video in. You earned a sub. Thank you!

@robwarner1858 4 ай бұрын

Amazing video. Lost me through a fair bit, but I came away understanding more than I ever have on the subject. Thank you.

@ShawhinTalebi 4 ай бұрын

Glad it was helpful :)

@aalaptube 4 ай бұрын

Very good information! Answered a lot of core questions I had.

@ShawhinTalebi 4 ай бұрын

Glad it was helpful!

@shih-shengchang19 3 ай бұрын

Thanks for your video; it's awesome. You explain everything very clearly and with good examples.

@ShawhinTalebi 3 ай бұрын

Thanks for the feedback, glad it was clear :)

@ifycadeau 7 ай бұрын

Love these videos! Keep it up Shaw!

@ShawhinTalebi 7 ай бұрын

Thank you 🙏

@DigsWigs2022 16 күн бұрын

Great explanation. I will have to watch it a few times to have a basic understanding 😂

@rezNezami 5 ай бұрын

excellent job Shawhin. Merci.

@ShawhinTalebi 5 ай бұрын

Thanks Reza, glad you liked it!

@MegaBenschannel 5 ай бұрын

Thanks for the great and pack expose. 😀

@ShawhinTalebi 5 ай бұрын

Glad it was helpful 😁

@randomforest_dev 2 ай бұрын

Awesome Video! Thanks.

@funnymono 2 ай бұрын

Exceptional material

@Kiririn 3 ай бұрын

thank you for making this

@ShawhinTalebi 3 ай бұрын

Happy to help!

@LezzGoPlaces 2 ай бұрын

Brilliant!

@romantolstykh7488 2 ай бұрын

Great video!

@echofloripa 7 ай бұрын

Wow, what a great content, thanks for that!! In LLM Fine-tuning, is there also a suggestion table between number of trainable parameters and tokens used (dataset size)?

@ShawhinTalebi 7 ай бұрын

That’s a great question. While I haven’t come across such a table, good rule of thumb is 1k-10k examples depending on the use case.

@echofloripa 7 ай бұрын

@@ShawhinTalebi thanks for the quick reply! What about the number of trainable parameter, should we worry about that? What if my number of examples is smaller than that let's say a 100 to 200?

@ShawhinTalebi 7 ай бұрын

@@echofloripa IMO you've got to work with what you've got. I've heard some people get sufficient performance from just 100-200 examples, but it ultimately comes down to what is acceptable for that particular use case. It might be worth a try. Hope that helps!

@YohannesAssefa-wk5oo 4 ай бұрын

thankyou bro for your help

@ShawhinTalebi 4 ай бұрын

Happy to help!

@PorterHarris 3 ай бұрын

Great content Shaw! Next step Im having troubles figuring out, is there a way to run locally an existing GPT and do prompt engineering or model fine-tuning on it with my own training data?

@ShawhinTalebi 2 ай бұрын

Thanks! While this depends on your local machine specs, the short answer is yes! My next video will actually walk through how to do this using an approach called QLoRA.

@SamChughtai 4 ай бұрын

Thanks, Shaw!! Great video and excellent data, would love to be your mentee, sir!!

@ShawhinTalebi 3 ай бұрын

Thank you for your generosity! I don't currently do any formal mentorship, but I try to give away all my secrets on KZfaq and Medium :) Feel free to share any suggestions for future content.

@akramsystems 2 ай бұрын

This is Gold

@ShawhinTalebi 2 ай бұрын

Glad it's helpful :)

@vijayakashallenki7275 Ай бұрын

Waiting for the complete AI-ML playlist! sir please

@user-cu7vl1jj9v 2 ай бұрын

Very very good!!

@arpadbrooks5317 5 ай бұрын

very informative thx

@ShawhinTalebi 5 ай бұрын

Happy to help!

@mitdasondi2171 Ай бұрын

watching this right before my interview.

@ShawhinTalebi Ай бұрын

Good luck!

@mitdasondi2171 Ай бұрын

@@ShawhinTalebi cleared 1st round, now its on Thursday, i hope your luck brings me my dream job ❤️

@siyufan1084 7 ай бұрын

Arrived right on time! The quality of the video is consistently excellent as always

@ShawhinTalebi 7 ай бұрын

Great to hear! I'm glad they are helpful :)

@gRosh08 Ай бұрын

Cool.

@wilfredomartel7781 5 ай бұрын

🎉❤❤❤amazing video

@ShawhinTalebi 5 ай бұрын

Thank you 🙏

@nobafan7515 2 ай бұрын

Thank you for the video! I was wondering if you can help me. Lets say i ask gpt if romeo and juliet was a comedy or a tragedy, and the only data it has was put in by people that didnt have time to fact check the data, and i wanted my own gpt (lets say this is one of the tiny ones that can easily run on my laptop) so it can explain the history of it so it can explain to me the facts of it. Do i need to dive in the llm model and find that specific data to correct it? Can i fine tune it to improve it (lets say i have a gpu big enough to train this llm)? Is the model fine, but i need a different gpt?

@ShawhinTalebi 2 ай бұрын

If I understood correctly, the question is on how to ensure the LLM gives accurate response. While there are several ways one can do this, the most effective way to give a model specialized and accurate information is via a RAG system. This consists of providing the model specific information from a knowledge base depending on the user prompt.

@vsudbdk5363 5 ай бұрын

Any resources on enrichment of prompt template, I feel in my case difficult one to understand and implement as an LLM returns response based on how we define the template overcoming unecessary context...

@vsudbdk5363 5 ай бұрын

Recently begun exploring Generative AI need like proper guidance on where to learn and do the code part, ik it will be a long journey understanding the math behind it, learning concept and code, staying all night for checkpointing metrics, performance and all.. thank you

@ShawhinTalebi 5 ай бұрын

Great question. The video on prompt engineering might be helpful: kzfaq.info/get/bejne/ZsmWatmrssXIimQ.html

@ShawhinTalebi 5 ай бұрын

That's a good mindset to have. AI is an ocean, with endless things one can learn. This playlist could be a good starting place: kzfaq.info/sun/PLz-ep5RbHosU2hnz5ejezwaYpdMutMVB0

@vsudbdk5363 5 ай бұрын

@@ShawhinTalebi thank you very much

@philtoa334 7 ай бұрын

Nice.

@Nobody2310 19 күн бұрын

what is the most basic technical artifact that is used/required to build any LLM? Is that an existing LLM such as Llama 2?

@ShawhinTalebi 14 күн бұрын

I am not quite sure of the meaning of "most basic technical artifact," but here's one perspective. There are two ways to build an LLM: from scratch and fine-tuning. When training from scratch, the essential piece is the training data used to develop the model. When fine-tuning, the essential piece is the pre-trained model you start from (e.g., Llama2). Hope that helps!

@ronakbhatt4880 6 ай бұрын

@17:08 isnt weight of decoders are wrong if 0 is the weight of token to the future token to it?

@ShawhinTalebi 5 ай бұрын

Sorry I didn't understand your question. Could you rephrase?

@rajez.s7157 4 ай бұрын

Can Ray clusters be used here for mutiple GPUs training of LLMs?

@ShawhinTalebi 4 ай бұрын

I haven't used Ray clusters before, but skimming their website it seems like it was specifically made for ML workloads.

@abcoflife6420 5 ай бұрын

Thank you so much for rich information, my target is to DIY one from scratch .. 😢 for sure it wont be billions of tokens, I want to make it practical for example for home management, or school reporting system ... instead of static reports . to enable it to create and run its own sql queries and run it .. 😅

@ShawhinTalebi 5 ай бұрын

Happy to help! To make something practical I'd recommend using an existing model fine-tuned to generate SQL queries e.g. huggingface.co/defog/sqlcoder

@abhishekfnu7455 2 ай бұрын

Is there a way to use Data Dictionary to train LLM model to generate SQL queries later on?

@ShawhinTalebi 2 ай бұрын

Yes, but you will likely need to transform the data a bit before it can be used for fine-tuning. I give a concrete example of this here: kzfaq.info/get/bejne/arhxqa2lmMy7n30.html

@Nursultan_karazhigit 2 ай бұрын

Hello , Thanks . Do you know is it possible to create an own LLM for own startup?

@ShawhinTalebi 2 ай бұрын

Of course this is possible. However, it is rarely necessary. I'd suggest seeking simpler (and cheaper) solutions before jumping to training an LLM from scratch.

@Nursultan_karazhigit 2 ай бұрын

@@ShawhinTalebi thanks

@user-oo2co6xb8u 5 ай бұрын

Hi, i have domain specific pdf files . How do i train using transfer learning? Please advise

@ShawhinTalebi 5 ай бұрын

Depends on what you mean by transfer learning. If you simply want to extract knowledge from a PDF I'd recommend exploring RAG or using off-the-shelf solutions like OpenAI Assistants interface. Happy to clarify, if I misinterpreted the question.

@dohua_ai 6 ай бұрын

So my dreams about own LLM are broken(( So as i understood the only way to build some personal LLM is FineTuning? Atleast while cheap ways of training not appeared yet...

@ShawhinTalebi 6 ай бұрын

I wouldn't give up on it! My (optimistic) conjecture is as we better understand how these models actually work we will be able to develop ones that are much more computationally efficient.

@hashifvs519 4 ай бұрын

Can you post a video onu continual pretraining of llms like Llama

@ShawhinTalebi 4 ай бұрын

Thanks for the great suggestion. I’ll be doing more content on fine-tuning so that will be a good topic to cover there.

@amparoconsuelo9451 5 ай бұрын

Can a fine-tuned LLM be repurposed and re-fine-tuned for more than one task?

@ShawhinTalebi 5 ай бұрын

Yes it can! In fact, that is what OpenAI did with their RLHF technique to create their InstructGPT models

@TheIronMason 2 ай бұрын

When it comes to transformers. Are you saying they're more than meets the eye?

@ShawhinTalebi 2 ай бұрын

That's a good way to put it 😂

@issair-man2449 5 ай бұрын

Hi, hoping that my comment will be seen and responded... I FAIL to understand: If a simple model learns/predicts, couldn't we prompt it to delete the trash data and train itself by itself autonomously until the model becomes super intelligent?

@ShawhinTalebi 5 ай бұрын

LLMs alone only do token prediction, as discussed in the first video of this series: kzfaq.info/get/bejne/qqx4mLiEr6e9knk.html While an AI system could in principle train itself, it would require much than just LLM to pull that off.

@hari_madh 3 ай бұрын

bro i want to build an LLM.. does this video help me learn myself and build LLM myself? possible? (i did not see it till now)

@ShawhinTalebi 3 ай бұрын

While this video may be a helpful first step, more resources will be necessary. Here are a few additional resources I recommend. - kzfaq.info/get/bejne/oamTa6meqsiVno0.html&ab_channel=AndrejKarpathy - huggingface.co/learn/nlp-course/chapter1/1?fw=pt

@techdiyer5290 4 ай бұрын

What if you could make a small language model, that maybe only understand english, can understand code, and is easy to run?

@ShawhinTalebi 4 ай бұрын

That is a compelling notion. If we can get there, then it would make this technology even more accessible and impactful.

@shrinik1969 4 ай бұрын

Size = accuracy...small may not give u what u want

@Decapodd 2 күн бұрын

NanoChatGPT

@jackflash6377 6 ай бұрын

Just now asking GPT4.0 to help me with training text. It is not allowed to assist in training any LLMs and would not give me anything.

@ShawhinTalebi 6 ай бұрын

I believe it’s now against OpenAI’s policy to use their models to train other models. You may need to look to open-source solutions eg Llama2, Mistral

@petevenuti7355 5 ай бұрын

@@ShawhinTalebi . How could it possibly stop it ? If the model being trained fed the prompt and used the response for reenforcment and alignment?

@guerbyduval4104 Ай бұрын

Do you have a course on how to do it as a programmer instead of *like a chat gpt talker* ?

@ShawhinTalebi Ай бұрын

I don't have a from scratch coding tutorial yet. But I am a fan of the one from Andrej Karpathy: kzfaq.info/get/bejne/oamTa6meqsiVno0.html

@lyonspeterson1094 Ай бұрын

Good contents. But when I watch the video, there are so many ads. I;m even confused what I am supposed to watch.

@crosstalk125 4 ай бұрын

Hi, I like your content. But I want to point out that what you are calling tokenization is vectorization. Tokenization breaks documents/sentences/words into subpart and vectorization converts tokens into numbers. Thanks

@ShawhinTalebi 3 ай бұрын

Thanks for raising that point. Here I'm lumping the two together, but these are 2 separate steps.

@catulopsae 28 күн бұрын

What does it mean the amount of parameters???

@ShawhinTalebi 28 күн бұрын

Good question. A model is something that takes an input (say a sequence of words) and produces an output (e.g. the next most likely word). Parameters are numbers which define how the model takes inputs and translates them into outputs.

@catulopsae 23 күн бұрын

@@ShawhinTalebi thank you

@Sunnyangusyoung 2 ай бұрын

What if I don’t want to build my model but work for someone who is building one.

@hypercoder-gaming 5 ай бұрын

When you were calculating the cost, you estimates that a 10b model would take 100k GPU hours but Llama 2 took 180k GPU hours and that was 7b. These estimates are way off. How is it that 100b costs less than 70b?

@ShawhinTalebi 4 ай бұрын

The numbers from Llama 2 were only meant to give an idea of scale. More precise estimates will depend on the details of the use case.

@PabloPernambuco Ай бұрын

Now, I am discovering my low QI... 0,001% of learning...😂

@MaxQ10001 4 ай бұрын

Is it just me, or does the math on the "how much does it cost" not make sense? 7b uses 180,000 hours, so 10b used 100,000 🤔 hours

@ShawhinTalebi 4 ай бұрын

This is only meant to give a sense of the cost's scale, so I round to the nearest order of magnitude :)

@Joooooooooooosh 3 ай бұрын

Wait how did we get from $180K for a 7B model to $100K for a 10B model...

@ShawhinTalebi 3 ай бұрын

This is what we Physicists call an "order-of-magnitude estimate"

@imaspacecreature 3 ай бұрын

Some of you need to look into your own "internal libraries", this video is someone attempting to teach you where to get the fish. Some of you get so hungry for the fish, but won't even understand the water it resides in.

@jamesmurdza 4 ай бұрын

The matrices at 16:40 don't look right to me. I think the words labelling the rows should go from top to bottom, not bottom to top.

@ShawhinTalebi 4 ай бұрын

Good catch! Yes, the word labels are inverted on the Y axis. A corrected visualization is provided in the blog: medium.com/towards-data-science/how-to-build-an-llm-from-scratch-8c477768f1f9?sk=18c351c5cae9ac89df682dd14736a9f3

@MrSpikegee 2 ай бұрын

The cost estimation does not seem correct, you have to take into account the time taken to train to estimate the hardware buying scheme, here you estimate as if we wanted to train the model in one hour - if you accept for eg. 4 days approx. 100h, you need only ten A100 which would be 100k€, so less than with the renting option.

@ShawhinTalebi 2 ай бұрын

I may be missing something here. Based on the numbers in the Llama paper, training 10B and 100B parameter models with 10 A100s, would take about 10,000 hr (1.1 years) and 100,000 hr (11 years), respectively.

@Vermino 7 ай бұрын

Just my prediction on data curation & copyright, what is going to happen is the companies of LLM's will do what Google did back in the day with scraping websites. Then once legislation passes, they will say "if you don't want your data crawled, opt-out with robots.txt". Right now it's a grey area and companies are building their data as quickly as possible to get in before the regulation. "Better to ask for forgiveness, rather than permission."

@ShawhinTalebi 7 ай бұрын

I can see that. It seems to be a hot-button topic these days.

@gamea9g349 Ай бұрын

why cant they use algorithims insted of this huge data sets

@ShawhinTalebi 28 күн бұрын

This is the big paradigm shift with machine learning (ML). Traditionally programmers implement logic into computers line by line. However, there are some tasks that are difficult (if not impossible) to explicitly program computers to do (e.g. driving, responding to open-ended prompts, etc). For these tasks ML allows us to program computers with data.

@DYLOGaming 26 күн бұрын

Well this was a video that explains things conceptually but does 0 building from scratch

@ShawhinTalebi 21 күн бұрын

Sorry for the lack of code in this one, maybe other videos in the series will be more helpful. An end-to-end technical guide on building an LLM isn't something I can fit in one video, but I hope to share more videos toward this end in the future.

@guerbyduval4104 Ай бұрын

3:23 What is the point of talking gpu hours and money spent on cloud. I have and Nvidia RTX 3070 with 5120 cuda cores. I don't care spending money to train. I just can train on my laptop.

@ShawhinTalebi Ай бұрын

That was to give context on training an LLM from scratch. This is negligible for smaller models.

@bittertruthnavin 4 ай бұрын

90 % of the people out there are just trying learn how to drive a car, not how to build a car. All these youtubers are showing is how to build a car which further complicates the process of new AI adoption and demotivate people. I just saw video by this lady somewhere and she showed she trained mistral 7b LLM in less than 1 hours for $1. Folks keep looking for better and simple information, remember we don't need to know how to build a car in order drive one.

@saisriramakarthikeyakollur3989 4 ай бұрын

I'm curious to know more about mistral 7b. Any chance you could share your source? All I found was the trained model on huggingface

@ShawhinTalebi 4 ай бұрын

Excellent point. Using these models to solve problems is much more impactful than knowing how they work. Other videos in this series are more hands-on than this one. Suggestions for future content are also appreciated :) kzfaq.info/sun/PLz-ep5RbHosU2hnz5ejezwaYpdMutMVB0

@Drdozzer 3 ай бұрын

What if what I want to do is build a car 😅

@scotthjackson5651 3 ай бұрын

I literally want to know about how to build an LLM.

@imaspacecreature 3 ай бұрын

This comment is extremely concerning. Do what you will, but a LOT of humans are PROGRAMMED CONSUMERS. Aka, you are like a Tool, you only do what you are given, like a task. Don't uplift or look down on other for wanting to know how things work. And also, some people drive cars and don't even know *how to drive, and they kill people every minute on a roadway. You should rethink your own logic on this, before you even entertain a "artificial intelligence"..

@genkidama7385 Ай бұрын

internet is full of trash to be honest. train llm on books would be a great leap for mankind. code datasets are full of "oh its too complex" and only produces comments instead of code. Machine speaking as a human and complaining about things beeing too complicated to do. thats how lame datasets are. computer is a computing device and refuses to work because the dataset says things are too complex. jesus. unbelievable levels of trash.

@ShawhinTalebi Ай бұрын

The greatest challenge is that their aren't enough (high-quality) books to keep scaling these models. How we solve this problem is going to define the next wave of LLMs. (Maybe we'll switch to lectures and other videos)

@handleking1 Ай бұрын

Bro doesn't even know how to spell NVDIA correctly.

@ShawhinTalebi Ай бұрын

Spelling was never my strong suit XD

@ButchCassidyAndSundanceKid Ай бұрын

A bit disappointed with this video, there is no coding !

@ShawhinTalebi Ай бұрын

Thanks for the feedback. Yes this one stays relatively high-level since a coding video would be (much) longer. For a hands-on guide I'd recommend Andrej Karpathy's video: kzfaq.info/get/bejne/oamTa6meqsiVno0.html

@thechoosen4240 4 ай бұрын

Good job bro, JESUS IS COMING BACK VERY SOON;WATCH AND PREPARE

@ShawhinTalebi 4 ай бұрын

🙏

@nick066hu Ай бұрын

Thank you for putting together this video, helped me a lot to understand LLM training. One question: with the advent of trillion token models and beyond, I wonder where will we get all that training input data from. I guess we already consumed what all humanity has produced in the last 5000 years, and by adding another 10M digitized cat videos, the models will not be smarter.

@ShawhinTalebi Ай бұрын

Good question! I suspect there is still much content out there that hasn't been touched by LLMs i.e. non-digital text and proprietary data. Nevertheless, this content is still finite and the "just make a bigger model" approach will eventually hit a limit.

@gameplaychulangado494 3 ай бұрын

i am from brazil and my english is not so nice but i did understand you almost well. I think that biggest challenge will be math and matrix 3D and more. 🥲

@ShawhinTalebi 3 ай бұрын

Glad it made sense :)