Let's build the GPT Tokenizer

Рет қаралды 538,230

Күн бұрын

The Tokenizer is a necessary and pervasive component of Large Language Models (LLMs), where it translates between strings and tokens (text chunks). Tokenizers are a completely separate stage of the LLM pipeline: they have their own training sets, training algorithms (Byte Pair Encoding), and after training implement two fundamental functions: encode() from strings to tokens, and decode() back from tokens to strings. In this lecture we build from scratch the Tokenizer used in the GPT series from OpenAI. In the process, we will see that a lot of weird behaviors and problems of LLMs actually trace back to tokenization. We'll go through a number of these issues, discuss why tokenization is at fault, and why someone out there ideally finds a way to delete this stage entirely.
Chapters:
00:00:00 intro: Tokenization, GPT-2 paper, tokenization-related issues
00:05:50 tokenization by example in a Web UI (tiktokenizer)
00:14:56 strings in Python, Unicode code points
00:18:15 Unicode byte encodings, ASCII, UTF-8, UTF-16, UTF-32
00:22:47 daydreaming: deleting tokenization
00:23:50 Byte Pair Encoding (BPE) algorithm walkthrough
00:27:02 starting the implementation
00:28:35 counting consecutive pairs, finding most common pair
00:30:36 merging the most common pair
00:34:58 training the tokenizer: adding the while loop, compression ratio
00:39:20 tokenizer/LLM diagram: it is a completely separate stage
00:42:47 decoding tokens to strings
00:48:21 encoding strings to tokens
00:57:36 regex patterns to force splits across categories
01:11:38 tiktoken library intro, differences between GPT-2/GPT-4 regex
01:14:59 GPT-2 encoder.py released by OpenAI walkthrough
01:18:26 special tokens, tiktoken handling of, GPT-2/GPT-4 differences
01:25:28 minbpe exercise time! write your own GPT-4 tokenizer
01:28:42 sentencepiece library intro, used to train Llama 2 vocabulary
01:43:27 how to set vocabulary set? revisiting gpt.py transformer
01:48:11 training new tokens, example of prompt compression
01:49:58 multimodal [image, video, audio] tokenization with vector quantization
01:51:41 revisiting and explaining the quirks of LLM tokenization
02:10:20 final recommendations
02:12:50 ??? :)
Exercises:
- Advised flow: reference this document and try to implement the steps before I give away the partial solutions in the video. The full solutions if you're getting stuck are in the minbpe code github.com/karpathy/minbpe/bl...
Links:
- Google colab for the video: colab.research.google.com/dri...
- GitHub repo for the video: minBPE github.com/karpathy/minbpe
- Playlist of the whole Zero to Hero series so far: • The spelled-out intro ...
- our Discord channel: / discord
- my Twitter: / karpathy
Supplementary links:
- tiktokenizer tiktokenizer.vercel.app
- tiktoken from OpenAI: github.com/openai/tiktoken
- sentencepiece from Google github.com/google/sentencepiece

Пікірлер: 985

@gustavojuantorena 4 ай бұрын

I'm amazed at the times we live in. One of the top AI experts in the world is sharing free tutorials that teaches technologies in great detail including examples and code. Thank you very much.

@von.changfeng 4 ай бұрын

Yeah, it's unbelievable, yet very helpful

@neuralthink 4 ай бұрын

@@von.changfeng We're living an incredible era.

@hazel8king 4 ай бұрын

unprecedented

@louisyuliu7200 4 ай бұрын

can't agree more

@DmitryBaranovskiyMrBaranovskyi 4 ай бұрын

@@louisyuliu7200 You can come here to Ukraine and.. find out that times r not the best. But ye. Respect to Andrej to sharing knowledge

@jstello 4 ай бұрын

The fact that Andrej prefers being here with us over developing self driving or developing AGI is surely one of the great things about living in the present times.

@osalbaro 4 ай бұрын

Not to worry, he's simply applying the Feynman method.

@jibbscat5146 4 ай бұрын

A True Master

@web 4 ай бұрын

Is Andrej hype-phobic and this means self-driving and AGI application research reached plato? That would explains why he is dedicating himself into teaching basics so more students go into the field

@domg7359 4 ай бұрын

@@web No, not at all. It's moving faster than expected and he wants the knowledge spread out and not contained to corporations before it's too late. Quite the opposite of your guess. Models need to be open-source or there will be huge disparities in capabilities.

@for-ever-22 4 ай бұрын

@@webHe’s an advocate for free and accessible knowledge for all plus he simply choose to do what he loves sharing knowledge with everyone

@SydneyPanda2016 3 ай бұрын

You could have made a million bucks or spent time on a beach with a cocktail. Instead you chose to do this for the benefit of unknown strangers. You are an amazingly talented and generous man. Thank you so much!

@vlad1655 2 ай бұрын

Hah, so you know his net worth?)

@The_Uncertainity_Principal 2 ай бұрын

he is already likely worth 100M+ lmao . and rightfully so.

@adrielomalley 22 күн бұрын

RobinHood_of_2024 = Andrej_Karpathy

@MShahbazKharal 4 ай бұрын

This guy left Tesla, OpenAI, and God knows how many other companies so he can teach us how the state of art AI works, from scratch, clearly explained. Such a pure gem. Thank you very very very much.

@13thbiosphere 4 ай бұрын

200,000 people watching probably only about 500 of those can actually implement any of it... One in a thousand probably more like

@akzsh 4 ай бұрын

what do you mean? (like 500 of those will try to implement it themselves?) @@13thbiosphere

@akf2000 4 ай бұрын

@@13thbiospherevery true but a lot of people, like me, just like watching people who really know their shit, in whichever area

@simonvutov7575 4 ай бұрын

@@13thbiosphere Yeah thats completely true, but I would argue it is still worth it even just for those 500 people. 500 new AI researchers could make a very significant difference in the field. (btw im one of those 500 that can implement lol)

@jcon2060 4 ай бұрын

@@simonvutov7575 pathetic

@julianhunt8177 4 ай бұрын

Whenever Karpathy leaves a job the ML world jumps forward a bit with his educational content

@kuoldeng4568 4 ай бұрын

I cannot express enough how grateful I am for your incredible tutorials. As a 44-year-old South Sudanese individual with three children and a full-time job, recently relocated from the UK to the US, my days are filled to the brim with responsibilities. However, I have made it a priority to utilize every ounce of my free time to follow your series of lectures. I must say, I have never been as excited about learning a new topic as I am now. Your clear explanations and engaging content have ignited a passion for learning within me that I never knew existed. Thank you from the bottom of my heart for all that you do.

@n11ahkv 4 ай бұрын

Welcome to America!! Great to have you.

@myhandle__ 4 ай бұрын

Internet is amazing

@MrEmbrance 4 ай бұрын

calm down

@que_93 4 ай бұрын

You being one among them, isn't it? Think... @@MikeDonaldson-eh2ru

@davemarko9480 4 ай бұрын

❤❤❤ Am South Sudanese, watching this from Nairobi, Kenya

@niceguysayshi5765 4 ай бұрын

When andrej uploads i immediately drop everything and watch the complete video. Thank you so much for your valuable content!

@pranavnair5232 4 ай бұрын

Same here!

@finojose 4 ай бұрын

me too

@teslabot5650 4 ай бұрын

I tried to watch his very first video. dint understand a thing.

@niceguysayshi5765 4 ай бұрын

@@teslabot5650 it takes some time. You need to have basic programming skills to understand this. Also algorithms help a bit.

@xspydazx 4 ай бұрын

yes the first few were not easy but after while it has been very good to listen to and understand the process to think with regard to some projects... as you note the projects are often simple and not over complicated!

@DavidOndrej 4 ай бұрын

This is honestly better than OpenAI releasing a new model

@xspydazx 4 ай бұрын

yes building the components and explaining the full concept is truly open source .... a paid resource is not open source .... the aim is to learn how to make things for yourself so that when such things are removed or censored they are already out in the publics hands ... how can you regulate people own experiments! .... next we need to be able to build a full model with the components with only our own data and not "The common crawl or Wiki data" hence a closed domain llm.... as it should be! everybody should have the right to create their own AI! with their own DATA !... in the future the merging of models to have multiagent systems is the way .... so plug in models into the multi aGI!

@extensionalan6060 2 ай бұрын

Ha ha, if GPT-5 isn't the next release, it means OpenAI hasn't released anything at all. Incremental releases are just tiring because it seems like they've only fixed the bugs from the previous model using hundreds of millions of high-quality human feedback in 2023, and nothing else is really happening.

@yoJuicy 4 ай бұрын

Andrej, content like this is advancing the industry more than most companies are. Thank you for educating us. You are pushing everybody forwards!

@coder_rc 4 ай бұрын

We're so back!!!!

@TheLokiGT 4 ай бұрын

He's even more back :)

@coder_rc 4 ай бұрын

@@TheLokiGT he never left

@ultrasound1459 4 ай бұрын

@@coder_rche is inside u😢💀

@coder_rc 4 ай бұрын

@@ultrasound1459 I don't mind

@amanmamgain3904 4 ай бұрын

Its so over

@entonbot8375 4 ай бұрын

the intrinsic value in andrej making videos for the world is huge. Please andrej keep doing this, no other jobs can output the alpha you're delivering here, pushing the boundaries of the best educational content on the planet. You are born for this.

@sanesanyo 4 ай бұрын

Thanks, Andrej, for taking your time and sharing your knowledge.

@emranmohammadabuanas9476 4 ай бұрын

Andrej, you're really good at teaching. We're lucky that you're spending your valuable time to prepare these free contents.

@TheMoefd 4 ай бұрын

Been waiting on the lecture to drop ever since we saw the Github repo go public. Andrej, on behalf of the academic and professional communities, thank you for putting the effort into creating this open knowledge. On a personal note, I've been following your work since the early OpenAI days, and your passion during keynotes/presentations has supported my choice of pursuing a PhD in the field. Thank you Andrej.

@Dyas777 4 ай бұрын

The attention to the details... You prepared this material briliantly. Thank you.

@Alilinpow2 4 ай бұрын

Feynman-class teaching abilities. Amazing. Thank you very much as always Andrej

@LLMTokenizationID 4 ай бұрын

Thank you Andrej Karpathy. Thanks for teaching Python libraries, GPT generator codes. Great work

@JH-bb8in 4 ай бұрын

Congrats Andrej on your new venture!

@ultrasound1459 4 ай бұрын

What is his venture?

@shringmusic6988 4 ай бұрын

@@ultrasound1459This 🙂

@Daukposse 4 ай бұрын

The level of detail and nuance share in this video is insane!

@Pythoncode-daily 4 ай бұрын

Thank you, Andrey! In an environment where 99% of conversations about LLM are marketing, technical videos are especially valuable and interesting!

@arildboes 4 ай бұрын

Thank you very much for this. I'm from a traditional programming background, and your videos on AI are really excellent. I have followed the previous sessions, being able to implement the things correctly, and compare some of the stuff, with your results. That is awesome! Good luck.

@sebah1991 4 ай бұрын

I haven't watched the video yet but I just wanted to say I really appreciate you taking the time to teach this. I loved all of your previous videos. You are an excellent teacher and your style makes it super easy for me to learn.

@MWileY-nj1yb 4 ай бұрын

Thank you so much Andrej! Your course content is accessible, concise and palpable to boot!

@prodrectifies 4 ай бұрын

Amazing! Was waiting patiently for another upload

@netanelmad 4 ай бұрын

Liking before watching the video. So glad to see you continuing to upload!

@von.changfeng 4 ай бұрын

Wow, I just got a notification about your new video, I love these videos, thank you for this content Andrej, this is super informative

@Parisneo 4 ай бұрын

Welcome back Mr Karpathy. Your videos are among the best I have ever seen about LLMs. Thank you very much for sharing this stuff with us.

@scottsun345 4 ай бұрын

Thank you, Andrej! Amazing lecture! Your contents are filling in the nooks and crannies that other people rarely talk about or go deep into. I always learn a lot!

@antoniobarbalau1107 4 ай бұрын

Oh my god. I never understood tokenization (hoped i would remain on vision :)) ). This is golden. Thank you so much ❤❤❤❤ you are my hero, I've learned deep learning from your cs231 while in college (we did not have deep learning as an option) and that got me a ML job. Thank you so very much ❤❤❤

@alainherreman3685 4 ай бұрын

Sorry Andrej, but you've missed the mark: you tell us that you don't like this topic, but you've still managed to make it fascinating!

@johnnypeck 4 ай бұрын

I had other plans but you said "space egg" and I was hooked. You've gone and done that engaging professor thing again. Thank you. I think you doing this is incredibly impactful. It's a real inspiration. Please keep it up.

@minister1005 3 ай бұрын

Thank you, Andrej. This series taught me so much about llms. I never thought I could get so excited watching educational videos. My heart started pounding when I started watching this. Such a quality video. So good 👍

@projects_with_mufty1621 4 ай бұрын

It’s about time. Haven’t watched it yet but I know it’s going to be good and educative. Thanks Andrej for this video.

@arunkallaje 4 ай бұрын

We are so back 😊 Thanks!!

@ChairmanHehe 4 ай бұрын

thank you so much for continuing this series andrej- its been absolutely foundational for me

@drogoknez1488 4 ай бұрын

Andrej, I hope you know how ridiculously great teacher you are, and thank you for taking the time to share it with us!!

@nanow1990 4 ай бұрын

Thank you very much for this ❤

@mb345 4 ай бұрын

Yes! More tutorials! Thank you Andrej!

@BradCordovaAI 4 ай бұрын

He's back in the game! Thanks for posting it. Such a great combination of feeling your passion, very detailed serious information, the clarity and simplicity of the explanation.

@flokleiser 4 ай бұрын

the fact that this is available completely for free is absolutely amazing, the passion and amount of time/effort you spent on this is reflected by the pure quality, i cant thank you enough for doing this andrej

@nirajabcd 4 ай бұрын

You are one of my favorites in ML/AI community. One of the greatest mentors.

@ZhengQu 4 ай бұрын

Andrej was wrong about one thing though. This video is not dry and boring at all as he indicated in the video. I watched the two hours plus video with joy and excitement. Thank you!

@user-vh3jg4uk7o 4 ай бұрын

I don't which tokens would best be used to say "Thank you" but I feel so lucky to even get to watch and learn from you.

4 ай бұрын

This is an excellent video on the explanation of LLM tokens. Great job Andrej!

@justjeremiah4255 4 ай бұрын

OpenAI was such a closed cult, Andrej gave himself to the world. 🌹❤️🌹 Andrej deserves his flowers.

@sankafoodey 4 ай бұрын

Watching from Muqdisho, Somalia. Much ❤ & appreciation to you Dr Karpathy

@rufex2001 3 ай бұрын

You’re a hero, Andrej! Thanks for the massive contributions!

@piper_of_the_dawn 2 ай бұрын

The best thing I have learnt from Andrej's videos is 'How to learn?' Break things down to axiomatic levels and then connect those pieces together! Its truly enlightening!

@nicolasjorquera8465 4 ай бұрын

It’s going to be beautiful 😮

@BisratTekle 4 ай бұрын

I am watching this late at night on my phone when I should already be sleeping. Mr. Karpathy’s videos are pure treasure.

@user-th7jy6zy1f 4 ай бұрын

Thank you for a high quality session. Hats off to Andrej

@tarandeepsingh9197 4 ай бұрын

Awesome content Andrej. Thank you for spending your time and sharing the knowledge with all of us.

@KashyapArjun_ 4 ай бұрын

He's back!

@SanthoshKumar-dk8vs 4 ай бұрын

Here we go💥

@tough_year Ай бұрын

wow. I am amazed by how intuitive this is.

@jmvillwock 4 ай бұрын

Thank You Andrej! I really appreciate you taking your time to put this together for everyone. You are so great at this. I know you really do not love doing this - but know you are making an impact.

@itsm0saan 4 ай бұрын

I really feel guilty watching this for FREE. Thanks so much Andrej ❤🙏🏻

@Adhil_parammel 4 ай бұрын

We are so back🎉

@zyzzyva303 4 ай бұрын

Actually, a fantastic insight into the issues. Thank you Andrej. Please continue.

@frankl1 4 ай бұрын

The best tutorial on Tokenizer I have ever seen. Thanks a lot for sharing your knowledge with the world.

@RAKIBKHAN-gm9dx 4 ай бұрын

Finally 💙

@coralexbadea 4 ай бұрын

1:27:50 The moment when you realise there is more to life than research. 😅😂

@mennovanlavieren3885 4 ай бұрын

Quickly ignored it like a pro, don't let it interfere.

@mdc8223 4 ай бұрын

Thank you very much for your patience and for record this!! :D

@dslkgjsdlkfjd 4 ай бұрын

This video is so valuable. This is a complete game changer. I have a lot of love and appreciation for what you do. Thank you so much for your time and effort!

@heyrishiraj 4 ай бұрын

The Lord is back.. Now make OpenAI secrets truly OPEN!

@noone-ld7pt 4 ай бұрын

You are providing a deep understanding of the most important technology in history to the public and I just want to thank you so much for it.

@iJustinCabral 2 ай бұрын

Just want to say thank you for the entire series of Zero to Hero. I'm about to finish up this final video and I can't believe how much I've learned about neural networks, back prop, and LLMs over the course of the lectures. Truly thankful to you for taking the time to spread your knowledge amongst the community. Let us all be so kind with the knowledge we hold in the future.

@vacsa 4 ай бұрын

Thank you for taking the time to prepare these videos for us!

@rishika1109 8 күн бұрын

This is such a great tutorial! Very well explained through the use of the notebook, which demonstrates everything live. Thanks for putting this out!

@dmihalache 3 ай бұрын

Thank you, Andrej, for all these lectures; you’re an amazing and wonderful teacher. It’s such a huge privilege for all of us to be able to learn from you, and we’d be so lucky if you’d continue the series of lectures. You truly make a huge difference for ML learners, both beginners and advanced.

@ndamulelosbg8887 4 ай бұрын

We really appreciate your generousity Andrej, this is great content🔥

@kasper52786 4 ай бұрын

I’ve been waiting for a new tutorial from you for a long time. Thanks Andrej 😊

@JohnWalz97 4 ай бұрын

Andrej, you are an amazing human for sharing this knowledge to the world for free!

@willscui 3 ай бұрын

The final blooper reels are great. Thanks for the hard work to make a dry topic fun.

@paweg344 3 ай бұрын

Great video Andrej, I really appreciate your sharing of knowledge and the clear way in which you do it.

@user-kl7gd9sz6q 4 ай бұрын

I loved the bloopers at the end. Thanks Andrej, you're the best.

@santhanamss 4 ай бұрын

I love that you are going back to examine these fundamentals deeply and also teaching the world about them. I think there is too much blind marching to create solutions on top of LLMs. Stepping back is how real innovation is going to happen. Lots of respect for your style of explaining things and the intentions behind this.

@user-uq5qw1fk3d 4 ай бұрын

Thanks so much for all of your videos Andrej!

@sumankumargunnala3691 4 ай бұрын

You are an inspiration to many...sharing knowledge with clear intentions of educating others is amazing...

@drhxa 4 ай бұрын

We love you Andrej, thank you so much!

@readcoderepeat 4 ай бұрын

How are you so good at teaching... Every single video I'm in awe at how clear the explanation is!

@eugene_volen 4 ай бұрын

Thank you very much for continuing to post these videos!

@asdrofllmao 4 ай бұрын

This is fantastic, thank you for making this really dense stuff accessible to the layperson, please keep them coming!

@isaccpdev 4 ай бұрын

Andrej… you’re the best, thanks so much for your lectures!

@qianma853 3 ай бұрын

I can't express my appreciate. I was inspired by your past podcast. I can pick myself up to keep learning. You saved me. Thank you! :)

@snehotoshbanerjee1938 Ай бұрын

Thanks Andrej for bringing this level of knowledge to us. We are grateful!!

@user-xk6ot7ft2j 4 ай бұрын

I’m so happy and thankful for these videos :) Thank you Andrej!

@panth5501 4 ай бұрын

Thanks for sharing your awesome knowledge!! Glad to have you back : )

@prabalmodi3454 8 күн бұрын

Thank you for this video, this has broadened my understanding of tokenization and large language models.

@project-hq 4 ай бұрын

Another amazing video, thanks for sharing all your time and knowledge with us!

@user-sd4to4os8r 4 ай бұрын

Special thanks for end credits. Thanks for another amazing video.

@tonyzhao5624 2 ай бұрын

Thank you so much for all the efforts u put in videos! It's truly remarkable to unveil this.

@IstvanF 4 ай бұрын

Thank you very much Andrej and welcome back!

@avishakeadhikary8154 4 ай бұрын

The fact that you're still doing this motivates me so much to do something like this myself. Proud to be in the same generation as yours.

@slaznum1 4 ай бұрын

Thank you so much for this. This is a beautiful example of problem decomposition and various solution alternatives!

@kodiko 2 ай бұрын

Andrej sir, your videos are really really good! You explain in such a way that you make the stuff look easy. Sometimes certain concepts are still a bit complex and hard to handle in one go but no one walks out saddened after watching your videos, Your positivity is infectious!