how the tokenizer for gpt-4 (tiktoken) works and why it can't reverse strings
Рет қаралды 1,905
5 ай бұрынchris breaks down the chatgpt (gpt-4) tokenizer and shows why large language models such as gpt, llama-2 and mistral struggle to reverse words. chris looks at how words, programming languages, different languages and even how morse code is tokenized, and shows how tokenizers tend to be biased towards english languages and programming languages,
@ernestuz 2 ай бұрын
The funny thing is the most complete the vocabulary the less pressure in the upper layers, so it's not only cheaper because of fewer tokens, but in processing, I wonder if somebody has prepared a semi handcrafted tokenizer, where, let's say the first 30K tokens come from a dictionary and the rest is generated.
@chrishayuk Ай бұрын
exactly. tbh, i wouldn't' be surprised if someone goes that direction
@feniyuli 3 ай бұрын
It is very helpful to understand how the tokenization works. Thanks! Do you think data that we encode using tiktoken will be sent to the AI?
@chrishayuk Ай бұрын
definitely not, it's all local
@ilyanemihin6029 4 ай бұрын
Thanks, very interesting information
@chrishayuk 4 ай бұрын
glad it was useful
What makes LLM tokenizers different from each other? GPT4 vs. FlanT5 Vs. Starcoder Vs. BERT and more
Jay Alammar
Рет қаралды 15 М.
Chris Hay
Рет қаралды 1,2 М.
Sameer Gaming
Рет қаралды 5 МЛН
MaviGadget
Рет қаралды 9 МЛН
KOMPUKTER
Рет қаралды 181 М.