Nice video. Thanks for this. I could be wrong but one potential error I see: In this video, you said that "You can’t do KV cache because you change the embeddings with every token you add." I don't think this is necessarily true, at least not for decoder architectures like GPTs. The previous tokens don't attend to the new tokens -- they only attend to tokens to their left (there's a causal mask). When you add a new token, the relative position between the previous tokens don't change. For example, if you add a 6th token to a sequence, the distance between token 1 and token 4 haven't changed at all; therefore, the KV cache is still valid. It seems to me that yes, relative position embedding is inefficient, but not because it invalidates KV cache; rather, it's because every time we add a new token, it needs to attend to all previous tokens twice: once for the regular attention calculation, once for the relative positional embedding

I am viet-teochew so our accents and words are a bit different, it feels weird to hear someone else speak teochew with different accent as ive only ever heard my family speak it haha An example i think is: the word “to like” is 哈 hah

I am teochew nang! But my family speaks a Vietnamese Teochew so our accent is a bit different and some words are different

Thank you very much. Great video! Clear, concise and yet covers most of the necessary details.

Amazing. Some of my colleagues work on KV cache, and this video was a great introduction to the topic. Thank you!

Superb

Can you make video on grouped query attention (GQA) and sliding window optimisation?

that was really nicely done. as a non-expert, I feel like I can now have a great general idea of what a quantized model is. thank you

Finally! My 4th video and I was lost but this one did the trick!

dude you are the mf goat

Amazing video! I was already really blown away when using whisper for the first time in Cantonese. I was surprised that it was able to work even on a "dialect" of Chinese. To see it being able to be further finetuned to other dialects really shows how well the model scales with new data. Also, great work on building out the pipeline for the Teochew dataset. Never would have thought to use those dialect tv shows as the base. Brings me back to the times when I'd hangout at my grandpa's place and he'd be watching Hokkien dramas. It'd be interesting to see if training a seperate class for Hokkien would improve both the Hokkien and Teochew performance for the model since they're supposedly similar. Not a speaker of either, but at least what I know from my mom who's a native hokkien speaker, it would seem that both dialects would be rather mutually intelligeble. Perhaps the model might be able to pick up on both and hence have "more data" to work with.

Nothing really new about this, it seems that big tech companies really do have it easier when publishing research

I haven’t read the paper yet but my understanding is that we sample from q(x) - p(x) because we want the most surprising token that the draft model does not anticipate. It should maximize the entropy but then it should have log on the equation, anyway, I gotta read the paper to understand the math.

Very clear explanation, thank you!

Thanks a ton for this crisp and precise explanation of why we use caching in transformers.

great video! very helpful, i did the same for moroccan dialect arabic, i have a quetion how did you make the tool to get the histogram and the search box please ?

great explanation,really helped me!

I picture this like mapping out a vector space in lower resolution by using a tree structure.

Excellent video, great high level overview!

Great video. No I understand the importance of 'Time to 1st token'. I like the short ones that are to the point on a topic. Learning in smaller chunks works well for me.Thanks!

Another great video I have a question: is RVQ solely for compression or could one conceivably do some processing of an RVQ to operate on it as a representation of the data rather than on the uncompressed data? Eg teach a model to classify sounds based just on the RVQ.

Hey There, I am trying to reach out to you via email. Could you please check? Anyway, here is my question: why does encodec's encoder output 75 frames of 128 dimension per second? I mean, don´t convolutions always just reduce dimensionality, why do they increase? I would expect a single array with less elements in the time dimension. Could you please help. Thank you

I thought about using voronoi cells nearest neighbour lookup for compressing latent spaces myself, but I also thought that some processes that generate the lantent space centroids of interest can also benefit from weighted voronoi tessellation / power diagrams, where maybe depending on density of points or other features we can weight that particular cell to make it more relevant.

Nicely done!

If you turn ur voice tool into an extension that can work on any web page on chrome, I would be interested. The way it is now can be helpful but I have better alternatives, like I can just use chatGPTs speech to text feature which is very good.

Isn't batch decoding a bit impractical to be assumed in estimating the KV cache footprint of OPT 30B? I'd say for bsz = 1 (online decoding) this is still not that significant ~= 1.4 GB.

*What is RVQ?* * RVQ is a technique to compress vectors (like audio embeddings) into a few integers for efficient storage and transmission. * It achieves higher fidelity than basic quantization methods, especially at low bitrates. *How RVQ Works:* 1. *Codebook Quantization:* A set of representative vectors called "codebook vectors" are learned. Each vector is mapped to the closest codebook vector and represented by its index. 2. *Residual Calculation:* The difference between the original vector and the chosen codebook vector is calculated (the "residual vector"). 3. *Iterative Quantization:* The residual vector is further quantized using a new codebook, and a new residual is calculated. This process repeats for multiple iterations. 4. *Representation:* The original vector is represented by a list of indices, each corresponding to a chosen codebook vector in different iterations. *RVQ in EnCodec (An Audio Compression Model):* * EnCodec uses RVQ to compress audio embeddings, achieving good quality even at low bitrates (around 6kbps). * The number of RVQ iterations controls the bitrate and quality trade-off. *Learning Codebook Vectors:* * Initially, K-means clustering can be used to find optimal codebook vectors. * For better performance, codebook vectors are fine-tuned during model training: * *Codebook Update:* Codebook vectors are slightly moved towards the encoded vectors they represent. * *Commitment Loss:* The encoder is penalized for producing vectors far from any codebook vector, encouraging it to produce easily quantizable representations. * *Random Restarts:* Unused codebook vectors are relocated to areas where the encoder frequently produces vectors. *Key Benefits & Applications:* * RVQ enables efficient audio compression with smaller file sizes than traditional formats like MP3. * It has potential applications in music streaming, voice assistants, and other audio-related technologies. i used gemini 1.5 pro to summarize the transcript

:smiley: 😄

how can i connect with you

great video. thanks

your explanation is amazing. thank you for your work

Great summary, thank you.

And if one was to quantize a distilled model? Is the outcome any good?

excellent video sir

Is that your wife voice in the end of the demo

nice video and great voice writer, I have tried implementing it with transformers js package and its whisper model but no luck yet since processing is heavy

♥

Thanks for sharing, I am wondering how target model check the generated tokens of draft model and produce probability distribution q of x for each token?

Hi. Thanks for your video If my encoder produce series of tags for each word in input sentence and I want to use that tags for generating text that is correct based on input and generated tags of encoder, how can i use decoder for this?

Teo Chew also has subdialects like many other dialect like Hakka also has many sub dialects.

awesome project and well-explained talk! I am also a cs student from Teochew. Have learned a lot from the video. 👍

I was talking to ChatGPT as to why it only produces 1 token at a time, and it suggested I should checkout "Non-Autoregressive models"

Thank you for explaining it!

nicely done

Awesome.

Absolutely amazing explanation! Keep it up man

made very simple, but one more variable is choosing right draft model. Suppose if one chooses that is too too away from larger one's distribution then its also a problem.

For this to work, the two models need to have identical tokenizations right? Is there any way around it?

nice video

Love your video, thanks! If I had to give one request/critique, it'd be that I wish there were some slides in here similar to Samuel Albanie's videos that are quite information-dense recaps that could be lifted out of the presentations and put into our notes (or into a powerpoint for a paper club, or something).