Stanford CS25: V3 I Retrieval Augmented Language Models

Рет қаралды 130,129

3 ай бұрын

December 5, 2023
Douwe Kiela, Contextual AI
Language models have led to amazing progress, but they also have important shortcomings. One solution for many of these shortcomings is retrieval augmentation. I will introduce the topic, survey recent literature on retrieval augmented language models and finish with some of the main open questions.
More about the course can be found here: web.stanford.edu/class/cs25/
View the entire CS25 Transformers United playlist: • Stanford CS25 - Transf...

Пікірлер: 56

Күн бұрын

Good lecture, offers a good summary of the literature on RAG.

@erniea5843 3 ай бұрын

I love that this content is freely accessible to everyone. Lots of helpful information being shared here

@nintishia 3 ай бұрын

The amount of research work on retrieval augmented generation for large language models has exploded in recent times. Thanks to the speaker for directing attention to the most significant bits.

@Arvolve 3 ай бұрын

So many great ideas here! Fantastic resource, thank you.

@capucinnolover 2 ай бұрын

Nicely explained in just right technical details. Thank you!

@99BLACKLP 3 ай бұрын

Great insights and video!

@loopaal 3 ай бұрын

this is what I needed, thank you sooooo much!!!!

@velociraptor75013 Ай бұрын

Excellent content, thanks for the references!

@dongxu9013 3 ай бұрын

The best talk about RAG so far

@morespinach9832 3 ай бұрын

How many have you heard, right here on KZfaq? Lots of actual hands on info out there and in 1/3rd the time. This one had loads of intro forever before getting into specifics.

@edwardmitchell6842 2 ай бұрын

@@morespinach9832 Can you recommend some sources? I'm compiling a list.

@yangzju Ай бұрын

@@morespinach9832 can you recommend some videos?

@comunedipadova1790 21 күн бұрын

@@morespinach9832 What lecture do you suggest for a more practical (code) view?

@morespinach9832 20 күн бұрын

@@comunedipadova1790 plenty of them - search for these keywords.

@ronitakhariya4094 3 ай бұрын

just the thing i wanted thank you so much.

@reslleygabriel 3 ай бұрын

Awesome content, thanks for sharing!

@dongmo6546 3 ай бұрын

Thank you

@NerdyXRPanda 3 ай бұрын

Thank you.

@deeplearningpartnership 3 ай бұрын

Really cool.

@muhannadobeidat 2 ай бұрын

It is good to see the whole spectrum of options, but what would be a practical way to get started on this? His old colleagues in HF actually have an excellent book that goes through many different ideas including RAG in chapter 7 where they do an excellent job explaining context and giving you options for implementation.

@johntanchongmin 3 ай бұрын

Pros: Gives a brief overview of many RAG methods Cons: No intuition given which is the key reason for why it works for the different methods Would have preferred more insights rather than just describing the papers, but overall thanks for the video!

@duongkstn 3 ай бұрын

Hi, about Atlas. You said that we can update the Retriever. At 42:00 is some retriever loss, but what about pair label (question, positive paragraph, negative paragraphs) like normal retrieval model - do they contribute to the retriever loss ? I read codebase of Atlas, and do not find that kind of loss

@kassy11jp 3 ай бұрын

Are Socratic models also a type of Multimodal RAG?

@user-ij2rm6yl2t 2 ай бұрын

3:34 What chatgpt was really about - fix the user interface to LM 12:00 Frozen RAG

@ritikdua7225 2 ай бұрын

does anyone have list of research papers metioned in this video?

@trinityblood5622 3 ай бұрын

Could you please share the slides?

@ginogarcia8730 3 ай бұрын

WE ARE IN THE FUTUREEEEEE

@amansinghal5908 3 ай бұрын

Are there assignments for this?

@samferrer 3 ай бұрын

Actually we can see a first sign of language models in Shannon's 1948 paper, A Mathematical Theory of Communication.

@ssssssstssssssss 3 ай бұрын

Language models started to appear in the 1800s.

@therainman7777 3 ай бұрын

Language models actually appeared sometime during the Cretaceous period, though scientists aren’t quite sure of the exact year. They think the stegasaurus might have had something to do with it; all these people who think OpenAI invented them are so wrong.

@AIPoker-tj6lr 3 ай бұрын

What about scann by google ??

@lambertch Ай бұрын

Shouldn’t it be Shannon, 1949?

@deconcoder 3 ай бұрын

THe bit about where this came from was funny, but your search just gave a less silly answer. Go reread Shannon's 1948 paper that invented Information Theory. Yes he did not talk about using neural nets (which did not exist) but he did talk about probability.

@Arthurlvaz 2 ай бұрын

It's not entirely clear what "frozen rag" and "retrieve rag" refer to.

@jstello 3 ай бұрын

I love how everyone tries to hide the fact that OpenAI is 100% the reason everyone is watching this video

@pw7225 3 ай бұрын

No one is hiding anything

@a3mia3mi82 3 ай бұрын

“Ignorance” is not equal to “facts”

@therainman7777 3 ай бұрын

I know, isn’t it hilarious? The bitterness and jealousy is so transparent. It reminds me of the scene from The Social Network where they’re trying to claim credit for his creation and he says “You know it’s really not that complicated. If you guys were the inventors of Facebook, you’d have invented Facebook.” Yann LeCun likes to drone on on Twitter about how everything OpenAI is doing is “old” technology. Well if it’s so old, how come hundreds of different companies filled with smart people were trying for years to make a chatbot that was worth using but until OpenAI no one seemed to manage it? Yes, OpenAI didn’t invent the Transformer. We know. Who cares? They clearly solved dozens of incredibly difficult engineering problems that no one else had been able to solve, and gave the world a language-based AI that was actually _useful._ As evidenced by the fact that it was the fastest growing app in human history, by an extremely wide margin. And as soon as they do it all these idiots turn up basically saying “I could have done that 10 years ago, I just chose not to.” Yeah, ok. It’s so pathetic.

@Gingnose 3 ай бұрын

Maybe because only you are the one that is obsessed by the idea of starting learning something new but merely due to its "novelty and coolness" is something to be ashamed of and you project that idea on others which make you think others are watching this video with the same reason but hinding that information (and trying to look smart) because it is emabrassing to declare and you think that way too. However I will remind you that trying to learn new things whether it is because it is fashion or cool is nothing to be ashamed of and a great excuse to start new things.

@srh80 3 ай бұрын

Whats an Open Al?

@mahkhi7154 3 ай бұрын

Us Mandalorians CYBERMEN: Understand The Material - Given To You By Minions, better Than The Minions. Your Gods Gods. Ask Them.

@therainman7777 3 ай бұрын

Could you try writing that again but having it not be nonsense this time?

@starmountpictures 3 ай бұрын

meh

@mahkhi7154 3 ай бұрын

You're a P1G. Memorising and Repeating without Understanding. What is a Sentence? A Collection of Concepts. The Brain can Learn 10's of Thousands of concepts. It Has Hardware To Help it. Concept: "Doing" brings up pictures of Making a Cake or Assembling a Chair. Concept: "What" brings up Pictures of "Asking someone a Question" or Deciding Colour Show to Buy. A Computer Doesn't Work Like That. All it Understands is Numbers and Maths. All These "Concepts" have To Be Encoded in Numbers and Maths and Rules Applied To Them. The Computer Doesn't Know What The Character "B" is. All it Know That if it Finds The Number "66" in Memory, it Draws Something That Looks like "B". Spelling. It Doesn't Know What Spelling is. All it Knows is "66,65,84" Valid for word BAT. BAB 66,65,66 is an Invalid String of Numbers and Not in the Dictionary. 10,000 Concepts Need - Modelling and Rules Creation (100's of Thousands) - For the Computer To Understand. That isn't an Easy Thing To Do: Trillions of Lines of Java Code.

@therainman7777 3 ай бұрын

Maybe if you were a computer maybe you would understand that you Don’t Need To Capitalize Every Word when writing English. And trillions of lines of code? You know nothing about coding; in all of human history everyone out together has not written a trillion lines of code, let alone in a single computer or a single program. P.S. Your comment was utter nonsense, it contributed nothing, and it meant nothing. Have a nice day.