SPLADE: the first search model to beat BM25

Рет қаралды 18,487

Күн бұрын

AI-powered search is heating up, but it's nothing new. Google, Netflix, Amazon, and many more big tech companies have all powered their search and recommendation systems with “vector search”.
In this video, we'll talk about sparse and dense vector search, their pros and cons, and how the latest sparse embedding model called SPLADE can help us eliminate many of the downsides of traditional sparse embedding methods like TF-IDF and BM25.
SPLADE can be used as an alternative to dense embedding models like OpenAI's text-embedding-ada-002, Cohere's embed endpoint, or sentence transformers. But more interestingly, it can be used alongside these dense embedding models to give us the best of both worlds.
🌲 Pinecone article:
pinecone.io/learn/splade
🎙️ AI Dev Studio:
aurelio.ai/
👾 Discord:
/ discord
🎉 Subscribe for Article and Video Updates!
/ subscribe
/ membership
00:00 Sparse and dense vector search
00:44 Comparing sparse vs. dense vectors
03:59 Using sparse and dense together
06:46 What is SPLADE?
09:06 Vocabulary mismatch problem
09:51 How SPLADE works (transformers 101)
14:28 Masked language modeling (MLM)
15:57 How SPLADE builds embeddings with MLM
17:35 Where SPLADE doesn't work so well
20:14 Implementing SPLADE in Python
20:38 SPLADE with PyTorch and Hugging Face
24:08 Using the Naver SPLADE library
27:11 What's next for vector search?

Пікірлер: 57

@jamesbriggs Жыл бұрын

To install the naver labs splade library you need `pip install git+github.com/naver/splade.git`

@JulianHarris Жыл бұрын

Came here curious about SPLADE, discovered a super understandable introduction to transformers and attention networks. Thank you!

@jamesbriggs Жыл бұрын

I really wanted to get the point across about SPLADE but there was a lot of foundational stuff to cover from sparse vs. dense, transformers, etc - so I'm glad the extra info helped :)

@zazouille2264 Жыл бұрын

Agreed. Great video. Nicely layered. Thank you OP

@magicofjafo Жыл бұрын

I agree!

@kevon217 Жыл бұрын

Great tutorial as always. Your slide animations are next level!

@ArnavJaitly Жыл бұрын

James, this is awesome and very relevant to my current project! Thank you for your efforts in putting this together and sharing it, much appreciated!

@jamesbriggs Жыл бұрын

awesome, good timing!

@abhayr Жыл бұрын

Amazing explanation. Thx for sharing

@IamalwaysOK Жыл бұрын

Hey James, as usual, thanks a ton for your awesome videos! I've got a quick question for you. Have you ever thought about using a knowledge graph alongside SPLADE to expand terms? And is there any way we can embed that knowledge into sparse vectors using transformers? Curious to hear your thoughts on this!

@salesgurupro Жыл бұрын

Amazing. Thanks for such a great explanation 😊

@jamesbriggs Жыл бұрын

you're welcome!

@shamaldesilva9533 Жыл бұрын

dude you are a gold mine when it comes to these topics 😍😍 .

@jamesbriggs Жыл бұрын

thanks man it's appreciated!

@Sky-ec9eu Жыл бұрын

This is incredible. Thanks James!

@jamesbriggs Жыл бұрын

you're welcome!

@avidlearner8117 Жыл бұрын

Fantastic content! Especially since I'm building an app and need to find a proper solution for data retrieval....

@ylazerson Жыл бұрын

very fascinating - thanks!

@jamesbriggs Жыл бұрын

glad you enjoyed it!

@nhatpham4053 Жыл бұрын

awesome works

@lutune Жыл бұрын

Have you built any of these apps? Your content is so great, as you get into more media, some development of those apps could really help with putting this into a visual space

@jamesbriggs Жыл бұрын

started building some demos and testing splade a couple months ago, will be sharing more soon - it's really cool though and I intend on making it a big part of my "go to toolkit" in the future

@lutune Жыл бұрын

@@jamesbriggs Your DC seems to be getting a lot of new people! ill get some things updated on there today for ya

@thedude9270 Жыл бұрын

Thanks for the tutorial! Is it possible that you could also share a colab or video explaining what would then be upserted as a Pinecone vector?

@aurkom 8 ай бұрын

Really enjoyed this one.

@danrosher6658 Жыл бұрын

Great talk, thanks James ... Would an alternative to the cosine sim to compare query/doc is to index the tokens and weights for docs (from SPLADE model outputs) , also convert a query to tokens(and weights) , then return docs having the query tokens where the doc weight > query token weight for each token? .. would this work ?

@user-vc8yk8ox3k Жыл бұрын

Hello James, the above pinned method for pip install splade is not working and giving error like "error: subprocess-exited-with-error" so, Can you please let what is the issue or what alternate we can use if not this.

@snack711 Жыл бұрын

i am surprised how "orangutans" got split into tokens. i thought "orangutan" surely had to be a token itself.

@alivecoding4995 Жыл бұрын

Which graphics library do you use for these Transformer illustrations? Are these pre-built assests?

@user-ih1dx6wn9c Жыл бұрын

Thank you! when using embeddings and asking the model gpt-3.5 some question like "write me some code that use this and that" does the model automaticlly search in the embedding too in order to give the answer?

@jamesbriggs Жыл бұрын

gpt 3 doesn't, you need to add a knowledge base to do this, like I do here kzfaq.info/get/bejne/qNhxdsuhx93dl3k.html

@johannamenges3095 Жыл бұрын

But is Faiss still a solid solution for a semantic search engine? Cause I am at the moment working on a search engine with Faiss algorithm

@gorgolyt 7 ай бұрын

Great video. But you should link to the SPLADE paper(s). Are you just talking about the original paper here?

@leventk.1611 9 ай бұрын

13:02: low proximity = high semantic similarity. Not high proximity. :D

@SinanAkkoyun Жыл бұрын

How does this compare to the new OpenAI embeddings?

@EkShunya Жыл бұрын

what tool do you use to make the diagrams ?

@jamesbriggs Жыл бұрын

excalidraw!

@abhinavkulkarni6390 Жыл бұрын

Hey James, Can you please compare SPLADE with ColBERTv2 - both of which are designed to alleviate the problems of desnse passage retrievers?

@jamesbriggs Жыл бұрын

I haven't read into the colbert models, I understood them to not be hugely scalable? I can look into it if they're of interest

@biaoliu9297 Жыл бұрын

Is there a multi-language version model?

@jeffsteyn7174 Жыл бұрын

That's interesting. What does pinecone use, sparse or dense?

@jamesbriggs Жыл бұрын

now it can use both, I'll talk about it in the coming days or you can refer to here github.com/pinecone-io/examples/blob/master/search/hybrid-search/medical-qa/pubmed-splade.ipynb - for an example

@sndrstpnv8419 7 ай бұрын

code deleted pubmed-splade.ipynb @@jamesbriggs

@RatafakRatafak 6 ай бұрын

Is it important? If you use cosine similarity for both dense and sparse embeddings, it should work in any case.

@AnonymousIguana 7 ай бұрын

So, SPLADE vector generation is just as computationally intensive as dense vector generation? My understanding is that SPLADE requires real-time inference from a sophisticated model like BERT at query time. Isn't that very problematic?

@RatafakRatafak 6 ай бұрын

Looks like so. Sentence-BERT is equally computationally intensive thant this SPLADE.