Both LangChain and Llama Index have added Semantic Chunking (level 4) to their libraries LangChain: python.langchain.com/docs/modules/data_connection/document_transformers/semantic-chunker Llama Index: llamahub.ai/l/llama-packs/llama-index-packs-node-parser-semantic-chunking?from=all

Single and multi hop explained concerning the semantic splitting. Nice !!

First video I came across that actually explain langchain in detail so that a layman can understand how it actually works

With the continuous influx of short form content, props to you for making this so interesting to watch. Didn't even realise it was an hour long. Loved every second of it. Thanks!

Why did KZfaq take so long to recommend me this channel? Incredible work!

This is an insanely detailed from first principles tutorial. Thank you for taking the time to put this together.

Hi Greg, many thanks for the work you put into this and to help all of us learn. Great clarity, depth and tempo! 💪

I thought the explanation and showing your experimentation for semantic splitting was creative. Thank you very much.

what the ___. how good can a tutorial be. such a gem of a video. thx for making this. new to ml and found this very helpful

Your channel is a gem, thank you!

Wow! I hadn't even thought about Agentic Chunking! I need to try this. I did some extensive experimentation with chunking on a project at work for a clinical knowledge base and I found that chunking strategies can make the difference between an ok retrieval and an awesome retrieval that works across a higher percentage of queries.

Thanks for this Greg. I've been looking at agentic chunking for a while and this video really helped me with implementation. Not heard of you before I searched but now subbed. Thanks a lot :)

Nice vid, Greg! You're on the cutting edge with some of these splitting techniques. Well done. 😎

Not a comment on the context, but I think that the style of the thumbnail is very smart. It reminds me of the Wired 5 Levels of Difficulty style. I think if the aesthetic is softer, it can be even more popular.

Hi Greg, thanks for the video. It's awesome to have someone publishing good content who's doing the exact same thing as me. Hope to see more videos on advanced topics like this!

you really deserve that like buttons really thanks for this out of the world content

man it took me 3 weeks to find you. thank you please keep on coming.

Amazing!! I am fascinated by how document specific splitting or the bonus level also ties with how we structure our data schema. E.g. extracting metadata like "Introduction" in level 3 or applying a summary to the podcast and indexing that to then link to the raw clip in the bonus level. All amazing, super useful stuff -- I am a bit skeptical on embedding based splitting though, maybe just need to dive in further! Mostly bullish on level 5: agentic splitting with multimodal llms that kind of blend levels 3 and 5

Thanks I was thinking about solving my own Retrieval problem. I already got the small crude proof of concept using just simple chunking, embedding, RAG, etc. Now I need to get bigger user inputs that are in bigger pdf files. I thought using agents for it to get around the context window, you agentic chunker is a good starter and does make intuitive sense. I will try this route.

Thanks greg! Love the long form instructional video :D Greatly appreciated

Liked this semantic splitting! Cool stuff you´ve done there!! Also agentic chunking. Pretty cool!!!

While running the code elements=partition_pdf(filename=filename,strategy="hi_res",infer_table_structure=True,model_name="yolox") in my Jupyter notebook, I encountered errors such as TesseractNotFoundError. If anyone has faced this issue or knows how to solve it, please guide me as soon as possible. Great work, sir. I can confidently say that such a combination of content and explanation is unparalleled in the KZfaq world. @DataIndependent

Thank you very much! Great video!

Best chunking video to date.

thank you so much! well done!

Another banger hit from Greg! How does he do it. Love this video!

This is an amazing professional content! it hits the point directly

Extremely helpful, thanks for the great tutorial!

amazing video ! very helpful ! thanks !

Love your videos, especially this one. The information density and presentation is off the charts. It is so altruistic of you to put this out there for free. I am especially interested in the semantic chunking. One use case is transcripts which often have distinct conversation blocks or qhestion answer pairs. Since it is important to capture the question and answer for full context, i was wonderinf what methodology might work best. Alternatively, semantically chunking a document vs pre-defined themes - sort of the opposite direction as the agentic chunker. First generate or define the overarching themes or buckets, then assign chunks to them. It seems that there is some real possibility in the semantic chunking methods. 🎉 Looking forward to experimenting more. Thank you again.

That was really helpful, thank you for information you're sharing!

Another great video - thank you:) In my case I need to try Semantic Splitting and Document Specific Splitting.

great video, thanks you

This video should have a milliion views already. Amazing work

great lectures, great teacher

This video is a piece of art ❤

wow it has been very long time since I made a comment. This content is outstanding! Thank you for creating such a great video.

That was great! Semantic and Agential ideas are definitely a way forward. Branching off that, here's a thought: building a meta-transformer that uses a classic-transformer through multi head attention to associate high dim vect between semantic chunks > more efficient parallel processing and capturing more nuanced relations between chunks & macro managing the splitting iteratively GPT formatting: Proposed Meta Transformer Approach: Chunk-Level Semantic Analysis: The meta transformer, as you propose, would operate on semantically split chunks, not just individual tokens. High-Dimensional Semantic Space: Each chunk (sequence of tokens) is mapped onto a high-dimensional semantic space. Iterative Mapping for Optimal Chunking: Through multi-head attention, the model would iteratively determine the best separation points for these chunks.

Incredible! I love the approach to Semantic Splitting. I'm working on creating AI tools that will analyze customer interviews (i.e. founders or user researchers talking to customers and then using AI for the analysis/synthesis). In those transcripts, there are multiple speakers. I'm incorporating your approach here and trying to find a better way to chunk those transcripts by the topic of conversation. Thanks a ton for sharing your work!

Great stuff!

Great video, starting out with naive and easy to understand methods of text chunking, ending up with novel ideas that may point to the future

This was next level

Thank you.

Fantastic Video. Been thinking about level 5, a brilliant way to approach chunking, and I see other applications. Level 4 is clever. Retrieving in syntheticI believe will be the standard as time moves on.

Great content. FYI - Google’s Gemini models are built to be Multi-Modal from the outset, so seems to overcome some of the challenges you mentioned when combing text and images.

Love the upload

I really enjoyed this thanks. I’ve had good IRL business results with your tiers 2 and 3. I’ve used semantic search quite a bit and my jury is still out on the match score’s reliability to granular levels like decision-making breakpoints. So I would probably find tier 4 still more of an aspirational novelty. I like the concepts of 4 & 5 though on the more distant horizon. As an aside - the term “naive” a lot of folks are using lately in the Langchain llamaindex crowd makes me roll my eyes. It just smells like smug Silicon Valley 20-something (not specifically throwing shade at you I’m seeing it all over the place). If someone is chunking a set of documentation and the content is divided into topics by markdown tags they’d call your “tier 3” implementation naive even if it’s clearly the most practical way to chunk the data and achieve an outcome. I would love to see a term arise to discuss the simple-but-often-practical methods with less negative baggage.

Congrats that's ! That's an excellent job ! I hope you will continue your work and more benchmark will come. I am particularly curious if the benefit of semantic & agent chunking are minored or majored when code, html , csv is chunked.

29:13 Now there is GPT4-O ... So you were right with the prediction 😂

Fantastic!

Buy you a cup of coffee ? How about a Starbucks franchise ! This is some very powerful material.. Looking forward to implementing these into my pipelines ! THANK YOU !

Thanks!

I also never thought of purpose built chunking or semantic proxies like that. Adding question hypotheticals as the embedding could be extrapolated to other use cases.

Love it

This might just work for my meeting transcripts. Ts similar to something David Shaperio did. Where knowledge bases articles are written. And then reserved and updated during a conversation. I like the idea of using propositions and doing this at the article level.

Great video. Some concepts in it overlap with the RAPTOR paper for RAG

I feel like my mind was blown, brought together then blown again by 'level 4 - semantic search' part of the video

Imagine langsmith for chunking. Something like the agentic flow you have but all chunk titles, summaries ECT... Are kept and able to be used for future tuning and other iterations.

That agentic chunking really does sound like an interesting approach . How can we predefine the topics instead of them being automatically generated?

What a great video. It would have taken me forever if if I was to research and learn more about this on my own. What a life safer. Do you have a video or a good resource about optimizing other RAG hyperparams and about reranking of chunks?

Great video! it helped me clarify the past and present of all the chunks. I have a question, in agent chunking, there can be an issue of having too much content on a single topic. In extreme cases, an entire book might be about one topic. How should we further break it down in such situations?

If I could like this video twice, I would have

Nice video, next level of chunking! Are you planning to have soon max_chunk_size?

Human beings are always continuously learning. LLM’s should have all the abilities that we have.

Theory & Importance of Text Splitting: Context Limits: Language models have limitations on the amount of data they can process at once. Splitting helps by breaking down large texts into manageable chunks. Signal-to-Noise Ratio: Providing focused information relevant to the task improves the model's accuracy and efficiency. Splitting eliminates unnecessary data, enhancing the signal-to-noise ratio. Retrieval Optimization: Splitting prepares data for effective retrieval, ensuring the model can easily access the necessary information for its task. Five Levels of Text Splitting: Level 1: Character Splitting: Concept: Dividing text based on a fixed number of characters. Pros: Simplicity and ease of implementation. Cons: Rigidity and disregard for text structure. Tools: LangChain's CharacterTextSplitter. Level 2: Recursive Character Text Splitting: Concept: Recursively splitting text using a hierarchy of separators like double new lines, new lines, spaces, and characters. Pros: Leverages text structure (paragraphs) for more meaningful splits. Cons: May still split sentences if chunk size is too small. Tools: LangChain's RecursiveCharacterTextSplitter. Level 3: Document Specific Splitting: Concept: Tailoring splitting strategies to specific document types like markdown, Python code, JavaScript code, and PDFs. Pros: Utilizes document structure (headers, functions, classes) for better grouping of similar information. Cons: Requires specific splitters for different document types. Tools: LangChain's various document-specific splitters, Unstructured library for PDFs and images. Level 4: Semantic Splitting: Concept: Grouping text chunks based on their meaning and context using embedding comparisons. Pros: Creates semantically coherent chunks, overcoming limitations of physical structure-based methods. Cons: Requires more processing power and is computationally expensive. Methods: Hierarchical clustering with positional reward, finding breakpoints between sequential sentences. Level 5: Agentic Chunking: Concept: Employing an agent-like system that iteratively decides whether new information belongs to an existing chunk or should initiate a new one. Pros: Emulates human-like chunking with dynamic decision-making. Cons: Highly experimental, slow, and computationally expensive. Tools: LangChain Hub prompts for proposition extraction, custom agentic chunker script. Bonus Level: Alternative Representations: Concept: Exploring ways to represent text beyond raw form for improved retrieval. Methods: Multi-vector indexing (using summaries or hypothetical questions), parent document retrieval, graph structure extraction. Key Takeaways: The ideal splitting strategy depends on your specific task, data type, and desired outcome. Consider the trade-off between simplicity, accuracy, and computational cost when choosing a splitting method. Experiment with different techniques and evaluate their effectiveness for your application. Be mindful of future advancements in language models and chunking technologies. Further Exploration: Full Stack Retrieval website: Explore tutorials, code examples, and resources for retrieval and chunking techniques. LangChain library: Discover various text splitters, document loaders, and retrieval tools. Unstructured library: Explore options for extracting information from PDFs and images. LlamaIndex library: Investigate alternative chunking and retrieval methods. Research papers and articles on text splitting and retrieval.

Gold.

Thanks, Excellent video about chunking strategies👍 Question: Can i store the pulled html table using unstructured in a vector database together with a normal text and asking question (RAG)?.

Could attention be used here instead of the embeddings? Input every 2 sentences with overlap into an encoder. Above a certain threshold of "attention" from one sentence to another, have both in the same chunk

Love this! I'm working with transcripts where semantically generated chunks can be quite large. These chunks need to be further divided to fit the limits of the embedding model. Given this, isn't semantic chunking unnecessary if we ultimately have to recursively break down the larger chunks into smaller ones?

Rip to semantic search for large datasets 😂! But interesting approach.

Agent chunking is a paradox. We aim to spilt the document into concise units to eliminate the noise so that the LLM can generate better answers. But we are asking the LLM to figure out the concise units by dumping all the propositions.

Great tutorials . Are there any courses or book written by you. Your explanation is excellent . Thank you. Can you please share the code which was shown in the demo.

Hi Greg, Nice video! As for 4, did you considered to use fine tuned NLI models? i.e. combine 2 sentences if the model predicts entailment/natural relationship?

Hi... Any suggestion like how we can handle large chunks s some of the chunks are having token length greater then 4k !!

@DataIndependent Hi Greg. Which type of splitting would you recommend when working with bank statement, invoices, balance sheets etc.

Great video. Thanks for sharing. The level 5 implementation doesn't rewrite the proposition (e.g. It would still say "He likes walking" not "Greg likes walking"), or am I missing something!? I guess that would be another level of improvement? Any ideas how to implement that rewrite?

So, as you are using Langchain and Llama-Index - what do you prefer for which task? What are the pros and cons of each? I´ve also used both and have manifested an opinion.

Hey Greg Thanks for an amazing tutorial, really loved all the strategies. I have couple of doubts. So if we do agentic chunking wont it cost too much and also considering the scale of the data we have and also i believe it would be very costly to update with a new item aswell every time. What about retrival strategies, Also how to tackle retrival strategies in QA chat setup. Thank you once again.

just found the channel and i already know its gonna be big. remember me when you're famous

Hi Sir, what is the best chunking method to process the complex pdfs such as 10K reports. 10K reports will have so many TABLES, How to load those tables to vectorDBs?

Great content! I have one quick question though, You have specified that typically you go with chunk sizes around 2000-4000 characters. But isn't it a problem for the embedding stage? I believe 4000 characters roughly corresponds around 600-1000 tokens, popular small-sized sentence transformers (for embedding purposes) typically have context size around 512. What am I missing here? How do you meaningfully embed the long chunks? Any suggestions? Thanks in advance.

Running the code, it always throws an error when using unstructured --> "No module named 'unstructured_inference.inference.elements' " Anyone solved it?

I guess semantic search is what already vector databases are supporting.. and i don't find any advantages by doing sentence split, then calculating cosine distance and putting them in same bucket.. am i missing something here?

Hi Greg, appreciate the fantastic breakdown of text splitting for LLM. Personally I keep finding LLM's having trouble retrieving whole chapters or pages. I was wondering why we wouldn't split based on page number instead of paragraph, would that slow down the LLM ? I'm temped to train a local LLM on chunking based on user input.

Is there a way to dynamically change the chunk size ? I have text where I want to split according 4 anchors let’s say. The 4 anchors have x amount of text in between them. So chunk size can say constant, and I’m trying to use regex to split the text.

Awesome, I'm trying to do a similar thing with semantic chunking on historic chat messages, but every new message that comes in means you have to re-do the chunking. Can you think of a better way of chunking chat message history.

8:52 Thou shan't troll the beginners for you were one too. - Pro Code, Rule 4.

Has someone solved this issue when running the function partition_pdf(). I get this error: module 'PIL.Image' has no attribute 'LINEAR'

Hello Greg, Great video! ANy chnace we can get access to the Agentic chunking code?

I've a better way for imsgesyin PDF, I converted PDF to markdown and turn all images to markdown image reference, then put all the images in a separate folder, that way my embeddings show markdown text with markdown images which can later be turned into html images in the chatbot

Does anyone have an example of agentic chunking (level 5) as javascript?

When I try to run the same code for reading tables from pdf and saving image from pdf my kernel shutdown and gives message that it will restart again. How to overcome this? Thanks

One useful technique for performing sentence embedding is to apply COREFERENCE RESOLUTION, with a cheap model like Haiku: """Identify pronouns, definite noun phrases, and other referring expressions in the text (for example "it", "he", "she", "they", "this", "that", "those", "their" etc.). Determine the antecedent or the entity to which each referring expression refers. Apply coreference resolution by replacing the referring expressions with their antecedents or a more explicit description of the entity."""

Maybe I'm missing something here, but isn't there the risk that in the Date&Time chunk you end up getting a chunk that reads "The month is October" and another that reads "The year is 2023" but that don't relate to the same context in the original source? In other words, are we sure that those wouldn't be placed togheter unless it was actually October 2023 in the source?

How would you suggest to split legal documents ?

cool ideas, another way is keybert till keyword repetition stops

OMG this video is so precious for me❤. I am a web dev and started studying LLMs for fews days now. I had no idea about "splitting" "embeddings" "retrieval" etc. You really well explained in here. Thanks!

Unstructured is a great package... But very very slow. If you need to work with big pdfs you will need 10+mins to extract all images and tables...

good video but you speak so fast

Hey greg, do you have a discord?