Never commented on a KZfaq video in 15 years but I need to now. thank you so much for your work and questioning the status quo.

44:59 I came into conclusion that maybe I should get 3 months off from AI and just get a long holiday, I'll come back when it will be all sorted out lol

The examples that you give are rag for specifically causal reasoning. The gains are much higher when rag is used for domain specific knowledge.

Love the undertone not sure everyone gets it 😅 interesting take on the VCs trying to force the scientists to make it work to get their ROI out of failed tech

38:10 1000 dimensions and 10% efficiency. We need to make more dimensions, like 10 000 or 100 000 000 000 to get 15% efficiency 😵‍💫

A commenter on an earlier video suggested using quaternions instead of vectors. I wonder if that approach might actually save the VCs and startups from total ruin? Maybe worth a try? I do love that we are full circle back to building models though in the meantime! Outstanding work professor, as always! Thank you so much for creating and sharing these videos!

The use of an 'apply logical reasoning' step should not be thought of as a weakness of the approach, because reasoning is inherently recursive. Any set up that doesn't have something like this is fundamentally bounded, no matter how well built or trained. That can be extremely powerful in an idiot savant way, but can never really be called intelligent. There is no finite machine that could answer any question in one step. I am completely with you that we are not doing it quite right, but I don't agree with you that parametric memory is the way forward (didn't bring it up here, but you did on the last one). It has fixed size, It doesn't scale efficiently, and I don't think you can really call a system that needs to train heavily on a reasoning technique before using it a reasoning engine. For that moniker it needs to be able to have a logical technique explained to it, and immediately apply it to a problem without retraining. It needs to be able to reason about reasoning, and do it in a one shot setting. The reason it needs to be able to one shot reasoning is that complex reasoning is not bounded in complexity, so it needs some way of unbounding it's memory and run time. The obvious way of doing this is to let it fill in more tokens. At that point you might have some tokens saying something like "we should try induction, induction is done by ...", and the model needs to be able to follow that recipe. That will probably involve searching for further decompositions and relevant facts, often dead ends, though training can improve how this search occurs. Importantly you need to be able to mix information from global memory with local context specific information, so it absolutely makes sense for them to be in the same 'language'. Maintaining knowledge as a separate database is the only way to build something that really scales, and having this database also holding all but a minimal set of foundational logical tools seems sensible. The specifics of how you do this are hard though.

in regards to your aversion on RAG (which i can relate to): how to build an ai system, that is able to cite back from texts (or video/audio), if not using embeddings, metadata & rag? lets say you want to build an domain specific ai tool. it should gather information from different sources and when asked about something from within that domain, an answer should be provided including the reference. (page of a document, timestamp from audio/video, text block from a webpage). i struggle to see when to use which strategy.

Are you familiar with the reasons for conducting re-ranking step? Specifically, given the premise of extracting relevant document candidates using only DPR, I'm curious about your perspective on why we'd need to conduct re-ranking using a cross-encoder, in addition to extracting relevant document candidates by computing cosine similarity with a bi-encoder.

20:11 This is a framework for large companies or suicidal startups

With a long enough pre-prompt and context window? Can you trigger a grokked transformer phase transition through in-context learning only?

Just a basic question - isnt reasoning also able to be done via search in semantic graph as shown in that one paper? They where able to visually show the trace needed to solve the task. why does one need a grogged transformer? shortest path search between semantic concepts or so on

So you have to train a transformer for this? Can we fine-tune a base model on our data? Is this what X's Grok is doing ?

Is there a grokked gpt4 level model on ollama?

any github project on that?

Are you monetized on KZfaq yet? 😃