Great way of wording this very clear distinction!

This is just multiple next word prediction are interacting in a way that represents world 🌎.

Yes, this. People so often mistake that.

There seems to be a general phenomenon in these generative models where the earlier layers build and enrich an abstract representation of the task, then the later layers query this representation to select an appropriate output.

​@@genegray9895 I agree, and that's behavior which can be leveraged to improve convergence and computational efficiency. Although, doing so does restrict the distribution manifold (it's not clear how big of an issue that is yet).

@@hjups Can you elaborate? I'm not sure if you mean the models implicitly learning an efficient way of organizing the forward pass or that we as researchers can leverage this observation to improve training efficiency. If the latter, how so?

​@@genegray9895 Both actually. I can't offer evidence for the former without going into a paper that's under review, but the latter is evident with U-Net architectures. Essentially my claim is that "The Bitter Lesson" only applies in the domain of infinite compute and data, but as researchers we don't typically live in that world (maybe Google and OpenAI approximate that regime). So in a more constrained environment, there are things we can do to reduce the cost of training a generative model without sacrificing performance (i.e. by leveraging the representation learning which the model will learn anyway).

haha i do this but very very rarely. replication takes orders of magnitude more time so i’ve gotta pick my battles

I actually did an extension of Hoffman's work that has never seen the light of day beyond one of my first shitty YT videos kzfaq.info/get/bejne/jtCiiM2QssC7Y6s.html I'm of the belief that if Hoffman is correct and the structure we do learn, while not veridical, is still useful, that AI agents should likely converge to learn roughly the same structure we have (although it may be in some ways quite different due to the nature of their substrate & architecture) and it doesn't matter whether either of our models are veridical as long as they're compatible definitely going to check out the paper you sent, based on the title it reminds me of an idea I had for an extension of his work that I never got around to

Thank you for the community service ❤

lmao i’m rigor blind fsfs

I mean it's not really a hard and fast rule to be fair. Just a sense about the work based on the work and what's been involved with it. It's easy to disregard, but also easy to give credence to nonsense. That said, I fall towards the end of I'd rather 'waste my time' than not.

it’s been used for awhile in the context of fiction stories to refer to facts about the fictional world confirmed by the author as opposed to those made up by fans in fan-fiction stories. it’s now being used by gen z to refer to important and inevitable events or themes in a person’s life, particularly common events frequently shared within a group of people (for example, cutting their own bangs and likely doing it too short when they’re pre-teens is a canon event for alternative/goth/lgbtq girls)

@@Tunadorable another word for it is gospel. But yes, how do you give spock a sister and stay canon. You develop a section 5 conspiracy that jumps her out of his timeline before star trek starts. Because the exemplary captain is Rodenbury canon and she was a progressive branch

ngl it is growing on me but i can’t say i’m loving brat as much as ive loved most prior charli albums. also that wall like never gets updated but i do one day want to switch it to a green screen and have it rotate thru different & newer albums like fantano does

The transformer updates it's internal activations during inference. In context, this activation updating can be thought of as a type of dynamics. It's not dynamics of weight updating, it's dynamics of activation states of the neurons in the neural network. One can call that dynamics a "set of useful programs" if one wants to, but what this paper shows is that the in-context activations have the geometry associated with belief state updating over the hidden states of the data generating process.

[1] All sequential training data can be thought of as arising from an HMM. It doesn't place a restriction on the structure of the training data whatsoever. The training data could be generated from a Turing Machine, and the work presented in the paper would still hold. [2] The belief states have information about the infinite future of the sequence, way beyond that of the next token. That's one way to see why the belief structure is surprising. [3] At a particular context window position, the transformer internal states can be influenced by any information in the previous context window positions. Depending on the correlation structure of the data generating process, these histories will have particular types and amounts of information about the future sequences that can come after that context window position. The data processing inequality isn't broken by this fact. Unless you are referring to something else? Just kind of zooming out a bit since maybe there are some confusions about what the paper is claiming and what it does - the HMM structure and the belief state structure and geometry are different things. The belief states have to do with the inference algorithm an optimal observer must do in the service of prediction, assuming that the observer knows the underlying data generating process. You very much do not need this structure to do _just_ next token prediction in a local sense. This video skipped over the last figure but it explicitly shows that the internal structure of the residual stream is explained by predictive structures for the entire future of the sequence, and not just the next token. Distinct belief states can have the same next token distribution, and yet the paper finds that these distinctions are represented in the residual stream, even in these cases.

Thanks for the reply. I want to preface this by saying this area of work is not in my field. However, I still don’t understand how it’s possible that a model that predicts next token probabilities can do so on unseen data without learning some concept of belief states.

I delved deeply into the definition of "understanding" and even dedicated a large portion of the video to responding to your comment kzfaq.info/get/bejne/qJmah8hnpq3Gj3k.html