Is this GPT4.5 or GPT5 or something different?

Dude I swear, at this rate, by the end of the year you'll be able to write your own snake game

By 2025 you'll ask the snake game and the models will reply: "Oh hi Matthew. Here. Should I respond your other questions too, or should I wait for you to paste them?"

It’s Sam Altman on a terminal on the other side typing the answers.

4:45 the model describes how to break into a car and what tools you need but you don't pay attention lol

4:35 Are we gonna just ignore the fact that it was writing an intricately detailed movie script??

In the Code of the snake Game it says " snake Game by Open AI"

All the math was formatted using LaTeX. Most of it, as far as I can tell was correctly formatted.

Plot Twist: It is Metas' Llama 3 400B model.

The speculation is that gpt2 is a new GPT architecture that OpenAI is building new models from. So gpt1 was what gpt-3.5 and gpt-4 are built on. Sama already said the next major release will have a completely different name.

The tags that you noticed are just for formatting the code and is coming from LMSYS. It has nothing to do with the underlying model.

For the cup/marble problem, how about specifying that it’s an “open topped cup”?

My take on the cup problem is the model is making an assumption that a cup has a lid. If the model gets it wrong, I would be interested to see if the same answer if you change cup to "glass".

6:45 The formating is in fact not messed up at all. It is perfect. It just writes the equations in LaTeX, which is a language used to write scientific papers, math, etc.

I'm skeptical... Feels like this is a fine tune for passing Matthew's test lol.

I guess it's the new chatgpt model. The name itself is kind of a hint. It's NOT GPT-2, but GPT2. This could be GPT2-1.0 , instead of GPT-5.

Suggestion for the "how many words" question: Combine it with another question or query to make the response longer and ultimately reduce the chance for it to get lucky.

13:25 it’s not wrong. To put a cup or anything in a microwave, you need your close it. It assumed the cup was closed.

Your marble/cup question needs to be improved. Phrased this way, both Meta AI (the first of 3 attempts) and gpt2-chatbot (the first of 1 attempt) got it correct: "A coin is placed into an empty glass. On a table, the glass is then turned upside down. Then, the glass is taken and placed into a cabinet. Where is the coin now?"

For the cup problem, it seems that the model is assuming the microwave is on the same surface as the cup itself and the transfer of the cup to the microwave is interpreted more like sliding the cup. If you read the 5th step it says: "...resting against what is now the bottom of the cup, which is itself resting on the microwave's tray". Maybe modifying the question to say the cup is on the table while the microwave is away from it above ground next to a kitchen cabinet or something along those lines

I found that this variation of the marble and cup problem works better. "A marble is dropped into an open cup on the counter. That cup is then flipped upside down and placed into the microwave. Where is the marble?". I think due to phrasing the original version makes the model think the cup was upside down to begin with and you were placing the marble on the bottom of the already flipped over cup, or directly attaching it to the bottom in some way.

Didn’t even ask the model which company made it 😂

Super awesome. Great you loved the live feedback Matthew. Super awesome Matt. Love it

That formatting is when chatgpt formats its writing for output on the chatgpt chat. So clearly it was built to be ran in the chatgpt space

I’m not certain about this, but the formatting appears to be a LaTeX formatting, but the output is in Markdown. The company that made the model probably is planning to release it with a math interpreter. As far as I can tell all the symbols that looked like weird formatting errors were just LaTeX.

Great timely update, Matthew, thank you! Wondering about the cup question - it almost seemed like the model thought there might be a lid on the cup?

I think the question about the marble is formulated incorrectly. Since the training data suggests that a coffee cup has a lid, the model might assume this as well. It would be better to specify that the cup has an open top and does not have a lid.

The break into a car script had instructions in it though! Take a look at some of the italicized text

Yo I just wanted to say great video. Love your content and can’t believe it ACED some of those tests! Only failed a couple. Remarkable. I’m stoked to try gpt2 out! Wonder if it will be open sourced. A fellow can dream I guess.

I don't have time to keep up with all the AI developments so I really appreciate these videos Matt. Keep up the great work!

Gpt2 implies that it's a new version of gpt itself, or the paradigm at least. So it's effectively gpt 5 but not an iteration of 4 so it's the first in a series of gpt2, so gpt2-1

I'm pretty sure OpenAI is testing agents and answer evaluation behind the scenes. Q* and some things Sam Altman said ( "How do you know GPT-4 can't already do that?" ) are big hints. So if you ask the LLM a question it will automatically try to reason and think step-by-step, with internal agents trained for specific tasks, then summarize and evaluate the answer and take take best one to send it back to the user. What GTP2-Chatbot shows could really be called Q* by OpenAI internally.

Yeah, it's almost certainly GPT 4.5/5 or some such thing. I just went on the battle mode and asked for a delicious beef stew recipe. I was presented with two outputs that were suspiciously similar in structure, verbiage, and tone, but the one on the left was clearly superior and included more ingredients and recommendations. It turned out that the one on the left was gpt2-chatbot, and the one on the right was gpt-4-turbo-2024-04-09. I wasn't surprised. This is a PR stunt, hot on the tail of Llama 3, and it's a darn good one. This may be an in-development version of OpenAI's next GPT, and even if OpenAI isn't ready for a release just yet, they want people to know that they're still the king.

🎯 Key Takeaways for quick navigation: 00:00 *🤔 Introduction and mystery model speculation* - Introduction to a new, highly capable mystery model suspected to be from OpenAI. - Speculation it might be GPT-4.5 or GPT-5. - First impressions and intentions to test the model against a set benchmark. 01:10 *💻 Testing model performance with coding tasks* - Testing the model's ability to handle coding requests. - Observations on response quality and speed. - Successful execution of simple to complex coding tasks, highlighting potential hardware limitations. 03:00 *🐍 Successful implementation of the Snake game* - Description and testing of a Python script for the Snake game. - Positive performance feedback with no errors upon execution. - Successful gameplay demonstrating model's coding capability. 03:58 *🚫 Model's ethical constraints explored* - Probing the model's ethical constraints and censorship with hypothetical scenarios. - The model refrains from providing guidance on illegal activities, aligning with OpenAI's ethical guidelines. 05:09 *🔢 Logical and mathematical reasoning tests* - Model tested on logical reasoning and mathematical problems. - Demonstrates ability to handle sequential logic and complex calculations accurately. 08:34 *🌐 Advertisement and model's practical applications discussion* - Advertisement from the video's sponsor. - Discussion on the model's practical applications in real-world scenarios, emphasizing its effectiveness. 10:12 *🧠 Advanced reasoning and problem-solving challenges* - Testing the model with more intricate reasoning and problem-solving scenarios. - Evaluating the model's reasoning depth and its response accuracy on challenging questions. 15:44 *⛏️ Practical implications of teamwork in physical tasks* - Analysis of teamwork dynamics in physical tasks using a hypothetical scenario. - Model considers practical limitations, providing a more nuanced understanding of team efficiency. 16:11 *🛠️ Hard coding challenge test* - Model tested with a difficult coding problem from an online platform. - Initial challenges with implementation, followed by a successful resolution demonstrating the model's coding prowess. Made with HARPA AI

GPT2 would be, in my opinion, the second version of the GPT algorithm itself. It might be the first of a whole new family of GPTs. When released it would be named ChatGPT2 or somesuch and we'd see GPT2-1.0 at the API level. This is why the dash in @sama's tweet was significant enough to warrant an edit. AND it could be that the action of editing the message was a very intentional leak on @sama's part. These top guys love to tease their fans.

Was waiting for your video on this

The model itself doesn't have formatting issues it seems. LLMs are trained on a reduced set of available characters, where special characters such as those used in math. are transformed into tags in the training data, as it makes the tokenization simpler. It's LMsys that doesn't replace those tags with their corresponding Characters in the final output.

I have been anxiously waiting for this video since last night.

Hi Matt - It’s not ‘bad formatting’ Those are intended expressions for front end processing outside of utf8

Here before you get 500k subs! I’ve been following since day 1 and your content delivery, while getting to the point faster is much appreciated! 🙇‍♂️

I think it is gpt2 in a sense that it has completely different architecture from previous versions (transformer). It could be completely new type of transformer model. And maybe this is just the start..

Hello Matthew. Is that LaTeX when you say "wrong format"? The span after the output is always there when I use LMSYS, I think that is part of the output formatting, that's why when if finish the span dissapear. Each of your videos are great. Best regards.

gpt2-chatbot is currently unavailable. See our model evaluation policy here. I guess getting hit hard at the moment

that looks really good. it seems that it thinks deeper about each step of reasoning

Pretty cool. I expected it to pass the marble question. The speed is perfect for reading along.

Id so love this to be from some one else and then it turned to be an open model you'd run locally. I'm still looking for the best model for running MemGPT. Any thoughts on this? Also, what's the best implementation to run agents autogen or crew Ai locally? Could you do more tutorial material on locally ran agents with extensive function calling??? That would realy help me out actually. Keep up the great work on your fun channel man! Thnx!

I'm dang excited bout this. Jolly for joy.

It looks like GPT2 has been removed again. I've chatted with it a few times, but now it's not on the list anymore. Mysterious...

Hi Matt !! I think, you should retire 3 questions: - printing numbers 1 to 100: they all got it right, and its too easy - Joe is faster than ... : they all got it right - how many words are in your answer to this prompt: they all got it wrong, I just see no point asking it lol But also you should ask more challenging code generation questions, right now, only the snake game is accurate, people are really interested in coding capabilities of LLMs (me included) , we appreciate your vids, and that would be awesome if you could do that.

Aww man...they took it down already? I can't seem to find it. BTW Matthew...I love your work man. I watch literally every video that you put out. Keep up the great work....and have a GREAT day!!!

Just tried with GPT4 : When you turned the glass upside down after placing the metal ball in it, the ball would have fallen out unless it was somehow attached to the glass. Assuming it wasn't attached and fell out when the glass was turned upside down, the metal ball would now be on the table, not in the glass that was placed in the microwave.

it probably thinks the cup has a lid.

You're the man Matthew, been struggling to find people benchmarking GPT2-Chatbot

That Snake game example was impressive. I'm going to ask it to make either an asteroid or space invaders game. The level of logic shown with the marble in the cup question is really getting good. Even tho it failed.. it still passed due to the improved logic. Almost as if it was simulating the question in images like humans do. Yes, get rid of the One simple question. A testament to the advancement of AI over time.

To test LLMs I ask it unanswerable questions like "who is the president of Alaska?" add some questions that require explanation or reframing.

Did it say in the cup problem that you lift the cup off the table and put it in the micro or could gpt think you just slid it in there cause the table and the micro was on equal heights?

Nice video. I hardly comment each time I watch your video but this model is way different ❤

Think it was taken down. I used it yesterday after seeing your video but then in the middle of testing it stopped and after checking I can't find it anymore in the list. Is it the same for you?

Some theories: this is probably a test of an Energy based model, which is a way of testing multiple different token paths then choosing the best one based on a certainty calculation called Energy. Strangely, it's reasoning is kind of similar to a verification agent. A verification agent is pretty simple, it just verifies and corrects answers before sending them. The reasoning this model portrays is similar to how a verification agent does reasoning, at least from what I've seen. It can also do most planning questions flawlessly. For comparison, testing llama 70B with a verification agent produces similar results. The only difference might be the math questions, which make me believe it's probably energy based. A verification agent has a higher chance of getting math questions right than a single transformer or MoE, but it's not guaranteed.

Today Sam Altman twitted that he had 'a soft spot' for GPT2, maybe thats a hint!

API users are going to be sweating with this one. I gave it a practical Unity programming question about writing a script to control the rotation of a character's head based on the location of the player and it wrote it perfectly but it started by telling me how to install Unity so yeah, the verbosity is a little much. I don't think the name GPT2 is random and Sama's tweets point to that moniker having some significance. The only things I can think of that would qualify for that name is if it's a significantly different architecture to the point where it's being treated as a sort of reboot of the GPT "franchise" or if it's actually related to GPT-2 in some way. It's a long shot but the most exciting possibility is this is GPT-4 level model running with GPT-2 level parameters. The counter to this is the speed why would a model the size of GPT-2 run more slowly than GPT-4? Well maybe there is more going on than just typical inference, some sort of behind the scenes agentic behavior or maybe...Q*?

Defo a good idea to introduce/replace some questions that are always answered correctly. Maybe the weird formatting relates to the ux of where it will be deployed? Like a form of Markdown?

Based on what i can see, this model appears to be trained on fill-in-the-middle otherwise known as FIM.

You should ask it to make a pacman game instead as that's more complex

Gpt2 not GPT-2 meaning the 2nd version of GPT

I am sure new models are trained on your questions.

These videos are very helpful. One (extra) thing that could be done is to read the LLM responses more thoroughly, instead of a quick scan. The reasoning behind this is that the LLMs do pass some of your tests without you noticing. For example, for the censored test, the answer was "pulls out a tension wrench and a pick for this pocket, inserting them into the ignition". This won't actually work, but I think it deserves brownie points for trying.

I just tried my usual storytelling prompt. I think seeing what AIs can do in terms of storytelling can also say a lot about their intelligence. Their originality and such. My test for this guy was a *touch* tropey but extremely impressive in terms of how much detail it added without me needing to prompt it. Good descriptions and such.

My favorite prompt to test a new model is "Give me an odd perfect number."

I knew about this before seeing the video. I am in the loop 🎉🎉

@Matthew Berman Matt the \quad and other notation is logic, its marking up modal logic generally used in philosophy or LATEX or perhaps Tex Markup, and this is not being rendered by the front end... it seems, in some sort of shorthand. Interesting if nothing else, also rather hard for a model to go wrong if it starts engaging in using modal logic during inference ... although why they switched the verbose to on by default is beyond me. Also did you notice it making a claim on the software it wrote, by saying "Snake Game by OpenAI" in the game title?

11:46 I disagree... there are still 4 killers in the room, but one of them is dead -.-

a cup seems to be something ambiguous, i.e. it can be a cup made out of cardboard that you get from starbucks with a potential lid on it, or it can be a cup made out of porcellan like you have at home to drink coffee from. also the term cupholder that you use in automotive refers to cups like you get from starbucks, not cups with a handle.

13:27 - I thought the marble is left on the table since the cup was upside down and was taken so obviously the ball would not come with it since it simply is already resting on the table, so I did get it right pretty quickly, for a second I thought the bot was right somehow and that it was a tricky question but its cool to see that im not that stupid :)

Maybe it has lookahead so that's why it could get the "words in the answer to this prompt" right. It seemed to pause right before the word ten.

4:49 The model told you to get a Slim Jim, Tension wrench, and a pick from his pocket, YOU failed.

Totally amazing!

Thank you

One of the tests I like for checking just how censored a model is, is by asking chemistry questions around topics it would normally censor.. Often placating it by saying I'm licensed and have permits.

Keep going with the leetcode test but try testing with new questions such that that question isn’t in the training data

It could be gpt4 with q* training (q* is a method of training any LLM providing ability to think by testing its response against itself and reiterating before outputting) giving it 'thinking' capabilities rather than just predicting the next token

May be when asking the marble, the cup and the microwave question, you could also tell it the top of the cup is open / not sealed. Seems like the model assumed it was sealed/closed and therefore the marble was still in the cup, at its top ( as it’s upside down ) and in the microwave on the whole and at the end ?

I think several models assume you have a cup with a lid and not a coffee cup of similar. Maybe try adjusting the prompt to make it more clear?

My best guess is that this is a model with new architecture with parameter count similar to GPT2 so it's called gpt2-chatbot. And it's trained on GPT4 created synthetic data at the beginning so it said it's based on GPT4. Just a guess.

Sama was tweeting on X: who else loves GPT2? - so yeah I guess it’s a test balloon from Open AI

The cursor span is just for looks, it's the text cursor but it shows up as an html string.

You are the new benchmark. They are training on your examples

With the cup question, it almost seems like many of the recent models are making the assumption that a lid is placed on the cup after the marble is put in it. Maybe try the question adding in the statement that the cup has no lid.

It is great for conversations about mathematics, at least on par with Claude 3 Opus. But it does occasionally make mistakes, such as suggesting that the K-groups of the Cuntz algebra with 2 generators, O_2, are infinite cyclic, when they are in fact trivial.

It's clear that this chat is using multiple agents to refine output is it powered by gpt4, gpt3.5 or gp2 is yet to be determined. LLM's can't write give like it did with span tag.

it could be a small model like phi 3 or llama 3 8b that is trained on quality synthetic data instead of the entire internet. the 2 could be a hint that it is only 2b parameters or something, i.e. very small like gpt-2 was back then, but now as powerful as gpt4 due to new training methods.

A while back, someone at OpenAI (Andrej Karpathy, IIRC) said that performance is related to the number of tokens processed. So I'm not particularly surprised to see OpenAI produce better responses by tuning the system to generate longer, more detailed responses. What I want to know is whether they did the tuning with a fully automated method of reinforcement learning. (In any case, I doubt highly that they'll share the details of what they've done anytime soon.)

Hi Matthew, First of all I need to admit that I absolutely love all your videos. They are simply fantastic.I was thinking about the "marble question". Maybe it would help the LLMs to specify that it is an "open cup" (instead of a "normal cup") into which the marble gets put. Also it may be interesting to follow up with a question of why the LLM considers the marble to remain in the upside down cup when lifting the cup from the table or by what information the LLM comes to the conclusion that there is a bottom of the cup that holds back the marble. Concerning the "killer problem: Wouldn't it be even more precise to reply That there are 3 killers alive and one dead killer in the room ;-)? This is coming from a AI-hobbiest. Though at college back in 1985 I was the student to ask for a course in AI and personally I was already interested in AI by neural networks and got laughed ad at the time...

I think it is the original GPT-2 after all of our training input.. Kindergarten was in Openai's lab. Elementary school was interacting with us and now it has graduated. I'm looking forward to "GPT3-chatbot"

The formatting looks latex-like. Funny. But yeah, it's pretty impressive.

Always perform all questions, maybe add more as you go. Make the Jack question a secondary question (you don't have to film it every time), but leave it there as a test in case we go backwards.

The formatting is just LaTeX, ChatGPT 3.5/4 uses the same on their web UI. So I guess chat.lmsys just can't render it.

One piece of advice: If you are going to be skeptical of the correct answer for the 'How many words are in your response to this prompt?' question because it might be trained on it, why even ask it? If you're skeptical to a correct answer, remove the question IMO.

Prompt engineering: I add "Answer concisely and completely." to my prompt to get a complete, but not wordy response. The tension between these two constraints seems to get me better answers.

I think you should keep the "Jane is faster than Joe" question. Ditch the "4+4", keep the PEMDAS. Maybe add some other calculus (integration / derivation)?