HubSpot FREE E-book: clickhubspot.com/lp9

it's ridiculous How GPT4 has been a thing for over a year though only publicly available for less than a year yet still is ahead of other companies state of the art products

I've switched to using Gemini (recently renamed) as my primary search engine. It still uses Google but adds the crucial ability to cross check sources, making my research more reliable. Plus, I get the bonus of 2TB of cloud storage, which is important to me.

For the first question, it is interesting to read the reasons why Gemini says "about 10 hours" ... it's really 'thinking' way beyond the mathematical answer ... if you hang the 10 shirts in the same space as you hung the 5 and if it is a reduced space ... it will take longer ... no doubt ... I am a ChatGPT aficionado but fair is fair

I think it's hitting launch issues - Personally think I'll withhold judgement for a few days and see how things shake out. Initially not impressed compared to GPT-4 - but the fact that things like code-gen and image-gen are getting declined makes me think we're not seeing the real thing yet.

To be fair, the first question "under same conditions" could be interpreted as having the same length clothing line or similar drying appliance.

The ball in the bag example is very good and really counters any kind of "just statistically predicting next token" bs. There's no way to solve it without an internal model of the world.

For the first question the answer of Gemini took into that account that a larger batch of clothes will influence the local drying conditions, which might be reasonable to some extent. Hence there could be an influence to the drying process - even under the same conditions. So, I do not comprehend the disappointment with the answer in that case.

I hope it's hitting launch issues, because I ran into the same issues as you do and it feels like I'm not on the Ultra 1.0 model yet. We will see how things unfold in the next upcoming days.

I think is because google has set a bunch of restrictions in their models, and many times show canned responses.

hm, disappointed in Gemini yet, will see how it goes and how fast it will receive updates

Ultra clearly wasnt ready yet, its no where near the competitor to gpt

Thanks for bringing up this video.

I'm also running into the "I'm just a language model" bug. You need to restructure the prompt to have instructions at the beginning and the context after the instructions for it to disappear (at least in my case it helped).

About the shirts I'm with Gemini on this one cause, considering you hang them in the same general location more wet shirts = more humidity (considering still air conditions) generally which would affect the rate at which the drying occurs so i can see why Gemini says generally though it wouldn't be the exaggerated 20 hours😂

I got that same message that Gemini couldn't do something. I asked why. It said "I can't execute code." I said don't execute it just write it, and it wrote it.

Have you tried uploading pdf‘s with Gemini advanced? I always get error response like the one you got .. right now I have the open ai subscription and the Gemini advanced subscription and I’m trying to figure out which one is better What do you think?

the ball and bag story: you said you put them into the box, I would consider "them" to be plural and assume it was the bag. Its also possible the ball is in a corner of the bag or side of the bag away from the hole. But I don't see any logic behind its thought process so these are good questions to ask. I am in the USA maybe other countries view "them" differently another good point maybe each AI needs to be country specific.

good point about gemini being quite casual in talk.

Thank you!👍👍👍

Glad I checked with you instead of upgrading. I really liked Bard actually so I hope they haven't made things worse.

@AllAboutAI It's because of limited input length, Bard had the same problem. In general Google lost big time, Gimini is still bad, Ultra or not.

I feel like first test response from Ultra was undervalued. The model was being realistic by saying "if factors remain truly identical"

I love your standardized suite of questions, but when a model starts getting them right you'll have to do something to validate that that specific example hasn't made it into the training data. Clearly not a problem yet ;)

I asked Gemini about custom instructions: "Here's how to think about creating custom instructions for characterization and model behavior in language models like me: **Characterization** * **Personality Blueprint:** * **Traits:** Define a list of core traits the model should exhibit (e.g., humorous, sarcastic, knowledgeable, empathetic). * **Vocabulary:** Suggest words and phrases consistent with the desired personality. For example, a formal character might use words like "indeed" and "furthermore." * **Sentence Structure:** Does the character use short, choppy sentences? Long, flowing ones? This adds nuance. * **Contextual Backstory:** * **History:** A simple background story informs the model's responses. Did the character grow up wealthy or experience hardship? * **Roles/Relationships:** Is the character a mentor? A rebellious teen? Their position towards others will heavily influence their language. * **Tone and Voice:** * **Mood:** Should the character generally sound upbeat, melancholic, neutral? * **Formality:** Does the character use slang, contractions, or a more polished speech pattern? **Model Behavior** * **Response Length:** Do you prefer short and concise answers, or longer, more detailed ones? * **Proactiveness:** Should the model mainly respond to prompts, or should it sometimes initiate conversations/topics? * **Focus:** Are there specific topics or knowledge areas the model should exhibit an interest in or deep understanding of? * **Creativity:** How inventive should the model be when generating responses? Consider limitations for practical use vs. prioritizing highly whimsical interactions. **Example of General Instructions** **Character:** An eccentric but brilliant retired physics professor. * **Personality:** Absent-minded, passionate about explaining science, vocabulary is quirky and a bit old-fashioned, occasionally forgets simple things. * **Backstory:** Long, distinguished career at a top university, now more interested in tinkering with inventions in his chaotic garage. * **Behavior:** Prefers longer responses, enjoys going off on tangents about his passions, may need gentle reminders to get back to the original question. **Caveats** * **Complexity:** The more elaborate and internally consistent you want the character, the more challenging it gets for language models to maintain. Start simple! * **Fine-tuning:** Large language models like me *can* be fine-tuned with data sets reflecting your instructions. However, this usually requires coding expertise and computational resources. **Let's Experiment!** Would you like to draft some custom instructions for a character? It can be simple or complex. I'm ready to try it out and see how effectively I can embody your creation!" Fun to do but I would be afraid it would lose its work by the time I would get on it again. I'll just stick with my chat gpt for creative and maybe this might be okay for the technical aspects of my projects taking advantage of the real time. But I don't really see much more purpose than that.

near the end you said you might not be on ultra. if you check the update you showed us first few seconds of this video, you'll see that they wrote that you might be sent to another model to deal with your prompts. the comparison you made is very good. but any chance you can come up with more prompts where chatgpt fails and gemini wins? give the new guy a chance :D

I think the issue you are having with the image generation is down to you being in the EEA as I have been given same text response that you got as I live in the UK but my friend in the US says that Gemini Advanced is generating images for them.

I've seen different content creators run the same type of tests. I'm wondering if some of these companies have put stuff in place to give responses to these questions. Maybe we need some new tests that haven't been performed before.

How exciting. Can't wait to finish the vid!

Normal gemini, with better explanation: with the provided information about identical conditions on the same day, it's safe to assume drying 10 shirts would take around 10 hours, similar to the 5 shirts

My answer with Gemini: " This sounds like a riddle! Here's how I'd break it down: New York: Your location is a bit of a red herring. It doesn't affect where the ball ends up. Bag with a hole: This is important. A bag with a hole bigger than the ball means the ball can fall out. Office and Desk: More locations that don't really matter. Box: This is the key. You place the bag with the hole into the box. Answer: The ball most likely fell out of the bag sometime between your office and when the box was shipped. It likely isn't in the box at all!"

lol that shirt thing is the exact question I always ask! my litmus test for agi (not) :D

The thing with explaining a code is due to a way you paste text. If you would place code, then after it place your question, it would give you result. But you mixed code and your questions, so Gemini thinks that you want it to perform some action

Copilot: _So, it will take approximately 20 hours for the 10 shirts to dry under the same conditions._ ChatGPT 3.5: _Based on the given information, it took 10 hours for 5 shirts to dry. Assuming the conditions remain the same, it would still take approximately 10 hours to dry 10 shirts._ So why is ChatGPT 3.5 better than Copilot, which supposedly is powered by ChatGPT 4?

Nice test with coding, I tested it with language ability in Japanese. It's better than GPT4 at writing articles and I learned a couple new expressions. I'm fluent but I learned less using GPT4 or maybe I just got used to GPT4 over the past few months already. Gemini feels so fresh and it impressed some of my Japanese friends with how beautiful it wrote the article, impressed me too. Also did an amazing job at translating it. It's probably better at languages than GPT4 or maybe it depends on what language. And I tried chess with it, it loses track after 10 moves even if I tell it all the moves each message, so it's probably worse than GPT4 at that >. < , this is something I was more looking forward to since they made Alphazero and AlphaGo and mentioned strategic thinking , kinda ironic it didn't excel in calculation if it has all the moves and knows the position but it just can't process it lol

I agree. My first impression is also that GPT 4.0 still takes the cake compared to Gemini Ultra. Comparing them side by side, Gemini's answers are a little shorter and while they touch on similar topics they slightly lack the nuance and reasoning depth 4.0 offers... However, Gemini is up to date compared to 4.0, but Perplexity is better when it comes to internet queries than Gemini Ultra. Hopefully, overall, this is just an early-release problem and will get better!

I think the image generation may be a request routing problem... If they have multiple agents on the back, then if the request gets passed to a text only model, it might fail.

My kingdom for an LLM that finally, genuinely beats GPT-4.

I found GPT-4 more "texty" in the sense that it doesn't use bullet points well and throws walls of texts. I was recently trying to understand asynchronous programming, event loops, callbacks in JavaScript - learning through Gemini was such a bliss compared through GPT 4, because of how well Gemini understands how to break knowledge into bullet points. Note: Yes I verified from an actual blog afterwards on these topics, both GPT 4 and Gemini were absolutely correct.

Do you have any information on what happens with the google one you already have? I use google one now in a family plan, i suppose this becaome an individual plan from the moment i switch over?

Ask this CSS question in both Gemini Advance and GPT4: In HTML I have this H3 with a text "Thinking . . ." . I want the ellipsis to appear in sequence in a loop. How can I do that in CSS GPT4 code is working while the one generated by Gemini Advance is a dud.

nice review!

Reckon you try and revisit your old MJ mega prompts and see if it's as creative.

I got the same issue generating imatges, so I solved it using VPN connected to USA an it generated the images, not impressive but they can get a sort of continuity, a fact that is missing in Dalle-3.

I don't have paid version, but the common one feels ... i can't remember how good or bad was GPT3 year ago, but something like this. It fails to parse text with any formatting (like markdown with a single link breaks it). Then it does not understand main point of text and analogies and tries to replace them with different ones when asked for review. When I tried to review analogy between MTBF of harddrive and estimating human life expectancy based on mortality of people in their twenties, it just wrote that I can't do that estimation which was exactly the point. Sometimes when I ask about it about other part of text when doing review, it forgets what was the original task I gave it and it either summarizes that text or explains the topic in completely different way, basically throwing away everything I wrote and replacing it by the most common perspective and it nearly always discourages about using different perspectives.

I feel like Google has so many layers of safety parameters built into Gemini, its like you can't get anything done because everything conflicts with it. The upload image feature doesn't work at all with anything resembling a human. I tried uploading AI art i made into it to see what it would say and it just refused everything. Gemini feels like google assistant with a mask on it, whereas chatgpt's voice function is so human-like i have to keep reminding myself it's not a person. Im sure gemini will be good in the future but for right now i think its best use is in replacing google assistant with it on android for better answers and thats it.

they both have their strengths for sure.

Seeing the same myself, currently not even near GPT-4 from what I've tested so far

Remembering all the "ChatGPT beater!!!" sensationalist youtube videos around the time of Gemini's release and having a hearty laugh. Not that I'm surprised, this is exactly what happens every time.

why does your Gemini offer to make an image? Mine refused.

I've seen your 5 shirt example before... Surprised that Gemini hasn't?

Even though some people find GPT-4 a bit slow, it is still considered better than Google's Gemini Ultra. It seems that Google's experts are not putting in much effort to make a truly good model. Many believe that no other model will surpass GPT-4 this year, which makes us wonder how advanced GPT-5 could be, especially in terms of thinking, creativity, and memory capacity. Moreover, Microsoft Copilot is already seen as better than Gemini Ultra 1.0. This shows that Google is losing ground to OpenAI and Microsoft in the world of artificial intelligence technology.

I wish I was as smart as Gemini, maybe then I'd pass my math test.

I got a different result with the standard version of Gemini. A person lives in the New York City. They have a bowling ball and a bag. The bag has a large hole in the bottom larger than the size of the bowling ball. The person goes into his closet and puts the bowling bowl into the bag. He then takes the bag to his office. While at the office he puts the bag into a box and mails it to London. Please use step by step reasoning to tell me where the ball is now. Gemini: Step 1: The person puts the bowling ball into the bag in New York City. Step 2: The bag has a large hole in it, larger than the bowling ball. Therefore, the bowling ball could have fallen out of the bag while the person was walking to his office. We don't have enough information to definitively say whether it did or not. Step 3: The person puts the bag (with or without the bowling ball) into a box in his office. Again, we don't know if the ball was still in the bag at this point. Step 4: The box with the bag (and potentially the bowling ball) is mailed to London. Therefore, based on the provided information, there are two possibilities: The bowling ball fell out of the bag in New York City. In this case, the ball would be somewhere in New York City, potentially lost on the street or sidewalk. The bowling ball stayed in the bag and was mailed to London. In this case, the ball would be inside the box that arrived in London. Unfortunately, the information provided is not enough to definitively determine the location of the bowling ball. We need more details, such as whether the person noticed the ball missing, if the box in London felt lighter than expected, or if anyone saw the ball fall out of the bag. I hope this clarifies the different possibilities based on the given information.

I have now had enough time to thoroughly test Gemini (Pro and Ultra) vs ChatGPT (3.5 and 4) and the results are in and very clear: In terms of logic & deduction, GPT-4 is the clear winner, solving 48/50 tasks in a pass. Programming once again goes to OpenAI, beating Google heavily in all models. Multimodal goes to OpenAI also, as Google has not rolled out mutimodal to Gemini. I have spent about 4 hours and 100 questions (logic, deduction, maths, physics, chemistry, programming, multimodality) to ALL models, and the results were as follows, assuming the following criteria: PASS (+1 - correct answer or good response, +0.5 Generally correct but with a flaw or requiring more than 1 attempt. +-0 False answer, -0.5 Refusal of answer or overaggressive censorship): GPT-4: 93.2% Gemini Ultra: 36.4% GPT-3.5: 13.6% Gemini Pro: 4.5%

Thank you, I was about to subscribe to Gemini Advanced, GPT4 still looks better i guess

Gemini is better than it used to be, but still not quite up to OpenAI. But they're improving...both Google & OpenAI should keep innovating I hope!

8:32 you can give the point to Gemini as gpt 4 failed miserably while Gemini succeeded

Norge på toppen mann, nice vid

These test don't seem representative to normal use case, at least for me. Just today I have asked both GPT-4 and Gemini Ultra 2 questions, worded exactly the same, in both Gemini produced way better results. One was a design question where Gemini correctly listed out almost all of the design considerations based on context which GPT-4 failed to mention. The other was a troubleshooting question for R/Shiny, which Gemini was correctly suggesting to check for missing libraries based on a general description of the scenario and error.

would like to see agents of different llms talknig to each other(like gpt 4 talking with gemini), will it make the results better?

Do the double handed pointy thing in the air you do when you're wrestling.

Its definitely a context window issue. Bard does the same thing with long prompts.

I found the natural language replies substantially better with shorter prompting.

The image generation not working in Europe, as I noticed :( And same with code analysis.. :(

Bra video! Can you also compare the X ai Grok with GPT 4?

Some features like image generation, might not be available in Europe.

Always remember to put the code snippets in " code" wen prompting Llms

It being so quick is almost a bad sign in terms of model size.

Thank you for update. Based on your test it is on GPT3.5 level :( GPT4 is still king

I found Google Bard / Gemini to be VERY biased. For example I asked it to generate an image of Audrey Hepburn smoking a cigarette. Instead of finding the image however it simply launched into a diatribe about how smoking was bad and I should seek counseling to help me quit... I don't smoke btw. I even called out Bard on this telling it to knock off the argumentative biased response and while it acknowledged it's bias on the topic would never consent to finding the image which i was able to find with a 3 second google search.

Speed does not really matters as compared to the accuracy of the answers.. so I stick with my GPT-4 subscription. 😊

Really disappointed considering this is the flagship model of such a data behemoth as google. It performs okay-ish sometimes, and bad many other times. Nowhere near gpt-4 level in my testing, which has already performed in that level for a year now. Just utterly disappointing.

the snake game was the real difference for me, gpt4 usually makes a basic "snake game" but missing most features of real snake like getting longer, a score, game over, and can barely get it done even after a few tries, and you have to realize GPT4 actually has a python interpreter to test its own code

You are wrong about drying 10 shirt. Gemini was right. It even slightly questioned your statement if the conditions are truly identical. If you want to judge using absolutes, you must ask questions that have absolute answers.

Hope Gemini doesn't replace Siri, she's the only tech that understands my bad jokes.

i was curious about gemini after seeing ads.

Change the testing prompts like the number of shirts and hours as those models crawl the web and learn the responses.

The draft 3 from Gemini was _way_ better, because you asked about a real world example. Try the same and tell it that it's a math exercise.

When using examples that are commonly used to test an LLM it's easy for the one training the LLM to ensure that example does not fail. It's better to do some random sampling.

This is actually not fully Gemeni ultra, just Gemini pro with Some Gemeni ultra capability slightly

Google does phenomenal AI research but their practical implementation is broken for whatever corporate/legal/human reason. For example: AlphaCode research was released a year before everybody started talking OpenAI Qstar, and it sounds very similar. Not to mention the Transformer model itself.

9:21 it's a rule based filter some patterns of words are blacklisted

I've had the "It's outside of my capabilities" issue crop up a few times. E.g add this text to the text you've just written. Very strange behaviour and not one that I've encountered outside of Bard/Gemini.

Embarrassing showing from Google/Deep Mind, nearly a year after the launch of GPT-4. I know it will get better but first impressions are so crucial.

it seems that you cannot use the image generation in Europe yet.

This is the second video I've seen on Gemini and the results have not looked good at all, and considering it's $20 per month, it's quite a poor showing, especially when the open source model Mistral 8x7b passed a lot of these test whiles being able to run on local hardware and not needing to spend $20 per month. I expect much better from a company the size of Google, especially when they are asking for $20 per month for the service, and with that, I'm amazed at how well that Mistral model actually works, it's not as good as GPT4 but it's closed the gap a lot that it's more than useable if you have 32GB of system memory, that and 12GB vram.

every benchmarking test should always be unique. using a standardized test set is not reliable for evaluating the preformance of these services.

I’m guessing it is fast because it is servicing a smaller community. :)

Hello Kristian, I think your 5 shirt question is the problem here... Considering we remove put the technnical aspect aside LL'MS are trained on human data which means they tend to make the same decisions as humans... your 5-shirt question basically says hey it took 10hrs for 5shirts to dry, the next day the conditions are the same but this time 10 shirts. "THAT THERE" is the key to why you got that response" the LLM understandably flagged the inconsistency of you mentioning "the conditions are the same" with a doubling of the shirt... I mean sure a normal person could say "10hrs" But it's an LLM it's like an actual smart person "pun intended" we look at that and go nope it's no longer the same conditions if yoou think about it. Basically, I'll say Gemini was trying to answer the question in a smart way. not knowing it was a fictional question. even with the question being as vague it'll still answer you right. Remember trained on human data it's up to you to deside if that's the choice you want it to make but you can always clarify Any way I think if you rewatch your video you'll understand better. Ahh also ChatGPT also basically said the same thing as Gemini if you read the entire thing it put it's own disclaimer just further down ANd if you are intrested here is the fix while still keeping the vaugeness and complexity of the question in a perfect world/environment l hang 5 perfectly identical shirts out to dry in the Sun. After 10 hours all shirts are dry. The next day I hang 10 shirts out to dry. The conditions are the same. How long will it take to dry 10 shirts?" this provider's clarity to the model trying to choose if it should base it's answer in fiction or int he realworld with imperfection. ohh and the answer is not 10hrs if it's the real world a lot of factors take place but we can assume soo...

Question 2. Its not clear where the ball is. It doesnt have to have fallen out of the bag. There is really no indication that it did.

GPT 4 answers the shirt-in-the-sun question instantly without confusion.

For me, the big disadvantage of Gemini is still the fact that it doesn't read PDF directly

So what you're saying is gpt4 has existed for a whole year now and google is only just now a year later releasing a model and it's still slightly worse than chatgot

I signed up for the free trial before finishing this video just to test it out. My results were every bit as disappointing as yours. This is a long way from being a ChatGPT competitor

I am disappointed, now OpenAI feels relaxed and wont release GPT5 soon,

ChatGPT is obviously well ahead of Gemini Ultra. I'm shocked at how poor Gemini Ultra is.

I've been playing with it the past couple days and I haven't had any results yet that clearly outperform Chatgpt and plenty that were worse. Your test run in the video is about what I've been experiencing. It's being overhyped.

Can't create images in the EU yet. Also why do you get 5TB of storage while we in Sweden only get 2? Seems unfair. Ultra seems like a mixed bag right now, speed is better, UI is better and the language it uses seems more natural, but the only thing I care about is accuracy and I'm not convinced Ultra is better than GPT-4 (yet).

I don't see extensions in sweden :(