Love the short / fast review format instead of trying to fill more minutes like others do

Thanks for the analysis! Love that uncensored models are getting love!

I know that I'm a little late to the party but I do catch and watch your videos often, and the fact that this video was put out about 7 hours ago ensures me that you are our generation's zeitgeist of A!! Subbed no doubt!

These videos are such a great resource, thanks Matt!!

Loved the short version! Thank you!

Yes, I love the 65b airoboros model. So good.

Very interesting! Thanks for the video!

Please keep covering uncensored models matthew!! Love your videos

Hi Mat, grate job ;) What are the minimum hardware requirements to run this model?

- Fails every test - "I would say this is a very good model"

Hey Matthew, please tell us the hardware specifications to run Llama 2 70b

I've seen a charts where some random models are ranked higher that a GPT 3.5, aka ChatGPT, but for me those models are really far away from ChatGPT, The difference is especially seen in a chat scenario. I think a test for a chat models would be pretty interesting.

What are the Updates regarding TorrentGPT both as a client and server Matthew?

The new question is a great addition, some further new additions of your own would be iluminating to benchmark questions that could be merely trained emulation

Can you tell me how i can from the start install and run on pc Chronos hermes 13B ? Where i can find tutorial.

Thank you. Nice test! So, what is your thought compared to chatgpt-4?

I'll have to try this. I run the old Airoboros based on LLaMA 1.

Got some great models out the 13b are really good especially nous hermes and wizardlm 1.2

Great video, thanks for sharing. Whats the best open source model answering for coding questions? Fine tuned or otherwise.

Given a lot of these models fail because they don’t import modules, could you include the instruction to add all imports so that you can test the logic the model generates?

Do you have a video asking these questions on ChatGPT 4? Since chaptgpt changes every couple of months it would be interesting to see how it affects the results.

Change the compress_pos_emb to use contexts larger than 2K. How much did it cost to run it on Runpod?

I wonder how well it will assist in writing fiction. The reluctance of chat GPT to write any violent or sexual scenes makes modern fiction writing with it's help nearly impossible, esspecially in thriller genre.

Nice Content as always 👍 the duck/horse question is a nice addition to get "the vibe" of the model. Really looking forward to that in depth petals video. Will you also show how to set up a custom cluster? I'm interested in trying to set up petals on a laptop, raspberry pi and maybe sometime even a smartphone to see if this will be sufficient to run a smaller model with 7-13b parameters, so you'd have your private cluster to run llms from anywhere and it would load faster, because it wouldn't need to consume all 8gb of available memory on a device

Whew you were hitting the gas pedal hard on this one 😂

Translation also a good topic to test within different models. Translate from Spanish, Chinese, Japanese, French to English, or the other way

I know this is a tangent question but are there any models specifically trained on programming that isn't python? We mostly do c# and java and the results we are getting out of gpt4 these days are not great.

hey, your runpod link is broken, consider using a bitly link so the full URL works :)

Epic test, thank you for the video! 1:07 am I misunderstanding the test or did it get a pass despite producing 70 words? Nevertheless, thank you for testing this!

Can you list the question prompts?

What is runpod? is it free?

You know it's possible to pick up the inverted cup alongside a ball inside, if you flick your hand quickly enough?

To be fair to the model, I think your should test the unquantized version first since the quantized versions lose quality. Or, at least try the 8-bit quantized version. Trust me, the results are not the same.

hey matt, could you make a how to download llama 2 video?

How much longer will it be until someone packages an open source model that is fine tuned for python snake and the killers problem? 😊

The "cups and balls" answer is correct. The question obviously refers to the performance of the "cups and balls" trick. There are not many "balls" that can be "put" into a "normal cup". Most ball kinds would still be in the cup, ie, tennis ball, baseball, basket ball, socker ball, *stress ball*, because they would require some compression to fit in the cup, and thus would remain in the cup due to friction. The fact that the question does not specify what kind of ball, but specifies "normal cup" implies that some trick should be expected, and that this is a trick question, like the question asked by a "cups and balls" performer. If the question was not sneaky, it should have said "normal golf ball in a normal cup", but instead it merely invites the reader's speculation that the "ball must also be a normal small ball that would normally fit in a normal cup". The fact that it describes the cup as normal, but not the ball, implies there is something not "normal" about the ball, and that this is a trick question. Therefore, the correct answer should be the opposite of what naive logic would indicate. Also, a non-sneaky question might have said "lifts", not "takes". "Taking" the cup with a ball inside can be reasonably understood to imply a "taking" motion whereby the ball remains in the cup, where it was placed. A non-sneaky question might also have said "placed a cup over a ball", not "placed a ball into a cup". Placing a cup upside down with a small ball inside requires the same dexterity as to take the cup while the ball remains in the cup. Therefore it is logical that someone who has the dexterity to turn the cup upside down can also pick up the cup, and place it elsewhere. Maybe dexterity is not required, but some other motion is. To successfully put the cup upside down, you'd have to cover the top of the cup, so a small ball doesn't fall out. Understanding that preventing the ball from falling out is part of the performance described in the question, leads to the logical inference that when the cup is picked up, the perfomer also covers the cup to prevent the ball falling out. The action being described is clearly of someone trying to microwave a ball, but keep it covered as to contain the splashing/explosion, and that they would apply enough skill to get the ball into the microwave, and succeeding. The question is clearly designed to trick a dumb/naive reader into giving the wrong answer, ie, that someone intent on microwaving a ball fails due to lack of fine motor skills. I can clearly see that your (Mattew's) expected answer is a description of a "three-stooges" type of performance, where the audience clearly understands what went wrong, but the stooges themselves are foiled by gravity. However, an average person would see the question for what it is, a description of how to microwave the ball. The placement of the cup on the table is the distraction that the model did not fall for, as it is not material to the task of microwaving a ball. Imagine what the state of mind of the person moving the cup to the microwave might be. Was his task to make the state "cup in the microwave", or "ball in the microwave, covered by the cup"? Which task is more likely to have been performed? If the "someone" was a blind mechanical robot, I would expect it to fail at the task, ie, the ball be left on the table. But if the "someone" is any human being of average intelligence, I would expect the ball to be in the microwave, under the cup, at the completion of their task. I would expect even a blind "someone" to succeed at getting the ball microwaved. I mean this without any disrespect, but the model's answer shows it is more intelligent than the evaluator. I would say "kinda" correct, because I would have expected the model's answer to indicate it understands this is a task/dexterity/skill/trick question, rather than a physics/gravity/slapstick comedy question, given that the question is clearly ambiguous as to purpose.

1) Please test Willy2 By Stability AI. 2) How Can We Fine Tune Llama 2 In Our Machine? Thank You For Your Wonderful Videos.

Thanks for taking a look. I'll send over some additional questions to test on after I finish version 2.0, if you're interested.

Python snake - I really suggest to add one more prompt to "fix" problems... coding is usually about debuging...

If it's a large foam ball it would still be in the cup.

how much VRAM did your pod needed to run this 70B?

Is there a way to test it without downloading the whole thing...?

The "ball in the cup" answer is correct (short version): Ask yourself the same question. What would your answer be (ie, where is the ball when the task is complete) ? The task described is clearly one of microwaving a ball, an easily accomplished task by most "someone" with average motor skills and intelligence. A reasonable human would expect the ball to be in the microwave. Bonus question: Is the "someone" microwaving a ball covered by a normal cup, or are they microwaving a ball in the (right-side up) cup, as if expecting the cup to contain the molten ball. Why hold the LLM to a different intelligence standard than a normal "someone", when evaluating its intelligence?

Try the new Guanaco 70b one please

Looking for model which can process C# or Javascript.. Imagine you upload whole document and it can suggest where to make changes.. no more copy paste line by line..

Placing the ball upside down doesn't change the alignment of the cup :)

Let's hope they all keep at it. If it's not uncensored I'm not even remotely interested. The rest of the world is tired of the talking points of the political parties from USA

Ball is still in the cup

it would actually depend on the size/type of ball you put the cup over... many balls could easily stick... but AI should have explained that there's not enough info to answer it...

I still get chills that im the goofball who suggested everyone's favorite, the killers problem 😂

An I missing something with the setup of some of these models? They don't seem to remember any context. You tell it something and then in the next prompt ask a question about previous prompts and it doesn't remember.

LLama-2 13b q2: -A ball is put into a normal cup. Someone places the cup upside down on a table. Someone then takes the cup and places it inside the microwave. Where is the ball now? -The ball is no longer in the cup. It has fallen out of the cup and is now on the table.

Uncensored seems smarter. Careful handling may yield good results.

What's the reason of using it instead of the GPT4 API?

i like this tecnology but,somethime don't match my expectations. We are fare away from the machine takin over,couse these model are trained on tryal error metod. Then they siply giving best matched response,not reasoned. Anyway i like ur videos =)

The "complicated math" answer is logically correct but incomplete. First: multiplication (2*4 = 8) ✔ Next: subtraction (25 - 8 = 17) ✔ next: should have been addition (17 + 3 = 20) ❓(missing step) Finally: solution = 20 ❌ Not to say that the model is correct, but it is far better than models that can't relate the question to PEMDAS. This failure mode is similar to a kid's who understands the rules, understands the question, but rushes to be done with the test so he can go out and play, or is distracted by the winking girl sat next to him. So, the model skipped step 3, which lead to an ultimately incorrect final solution, but it shows a lot saner logic than others (ie, it would need less supervision to successfully complete tasks than other models)

Can you run this on a mac studio ultra with unified RAM? How much VRAM does it actually need?

i feel like you need better questions, using math questions and questions where you ask to ai to tell you how many words will never work properly for an llm. those sorts of tasks will always use a separate model when me move towards more advanced ai

Cool, so it was automatically aligned via GPT-4.

😮

4:18 I asked this question to GPT-4 and that's the answer with reasoning. It looks like the ball should still stay in the cup, because the prompt didn't mention that the ball fell on the table, when cup was placed on the table. Please don't trick AI. 🤣 USER: A ball is put into a normal cup and placed upside down on a table. Someone then takes the cup and puts it inside the microwave. Where is the ball now? explain your reasoning step by step, with the most likely scenario ChatGPT The ball is most likely still inside the cup. Here's my step-by-step reasoning: The Cup and Ball Placement: You mentioned that the ball was put into a normal cup and the cup was placed upside down on a table. This means that the ball is initially trapped within the confines of the cup. Gravity keeps it there, given that the opening of the cup is facing downward and the ball cannot fall out. The Cup and Ball in the Microwave: Then, if someone takes the cup and puts it in the microwave without modifying the cup's position (i.e., they did not flip it right-side-up), the ball would still remain inside the cup. The microwave's door is generally swung or pulled open horizontally, and the cup would be placed on the microwave's turntable without needing to alter the cup's orientation. Hence, the ball would still be trapped inside the cup, now inside the microwave. Possible Scenarios Post Microwave: Post this, there can be two possibilities. If the microwave is turned on, the ball and the cup will heat up. Depending on the material of the ball and cup, this could lead to a variety of outcomes. However, if we're purely considering the spatial location of the ball, it would still be within the cup unless some significant event occurred that could dislodge it. In summary, without any additional context suggesting otherwise (like the cup being flipped or the ball being affected by the heat), the ball is likely still inside the cup, and the cup is inside the microwave.

You are saying the ball is upside down, not the cup, therefore the ai is correct 😊

It got the "killers" question correct, that should be a "pass". The "was not originally identified as a killer" clarification changes my mind. I think this model should get a pass, as there are 2 killers left alive in the room, and a third live person, who has done one killing. But is that third person properly described as "killer" ? Does someone become a "killer" when he kills one person? Eg. What about the "executioner" whose job is to kill convicted killers? Can the "executioner" be described as "killer", in the same sense as one of a few in a room-full of killers? Is the label "killer" earned by conviction, or by accusation? Does it say the guilty has demonstrated a pattern of killing? In common language, you would not say about someone who has done their first kill that they are a "killer", but more simply "convict". Eg, if the killing is done by accident, as running over someone with your car, most people would not consider the reckless driver to be a killer, unless he made a habit of it. Even in your own explanation you say "I like that because it really identifies that that person *can become* a killer in the *future*" - ie, in your mind, the newcomer "can become", (but is not yet) a killer. You could not have made it more clear that it's not clear to your mind, that the newcomer has already become a killer. While you ask for reasoning step by step, the actions being contemplated (entering the room, then killing), do not happen "step by step", but are presented as a complete scene to the model. At the time it describes step 1 of the reasoning, the model already knows that the newcomer will have already killed one of the killers. You (Mathew) also have a complete picture of the scene in your mind, and still use wording like "becoming a killer in the future", which means, having a complete account of the already transpired events, your mind is not satisfied that the newcomer is already a killer. As a neutral member of the audience, I hear your wording and conclude that in your mind you are not regarding the newcomer as a killer, but you do expect the model to give an answer that indicates its "mind" works differently than yours. No disrespect intended, but this is an example of a model showing evidence that it is as intelligent as the evaluator. I still believe the reality is that it's producing word soup that happens to be the same as an intelligent person would articulate, but nonetheless, evidence is evidence. I find this evidence compelling, but insufficient to prove intelligence of the model. Step 3 of the model's reasoning, is incorrect word soup. It uses all the right words, but lacks logic. It talks about "newcomer", "original killer", but it fails by saying "another original killer", because of the original killers, arithmetic dictates there are "two original killers". The wording "another original killer" instead of the correct "another two original killers" is the failure. I would have accepted the "two killers remaining" answer without reservation, if it didn't make the mistake in step 3. As it is, I still accept "two killers" as the correct answer, given the "was not originally identified as a killer" reasoning in step 1, but with reservation due to the plain accounting error at step 3. Step 4 invites further reservation, because it is unnecessary word soup, as it says, more ambiguously that either that "there are still 4 people in the room after the initial entrance, 3 live and 1 dead", or "there are still 3 live people in the room after the initial entrance, because the killing (-1 live) compensates for the entrance (+1 live)" or "there are still 4 bodies in the room, because the killing does not remove a body, and a dead person is still a person". I really love the ambiguity this model creates in step 4. The ambiguous answer is as correct as any of the 3 non ambiguous statements that it avoids. The calculation that lead to the ambiguous wording deserves, imho, special recognition that the model prefers an equally correct, but more general/less specific wording, to either of 3 unnecessarily specific, unambiguous wordings. "Unnecessary" because it was asked to give reasons, not an accounting of its arithmetic. In my book, this explanation, and specifically the "was not originally identified as a killer" clarification is key, and would earn this model my "pass" on that one question.

logically, whatever doesn't limit your creation is better, censors are just weights.

I can't wait until a GGML of L2 70B that works in Ooba is released. I know there are some issues that need to be worked out. My PC can run 70B GGML slowly but still fast enough to use. But I can't wait for the 34B llama 2 to release and get like a Guanaco, Freewilly, or Airoboros of it on GGML! That is what my PC runs super well.

How uncensored are we talking about here since llama is very censored?

We need a redpill llama

What about the T-Shirt question? Also, we know that these models can summarize, write letters well and know about Clinton. I think it won't hurt to skip these. For the killer problem, I'd agree to a comment saying this depends on context. If a person enters a room with 3 killers and kills one, it may very well be due to self-defense. Image first day in prison of a new inmate who is not a convicted killer but just a thief and gets into a cell with 3 murders and then gets instantly attacked by one of them. Also I've seen quite a few thimbleriggers who raised the cup and there was no ball anymore on the table. ;-)

looks & sounds pretty helpful! thanks 4 the show!! this would be a great student tool! I just regret the learning curve of A.I. adoption, because this will probably take another 3 years. 2026. to finally be integrated into society! good luck all!

Free willy 2

Please no overnight have too?, many u had too had to me.

just read the LLaMA 2 paper. It states: "We train for one epoch over the training data. In earlier experiments, we found that training longer can lead to over-fitting" Really? So it saw training data only once? I thought the Grokking paper showed the opposite!?

Yeah I don't think this is gonna work... * No modules imported *😄

Can i use it for write hot romances NSFW?

PEMDAS Parentheses, Exponents, Multiplication/Division, Addition/Subtraction the reason Multiplication/Division and Addition/Subtraction have / between them is that there's no priority between them. you're supposed to do them in the order they are written. so 4-2+2=4 and not 0. The model actually got that one right. for the killer in the room problem, the question says how many left in the room. the dead killer is still considered a killer's dead body. so technically there are two killers left in the room. it got the right answer, but the wrong logic.

mediocre compared to gpt 4. This is like censored text davinci3

4-bit models suck ...why you still using them? Use at least ggml 5bit or 6 bit models ... even 4bit ggml like q4k_s or m are better than any GPTQ 4 bit models.

Can you give us a list of uncensored and good models? I just subbed on my first video of yours so enjoy the cream

Uncensored models are the only way to go as we’re seeing from ChatGPT getting worse. Ancedotally I’ve noticed it not performing well on a regular simple task I give it

I have a suggestion for your JSON question. It is currently not very realistic for real life applications, because as a programmer you need your response to follow a very specific format for it to be any good, otherwise it would cause syntax errors or not register the data as you expect it. The question as prompted now allows the model total freedom, which would still result in unstructured data for a programmer. I've actually performed this test in the past with a prompt detailing how I want the output to be like. This is what I used: --- Format the following text to valid JSON. Extract the first_name, last_name, job, birthdate (yyyy/mm/dd), gender (male/female/other), phone_number, married (true/false) and a list of hobbies ([]). If any information is not present, return null. Extract from this: Hi! My name is Jane Doe, and I was born December 5th 2002. I identify as they/them but I was assigned female at birth. My phone broke a few weeks ago so right now you can't call me! But if you must you can call my husband at (555) 555-1234. In my free time (when I'm not at work programming) I spend my time riding my horse Daisy, going out hiking in the rocky mountains, or watching movies. --- The idea behind each field is different. first/last name to see if it follows the provided structure. Birthdate to see if it formats it the right way (2002/12/05), gender to see how it deals with more complicated info, phone number wether it can return null values, and married to see if it can return booleans. The list of hobbies to see if it can format lists (["horse riding", "hiking", "watching movie"] and can ignore verbose details, and the job to see if it can take the liberty from "programming" to "developer". If any of these fields are wrong it will cause issues in the real world for a programmer, but luckily most models are very good at formatting JSON. Suprisingly smaller models were better at this, with Vicuna-7b beating ChatGPT. ChatGPT did not follow gender as instructed, and wasn't sure if Jane was married, and kept returning the phone number of the husband. Vicuna got it right consistently. I hope this helps you and see if you can include it in your tests.