What tests should I add to the new coding-specific rubric?

VS Code tutorial would be great!

Yes, please. A tutorial for LLM in VS Code would be great. How do we deploy them, local Vs cloud and how would a basic workflow look like? Thank you 😇

I love it!! Snake works perfectly! The 7b model is also impressive! Reaching also pretty close to GPT4 for a 7b and its also capable of creating Snake Locally!

Please make the tutorial, this looks like a ton of fun!

I appreciate these overviews, and this is great to see! I teach undergrad CS and before teaching full-time my work was in creative text generation, creative agents, and using AI to support creativity (glad I defended before GPT-2 dropped). I've been hoping to find coding models we could run on our local servers for (at least a subset of) students to eventually use, but don't have the time to keep up this the wild progress without videos like these. +1 for sure on VS Code usage, anything on deployment and practical workflow integration is fantastic. Great vid as always! Oh, and in terms of some niche use cases for different kinds of code benchmarks, beyond generation, code reviews, checking if requirements are met or violated, and checking test case coverage have all been almost fully hallucinations in the limited tests I've done, so these are always appreciated (assuming the model is meant to support instruct)

For formatting the !=, i think the term is ligature and different fonts can include them. Ligatures are special characters in a font that combine two (or more) troublesome characters into one. For instance, in serifed text faces, the lowercase f often collides with the lowercase i and l. To fix this, the fi and fl are often combined into a single shape (what pros would call a glyph).

a local copilot would be wild if you can get that to work!

Insane. What parameter model did you use for the test, you didn't say I think? EDIT: If it was the 33B then I think it would be worth the hassle to test the other models or at least the 3B or 7B because that's I think what pretty consumer level hardware can run pretty easily

YES!!!!! VS Tutorial for the win!!!!!

Really appreciate everything you do. Thank you!! Tip delivered!

You are amazing, one of the few channels that I have with all the notifications enabled. Of course, I would love that you make a video with the installation of local VSCode programming assistant. Thank you very much.

As a programmer with 20+ years of experience I can say that now I'm starting to become impressed. The code (in this case quick sort) identification element is extremely valuable on its own. Finally an local AI model has as much smarts as a high schooler.

My favorite moment by the end of the week when I discover new impressive project with your video 😊. Lovit

This was one of the better code testing processes I've seen you do on model's. I don't expect most of them to get things on the first try but after some encouragement. It would be great to see creative and effective ways to use open source llm models to build useful applications.

I love the direction this is going 🙂 Thanks for sharing 🙂

Thank you for doing that multi-turn error testing, that's what all coding LLMs testing should go into!

Thanks for this deepseek video. I have tested it and it really blows my mind. A copilot video will be highly appreciated.

I love your code model testing!!

Finally... this is awesome. Thank you for reviewing this model.

Yes please to the VS code open source code pilot. Folks working behind a corporate firewall would be rescued by that!!! 🙏

Love this!

The fact that AI is becoming a common topic between the United States and China highlights some fundamental interests between both Nations. We both want to pursue technologies and the betterment of our people. This should be a technological space that foreign Nations can find common ground on. Maybe AI can become the platform from which we as a global society interact within an open discourse.

I have a suspicion that the snake game was in their training dataset, that's why it always nails it.

Awesome video, please make a video about using an open source llm as a copilot on visual studio code. That sounds very interesting.

Go for it Love t see the Visual Studio off line integration, thank you Matthew

The ≠ formatting is a rendering option. VS code supports it. Under the hood it's still the same text.

Wild! Loved it!

The score-results vs GPT-3.5 and GTP4: I'm guessing all the scores are for un-quantized models? It would be interesting to see how the scores were for all different quantized versions of the open models too.

Amazing. Super duper keen to see how to integrate this as a copilot substitute in VS code!

Super cool! +1 for a tutorial on the vscode integration, is possible to use a local LLM for coding without the cut and paste in the files?

Let's get that tutorial, save us some time haha. Love your work Matthew, thanks for your contributions to the AI space.

Hey Matthew, Really enjoy your videos. Just a question on the code challenge section. If you change the engineer prompt to "Please find the issues with this code and explain in detail: ". You will find the outcome to be more inline with expectations.

You mentioned Phind and I wanna make sure you know that while Phind did release an open model Phind/Phind-CodeLlama-34B-v2 but later iterations have all been proprietary and behind a commercial cloud offering. So when people say "phind is the best" they generally aren't talking about Phind/Phind-CodeLlama-34B-v2 rather they are taking about the proprietary commercial offering.

So what's next on the game test front? Breakout? Space Invaders?

Very good, Sir. Thanks for the video.

Thank you for another interesting video. Please do make a video that implements a local copilot. I use Olama, Litellm, Conda, and VScode on my M1 Mac Mini. I've also installed Autogen and Langchain with this setup. So far, my only problem has been using Docker to run code from Autogen, but it runs fine on the CPU. IMO, you are working on the most important projects for helping the community march towards AGI.

Really happy that you changed the way you test the coding models, it really hurt me to see you only ask one very basic coding question that every model was getting right and then a super complex question that no model was getting right. With this new format the improvements are more visible. Also would love to see a way to use this as an AI code completion like copilot. (Maybe find a way to use it in Visual Studio as well as Visual Studio code, I've found some projects for VSC code but none for VS :( )

this is AWESOME!

Thanks for this powerful information

Yes, please. A tutorial for LLM in VS Code would be great.

Would love copilot tutorial

Very impressive !!!

This year: Snake Next year: Optimizing Crysis to run on a raspberry pi

Very cool :) Can you make a few videos about AI code assistants for C++ ?

I would be really interested in seeing a video on how to set up this model in VSCode!

Vscode tutorial of course - but would like to see the code suggestion/completion feature as well in it

When running the aider benchmark tests it also scored very well

Good stuff as per usual. This one hits all my needs... So let's get this.into autogen and see what kind of damage we can do.

This is the ONLY model aside from OAI 3.5 and 4 that will actually write good code, python, js, bash, or whatever I have tried. Great Video!

Definitely want to see how you set this up. I wonder if you can ask it to increase the size of the tiles in Snake game without breaking? Aider couldn't do that.

A tutorial for LLMs in VS Code would be super awesome!!

I test this model to beat leetcode. It works like a beast, almost every coding challange is solved even for the hard problems. I guess we have a practical coding model already. Even better, it's open source.

Yes I need to know how to run this model in vscode on local

Awesome! I want this as an API.

9:38 - it's just a 'font ligature' feature. working with a special font and modern editors

Thanks!

Definately the best coder!

Please I would LOVE to see this tutorial in how to do It to run it on local. Thanks a lot for your great work and aim to share knowledge with everybody.

Do that vs studio guide. It sounds great, I will definitely watch it.

Yes please! Can we please have a tutorial for DS integration in VS or any other IDE. Thanks!

We need a 70b coding model!!!! It would crush GPT-4

This is really handy.

Bro how you said that the background is ugly lol😅😂😂 which is true. And that’s the thing I like about you is that you are true to the core, brutally honest. Who can agree

Interesting that it used the unittest framework which is pretty much secondary to pytest these days. I would ask it specifically to use pytest. I had this problem all the time when generating python code with tests. It’s also very interesting to generate test fixtures as well with parameters to make really good unit tests. Pytest is awesome.

First open source LLM into awesome level

Yeah, a tutorial for how to install an LLM as a copilot alternative would be fantastic.

It would be great if you could briefly discuss the licenses these adopt and the type of hardware you’d need depending on the parameter sizes.

Hi Matthew, fantastic stuff, please do make a tutorial for this DeepSeek Coder AI in VS Code.

Vs Code Tutorial would be really great

Tutorial +Pointer App, thank you

That'd be great! A tutorial on using it with VS Code

Yes please do the local coder tutorial!

Your own free local copilot? Hell yeah, that would be interesting!

Isn't it great to be astonished almost on a daily basis by the progression of AI ? I will have to get this going sometime this weekend to see if it can help me finish up my Inventory management app. One question: Say I wanted to use an LLM to make a local AI for use by my company and website. Can you give an overview of what is required as far as training etc..?

hi, pls let us know how to setup the LLM in visual studio as a replacement to copilot. thanks.

Woah❤

I'd really like to see a video on how to combine an embedding model with something like this. For some reason, (using llama_index) whatever model I use to generate the index/embeddings, it never works together with the model that I want to use to retrieve from the vectorindex (sometimes I get shape/dimension errors, sometime other errors), it's really frustrating!

Could you test the models on strongly types languages instead of dynamically typed languages like Python and JavaScript ?

The first model to create the Snake Game !

autogen expert when?!!!

+1 for visual code intergration, would also like to know how it performs on C#, and if I can add the libraries and documentation and preferences what I normally use to this model like embed it. Similar like what you can do in a custom GPT

pretty cool

it’s a slept on model. Competes very well against the GPT4 coders for sure

Hi Matt: I find your videos very informative. Is it possible that DeepSeek was pre-trained to ace the Snake game? I am scientist not an experienced coder. I used DeepSeek locally and was able to get the Snake game to work. However, It failed miserably when I prompted it to create another application (e.g app to analyze stock performance). GPT4 gave me excellent code for the same and other tasks. You may need to refresh your rubric; what do you think?

I think it'd be super interesting to see if we can get this to work with aider

Deep seek inf loops so much!

Maybe it's good on basic, widely known code like snake. But it fails on something more complex which wasn't included in dataset I guess.

A tutorial will be highly appreciated, thanks

hey thanks for the awesome video. where can i learn more about how such tools are developed. i'd like to discover how the sausage is made

Using it as a VsCode editor would be awesome!!!

Vscode lesson !! thank you

When are you gonna release the llm finetune vid?

VS Code Copilot Tutorial would be helpful!

Dude, i can't believe you have to ask if we want to know how to hook this up to VS code. This would be next level

I use VSCode and would love a free copilot tool to help me out. Yes please to the tutorial!

My first game on a homecomputer was Lemonade on an Apple II. Many moons have passed since then...............

Yes we need tutorial on vs code

Tried 7b locally, impressive!

I want to see an integration with visual studio :D