Is this the ultimate unlock for open source models to compete with closed source?

This is a serious oportunity for Groq to just employ this transparently to the user. They could have a great competitor to frontier models.

Tshirt merchandise : "I am going to revoke these API keys" 😂😅

Hey Matthew - Love your vids - they are tremendously informative.

I love that open source local models are so powerful that they can compete with the giants.

Thanks for the clear and concise instructions. Worked flawlessly first time. Now we just need a UI to work with it, including an artifacts window of course

Great! I will certainly try this out

Awesome instructions video. Thanks a lot!

I wonder how good it will be with Claude 3.5 Sonnet + GPT4o + Gemini 1.5 Pro

Great tutorial thanks so much Matthew!

Hey Matt, remember GPT-Pilot? Apart from revisiting the repo (it's done amazingly well) there is an interesting use case in relation to MoE. Remember how GPT-Pilot calls an API for its requests? Wouldn't it be interesting to see how it performs if it were to call a MoE "API"? It would require to expose the MoE as an API, but it would be very interesting to see as it would enable developers to piece together much cheaper models to achieve great outcomes, the likes of 3.5 sonnet.

Wow 🤯 I really appreciate your work, keep going pro ❤

Stoked you're using Groq! 🙌 The speed is mind-blowing! IMO, the results quality have been useful to me.

Great video Matthew! Very useful. Previously I had problems using crewai and max tokens with groq and will have to check how to resolve it.

I wonder how asking the same model multiple times with different temperatures might work. For instance, you ask the hot head, you ask the cold head, and a medium head integrates this. I think most LLM models would hate it this way, bu it's clearly the future that we will have a unified frameworks with several LLM's asked several time and some model integrating these answers. No single model can compensate for integrating data from several sources.

Thank you. This is brilliant!

Thank you for the inspiration.! I created a version using groq and open-webui as a pipeline.

Sir, you are a legend.

We need to start comparing things to claude 3.5 sonnet too! But I love MoA concept.

Nice went smoothly

this setup is actually worth it.... good video man :)

great !! just great ! thanks!

Fantastic!

Just glancing at your headshot, I thought you were wearing a massive chain, like a wideboy from the Sopranos. Then I realised it's just the way your T-shirt is folded 😂

Hey @matthew_berman , How would I go about having this whole process in a Docker container or have it as an API endpoint to where I can then connect Groq's speech to text, and that returned text is what gets passed to the prompt of MOA? Thanks and love the vids!

This is awesome. What would be even better is getting a GUI setup for this so that it can be used more for coding. I maytry this later.

AMAZING AI NEWS!

groq is an absolute marvel

Thanks much! I might actually give that a try considering you did all the heavy lifting. Seems like the AI orchestrators such as Crew AI or LangChain should be able to do this as well.

Just a thought if you fork the repo then upload your changes we could just download your fork and try it out?

Oh boy, Groq's inference time is truly impressive. As most guys, however, I thought you were talking about Grok. Groq is in fact just mostly Llama on steroids. It's a pitty they cant offer larger models so far. But seeing how Groq works gave a glimpse of the speed of LLM chatbots in a year or two.

Groq is awesome !

What if we use mixture of gaint models like 4o with claude opus and sonnet 3.5 and other state of art models with groq combined

Trying right away!! Thank you very much.. I wonder if I expose my personal mixture of agents to someone else and they use my model as their primary model?? Or thousands of such a model interconnected to each other.. a mesh of a model looped within themselves what will happen??

Any idea if this could be used as a sort of intermediary endpoint? As in I point to a local machine hosting this script as though it were a locally hosted model. If this can perform on par with or better than gpt-4o with this kind of speed then it would be incredible to use as a voice assistant in Home Assistant.

The title of your video is misleading. Your tutorial shows how to setup MoA with Groq, but you haven't demonostrated how it's "Better Than GPT4o AND Fast". Why didnt you test the MoA with your benchmarks?

If Matthew with his insane local machine can’t compete, I am convinced Groq is the way to go for MoA. Super interesting to see how it nails the most difficult task in the rubric and fast! ❤

I've been thinking about this approach also. Very helpful vid. Thanks.🙏🏻. I'm curious if on top of MoA if MCTSr would get the answer closer to 💯? And once SoHu comes online how awesome are those speeds gonna be?!

What is the best local private AI models RAG app?

All of these enhancements could be something you contribute back to the project in the form of configuration. Would make it a lot easier for the next wave of users

Wow. That is good.

How many tokens are used for calling openAI API? Would be wonderful, if you could show how to leave OpenAI out. And full benchmark test please. Thanks Matthew!

Just wondering how can I use this groq setup in Continue?? ]

Matthew, how are you? Groq hosted another one, I inserted it in this version of MoA that you taught us. Now, with 5 LLMs at the same time. The new model is gemma2-9b-it.

Groq? We love local LLM

Good idea, but we need to know the cost comparison as well.

I'm wondering, have any test been done with much smaller models where you can have 2 or 3 running locally on your own hardware to see if it improves quality over any of the 2 or 3 on their own? I ask because with how APU's are developing, dedicating 20-30-40GB or more to A.I. use wouldn't be that big of a deal with how cheap memory is getting.

Can I make an "API" call to this MoA and use it say, with DSPy? Have you ever considered making a tutorial on DSPy?

Need the follow up tho. What is mixture of agents? How can we use it? Do we get to edit the agentic framework and structure ourselves? What possibilities now exist with this tool? I need moooore! (Amazing video and value, I never miss your posts!)

would there be a way to integrate this with a front end like openwebui?

How does this compare when there is RAG with structured and unstructured data???

Can we use OpenAI as the main engine along with Groq?

Hi Matthew, I got an error: Traceback (most recent call last): File "D:\ai\MoA\bot.py", line 1, in import datasets ModuleNotFoundError: No module named 'datasets' But I do installed it

Little tip: conda create -n python= && conda activate

How is this any different than Autogen or CreawAI?

Could you put your version in a github repo?

see now we need that but uncensored

We got Moe, MoA, CoE, is there anything else?

I love that , i hat open ai 😅

Not all of those Apple sentences make sense Apple.

Can you explain how this many layers of models actually know HOW TO produce the best answer possible? How do they know what answer is better or more correct?

The biggest limitation for Groq is API limit. After some use, you cannot use it anymore.

Is there any chance you’re gonna look into the new algorithmic way to produce LLM’s, this is a transformers killer supposedly, I would think that that would be really relevant to viewers.

Excellent explanation of how to write code that uses GROQ as a manager of a mixture of agents. But you just went too fast for me to catch all your changes that make it all work. Could you write this in your newsletter or create a video for the slow, metal programming types? Thanks again. Groq AI is making programming more accessible for non-power users like most of us.

But it cant process pictures?

why use conda and not pip ?

When KZfaqrs has better security than Rabbit R1. He revokes API keys.

What the coast of running this in a one million contect input and output

Groq is amazing, but can't do this setup because I don't have OpenAI API key and I don't use it, let's wait for a way to completely use it without the need for OpenAI at all

Is goq censored?

Quietly this is what Sonnet 3.5 is and the Anthropic secret. That why the API doesn't work well with using so much function calling.

Has anyone created a comparision between MoA and other multi-agent systems that can utilize LLMs? (Like Autogen) Because to me, this looks exactly like an Autogen network with a few simplifications, like no code running, and no tool or non-LLM agent usage. So, if this is not better, or even worse than Autogen, then it might not worth to use it. Since Autogen has a lot more features (like the code running, which was mentioned in the last video). Also, the results compared to a single (but much bigger) LLM, looks kinda obvious to me. Since the last modell receives a lot of proposed outputs, next to the prompt, and it only has to filter the best ones. This task is a lot easier than generating the correct one, for the first time, only from the prompt. And with the base idea behind MoA is this, the results are expectable.

nice AD

Ok 🎉

"4. The old proverb says that eating an apple a day keeps the doctor away apple." 🙃

all about AI ragmodel running uncensored v2w/ MOA using groq MOA LIBRABRY blackfridays gpts library ENOUGH SAID YOUR WELCOME I WOTE MY OWN API KEY ON OPEN GUI i built LOLOLOL

This is cool but can you write some real multi-document code with these things? I don't need to make 10 sentences ending in apple. Most the things I've tried so far can't write code and I struggle to figure out their use.

Interesting but I don't know WTF You're doing but it looks great. Unless there's an easier setup and install, for me right now it's easier to just use chat GPT.

Thats why (according to the small agents vs response-time) Microsoft specially, with all the others have whole datacenters trying to find out how a swarm of millions of small agents can work seamlessly. What else should it be? A giant multi-trillion parameter model with the name Goliath? haha

Brave browser local llm test o.o

Groq? Sounds like Grok doesn’t it?

I tried asking MoA if A=E, B=F, C=G, D=H Then E=? It got it wrong 😂 MoA’s answer:- “I” But it’s amazing 🤩

There's so many advances coming so fast with all of this, I wonder if the real value in your content is the rubric. Or, more accurately, improving your rubric. Right now I think it's barely version 1.0, and it needs to be v5.0. And try adding a medical or legal question for gosh sakes.

This is called Narrow AI.

dude, all your videos talking about beating gpt-4o, and we haven't seen any!

I just read that GROQ no longer wants to sell cards directly for 20K but instead wants to offer a SaaS model ? This seems to contradict the benefits of running LLMs locally for privacy reasons, because now you're sending tokens out to a 3rd party web-service. I don't know why this is an either / or decision. LOTS of SMBs can afford to invest 20K in hardware. Total outlay for a serious LLM rig would have been something like 30K, which is barely half a year's salary to an entry level position. Bad move I say, but the good news is there will be competitors that correct this decision if Groq doesn't, and soon !

Honestly Matthew why do advertise this? It has been months and we still are stuck with their free package with a rate limit that you can bring no app to production yet . Waste of time and integration

First😊

Second

Groq is terrible, worse than GPT-3. Why do you keep chilling for it?

Groq is too filtered/censored... too shameful not courageous.... too weak impotent.....

too technical too much effort unfriendly interface

I finally unsubscribed. Don’t know why it took me so long.

You are gaining weight?

Groq is nowhere near CGPT or Claude... and all of them need a lot of attention and are far away from SAI. There is currently only SUPER-FLAWED and SUPER-OVERRATED.

As usual, too much coding...

@matthew_berman Attention, you didn't revoke keys before publishing this video!

how about? MoA-Openrouter