I would recommend adding 'Langchain' to the title of the video, most of this is very langchain specific, for those specifically searching for that.

Thank u sharing video of ollama with phi3 to run locally, hope u would come up wid more such videos to use ollama locally for different tasks. Pls mk more videos on phi3, llama3 with ollama.

Excellent as usual! For Phi3 3.8b latest it works fine with: prompt_phi = PromptTemplate.from_template( """{context} Human: {question} AI:""" ) Otherwise you will get validation errors. All the best Sam!

Thank you for doing this with Ollama, this was an really good explanation and helped me a lot!

Ah really needed this, I kept feeling, I want to learn function calling with llama3. Feels so good to use a local model with function calling, and langchain made it really easy to do. Love to experiment with it now, thank you so much for this video❤️❤️❤️ and thanks to langchain for making it easy to do function calling ❤️❤️❤️

Amazing video 🙏🏻 Currently using crewai

For local models I’ve found it’s helpful to at extra context at the very end of the prompt, in the assistant reply section (not the instruction section), kicking things off with “Sure, here is your JSON:” and then adding markdown syntax for preformatted text and then letting one of the end symbols be the final three backticks to close the markdown. It’s also helpful to write a custom grammar (like with llama.cpp) to constrain output to a specific schema even. (Depending on your setup this could slowdown inference if the constrained generation part isn’t running on the GPU.)

Thank you so much. Super helpful.

Very cool! I've been using the instructor library with pydantic for structured output and had a lot of success on openai models, but it didn't work very well with local llms. I'll definitely try out your approach!

Very useful, thanks!

I see you use a mac mini, could you talk more about what model and OS setup? Thinking of fun things to do with my 2011 2ghz i7 16gb ddr3 ram, a local something on my network if I could.

Thank you for the informative videos as always. One note: if you want to run things all locally and want a lot better throughput, running the models using vLLM and serving the API with vLLM's OpenAI-compatible server is definitely the way to go. If you have a 24 GB VRAM GPU like a 3090 or 4090, you can run a GPTQ or AWQ quantized model, or just the full FP16 model and serve a large number of concurrent clients. With batching, you can get thousands of tokens per second in aggregate for responses if you run a lot of parallel clients.

Thanks for the video! I'd like to see an example of using DSPy to optimize a local model so that it can use tools more reliably. I'm actually not sure if this would work but I'd like to find out. 😃

The biggest issue with function calling is that the way everyone suggests to use it is not very viable / economical, if you want your model to choose one out of many functions to call. I'll elaborate: in order for LLM to pick a function to use, you need to announce all those tools in advance and make sure it hasn't forgotten them, if you're going into multy turn chat. This means more context will be used just to make model aware about all these extra tools you want it to use and less context will be available for responses. There's probably some semantic router needs to be introduced in-between to give model only those tools which might be relevant to current question

Thanks for the code and the explanation. In order to be usable, you should be able to execute the function feed the info back into the history of the conversation with the result of the function and then the llm should be able to use the results from the function to write the last message. For instance, lets say that the weather tool responds with just the temperature and nothing else, then the LLM should be able to respond back 'in Singapore the current temperature is ..." and in the same language as it was asked from the user.

Great video as always! In future videos, could you please show how to do this with Ollama and langchain running on separate computers? I'd like to develop on Laptop or Colab with just inference running on my Desktop PC. And since Ollama doesnt currently do API keys, how do we secure the inference server and access it from a Colab notebook? Thanks!!

would love if you can explain this using the ollama python package. As someone else said this is very specific to langchain and i just cant find good information on how to use function calling with ollama.

Great video. I was hoping this would give me a reason to try LangChain vs my own prompt/post-parsing for a web ui, but I'm actually getting better results than this demonstrates. I'm using llama3-8B via LM Studio. I think until these guys get their sh*t together and create a standard for output, this is going to be similar to the browser wars (standards). At the very least, they should all conform to current markdown standards or accept a config/spec for default output. Whoever comes out with an open source competitive model that does this is going to be the clear leader... for me anyway. ...And if such a model exists, please point me to it!! :)

I have found Phi-3 truly impressive for its size, getting good results even for general inquiries. I almost wonder if you could just use Phi-3 if you don't need a super refined response. It's so light on resources comparatively for an LLM.

Hi< Is that because we try to give it as much more accurate and better machine-readable input, so the model does not have to 'think' too much that it can follow the correct format like JSON and some basic function, and it can meet some complex requirements also. The way is more efficient and energy-saving.

Phi3 excels at creative language tasks, surpassing even GPT-4 in my tests. GPT-4 itself ranks Phi3's lyrics higher :)

Thanks for sharing this, how can I use this json output funcution call format to combine the langchian agent functuion call framework , which. Use the llm.blind_tool to replace the llm=ChatOpenAI()? Will this work? Thanks

wow Sam!!!, this video is really helpful but i am facing challenge in running it on server as the response is not coming within 1 min and i am getting 504 Gateway Timeout error, i have used ollama docker image to install ollama but i am not able to find how to increase gateway timeout to 10 mins instead of default 1 min. Can you please help if you have faced such issue?

all looked well and good until you try feeding a question into the 'agent' that doesn't relate directly to: "get the current weather in a given location". I thought the whole point of function calling/tooling was to present the LLM with the opportunity to use tooling if necessary.

How to use chatOllama along with function calling. i want to pass messages along with functions same as open ai v1/chat/completions api provides.

How do I incorporate function calling with follow up questions and memory. Say a user asks “what is the weather”. The model should be able ask “what place are you requesting for” and say the user replies “California” It should then make the function call with the mentioned arguments. Please let me know which direction I should look in order to achieve this.

Is possible to fine tune a small language model for function call? For example, if we look to BERT models that perform zero-shot classification we can pass a set of labels to it, so maybe is possible to use a similar approach to get a very performatic model just for function calling, since LLMs are very huge and almost every time requires a GPU. I know that phi3 is very small but in my machine it takes like 3Gb of GPU.

I have been running gpt-pilot with Llama3-70b-instruct.Q5_K_M for a couple of weeks. The biggest problem I have, as far as I understand, is not function calling but rather the stability of the framework. It starts developing a bunch of files, but when I provide feedback, it may abandon the old files instead of correcting them, and starts creating a new set of files. Basically makes a mess.

Hey, thats very helpful to understand how to run these models locally. Can u/anyone tell, how to actually do actual function call and pass that response to llm? Is it possible without LangGraph??? I want llm to decide which tool to call, once he decide that, llm should do entity extraction and then invoke tool, then returns ans back to llm and gives it to user. This was easy with AgentExecutor in OpenAI examples. Similar thing possible in Ollama?

What was the name of the paper that shifts the probabilities to get json as response more likely?

At last someone finds a good use for agents - to give them some tasks you want accomplished and give loose it free overnight to use internet :)

Can I use function calling with llama.cpp?

Well, actually, where are the functions? I only see a Json string.

has someone tried to load the models other than using ollama like the huggingface transformer pipeline or in other words I would love to know how torun these models in Linux based servers like databricks where I am unable run ollama application in the background like in my windows PC?

You are not using latest version. It’s now called “bind” not bind_tools

very cool, but this is kind of useless unless you can mix text responses and function calling with the same prompt

I hope you to make react agent tutorial with ollamafunction..!