someone just commented about finding another way to upgrade the container. I can't find the comment now, so if this was you, post again. But no, do not upgrade the install inside a container, that’s a whole lot of work for no benefit. The models are stored in the volume you mounted as part of the install, so deleting the image will not affect the models. If you have gone against the recommendations and stored models inside the container, then best approach is to move them to the correct spot and update the container.

I really wish more videos were made like this. No nonsense, gets straight to the point, clear, concise. Thank you.

Matt your a great teacher, no one explains things like you do. They just read the command in one sentence and do not explain the actual function of that command in parts. Lots of videos showing how to do something and 75% never work. So thanks so much!

Thanks Matt this helped me understand the Docker side of things. Namely keeping the models in a volume. I will restructure my project based on this. Keep it up ❤

thank you Sir! You just took the mystery of how to set this up right. I love me some docker. It really helps to keep the work stuff separated from the personal project stuff.

Does it make sense to use ollama in production as a server?

much respect for the way you deliver information

Straight to the point, no fluff, very informative. Very updated. You just earned a fan/subscriber. Howdy Matt 🎩

Learned that containers can be remote and the alias. Yet another great video! I need to take advantage of that. I have a bunch of RPI security cameras and remote containers might make administration even easier!

This is my new favorite channel! I learned like 10 things just in this video. I love learning about AI, modern tools such as docker and tailscale, and modern hosting platforms and services. Thank you!

In the realm of ones and zeros and LLM models, Matt is the undisputed sovereign.

Hi Matt, your videos are super useful and right on point. Thank you putting this together. I have a quick ques on this topic. I have created a RAG streamlit app in python using Ollama llama3 and ChormaDB. The app runs fine on my Mac localhost but I wanted to create a docker image of this app. I am unable to figure out how to include Ollama llama3 in my docker image. Can you help point to any resources which can guide me on this or cover this in one of the topics? Again,thanks a mil for the content. Great stuff!!! Cheers

This is my new favorite content, the way you explain it just beams directly into my brain and i get it right away. Thank you. Is there a way to show support, donations or similar?

There are not many people that can explain these steps in such an easy and entertaining way as you do Matt. I often pride myself in being able to do so, but you can be my teacher. I often find myself watching the progress-bar beause I don't want it to end (seriously :-))! A request: could you do an explainer video on how to train a model (say Microsoft/Phi-2) on your own dataset and deploy the trained model? OpenAI makes it super easy by deploying a JSONL file and after a while it 'returns' the trained model. But I want to train my own models. I have been looking around YT but get lost in parameters, incorrect JSONL-files (or csv)., etc. Surely, this must be easier. (hopefull your answer is "it is easier, and don't call me Shirley") Thanks so much again. You have a happy subscriber (and many more to come)> Kind regards, Bas

Hey @Matt Williams! Thanks for the great content! Am I crazy, or did you do a version of this somewhere where you added a file to /etc/systemd/system that called up docker at startup and ran ollama that way? i.e. letting systemd handle the startup and restarting of the container as a service?? If not, would you?! Please and thank you!!?? 😅

Thanks, Matt! It was a straightforward tutorial.

Ive been running a lot via Docker but when I found out about the difficulty of GPU pass-through (on any machine) I have been swapping things over to proxmox which does have a GPU pass-through *and* can also use CPU to emulate GPU as it is needed ... what do you think about running on Proxmox?

A fantastic, clear instructional. Thank you so much! This helped me a ton.

How you speak in your videos is refreshing thank you 🙏🏻

what an amazing video! Thank you lots for providing for the tech community

how do you swap models in the same container? I think I'm doing it wrong and it's affecting my container memory

This is exactly what I've been looking for! Could you please tell (or maybe create a video) how to use ollama completeley offline? I have a PC that I can not connect to the internet.

Could you kindly assist me in clarifying how to specify the model name when running the ollama Docker command? For instance, I aim to utilize the mistral and llama2:13b models in my project. Thus, I request our dev-ops team to launch an ollama container configured with these specific models.

Does it make possible to limit resource consumption of ollama? I'm looking for some way to run a background computation and I don't really care about how much time it takes (if it is able to process a stream's avg load), but it would be annoying if it would be hanging the main activity on the machine.

🔥good stuff as always Matt!

So, I think you cleared up most of the problems that i have been having trying to get this setup. But i have one last one that i just cant seem to get past. So, my setup is on proxmox and i first tried to create a lxc container then once i have my NVidia passthrough working for my gpu i installed ollama and downloaded my first model. That all went fine then i tried to see if the api was listening on port 11434 by opening a browser and going to the address:11434. according to the documentation i should get a message that ollama is ready. unfortunately, i get no errors the page simply doesn't open. So i approached it from the other side and just created a lxc and installed docker and portainer on it. much to my surprise when i navigated to the address i got the message ollama is ready. My questions is why? im sure this is something easy that im missing but 24 hours later i am still not sure why. any ideas?

Excellent, Matt! For some reason, I had to run docker commands with "sudo" , to use my GPUs.

Excellent content.

Searching for a full airgap install on docker to use on Kubernetes. This is a start ^^. Thx

Nice and clear tutorial. Thanks! 😀

Great video. Now i'm curious about how you setup ollama on brev ... What is your recommended setup & host service for using Ollama as an endpoint?

hi Matt, think is an excellent video, thank you. I have an app built with ollama, How do I containerize the entire thing with Docker? My aim is to push this image online and share around. Thank you in advance.

Great video as always Matt, I love them. I would like to know how to load a custom model in docker with a model file. Thank you so much.

Hi Matt, thank you for such a clear an concise explanation!! I have a question that may or may not apply in this context, and I'll let you be the jugde of it. I'm running on CPU on an 8 virtual core server with 30Gb RAM and NVme disk on ubuntu 22.04, and the performance is kind of poor (and I clearly understand that GPU will be the straightforward way to solve this). But I've noticed that when I run the models, for example Mistral 7b, ollama only uses about half the CPUs available and less than 1 Gb of RAM. I'm not sure why it is not using all the resources available, or if using them will improve the performace. Anyway it would be great to have your advice on this, and if it is something that can be improved/configured how would you suggest to do it? Thank you very much!!!

I would like to suggest the ollamas support to embeddings, when it becomes available through REST API. If they really choosed the nomic-ai/nomic-embed-text-v1.5-GGUF, it would be perfect as this model is multi-language

Any idea where I can see docker logs for local install (i.e. not docker) on mac....

Kindly make a tutorial to fine tune an open source LLM model on many pdfs data. The fine tuned LLM must be able to answer the questions from the pdfs accurately.

thanks for your help!

Does it consume less resources and run better with OrbStack instead of with Docker Desktop?

Cool tutorial. Can you also show how we can integrate ollama docker with other programs, say, langchain script inside docker. How to connect both of them together or separately? Thanks!

Nice video. Could you also please make a demo video on how to use ollama via nix (nix shell or on nixos)?

Can I run Ollama without having access to internet? I would like to run it locally, with no internet connection at all.

you're amazing bro

Make you Ollama even better by installing Open WebUI in a second container. This even runs on my Raspi5!

Hi bro. When try to deploy ollama on awa lambda with ecr docker image I am getting error can you please help me Error:http:connecterror:[errno 111] connection refused Thank you

I recently had the opportunity to try Ollama in docker and it worked pretty much as shown in this video. I do think it would be nice if it was somehow possible to start a container and have it ready to serve a model immediately but i couldn't find an easy way to do this. You basically have to run one docker command to start Ollama, then wait a bit, then run another docker "it" command to tell Ollama to load whatever model you happen to need. How do i achieve the same thing using just one single docker command?

I am wondering where I am going wrong. I followed step-by-step but when I run ollama run llama3 I get 'ollama: command not found'

did you hear about jan ai? Would be got a Tutorial for docker. Thanks

hey matt, appreciate your content - has been very helpful to get everything running so far! I am on a windows 11 pc and managed to get ollama + anythingllm running on docker and communicate w/ each other. Now I want to try to get llms from hugging face to run in the dockerized ollama. I saw how it works, if you have ollama installed directly on the system. But how do I approach this with using docker? Thanks in advance and keep it up 👏

is it possible to manage multiple instances of ollama on docker for scaling the ollamas for production? how ?

Running the docker container as ROOT user is not secure,is there anyway to run it as non root user

Juat dound this channel. Coould you make a video tutorial on how to use inside vscode for code competions?

I wish there was a clean way to launch an Ollama docker container with a preconfigured set of models so it would serve and then immediately pull the models. We are overriding the image’s entry point right now to run a script shell that does this…

would ollama still work on a machine with no graphics card?

some of the model links r broken so I had to add it to requirements and edit the Dockerfile

In wsl2 docker version DONT PUT MODELS OUTSIDE WSL2 on mounted windows drive - I/O performance will be x15 times slower.

For Windows, the recommended way would be to use WSL(2), since that's a container in itself

Hi, would you also share podman commands ? Did you give it a try ?

how to connect this ollama server with a streamlit app and run both on docker

Loved the video…can you do something similar for LocalAI. Thanks!

It's better to write a docker file and package it as docker image

Hey Guys.... Mistral just launched their new model named Large!

If anyone has insights on a particular LLM model that has low halucination rate on Kubernetes native resource generation, please leave me a comment ;-). Thx

How to run as a dockerfile

Mac not allowing GPU pass through is a huge limitation

VID SUGGESTION ~ (Resolve Error Response: Invalid Volume Specification) - Thanks test@xz97:~$ docker run -d --gpus=all -v /home/test/models/:root/.ollama -p 11434:11434 --name ollama ollama/ollama docker: Error response from daemon: invalid volume specification: '/home/test/_models/:root/.ollama': invalid mount config for type "bind": invalid mount path: 'root/.ollama' mount path must be absolute.

waaiiiit wait wait a sec... i specifically remember in a vid (don't remember which, it's been months) that on Mac in order for ollama to utilize "metal" 3d acceleration that it needs to run in docker... strange 🫤