Using Ollama with Quadlets!

There are numerous posts around the internet about getting Ollama running to do local AI, but not many seem to know about the convenience of Quadlets! With Docker images and Podman, you can create what are known as Quadlet files that will do all the leg work of running things with the bonus of containerizing. I use Fedora Silverblue, which is a container-focused OS, and Podman is readily available to us Fedora Atomic users. With my AMD GPU, I’m using the :rocm version of the Ollama image. A lot of this post is AMD/ROCm centric, so adapt as necessary. Start off by putting this file in ~/.config/containers/systemd/ollama.container to get Ollama set up.

[Unit]
Description=Ollama Container with ROCm Support

[Container]
ContainerName=ollama
Image=docker.io/ollama/ollama:rocm
AutoUpdate=registry
AddDevice=/dev/kfd
AddDevice=/dev/dri
Volume=ollama:/root/.ollama
PublishPort=127.0.0.1:11434:11434

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=default.target

This will run the ROCm-variant Ollama image with Podman! For reference, instructions for getting Ollama going with Docker (and equivalently Podman) are here at the Ollama Docker Hub page, which lays out the particulars of the Docker method for Ollama. To get this up and running, a few commands are necessary though…

grep '^video:' /usr/lib/group | sed "s/\$/$USER/" | sudo tee --append /etc/group
grep '^render:' /usr/lib/group | sed "s/\$/$USER/" | sudo tee --append /etc/group
sudo setsebool -P container_use_devices 1
systemctl --user daemon-reload
systemctl --user start ollama.service

Using ROCm with Ollama requires your user to be a member of the video and render groups. Sadly on Fedora Atomic, adding group membership is not well implemented yet, so it goes like this. If you’re using something else, traditional usermod is all of course. If you’re not already a member of these groups yet and this step is necessary, don’t forget to log out and back in.

Next, if you’re using SELinux on your platform, you’ll need to allow the Ollama container to use the /dev/kfd and /dev/dri device files for ROCm with setsebool.

Then, just reload systemd and Podman will create an ollama.service systemd unit for you. Turn it on, and you now have the Ollama image running as a systemd service. curl http://127.0.0.1:11434 should confirm this. You can pull models like so…

podman exec ollama ollama pull <model>

Want to get Open WebUI running? Next Quadlet (put it in ~/.config/containers/systemd/open-webui.container)…

[Unit]
Description=Open WebUI Container
After=ollama.container
Requires=ollama.container

[Container]
ContainerName=open-webui
Image=ghcr.io/open-webui/open-webui:main
AutoUpdate=registry
Volume=open-webui:/app/backend/data
PodmanArgs=--network pasta:-T,11434
Environment=OLLAMA_BASE_URL=http://127.0.0.1:11434
PublishPort=127.0.0.1:8080:8080

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=default.target

And of course start it with…

systemctl --user daemon-reload
systemctl --user start open-webui.service

You might be curious about the PodmanArgs= line in this Quadlet. Without this, the Open WebUI container will be unable to find Ollama. In these containers, the loopback does not refer to the host but only the container itself. This line grants TCP port mapping from the container to the host on the Ollama port, which allows the Open WebUI container to see the Ollama socket listening on the host. You could also set up a network bridge with Podman to get the containers to talk with each other, but that is overkill. I would add that this method is rather efficient in terms of overhead and latency. If you’re lazy, you can do a --network host-type setup, and instead of the TCP port mapping add Network=host to your Quadlet. Finally for those curious, mapping over the IPv6 loopback hasn’t been working for me, so until that’s fixed just stick with good old IPv4.

Now let’s top it off with auto-updates.

systemctl --user enable --now podman-auto-update.timer

That’s it! Podman will now automatically update your Ollama and Open WebUI images at some random time a little after midnight each day. I’ve seen people say enable podman-auto-update.service, but don’t do that. That service is there just to be activated by the timer, and turning on the service unit does nothing. If you want to get a glance at your Podman auto-update history, look at journalctl --user -u podman-auto-update.service.

Enjoy your containerized local AI inferencing!

EDIT (2025-08-25): Quadlets don’t require enabling, just start them. :)