# Set Up Ollama

[Ollama](https://ollama.com) runs a language model as a local service on port `11434`. It is the default provider for `splam`'s [Chat panel](01.09-how-to-chatbot.md), and the reason that panel can be the default at all: nothing you type and nothing the model reads leaves the host.

This page covers Ubuntu 24.04 and the distributions built on it, including Pop!\_OS. Commands assume `sudo` and a `systemd` init, which is what you already need for `splam` itself.


# Check the machine can run it

A local model is the one part of this setup with hardware requirements. Three commands tell you where you stand:

``` bash
free -h                     # RAM: the "available" column, not "total"
df -h /usr                  # disk: models land under /usr/share/ollama
lspci | grep -iE "vga|3d"   # graphics
```

Size the model against available RAM. A working rule is that you want roughly twice the download size free while the model is loaded, leaving room for the context window.

| Model           | Download     | Comfortable in |
|-----------------|--------------|----------------|
| `llama3.2` (3B) | about 2 GB   | 8 GB RAM       |
| `llama3.1` (8B) | about 4.7 GB | 16 GB RAM      |
| `qwen2.5:14b`   | about 9 GB   | 32 GB RAM      |

Graphics decides speed, not whether it works. Ollama accelerates on NVIDIA through CUDA and on AMD through ROCm. Everything else, including Intel integrated graphics and the NPU on recent Intel laptop chips, runs on the CPU:

``` bash
nvidia-smi                  # NVIDIA present if this prints a table
ls /dev/kfd                 # AMD ROCm present if this exists
lscpu | grep -o avx2        # CPU path wants AVX2, which any recent chip has
```

Neither of the first two existing means CPU inference, which is supported and correct, just slower. On CPU prefer the 3B model. The 8B models answer better but you wait long enough per reply that the panel stops getting used.


# Install it

The official script is the supported path on Ubuntu. There is no `apt` repository:

``` bash
curl -fsSL https://ollama.com/install.sh | sh
```

Read it first if that matters to you, and it should on a regulated host:

``` bash
curl -fsSL https://ollama.com/install.sh | less
```

The script needs root and makes four changes worth knowing about before you run it on a machine you have to account for:

| Change | Where |
|----|----|
| The `ollama` binary | `/usr/local/bin/ollama` |
| A system user and group named `ollama` | `/etc/passwd`, `/etc/group` |
| A `systemd` unit enabled at boot | `/etc/systemd/system/ollama.service` |
| Downloaded models | `/usr/share/ollama/.ollama/models` |

That last one is why the disk check above looks at `/usr` rather than `$HOME`. Models are owned by the `ollama` user, not by you.


# Confirm the service is up

The unit is started and enabled by the installer:

``` bash
systemctl status ollama
```

Ask the service itself, which is the check that matters:

``` bash
curl http://127.0.0.1:11434
```

It answers `Ollama is running`. If the unit is masked or you would rather not have a boot service, `ollama serve` runs the same server in the foreground and the rest of this page is unchanged.

It binds to `127.0.0.1` by default, so nothing off the host can reach it. Leave it that way. If some other tool has already claimed `11434`, find it with `ss -tlnp | grep 11434` before changing anything.


# Pull a model

`llama3.2` is what `splam` asks for unless told otherwise:

``` bash
ollama pull llama3.2
```

Confirm what you have, and check the size against what you measured:

``` bash
ollama list
```

Then talk to it directly, before involving `splam` at all. This separates a model problem from an app problem:

``` bash
ollama run llama3.2 "reply with the single word: ready"
```

An answer here means the service, the model, and the hardware are all fine, and anything that goes wrong afterward is configuration.


# Point splam at it

Nothing to configure. `ollama` is already the default provider and `llama3.2` the default model, so with the `chat` extra installed the panel finds it:

``` bash
.venv/bin/pip install -e ".[chat]"
shiny run splam.app:app
```

Open the **Chat** tab and ask something only your own admin-task notes can answer, such as "what's the compliance note for this service?". An answer in your own wording proves the whole path. See [How-To: Chatbot](01.09-how-to-chatbot.md) for what the panel can and cannot do once it is talking.

To use a different local model, pull it and name it:

``` bash
ollama pull qwen2.5:14b
SPLAM_CHAT_MODEL=qwen2.5:14b shiny run splam.app:app
```


# Keep an eye on disk

Models accumulate. Every `pull` is a few GB and nothing removes them:

``` bash
du -sh /usr/share/ollama/.ollama/models
```

``` bash
ollama rm llama3.1
```

To keep them somewhere other than `/usr`, override the service environment rather than editing the unit file, which an update will overwrite:

``` bash
sudo systemctl edit ollama
```

Add:

``` ini
[Service]
Environment="OLLAMA_MODELS=/srv/ollama/models"
```

Then create the directory, give it to the `ollama` user, and restart:

``` bash
sudo mkdir -p /srv/ollama/models
```

``` bash
sudo chown -R ollama:ollama /srv/ollama
```

``` bash
sudo systemctl restart ollama
```


# Update it

Re-running the install script upgrades in place and keeps your models:

``` bash
curl -fsSL https://ollama.com/install.sh | sh
```

``` bash
systemctl restart ollama
```

``` bash
ollama --version
```


# Remove it

Uninstalling is four steps, and none of them is `apt remove`:

``` bash
sudo systemctl disable --now ollama
```

``` bash
sudo rm /etc/systemd/system/ollama.service /usr/local/bin/ollama
```

``` bash
sudo rm -r /usr/share/ollama
```

``` bash
sudo userdel ollama && sudo groupdel ollama
```

With the service gone, `splam` shows the same setup help it showed before the install. The app keeps working; only the **Chat** tab changes. To take the chat code path out entirely, uninstall the extra instead, as described in [Turn it off for everyone](01.09-how-to-chatbot.md#turn-it-off-for-everyone).


# When it doesn't work

| What you see | What it means |
|----|----|
| `Can't find locally running ollama.` in the Chat tab | The service isn't running. `systemctl start ollama` |
| `Unit ollama.service could not be found` | The install script didn't finish. Re-run it and read the output |
| `model "llama3.2" not found` | Installed but nothing pulled. `ollama pull llama3.2` |
| `ollama: command not found` after installing | `/usr/local/bin` isn't on this shell's `PATH`. Start a new shell |
| Replies take minutes | CPU inference on too large a model. Pull `llama3.2` and set `SPLAM_CHAT_MODEL` |
| The service dies partway through a reply | Out of memory. Check `journalctl -u ollama -n 50` for the OOM kill, then use a smaller model |

The service writes to the journal like anything else on the box, so the same **Logs** tab you use for the rest of the system works here:

``` bash
journalctl -u ollama -n 100 --no-pager
```


# Further reading

- [Ollama's model library](https://ollama.com/library), for models beyond the three named here

- [How-To: Chatbot](01.09-how-to-chatbot.md), for the panel this feeds

- [Explanation: Chatbot Scope](02.04-explanation-chatbot-scope.md), for why local is the default