Ollama

Ollama is an application which lets you run offline large language models locally.

Installation

Install the ollama package, which provides a daemon, command line tool, and CPU inference.

For GPU inference:

Install ollama-vulkan for inference with Vulkan.
Install ollama-cuda for inference with CUDA.
Install ollama-rocm for inference with ROCm.

Tip On AMD GPUs from RDNA 3 onward, ollama-vulkan is usually faster and uses less power than ollama-rocm and avoids ROCm's unsupported-GPU issues (see #ROCm is not utilizing my AMD GPU). Integrated GPUs are skipped automatically; set OLLAMA_IGPU_ENABLE=1 to allow them.

Next, enable/start ollama.service. Then, verify Ollama's status:

$ ollama --version

If it says Warning: could not connect to a running Ollama instance, then the Ollama service has not been run; otherwise, the Ollama service is running and is ready to accept user requests.

Next, verify that you can run models. The following command downloads the latest 270M parameter model of Gemma 3 and returns an Ollama prompt that allows you to talk to the model:

$ ollama run gemma3:270m

>>> Send a message (/? for help)

Usage

The Ollama executable does not provide a search interface. There is no such command as ollama search. To search for a model, you need to visit their search page.

To run a model:

$ ollama run model

To stop a model:

$ ollama stop model

To update a model:

$ ollama pull model

To remove a model:

$ ollama rm model

To view locally available models:

$ ollama list

Configuration

System-wide configuration

Ollama is configured through environment variables. Set them for the service in a drop-in file, then restart ollama.service:

/etc/systemd/system/ollama.service.d/environment.conf

[Service]
Environment="VARIABLE=value"

Commonly tuned variables:

Variable	Explanation
`OLLAMA_FLASH_ATTENTION=1`	Faster prompt processing and smaller key-value cache; prerequisite for key-value cache quantization.
`OLLAMA_KV_CACHE_TYPE`	Key-value cache quantization, requires flash attention: `f16` (default) `q8_0` (half the VRAM of `f16`, negligible quality loss) `q4_0` (half the VRAM of `q8_0`, noticeable quality loss)
`OLLAMA_CONTEXT_LENGTH`	Default context length per request, e.g. `32000`; if unset, it scales with available VRAM (4k/32k/256k). Models can override it (see below).
`OLLAMA_NUM_PARALLEL`	Concurrent requests per model; each slot multiplies the key-value cache allocation.
`OLLAMA_MAX_LOADED_MODELS`	Number of models kept loaded per GPU.
`OLLAMA_KEEP_ALIVE`	How long a model stays in memory after the last request (default: `5m`); `0` unloads immediately, `-1` keeps it loaded forever).
`GGML_VK_VISIBLE_DEVICES=0`	On systems with multiple GPUs (e.g. discrete + integrated), pin inference to one Vulkan device (`0` = first in the startup log).
`OLLAMA_HOST=host:port`	Listen on all interfaces instead of loopback only; required for containers (e.g. Podman/Docker reaching the host via host.containers.internal) and other LAN hosts.
`OLLAMA_IGPU_ENABLE=true`	Use the integrated GPU for inference. By default they are ignored.

See ollama serve --help and the upstream FAQ for the remaining variables.

Note Vulkan support is enabled by default when ollama-vulkan is installed; OLLAMA_VULKAN=1 is not required.

Warning The Ollama API has no authentication. With OLLAMA_HOST=0.0.0.0, anyone on the network can use the models - restrict access with a firewall if the network is not trusted.

Per-model parameters

To run a model with custom settings, bake them into a new model with a Modelfile:

Modelfile

FROM gemma4:26b
PARAMETER num_gpu 29
PARAMETER num_ctx 32768

Directive	Explanation
`FROM`	An already pulled model, as listed by `ollama list`.
`PARAMETER num_gpu`	Number of model layers offloaded to the GPU. Speed rises with each extra layer until VRAM fills. One layer too many spills into GTT (system RAM over PCIe) and speed collapses, even though the model still reports "100% GPU". Benchmark a few values - the optimum is the most layers that fit just under free VRAM. For example, on a 16 GiB RX 9070 XT running `gemma4:26b` (Q4_K_M) at 32k context, 29 of 31 layers hit ~88 tok/s with most of the VRAM available; 30 layers dropped to ~75 tok/s (spill) and the auto-offload default gave ~37. With a browser holding ~5 GiB of VRAM the optimum fell to ~24 layers. On an 8 GiB GPU only ~13 layers of this model fit and generation is mostly CPU-bound - there, prefer a model that fits in VRAM entirely (a 7-9B model at Q4_K_M takes ~4-5 GiB) at 8-16k context.
`PARAMETER num_ctx`	Context size: the maximum number of tokens the model sees at once (prompt + response). The key-value cache shares VRAM with the layers: the larger the context, the fewer layers fit - retune `num_gpu` after changing it.

Then, create the model with:

$ ollama create gemma4-tuned -f Modelfile

ollama create does not copy weights - the store is content-addressed, so a tuned variant only adds a small manifest. The same parameters can also be passed per request in the API options field, which overrides Modelfile values.

Ollama does not keep the source Modelfile; the parameters are stored as layers in the model store (/var/lib/ollama, manifest under manifests/registry.ollama.ai/library/name/tag). To change a created model, reconstruct its Modelfile, edit it and run ollama create again with the same name (only the small manifest is rewritten):

$ ollama show --modelfile gemma4-tuned > Modelfile
$ ollama create gemma4-tuned -f Modelfile

See the Modelfile reference for the remaining instructions and parameters.

Troubleshooting

ROCm is not utilizing my AMD GPU

You may have used utilities like amdgpu_top to monitor the utilization of your GPU during an Ollama session, but only to notice that your GPU has not been used at all.

Without configuration, ROCm simply ignores unsupported GPUs, causing everything to be computed on CPU.

Note Verify supported GPUs by consulting ROCm System Requirements.

To work this around, create a drop-in file for ollama.service:

/etc/systemd/system/ollama.service.d/override_gfx_version.conf

[Service]
Environment="HSA_OVERRIDE_GFX_VERSION=X.Y.Z"

Where X.Y.Z is dependent to the GFX version that is shipped with your system.

To determine which GFX version to use, first make sure rocminfo has already been installed. It should be pulled into your system as an indirect dependency of rocblas, which is required by hipblas, which is required by ollama-rocm.

Next, query the actual GFX version of your system:

$ /opt/rocm/bin/rocminfo | grep amdhsa

You need to remember the digits printed after the word gfx, because this is the actual GFX version of your system. The digits are interpreted as follows:

If the digits are 4-digit, they are interpreted as XX.Y.Z, where the first two digits are interpreted as the X part.
If the digits are 3-digit, they are interpreted as X.Y.Z.

Then, find all installed rocblas kernels:

$ find /opt/rocm/lib/rocblas/library -name 'Kernels.so-*'

You need to set X.Y.Z to one of the available versions listed there. The rules are summarized as follows:

For the X part, it must be strictly equal to the actual version.
For the Y part, mismatch is allowed, but it must be no greater than the actual version.
For the Z part, mismatch is allowed, but it must be no greater than the actual version.

After setting the correct X.Y.Z, perform a daemon-reload and restart ollama.service.

Then, run your model as usual. You may wish to monitor GPU utilization with amdgpu_top again.

Models are not removed after uninstalling Ollama

You can manually remove the model files. They are stored in /var/lib/ollama/blobs.