> ## Documentation Index
> Fetch the complete documentation index at: https://bastani-feat-pi-model-runtime-parity.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Llama cpp

# llama.cpp

Atomic supports the [llama.cpp](https://github.com/ggml-org/llama.cpp) router server. The router discovers multiple GGUF models and loads or unloads them on demand.

Use a current llama.cpp build with router support. Follow its [build instructions](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md) or install a [prebuilt release](https://github.com/ggml-org/llama.cpp/releases).

## Start the router

Start `llama-server` without `--model` or `-m`; those options start single-model mode instead of router mode.

```bash theme={null}
llama-server \
  --models-dir ~/models \
  --no-models-autoload \
  --jinja \
  --host 127.0.0.1 \
  --port 8080 \
  -ngl 999 \
  -c 32768
```

* `--models-dir` discovers local GGUF files.
* `--no-models-autoload` leaves loading under explicit `/llama` control.
* `--jinja` enables compatible chat templates and tool calling.
* `-ngl 999` offloads as many layers as possible to the GPU.
* `-c 32768` sets each model's context window. Omit it to use the model's native context, which may require substantially more memory.

Single-file models can sit directly in the model directory. Put multimodal and multi-shard models in separate subdirectories with their projection or shard files. Restart the router after manually adding files. Per-model context sizes and other options can be set with [model presets](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md#model-presets).

## Configure Atomic

Run:

```text theme={null}
/login llama.cpp
```

Enter the router URL and optional API key. The default URL is `http://127.0.0.1:8080`. The same values can be supplied without `/login`:

```bash theme={null}
export LLAMA_BASE_URL=http://127.0.0.1:8080
export LLAMA_API_KEY=optional-secret
atomic
```

If the server requires a key, start `llama-server` with the matching `--api-key`. Keep `--host 127.0.0.1` for local-only access.

## Manage models

Run `/llama` in interactive mode:

* Select an unloaded model to load it, or a loaded model to unload it.
* Select **Download model…**, search Hugging Face, then choose a repository and quantization. Exact `owner/repository[:quant]` values also work.
* Press Escape during a load or download to confirm cancellation.

Hugging Face search uses `HF_TOKEN` when set, then checks `$HF_TOKEN_PATH`, `$HF_HOME/token`, `$XDG_CACHE_HOME/huggingface/token`, and `~/.cache/huggingface/token`. Unauthenticated search has lower rate limits. Atomic warns before gated downloads and links to the access page. Because llama.cpp performs the download, its process must also have `HF_TOKEN` for gated repositories.

Atomic asks before unloading other models, never silently unloads models, and never deletes model files. `/llama` always displays the router's current state because other clients may share it. Only loaded models appear in `/model`; load one first, then select it there. Atomic persists the last successful loaded-model catalog in `~/.atomic/agent/models-store.json` (or the active custom agent directory), so those entries remain selectable after restart and before the first successful refresh. If that first online refresh fails, Atomic reports the original router error while continuing to expose the validated persisted catalog; a later successful refresh replaces those stale loaded-state entries without duplicates. If the router disconnects, choose **Retry** to reconnect and refresh state without replaying the interrupted operation.

Each llama model uses the router-reported loaded context (`meta.n_ctx`, then training context, otherwise Atomic's fallback) for both `contextWindow` and `maxTokens`; Atomic no longer applies a separate 16K output cap. The server remains authoritative and may impose a smaller practical generation limit.

## Troubleshooting

```bash theme={null}
curl http://127.0.0.1:8080/health
curl http://127.0.0.1:8080/models
```

* **No models in `/llama`:** Check `--models-dir`, the directory layout, and restart the router.
* **Model missing from `/model`:** Load it with `/llama` first.
* **Load fails or uses too much memory:** Lower `-c` or unload another model.
* **Server is not in router mode:** Start it without `--model`, `-m`, or `-hf`.