> ## Content Index > Fetch the complete content index at: https://computefit.dev/llms.txt > Use this file to discover other available public pages before exploring further. # Ollama vs llama-server: Which Runtime Should You Actually Run Your Local Models Through? - URL: https://computefit.dev/ollama-vs-llama-server-runtime-decision/ - Published: 2026-08-24T07:15:37.000Z - Updated: 2026-08-24T07:42:27.000Z - Description: Same engine, two different operating models. Ollama defaults to registry pulls and a VRAM-tiered context guess; llama-server asks you to set the window yourself and exposes layer split, KV-cache dtype, metrics, and API keys. A decision guide for both. - Author: Alex Vale - Tags: Local AI, Ollama, llama.cpp ## The short version Ollama and `llama-server` do roughly the same job - they load quantized model weights on your own hardware and answer chat, completion, and embedding requests over HTTP. They package that job differently. Ollama is a complete model-management and serving app with a registry you pull from; llama.cpp's server is a plain C/C++ program that you point at weight files (or a Hugging Face repo) and configure yourself. The choice between them comes down to three things: how you get models, which memory defaults you accept out of the box, and how much control you want over offloading, KV-cache format, concurrency, and observability. - If one person is using the machine and you mostly want working models fast, start with Ollama. - If you need explicit control - a fixed context budget, quantized KV cache, exact layer offload split, metrics endpoint - run `llama-server`. - Either way, switching is cheap: both consume the same `.gguf` files. The decision below costs nothing later. ## A framing note first: these are not two competing engines The inference core shared by much of the local AI tooling lives in llama.cpp - that layer was covered in detail in [the llama.cpp internals post](https://computefit.dev/what-is-llama-cpp-under-your-local-models/). On top of it, two different serving products have grown: Ollama on one side, and the standalone `llama-server` binary (with its `-hf` downloader and web UI) on the other. Practically this means that running the same quantized file through either runtime usually gets you comparable output quality - they are not two different approximations of your model. What genuinely differs is everything around the engine: where models come from, what a fresh install assumes about context length and GPU placement, how many parallel workloads it will accept before queueing or refusing them, and which knobs exist at all. ## The three places they actually differ ### 1\. How you acquire models Ollama's default path is its registry: `ollama pull llama3.2` (or any published tag) and the model lands in a platform-specific local store (`~/.ollama/models` on macOS, `/usr/share/ollama/.ollama/models` on Linux with the standard installer), per its FAQ. You never touch individual files unless you want to. Finding something outside that registry is a supported but narrower path: Ollama's import docs describe importing GGUF files via a one-line `Modelfile` (`FROM /path/to/file.gguf`, then `ollama create my-model`), and safetensors weights for named architectures (Llama 2/3 families, Mistral 1/2 and Mixtral, Gemma 1/2, Phi3). You can also quantize an imported FP16/FP32 model during creation with `--quantize q4_K_M` or similar. `llama-server` has no registry step at all. It runs a local file path (`-m models/file.gguf`), pulls directly from Hugging Face repositories (`-hf user/repo:quant`, defaulting to Q4\_K\_M when no quant is given, plus automatic multimodal projector download), Docker Hub model repos, or serves static files from a mounted directory. Whatever .gguf you converted - including with llama.cpp's own `convert_hf_to_gguf.py`, which is the same converter Ollama's import docs point you to - can be pointed at directly. ### 2\. What a fresh install assumes about memory This is where day-one behavior diverges most, and it matters on tight cards (the context math itself in [the hidden cost of context length post](https://computefit.dev/context-length-hidden-vram-cost-local-llm/)). Ollama's current documentation gives default context lengths as tiers by installed VRAM - 4k under 24 GiB, 32k for 24-48 GiB, and 256k at 48 GiB or above - overridable per session with `/set parameter num_ctx`, per request via the `num_ctx` option in its API, or globally with `OLLAMA_CONTEXT_LENGTH`. (Its FAQ page also states a flat default of 4096 tokens; treat the context-length page and your live `ollama ps CONTEXT` column as what your install actually does.) Its own guidance is that workloads like agents, coding tools, and web search should be set to at least 64,000 tokens. `llama-server`'s context flag `-c/--ctx-size` defaults differently: a value of 0 means "use whatever the model file says". That is arguably the more honest default - it never silently promises a window larger than what you explicitly requested - but on a machine where nobody has set anything, your effective window depends entirely on the metadata baked into that quant. GPU layer placement is explicit rather than automatic: `-ngl` chooses how many layers live in VRAM (an exact number, or auto), `--split-mode` and `--tensor-split` control multi-GPU topology, and a `--fit` option (on by default) will adjust unset arguments to fit device memory with roughly 1 GiB of margin per device. You can also pin exact layer counts when you co-locate other GPU workloads and Ollama's automatic split is not the arrangement you want. Both expose KV-cache quantization, but with different scopes: Ollama's `OLLAMA_KV_CACHE_TYPE` (f16 default; q8\_0 at roughly half the memory, documented as the recommended non-f16 choice; q4\_0 at about a quarter) applies to all models on that server. For llama.cpp you set `-ctk`/`-ctv` per process and can pick from a wider list including f32, bf16, and 5-bit/4-bit variants (f32/f16/bf16/q8\_0/q4\_0/q4\_1/iq4\_nl/q5\_0/q5\_1), with `--override-tensor` for per-tensor control. Flash attention is on automatically where the backend supports it in both; each also has a force flag (`OLLAMA_FLASH_ATTENTION=0/1`, or `-fa on/off/auto`). The quality caveats are similar: models with high grouped-query-attention counts can be more sensitive to KV quantization precision. ### 3\. Concurrency and observability Ollama's documented model: multiple loaded models when memory allows (server-side default maximum is 3x the number of GPUs, or 3 on CPU), parallel requests per model controlled by `OLLAMA_NUM_PARALLEL` (default 1) with required RAM scaling as parallelism times context, and a queue capped by `OLLAMA_MAX_QUEUE` (512). Idle models unload after five minutes unless `keep_alive` says otherwise. Multi-GPU placement follows published rules: entirely on one GPU if it fits there (better for PCI bus traffic), spread across GPUs if not. `llama-server`'s model is slot-based: server slots via `-np` (auto by default) with continuous batching enabled by default, so concurrent requests share the model rather than multiplying its loading. It also has a health endpoint (`/health`, public), an optional Prometheus-compatible metrics endpoint (`--metrics`), and prompt caching on by default that re-uses computed context across repeat prefixes - relevant when agents send long shared system prompts repeatedly. ## Reference: where each runtime stands feature-by-feature | | Ollama | llama.cpp llama-server | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------- | | Model acquisition | Registry pull; GGUF/safetensors import for named architectures; quantize-at-create | Local .gguf path, Hugging Face repo via \-hf, Docker Hub repos; no registry step | | Default context length | VRAM-tiered per current docs (4k/32k/256k) | Model file metadata (\-c 0) until you set it explicitly | | GPU layer placement | Automatic; documented one-GPU-first multi-GPU rules | Explicit: \-ngl, split modes, tensor splits, fit-margin control | | KV cache dtype | Global server option (f16 default, q8\_0, q4\_0) | Per-process K/V flags with a wider type list; per-tensor override | | Concurrency model | Parallel loaded models + parallel requests (n times ctx), queue with 512 default cap | Server slots + continuous batching, prompt cache reuse | | API surface | Native /api/\* plus OpenAI-compatible routes - detailed in [the Ollama endpoint post](https://computefit.dev/ollama-openai-compatible-endpoint/) | OpenAI-compatible chat/completions/embeddings routes, Anthropic Messages compatible endpoint, native endpoints, health check | | Model lifetime | 5-minute idle default; keep\_alive, stop command | Loaded until you stop it (optional sleep-on-idle via \--sleep-idle-seconds) | | Observability | ollama ps for loaded split and context; log files per troubleshooting docs | Health endpoint, optional Prometheus metrics endpoint, performance timing flags | | Packaging | App + server installers with auto-updates (macOS/Windows), script for Linux | Release binaries, CMake build, published Docker images including CUDA builds | ## If you migrate: a parameter mapping | You want to carry over | Ollama side | llama-server side | | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | | Requested context length | /set parameter num\_ctx N, per-request "num\_ctx": N, or OLLAMA\_CONTEXT\_LENGTH | \-c N | | A resident model | keep\_alive: -1 in the request, or OLLAMA\_KEEP\_ALIVE=-1; unload with ollama stop or 0/short values | Stay loaded by default; optional sleep via \--sleep-idle-seconds | | Cheap KV cache for long context | OLLAMA\_KV\_CACHE\_TYPE=q8\_0 (server-wide) | \-ctk q8\_0 -ctv q8\_0 wider type list available; \--override-tensor per tensor | | Flash attention on/off | OLLAMA\_FLASH\_ATTENTION=1/0 (auto otherwise) | \-fa on/off/auto | | Exactly N layers in VRAM | Automatic; not normally exposed as a first-class request option the way llama.cpp's \-ngl is - verify what your version offers before relying on it | \-ngl N, split modes, tensor splits | | Listen address and port | OLLAMA\_HOST; default 127.0.0.1:11434 | \--host/\--port; default 127.0.0.1:8080 | | Parallelism budget | OLLAMA\_NUM\_PARALLEL (default 1) times ctx, plus loaded-model cap and queue | Serve slots (\-np) with continuous batching | The honest footnote on that last section: feature parity is per-install. Both projects ship frequently, and the switches above reflect each project's current documentation rather than a permanent contract - confirm the specific flags in whichever versions you actually run. ## Situations and what I would pick **A solo developer on a 24 GB-class laptop running agent tooling against an OpenAI-compatible endpoint.** Ollama. Registry pulls, one env var for context, `ollama ps` to verify the split - and if the defaults stop fitting your workloads, the [Qwen 3.8 on Ollama settings post](https://computefit.dev/qwen3-8-27b-ollama-settings-that-matter/) on this site walks through exactly which knobs matter in that position. This is the path of least friction by design. **A shared box where a 24 GB card also renders video or hosts other CUDA processes.** llama-server. Pin `-c`, set the KV cache to q8\_0 (or lower if you need room and can tolerate it), cap offload with an explicit layer count, and let `--fit`'s margin keep a headroom you actually chose instead of one inferred from what happens to fit. The metrics endpoint gives you numbers for "how much is this server holding" that Ollama only shows through `ollama ps`. **A small team where several services hit the same local model.** Both are defensible, and I would test both rather than guess: Ollama has documented knobs for parallel requests, loaded-model ceiling, and queue depth out of the box; llama-server offers slots plus continuous batching, prompt-cache reuse across repeat prefixes, API keys (`--api-key`, or a key file) and an SSL build option if anything beyond localhost is involved. If you will expose either one on a network beyond your LAN, that exposure decision matters more than which binary answered better: put it behind a reverse proxy with auth in both cases. **A specific decode feature you need - speculative decoding with a draft model, MoE expert offload to CPU, the 1.5/2-bit integer quantizations llama.cpp advertises in its README.** Check each project's current docs before assuming parity; as of writing, they document quite different sets of server-side switches. If the feature you need has first-class support on only one side, that is a legitimate deciding factor rather than a temporary annoyance to work around. ## Honest limitations Ollama's documented surface gives you fewer dials - deliberately; the FAQ reads as if "set context, check `ollama ps`, move on" is the intended operating mode. The exact default numbers also drift between its own pages across versions, so a live install beats any article (including this one) when it comes to what your box actually assumed. llama.cpp's server hands you more of that surface, and with it: no automatic model fetching from a friendly registry beyond HF/Docker repos, configuration as roughly a hundred flags rather than env vars plus CLI verbs, and lifecycle management (start, sleep, API keys) that is yours to script if you want anything non-trivial. It also uses "slots" where Ollama uses parallel-loaded-models vocabulary - when reading either set of docs mentally translate before comparing queue behavior. This post compares the servers, not their CLI chat tools; the underlying inference quality discussion for a given quant is covered in the Qwen3.8 series and the context-cost post rather than repeated here. ## Sources - Ollama docs - FAQ (context default text, model storage paths, keep\_alive, queue/parallelism limits, multi-GPU rules, flash attention and KV-cache quantization): https://docs.ollama.com/faq - Ollama docs - Context length (VRAM-tiered defaults, 64k guidance for agent workloads, OLLAMA\_CONTEXT\_LENGTH): https://docs.ollama.com/context-length - Ollama docs - Importing a Model (GGUF and safetensors import paths, quantization at create time): https://docs.ollama.com/import - llama.cpp repository README (backends, quantization types, quick start with `llama serve -hf`): https://github.com/ggml-org/llama.cpp - llama.cpp server tooling documentation (API surface, `-c`/`-ngl`/split modes/KV cache flags, slots and continuous batching, metrics, CORS guidance): https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md ## Conclusion Pick Ollama when the job is "have working local models on this machine without thinking about buffers" - its registry, defaults, and `ollama ps` verification loop are the smoothest path from install to a useful endpoint, and agent frameworks speak it fluently through its OpenAI-compatible routes. Pick `llama-server` directly when "working" must mean an explicit contract: a context window you set yourself, quantized KV cache where quality tolerance allows it, exactly the layer split that coexists with other GPU work, and metrics if you need numbers beyond a table in a terminal. The two can also coexist on one machine (different ports, different roles), and because they share files rather than formats - Ollama's own import docs even point at llama.cpp's converter - switching is a reconfiguration, not a re-download. Set your context and KV-cache expectations from the memory math in the companion post, pick the runtime that matches how much control you want today, and revisit the decision when your hardware tier changes rather than when marketing cycles do.