> ## Content Index > Fetch the complete content index at: https://computefit.dev/llms.txt > Use this file to discover other available public pages before exploring further. # Running Local RAG with Embeddings in Ollama: Chunking, Dimensions, and the Failure Modes Nobody Warns You About - URL: https://computefit.dev/local-rag-embeddings-ollama-chunking-dimensions-failure-modes/ - Published: 2026-08-25T01:15:19.000Z - Updated: 2026-08-25T02:22:57.000Z - Description: You do not need a vector database or extra VRAM to give your local LLM semantic search over its own documents. You need the right chunk sizes, the right embedding tag, and an index that fails loudly instead of silently cross-model-searching. - Author: Alex Vale - Tags: Local AI, RAG You have already spent time making a chat model work locally: [context length has real VRAM cost](https://computefit.dev/context-length-hidden-vram-cost-local-llm/), an idle model will be evicted unless you tell the runtime to keep it resident (covered in our article on Ollama’s keep-alive), and Ollama can serve everything through an OpenAI-shaped API that your existing tools already know how to speak ([how far that compatibility actually reaches](https://computefit.dev/ollama-openai-compatible-endpoint/)). There is one piece of the local stack those posts never touched: when the model needs to answer about *your documents*, something has to decide which parts of your corpus are worth putting in front of it. That something is an embedding model, and on 16–24 GB hardware it is cheap enough that most people skip it entirely — or build the wrong pipeline around the wrong kind of model. ## The short version - An embedding model is a small text-only model whose entire job is to map passages into vectors, so you can store them and search by meaning instead of exact words. It does not read documents on your behalf, cite sources, or answer anything: it only represents overlap between texts as numbers. - You run one alongside your chat model on the same Ollama instance. `POST /api/embed` (or the OpenAI-compatible `/v1/embeddings`) returns L2-normalized vectors in a single call, and batching is built in — pass an array of strings, get back an array of vectors. - Sizes across common options range from 274 MB to 4.7 GB — an order of magnitude smaller than the chat LLM sitting next to it on your machine. - The practical failure modes are almost never about GPU horsepower: silent truncation when input exceeds the embedding model’s short context window, vectors and queries produced by two different models after a tag got pulled in the background, and trusting benchmark tables instead of testing retrieval on your own corpus. ## What an embedding model actually does You already know why “just load all my documents into the prompt” stops working: every token you feed has a memory price tag and slows generation, so long sessions run out of hardware before they finish reading. Keyword search is the other common local option, but it silently fails on paraphrase — ask about “the incident where exports stopped” and exact tokens will not surface a note that says “data dump failure.” Embedding models close that gap at the retrieval step. You pass text in, it returns a numeric vector: Ollama’s own API documentation example shows ten small decimal values, roughly between -0.002 and +0.13, and `/api/embed` returns vectors that are L2-normalized (unit length), which means cosine similarity between two vectors reduces to an ordinary dot product in your comparison code. Ollama’s documentation states the role directly: embeddings “turn text into numeric vectors you can store in a vector database, search with cosine similarity, or use in RAG pipelines,” and typical length is 384–1024 dimensions depending on model. Your actual dimension count depends on the tag you run — verify it from the first batch response before writing the storage schema around an assumption. The endpoint story matters more than most readers expect, because it decides how much glue your pipeline needs. Ollama exposes a native `POST /api/embed` — `{"model": ..., "input": ...}`, where `input` can be one string or an array of strings for batch — and, on top of that, the OpenAI-compatible `/v1/embeddings`. The compatibility surface supports a plain string and an array of strings as input plus the `dimensions` field (the token-array variants are documented as not supported). Concretely: any tool that already calls OpenAI’s Embeddings API can point at local Ollama with no code changes, which is what makes a local RAG stack usable from existing agent frameworks instead of requiring bespoke plumbing. The native response also carries timing fields (total duration and load time in nanoseconds) plus `prompt_eval_count`, the number of input tokens that went into producing each vector — useful for confirming that truncation is not quietly eating your chunks. ## Picking an embedding model: sizes, context windows, dimensions Ollama’s documentation names three models as its recommended embedding options (embeddinggemma, qwen3-embedding, all-minilm). The table below pulls file size and context window from the Ollama library pages; dimension counts come from each project’s model card where I could verify one. Every entry loads comfortably in system RAM on a 16 GB machine — embedding inference is small enough that GPU offload usually buys nothing, which matters if your VRAM belongs to a chat LLM. | Model (Ollama tag) | Download size | Context window | Vector dimensions | Rough fit | | ------------------------------------------------ | ------------------------------------------------- | --------------- | ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | nomic-embed-text:latest | 274 MB | 2K | 768 (fixed) | The conservative English-first option for personal notes, docs, and code comments in English | | embeddinggemma | 622 MB (current tags observed: :latest and :300m) | 2K | Not verified in sources I could read for this article; check before building your storage schema | One of Ollama’s three officially recommended embedding models (per docs.ollama.com/capabilities/embeddings), with a 2K context window like nomic-embed-text — verify behavior on a sample of your own corpus before committing. | | qwen3-embedding:0.6b / :4b / :8b (:latest = :8b) | 639 MB / 2.5 GB / 4.7 GB | 32K / 40K / 40K | 1024 / 2560 / 4096; each tag supports Matryoshka-style dimension truncation (MRL) | Multilingual corpora and long documents; the :4b tag is the middle option if you want multilingual plus 40K ceiling without needing 8B headroom for it | Three rules that make most of this table irrelevant to your decision and keep it consistent: 1. **Pick by languages and chunk length, then verify dimensions empirically.** An English-only notes corpus has three defensible default answers; a mixed-language household archive does not. Whatever you choose, the first embed call returns the actual dimension count — use that value (not the table) to size your storage, because tag variants and project releases differ. 2. **“Supports 40K context” is a ceiling for an individual chunk, not a recommendation for one.** Quality degrades long before you fill a window in most personal pipelines; small chunks that clearly contain the fact beat sprawling ones that contain it plus noise. We will come back to this when discussing truncation, because the two settings interact. 3. **Dimensions mostly affect storage and not quality at home scale.** A 768-dimensional float vector is about 3 KB per chunk; a 4096-d one is roughly 16 KB. Ten thousand chunks: about 31 MB versus 157 MB of raw floats — immaterial next to the corpus itself, but if you keep tens of thousands of passages and want them in RAM for dot-product search, this is where disk starts to matter. ## Chunking and context windows: where local RAG actually breaks The most under-appreciated configuration decision in a home RAG pipeline is not top-k or which vector database. It is chunk size, and it is controlled by the embedding model’s context window — not the chat LLM’s. Three documented things make this concrete: 1. **Silent truncation.** Ollama documents `truncate` on `/api/embed` as defaulting to true: inputs longer than the context window get truncated rather than error. A pipeline that tests fine against a general chat LLM accepting long prompts can pass 3K-token chunk text into an embedder capped at 2K and produce vectors for only the first N tokens of each passage — with no symptom in your logs. 2. **Flip it if you’d prefer failure on over-length inputs**: `"truncate": false` makes Ollama return an error instead. Both behaviors are normal configuration, not bugs — set chunk sizes so the question does not matter and pick whichever mode matches your testing stage. 3. **The response tells you what happened**, for better or worse: `prompt_eval_count` reports how many tokens of the input actually went into producing each vector. If it is systematically lower than the chunk length you feed, your embedder is truncating and retrieval quality has a ceiling with no error message to find. A word on what embeddings genuinely are blind to: distinctive exact strings — IDs, serial numbers, version strings, error codes with unusual tokens. Vectors do not have an affinity for them the way they do for topical overlap, so a corpus heavy in identifiers needs a second index alongside the vectors rather than replacing it. Small local pipelines usually keep a cheap keyword path and merge candidate lists together. ## The minimal stack: JSONL + numpy, no vector database Before reaching for pgvector, Qdrant, or LanceDB — all of which are reasonable choices at larger scale — note that the retrieval side of a personal RAG is small enough to live in one Python file. This shape (JSONL corpus + numpy sidecar) runs against Ollama’s native API with only `numpy` as an added dependency: import json import urllib.request import numpy as np OLLAMA = "http://localhost:11434" EMB\_MODEL = "nomic-embed-text" # pin the tag; see "swapping models later" below def embed(texts): """texts: list\[str\]; returns a float32 (len(texts), dims) array. Ollama's /api/embed L2-normalizes each vector for you.""" req = urllib.request.Request( OLLAMA + "/api/embed", data=json.dumps({"model": EMB\_MODEL, "input": texts}).encode(), headers={"Content-Type": "application/json"}, ) with urllib.request.urlopen(req) as r: out = json.loads(r.read())\["embeddings"\] arrs = \[np.asarray(v, dtype="float32") for v in out\] return np.stack(arrs) \# --- build (run once per corpus change; chunk defensively beforehand) -------- CHUNKS\_FILE = "chunks.jsonl" # one JSON object per line: {"id": int, "text": str} rows = \[json.loads(line) for line in open(CHUNKS\_FILE, encoding="utf-8") if line.strip()\] ids = \[r\["id"\] for r in rows\] texts = \[r\["text"\] for r in rows\] BATCH = 32 # /api/embed accepts arrays; batching keeps requests small and progress legible matrix = np.row\_stack(\[embed(texts\[i:i + BATCH\]) for i in range(0, len(texts), BATCH)\]) \# the probe doubles as your dimension-count verification: read len(probe) live, \# never assume dimensions from a table (this article's included). probe = embed(\["dimension check"\])\[0\] dims = int(len(probe)) assert matrix.shape\[1\] == dims, "chunk shapes disagree inside one batch" np.savez("embedding\_index.npz", model=np.array(EMB\_MODEL), ids =np.asarray(ids, dtype=np.int64), vectors=matrix.astype(np.float32)) \# --- search (run on every query) --------------------------------------------- def retrieve(query\_text, top\_k=5): idx = np.load("embedding\_index.npz") if str(idx\["model"\]) != EMB\_MODEL: # fail loudly rather than cross-model search raise ValueError(f"index built with {str(idx\['model'\])!r}; running {EMB\_MODEL!r}") q = embed(\[query\_text\])\[0\] # also L2-normalized by Ollama, so dot == cosine sims = idx\["vectors"\].astype("float32") @ q top\_slice = np.argpartition(sims, -top\_k)\[-top\_k:\] # cheap: no full sort required for the slice ranked = sorted(top\_slice, key=lambda i: float(sims\[i\]), reverse=True) return \[(int(idx\["ids"\]\[i\]), float(sims\[i\]), rows\[int(i)\]\["text"\]) for i in ranked\] Deliberate choices worth pointing at. The index stamps the embedding model that built it and `retrieve()` refuses to run against a different one — that is what prevents silently cross-model searching after an upgrade or a mistyped tag on another machine. Dimension count comes from reading the first live response rather than trusting the table in this article or from memory, because dimensions are exactly the value that differ per tag and deserve verification at build time before the schema around them goes anywhere permanent. Batch size 32 is reasonable — a middle point between roundtrips for small corpora and request sizes you would not want to eyeball on an untrusted network — but it exists because array inputs are supported, so there is no reason to burn `prompt_eval_count`’s usefulness over individual calls. And the retrieval result deliberately carries chunk text alongside score so a human can sanity-check the top few before feeding them to the chat LLM; automatic grounding alone will not catch a retrieval miss in a personal corpus. ## Making retrieved text survive into an honest answer The step everyone under-invests in locally. You can get three classes of bad output from a perfectly working retriever, and none of them should be attributed to your embedding model: 1. **Answer contradicts the source.** The classic local-RAG failure: you asked for grounding against retrieved text, and it grounded against its own priors instead. Stronger prompt framing helps — name what to ignore if not present in passages, require citing chunk IDs inline with claims, grade results by whether their cited ID actually contained the supporting sentence. 2. **Citation drift: correct fact from one retrieved chunk while a second fabricated detail borrows the same citation format.** Split citations out and verify each as its own unit against the source text before you present it to yourself; doing this in a single prompt (list cited IDs, then assert whether each was used) is worth more than hoping on instinct because the checks are small enough that even “semi-automated” beats “manual.” 3. **Similarity looks fine but top-k simply does not contain the answer** at all — a retrieval coverage failure, usually from chunk-size choices or language mismatch. This one is only visible once you measure it on your own corpus, which raises the next section’s point about trusting other people’s numbers. ## Benchmarks: where vendor-reported numbers end Qwen3-Embedding’s official technical report lists benchmark scores for each size variant, including an MTEB Retrieval result (MTEB-R) of 61.82 in their English-evaluation table. Those are vendor-reported numbers measured on standardized retrieval benchmarks — quotable as a relative ordering among the sizes within that family, and not transferable to your notes without testing them there. Two features from the same report matter more for local users than any score: instruction-aware prompting (a prefix tunes retrieval behavior per task or language) and multilingual coverage across 100+ languages — both are things you would end up tuning by feel for weeks if you never read their docs, and neither shows up in a headline number at all. ## What goes wrong in a home RAG setup (and what is cheap about fixing it) - **Your laptop GPU’s VRAM is not the constraint; disk and RAM are.** At these sizes, embedding inference fits comfortably inside system memory on 16 GB machines; even the 4.7 GB :8b tag is a question of storage headroom rather than needing offload — that one matters when you are deciding whether to keep it resident next to your chat LLM. - **Tapping two different models accidentally: Ollama keeps `:latest` as a pointer and pulls rotate it under you.** Pin the tag in configuration or code (a full, explicit tag like `nomic-embed-text:latest` is still only pinning what someone might quietly pull over — prefer locking to an immutable tag name if your project has one). - **Silently degrading retrieval after a background update:** the stamped-model guard in the code above is cheap insurance against this class of surprise across multiple sessions. Re-index when you intentionally change models rather than mixing old-vector and new-query pairs. - **Chunk sizes outgrowing what `prompt_eval_count`’s reporting shows:** if that number diverges from your intended chunk token count, stop guessing and read the real tokens — the difference tells you exactly how much of each passage was lost. Ollama’s documentation also names the OpenAI-compatible `/v1/embeddings` surface (which is what existing agent-framework glue points at by default rather than a custom client), and Qwen3-Reranker models exist in the same ecosystem to re-score small candidate lists when initial retrieval under-covers. Both surfaces are documented APIs rather than needing bespoke plumbing, and reranking can share your single GPU slot with embedding inference without fighting over VRAM on 24 GB hardware — the tradeoff shifts slightly if you are running everything on one machine that also hosts a chat LLM as our article on [the cloud-versus-local split](https://computefit.dev/cloud-ai-vs-local-ai-agents/) frames it. ## Cheap mistakes worth catching early in personal RAG setup 1. Treating the index as throwaway instead of versioning it like any other artifact: corpus grows, chunks change, old vectors remain unless you re-embed — make changed passages trigger re-embedding rather than waiting for a full rebuild. (experience, not spec) 2. Assuming headline multilingual numbers transfer one-to-one to the specific language pairs that matter in your archive: supporting “100+ languages” as stated does not mean every one performs on par with English retrieval scores from benchmark sets. Test a handful of your own text pairs before trusting coverage claims implicitly. 3. Leaving `truncate` at its default true while debugging “weird low relevance,” when over-length chunks are the actual root cause you cannot see: once chunk generation is deterministic, flip it to false temporarily so any remaining length bug fails loudly instead of silently degrading your embeddings. ## Conclusion: what a local embedding model buys (and does not buy) on your machine If the pipeline question is “can my notes be searched by meaning without touching an external API,” yes: - You add one small model, Ollama manages it like any other tag (`ollama pull`, keep-alive applies identically), and `/api/embed` or `/v1/embeddings` gives you a clean way to batch-embed your corpus. Sizes I verified above range from 274 MB to 4.7 GB — RAM-side, not VRAM-side for realistic home hardware. - Chunk size should come from the embedding model’s context window (together with `truncate`’s behavior under it), not from what your chat LLM accepts — that keeps the most common silent failure mode out of the build by construction, and `prompt_eval_count` lets you verify it kept working after any changes upstream. - Skip vector databases until corpus size or access patterns actually force you into one: JSONL plus a sidecar matrix works well enough at personal scale to tell whether you need more before buying it. (experience, not spec) The model choice itself is “pick the smallest option that covers your languages, test on samples of your own corpus, and record which tag was used” rather than a big upfront investment — dimensions in particular should come from reading the first real response, not from trusting any table, this article’s included. Where durable gains live in home RAG is retrieval discipline: chunking quality, loud failure on cross-model search or truncation, reranking small candidate lists when precision actually matters. That sequencing tracks what the rest of [what hardware actually buys you locally](https://computefit.dev/how-much-hardware-do-you-need-for-local-ai-agents-a-practical-vram-ram-and-cpu-guide/) has suggested all along: configuration you can audit beats a bigger number next to the word “model”, every time.