How We Test

This page describes our default approach when articles include original measurements of local AI running on consumer hardware: what "24GB Lab" refers to, which metrics we may use, and what we deliberately do not claim.

What the 24GB Lab means

The 24GB Lab label on this site refers to our own reference environment: a GPU with around 24 GB of VRAM, rather than cloud GPUs rented for testing — hardware that is ours and where results can be tied back to identifiable machines. Our current lab baseline uses an NVIDIA RTX 5090 Laptop GPU with 24 GB of VRAM. When an article reports measurements from our current lab, the exact hardware used is identified in that article.

Metrics we typically use

Where an article includes original measurements, depending on the question it is trying to answer, we may report metrics such as time-to-first-token (TTFT), sustained output speed in tokens per second over a stated number of generated tokens, and peak GPU memory / system RAM usage during that run. The prompt length, context size, temperature or other settings used are noted alongside those numbers where they matter for the specific workload covered by the article, rather than assumed identical across unrelated tasks.

Making results replicable

Where feasible we include enough detail to help you reproduce a measurement on similar hardware: model identifier and version, quantization file level, runtime (e.g. which version of Ollama or llama.cpp), and other settings that genuinely affect the result. Where it materially helps reproducibility, we also include the exact command or configuration used.

Cloud comparisons (only where we actually use them)

Some articles compare how a model behaves under constrained local memory against runs where VRAM scarcity is not the limiting factor. If we perform a cloud comparison, the provider and test conditions are identified in that article rather than presented generically here; such results are labeled as non-local comparisons used only to show limits where there is no longer meaningful memory pressure.

What we do not claim

We avoid treating benchmark/eval scores produced by different harnesses, datasets or tokenizers as interchangeable without naming which one produced each figure and why; and when your exact machine configuration differs from ours in ways that affect a result — clock-speed tier, unified-memory architecture versus discrete VRAM, CPU fallback behavior — we note the difference in-line instead of assuming parity between hardware classes that look similar on paper at the same nominal VRAM size.