Running Local LLMs on a Budget CPU, and When to Fine-Tune vs. Use RAG

Running Local LLMs on a Budget CPU, and When to Fine-Tune vs. Use RAG

Running a large language model on your own computer is much more accessible than most people think. You don’t need a data center or an expensive GPU to get useful results. This post walks through the whole path, from running a model locally to fine-tuning one for a specific task, using a real example: classifying customer complaints into consistent JSON.

My setup

• CPU: AMD Ryzen 5 5600G (6 cores / 12 threads)
• RAM: 19 GB
• GPU: integrated Radeon Vega only, no dedicated VRAM
• Disk: plenty of space

That means inference runs on the CPU. It’s slower than a GPU, but small and mid-sized models are perfectly usable for chat and experiments.

Step 1: Run a model locally

The easiest route is Ollama:

curl -fsSL https://ollama.com/install.sh | sh
ollama run qwen2.5:7b

Ollama downloads a quantized model automatically and gives you a chat prompt right away.

Step 2: Understand quantization

Models are normally stored with 16-bit weights. Quantization reduces that precision, usually to 4 or 8 bits per weight. The effects are:

• The file gets roughly 3-4× smaller.
• Memory use drops, so bigger models fit in RAM.
• Quality drops only slightly for most tasks.

A common format is Q4_K_M, a good balance for most people. As a rough rule, a Q4 model needs about 0.6 GB of RAM per billion parameters, plus some overhead for context.

Approximate fit on a 19 GB machine:

Size	Q4 RAM needed (approx.)	Verdict
3B	~2-3 GB	Fast, great for experiments
7-8B	~4-5 GB	Comfortable sweet spot
14B	~9 GB	Fits, but slow and tight on memory

Step 3: Tune the runtime (not the model)

“Tuning” can mean two very different things. The first is tuning runtime settings, which makes inference faster without changing the model:

• Threads: set to your physical core count (6 here), not 12. Hyperthreads often don’t help LLM inference.
• Context size (num_ctx): larger values use more RAM and slow things down. Start around 2048-4096.
• Quantization level: Q4 is faster and lighter; Q8 is more accurate but bigger and slower.
• Model size: this has the biggest impact on speed.

In Ollama, you can set these with a Modelfile:

FROM qwen2.5:7b
PARAMETER num_thread 6
PARAMETER num_ctx 4096

ollama create my-model -f Modelfile

Feeding a model lots of data doesn’t automatically make it slower. What slows things down is the length of the prompt you send and the size of the model.

Step 4: Fine-tuning vs. RAG

The second meaning of “tuning” is fine-tuning: training the model further on your examples. The key question is what you actually need.

Use RAG when you need the model to answer from specific documents:

• The data changes often. Swap files, no retraining needed.
• You want answers you can trace back to a source.
• The knowledge is factual: manuals, FAQs, archives, internal docs.

Use fine-tuning when you need consistent behavior:

• A fixed output format, such as always returning the same JSON structure.
• A consistent tone or style.
• A repetitive, narrow task with clean labeled examples.

One important caveat: fine-tuning teaches how to answer, not what facts to know. The model can still forget or hallucinate facts. For knowledge, use RAG.

A sensible order of attack:

1. Try a good prompt first. Often that’s enough.
2. Add few-shot examples (2-5 in the prompt).
3. Add RAG if the model needs your data.
4. Consider fine-tuning only if format or style is still unstable.

Step 5: A realistic fine-tuning example

Let’s say you want to classify Indonesian customer complaints into a fixed JSON format:

{"kategori":"pengiriman","prioritas":"tinggi","ringkasan":"Paket terlambat 2 minggu, status stuck di gudang"}

Each training example is a conversation in JSONL format:

{"messages":[{"role":"system","content":"Klasifikasikan keluhan pelanggan. Balas hanya JSON."},{"role":"user","content":"Paket saya sudah 2 minggu belum sampai, resi JNE123 masih di gudang"},{"role":"assistant","content":"{\"kategori\":\"pengiriman\",\"prioritas\":\"tinggi\",\"ringkasan\":\"Paket terlambat 2 minggu, status stuck di gudang\"}"}]}

Aim for roughly 300-1000 examples, plus about 50 held out for testing.

Labels matter more than volume. The model learns the pattern in your data. If similar complaints get different priorities, the model learns that inconsistency. Write down your labeling rules before you start, for example:

• High: payment problems, app crashes, lost packages
• Medium: delayed packages, damaged products
• Low: general questions

Step 6: Choose a base model

Four things matter:

1. Size: 3B trains comfortably on a free Colab T4 GPU with 4-bit QLoRA, and 7B is possible but slower.
2. Instruct variant: pick the one labeled Instruct or Chat. It already follows instructions, so fine-tuning is faster and more stable.
3. Language support: test a few Indonesian examples, especially casual phrasing and abbreviations. Qwen2.5 handles this reasonably well.
4. License: check it allows your intended use.

My starting pick is Qwen2.5-3B-Instruct. It’s quick to train and light to run on CPU. Move up to 7B if the results aren’t good enough.

Step 7: The full process

1. Define the task and rules. Write the output format and labeling criteria.
2. Test the baseline. Run the base model with prompts and few-shot examples on your 50 test cases. If it’s already good enough, stop here.
3. Collect raw data. Gather real complaints and remove personal information.
4. Label. Write the JSON output for each example, following your rules. A stronger model can draft labels, but check them by hand.
5. Format the dataset. Save as JSONL, then split into train.jsonl and test.jsonl. Never train on test data.
6. Set up training. Open an Unsloth notebook in Google Colab, select a T4 GPU, and load the base model in 4-bit.
7. Train with QLoRA. Attach LoRA adapters and train for 2-3 epochs. Watch the loss, but don’t trust it alone.
8. Evaluate. Run the 50 test cases and measure valid-JSON rate and label accuracy. Compare against the baseline from step 2.
9. Export. Merge the adapter, convert to GGUF, and quantize to Q4_K_M.
10. Run locally.FROM ./complaint-model.gguf PARAMETER temperature 0.1ollama create complaint-classifier -f Modelfile ollama run complaint-classifier
11. Iterate. When the model gets something wrong, add that case to the dataset and retrain.

The two steps that decide success are labeling (step 4) and evaluation (step 8). Most failed fine-tunes come down to inconsistent labels or skipping a proper comparison against the baseline.

Closing thoughts

You don’t need expensive hardware to explore local LLMs. Start small with Ollama, learn how quantization affects speed and quality, and only reach for fine-tuning when simpler approaches stop working. Measure everything against a baseline, and let your evaluation set, not your intuition, decide what to keep.