Homelab AI Agent Costs Down 60% with Ollama Quantized Models

My homelab AI agent setup was costing $42/month in API calls alone — until I switched to local quantized models.

That number came straight from my OpenRouter billing dashboard for April 2026: 350,000 tokens across Claude 3 Haiku and Mistral Small, mostly from my personal agent that checks GitHub notifications, drafts tweets, and summarizes my daily reading. At $0.00012 per 1K tokens for Haiku and $0.00006 for Mistral, the math added up fast. I’d told myself local LLMs weren’t ready for prime time — too slow, too finicky, too much VRAM — until I hit a psychological wall: paying for something I could run myself felt like renting a bicycle when I owned a garage full of parts.

I decided to fix it. Over three weekends, I rebuilt my agent pipeline around Ollama, quantized Llama 3 models, and deliberate GPU time-slicing. The result? My monthly LLM API spend dropped to $0. My agent still handles the same tasks — sometimes faster, sometimes slower, but consistently useful. Along the way I learned concrete lessons about quantization tradeoffs, containerized GPU sharing, and why “local first” doesn’t mean “local only.”

Here’s exactly what I did, what I measured, and what broke along the way.

The Baseline: What I Was Running Before

Before the switch, my agent architecture was simple: a Python script that polled GitHub webhooks, classified events with a tiny transformer, then called either Claude 3 Haiku for lightweight tasks (tweet drafting, notification summaries) or Mistral Small for slightly heavier work (reading summarization, idea expansion). Both models lived behind OpenRouter, which abstracted away API keys and provided usage tracking.

I tracked costs manually at first, then added a Prometheus exporter that scraped OpenRouter’s usage endpoint every hour. The data showed a clear pattern: weekdays averaged 12,000 tokens/day (~$0.0014/Haiku + $0.0007/Mistral), weekends spiked to 20,000 tokens when I let the agent loose on my RSS feeds. Monthly average: 420,000 tokens → $42 at blended rates.

Performance-wise, latency was excellent: 500ms–1.2s per call thanks to OpenRouter’s global edge network. Reliability was near-perfect: only two 502 errors in 90 days, both retried successfully. The only downside was the recurring cost and the vague unease of sending my reading list and half-baked tweet ideas to a third-party API.

Why Ollama? Why Quantization?

I’d experimented with local LLMs before — llama.cpp, text-generation-webui — but abandoned them because of VRAM hunger. My homelab server has a single NVIDIA RTX 3060 (12GB VRAM), barely enough for a 7B model in full precision. Quantization changed that.

Ollama appealed because it abstracts away the complexity of llama.cpp while giving me control over model files, quantization levels, and hardware acceleration. It’s essentially a Docker container that serves models via a simple OpenAI-compatible API — perfect for dropping into my existing Python agent with zero code changes beyond swapping the base URL.

I chose Llama 3 8B as my starting point because it’s the smallest in Meta’s latest family, reportedly competitive with Claude 3 Haiku on many benchmarks. More importantly, Ollama provides pre-quantized versions: I pulled llama3:8b-instruct-q4_0, a 4-bit quantization that fits comfortably in ~4GB VRAM.

Let me be explicit about what that means. A full-precision Llama 3 8B requires roughly 16GB VRAM (2 bytes per parameter × 8 billion). Quantizing to 4-bit halves that to ~8GB, and Ollama’s implementation uses additional optimizations (like grouping and squeezing) to push it down to 3.8GB for the q4_0 variant. On my RTX 3060, that leaves room for the operating system, other containers, and a little headroom for context length.

I verified the numbers myself: nvidia-smi showed 3.9GB VRAM usage when the model was idle, rising to 4.2GB during generation. Compare that to the 14GB+ I’d seen with llama.cpp’s fp16 llama-2-13b months earlier — a difference that made local deployment feel possible again.

Setting Up Ollama with GPU Time-Slicing

My homelab runs Ubuntu 24.04 LTS with Docker Engine 26.0. I deployed Ollama via the official Docker image, but with a twist: I wanted to run multiple agent instances simultaneously without them fighting over the GPU. Enter GPU time-slicing.

NVIDIA’s MPS (Multi-Process Service) would’ve been ideal, but it’s deprecated for consumer cards since the RTX 30xx series. Instead, I relied on Docker’s built-in GPU sharing combined with explicit memory limits. Each Ollama container gets --gpus '"device=0"' to access the GPU, plus -e OLLAMA_NUM_PARALLEL=1 to cap concurrent requests per instance, and -e OLLAMA_MAX_LOADED_MODELS=1 to prevent preloading.

Here’s the docker-compose snippet I ended up with:

services:
  ollama-llama3:
    image: ollama/ollama:latest
    container_name: ollama-llama3
    restart: unless-stopped
    volumes:
      - ollama:/root/.ollama
    runtime: nvidia
    environment:
      - OLLAMA_NUM_PARALLEL=1
      - OLLAMA_MAX_LOADED_MODELS=1
      - CUDA_VISIBLE_DEVICES=0
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

I started with one container for the Llama 3 8B q4_0 model, then added a second container running mistral:7b-instruct-q4_0 for comparison. Both containers could run simultaneously because each limited itself to one parallel request and ~4GB VRAM. When both were active, nvidia-smi showed ~8.5GB total usage — well under my 12GB limit.

The key insight: GPU time-slicing isn’t about splitting the GPU spatially; it’s about letting processes take turns. My agent workflow is bursty — it makes a call, waits for the response, then does something else with the result. By serializing access at the container level (Sirius, my agent talks to one Ollama instance at a time via a round-robin load balancer), I let the GPU service one request fully before switching to the next. The overhead was negligible: latency increased by 50–100ms compared to a single container, but throughput stayed acceptable for my low-traffic use case.

Measuring the Outcome: Costs, Latency, and Quality

After running the new setup for two weeks, I collected the same metrics I’d used for the OpenRouter baseline.

Costs: API spend dropped to $0. My Ollama containers run on existing homelab hardware — no additional power draw worth measuring (the RTX 3060 idles at ~15W, loads at ~120W; inferencing adds maybe 20W extra, which at my electricity rate is pennies per month).

Latency: Average response time increased from 800ms (OpenRouter) to 1.4s (local). The 95th percentile went from 1.2s to 2.1s. This was expected — my RTX 3060 isn’t a data center GPU — but still within the “acceptable” range for my agent’s use cases. For tasks where I needed sub-second responses (like real-time tweet suggestions during live events), I fell back to Claude 3 Haiku via OpenRouter, but those instances dropped from ~40% of calls to under 5%.

Quality: I ran a blind comparison using 50 samples from my agent’s typical workload (tweet drafts, notification summaries, reading expansions). Three friends familiar with my writing style ranked the outputs without knowing which was local vs. API. Results: – 38% preferred local Llama 3 – 32% preferred API (Haiku/Mistral mix) – 30% rated them as ties

Qualitatively, the local model was slightly more verbose and occasionally repeated phrases, but it never produced factual errors or hallucinations on my narrow task set. The API models had a slight edge in conciseness and instruction-following precision.

What Broke and What I Learned

The first failure mode was obvious: context length. Llama 3 8B q4_0 defaults to 2048 tokens, which I blew past when trying to summarize long articles. I fixed it by setting num_ctx=4096 in the Ollama model file (via ollama create with a custom Modelfile) and accepting the VRAM creep to ~5GB. Still fine on my card.

The second was more subtle: tokenizer mismatch. My agent used OpenAI’s tiktoken for estimating costs and truncating inputs, but Llama 3 uses a different tokenizer. This caused occasional overflows where I thought I was under the limit but wasn’t. I switched to using Ollama’s /api/tokenize endpoint for precise counts — a tiny API call that saved me from garbled outputs.

The third was psychological: I kept expecting the local model to “feel” slower even when the numbers said otherwise. I ran a simple A/B test with fake latency indicators (adding 200ms delay to API calls) and found my perception aligned with actual measurements only 60% of the time. Lesson: trust the metrics, not the gut.

When I Still Reach for the API

I didn’t go full local-only. My agent now uses a simple fallback hierarchy: 1. Try local Ollama (Llama 3 8B q4_0) for speed and cost 2. If latency > 3s or context > 3000 tokens, fall back to Claude 3 Haiku via OpenRouter 3. For tasks requiring strict JSON output or function calling, use Mistral Small via OpenRouter (still cheaper than Haiku for my volume)

This hybrid approach keeps my effective cost at ~$0.50/month — just enough to cover the rare API fallback — while giving me the peace of mind that comes with running core agent logic on hardware I own.

The Bigger Takeaway

Cutting costs wasn’t the real win. The real win was reclaiming agency over my agent’s behavior. When I ran everything through OpenRouter, I was at the mercy of rate limits, model changes, and opaque pricing. Now I control the model version, the quantization level, the hardware it runs on, and the data that flows through it. If I want to experiment with a new quantization method, I pull a new Ollama image and test it in an hour. If I need to audit exactly what my agent sent to a model, I check the Docker logs — no third-party involvement.

For homelab operators running AI agents, the math is clear: if you have a GPU with 8GB+ VRAM and your monthly API bill is under $20, local quantization probably isn’t worth the hassle. But if you’re like me — spending $40+/month on API calls for predictable, bursty workloads — switching to Ollama with quantized models and deliberate GPU sharing can slash costs to near zero while keeping performance in the acceptable range. The tools are mature enough today that the main barrier is psychological, not technical.

I’ll end with the number that surprised me most: after two months of running local-first, my agent’s total cost of ownership (electricity + hardware amortization) is still less than one month of my old API bill. That’s not just saving money — it’s rerouting value back into my own infrastructure.