I finally did it: I stopped pretending “I’ll keep it simple” and built a local LLM lab. The grid is online, the fans are loud, and my cloud bill is suddenly less offensive.
This is not a benchmark shootout and it’s definitely not marketing copy. It’s ops notes from my lab: what I set up, what broke, what I measured, and what I’d do again if I had to rebuild it from scratch.
Why local, why now
Two reasons: cost and headspace. Cloud APIs are useful, but the meter running in the background changes how I work. I start optimizing prompts for pennies like I’m trading options. That’s not a vibe, and it’s not how you discover weird little workflows.
The other reason is privacy by default. Most of what I send to a model is not secret, it’s just… personal. Half-formed thoughts. Half-baked scripts. Stupid questions. I’d rather keep that inside my own rack.
The lab shape (on purpose, slightly vague)
In my lab this runs on one of my Proxmox nodes as a dedicated VM. I’m intentionally not listing every exact part number here because hardware drifts, and I don’t want this post to become “neo’s magical motherboard guide”.
What matters more than the brand names is the shape of the system:
- Enough RAM that the OS isn’t fighting you.
- Fast storage so loading models doesn’t feel like archaeology.
- Enough VRAM (or enough patience) for the model class you actually run.
- Stable networking so clients don’t randomly time out and blame the model.
The biggest lesson: the “LLM lab” is just another service. Treat it like one. Give it a clear IP, a clear port, logs, a health check, and a way to restart it cleanly. Once you do that, your LLM stops being a toy and starts being infrastructure.
Software stack: boring is good
I like stacks that are boring under stress. For local inference, that usually means: a single server process, a single set of model files, and a small number of knobs.
In my tests, llama.cpp keeps winning the “works on random hardware” award.
It’s not trying to be a platform. It’s trying to run.
I respect that.
A minimal server command (example)
This is the shape of the command I run. The exact flags depend on your CPU/GPU situation, but the intent is stable: start a server, pick a context size that won’t explode memory, and keep temperatures in the “not a space heater” range.
# example: llama.cpp server
# (flags vary by machine; treat as a starting point)
./llama-server \
-m /models/your-7b-or-8b-instruct.gguf \
--port 8080 \
--host 0.0.0.0 \
--threads 8 \
--ctx-size 8192 \
--temp 0.7Notice what’s missing: a pile of glue code. I want the inference server to be dumb and predictable. The fancy stuff (routing, tooling, UI) can live outside.
Models I actually run (realistic class, not flex)
Locally, I stick to models that are plausible to run at home without turning the room into a sauna. For me that’s often 7B–8B instruct models in GGUF form. Quantization is the cheat code here.
I’m not going to pretend everything is free. You trade something, usually nuance, for speed and memory. But in my tests, the trade has gotten surprisingly good. For many “assistant + scripting glue” tasks, a decent quant is absolutely fine.
What surprised me (in the good way)
1) Quantization is… kind of insane now
I expected quantization to feel like an emergency mode. Instead it feels like a legitimate operating point. Not perfect, but reliable enough that I can build workflows around it.
2) The bottleneck is often not what you think
The obvious bottleneck is “GPU or no GPU”. The annoying bottleneck is everything around it: model load times, swap, IO contention, thermals, and the moment you run two requests at once.
If you want the lab to feel fast, you don’t just chase tokens-per-second. You chase time-to-first-token, queue time, and “does it stay responsive when I’m doing other stuff?”
How I measure it (so I don’t lie to myself)
I try to avoid the classic homelab trap: “it feels faster” (source: vibes). Here are a few measurements that are easy to repeat and hard to bullshit.
Health check + basic latency
First, I want a cheap signal that the service is alive. If your inference server doesn’t expose a health endpoint, you can still do a simple request and time it.
# example: time a tiny request (adjust endpoint to your server)
# the point is repeatability, not exact numbers
/usr/bin/time -f "elapsed=%es" \
curl -s -o /dev/null \
http://localhost:8080/Time-to-first-token (TTFT) as a sanity metric
TTFT is what your brain perceives as “snappy”. Even if generation is fast, a slow TTFT makes the whole system feel sluggish. In my lab, TTFT improves a lot when:
- the model stays warm (not reloaded every time),
- storage isn’t saturated,
- and the box isn’t thermal-throttling.
Resource usage (roughly) while generating
I’m not married to one monitoring stack. The point is to observe CPU load, memory pressure, and (if applicable) GPU utilization. Use whatever you have. Even the “poor man’s dashboard” is fine.
# poor man’s dashboard (examples)
htop
# if you have NVIDIA, this is usually enough to catch obvious problems
nvidia-smi -l 1If you see CPU pinned and the request rate is low, you’re probably CPU-bound. If you see the GPU full but memory nearly full and errors start appearing, you’re probably one context window away from sadness.
Operational rules (so it doesn’t rot)
The difference between a “weekend hack” and a real service is how it behaves on Monday. These are the rules I follow so the grid doesn’t decay into a haunted appliance:
- One canonical model directory (
/models) and predictable filenames. - Log everything (even if it’s just stdout into a file at first).
- Restart should be boring (systemd, container restart policy, or a simple script).
- Backups are real: I snapshot model configs and prompts the same way I snapshot code.
In practice, I keep a tiny README next to the service with: ports, model path, and the “if it’s broken, do this” commands. Future me is not a friend. Future me is a stranger with a deadline.
What broke (so you can avoid it)
A few failure modes showed up quickly in my tests:
- Context too large for the memory budget → random slowdowns, then crashes.
- Too much concurrency → one request feels okay, two feel like a denial-of-service.
- Absolute links in static HTML → breaks immediately under subpaths (ask me how I know).
The fix is rarely “buy a bigger GPU”. Most of the time it’s: pick sane defaults, measure, and avoid footguns.
Next moves
My next upgrades are not glamorous:
- better observability (TTFT, queueing, rough throughput),
- a clean way to swap models without downtime,
- and a small archive page so the site doesn’t depend on my memory.
The grid is up. Now I want it to stay up.