Although the LLMs on ChatGPT, Perplexicity, and other cloud-based AI platforms have terrific reasoning capabilities, locally-hosted models have some major perks up their sleeve. Since they run on local hardware, you don’t need to spend extra bucks every month or deal with rate limits to run multiple inference jobs in a short amount of time. Likewise, local LLMs are significantly better for productivity tasks centered around financial records, academic details, personal projects, and confidential files, as they respect your privacy instead of storing data on some random company’s cloud.
The drawback, of course, is that you need a lot of processing prowess to drive models that don’t devolve into a hallucinating mess… Or, at least, that’s how it used to be when local LLMs started making the rounds. These days, it’s entirely possible to get powerful LLMs up and running on consumer-tier graphics cards at respectable speeds.
Related
Two old GPUs I salvaged are doing more AI work than a brand new $2000 card, and I won’t be upgrading anytime soon
I built a local AI setup out of two old GPUs that sell for cheap, and it beats a single new card
Mixture-of-Experts models are game-changers for GPUs with limited VRAMs
My 12GB VRAM card runs a bulky 35B model at respectable speeds
Despite sneering at the low reasoning capabilities of locally-hosted LLMs, mixture-of-experts models single-handedly made me move my entire AI stack to my VRAM-constrained cards. Sure, I’ll admit that it’s technically possible to load conventional LLMs into GPUs with limited VRAM, but each workaround has some major drawbacks. Opting for a highly quantized model in the INT2 range lets you accommodate it on weak GPUs, but you’ll end up degrading its accuracy big time. Alternatively, you can move different LLM layers to the CPU using the -ngl flag, thereby letting a bulky LLM fit inside limited VRAM. However, the CPU acts as a huge bottleneck for this setup and tanks the token generation rate for the LLM.
Meanwhile, MoE models have a downright ingenious architecture that solves the performance vs. reasoning capability conundrum. Rather than featuring dense layers like traditional LLMs, MoE models have smaller feed-forward networks called experts that run side-by-side. During inference tasks, the router selects only those experts that are specialized enough to generate accurate responses, while ignoring the rest. As such, you can have the router and attention mechanism remain on the VRAM and offload the expert weights onto the system memory to run these LLMs without causing their performance to plummet.
It goes without saying that the RAM + VRAM combo needs to be enough to fit the LLM. But when you’re trying to run, say, a 25B+ model on a GPU with half as much VRAM, MoE models are absolute game-changers. To put this into perspective, I’ve got an RTX 3080 Ti with 12GB VRAM that I bought ages ago, and as long as I use the –n-cpu-moe flag with a value of 25+ in llama.cpp, my outdated GPU runs Qwen3.6-35B-A3B (Q4_K_M) at least 25 tokens/second.
Related
I ran local LLMs on a “dead” GPU, and the results surprised me
My Pascal card may not be ideal for intensive workloads, but it’s more than enough for light LLM-powered tasks
That’s a 35B model we’re talking about, and if it were a conventional LLM that needed the -ngl workaround, it’d be so slow that I’d have to measure its performance in seconds per token. I use this setup primarily for my coding tasks, and it serves as everything from my VS Code companion to the underlying LLM powering my Pi agent harness and Pulse monitoring server. Capabilities-wise, Qwen3.6-35B-A3B is as fantastic at creating fully-functional Terraform configs as it is at solving complex server malfunctions from logs and generating YAML snippets, Arduino code, and Jinja2 templates for HASS.
In a similar vein, I often deploy a Gemma-4-26B-A4B (Q4_K_M) instance on my old GTX 1080, and this decade-old Pascal card can run the LLM at 14+ t/s. Sure, it’s not fast enough for something like Home Assistant conversation models, but for running inference operations on Open Notebook, Paperless-GPT, and even agent harnesses in the background, my retired gaming companion + Gemma 4 26B A4B are more than enough. And since we’re on the subject of responsive LLMs…
The lightweight king, Gemma-4-E4B, can fit on extremely weak GPUs
And it’s significantly better than most sub-8B models
Low-parameter models are infamous for devolving into looping answers and hallucinated responses, but there are certain LLMs that manage to punch above their weight, despite their low resource footprint. Gemma-4-E4B is one of them, and while it’s definitely not as powerful as its 26B-A4B sibling, it has a neat way of cramming a lot of reasoning prowess inside a tiny LLM.
Gemma-4-E4B gives every decoder layer its own embedding table (called Per Layer Embedding) that relies on quick lookups instead of running bulky matrix operations. So, it effectively has the same footprint as a 4.5B model, even though it possesses a knowledge base comparable to 8B models.
If you’re looking for some raw numbers, my RTX 3080 Ti manages around 100 tokens/second with this lightweight model, while my GTX 1080 often manages to hit half as much, though its average lies somewhere in the 35–40 t/s range. Heck, I’ve even tried this tiny LLM on an Intel Arc A750 and an old laptop with a GTX 1060, and both of these underpowered cards manage to hit around 30ish tokens/second with this LLM. I typically use the Gemma-4-E4B running on the GTX 1080 as the centerpiece of my HASS voice assistant pipeline, but it can even handle typical home server automation tasks without throwing too many errors.
Embedding models are incredible companions for your LLMs and can run off typical CPUs
The same goes for TTS and STT models
Since this article is about home server AI, I can’t end it without mentioning a handful of other models that you can run on cheap hardware. Take embedding models, for example, which have the sole purpose of turning text, images, and other inputs into vectors and mapping them into a vector space, where similar elements lie close together. Then, apps such as Open Notebook and Blinko can use these maps to add context to my queries, thereby providing some much-needed information to my LLMs during inference tasks.
The best part? Barring a few, most embedding models are extremely lightweight and can run on simple processors. I typically use nomic-embed-text-1.5, and it meshes well with every app that supports embedding models.
Finally, text-to-speech and speech-to-text models deserve a special mention. Unlike image or video generators, many of the popular TTS and STT models don’t need a powerful graphics card. I’ve run mine off a mere Xeon CPU in the past, and only switched to a Docker-based Speaches instance that connects to my GTX 1080 when I wanted extremely quick response times for my local Open WebUI voice assistant setup.