What does a local AI server take?

The short answer so far: 24GB of graphics memory looks like the sensible starting point for serious work, and memory is the thing to budget for, not the processor. I haven't bought anything yet. These are my research notes, so treat them as that.

The reason I'm looking is on the Dev Partners blog, in what happens to the tools if the AI bubble bursts. In short, we use AI every day, and I'd like to be less exposed to one supplier's price list. GitHub moved Copilot to usage-based billing in June, saying the old model was "no longer sustainable". I doubt it'll be the last.

The little mini PC in the office has 12GB of memory and no proper graphics card, so that's out.

Decide what the job is first

This changes everything. An assistant that summarises documents and answers staff questions is one job, and I've written about when that's worth doing. A coding agent that reads a whole project and works for an hour is another, and it's far hungrier. That's the one I'm researching.

It also matters how many people use it at once. Most of the advice you'll read assumes one person at one desk.

How much memory

The whole model has to fit in memory, ideally on the graphics card, with room left for what it's working on. Kilo's comparison of local coding models is the most useful guide I've found, and it breaks down roughly like this:

8GB. Small models. Useful for bounded tasks, not for an agent running loose in a codebase.
12 to 16GB. Gemma 4 12B, a step up, with limits.
24GB. Qwen 3.6 27B at Q4, with about 64K of context. Kilo rates this the strongest general pick for one card.
48GB or more. Qwen3-Coder-Next. Kilo mentions two RTX 3090s for about 128K of context.

The "Q4" means the model has been compressed to fit, trading a little quality for a lot of memory. Those are guide figures and other guides disagree, so I'd check them against the model cards before spending a penny.

Licences are better than they used to be. Qwen3.6-27B and Qwen3-Coder-Next are both Apache 2.0, so you can use them commercially.

What to buy

There seem to be three routes.

A proper graphics card. A used RTX 3090 keeps coming up, because it has 24GB. Used prices have been rising, so look at sold UK listings rather than trusting me. New is painful. An RTX 5090 has 32GB and is about £4,300 here, against an official £1,799. It's the same memory squeeze that's pushing up hosting bills.

Shared memory machines. Mini PCs with AMD's Ryzen AI Max+ 395 and 128GB, or Macs, let the model use the main memory. That fits much bigger models. My understanding is they run slower than a good graphics card, and I'd want to test that with a real job. They aren't immune to prices either: Framework put its desktops up by as much as $460 because of memory costs.

Renting. You can hire a GPU server by the hour. It's a cheap way to test before buying, and it's the first thing I'd do. The catch is that your code is on someone else's machine again, which rather depends on why you wanted this.

Don't forget the boring bits. A 5090 can draw 575W on its own, so think about the power supply, heat and noise, and where this thing will live. Nobody wants it under their desk in July.

The software

Ollama looks like the easy way in, and Open WebUI gives you a chat screen on top. Two settings stand out from the docs.

The first is context, which is how much the model can hold in its head at once. Ollama's default is 4K on a card under 24GB and 32K up to 48GB. That's fine for chat, but too small for a coding agent. OpenCode's docs suggest starting at 16K to 32K if tool calls aren't working, and I suspect a lot of "local models are rubbish" comes from this one setting.

The second is sharing. Ollama handles one request per model at a time by default. You can raise that, but it says the RAM needed scales with parallel requests multiplied by context length. So four people at 32K isn't a 32K problem, it's a 128K one. For several people at once, the guides point to vLLM, and for more control to llama.cpp.

Would I expect it to match the cloud?

No. Kilo says hosted models often give better speed, context and tool reliability for demanding agents, and I'd go in expecting that. My plan would be a mix: the best hosted models for the hard thinking, and a local one for the routine work and anything that shouldn't leave the building.

Security matters too. Ollama shouldn't be open to the internet. I'd keep it behind Tailscale or similar, like anything else on a private network.

What I'd do next

Rent a 24GB card for a day or two.
Give Qwen3.6-27B a real job on a real codebase, with a sensible context setting.
Measure speed, and how often I have to step in.
Do the sums against what we're paying now, before buying anything.

I'll write up what I find, including if the answer turns out to be "not worth it". If you're looking at the same thing for your own business, the private AI page has how I'd set it up, and for bespoke software on top, the team at Dev Partners can help.