Running LLMs on an RX 580

Tue, 20 Oct 2026 00:00:00 GMT

An AMD RX 580 is an inexpensive old gaming GPU that is mostly considered obsolete now, but it is still useful for running models locally, as long as one has realistic expectations.

Not too long ago, I bought an old Sapphire Nitro+ RX 580 8 GB to turn some disused parts into a low-spec gaming PC. It ended up on the shelf before long, at the lowest rung of the GPU hierarchy in my house. But the motherboard in my primary ATX PC had a second PCIe slot going unused, so I put the card back into service for running local models.

Caveats

The RX 580 is a mid-range gaming GPU from 2017. Mine has 8 GB of VRAM, which makes it the high-end version of the card for the time. It was a very competent gaming GPU for several years, but it is now mostly obsolete. For local AI it is also not top of the line: 8 GB is considerably memory-constrained for dense models — Qwen3.8 27B’s smallest quant is 13.1 GB, so it cannot fit without offloading layers to the CPU. Its memory bandwidth is 256 GB/s, only about 1.3× a MacBook Pro M2 Pro’s 200 GB/s, which constrains inference speed. And as an AMD card it has no CUDA, so local image-generation tools need workarounds.

Setup

On Linux the card is no longer supported by AMD’s ROCm stack, so you will be using Vulkan. Setup was mostly a breeze, and with quantized versions of popular models at 12B parameters or fewer, the RX 580 is a very usable tool.

I used llama.cpp running as a service to expose the card as a general-purpose inference device on my network. --models-max 1 is essential for keeping llama.cpp from running out of VRAM. llama.cpp will automatically swap models defined in the presets.ini file as needed. The following is the systemd service definition I went with.

[Unit]
Description=llama.cpp model router 
After=network.target

[Service]
Type=simple
User=[user]
WorkingDirectory=/home/[user]/git/llama.cpp
# pin to the AMD Vulkan ICD so the RX 580 is always Vulkan0
Environment=VK_DRIVER_FILES=/usr/share/vulkan/icd.d/radeon_icd.json
Environment=MTMD_BACKEND_DEVICE=Vulkan0
# access /dev/dri/renderD* without a local seat ACL
SupplementaryGroups=render
ExecStart=/home/[user]/git/llama.cpp/build/bin/llama-server \
    --models-preset /etc/llama-gemma/presets.ini \
    --models-max 1 \
    --host 0.0.0.0 \
    --port 8080
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

MTMD_BACKEND_DEVICE is not optional. --device only places the LLM; the multimodal projector defaults to device 0, which on a multi-GPU box is the wrong card, and the load fails with a hard out-of-memory abort rather than a clean error.

Methodology

I chose Ornith 1.5 9B and Mistral Nemo 12B for their complementary strengths considering the VRAM limit. For this series of benchmarks, Ornith was doing VLM work on a fixed set of 12 images. Nemo was doing text inference, running a fixed set of 6 prompts. Requests used a nonce per call to defeat KV-cache reuse, with temperature=0 and max_tokens 250 (VLM) / 300 (text), and server-reported timings and GPU power were sampled per request.

Baseline at the production quants

The first job is fitting the model into 8 GB. I picked quantized versions of both models, used KV cache quantization where it helped, and for the multimodal model fit the mmproj into VRAM as well.

Model (quant)	mmproj	Context (tokens)	Other settings
Ornith 1.5 9B (i1-Q4_K_S)	Q8_0	49,152	flash-attn on
Mistral Nemo Instruct 2407 12B (IQ4_XS)	—	12,288	q8_0 K/V cache, flash-attn on

I noticed in the process of benchmarking that power consumption was not hitting the card’s rated TDP. That alone doesn’t prove anything, but behavior under a power cap does: pulling the cap down to 110 W left decode nearly unchanged (−9% on Ornith, −2% on Nemo) while prefill fell 26–34%. Decode is bandwidth-bound; prefill is compute-bound.

Model	Prefill tok/s	Decode tok/s	Power (W, mean)	Effective BW
Ornith 1.5 9B Q4_K_S	117.3	23.68	161.5	126.7 GB/s (49.5%)
Mistral Nemo 12B IQ4_XS	105.7	18.77	160.0	126.6 GB/s (49.4%)

Those numbers use the quants I chose for fit and quality. The obvious next question is whether the quant format is itself leaving speed on the table. K-quants and IQ-quants are the sensible default on NVIDIA cards, but Polaris reports int dot: 0, fp16: 0 and no matrix cores, so there is no fast shader path for their unpacking and the dequant cost may dominate.

Quant format: K-quant vs Q4_0

I benchmarked the formats directly with llama-bench (pp512 / tg128, three runs, flash attention on, q8_0 KV) and then confirmed the winner end-to-end.

Model	Quant	Size	Prefill tok/s	Decode tok/s	Decode vs production
Ornith 1.5 9B	Q4_K_S (production)	4.97 GiB	214.0	24.5	—
Ornith 1.5 9B	Q4_0	4.95 GiB	221.6	29.8	+22%
Mistral Nemo 12B	IQ4_XS (production)	6.27 GiB	151.0	19.4	—
Mistral Nemo 12B	Q4_0	6.60 GiB	157.6	23.5	+21%

Legacy Q4_0 dequantizes much faster than every K- or IQ-quant on this card. At roughly the same size it decodes about 22% faster, and it beats Q3_K_M even though Q3_K_M is 13–15% smaller. Bytes saved do not translate because dequantization cost dominates. IQ4_XS is not slower than the K-quants here, so the gain is Q4_0 being cheap rather than IQ being expensive.

End-to-end on the same harness, the gain held: Ornith went from 23.38 to 28.20 tok/s (+20.6%) and Nemo from 18.83 to 22.77 (+20.9%), for 1–4 W more.

The catch is quality. On wikitext-2 (100 chunks, 512 context) Q4_0 costs 2.7% higher perplexity on Ornith (10.571 → 10.854) and 4.9% on Nemo (6.659 → 6.988). A small spot check (five images and three prompts) showed no visible degradation, which is exactly why the perplexity run matters: the sample was too small to see a real but small loss.

So Q4_0 isn’t a free win. It is about +21% decode for +2.7–4.9% perplexity, which is a good trade for bulk captioning or judging and a bad one where the last bit of fidelity matters. Q8_0 is not an option for either model at this size: 9.5 GB and 13 GB do not fit in 8 GB.

Free improvement: a newer Mesa

My system had Mesa 25.2.8 installed when I started, and moving to Mesa 26.2.3 produced a free bump in speed without any penalty in quality. Both sides of each arrow are reruns from the same session.

Model	Quant	Prefill tok/s	Decode tok/s	PPL cost
Ornith 1.5 9B	Q4_K_S	117.7 → 122.4 (+4%)	23.38 → 26.48 (+13%)	—
Ornith 1.5 9B	Q4_0	119.4 → 126.1 (+6%)	28.20 → 28.81 (+2%)	+2.7%
Mistral Nemo 12B	IQ4_XS	103.7 → 103.8 (0%)	18.83 → 18.49 (−2%)	—
Mistral Nemo 12B	Q4_0	92.3 → 99.8 (+8%, noisy)	22.77 → 23.08 (+1%)	+4.9%

Prefill improves by up to ~8%; the big decode win is on the K-quant. Perplexity is untouched (the same Ornith model measured 10.5705 under both drivers, identical to four digits), so this is a pure win.

It also changes the quant decision. On Ornith, Q4_0’s advantage over Q4_K_S shrinks from +22% to about +9% (28.81 vs 26.48) for the same +2.7% perplexity, so staying on Q4_K_S is now the better trade. Nemo is unaffected by the driver, so Q4_0 still buys about +25% (23.08 vs 18.49) for +4.9% perplexity.

Power consumption

The RX 580 has a 185 W reference TDP (this card reports a 183 W power cap), and it’s not as power-efficient as more recent mid-range cards. I found that undervolting was an effective way to cut power with almost no penalty on prefill or decode. Capping power, by contrast, hits prefill harder, because prefill is the compute-heavy phase and decode is mostly waiting on memory.

These runs are from the same post-reboot state at 1158 MHz, where the card would no longer clock any higher, so the baseline here is deliberately not the 122.4 tok/s row above. The comparison is voltage-for-voltage at a fixed clock.

Model	`max_voltage`	Prefill tok/s	Decode tok/s	Busy power (W)	tok/s/W
Ornith 1.5 9B Q4_K_S	1106 mV (stock)	110.5	25.12	152	0.166
Ornith 1.5 9B Q4_K_S	950 mV	111.1	25.61	110.9 (−27%)	0.231 (+39%)
Mistral Nemo 12B IQ4_XS	1106 mV (stock)	97.9	18.03	131	0.138
Mistral Nemo 12B IQ4_XS	950 mV	96.3	18.32	99.5 (−24%)	0.184 (+33%)

The card will not go below 950 mV under load: the memory runs at 2100 MHz during decode and its lowest voltage point is 950 mV, so the core rail cannot drop further while memory is busy. Settings below 950 mV only apply in brief low-memory-clock moments and add instability for no saving.

Idle power

The card is quite power-hungry at idle, consuming about 30 W when doing nothing. It’s worth exploring this further, because I was not able to get idle consumption lower than that with my system configuration.

Worthwhile?

The configuration I settled on depends on the workload:

Workload	Choice	Why
VLM, quality matters	Ornith 1.5 9B Q4_K_S	Mesa 26.2.3 gives +13% decode at no quality cost; Q4_0 is now only +9% for +2.7% perplexity
Text, throughput matters	Nemo 12B Q4_0	+25% decode for +4.9% perplexity
Text, fidelity matters	Nemo 12B IQ4_XS	the production quant
Everything	950 mV at the stock 1158 MHz	−24–27% busy power with no measurable speed loss

Should you buy one of these cards for the purpose? With GPU prices these days, it might be tempting. I wouldn’t recommend building a PC around one, but if you’ve got one lying around and a free PCIe slot, it is still worth using.

gradient.garden