Running large language models (LLMs) on your local machine has become increasingly popular, offering privacy, offline access, and customization. Ollama is a ...
The 3bpw weights are 13 GB, say another 1.5GB for some q5_q4 context, and you are looking at 14.5GB-15GB or so. It will be tight, but it will be leagues smarter than 14Bs.
24B Mistral models will fit much more easily. No need to CPU offload those on a 16GB card, you just need to be careful with your settings.
I got both IK and Croco built and running on Vulkan, but they both seem abysmally slow compared to Kobold, whether I run one of the GGUFs I already had or a trellis quant I just made. It’s using the GPU, so not sure why. I’ll keep an eye on the projects and try them again in the future.
Is it 3000 series or newer?
If so, with exllamav3, you can squeeze 32Bs in that 16GB card with relatively little loss. For instance: https://huggingface.co/turboderp/EXAONE-4.0-32B-exl3/tree/3.0bpw
The 3bpw weights are 13 GB, say another 1.5GB for some q5_q4 context, and you are looking at 14.5GB-15GB or so. It will be tight, but it will be leagues smarter than 14Bs.
24B Mistral models will fit much more easily. No need to CPU offload those on a 16GB card, you just need to be careful with your settings.
I have an rx 6800. Looks like exllamav3 doesn’t support AMD cards yet… I’ll keep an eye on it, so I can try it when ROCm or Vulkan support is added.
Ah. You can still run them in exllamav2, but you’re probably better off with ik_llama.cpp then:
https://github.com/ikawrakow/ik_llama.cpp
It supports special “KT” quantizations, aka trellis quants similar to exllamav3, and will work with vulkan (or rocm?) on your 6800.
Quantizing yourself is not too bad, but if you want, just ping me, and I can make some 16GB KT quants, or point you to how to do it yourself.
It’s also a good candidate for Qwen3 30B with a little CPU offloading. ik_llama.cpp is specifically optimized for MoE offloading.
I got both IK and Croco built and running on Vulkan, but they both seem abysmally slow compared to Kobold, whether I run one of the GGUFs I already had or a trellis quant I just made. It’s using the GPU, so not sure why. I’ll keep an eye on the projects and try them again in the future.