So my relevant hardware is:
GPU - 9070XT
CPU - 9950X3D
RAM - 64GB of DDR5
My problem is that I can’t figure out how to get a local LLM to actually use my GPU, I tried Ollama with Deepseek R1 8b and it kind of vaguely ran while maxing out my CPU and completely ignoring the GPU.
While I’m here model suggestions would be good too, I’m currently looking for 2 use cases.
- Something I can feed a document too and ask questions about that document (Nvidia used to offer this) To work as a kind of co-GM to quickly reference more obscure rules without having to hunt through the PDF.
- Something more storytelling oriented that I can use to generate background for throwaway side NPCs when the players innevitably demand their life story after expertly dodging all the NPCs I actually wrote lore for.
Also just an unrelated asside, Deepseek R1 8b seems to just go into an infinite thought loop when you ask it the strawberry question which was kind of funny.
llama.cpp
The Only Inference Engine You’ll Ever Need™
I found this guide which seems very comprehensive but has a few sections where it assumes knowledge I don’t have and doesn’t suggest a clear route by which to gain said knowledge.
For the section just following “Grab the content of SmolLM2 1.7B Instruct” I assume it boils down to run this prior program called MSYS and run this command through it? “GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct”
That’s for quanting a model yourself. You can instead (read that as “should”) download an already quantized model. You can find quantized models from the HuggingFace page of your model of choice. (Pro tip: quants by Bartowski, Unsloth and Mradermacher are high quality)
And then you just run it.
You can also use Kobold.cpp or OpenWebUI as friendly front ends for llama.cpp
Also, to answer your question, yes.