Running large language models (LLMs) on your local machine has become increasingly popular, offering privacy, offline access, and customization. Ollama is a ...
Try any of Latitude’s series. They’re ‘uninhibited’ dungeonmaster models, but they should be smart enough (and retain enough of that personality) for some flexibility:
It’s trained from Qwen A3B base, not instruct. Base models usually don’t have the severe ChatGPT-isms you describe, hence while I haven’t personally tried this model, it seems promising. And it should be fast on your Xeon.
Random thing, I did not get a notification for this comment, I stumbled upon it. This happens all the time, and it makes me wonder how many replies I miss…
I don’t run A3B specifically, but for Qwen3 32B Instruct I put something like “vary your prose; avoid repetitive vocabulary and sentence structure” in the system prompt, run at least 0.5 DRY, and maybe some dynamic sampler like mirostat if supported. Too much regular rep penalty makes it dumb, unfortunately.
But I have much better luck with base model derived models. Look up the finetunes you tried, and see if they were trained from A3B instruct or base. Qwen3 Instruct is pretty overtuned.
They may have been based on Instruct. It left such a bad impression, I didn’t play around with them much. Good to know for the future, though. I haven’t used DRY or mirostat really in the past, but I’ll try them next time I look at the Qwen3s.
Honestly I don’t use Qwen3 instruct unless it’s for code or “logic.” Even the 32B is soo dry and focused on that, and countering it with sampling dumbs it down.
Not sure if it’s too big, but I have been super impressed with Jamba 52B. It knows tons of fiction trivia and writing styles for such a “small” model, though I haven’t tried to manipulate its prompt for writing yet. And it’s an MoE model like A3B.
Interesting. I hadn’t heard of this one before, but the design sounds innovative. The biggest I run is 35B or 7x8B, but I’ll have to try and check it out.
Jamba is a killer model flying under the radar, though it does have a quirk I more recently discovered: no prompt caching in llama.cpp (yet).
If you have a 24GB GPU you can cram Nemotron 49B in it with no offloading, including the new reasoning version. It’s a monster at STEM stuff, and I can upload my special quantization (3bpw, with 4bpw KV heads, exllamav3) if you ask.
Qwen 30B coder is ridiculously fast for how smart it is at coding, just came out today…
TBH the last week or two has been nuts with new releases.
But FYI if you are looking for pure prose quality, I still use EVA Gutenberg 32B (based on Qwen 2.5 base) and Jonboro’s brand new QWQ 32B fine tune, as new models have not surpassed them IMO. But for creative writing, I tend to write novel style instead of multi turn, so YMMV.
I only have a 16GB card, and my CPU is new enough that it’s better to offload some layers of all but 7-8B models, so I haven’t tried exllama, but you’re making me think I should, if only for comparison.
I like Qwen 2.5 based models in the 14B size range, but I don’t think I tried the bigger ones. I tried the QWQ and didn’t really like it, but I haven’t seen this new one. You’ve given me a whole list of things to try, so thanks.
The 3bpw weights are 13 GB, say another 1.5GB for some q5_q4 context, and you are looking at 14.5GB-15GB or so. It will be tight, but it will be leagues smarter than 14Bs.
24B Mistral models will fit much more easily. No need to CPU offload those on a 16GB card, you just need to be careful with your settings.
Big thanks! I’m always looking for recommendations. I’ll check them out. It’s going to take me some time, since it’s very subjective. I used to look at numbers and scores, but they just don’t mean a lot. So I need to use every one for a while and see whether I like what they write. The MoE model is quite an improvement in speed already. It’s 3 times faster…
Turn it into an ik_llama.cpp k quant, and you should be able to squeeze even more out!
FYI you can find more models like this by looking up a base model (not the instruct) of interest, then clicking on the ‘finetunes’ category. For example:
One other thing. A lot of folks (like me) tend to use the base models, not instruct finetunes, in completion mode since they tend to be devoid of AI slop. But you have to prompt them different than a regular LLM: instead of multi turn conversation, you write out a starting block of text for them to ‘latch onto’, and get them to continue it from your cursor.
But prompt them right, and they will do literally whatever you want, devoid of any sycophancy or guardrails.
Mikupad is great for this since it also shows token probablities. So you can, for instance, click on a critial word, and see what ‘choices’ the LLM was considering internally as a set of branches, and regenerate from there.
Try any of Latitude’s series. They’re ‘uninhibited’ dungeonmaster models, but they should be smart enough (and retain enough of that personality) for some flexibility:
https://huggingface.co/LatitudeGames
Perhaps more optimally for your hardware, try this:
https://huggingface.co/Gryphe/Pantheon-Proto-RP-1.8-30B-A3B
It’s trained from Qwen A3B base, not instruct. Base models usually don’t have the severe ChatGPT-isms you describe, hence while I haven’t personally tried this model, it seems promising. And it should be fast on your Xeon.
The 30B-A3Bs I’ve tried have been suuuuuuuper repetitive. Do you have any specific settings to recommend to get them to work well?
Random thing, I did not get a notification for this comment, I stumbled upon it. This happens all the time, and it makes me wonder how many replies I miss…
I don’t run A3B specifically, but for Qwen3 32B Instruct I put something like “vary your prose; avoid repetitive vocabulary and sentence structure” in the system prompt, run at least 0.5 DRY, and maybe some dynamic sampler like mirostat if supported. Too much regular rep penalty makes it dumb, unfortunately.
But I have much better luck with base model derived models. Look up the finetunes you tried, and see if they were trained from A3B instruct or base. Qwen3 Instruct is pretty overtuned.
They may have been based on Instruct. It left such a bad impression, I didn’t play around with them much. Good to know for the future, though. I haven’t used DRY or mirostat really in the past, but I’ll try them next time I look at the Qwen3s.
Honestly I don’t use Qwen3 instruct unless it’s for code or “logic.” Even the 32B is soo dry and focused on that, and countering it with sampling dumbs it down.
Not sure if it’s too big, but I have been super impressed with Jamba 52B. It knows tons of fiction trivia and writing styles for such a “small” model, though I haven’t tried to manipulate its prompt for writing yet. And it’s an MoE model like A3B.
Interesting. I hadn’t heard of this one before, but the design sounds innovative. The biggest I run is 35B or 7x8B, but I’ll have to try and check it out.
Jamba is a killer model flying under the radar, though it does have a quirk I more recently discovered: no prompt caching in llama.cpp (yet).
If you have a 24GB GPU you can cram Nemotron 49B in it with no offloading, including the new reasoning version. It’s a monster at STEM stuff, and I can upload my special quantization (3bpw, with 4bpw KV heads, exllamav3) if you ask.
Qwen 30B coder is ridiculously fast for how smart it is at coding, just came out today…
TBH the last week or two has been nuts with new releases.
But FYI if you are looking for pure prose quality, I still use EVA Gutenberg 32B (based on Qwen 2.5 base) and Jonboro’s brand new QWQ 32B fine tune, as new models have not surpassed them IMO. But for creative writing, I tend to write novel style instead of multi turn, so YMMV.
I only have a 16GB card, and my CPU is new enough that it’s better to offload some layers of all but 7-8B models, so I haven’t tried exllama, but you’re making me think I should, if only for comparison.
I like Qwen 2.5 based models in the 14B size range, but I don’t think I tried the bigger ones. I tried the QWQ and didn’t really like it, but I haven’t seen this new one. You’ve given me a whole list of things to try, so thanks.
Is it 3000 series or newer?
If so, with exllamav3, you can squeeze 32Bs in that 16GB card with relatively little loss. For instance: https://huggingface.co/turboderp/EXAONE-4.0-32B-exl3/tree/3.0bpw
The 3bpw weights are 13 GB, say another 1.5GB for some q5_q4 context, and you are looking at 14.5GB-15GB or so. It will be tight, but it will be leagues smarter than 14Bs.
24B Mistral models will fit much more easily. No need to CPU offload those on a 16GB card, you just need to be careful with your settings.
Big thanks! I’m always looking for recommendations. I’ll check them out. It’s going to take me some time, since it’s very subjective. I used to look at numbers and scores, but they just don’t mean a lot. So I need to use every one for a while and see whether I like what they write. The MoE model is quite an improvement in speed already. It’s 3 times faster…
Turn it into an ik_llama.cpp k quant, and you should be able to squeeze even more out!
FYI you can find more models like this by looking up a base model (not the instruct) of interest, then clicking on the ‘finetunes’ category. For example:
https://huggingface.co/models?other=base_model%3Afinetune%3AQwen%2FQwen3-30B-A3B-Base&sort=modified
https://huggingface.co/models?other=base_model%3Afinetune%3Amistralai%2FMistral-Small-24B-Base-2501&sort=modified
This one’s also the perfect size for you, but has no finetunes yet: https://huggingface.co/baidu/ERNIE-4.5-VL-28B-A3B-Base-PT
One other thing. A lot of folks (like me) tend to use the base models, not instruct finetunes, in completion mode since they tend to be devoid of AI slop. But you have to prompt them different than a regular LLM: instead of multi turn conversation, you write out a starting block of text for them to ‘latch onto’, and get them to continue it from your cursor.
But prompt them right, and they will do literally whatever you want, devoid of any sycophancy or guardrails.
Mikupad is great for this since it also shows token probablities. So you can, for instance, click on a critial word, and see what ‘choices’ the LLM was considering internally as a set of branches, and regenerate from there.