Do you host your own ML / AI / LLM? What do you use, and what do you use it for?

  • atzanteol@sh.itjust.works
    link
    fedilink
    English
    arrow-up
    1
    ·
    edit-2
    1 day ago

    Yeah - I’ve been playing around more with the Qwen3-Coder-30B-A3B-Instruct MoE model and it’s still quite… Meh. I’ve been using llama.cpp and I’ve tried a bunch of tuning. It works and performs well enough (15t/s) but the output is just garbage. I can do some simple coding but I’m finding I’m fighting with it more than if I just wrote the code myself. Maybe I just have standards that are too high. Claude Opus 3.7 is just in an entirely different league…