When I first got into local LLMs nearly 3 years ago, in mid 2023, the frontier closed models were ofcourse impressively capable.

I then tried my hand on running 7b size local models, primarily one called Zephyr-7b (what happened to these models?? Dolphin anyone??), on my gaming PC with 8GB AMD RX580 GPU. Fair to say it was just a curiosity exercise (in terms of model performance).

Fast forward to this month, I revisit local LLM. (Although I no longer have the gaming PC, cost-of-living-crisis anyone 😫 )

And, the 31b size models look very sufficient. #Qwen has taken the helm in this order. Which is still very expensive to setup locally, although within grasp.

I’m rooting for the edge-computing models now - the ~2b size models. Due to their low footprint, they are practical to run in a SBC 24/7 at home for many people.

But these edge models are the ‘curiosity category’ now.

  • SuspciousCarrot78@lemmy.worldEnglish
    1·
    10 小时前

    I was playing around with a tiny amount earlier today (I use ProtonMail, so I figured why not).

    I can’t tell much about it. It seems very…safety theater / personality removed.

    Any idea of what models they use now? I get a feeling that the main brain is 14B (based on how it responds to questions / drops nuance).