I want to host some LLM’s locally and use more advanced models. Since new hardware is out of the question, I think I should be able to pull something off buying some yesteryear equipment on ebay etc. Did anybody attempt such a project? Does it scale horizontally? (I.e. can I connext two boxes to overcome single box slowness?)
It can vary a lot based on what qwen model you want to run, but generally the 27b dense or 35b MoE are currently the best balance between size and capability afaik.
If you can run two 16GB cards you can pretty much max out the context on the 27b model, but a single card like the 3060 12gb could still work well on the 35b MoE model with the excess spilling into system memory.
I saw in another comment you have cards from the 2010’s but if they don’t have at least 8gb I wouldn’t even bother