They said their’s is “comparable with the 8-bit models”. Its all tradeoffs. It isn’t clear to me where you allocate your compute/memory budget. I’ve noticed that full 7b 16 bit models often produce better results for me than some much larger quantied models. It will be interesting to find the sweet spot.
So are more bits less important than more paramters? Would a higher paramter or higher bit count matter more if the models ended up the same size?
I canceled when they raised the price. I am DONE with all the price increases the past year.
What’s crazy to me is they are using tons of copyrighted data to train but put in a statement saying you can’t use LLama outputs to train other models.
Picture seems to be gone? I don’t know how lemmy works. But here was the picture
If you don’t have a good GPU then just use gpt4all
Apparently I am an idiot and read the wrong paper. The previous paper mentioned that “comparable with the 8-bit models”
https://huggingface.co/papers/2310.11453