Just a stranger trying things.

  • 5 Posts
  • 39 Comments
Joined 1 year ago
cake
Cake day: July 16th, 2023

help-circle












  • There is a way to place the secret file (corresponding to the password) on a dedicated USB stick and have a script attempt to Mount it at boot to unlock the partition. If the USB stick is not found, it will revert to the password prompt. Perhaps this is the best of both?

    Make sure not to leave the USB stick plugged in, but rather only take it and and plug it in to boot then safely store it once booted, otherwise you are probably defeating the purpose of having an encrypted partition to begin with.

    I’ll add a link to read more about it shortly.

    Edit: here is one example to set it up (including to auto-decrypt ZFS) https://www.youtube.com/watch?v=7xOLxCwdi-I


  • Well yes, but also no.

    Whenever you search for a solution to your problem, it stems from the realization that something is a problem. But sometimes, you have a thing which has been done for a longtime, it was a problem with no solution and you’ve had to accept that. How would you determine one day that things can be done differently and better without constantly reevaluating everything? It’s not realistic.

    In my view, it is a perfectly reasonable question to ask “what problem does waydroid solve?” To figure out if you have that issue and you didn’t know of this solution.

    Sorry, just my 2 cents.


  • You’re right, but the model is also not quantized so is likely to be in 16bit floats. If you quantize it you can get substantially smaller models which run faster though may be somewhat less accurate.

    Knowing that the 4 bit quantized 8x7B model gets downscaled to 4.1GB, this might be roughly 3 times larger? So maybe 12GB? Let’s see.

    Edit: sorry those numbers were for Mistral 7B, not mixtral. For Mixtral, the quantized model size is 26GB (4 bits), so triple that would be roughly 78 GB. Luckily, being an MoE, not all of it has to be loaded simultaneously to the GPU.

    From what I recall, it only uses 13B parameters at once, so if we compare that to codellama 13B, quantized to 4 bits, that is 7.4GB, so triple that would be 22GB, so would require a 24GB GPU. Someone double check if I misunderstood something.

    24GB GPUs include the AMD 7900 XTX and the nvidia RTX 4090 (Ti), non-mobile.