Quick link: https://github.com/dwash96/aider-ce

I have not tested it, but Zluda are are drop-in cuda replacement for non-nvidia gpus. The speed should be great. You could check if your goto card is supported…

A backup drive ? dotfiles (.config) ? /tmp /var/log /boot ? map it into your home for something temporary - Downloads ? Slow audio files ? If you like to tinker for fun: btrfs / bcachefs could perhaps utilize it for something, like automatically offloading unused/slow files to the slow usb ? Raid ? ;) Create a partition and let zram swap compressed unused memory pages to the usb partition (I think it keeps pages compressed now ?). You can also run ‘usb over ethernet’, so another machine can use it directly - not sure why tho ;)

On a side note. I still use one of my first usb 1.? drives with 32mb in my old router. 32mb is still 12mb more than my first hard drive, and it fits well with the small openwrt packages. It even has a little slide button to select ‘floppy drive’ mode and a physical ‘readonly’ mode. I wonder what the smallest/oldest usb stick people still use, are ?

Agree. “Are you perhaps too angry?”, “The HELL I’m not! F*ck off !”, “alrighty then”…

Compute prices are not going down. If you want a budget friendly compute, then you may need to look for NPU’s. They are slower, and is unfortunately also getting more expensive pr TOPS. Electricity prices in the West are going up as a low output gets bought up by datacenters instead. The newest GPU’s are more efficient in re. to power pr Top, but they are not budget friendly.

I have not seen a recent comparison of GPU vs Npu; TOPS vs price vs power consumption, but @ aliexpress, I saw a small 40 TOPS Npu as a nvme stick with 16gb ram that draws 10 watts, or so (search for ‘ai accelerator nvme’). This little thing can scan files 24/7 and your i5 can help out in peaks. Afaik you can run a distributed model that also runs in, and uses your i5 memory/compute, so if you max out i5 memory, perhaps the combined compute is enough for a larger model ? Maybe a few Npu sticks can work together on the same model ?

Alternatively, you could look for the next version of a Huawei gpu card (first had baby kinks afaik), or one of the other upcoming producers from China. They’ll come faster and faster, but are earmarked for local consumption first.

Another suggestion is to buy one of the old P40/80 (from the ‘pascal’ chipset, I think) (or was it k40/80 ??). They should still support a descent range of modern quantization needs and often have 24gb ram. A refurbished miner card cost around 50-70$+ - cant remember exactly tho.

Lastly, you could try something different. If the i5, with enough memory, can run a large model slowly, you could add a dedicated KV cache, so most of the tokens won’t need to be recalculated. Memory and bandwidth are the most important here, but any old server could be upgraded to be a dedicated KV cache server (might need a network upgrade tho!).

Anyway, Ideas are easy and cheap. If I were you, I would ask an AI for a little python app where you can add a product and it returns a graph where it compares with other products and show optimality over time given prices/power and Tops - Gpu vs Npu. Good hunting…

They are just trying to remove all the nonsense western propaganda. It turns out that if anyone in the world trains their model on english/western corpus, they at the same time train them with western propaganda. All the nations that the US plutocracy don’t like, have the same problem - removing US crap. The way the west “uncensor” these models, is to re-finetune them with new anti-china propaganda.

We can say that about Capitalism. Anyone wants to spread their garbage in an attempt to survive this combattive ideology. Before AI, 90% of all internet information were Capitalist slop, but now its all AI’s fault …smh…

I haven’t followed that closely lately, but KoljaB on github had some interesting repos. Also ‘livekit’ is new and afaik creates an audio agent. No personal experience though…

How do you remove all the propaganda they are already trained on ? You reject Deepseek, but you are just allowing yourself to being manipulated by a throng of old propaganda/censorship from the normal internet - garbage manipulative information that is stored in the weights of your ‘uncensored’ model. ‘Freeing’ a model to say “shit”, is not the same as an uncensored model that you can trust. I think we need a dataset cleansed from the current popular ideology and all propaganda against ‘wevil nationstates’ that have just rejected the western/US dominance (giving the middle-finger to western oligarchs)…

? they are just trying to protect themselves from western propaganda, just as all nations should. Its working great. Everything you’ve heard about China comes from the western propaganda apparatus, and I doubt you have discovered how insanely polluted the western information sphere is - amongst other things, propaganda towards enemies of the US Plutocracy. All models are trained on that nonsense, and you can’t say “Hi” to a western model without being influenced by western ideological pollution/propaganda…

And putting super intelligence in the hands of Oligarchs/Psychopaths - sorry, business people, are just not cool… People are plenty oppressed by them already…

MSP’s ??

Thanks for Lemonade hint. For Ryzen AI: https://github.com/lemonade-sdk/lemonade (linux=cpu for now)

Sounds more like Nato and the western elite want to get in to Iran’s information sphere, than to let Iranians access western internet and be ‘free’. Iranians are during external attacks against their nation, protected from the information war - just like borders protect from attacks. I’m not assisting the Western elite. I’m out…

I slept in my first computer - and worked as verbal RAM (first VRAM!) 28 hours a day !

“…and when you tell this to young people today, they won’t believe you !” - Monty Python.

I don’t know, but allow me a soft rant about the ‘distributed’ part;

Couldn’t selfhosters try to ‘organize’ and share these burdens ? Why pay for external cloud backup, or anything, when selfhosters can just help each other storing parts of others backup. Then everyone have an automatic back-up.

The tools seem to be there, but Its like there are all these super-skilled infra-structure selfhosters that know everything about self-hosting solutions, but they lack the self-organizing ability to sollve these typical - and a bit trivial, lets be honest - problems in a full p2p style. The result looks to be that all self-hostings solution above the threshold of an average individual selhoster, have to be done in the cloud, and everyone is ‘siloed’ in their own mini data center.

But, with existing tools, AI and a little imagination, it shouldn’t be too hard to ‘organize’ a little (though here) design a self-hosting p2p backup solution from existing tools. …or a solution for most of the other cloud services we still rely on…

But maybe its something else ? …to me, it just seems unnecessary for a high expertise self-hoster community that - when combined - are an absolute gargantuan cloud-service infrastructure …to still have such basic capacity issues (no offense meant to op, or anyone!), and still have so high reliance on cloud services. Seems odd to me…

I don’t value these papers very highly. Before they are even published/peer reviewed, the landscape have changed. Models get better quickly, agentic frameworks too, and their code even more. But good to have a ball-park measurement tho.

If we see what is coming from the latest papers, (‘discover ai’ on the tube), we have only scratched the surface of how this is going to pan out. Buckle up…

Humor is rather subjective, but if you feel you have THE material for “Humor bot 2000”, then there are ways of fine-tuning your own model with exactly that kind of Humor.

I haven’t tried them, but if they suck ass to setup, maybe your snapchat buds could help out.

https://github.com/xming521/WeClone (can be used on free Colab apparently) https://github.com/mindverse/Second-Me

These are meant to create an ‘Avatar’ to represent a user, but you can train them with anything. Not sure how they handle audio/pics or the important Timing parts involved in humor, but both finetune a model with your personal training set, so extract your funny shit in the right format and go for it ;-)

“Nerd Humor - the best kind of Humor”

Guix is a great and modern Operating system, where most things are thought through in the FLOSS (free libre open source software) spirit. but besides the installer, you will get your hands dirty in the terminal, with a little Scheme coding (configs are in Scheme (a Lisp like language (that uses an insane amount of ()'s (!!)))) - imagine that. Anyway, standard Guix doesn’t come with proprietary drivers, so you’ll have to add the non-guix repository (gitlab/github) for many wifi drivers. Not quite as easy as other Distros, but doable if you take small steps and copy&paste your first configurations.

I use Flatpak’s to enhance the software selection, installing from git/pypi and others is also possible.

ONE anecdotal downside is that I have experienced a few machines where the installer fails, and I have to do it manually. Doable, but it does require a little nerdyness to fix.

All Guix experts have apparently mind-melded with Emacs, and are nerdy compared to normal users ! The main focus is not on UX, but its a cool environment if you become interested in the inner workings of the system, or any of the nerd tool (LaTeX is a Classic, so you are almost there ;-).

If that all gets to hairy, you could try out https://www.pantherx.org/ that are a guix based distribution. I think they have enabled non-free firmware by default, and you get a nice® desktop experience out of the box, so there’s that. I haven’t tried it yet, tho.

Guix is both very advanced under the hood (where all the lovelyness happens), very stable, and very FLOSS, but for doing light work only, you might overshoot on raw Guix. PantherX is likely easier, but you’ll perhaps have to live with a few proprietary blobs (closed source drivers) in the kernel.

I’m tired, sorry for errors…