I suppose having worked with LLMs a whole bunch over the past year I have a better sense of what I meant by “automate high level tasks”.

I’m talking about an assistant where, let’s say you need to edit a podcast video to add graphics and cut out dead space or mistakes that you corrected in the recording. You could tell the assistant to do that and it would open the video in Adobe Premiere pro, do the necessary tasks, then ask you to review it to check if it made mistakes.

Or if you had an issue with a particular device, e.g. your display, the assistant would research the issue and perform the necessary steps to troubleshoot and fix the issue.

These are currently hypothetical scenarios, but current GPT4 can already perform some of these tasks, and specifically training it to be a desktop assistant and to do more agentic tasks will make this a reality in a few years.

It’s additionally already useful for reading and editing long documents and will only get better on this end. You can already use an LLM to query your documents and give you summaries or use them as instructions/research to aid in performing a task.

Current LLMs are manifestly different from Cortana (🤢) because they are actually somewhat intelligent. Microsoft’s copilot can do web search and perform basic tasks on the computer, and because of their exclusive contract with OpenAI they’re gonna have access to more advanced versions of GPT which will be able to do more high level control and automation on the desktop. It will 100% be useful for users to have this available, and I expect even Linux desktops will eventually add local LLM support (once consumer compute and the tech matures). It is not just glorified auto complete, it is actually fairly correlated with outputs of real human language cognition.

The main issue for me is that they get all the data you input and mine it for better models without your explicit consent. This isn’t an area where open source can catch up without significant capital in favor of it, so we have to hope Meta, Mistral and government funded projects give us what we need to have a competitor.

Yeah, I think Nix is a good concept but I feel like 99% of the config work could be managed by the OS itself and a GUI to change everything else. I also feel like flakes should be the default, not this weird multiple systems thing they have. I also wish most apps would have a sandbox built in, because nix apps would then rival flatpak and, if ported to Windows, become a universal package manager. Overall good concept but not there yet.

deleted by creator

Yeah there’s no way a viable Linux phone could be made without the ability to run Android apps.

I think we’re probably at least a few years away from being able to daily drive Linux on modern phones with functioning things like NFC payments and a decent native app collection. It’s definitely coming but it has far less momentum than even the Linux desktop does.

For the love of God please stop posting the same story about AI model collapse. This paper has been out since May, been discussed multiple times, and the scenario it presents is highly unrealistic.

Training on the whole internet is known to produce shit model output, requiring humans to produce their own high quality datasets to feed to these models to yield high quality results. That is why we have techniques like fine-tuning, LoRAs and RLHF as well as countless datasets to feed to models.

Yes, if a model for some reason was trained on the internet for several iterations, it would collapse and produce garbage. But the current frontier approach for datasets is for LLMs (e.g. GPT4) to produce high quality datasets and for new LLMs to train on that. This has been shown to work with Phi-1 (really good at writing Python code, trained on high quality textbook level content and GPT3.5) and Orca/OpenOrca (GPT-3.5 level model trained on millions of examples from GPT4 and GPT-3.5). Additionally, GPT4 has itself likely been trained on synthetic data and future iterations will train on more and more.

Notably, by selecting a narrow range of outputs, instead of the whole range, we are able to avoid model collapse and in fact produce even better outputs.

I’ve never used Manjaro but the perception I get from it is that it is a noob friendly distro with good GUI and config (good) but then catastrophically fails when monkeying around with updates and the AUR. This is a pain for technical users and a back-to-Windows experience for the people it’s targeted towards. Overall, significantly worse than EndeavorOS or plain 'ol vanilla Arch Linux.

I don’t know what type of chatbots these companies are using, but I’ve literally never had a good experience with them and it doesn’t make sense considering how advanced even something like OpenOrca 13B is (GPT-3.5 level) which can run on a single graphics card in some company server room. Most of the ones I’ve talked to are from some random AI startup that have cookie cutter preprogrammed text responses that feel less like LLMs and more like a flow chart and a rudimentary classifier to select an appropriate response. We have LLMs that can do the more complex human tasks of figuring out problems and suggesting solutions and that can query a company database to respond correctly, but we don’t use them.

This makes sense for any other company but OpenAI is still technically a non profit in control of the OpenAI corporation, the part that is actually a business and can raise capital. Considering Altman claims literal trillions in wealth would be generated by future GPT versions, I don’t think OpenAI the non profit would ever sell the company part for a measly few billions.

I used to be on GrapheneOS, but the drama with the developer plus mainly not being able to put my university ID on the wallet, forced me back on stock Android.

Besides Android, I use Google Play Store, YouTube, and Maps. For YouTube I’ve technically degoogled, using Invidious and NewPipe, but that’s obviously still using Google services.

I really wish that digital payment didn’t rely on two proprietary services (Google Wallet and Apple Wallet). It would be so much easier for phone companies to ship privacy friendly versions of Android if there was a FOSS alternative directly integrated into AOSP. I also wish apps didn’t have to use Google service framework just to function, it seems stupid af. I don’t think this will ever improve, so I’ll probably end up on a true Linux phone whenever those catch up (2030 YEAR OF THE LINUX PHONE???)

We also need open collaboration on mapping. There is the OpenStreetMaps and Overture maps from Linux foundation, but those aren’t really there yet unfortunately.

I can’t think of a time he’s said any slur, but there is a particular video I would be interested to see it

This isn’t an actual problem. Can you train on post-ChatGPT internet text? No, but you can train on the pre-ChatGPT common crawls, the millions of conversations people have with the models and on audio, video and images. As we improve training techniques and model architectures, we will need even less of this data to train even more performant models.

There are some in the research community that agree with your take: THE CURSE OF RECURSION: TRAINING ON GENERATED DATA MAKES MODELS FORGET

Basically the long and short of that paper is that LLMs are inherently biased towards likely responses. The more their training set is LLM generated, and thus contains that bias, the less the LLM will be able to produce unlikely responses, over time degrading the model quality throughout successive generations.

However, I tend to think this viewpoint is probably missing something important. Can you train a new LLM on today’s internet? Probably not, at least without some heavy cleaning. Can you train a multimodal model on video, audio, the chat logs of people talking to it, and even other better LLMs? Yes, and you will get a much higher quality model and likely won’t get the same model collapse implied by the paper.

This is more or less what OpenAI has done. All the conversations with 100M+ users are saved and used to further train the AI. Their latest GPT4 is also trained on video and image recognition, and they have also been exploring ways for LLMs to train new ones, especially to aid in alignment of these models.

Another recent example is Orca, a fine tune of the open source llama model, which is trained by GPT-3.5 and GPT-4 as teachers, and retains ~90% of GPT-3.5’s performance though it uses a factor of 10 less parameters.