The problem in this case is specifically that it is not trained on humans. The LLM hallucinates nonsense and then recursively reads its own nonsense from the internet in some kind of shit Ouroboro. This problem doesn’t seem solvable with LLM and contrasts it with AGI (artificial general intelligence - the thing LLMs claim to be.) Since the model needs more and more data to build more satisfying responses, it’s susceptible to injesting from other LLMs.

A LLM must ONLY be trained on humans because they don’t actually understand reasoning or linguistic structure. You end up with a “invisible green dragons sleep furiously” response very quickly. But the LLM also can’t tell if the text it’s injesting is from a human or LLM.