That's a point that's often overlooked. I suspect many of the "amazing insights." LLM events only happen because training sets encompass an extensive range of knowledge and can arrive at conclusions previously unseen.
One could reasonably claim that the value of AI systems and very large training sets is not that it is an approach to AGI, but that it makes finding previously unseen connections possible.
Many moons ago, I attended a demo of a new software product, and without disclosing my disability to the vendor, I inquired about its accessibility features. They said that they don't do anything for the disabled because it's such a small market, and it wasn't profitable to accommodate the needs of disabled users.
I found myself irrationally enraged and had to walk away from the conversation. I thought about it when I calmed down and I realized I was feeling, "Who the fuck are you to tell me how I can live in the world?"
Dissecting these thoughts further led me to the understanding that, without accessibility, you are telling a class of people that they don't deserve access to education, government services, or commercial products.
Telling disabled people that they don't deserve access to any benefits of a civilized society is a long-standing and persistent attitude. It's roughly analogous to denying poor people health care, food, and basic shelter because all they deserve is what scraps we are willing to bestow on them.