An argument can be made that no single human being really understands how AI works. What they discovered when they added more processing power and more layers of pattern matching (what they call deep learning) for building large language models is that the chatbots became REMARKABLY humanlike in terms of their output. This was a downright shocking discovery, and this development alone suddenly diverted trillions of dollars of investment towards the development of AI. But according to the authors of the recent book, AI Snake Oil: What Artificial Intelligence Can Do, What It Can’t, and How to Tell the Difference, Arvind Narayanan and Sayash Kapoor of Princeton University, relatively little of that money has been spent on research that would attempt to understand WHY we are getting this result. It seems no one really knows, and worse, no one REALLY CARES. Instead, the agenda is to throw more and faster hardware at it, “FEED THE BEAST” to give it more power, more capacity, more memory, with no one truly understanding why it even works as it does. Is this more human folly unfolding before our very eyes? What is Creator’s perspective?

In a sense, this question is getting into the realm of philosophy—what constitutes a human being, human thinking, or a single human thought and its potential for being more or less complex, more or less a concrete description of demonstrable reality, or something simply conceptual and unprovable even, but nonetheless worthy of contemplation and discussion because it can relate in a more indirect way to human aspirations, goals, yearnings and potential future developments that are of value to at least consider and think about if not achievable at present in a concrete fashion. We would say what is taking place, and these authors are describing as summarized in your question, is that the AI capabilities becoming so sophisticated in their pattern matching ability, to extrude text that sounds human-made, has reached a level not readily matched by the average human being. In other words, that output sounds like a person and better than what a person could do spontaneously without considerable training, thought, serious research, and considerable time and effort creating a verbal description containing the same level of content and understandable descriptive wording to convey concepts and enable deriving useful conclusions about its meaning. We would say a simulation is still just a simulation. Just as the average person can in no way do the mathematical calculations of a high-powered super computer, the sophisticated chatbots and their output may well satisfy routine queries for information and even pass for high-level human produced information, but there is no way this can maintain a high level of sophistication and do more than regurgitate the training data set underlying the pattern matching allowing the current output to be generated. In other words, true originality will be a random occurrence, and while given the complexity of language, so that many ways exist to convey similar thoughts and an endless series of similar compositions might look like individual contributions by human originators, this will be only a simulation and not genuine human ingenuity and creativity on display. So again, the problem comes back to the fundamental reality here, the question needing to be asked is, what good is a simulation, really? If only for entertainment value, the downside is minimal other than perhaps using up precious life experience focused on a simulation of living and not living oneself in the real world, much akin to the many, many lost hours looking at screens at simulations of people living their lives or even true-life documentary recordings of real events, but still the events or ideas and thoughts of others, not the self. To some extent, this provides a training set of data to help people learn and grow, but at some point, people need to spread their wings and live their lives in a way to bring true rewards and meaning for them personally; that can never be supplied by watching others live their lives because it is looking at a separate reality from one's own.