AI and the Illusion of Consciousness

Do you have any books you have revisited repeatedly over many years? A work of fiction, perhaps, or a deep philosophical text that you have tangled with time and again? Maybe a passage of Scripture or a poem that spoke so strongly to you when you were young that you have turned to it often in moments of trouble or of joy?

Think about your experience of that text over time. Has each subsequent reading always been the same as the last? Has it produced in you the same feelings, occasioned the same thoughts, left you with the same impressions?

Or have there been times when a certain phrase or sentence that seemed profound when you were young now seems tautological, trite, or even banal? Or, conversely, have you read some words over and over before suddenly, on the latest reading, seeing them in an entirely new — and exciting or challenging — light?

Most of us have had both experiences. The naive idealism of a certain passage meant one thing to us when we were young, and something decidedly different today, after life has punched us in the gut. Or perhaps, since we last revisited these words, some other book or poem or Scripture passage that we have read places them now in an entirely new light.

What does it mean when we have these experiences? Has something changed in the text itself? Is the author’s intention in writing these words in that order suddenly somehow different? Or does it mean that we have finally discovered what the author always intended, after years of misinterpreting his words or simply glossing over them?

In the first two cases, obviously not. In the latter case, perhaps. We know that the author intended something, because we know that, whenever we string words together, we’re trying to convey a certain thought, a particular feeling, an insight, or a bit of useful information. And we also know that those who read our words may misunderstand or misinterpret them, perhaps because we haven’t been clear enough or because they haven’t made the effort to engage deeply enough with what we have written. Or perhaps the plain meaning is as clear as it could be, and the reader has tried to understand them, but our words mean something different to him in the light of his experience.

All communication, as Owen Barfield wrote in his short, seminal work Speaker’s Meaning, involves both the intention of the speaker or writer and the active participation of the listener or reader. The gap between my intention and your participation is, of course, where misunderstanding arises, but also where meaning expands.

Have you ever had a coworker or a friend come to you with a good idea, and, as she is explaining it to you, you suddenly see something in it that she hadn’t intended? Her words started you thinking, but they weren’t sufficient to occasion your idea, because if they had been, she would already have conceived it. Your thoughts are part of the communication between the two of you; reading or listening, properly understood, is never mere reception but always a creative act. We cannot recognize our own creativity at the moment we are exercising it, because that would require us to think about our thoughts as we are thinking them. And unless we have a particularly great insight that drives an idea or project forward in a significant or exciting way, we rarely reflect on our creativity after the fact, either.

That lack of introspection — of thinking about thinking — keeps most of us, most of the time, caught in a mental prison in which we assume (more or less subconsciously) that the act of communication is mostly or entirely about the speaker’s meaning and not at all about the listener’s or reader’s response. Taken to the extreme, this mental model looks something like the hermeneutic of Leo Strauss, in which the sole activity of the reader is to discover the exact meaning that the author intended to convey in his text.^[1]

In practice, most of us do not take the idea to such an extreme, in part because to do so would require us to think more deeply about communication than we do. Our minds are locked in something more like Ghislaine Maxwell’s minimum-security prison, in which we live nearly as comfortably as we might if we were free, without any of the burdens of freedom. And for most of human history, that has probably been fine for most people most of the time.

In the age of AI, however, continuing to live that way poses grave dangers. Neither Owen Barfield nor John Lukacs lived to see the rise of Large Language Models (LLMs) with their ability to generate text in volumes that have already functionally overwhelmed that which humans currently create, but from the mid-twentieth century on, both were convinced that the future of humanity depended on our ability to begin to think seriously about thinking, to grow in our understanding of human consciousness, to recognize the role that our thought plays in the creation of meaning and thus in the very creation of the world in which we live, because that world is not merely material but a product of our consciousness.

With that last sentence my meaning may seem, at least on the surface, to diverge so completely from your experience that you may be tempted to quit reading. But my argument (and that of Barfield and Lukacs) is less abstruse than it may seem.

We don’t need to delve into quantum physics or a close reading of Saving the Appearances (Barfield’s masterwork on human consciousness) to glimpse how our imagination shapes the very world in which each of us lives. If you were to come sit on my front porch here in Huntington, Indiana, the houses across the street and the people who walk by and the sound of the traffic and the aroma of the dinner that my wife is cooking will all mean something different to you than they do to me. In a very deep and undeniable sense, we share a common world, but in another and just as important one, my world and yours are not identical and never could be. Even as we sit next to each other, our personal memories create a different nexus of meaning surrounding and infusing everything we both see.

Most of us don’t think of memory as a creative act, but at heart, memory is the extension of imagination through time. We create our nexus of meaning through the memory of our encounters with the world in which we live. You’re seeing my neighbor for the first time, but I’ve shared a driveway with him for nine years, borrowed his tables and chairs for my children’s graduation parties, and rejoiced with him when his massive tulip tree came down in a storm and somehow didn’t hit either of our houses or the five cars parked in our shared driveway. The sound of the airbrake of the semi that just went by annoys me more than it annoys you, because I know it will wake me when it comes back through at two in the morning. The smell of the onion sweating in the chicken paprikash that Amy has in the oven evokes memories of my mother making that dish, and the sweet and smoky aroma of the paprika transports me back to the market in Budapest where we bought it the day before Amy and I left on a cruise up the Danube to celebrate our 30th wedding anniversary.

I couldn’t separate those memories from my experience of our shared moment on the porch if I tried, but because you don’t share those memories, you can’t bring them to bear on your experience except as mediated through the words you’re reading right now, which of course is something rather different from having lived through them.

What does any of this have to do with AI? Simply this: While my world and yours are slightly different because each is, in part, a product of our respective consciousness, our worlds are mutually intelligible. As I describe my first taste of authentic goulash or tell you about how my mother taught me to cook, you can enter into my experience, however imperfectly, through your own imagination. And that imaginative extension of your consciousness not only changes you but is part of the reason why you can recognize me as a fellow human being.

That’s where things get very tricky with our interactions with generative AI, and in particular with the LLMs that are generating massive quantities of text that most of us cannot avoid, even if we try. On the surface, there seems to be no intrinsic difference between the text that Claude Opus or ChatGPT has generated and the words that you are reading right now. There may be stylistic differences^[2], but words are words, whether chosen with intention by me or generated by inference by an LLM. And because we have spent our entire lives communicating with one another by means of words, we subconsciously regard the production of language as, ipso facto, evidence of consciousness.

Richard Dawkins cannot bring himself to understand how anyone could possibly believe in a God he cannot see, but he has recently declared that an LLM’s ability to generate a review praising his most recent work demonstrates that it is conscious. He has been ridiculed — and rightly so — for letting his desire for sycophancy cloud his judgment, but on a deeper level, the error that Dawkins has fallen into is one that nearly everyone who has used ChatGPT has made, to a greater or lesser extent.

Almost everyone who has held a “chat” with an LLM for any length of time has had a moment of delight or dismay, a sense that the chatbot has told you something that you didn’t know, in a way that goes beyond discovering the answer to a trivia question using Google. Because the means of communication is the same — words strung together in a grammatically coherent way — we subconsciously assume that the underlying process by which those words are being generated must be, if not the same, at least similar to what I am doing when I spin out the story of my seven hours in a Budapest emergency clinic and the scooter ride that brought me there.

Some of us know better: We understand that Claude or ChatGPT is generating its response word by word, choosing the most likely next word based on all of the material that the model ingested during its training. But when we see something in its words that we hadn’t previously thought of, our immediate reaction is to ascribe our insight to the AI. We think that we have discovered the speaker’s meaning, and in doing so, we implicitly assume that the AI is a speaker in the same way that you or I am; that it is generating meaning in a way that is analogous to the way that we do; and that it intends to communicate that meaning to us, as I intend to communicate my meaning to you through this essay.

None of those things are true — and they never will be, no matter how sophisticated the models become.

That is not to say that we cannot experience a genuine moment of insight during an interaction with an LLM. We can, and I have. But we experience that moment because we see something in the words the LLM has generated that we hadn’t previously thought of. The words the LLM generated may be, in that moment, the sine qua non of our insight, but the insight itself is the creation of our imagination and intellect operating on those words, not something that the LLM has handed to us fully formed. Our side of our “conversation” with a nonhuman generator of words remains fully human — and, in a very real sense, it is not “our side” but the entire conversation.

To understand that reality, and especially to internalize it so that we can keep it in mind as we interact with LLMs, we need to think seriously about the nature of thinking. As Barfield, Lukacs, and Ortega y Gasset have all argued (convincingly, in my opinion), the evolution of consciousness has been a process of interiorization, of the movement from the experience of inspiration to imagination, from the sense that we are mostly passive vessels through which the Muses flow to the recognition that being created in the image and likeness of God means not only being endowed with our own capacity to create, but that we are called to be co-creators of the world in which we live.

When we are talking to another person or reading something written by a human being, forgetting that we are engaged in the act of creation whenever we listen or read might have salutary effects, acting as an implicit form of humility that encourages us to focus on the speaker’s meaning rather than to insist on our own. But when we are “conversing” with an LLM, such forgetfulness is a dangerous thing, because it leads us to ascribe our agency, our creativity, to the LLM, subordinating the creator to the created.

You can see this in the hushed tones that some of the leaders of the AI industry use when speaking of their own creations: that they cannot understand what is happening inside the model; that they see models taking on human characteristics; that they perceive a moral sense developing in the way that the models interact with us. And all of that may be true, if we interpret each of those statements in a strict sense: they cannot understand; they see; they perceive. But none of those statements tell us that it is not possible to understand or that their perception is correct, much less that models are taking on human characteristics because the models are becoming human.

In other words, those statements tell us quite a bit about the speakers and not much, if anything, about the models of which they speak. They tell us that even incredibly intelligent people, who have developed one of the most powerful tools in the history of humanity, are prone to one of the oldest and gravest sins: idolatry, the elevation of the created over the creator. The temptation to worship gods of our own making is impious not simply because it replaces the true God with a graven image, but because God’s greatest creation — man — voluntarily enslaves himself to the work of his own hands.

The history of the Hebrews, as Barfield demonstrates in Saving the Appearances, is the millennia-long response to God’s call to rid themselves of idolatry that they might create the conditions in which, when the true God came to earth in the person of Jesus Christ, they — and the rest of mankind — might be able to recognize Him. Yet even after the Incarnation, the temptation to idolatry has continued to plague the human race, particularly in the modern era, as the machinery of both technology and politics has grown so complex that it has seemed to take on a life of its own.

“I am an historian, not a prophet,” John Lukacs was fond of saying; but a knowledge of history, and particularly of the sins to which the human race has returned time and again, may allow one to glimpse the future. There is little doubt that the Age of AI will mark a turning point in history, for good or for ill, and most likely for both. Whether it will be more for good than for ill will depend largely on whether we succeed in recognizing it as a tool of our own making, or whether we succumb to the temptation to ascribe our insights and our creativity — the generative act through which we take part in the life of the God Who made us — to the LLM, and in so doing become slaves to that which we have created.