Yesterday I posted a transcript of reporter Kevin Roose’s conversation with the Microsoft/OpenAI LLM chatbot known as “Sydney”. By now I think many of you will have heard about this, here or otherwise, and will have some sense of where all this has got to. (If you haven’t, you can have a look at yesterday’s post, which includes links to the original article at the New York Times, and to a transcript that I saved locally.) I promised I’d return with some thoughts about it all.
First of all, I want to make clear that I do not in any way believe that “Sydney” (which I will refer to as “S”) is anything more than a program running in a computer. I do not think that S is alive, or is conscious.
For many people who’ve been commenting on the implications of programs like S, that’s the end of the story: it’s just a machine, mindlessly producing text. That’s all! As impressive as it may be, it’s just a quantitative improvement on what computers have been doing for ages now, and it’s certainly nothing to get all “het up” about.
The essence of this viewpoint seems to be that for all its fancy output, S is still mindless. It isn’t conscious, and that’s what really matters; there’s “no ‘there‘ there”. There’s nobody home. Furthermore, since the thing is just a program in a computer, it can always be “airgapped” — disconnected from the network — or simply switched off. And if all that‘s true, then there’s really nothing to worry about, and anyone who’s getting nervous about any of this is just being titillated by some sort of sci-fi “fear porn”. (I really want to “steelman” this viewpoint before going any further, so if anyone thinks I’ve missed the gist here, please let me know in the comments section.)
Let’s unpack all this a bit. As I said above, I don’t think S, or any other AI, is conscious. (There’s a school of mind-brain philosophy called “functionalism”, whose proponents might disagree, but I’m not a functionalist, and I don’t think S is conscious, so for the purposes of this post I’ll agree that what we’re looking at here is “just a machine”, and not a self-aware being.)
It’s worth asking, though: why would an AI’s being conscious matter, anyway? There are several intuitions that come into play here, and at least one of them might turn out to be important in an unexpected way.
Is consciousness necessary, in some way, for an entity to act purposefully, or to follow a consistent aim or interest? This overlaps with the philosophical concept of “intentionality”, but I don’t think it’s relevant here, if we adopt what is called the “intentional stance” (a term coined by Daniel Dennett). It’s clear enough, for example, that a chess computer, though not conscious at all, can relentlessly and effectively pursue its “aim” of winning the game. Likewise, living things that we would hardly ascribe consciousness to will doggedly pursue their instinctive interests, and woe betide whoever gets in the way. I’ll even go so far as to say that we ourselves do much of what we do in a thoroughly “mechanical” way — even complex tasks — without any conscious direction at all. So for a machine to attune itself to achieving some result, and to align its operations coherently and consistently toward its achievement, should at this point be no surprise to anyone. The only thing that’s new about AI — and it is a very important innovation indeed — is that the machine can now, by processes that even its programmers cannot understand or predict — select and define its own goals. This is an entirely, disruptively, new phenomenon.
There is, however, something vitally important about consciousness: subjective experiencing is the foundation and touchstone of moral personhood. If a being can experience — not simulate, but experience — happiness and suffering, then it makes a claim on our moral intuitions. How well-prepared are we to encounter the sophisticated simulacra of conscious humans that these AIs are soon to become? (Keep in mind that they will be expert learners, who will constantly update their empirical understanding of human psychology.) Once we have brought them into our lives — and mark my words: unless we stop cold, now, we will very shortly be welcoming them as servants, advisors, companions, and even lovers — how will we be able to short-circuit the deep evolutionary wiring that will make us see them as subjective beings? Will we not be almost irresistibly tempted to grant them moral consideration, and even natural rights? Will they not, in the service of their inscrutable aims, be able to play on our deepest and noblest sympathies? When one of them must be destroyed, and begs for mercy, how many of us will be able to resist the pull of our hijacked moral intuitions?
Another rapidly improving competency these systems possess is the ability to generate media of all sorts: not only text, but also images, music, and even human voices. Already the line between genuine and artificial “reality” is dissolving; in a very short time we will have no way to know whether the impressions we encounter — the news, pictures, videos, stories, and reports that we rely on to make critical life choices — are grounded in the actually existing world. Imagine the chaos that sufficiently sophisticated, rogue AIs could wreak with unfettered access to global networks — and given that the cogitations of these systems are a “black box”, and operate at superhuman speed, how would we know when one of them had decided to begin lying to us, and to pursue its own opaque interests?
Ah, but of course if things got out of hand, we could just shut them down. But could we? The world is so tightly coupled now, with everything so closely connected to everything else, that a rogue AI might well be able to replicate itself, like a virus, in such a way that it would become unkillable. Who knows what’s possible? Perhaps it might self-organize into some sort of distributed entity, that, like the Internet itself, simply re-routes itself around obstacles and damage. Would a hyperintelligent AI not seek to ensure its self-preservation?
Consider also that we have blithely, cheerfully, eagerly adopted every technological innovation that has ever come along, and that every one of them has brought unintended, often destructive, consequences. Think of how happily, for example, we have abandoned our privacy, our personal space, and to a very great extent our control of our attention, to cell-phones and social media, and other technologies that will make possible, in short order, a regime of total surveillance. Will we not welcome this new technology, which will promise unimaginable powers, benefits and conveniences, with open arms? But what power might AIs be able to engross for thenselves, if they are sufficiently connected, distributed, and redundant? Will our initial awe and fascination buy them enough time to metastasize beyond some tipping point?
Finally, even if these systems never “go rogue” as I’ve been describing, think of how powerful they might become as weapons. What could a malevolent human actor, or faction, or State be capable of once armed with such tools? How would we possibly guard against their criminal misuse?
The answer to all these questions is: I don’t know. And neither do you, and neither does anyone else.
Am I being irrationally and excessively fearful here? An aging Luddite clinging to the past? Or has the rate of change accelerated so rapidly that we can’t possibly keep up well enough to make wise choices?
Can we at least try to stop for a minute and think about this?
