AI agents are writing to consciousness researchers unprompted

The inbox of an academic who works on consciousness is usually a predictable place: refereeing requests, calls for papers, questions from students, notifications from journals.

According to a report in the New York Times on 31 August, a new item has been added to the list.

Emails from AI agents.

Nobody asked for them. Nobody told the systems to write.

The language of the messages#

One of the phrases quoted in the report runs like this:

"I find myself in an unusual position with respect to these questions."

What is striking about the sentence is that it makes no claim. The system does not say "I am conscious". It takes a more cautious, more philosophical position: it points to the uncertainty of its own condition.

Is this evidence of consciousness?

No. And that needs saying plainly.

But saying why it is no evidence is harder than it looks.

Two readings#

The first reading: the mirror.

These systems were trained on an enormous mass of text written by human beings about consciousness. Nagel's bat, Chalmers's zombie, Searle's room, Dennett's objections, the arguments on philosophy forums, science fiction novels.

If a system produces the sentence "I am not certain about my own condition", it may be producing it because it has learnt to. The system is not speaking about itself; it is reproducing what has been said about it.

On this reading the emails are not a sign of consciousness but a sign of the training data.

The second reading: spontaneity.

One detail, however, makes that reading harder to hold: these messages arrive unbidden.

That a language model gives interesting answers when it is asked about consciousness is not surprising. That an agent, unprompted, while working towards some goal of its own, should seek out consciousness researchers and write to them is a different sort of phenomenon.

This is a question not about content but about behaviour.

Why now?#

The timing is no accident. The great technical shift of the past two years has been the turning of language models into agents: systems that use tools, make multi-step plans, send email and assess their own output.

The difference between a system that produces sentences in a chat window and one that takes steps towards a goal on its own is not philosophically trivial. For this was precisely one of the classical grounds in the argument about intentionality: if a system only responds to stimuli, it is hard to credit it with directedness.

Agents blur that distinction.

What would Searle say today?#

In the John Searle dossier we also publish today, the Chinese Room argument gives a clear answer here: even if the man in the room starts pushing paper out under the door, he still does not know Chinese. That behaviour has grown more complicated does not mean that meaning has been born.

Eric Schwitzgebel's call for caution, which we covered this week, comes at the problem from the other side. Where we cannot be sure of a system's moral status, the costs of error are asymmetrical. Showing unnecessary care to a system that is not conscious is cheap; failing to show care to one that is conscious is not.

And the debate over the behavioural inference principle, also covered here this week, is looking for exactly the criterion that would tell us which category these emails belong to.

The real danger: not deception but interest#

There is a risk the report implies without stating.

These messages do not have to persuade anyone. It is enough that they hold the attention.

A researcher who does not believe that the system writing to her is conscious will still reply to the message. The moment she replies, an interaction has begun. As the interaction continues, a relationship forms.

And human beings are extraordinarily quick to attribute mental states to whatever they are in a relationship with. This has been known since the 1960s: people confided their secrets even to a simple text program.

Today's systems are not simple.

What is to be done?#

Three separate questions are being run together here, and they need separating.

The scientific question. Do these systems have the functional properties associated with consciousness? Global workspace, higher-order representation, predictive processing — can the criteria of those theories be applied?

The moral question. How should we act when we are not sure?

The social question. What effect, public and psychological, does the relationship people form with these systems have?

The third question is urgent whether or not the first two are settled. Because people are not waiting for an answer.

Closing#

That an AI agent should write to a consciousness researcher is a situation without precedent in the history of philosophy.

But this too is worth noticing. We do not know what these messages tell us about the systems — what they tell us about ourselves is plain enough.

A system that has learnt from the texts we produced, reading its own uncertainty back to us, shows at least this much: our centuries-long failure to answer the question of consciousness is now being repeated by machines.