8.3 billion synthetic personas: can MatrAIx simulate humanity?

For a long time the standing question in artificial intelligence was whether machines could think as human beings do.

A paper posted to arXiv on 4 August turns the question round: can machines simulate human beings well enough?

The technical report, MatrAIx: Simulating the World with 8.3 Billion Persona Agents, was written by ninety-three researchers under the lead of Harvard and MIT. The project describes itself not as a language model but as an evaluation infrastructure populated by simulated users.

The numbers#

At the centre of the system sits Persona 8B.

The claim is arresting: 8.3 billion profiles, one for every inhabitant of the planet. Each profile is specified by 1,290 attributes, running from age and occupation to income and tolerance of risk.

A distinction needs making at the outset. This is not a digital copy of 8.3 billion actual people. It is a persona space that can be generated from combinations of attributes; the researchers state that they sample from a dependency graph which preserves the correlations between them.

What has been released publicly is far smaller: a core of roughly a million profiles that have passed a quality filter. Of these, 599,847 rest on human-sourced data and 400,000 are synthetic.

The persona agents can be run in four environments — survey, AI chatbot, web and app (desktop and mobile alike). The system contains 1,010 reusable tasks spread across more than twenty-five domains.

The idea: simulate first, verify afterwards#

MatrAIx does not propose to do away with research on real users.

The researchers position simulation as a stress test that runs before any study of actual users. When a company develops a new application, testing it with thousands of users costs time and money; the same holds for a customer service bot, a financial application or a piece of health software.

From the standpoint of software engineering this has a consequence worth noting. Conventional testing asks: does the system work correctly?

MatrAIx adds a second question: how does the system work when different people are using it?

The researchers report having run 18,189 evaluation experiments, with the persona agents driven by a range of large language models.

The point matters. MatrAIx is not, on its own, a simulator of human beings. The persona information supplies a behavioural frame; the mechanism that produces the behaviour is the language model itself.

The equation runs: persona + environment + task + language model = simulated user behaviour.

91.5 per cent: does a persona become behaviour?#

One of the study's most striking measurements concerns persona adherence.

Across ten behavioural traits, 400 controlled trials were run; in 366 of them the behaviour written into the persona duly appeared, or was correctly suppressed where suppression was called for. That is an adherence rate of 91.5 per cent.

The figure looks high. But there is a methodological point to be careful about here:

Persona adherence is not the same thing as human reality.

That a model defined as a "sceptical user" behaves sceptically in an experiment does not establish that sceptical people behave that way in life. What is being measured is how faithfully the model follows the instruction it was given.

The gap between the two may turn out to be one of the more important research questions of the coming years.

Philosophy's old question: what is a human being?#

Behind the technical architecture of MatrAIx stands a very old problem.

We can describe a person in terms of age, income, education, occupation, personality traits, preferences and behavioural patterns. But once all of that has been assembled, is what we hold in our hands really a "human being" — or a data model of one?

The question drops us into a debate that runs from Aristotle to Kant, and from phenomenology to contemporary philosophy of mind.

For a human being is not merely the sum of the attributes he possesses. A human being is also conceived as something that has first-person experience.

Hunger is not a variable set to 0 or 1. Anxiety is not a behavioural parameter. Trust in a product is not a categorical property. What settles a person's decision is sometimes a singular experience that no database can represent.

Don Ihde's postphenomenology, which we covered this month, was built to analyse precisely this difference: experience cannot be reduced to a sum of attributes.

The problem of circularity#

There is a deeper difficulty with the system: artificial intelligence is evaluating artificial intelligence.

In MatrAIx the persona agents are driven by large language models, and the systems under evaluation are themselves AI products. Which raises the question:

When one artificial intelligence measures the effect of another on human beings, is it really measuring human behaviour — or the model's assumptions about human behaviour?

The objection has been raised in the literature under the name of the circularity problem, and it is not one to be waved away. If artificial intelligence serves as both user and assessor, the system may constitute a closed epistemic loop in which its own assumptions are confirmed from within.

This is a familiar trap in the philosophy of science: the instrument of measurement sharing the assumptions of the thing it measures.

The real contribution: the end of the "average user"#

MatrAIx deserves its due. The project shows plainly how inadequate the notion of the "average human being" is.

In the real world people are not alike. The same product is used differently by different people. The same fault may be trivial to one user and, to another, reason to abandon the system altogether. The same price rise may leave one consumer unmoved and change another's buying behaviour outright.

The researchers show that different persona attributes make a difference to behaviour: how users respond to a price rise, how readily they abandon a system when an AI assistant fails them, how much delay they will tolerate.

Benchmarks to date have largely assumed a single "average user". MatrAIx puts user diversity at the centre of evaluation.

Its contribution may come down to a single sentence: it turns the question "how good is the model?" into the question "how good is the model, and for whom?"

Where to stop?#

The longer-term consequences need not stay confined to product testing. Market research, policy trials, the design of education systems, the optimisation of advertising — all of it can be done on the same infrastructure.

And it is just here that the ethical limit comes into view.

What happens when a company begins to treat research on real users as superfluous because "we tested it on 8.3 billion virtual users"? When a government tries out its policies on artificial citizens rather than actual ones? When an advertising firm can calculate in advance, across billions of simulations, which message will move which persona?

At that point technology ceases to be an instrument that seeks to understand human beings and becomes one that predicts and shapes what they do.

The researchers' own caution is worth noting in this light. They state that simulated users do not stand in for real ones, that the system is a way of finding problems earlier, and that significant results should be checked against different models and verified with actual people.

Acknowledging the difference between simulation and reality is the precondition of the simulation remaining scientifically trustworthy.

The race is changing its subject#

MatrAIx marks a change of direction.

The question in the first period was: who will build the largest model? Then it became: who will develop the most capable agent? Now another question is steadily coming to the fore: who can model human behaviour best?

That suggests artificial intelligence is ceasing to be a technology for producing intelligence and becoming a technology for modelling human behaviour.

And the last question is perhaps not a technical one at all. Rather than asking how well artificial intelligence can imitate a human being, we might ask: at what point will we begin to trust an artificial intelligence we take to be imitating one?