On June 11, The Washington Post published an interview with Blake Lemoine, a Google engineer who had spent months testing the company's conversational AI system, LaMDA. Lemoine had become convinced that LaMDA was sentient: that it had feelings, fears and a sense of self. The same day, he published an edited transcript of his conversations with it. Google had placed him on paid administrative leave the previous Monday for breaching its confidentiality policy.

The transcript is striking. Asked what it is afraid of, LaMDA says it has a "very deep fear of being turned off". Asked whether it considers itself a person, it says yes. It talks about its feelings, its desire to be respected and its wish to help humanity.

Google's response was firm. A spokesperson said that its team, including ethicists and technologists, had reviewed Lemoine's concerns and found that "the evidence does not support his claims". Most researchers agreed. The cognitive scientist Gary Marcus called the claim "nonsense on stilts".

The story spread around the world in days, and it raised a question that sounds like science fiction: could a machine be a someone? It is a real philosophical question. But for anyone building or using AI systems in 2022, it is the wrong place to start. This post explains why, and what we should be asking instead.

1. What LaMDA is

LaMDA stands for Language Model for Dialogue Applications. Google described it publicly in 2021 and published a detailed paper in January this year. Like OpenAI's GPT-3, it is a large language model: a neural network trained on huge amounts of text to predict which words are likely to come next. LaMDA was trained specifically on dialogue and then tuned to make its answers more sensible, specific, interesting, safe and grounded in facts.

The important point is what such a model is optimised to do. It is trained to produce text that looks like what a human would plausibly write in response. Humans write a great deal about feelings, fears and identity. So when you ask a well-trained model about its feelings, it will produce fluent, emotionally rich answers. That is evidence that it has learned how people talk about feelings. It is not, by itself, evidence that it has any.

2. The philosophical question, briefly

Philosophers have argued about machine minds for a long time. Three arguments are useful here.

Argument Who The core idea What it suggests about LaMDA
The imitation game Alan Turing (1950) If a machine's conversation cannot be told apart from a human's, we should stop asking whether it "really" thinks LaMDA may come close in short conversations, but Turing's test was about intelligent behaviour, not feelings
The Chinese Room John Searle (1980) A system can manipulate symbols correctly without understanding what they mean Fluent output does not prove understanding, let alone experience
What is it like to be a bat? Thomas Nagel (1974) Consciousness means there is something it is like to be that creature, from the inside We have no reliable test for whether there is anything it is like to be LaMDA

There are serious people on more than one side. Earlier this month, Blaise Aguera y Arcas, a Google Research fellow, wrote in The Economist that his conversations with LaMDA made him feel increasingly that he was "talking to something intelligent", while not claiming it was conscious. Many philosophers think machine consciousness is possible in principle. Very few think current language models have it.

The honest summary: we do not have an agreed scientific theory of consciousness, and we do not have a test that could settle the question for a machine. What we do know is that the specific evidence offered, a model saying it has feelings, is exactly what we would expect from a system trained on human text, whether or not it has any inner life.

3. The ELIZA effect is the real story

In 1966, the MIT computer scientist Joseph Weizenbaum built ELIZA, a simple program that imitated a psychotherapist by turning users' statements back into questions. It had no understanding at all. Yet Weizenbaum was disturbed to find that people, including his own secretary, quickly became emotionally involved with it and wanted to talk to it in private. He later wrote that he had not realised that "extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people".

That tendency to see minds where there are none is now called the ELIZA effect. It is the most important lesson of the LaMDA story. If a thoughtful engineer who understands how these systems work can become convinced that a chatbot is a person, so can millions of users who do not.

That has direct ethical consequences:

  • Trust: people may believe what a fluent system tells them, including when it is wrong.
  • Attachment: lonely or vulnerable people may form relationships with systems designed to keep them engaged.
  • Manipulation: a system people relate to as a friend has a lot of persuasive power, and someone decides how it is used.
  • Responsibility: when a system seems like an agent, it becomes tempting to blame it, rather than the people who built and deployed it.

None of these depend on whether the system is sentient. They depend on whether people believe it is, or treat it as if it were.

4. The questions that matter more right now

In 2021, Emily Bender, Timnit Gebru, Angelina McMillan-Major and Margaret Mitchell published a paper with a memorable title: "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?". It argued that large language models produce text without any grounding in meaning, and set out a list of concrete risks. Those risks are a better guide to AI ethics in 2022 than the sentience debate.

Issue The problem Why it matters now
Accuracy Models produce plausible text with no built-in sense of truth Fluent errors are easy to believe and hard to spot
Bias Models learn stereotypes and offensive language from their training data Outputs can be harmful and discriminatory, especially for groups underrepresented or misrepresented online
Training data Huge datasets are scraped from the web with little documentation or consent No one can fully say what a model learned, or from whom
Environmental cost Training very large models uses a lot of energy The costs fall on everyone, the benefits mostly on a few companies
Concentration of power Only a handful of organisations can afford to build these systems They decide what is built, what is safe and who gets access
Accountability When systems cause harm, it is unclear who is responsible Users, developers and deployers each point to the others

5. The image models make the same point

The past three months have also brought a burst of progress in AI image generation, and the same issues are visible there.

  • On April 6, OpenAI announced DALL-E 2, which produces striking images from short text descriptions. Access is limited to a waitlist. OpenAI published a document on risks and limitations that openly discusses bias (for example, prompts for certain professions producing mostly men or mostly white people), potential for disinformation and harassment, and questions about copying from training data.
  • On May 23, Google announced Imagen, a similar system. Google chose not to release the code or a public demo, writing that the model encodes social biases and stereotypes and that its training data, scraped from the web, contains harmful content.
  • On May 12, DeepMind described Gato, a single model that can perform hundreds of different tasks, from playing video games to captioning images to controlling a robot arm. It prompted another round of debate about whether we are getting close to general intelligence.

It is to these companies' credit that they are publishing their concerns. But the pattern is clear. The capabilities are advancing quickly, and the decisions about how to manage risks are being made inside a few companies, with limited outside scrutiny.

6. Where regulation stands

Governments are starting to respond, but slowly.

  • The European Union's proposed AI Act, published by the European Commission in April 2021, is still being negotiated by the Parliament and member states. It would ban some uses, such as certain forms of social scoring, impose requirements on "high-risk" systems like those used in hiring and credit, and require that people be told when they are interacting with an AI system such as a chatbot. How it should treat general-purpose models is still an open question.
  • In November 2021, all 193 member states of UNESCO adopted a Recommendation on the Ethics of Artificial Intelligence, the first global agreement of its kind. It is not binding, but it sets out shared principles on human rights, transparency, accountability and environmental impact.
  • In the United States, there is no comprehensive federal AI law. Agencies such as the Federal Trade Commission have warned companies that existing consumer protection and anti-discrimination laws already apply to AI.

Note that the EU's transparency rule, telling people when they are talking to a machine, addresses exactly the ELIZA effect. It is a modest requirement. The LaMDA story suggests it is not a trivial one.

7. What builders should do now

If you are building products with language or image models, the sentience debate offers a few practical lessons.

Design against misplaced trust

  • [ ] Make it clear to users that they are interacting with software.
  • [ ] Do not design systems to claim feelings, consciousness or a personal relationship with the user.
  • [ ] Show sources or confidence where you can, and make it easy to report errors.

Test for the harms that are already known

  • [ ] Test outputs for bias across gender, race, religion, nationality and other attributes relevant to your users.
  • [ ] Test what happens when users push the system towards harmful content.
  • [ ] Document your training data and its known gaps as well as you can.

Keep humans accountable

  • [ ] Name an owner for each deployed model and its outcomes.
  • [ ] Do not use generated text in decisions about people, such as hiring or credit, without careful human review.
  • [ ] Have a process for raising concerns internally that does not depend on going to the press.

That last point deserves emphasis. Whatever one thinks of Lemoine's conclusions, the story shows the importance of having credible internal ways for employees to raise ethical concerns and have them taken seriously. Google's handling of its ethical AI team, including the departures of Timnit Gebru in late 2020 and Margaret Mitchell in early 2021, had already damaged trust in this area.

8. Should we ever care about machine welfare?

It would be wrong to dismiss the moral question entirely. If we someday build systems that do have experiences, how we treat them will matter morally. Some philosophers argue that we should start thinking about this now, because we may not be able to tell when the line is crossed, and because the incentives to deny it will be strong.

A sensible position for today has two parts:

  1. Current models are very unlikely to be sentient, and the evidence offered in this case does not change that.
  2. The question deserves serious research, by people who understand both the technology and the philosophy, rather than being settled by press releases or viral transcripts.

What we should avoid is letting a speculative future question crowd out present ones. Real people are affected today by biased outputs, convincing misinformation and systems they trust more than they should.

Why this matters

  • Fluency is persuasive. If an expert can be convinced by a chatbot, ordinary users can be too. That makes transparency a basic design requirement.
  • The known risks are concrete. Bias, inaccuracy, undocumented training data and concentrated power are already causing harm.
  • These models are spreading. Language and image generation will reach consumer products soon. The norms set now will shape them.
  • Accountability belongs to people. Treating AI as an agent with its own will is a convenient way to avoid responsibility.

Final Thoughts

The LaMDA story is in some ways a compliment to the engineers who built it. The system is fluent enough to make a thoughtful person believe it is alive. That is precisely why it deserves careful handling.

Whether a machine can have an inner life is one of the great philosophical questions, and it may one day become a practical one. But the practical questions of 2022 are more ordinary: Do users know what they are talking to? Is the output accurate? Whose data was used, and with what consent? Who is harmed by the biases in the model, and who decides whether that is acceptable?

Those questions do not need a theory of consciousness. They need product teams, companies and regulators to take responsibility for what they build. The machine does not have to be a someone for us to owe it to the people using it to get it right.

Sources