Two events this autumn, three weeks apart, made the same philosophical question front-page news.

On November 2, a New York jury found Sam Bankman-Fried, founder of the crypto exchange FTX, guilty on all seven counts of fraud and conspiracy. Before FTX collapsed in November 2022, Bankman-Fried had been the most famous supporter of effective altruism, a movement that tries to use evidence and reason to do the most good. He had said publicly that he was making money in order to give it away.

On November 17, the board of OpenAI, a company governed by a non-profit whose mission is to ensure that artificial general intelligence benefits all of humanity, removed its chief executive, Sam Altman, saying he had not been "consistently candid in his communications with the board". Within days, almost all of the company's employees signed a letter threatening to leave, and on November 21, OpenAI announced that Altman would return under a new initial board. The board's reasons have still not been fully explained.

The two stories are very different, and it would be unfair to equate them. But both raised the same question, one of the oldest in moral philosophy: when you believe the stakes are enormous, how far can good ends justify questionable means, and who gets to decide?

This post looks at that question through the year's biggest debates, mostly about artificial intelligence, and at what philosophy can offer people making decisions under uncertainty.

1. A year of large claims about the future

2023 was unusual for how often serious people made claims about the long-term fate of humanity.

Date Event
March 14 OpenAI releases GPT-4
March 22 The Future of Life Institute publishes an open letter calling for a pause of at least six months on training systems more powerful than GPT-4
May 1 Geoffrey Hinton, a pioneer of deep learning, says he has left Google so he can speak freely about AI risks
May 30 The Center for AI Safety publishes a one-sentence statement: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war." Signatories include the chief executives of OpenAI, Google DeepMind and Anthropic
October 30 President Biden signs Executive Order 14110 on safe, secure and trustworthy AI
November 1 28 countries and the EU sign the Bletchley Declaration at the UK's AI Safety Summit, including the US and China
November 2 Bankman-Fried convicted
November 17 to 21 OpenAI board crisis
December 8 EU institutions reach a provisional political agreement on the AI Act

Not everyone accepted the framing. Researchers including Timnit Gebru, Emily Bender, Angelina McMillan-Major and Margaret Mitchell, authors of the 2021 "Stochastic Parrots" paper, responded to the pause letter by arguing that its focus on hypothetical future "digital minds" distracted from real harms happening now: exploitation of workers, concentration of power, bias and misinformation. In October, investor Marc Andreessen published a "Techno-Optimist Manifesto" arguing the opposite case: that slowing AI down was itself the danger.

Underneath these positions are different answers to philosophical questions that are much older than AI.

2. Consequentialism: the power and the trap

Consequentialism holds that the right action is the one that produces the best outcomes. Its most famous form, utilitarianism, developed by Jeremy Bentham and John Stuart Mill, says we should maximise overall wellbeing, counting everyone's interests equally.

The idea has real power. It explains why we should care about strangers, why we should look at evidence of what actually helps, and why a life saved far away matters as much as one saved nearby. Peter Singer's 1972 essay "Famine, Affluence, and Morality", with its image of a drowning child in a shallow pond, is one of the most influential pieces of applied ethics ever written. Effective altruism grew directly out of this tradition, and has moved large sums of money to well-evidenced charities, particularly in global health.

The trap is in the "maximise". If only outcomes count, then lying, breaking promises or taking huge risks with other people's money can, in principle, be justified when the expected payoff is large enough. Philosophers have known this for a long time, which is why most sophisticated consequentialists argue for strong rules in practice: honesty, promise-keeping and respect for rights tend to produce better outcomes than case-by-case calculation, especially because people are bad at predicting consequences and very good at rationalising.

After FTX's collapse, William MacAskill, one of effective altruism's founders, wrote that a clear-thinking effective altruist should strongly oppose "ends justify the means" reasoning, and said he would reflect on what had happened. Months earlier, in a 2022 interview, the economist Tyler Cowen had asked Bankman-Fried whether he would keep taking a bet with a 51% chance of doubling the world's value and a 49% chance of destroying it. His answer was, broadly and with caveats, yes. Taken literally, a strategy of always maximising expected value with no regard for risk leads almost certainly to ruin. That is a mathematical point, not only a moral one.

The lesson: outcome-focused thinking is a valuable tool, but it needs guard rails. Integrity is not a constraint on doing good. It is one of the conditions for it.

3. Longtermism: taking the future seriously

Longtermism is the view that positively influencing the long-term future is a key moral priority of our time. MacAskill's What We Owe the Future (2022) summarises the argument in three steps: future people count morally; there could be an enormous number of them; and we can make their lives go better, for example by reducing the risk of catastrophes that could end or permanently cripple civilisation.

Each step can be challenged:

Claim Strongest objection
Future people count Most people accept this in some form. We worry about climate change and nuclear waste for this reason
There could be very many of them Numbers far in the future are deeply uncertain, and multiplying them can make any small effect look overwhelming
We can influence their lives Our ability to predict the effects of actions decades or centuries ahead is weak, and confident long-range forecasts have a poor track record

Toby Ord's The Precipice (2020) estimated a one in six chance of an existential catastrophe this century, with unaligned AI as the largest single source at about one in ten. Those estimates are subjective judgements by a careful thinker, not measurements. They influenced how many people in AI labs and policy think about risk.

The philosopher Hans Jonas, writing in The Imperative of Responsibility (1979), offered a version of this concern decades earlier, about nuclear and environmental technology. He argued that when an action could have irreversible effects on the conditions for human life, we should give the prophecy of doom priority over the prophecy of bliss. Jonas's principle helps explain the appeal of caution, but it also has a weakness: almost any new technology can be described in ways that make the worst case sound possible.

4. Reasoning about small chances of catastrophe

The core difficulty in debates about AI risk is how to reason about outcomes that are uncertain, potentially extreme, and have no track record to learn from. Several philosophical tools help.

Expected value and its limits. Multiplying probability by size of outcome is the standard approach. It breaks down when probabilities are tiny and stakes are vast. Nick Bostrom's thought experiment "Pascal's Mugging" (2009) shows why: a stranger promises astronomical rewards in exchange for your wallet; however unlikely the promise, the size of the reward can make handing over the wallet the "rational" choice. Most people rightly resist this. A practical reading: be suspicious of any argument whose conclusion depends entirely on a tiny probability multiplied by a huge number.

Moral uncertainty. In Moral Uncertainty (2020), MacAskill, Krister Bykvist and Ord argue that since we are unsure which ethical theory is correct, we should act in ways that do reasonably well across several plausible theories. Applied to 2023's debates, that means we should not choose between current harms and long-term risks. A policy that ignores bias and labour exploitation fails on most views of justice. A policy that ignores catastrophic risk fails on most views of prudence.

The precautionary principle and its critics. Precaution says we should act to prevent serious harm even before the science is settled. Its critics, from economists to technologists, point out that inaction also has costs: medicines not developed, problems not solved. The useful version is proportionate precaution: the more severe and irreversible a possible harm, the more evidence of safety we should want before proceeding.

Who bears the risk? Kantian ethics adds something consequentialism can miss: people should not be treated merely as means. Imposing risks on others without their consent is wrong even when the expected payoff is positive. When a small number of companies or individuals make decisions that affect everyone, consent and legitimacy matter, not only calculations.

5. "AI safety" versus "AI ethics": a false choice

Through 2023, the discussion of AI risk often split into two camps that talked past each other.

"AI safety" focus "AI ethics" focus
Main worry Loss of control over highly capable future systems; catastrophic misuse Harms happening now: bias, surveillance, labour exploitation, misinformation, concentration of power
Typical proponents Some AI lab leaders, researchers in alignment, longtermist funders Researchers in fairness and accountability, civil society, affected communities
Typical criticism of the other Present harms are real but are not existential Talk of extinction serves industry by distracting from accountability and by making AI seem more powerful than it is

Both sides make fair points. Present harms are real and documented, and some loud warnings about the future have come from companies racing to build the very systems they warn about. At the same time, many of the people warning about catastrophic risk are serious researchers with no commercial stake, and some of the capabilities they worry about are improving quickly.

The 2023 policy responses suggest that the two concerns can be addressed together. The US executive order covers both safety testing of the most powerful models and civil rights, privacy and workers. The EU AI Act deal combines a risk-based approach to applications with obligations for general-purpose models. The Bletchley Declaration mentions both frontier risks and the protection of human rights. Whether these measures work will depend on how they are implemented.

6. Governance: who gets to decide?

The OpenAI episode was, among other things, a test of an unusual governance structure: a non-profit board, with a mission to benefit humanity and no duty to maximise profit, controlling a company worth tens of billions of dollars and backed by Microsoft. The board used its power. Within five days, pressure from employees and investors had reversed the decision.

We do not yet know the full story, and it would be wrong to guess at the board's reasons. But the philosophical point is clear enough. Good intentions written into a charter are not the same as accountable governance. Political philosophers from Montesquieu onwards have argued that institutions should not depend on the virtue of the people who run them: power should be checked, decisions should be explained, and those affected should have a voice. Those principles apply to organisations developing powerful technology, whoever runs them and however good their intentions.

The same applies to movements. Effective altruism's own response to the FTX collapse has included discussion about governance, conflicts of interest and the risks of relying on a few very large donors. These are institutional lessons, not just lessons about one person.

7. Principles for acting well under deep uncertainty

What can an individual, a team or an organisation take from all this? A short list of principles, drawn from several ethical traditions:

  • [ ] Keep integrity non-negotiable. Do not lie, deceive or break serious commitments for the sake of a greater good. The track record of people who did is bad.
  • [ ] Be humble about forecasts. Treat your predictions about the long-term effects of your actions as weak evidence, and be suspicious of reasoning that depends on them alone.
  • [ ] Avoid ruin. Do not take risks that could be catastrophic or irreversible, however attractive the expected value.
  • [ ] Do well across views. Prefer actions that look good under several ethical perspectives, not only the one you favour.
  • [ ] Do not ignore present harms for future ones, or the reverse. Both can be addressed, and usually through the same institutions.
  • [ ] Build accountable institutions. Check power, explain decisions and include the people affected. Do not rely on anyone's good character, including your own.
  • [ ] Listen to critics. People who disagree with your framing are more likely to see what you are missing.

Why this matters

  • High stakes tempt shortcuts. The more important people believe their mission is, the more tempting it becomes to justify questionable means. 2023 showed what that costs.
  • AI has made long-term ethics practical. Questions that used to be academic are now in executive orders, international declarations and company boardrooms.
  • Frameworks shape decisions. Whether you think in expected values, rights, precaution or justice changes what you would do about AI, and these frameworks are rarely made explicit.
  • Governance is ethics in practice. Good intentions need institutions that are accountable, transparent and resilient to the failures of individuals.

Final Thoughts

It is easy to read this year as a story about a few individuals. It is more useful to read it as a reminder of something moral philosophers have argued for centuries: good intentions are not enough, and the belief that you are working for something enormously important is precisely when you should be most careful about how you behave.

The questions raised this year will not go away. AI systems will become more capable, the arguments about how to govern them will get harder, and the temptation to justify risky behaviour by pointing to large stakes will grow. Philosophy cannot tell us exactly what to do. But it can help us notice when we are reasoning badly: when a calculation has replaced judgement, when a mission has excused deception, or when a small group has decided on behalf of everyone else.

The old answer still holds. Care about consequences, including those far in the future. But keep your promises, tell the truth, respect the people your decisions affect, and build institutions that do not depend on anyone being right all the time.

Sources