On Sunday, August 16, a crowd of students gathered in Parliament Square in London. Their placards and chants were not about climate change or racial justice, the issues that have brought many young people onto the streets in the past two years. They were about a statistical model. "F*** the algorithm," they chanted.

Three days earlier, A-level results in England had been published. Exams were cancelled in March because of the pandemic. Instead, teachers estimated a grade for each student and ranked them within each subject. Then the exams regulator, Ofqual, used a model to adjust those grades so that the national results looked like previous years.

About 39 percent of teacher-assessed grades were lowered, around 3 percent by two grades. Thousands of students lost university places. On August 17, the government and Ofqual reversed course and gave students their teachers' grades. On August 25, Ofqual's chief regulator resigned. Last week the Prime Minister blamed a "mutant algorithm".

The algorithm did not mutate. It did what it was designed to do. That is what makes it worth studying.

1. What the model did

The approach, which Ofqual called the Direct Centre Performance model, worked roughly like this:

  1. Take each school's results in each subject over the previous three years.
  2. Use them to predict a grade distribution for this year's students at that school, adjusted for the prior attainment of this year's cohort.
  3. Use the teacher's ranking of students to place each student into that distribution.
  4. For small classes, where statistics are unreliable, rely more on the teacher's assessed grade. For the smallest, use it unchanged.

The goal was reasonable. Teachers' estimates are generous on average, and if every estimate had been accepted, grades would have jumped far above any previous year. Ofqual wanted to keep results consistent with the past, so that a 2020 A grade meant the same as a 2019 A grade.

The problem was what the model treated as the unit of fairness.

2. Grading the school, not the student

The model did not ask "how good is this student?" It asked "how well do students at this school usually do, and where does this student rank among them?"

For most students the answer was close enough. For some it was badly wrong:

  • The outstanding student at a struggling school. If nobody at your school had got an A* in a subject in three years, it was very hard for the model to give you one, however good you were.
  • The improving school. A school that had turned itself around recently was held to its older, weaker results.
  • The small class. Because small classes kept their teacher grades, subjects and schools with small groups, which are more common in independent schools, were largely protected from downgrading.

The result was that the share of A and A* grades at independent schools rose by 4.7 percentage points compared with last year, more than twice the rise at state comprehensives. Nobody designed the model to favour private schools. It followed from the design choices.

In Scotland, a similar approach had already produced the same kind of result a week earlier. When results came out on August 4, pupils from the most deprived areas had their teacher estimates lowered much more than pupils from the most affluent areas. After a week of protests, the Scottish Government restored the teacher estimates for all 124,564 downgraded results on August 11. England followed six days later.

3. The philosophical problem: individuals and statistics

There is an old idea in moral philosophy, often associated with Kant, that people should be treated as ends in themselves, not merely as means. A related idea in law is that a person should be judged on what they did, not on what people like them usually do.

Statistical prediction often breaks this rule quietly. Insurance prices, credit scores and risk assessments all judge individuals partly by the group they belong to. We tolerate that in some contexts, especially when the stakes are small and the individual can do something about it.

The A-level model made the problem visible because:

  • The stakes were high. A grade decides a university place, and a university place shapes a life.
  • The individual could do nothing. Students could not sit an exam to prove the model wrong, at least not in time.
  • The group was not chosen. Nobody chooses which school they attend at 16 on the basis of its past A-level results in further maths.
  • Accuracy on average hid unfairness at the edges. The model could be reasonably accurate across the country while being very wrong for particular students.

That last point is the most important lesson for anyone building predictive systems. A system can be accurate overall and still be unfair to the people it gets wrong, especially if the errors fall on the same kinds of people again and again.

4. A summer of algorithmic accountability

The A-level fiasco is the most visible example this year, but not the only one.

In June, after the killing of George Floyd and protests across the world, three of the largest technology companies stepped back from selling facial recognition to police. On June 8, IBM's chief executive Arvind Krishna wrote to the US Congress that IBM would no longer offer general purpose facial recognition and opposed its use for "mass surveillance, racial profiling, violations of basic human rights and freedoms". On June 10, Amazon announced a one-year moratorium on police use of its Rekognition system. Microsoft said it would not sell facial recognition to US police until there was a national law.

Later in June, the New York Times reported that Robert Williams, a Black man in Detroit, had been wrongfully arrested in January after a facial recognition system matched him to a blurry image of a shoplifting suspect. It is believed to be the first known case of its kind in the US. Earlier research, including the Gender Shades study by Joy Buolamwini and Timnit Gebru in 2018 and a US National Institute of Standards and Technology report in December 2019, had found that many face recognition systems make more errors for some demographic groups than for others.

Two very different technologies, one simple statistical model and one deep learning system, ran into the same set of questions:

Question A-level model Police facial recognition
Who bears the errors? High achievers at lower-performing schools Disproportionately people with darker skin
Could the individual contest the decision? Only through a narrow appeals process Often only after arrest
Was the system open to scrutiny? The full method was published only on results day Often not at all
Who decided it was acceptable? A regulator under ministerial direction Often individual police departments

5. Lessons for anyone building decision systems

You do not need to be a regulator or a technology giant for these lessons to apply. Any team that builds a system to score, rank or select people, for hiring, performance reviews, credit, fraud or content moderation, faces the same questions.

Decide what fairness means before you build

  • There is no single definition. Equal accuracy across groups, equal outcomes across groups and individual fairness can conflict. Researchers have shown that, outside special cases, several common definitions cannot all be satisfied at once.
  • Choose deliberately, write down why, and say so publicly.

Look at the errors, not just the average

  • Break performance down by the groups that matter: school type, region, gender, ethnicity, age.
  • Look for people the system cannot get right by design, such as the outlier student at the low-scoring school.

Let outsiders check your work

  • The Royal Statistical Society offered to help Ofqual this spring, but withdrew after being asked to sign a non-disclosure agreement it found too restrictive. Independent scrutiny is cheapest before launch.
  • Publish your method and your test results before the decisions are made, not on the day.

Make it easy to contest

  • Every person affected by an automated decision should be able to understand why it was made and challenge it in time to matter.
  • In Europe, the GDPR already gives people rights around significant decisions based solely on automated processing. Even where the law is unclear, contestability is good design.

Keep a human answerable

  • "The algorithm did it" is not an explanation. Somebody chose the inputs, the objective and the trade-offs. That person or body should be able to explain them.

6. A checklist before you deploy a system that decides about people

  • [ ] Have we written down what "fair" means for this system, and who agreed to it?
  • [ ] Have we tested error rates for each relevant group?
  • [ ] Have we identified people the system structurally cannot get right?
  • [ ] Has someone independent reviewed the method?
  • [ ] Can affected people see why a decision was made and challenge it quickly?
  • [ ] Is there a named person accountable for the outcomes?
  • [ ] Would we be comfortable if every affected person read the design document?

Why this matters

  • Simple models can do serious harm. The A-level model was not artificial intelligence in any fashionable sense. The ethical questions were the same.
  • Legitimacy matters as much as accuracy. People accepted exam results for decades even though exams are noisy, because the process felt fair. A model that judged them by their school did not.
  • Errors that fall on the same groups are not random. They are a pattern, and patterns are a design choice.
  • Public pressure works, but late. This summer's reversals came only after people were harmed. Good governance should make them unnecessary.

Final Thoughts

It is tempting to see the A-level story as a one-off failure in an extraordinary year. It is better seen as a preview. More decisions about people, from who gets a loan to who gets a job interview, are going to involve models. Most will be less visible than exam results, and the people affected will not be able to march on Parliament Square.

The students who chanted in August were not against mathematics. They were asking for something more basic: to be judged as themselves, by a process they could understand and challenge. That is a fair thing to ask of any system that decides about people, and a good test to apply before we build the next one.

Sources