Better Decisions Cheatsheet: Pre-mortems, Decision Journals and Calibration

Most of us judge our decisions by how they turned out. The launch went well, so the plan was good. The hire did not work out, so the interview process is broken. It feels like learning, but it is mostly noise: outcomes mix decision quality with luck, and a single outcome cannot tell you which was which.

The good news is that you do not need more willpower or more intelligence to decide better. A handful of simple, well-studied tools make a measurable difference: imagining failure before it happens, checking how similar cases usually go, writing down what you expected before you know the answer, and scoring your probability estimates over time.

This cheatsheet collects those tools, where they come from, and how to use them in a team or on your own. Each is cheap: minutes per decision, not days.

1. Decision quality is not outcome quality

Poker players call it resulting: judging a decision by its result. Annie Duke, a former professional poker player, built her book Thinking in Bets (2018) around the idea that every decision is a bet under uncertainty, and that life outcomes are driven by two things: the quality of your decisions and luck.

Her opening example is Pete Carroll's call in Super Bowl XLIX: a pass from the one-yard line that was intercepted, and was heavily criticized as a blunder. Duke's argument is that, given the situation and the low base rate of interceptions on such plays, it was a defensible decision with a bad outcome. Critics judged it by the result.

Practical rules:

• Review the decision, not only the outcome. Ask: what did we know, what could we have known at reasonable cost, what did we assume, and was the reasoning sound?
• Watch the dumb luck cell. A risky shortcut that worked is not evidence that it is safe.
• Hindsight bias makes this hard. Baruch Fischhoff (1975) showed that once people know an outcome, they rate it as having been more predictable than people who judged it in foresight. After the fact, "we should have known" is often false. This is the main reason to write things down before you know the result (section 4).

2. The pre-mortem

Gary Klein described the pre-mortem in Harvard Business Review in September 2007 ("Performing a Project Premortem"). A post-mortem explains why a patient died; a pre-mortem happens before the project starts. The team assumes the project has already failed and works backwards to explain why.

Why it works:

• Prospective hindsight. Klein cites Mitchell, Russo and Pennington (1989, "Back to the Future: Temporal Perspective in the Explanation of Events"), who found that imagining an event has already happened increased the number of reasons people generated for it by about 30%. (Klein's article phrases it as the ability to correctly identify reasons; the study measured how many reasons people produced, not whether they were right.)
• Permission to dissent. Klein's diagnosis is that people are reluctant to voice reservations during planning. Framing failure as a given turns criticism into a contribution instead of disloyalty.

The procedure (about 30-45 minutes):

A template you can paste into a doc:

PRE-MORTEM: <project name>            Date: <yyyy-mm-dd>
Assume: it is <date + horizon>. The project has failed.

Reasons it failed (one per line, write alone first):
-
-

Top risks (after discussion):
| Risk | Likelihood (1-5) | Impact (1-5) | Early warning signal | Mitigation | Owner |
|      |                  |              |                      |            |       |

Changes to the plan we are making now:
-

Tips:

• Writing alone first is the point. If people speak before they write, the first loud opinion anchors the room.
• Include "boring" failures: the key person leaves, the dependency slips, nobody uses it, the budget gets cut.

3. Base rates and the outside view

When we forecast our own project, we take the inside view: we think about our plan, our team, our steps, and imagine how it unfolds. The outside view ignores the details at first and asks: how did similar projects go?

Daniel Kahneman and Dan Lovallo formalized this in "Timid Choices and Bold Forecasts" (Management Science, 1993), arguing that overly optimistic forecasts come from adopting an inside view anchored on plans and scenarios, while decision makers neglect the statistics of the past.

Kahneman's own story is the classic example. In the 1970s he led a team writing a curriculum and textbook on judgment and decision making for Israeli high schools. Asked how long it would take, the team's estimates, including his own, ranged from 18 to 30 months. Then he asked the curriculum expert on the team how long similar teams had taken. The answer: about 40% of them eventually gave up, and of those that finished, he could not think of any that took less than seven years, nor more than ten. The project was finished about eight years later.

The outside view in four steps:

Rules of thumb:

• Start from the base rate, then adjust. Not the other way around. Most adjustments are smaller than they feel.
• Use your own history. Your past estimates versus actuals are often the best reference class you have. A decision journal (next section) builds it for you.
• Pick the reference class honestly. "Projects like ours" tends to shrink to "projects that went well". If unsure, look at two or three classes and see how far apart they are.

4. The decision journal

A decision journal is a record of what you decided and why, written at the time of the decision. Its job is to beat hindsight bias: later, you compare what actually happened with what you thought would happen, not with what you now remember thinking.

Michael Mauboussin has told the story of asking Daniel Kahneman for one thing an investor could do to improve; Kahneman's advice was to "go down to a local drugstore and buy a very cheap notebook and start keeping track of your decisions." That is the whole method. The rest is a format.

A template (one entry per consequential decision):

DECISION JOURNAL                       Date: <yyyy-mm-dd>
Decision:            What I decided, in one sentence
Context:             The situation, constraints, time pressure
Options considered:  Including "do nothing"
Information:         What I knew; what I did not know; what I chose not to find out
Expected outcome:    What I think will happen, with probabilities
                     e.g. "70%: the feature reaches 20% weekly adoption by Q4"
Key assumptions:     What has to be true for this to work
What would change my mind:
State:               Energy, mood, stress (1-5). Tired and rushed decisions are worth flagging
Review date:         When I will check the outcome
---
Review (on the review date):
Outcome:             What happened
Decision quality:    Good / bad process, regardless of outcome (section 1)
Luck vs skill:       What was in and out of my control
Lesson:

How to make it stick:

• Only consequential decisions. Hiring, pricing, architecture, investments, big commitments. A few per month is plenty.
• Always include a probability. "Likely" can mean anything from 55% to 90% to different people. Numbers can be scored (section 5).
• Schedule the review. An unreviewed journal is a diary. Put the review date in your calendar.
• Review in batches. Every quarter, read the entries due and look for patterns: always too optimistic on timelines? Always underestimating adoption?

5. Calibration and the Brier score

You are calibrated if the things you say are 70% likely happen about 70% of the time. Calibration is learnable, but only with feedback: you need to record probabilities and score them.

The standard score is the Brier score, introduced by Glenn Brier in 1950 ("Verification of Forecasts Expressed in Terms of Probability", Monthly Weather Review) to grade weather forecasts. For yes/no events:

Brier = mean of (forecast probability - outcome)^2, with outcome = 1 if it happened, 0 if not.

Reading it:

• 0 is perfect, lower is better. Always saying 50% scores 0.25, so you need to beat 0.25 to show any skill on yes/no questions.
• Confident and wrong is punished hard. A 95% call that misses costs about 0.90. This is what makes the score honest: you cannot game it by being bold or by being vague.
• Two scales exist. Brier's original formula sums the error over both outcomes (event and non-event), which doubles the number: a range of 0 to 2, with 0.5 for always saying 50%. Superforecasting uses this 0-2 scale: 0 is perfect, 0.5 is hedged 50-50 guessing, 2 is always certain and always wrong. Check which one a number uses before comparing.
• Compare to a baseline. The Brier skill score = 1 - (your score / baseline score), where the baseline is always forecasting the base rate. Above 0 means you add information.

Tetlock and the Good Judgment Project

Philip Tetlock and Barbara Mellers led the Good Judgment Project in the forecasting tournament that IARPA (the US intelligence community's research agency) launched in 2011: thousands of volunteers forecasting hundreds of geopolitical questions, scored with Brier scores. Their team performed so well that after the first two years it was the only one remaining. Tetlock and Dan Gardner described the findings in Superforecasting (2015).

Mellers et al. (Psychological Science, 2014) tested what made forecasters better and found support for three drivers:

All three improved both calibration (probabilities match frequencies) and resolution (confidently distinguishing what will happen from what will not).

A small Brier score calculator

Score your journal entries, compare against simple baselines, and see where you are over- or underconfident:

from collections import defaultdict

# (forecast probability that the event happens, outcome: 1 = happened, 0 = did not)
journal = [
    (0.90, 1), (0.80, 1), (0.70, 0), (0.60, 1), (0.90, 1),
    (0.30, 0), (0.20, 0), (0.70, 1), (0.80, 0), (0.95, 1),
    (0.40, 1), (0.10, 0), (0.60, 0), (0.85, 1), (0.75, 1),
]


def brier(pairs):
    """Mean squared error of probability forecasts. 0 is perfect, lower is better."""
    return sum((p - o) ** 2 for p, o in pairs) / len(pairs)


def calibration_table(pairs, width=0.2):
    """Group forecasts into probability buckets and compare to the observed hit rate."""
    buckets = defaultdict(list)
    for p, o in pairs:
        buckets[min(int(p / width), int(1 / width) - 1)].append((p, o))
    for b in sorted(buckets):
        items = buckets[b]
        avg_p = sum(p for p, _ in items) / len(items)
        hit = sum(o for _, o in items) / len(items)
        print(f"{b * width:.1f}-{(b + 1) * width:.1f}: n={len(items):2d}  "
              f"said {avg_p:.0%}, happened {hit:.0%}")


score = brier(journal)
base_rate = sum(o for _, o in journal) / len(journal)
reference = brier([(base_rate, o) for _, o in journal])  # always forecasting the base rate
skill = 1 - score / reference

print(f"Brier score: {score:.3f}")
print(f"Always 50%:  {brier([(0.5, o) for _, o in journal]):.3f}")
print(f"Base rate ({base_rate:.0%}) every time: {reference:.3f}")
print(f"Brier skill score vs base rate: {skill:+.2f}")
calibration_table(journal)

# Brier score: 0.159
# Always 50%:  0.250
# Base rate (60%) every time: 0.240
# Brier skill score vs base rate: +0.34
# 0.0-0.2: n= 1  said 10%, happened 0%
# 0.2-0.4: n= 2  said 25%, happened 0%
# 0.4-0.6: n= 3  said 53%, happened 67%
# 0.6-0.8: n= 3  said 72%, happened 67%
# 0.8-1.0: n= 6  said 87%, happened 83%

With 15 forecasts the buckets are tiny, so treat the table as a hint, not a verdict. After 50-100 scored forecasts, patterns become real: if your 80-100% bucket keeps landing at 65%, you are overconfident at the top and should pull those numbers down.

Calibration practice

• Forecast things that resolve. Clear question, clear date, clear criteria: "Will the release ship by September 30?", not "Will the release go well?"
• Use the full range. If everything you write is 60-80%, you are hedging. Use 5% and 95% when you mean it.
• Update in small steps. Tetlock describes the best forecasters as updating often, in small increments, as news arrives, rather than in big swings.
• Keep a team leaderboard for recurring questions (delivery dates, metrics, deal closes). Tracking was one of the three drivers above.

6. Putting it together

A lightweight workflow for any decision that is expensive to reverse:

Reversible, low-stakes decisions do not need any of this. Make them quickly and spend the time saved on the few that matter.

Why this matters

• Outcomes are noisy feedback. Separating decision quality from luck is the only way to learn the right lessons from both wins and losses.
• Pre-mortems surface risks people already know but will not say in a normal planning meeting.
• Base rates beat optimism. Starting from how similar cases went corrects the planning fallacy more reliably than trying harder.
• Writing it down defeats hindsight. A decision journal is the cheapest debiasing tool available.
• Calibration is a measurable skill. With probabilities and Brier scores, "good judgment" becomes something you can track and improve.

Final Thoughts

None of these tools requires software, a budget or a new process framework. A shared doc for pre-mortems, a notebook for decisions, and a spreadsheet (or twenty lines of Python) for scoring are enough.

What they require is the habit of being explicit before the outcome is known: what could go wrong, how cases like this usually go, what you expect, and how sure you are. Do that for the few decisions that matter, check back honestly, and your judgment will improve in a way you can actually measure.

Sources

• Gary Klein, Performing a Project Premortem, Harvard Business Review (September 2007): the pre-mortem method; reluctance to voice reservations during planning; cites the prospective hindsight research.
• Deborah J. Mitchell, J. Edward Russo and Nancy Pennington, Back to the future: Temporal perspective in the explanation of events, Journal of Behavioral Decision Making 2(1), 25-38 (1989): prospective hindsight increases the number of reasons generated.
• Daniel Kahneman and Dan Lovallo, Timid Choices and Bold Forecasts: A Cognitive Perspective on Risk Taking, Management Science 39(1), 17-31 (1993): inside view vs. outside view; the curriculum project anecdote (18-30 month estimates, 40% of similar teams gave up, 7-10 years, finished about eight years later).
• Daniel Kahneman, Thinking, Fast and Slow (2011): retells the curriculum story as an example of the planning fallacy and the outside view.
• Baruch Fischhoff, Hindsight is not equal to foresight: The effect of outcome knowledge on judgment under uncertainty, Journal of Experimental Psychology: Human Perception and Performance 1(3), 288-299 (1975): hindsight bias.
• Farnam Street, How to Improve the Quality of Our Decision Making (2012): Michael Mauboussin's account, from a Motley Fool interview, of Kahneman's "cheap notebook" advice and the decision journal.
• Annie Duke, Thinking in Bets: Making Smarter Decisions When You Don't Have All the Facts (Portfolio, February 2018): resulting, decisions as bets, the Pete Carroll Super Bowl XLIX example.
• Glenn W. Brier, Verification of Forecasts Expressed in Terms of Probability, Monthly Weather Review 78(1), 1-3 (1950): the Brier score.
• Barbara Mellers et al., Psychological Strategies for Winning a Geopolitical Forecasting Tournament, Psychological Science 25(5), 1106-1115 (2014): training, teaming and tracking improved calibration and resolution.
• IARPA, Meet the Elite Team of Superforecasters Who Have Turned Future-Gazing Into a Science (reprinted from Entrepreneur): the 2011 tournament, about 500 questions, Good Judgment the only team remaining after two years.
• Philip E. Tetlock and Dan Gardner, Superforecasting: The Art and Science of Prediction (Crown, 2015): the Good Judgment Project, superforecasters, Brier scoring.