Probability and statistics for decisions

You almost never see everyone. You see a sample: the customers who answered a survey, the visitors during one week. Any number from a sample would come out a little different with another sample. Statistics is mostly the craft of saying how sure you can be, given that wobble.

Bigger samples wobble less, but a big sample collected in a biased way is still biased. Ten thousand survey answers from only your happiest users tell you about your happiest users.

A confidence interval gives a range instead of a single number. “Average order value is 42, with a 95% interval from 39 to 45” means the method used tends to produce ranges that contain the true value about 95% of the time. A wide interval is an honest way of saying “we do not know very precisely yet”.

When you report a result, give the range. Decision-makers handle uncertainty better than you might expect, and much better than a false sense of precision.

A p-value answers a narrow question: if there were truly no effect, how surprising would data like ours be? A small p-value, often below 0.05, says the data would be fairly unusual under “no effect”. It does not tell you the chance your idea is true, and it says nothing about whether the effect is big enough to matter.

Test enough things and some will look significant by luck. Decide what you will test before you look, and report effect sizes, not just p-values.

An A/B test randomly splits people between two versions and compares an outcome. Randomness is the key: it makes the groups alike in everything except the change, so the difference can be credited to the change. Decide the sample size and the metric in advance, and do not stop early the moment it looks good.

Simpson's paradox is a warning about combining groups. A treatment can look better in every subgroup yet worse overall, if the groups differ in size and mix. Always ask whether a hidden variable is shaping the total.