How to calculate a p-value, step by step
A p-value is the area in the tails of the null distribution beyond your test statistic — how surprising your result would be if nothing were going on. To calculate a p-value, take your test statistic, find the area under the null curve beyond it, and double it if the test is two-tailed. On the defaults — z = 1.96, two-tailed — the p-value is 0.049996, because the normal curve up to 1.96 holds 0.9750 of its area, the tail beyond it is 0.0250, and doubled it is 0.0500. Work it out in the p-value calculator.
The worked example: z = 1.96, two-tailed
The worked example, line by line. Step 1 — the statistic: z = 1.96, two-tailed, on the normal curve. Step 2 — the CDF: the area under the bell curve up to 1.96 is 0.9750, so 97.50% of the curve sits below it. Step 3 — the upper tail: 1 − 0.9750 = 0.0250, the area beyond 1.96 on one side. Step 4 — double it for two tails: 2 × 0.0250 = 0.0500, the table value. Step 5 — the exact figure: the calculator gives 0.049996, a hair under the table value, because the table rounds to four decimals. Step 6 — the verdict: 0.049996 is below 0.05, so the result is significant at 0.05. The same six steps work for any statistic: find the CDF, take the tail, double it if the test is two-tailed, and compare with 0.05.
What a p-value is
A p-value is the probability of seeing a test statistic at least as extreme as yours, if the null hypothesis were true. It is not the probability that the null hypothesis is true, and it is not the size of the effect — it is only how surprising the data would be if nothing were going on. A p-value of 0.049996 means a result this far out would show up about 5% of the time by luck alone. A p-value of 0.50 means the result is exactly what the null predicts, and a p-value of 0.0027 means it would show up about 3 times in a thousand. The smaller the p-value, the more surprising the data under the null — and the more reason to doubt it.
z-tests and t-tests
A z-test uses the normal curve and is for large samples or a known standard deviation; a t-test uses the t curve with degrees of freedom and is for small samples. The t curve has fatter tails than the normal curve, so the same statistic gives a larger p-value: t = 2.0 with 10 degrees of freedom is p = 0.0734, while z = 2.0 is p = 0.0455. With 10 degrees of freedom the 5% point is t = 2.228, not 1.96 — the t curve needs a bigger statistic to reach the same p-value, because its tails hold more area. As the degrees of freedom grow, the t curve closes in on the normal curve, and at 100 degrees of freedom the two are nearly the same. Pick the test that matches your sample: a z-test for a large sample, a t-test for a small one, and never a z-test on a small sample just because the p-value is smaller.
One tail or two
Two-tailed tests both directions — a result far out either way counts — while one-tailed tests one direction only. z = 1.96 is p = 0.0500 two-tailed and p = 0.0250 one-tailed, exactly half, because one tail is half of two. The choice has to come before you look at the data: picking one tail after seeing the result halves your p-value for free, which is why a one-tailed test needs a reason stated in advance — a new drug can only help, a machine can only run fast — and a two-tailed test is the default whenever the direction is not fixed. If you are not sure, test two tails: it is the honest default, and the p-value it gives is the one a reader will expect.
The 0.05 line and what it does not mean
The usual line is 0.05: below it, the result is called significant; above it, not significant. But 0.05 is a line, not a law — a p-value of 0.049996 is significant at 0.05 and a p-value of 0.0500 is not, and nothing real changes between them. Report the p-value itself, not just the word significant, and a p-value just over the line is weak evidence, not proof of nothing. Significance is not importance: a tiny effect with a huge sample can be significant, and a large effect with a small sample may not be, because the p-value mixes the size of the effect with the size of the sample. And a p-value says nothing about how likely the null hypothesis is — it is the probability of the data given the null, not the probability of the null given the data.
Common mistakes
The first mistake is picking the tail after looking — a one-tailed test chosen after the result is a two-tailed test with its p-value halved, and no reader will accept it. The second is reading p as the probability that the null is true — a p-value of 0.05 does not mean there is a 5% chance the null is true, it means the data would be this surprising 5% of the time if the null were true. The third is treating 0.05 as a cliff — 0.049996 and 0.0500 are the same evidence, and the line between them is a convention, not a discovery. The fourth is confusing significance with size — a significant p-value on a huge sample can be a trivial difference, and a non-significant one on a small sample can hide a large one. The fifth is testing the wrong curve — a z-test on a small sample gives a smaller p-value than the t-test it should have been, and the answer flatters the result. Work it out in the p-value calculator, and check the z-score calculator and the standard-deviation calculator before you decide what the answer means.
From a p-value back to a z-score
Sometimes the question runs the other way: what z-score gives a p-value of 0.05? The smallest z for a two-tailed p-value of at most 0.05 is 1.96 — at 1.95 the p-value is 0.0512, just over the line. For a p-value of at most 0.01 the smallest z is 2.58, and for a one-tailed p-value of at most 0.025 it is 1.96 again, because one tail at 0.025 is two tails at 0.05. The same idea works for a percentile: the smallest value for a z-score of at least 2 is 90 against a mean of 70 and a spread of 10, and the smallest value for a percentile of at least 95% is 86.5 — the z-score calculator works those reverses out for any value, mean and spread. Going backwards is the same calculation in reverse: pick the p-value you want, and find the statistic that gives it.