P-value calculator

A p-value is the probability of observing results at least as extreme as yours, assuming the null hypothesis is true. That conditional clause is the whole meaning, and it's the part most often dropped.

The null hypothesis is the boring explanation: no difference, no effect, no relationship. The p-value asks how surprising your data would be in a world where that boring explanation held. A small p-value means your data would be very surprising in that world, which is grounds to doubt it.

A p-value of 0.03 means: if there were genuinely no effect, you'd see a result this extreme roughly 3% of the time.

Compare it against your significance level, α, chosen before you look at the data.

α = 0.05 is the usual default. Use 0.01 when a false positive is costly, and 0.10 for exploratory work where missing a real effect matters more than chasing a false one.

Failing to reject is not proving

A p-value of 0.4 does not show the null hypothesis is true. It shows your data are consistent with it — and also with many small effects your test was too weak to detect. Absence of evidence is not evidence of absence.

Two-tailed

Asks whether there's a difference in either direction. The α is split between both tails, so at α = 0.05 each tail holds 0.025. This is the default and the right choice in most situations.

One-tailed (left or right)

Asks whether the effect goes in one specific direction. All of α sits in that tail, which makes the test more sensitive — a one-tailed p-value is exactly half the two-tailed one for the same statistic.

Z-score

For large samples (roughly n > 30) or when the population standard deviation is known. Also the basis for comparing two proportions, as in A/B testing.

T-score

For smaller samples where you estimate the standard deviation from the data. The t distribution has heavier tails than the normal, which is how it accounts for that extra uncertainty — so the same statistic yields a larger p-value. Degrees of freedom are usually n − 1.

As degrees of freedom rise the two converge; past about 100 the difference is negligible.

Chi-square

For categorical data — testing whether observed frequencies differ from expected ones, or whether two categorical variables are independent. Common with survey cross-tabs.

For a contingency table, degrees of freedom are (rows − 1) × (columns − 1). Chi-square tests are inherently right-tailed, since only large values indicate a departure from expectation.

Your historical average satisfaction score is 7.2 out of 10. After a service change you survey 25 customers and get a mean of 7.8 with a standard deviation of 1.4. Did satisfaction really improve?

Hypotheses. H₀: the true mean is still 7.2. H₁: it has changed.

Test choice. Small sample with an estimated standard deviation, so a t-test with df = 25 − 1 = 24.

Entering t = 2.143 with 24 degrees of freedom, two-tailed, gives a p-value of 0.0425.

Since 0.0425 < 0.05, you reject the null hypothesis: the improvement is statistically significant at the 5% level. Whether a 0.6-point rise is worth what the change cost is a separate question, and not one statistics can answer.

• Reading it as the probability of being wrong. p = 0.03 does not mean a 3% chance the finding is false.
• Treating 0.05 as a cliff. p = 0.049 and p = 0.051 carry essentially identical evidence. Report the value, not just the verdict.
• Testing until something works. Run twenty comparisons at α = 0.05 and about one will come up significant by chance. Correct for multiple comparisons.
• Confusing significance with importance. Large samples make trivial differences significant. Always report the effect size alongside.
• Picking the tail after seeing the data. Decide up front.
• Ignoring the assumptions. Every test assumes something about your data — independence, distribution, minimum expected counts. A p-value from a violated assumption is just a number.

Typical survey questions that call for a significance test:

• Has satisfaction changed since last quarter? — t-test on the two means
• Do two segments answer differently? — chi-square on the cross-tab
• Did variant B get a better response rate? — two-proportion z-test
• Is this month's NPS genuinely up? — compare with the confidence interval

One caveat worth keeping in mind: significance tests assume a random sample. Survey data often isn't — respondents self-select, and the people who reply differ from those who don't. The p-value quantifies sampling noise only. It cannot tell you that the wrong people answered.

Before running the test, check your margin of error — if the intervals around your two numbers overlap heavily, you already know the answer.