Uncertainty and hypothesis tests

An effect estimate, its uncertainty and a hypothesis test answer different questions. Reporting them together allows a result to be assessed without treating a significance threshold as a verdict on the treatment's clinical value.

Estimates and sampling variation

Inferential statistics use sample measurements to estimate characteristics of the population from which the sample was drawn. ^{1, 2, 3} Estimates can describe a population trait, a difference between groups or a change within a group. ⁴ A hypothetical illustration starts with a gnome measuring 10 cm and estimates average height as 10 cm; adding an 11-cm gnome changes the sample mean to 10.5 cm. ⁴ A sample consisting of unusually small or large gnomes would under- or overestimate population height, showing why the estimate needs a measure of precision. ⁴

Standard deviation (SD) describes variation among observations, whereas the standard error (SE) of the mean describes variation among the means of repeated samples of the same size. ^{5, 6, 7} For the ordinary mean of independent observations, SE = s/√n, where s is sample SD and n is sample size. ^{8, 9, 7} For a positive SD and n greater than one, SE is smaller than SD and becomes smaller as sample size grows for a fixed SD. ⁸ Using SE to describe variation among individuals can therefore make the data appear less variable than reporting SD. ¹⁰ SD is preferred for describing the data distribution, while a confidence interval is preferred for presenting an estimate's precision. ^{10, 2}

Confidence intervals

The sampling distribution can be pictured by repeatedly taking random samples of a fixed size, calculating each mean, plotting the means and returning the sampled observations to the population before repeating. ¹¹

A 95% confidence interval is produced by a procedure that covers the population parameter in about 95% of repeated samples under the procedure's assumptions. ^{12, 13, 14} It does not establish that the true mean lies inside the particular interval observed. ¹³ Different samples yield different estimates and intervals: in a teaching example, a party's mean monthly income was €2,446 with a 95% interval of €2,381–€2,511, while another party would yield different values. ^{15, 16}

The 95% level is conventional; 90% and 99% levels are also used. ^{12, 14} For the same data and construction, a 90% interval is narrower and a 99% interval wider than a 95% interval. ^{14, 17, 18} Interval width depends on three factors. ¹⁴

1. Confidence level: greater coverage requires a wider interval.
2. Data variability: greater variability widens the interval.
3. Sample size: larger samples narrow it; under the inverse-square-root relationship, halving the width requires four times the sample size.

^{14, 17}

An illustrative study of 100 patients defined response as HbA1c below 7.0% at study end and observed 33 responders. ¹⁴ Its response estimate was 33%, with an adjusted Wald 95% interval of 25%–43%. ^{14, 17} The limits are in the same units as the estimate, here the percentage of responders. ¹⁹ A 25% response rate and a 43% response rate may have different clinical implications, so the interval can reveal insufficient precision for a firm clinical judgment. ¹⁹

Construction and reporting

For the simple random-sampling interpretation, three conditions matter. ²⁰

1. The sample is drawn randomly from the population of interest.
2. Observations are independent; family-related recruitment can undermine that assumption.
3. Data are calculated and tabulated correctly, without systematic mistakes introducing bias.

²⁰

For a normally based interval, the estimate is combined with its SE and a critical value: the usual normal multipliers are approximately 1.96 for 95%, 1.64 for 90% and 2.58 for 99% coverage. ^{12, 21} A Student t critical value depends on degrees of freedom and approaches the normal value as degrees of freedom increase. ^{22, 23} Skewed distributions may call for transformation, nonparametric intervals or bootstrapping. ¹² Several interval methods exist, so the method used matters. ¹³

For a symmetric normal-based 95% interval, SE can be recovered as the interval width divided by 3.92; corresponding divisors are approximately 5.15 for 99% and 3.29 for 90% intervals. ²⁴ For an independent-group difference with a common within-group SD, the conversion is SD = SE/√(1/nE + 1/nC), where nE and nC are group sizes. ²⁴

Report the point estimate, its unit and both confidence limits, as in 8.6% (95% CI 8.4%–8.8%), rather than 8.6% (95% CI ±0.2%). ²⁵ This avoids confusion with SD or SE and saves readers from calculating the limits. ²⁵

Intervals for comparisons

For a two-sided comparison corresponding to the interval, a 95% interval for a difference containing zero, or for a risk ratio containing one, does not reject that null value at the 0.05 level. ²⁶ The comparison concerns the interval for the difference or ratio itself. ²⁶

An interval containing both clinically important and clinically unimportant values leaves uncertainty about clinical benefit, even if the result is statistically significant. ^{6, 27, 28} A reporting example gives a mean diastolic blood pressure reduction of 15 mm Hg, a 95% interval of 3.5–26.5 mm Hg and P = 0.01. ²⁹ If 15 mm Hg is clinically important but 3.5 mm Hg is not, the interval includes both clinically important and unimportant reductions. ⁶ Intervals containing only clinically important values, or only clinically unimportant ones, give a more definite clinical answer; interpretation therefore needs the range of effects, not simply interval width. ^{6, 27, 28}

Null hypotheses and P values

A null hypothesis states the relationship or value tested, often no difference between groups; the test rejects it or fails to reject it. ³⁰ A P value assesses the probability of a result at least as extreme as the observed result under the null hypothesis and the test’s assumptions. ³¹ A small P value does not give the probability that the alternative hypothesis is true or that the observed difference represents a true population difference. ³¹

A probability for one exact outcome is distinct from the probability used for a hypothesis test. ^{32, 31} In a fair-coin example, the binomial probability of exactly 80 heads in 100 tosses is 100!/[80!20!] × 0.5¹⁰⁰, approximately 4.23 × 10⁻¹⁰. ³²

A simplified diabetes trial illustrates a two-sided question. ^{33, 34}

1. Randomise participants to a new compound or look-alike placebo, keeping participants and study personnel blinded.
2. At 24 weeks, compare mean HbA1c: the null is that the means are equal, and the alternative is that they differ in either direction.
3. Set the significance threshold at 0.05 before testing.
4. Suppose the observed difference is one percentage point with SE 0.413 percentage points; the standardised statistic is (1 − 0)/0.413 = 2.42.
5. Under a standard normal reference distribution, the two-sided P value is approximately 0.0155, below 0.05, so the null is rejected while recognising the possibility of a false positive.

^{33, 34, 35, 36}

The significance threshold and whether the test is one- or two-sided should be specified in advance. ^{34, 30, 37} Conventional thresholds include 0.05 and 0.01, also called significance or alpha levels. ³⁸ Test results depend on sample size, sample variation and the method applied. ^{39, 40, 41} Data errors also affect the result, so a P value does not overcome problems in the data. ^{42, 43}

Errors and power

A type I error, or false positive, rejects a true null hypothesis; a type II error, or false negative, fails to reject a false null hypothesis. ^{44, 45, 46, 38} In drug development, a false positive can expose patients to an ineffective treatment and its adverse effects. ⁴⁵ The two error risks are denoted by alpha and beta respectively. ⁴⁴ Power is the probability of rejecting a false null for the specified effect, and equals 1 − beta. ^{44, 47, 48} Common design targets are 80% or 90%; an 80% target corresponds to a 20% false-negative risk. ^{44, 37, 49} Lowering the significance threshold from 0.05 to 0.01 reduces false positives but increases false negatives for a fixed study. ⁵⁰ The term type III error has also been used for asking an incorrect or inappropriate question. ⁴⁴

A P value above the chosen threshold means insufficient evidence to reject the null; it does not show that the null is true, that no effect exists or that groups are similar. ^{31, 46, 51, 47, 48} An underpowered study may simply be inconclusive about the clinically important effect. ^{47, 49, 48, 52} Reporting the effect and its interval helps distinguish uncertainty about an important difference from a claim that no effect exists. ^{31, 2} Accordingly, “there was no statistically significant difference between the means” with its P value reports the test result more accurately than “the mean values did not differ”. ⁵³

Multiple comparisons

Secondary endpoints, subgroup analyses and interim analyses can be clinically relevant, but multiple tests increase the risk of false-positive findings. ^{45, 54, 55} In the illustrative case of ten independent tests of true null hypotheses at alpha 0.05, the probability of at least one false positive exceeds 40%. ^{45, 56} The probability that all n such tests avoid a false positive is 0.95ⁿ. ⁵⁶

Bonferroni adjustment divides the overall alpha by the number of comparisons: ten tests with overall alpha 0.05 use 0.005 per comparison, while five use 0.01. ^{45, 57, 37, 58, 56} It is a conservative approach. ⁴⁵ The number used for adjustment should reflect tests performed, which may exceed those reported. ⁵⁹

The Benjamini–Hochberg approach targets the false discovery rate. ^{45, 54} It ranks P values from smallest to largest, compares each with (i/m) × q, where i is rank, m is number of comparisons and q is the target false discovery rate, and treats the largest value below its critical value and all smaller values as significant. ⁵⁴

Review of multiple testing should address three questions. ⁵⁴

• Were the comparisons appropriate for the study and disease setting?
• Were they prespecified in the protocol or analysis plan?
• Were appropriate measures used to control type I error?

⁵⁴

Failure to address multiplicity can overstate significance and support unsubstantiated efficacy claims. ⁵⁴

Statistical significance and clinical importance

A P value does not establish effect magnitude or clinical meaningfulness. ^{31, 38, 60, 2, 29, 61} A smaller P value need not imply greater clinical importance: large samples can make very small effects statistically significant. ^{31, 41, 61} An illustrative epidemiological study of four million people could yield a statistically significant odds ratio of 1.05 without establishing that the association matters clinically. ⁶²

Results should include the treatment effect and its confidence interval, rather than an isolated P value. ^{61, 2, 29, 63} Minimum clinically important differences are medical judgments, rather than consequences of a statistical test. ⁶¹ Claims of clinical importance should be anchored in quantitative, validated thresholds where available. ⁶⁴ Calling an effect “highly significant” or “borderline significant” adds a judgment to the P value; reporting the actual result avoids these modifiers. ^{58, 41}

The word “statistically” should be explicit when significance refers to a test, since clinical significance is different. ^{65, 66, 67} For example, a sentence reporting adverse events in 79% of treated patients versus 72% on placebo (P = 0.19) should specify that the difference was not deemed statistically significant. ^{68, 65} A following sentence beginning “similarly” does not clarify an earlier ambiguous use. ⁶⁵ A P value or confidence interval usually signals statistical usage, but does not always remove ambiguity. ^{66, 69, 67}

Interpretation should acknowledge uncertainty rather than reduce findings to significant or nonsignificant. ⁷⁰

Proposals about significance thresholds

A proposal discussed in 2018 and signed by 72 statisticians recommended lowering the default threshold for new discoveries from 0.05 to 0.005 and calling findings between those thresholds suggestive. ⁷¹ Its scope was limited to null-hypothesis significance tests claiming discovery of new effects, not confirmatory or contradictory replications. ⁷¹ The signatories argued this could improve reproducibility, while acknowledging different views and preferences among them for Bayes factors or posterior summaries under explicit model assumptions. ⁷¹ A concern was that the change could favour large studies and concentrate funding among fewer groups. ⁷¹

Five proposed changes to statistical interpretation accompanied that debate. ⁷¹

• Adjust for human cognition — Jeff Leek.
• Abandon statistical significance — Blakeley McShane and Andrew Gelman.
• State false-positive risk too — David Colquhoun.
• Share analysis plans and results — Michèle Nuijten.
• Change norms from within — Steven Goodman.

⁷¹

In a 2018 viewpoint, John Ioannidis suggested 0.005 for new discoveries could encourage fewer, larger and more carefully designed studies, while warning that attempts to cross the lower threshold could increase bias. ⁷²

In 2019, more than 800 researchers supported retiring statistical significance as a dichotomy, while retaining P values. ⁷³ They proposed describing confidence intervals as compatibility intervals, discussing compatible effects without overconfidence, and avoiding declarations of no difference or no association based only on a threshold crossing. ⁷³ They also discouraged replacing P-value categories with categories based on other statistical measures such as Bayes factors. ⁷³ Ioannidis argued for retaining a filtering role for significance, potentially with a stricter threshold, to avoid being overwhelmed by noise. ⁷³ He held that significance, beyond its statistical sense, is necessary for science and science-based action. ⁷³

Numerical reporting and checks

A P value should be accompanied by enough information to understand its calculation. ^{69, 67} Reporting should identify the actual value, alpha level, test, relevant assumptions such as pairing or independence, one- or two-tailed testing and software. ^{69, 67} Results should include absolute numbers, percentages and estimated group differences with 95% intervals; a statistical reviewer's advice is to avoid baseline significance tests in randomised trials. ⁷⁰

Reporting conventions differ by destination. ^{74, 75} A 2009 description of AMA journal style omitted leading zeros, used capital italic P and thin spaces around inequalities, expressed values below .00001 as P < .001 and values above .999 as P > .99. ⁷⁵ Omitting leading zeros can distinguish P values in busy parenthetical text and save table space. ⁷⁴ Consistency within the document matters, and editors may apply their journal's conventions. ^{75, 76} For rounded extreme results, a teaching recommendation was to use bounds such as P < 0.0001 or P > 0.9999 rather than display 0.0000 or 1.0000. ⁵⁹

Reported P values can be checked against the test statistic and degrees of freedom. ⁷⁷ A study of 49 psychology papers found 49 reporting errors, 96% with reported P values smaller than recalculated values; seven papers contained findings incorrectly presented as significant. ⁷⁷ A separate analysis of 281 psychology papers found incorrect assignment of significance or nonsignificance for at least one result in 15% of papers. ⁷⁷

In a text-mining study of biomedical literature from 1990–2015, investigators identified 4,572,043 P values in 1,608,736 MEDLINE abstracts and 3,438,299 in 385,393 PMC full-text articles. ⁶³ The proportion of abstracts reporting P values rose from 7.3% in 1990 to 15.6% in 2014. ⁶³ Values clustered at 0.05 and at 0.001 or below, and abstract P values were generally lower than those in full texts. ⁶³ Qualitative significance claims were more common than numerical P values, often without other quantitative information, and few articles reported intervals, Bayes factors or effect sizes. ⁶³

Reference distributions

A critical-value lookup depends on the distribution, degrees of freedom where applicable, and whether one or both tails are included. ^{21, 23, 78} For the standard normal distribution, 1.96 corresponds to upper-tail probability 0.025 and combined two-tail probability 0.05. ²¹ For a t distribution with five degrees of freedom, the corresponding two-sided 0.05 critical value is approximately 2.571. ²³ Chi-squared tables may instead label right-tail probabilities, which should be read on their stated scale. ⁷⁸