Statistically Significant vs Practically Significant: What a T-Test Can't Tell You
A t-test answers one narrow question: is the difference I observed larger than sampling noise would plausibly produce? It does not answer whether the difference matters. Those are separate questions, and treating a p-value as if it settled both is the most common way t-test results get misused in engineering and business decisions.
This article works through both directions of the mismatch: results that are statistically significant but practically meaningless, and results that are practically important but statistically inconclusive. It then covers what to report so that readers can judge both.
Two kinds of significance
Statistical significance means the p-value fell below your threshold, typically 0.05. It's a statement about evidence: the data are inconsistent with the null hypothesis of no difference.
Practical significance means the difference is large enough to matter for the decision at hand. It's a statement about consequences: the difference would change a cost, a risk, a specification, or a customer outcome.
The t-test computes the first. Only you can judge the second, because it depends on context the test knows nothing about: tolerances, costs, what the measurement is used for.
The relationship between them runs through sample size. Statistical significance depends on the ratio of the observed difference to its standard error, and the standard error shrinks with √n. Any non-zero difference, however trivial, becomes statistically significant if you collect enough data. Conversely, a large and important difference can fail to reach significance if the sample is small and noisy.
Case 1: significant but meaningless
Two production lines make the same shaft. Over a month you collect 500 measurements from each.
Line 1 Line 2
n 500 500
Mean (mm) 25.010 25.030
Std. dev. (mm) 0.15 0.15
SE = 0.15 · √(1/500 + 1/500) = 0.15 · 0.0632 = 0.0095 t = 0.020 / 0.0095 = 2.11, df = 998, p ≈ 0.035
Statistically significant. The lines differ. A report that stops there will send someone off to investigate why Line 2 runs high.
Now add the context: the specification is 25.00 ± 0.50 mm. Both lines are centered within 0.03 mm of nominal on a tolerance band of 1.00 mm. The 95% confidence interval for the difference is 0.020 ± 0.019, or (0.001, 0.039) mm. Even the upper end is 4% of the tolerance. The measurement gauge itself probably has more error than that.
The difference is real. It's also irrelevant. Nobody should spend an hour on it.
This pattern shows up constantly with automated data collection. When you have thousands of observations, the t-test will find differences everywhere, because there are always small differences between any two things. Significance stops being informative and the confidence interval becomes the only number worth reading.
Case 2: important but inconclusive
A supplier proposes a new alloy that should improve a fatigue-critical dimension. Testing is destructive and expensive; you get 6 samples of each.
Current Proposed
n 6 6
Mean (mm) 25.10 25.40
Std. dev. (mm) 0.35 0.35
SE = 0.35 · √(1/6 + 1/6) = 0.35 · 0.577 = 0.202 t = 0.30 / 0.202 = 1.49, df = 10, p ≈ 0.17
Not significant. A report that stops there says "no evidence of a difference" and the proposal dies.
Add the context: a shift of 0.30 mm on this dimension is expected to extend fatigue life by 20%, which is worth a great deal. The 95% confidence interval is 0.30 ± 2.228 · 0.202 = (−0.15, 0.75) mm. The data are consistent with the new alloy doing nothing, and equally consistent with it delivering a 0.75 mm improvement, more than double the target.
This result doesn't say the alloy doesn't work. It says six samples can't tell. The correct decision is to fund a larger test, because the potential upside is large and the current data haven't ruled it out. With 20 samples per group and the same means and standard deviation, t would be 2.71 and p about 0.010.
Underpowered studies that get interpreted as "no effect" are how good ideas get killed. A non-significant p-value with a wide confidence interval is an absence of evidence, not evidence of absence.
How to judge practical significance
The t-test won't do this for you, so decide in advance what difference would matter. Three common reference points:
The specification or tolerance. A difference that's less than 10% of the tolerance width is rarely worth acting on. A difference that consumes a substantial fraction of the tolerance is a problem even if the means are both technically in spec.
The cost or outcome it drives. If a 2-second cycle-time reduction is worth 13 minutes per shift, that's the threshold. If a 0.5% yield improvement is worth a defined sum per year, that's the threshold. Convert the measured difference into the unit that decision-makers care about.
A standardized effect size. Cohen's d expresses the difference in units of standard deviation. Values around 0.2 are small, 0.5 medium, 0.8 large. It's useful when there's no natural tolerance or cost, though it's a rough guide and shouldn't replace domain judgment where domain judgment exists.
Write the threshold down before you look at the data. Deciding what matters after seeing the result is a reliable way to convince yourself of whatever you already believed.
Read the confidence interval against the threshold
Once you have a practical threshold, the confidence interval does the work. There are four cases:
Interval position Statistically significant? Practically significant? Conclusion
Entirely above threshold Yes Yes Real and important. Act.
Excludes zero, entirely below threshold Yes No Real but trivial. Ignore.
Includes zero and threshold No Unknown Underpowered. Collect more data.
Includes zero, entirely below threshold No No Equivalent for practical purposes.
The fourth row is worth noting. If the interval includes zero but its entire width sits below the threshold that would matter, you have positive evidence that any difference is too small to care about. That's a stronger and more useful conclusion than "not significant," and it's the logic behind formal equivalence testing.
What to report
A t-test result that supports a decision should include:
The observed difference in the original units.
The 95% confidence interval for that difference.
The practical threshold and where the interval falls relative to it.
The t-statistic, degrees of freedom, and p-value, for completeness.
For Case 1: "Line 2 averages 0.020 mm higher than Line 1 (95% CI 0.001 to 0.039 mm; t(998) = 2.11, p = 0.035). The difference is statistically detectable but is under 4% of the tolerance band and not practically relevant."
For Case 2: "The proposed alloy averaged 0.30 mm higher (95% CI −0.15 to 0.75 mm; t(10) = 1.49, p = 0.17). The result is not statistically significant, but the interval includes the 0.30 mm target and extends well above it. Six samples per group cannot resolve the question; a 20-per-group study is recommended."
Both statements give the reader what they need. Neither reduces the finding to a single word.
Tools
A two-sample t-test calculator that reports the confidence interval alongside the p-value makes the comparison against a practical threshold direct: enter the summary statistics, read the interval, and check where it sits relative to the difference that would matter. Trying different sample sizes with the same means and standard deviations also shows how quickly a trivial difference becomes "significant" as n grows, which is the fastest way to build the right instincts.
For background on what the p-value and t-statistic represent and why the test is constructed the way it is, the Student's t-test overview covers the fundamentals.
Summary
Statistical significance tells you a difference is detectable. Practical significance tells you it matters. Large samples make trivial differences significant; small samples leave important differences unresolved. Decide what difference would matter before you test, report the confidence interval, and read the interval against that threshold. The p-value is one input to the decision, not the decision itself.