HLTH 511 Week 1 Discussion Board Forum 1 Example

Discussion Board Forum 1: Statistical Significance and Public Health Evidence

HLTH 511 Research Methods in Public Health, Discussion Board Forum 1

Thread by [Student Name]

The classmates are composites; the textbook, Scripture and the peer-reviewed sources cited are real.

News stories often announce that a food, habit or treatment "significantly" affects health, and readers assume the effect must be large. In research, however, significant has a narrow meaning, and confusing it with important can mislead public health decisions.

Jacobsen (2017) explains that hypothesis testing compares observed data with what we would expect if the null hypothesis, usually no difference or no association, were true. The p-value expresses how surprising the data would be under that assumption. A small p-value suggests the data are hard to reconcile with the null hypothesis, but it says nothing about how large or useful the difference is. The textbook also stresses estimation, using a point estimate and confidence interval to describe how big an effect is and how precisely it was measured, as a complement to testing.

Wasserstein and Lazar (2016) published the six principles the American Statistical Association adopted to curb misuse of p-values. Among them, p-values can show how poorly data fit a stated model, yet they do not give the odds that the hypothesis being tested is right; findings and policies should never hinge solely on clearing a cutoff like .05; proper inference requires full reporting and transparency; and the size or practical weight of an effect cannot be read from a p-value. The sixth principle is that, by itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.

A practical example shows the problem. Imagine a study of 100,000 adults finding that people who drink a certain beverage have systolic blood pressure 0.4 mmHg lower than others, with p < .001. The result is statistically significant because the sample is huge, but a difference of less than half a point has no meaningful effect on heart disease risk. A result can be certain enough to pass a statistical test and still be too small to change anyone's health. Reporting the effect size and its 95% confidence interval would show readers immediately how small the difference is. The reverse problem also occurs: a small pilot study might find a meaningful 6 mmHg reduction with p = .09 and be dismissed as a failure, even though the effect, if real, would matter a great deal. Both errors come from treating a single threshold as the whole story.

Proverbs 18:17 warns that the first person to argue a case sounds convincing until someone else questions him (English Standard Version Bible, 2001/2016). As a Christian and a future public health professional, I see careful statistical reporting as a form of truthfulness. Communities make decisions about programs and funding based on research, and they deserve an honest account of how large, how certain and how important a finding really is. Reporting results that way honors both the participants who gave their time and the neighbors whose health depends on decisions made from the evidence.

Reply to Elise

Elise, I agree with your suggestion that researchers should report confidence intervals, and I think it helps to explain why they are so useful. Jacobsen (2017) describes a confidence interval as a range of values that is likely to contain the true population value, given the data and the method used. Unlike a p-value, it shows both the size of an effect and how precise the estimate is. For example, if a smoking cessation program raises quitting by an estimated 8 percentage points, with the 95% interval running from 6 to 10, we know the effect is meaningful and estimated fairly precisely. If the interval were -1 to 17, the same 8-point estimate would be far less certain, since the true effect could be nothing at all or quite large. A confidence interval that includes zero for a difference corresponds to a result that is not statistically significant at the matching level, but it also tells us how big the effect might plausibly be. Wasserstein and Lazar (2016) recommend reporting such estimates alongside p-values because they give decision makers the information a single p-value cannot. In public health, where we must decide whether a program is worth funding, knowing the plausible range of its effect is often more useful than knowing whether it crossed a threshold. Your point would make our research reports far more useful to the communities we serve, because board members and health officers can read a range of plausible effects even if they never learned what a p-value is.

Reply to Nathan

Nathan, your thread raised the important point that a non-significant result does not prove that a program has no effect. Jacobsen (2017) explains that statistical power, the probability of detecting an effect when one truly exists, depends heavily on sample size, the size of the effect and the variability of the data. A small study may simply be unable to detect a modest but real benefit. For example, a pilot study of a community walking program with 30 participants might find a non-significant improvement in blood pressure, even though a larger study would show a meaningful effect. Concluding from the pilot that walking programs do not work would be a mistake. Wasserstein and Lazar (2016) warn against treating p-values above .05 as proof of no effect for this reason. Calculating the needed sample size before a study begins, and reporting the confidence interval afterward, can help prevent these errors. I also think your point has ethical weight: if underpowered studies lead us to abandon helpful programs, the people who would have benefited bear the cost. Designing studies large enough to answer the question is part of being responsible with participants' time and with public resources. When a large study is not possible, researchers can still describe their pilot honestly as preliminary, report the confidence interval and call for a properly powered follow-up rather than a final verdict. That humility about what a single small study can show protects the public from both false hope and false discouragement about programs that may still work.

References

English Standard Version Bible. (2016). Crossway. (Original work published 2001)

Jacobsen, K. H. (2017). Introduction to health research methods: A practical guide (2nd ed.). Jones & Bartlett Learning.

Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129-133. https://doi.org/10.1080/00031305.2016.1154108