BUSI 600 Week 8 Group DB Forum 4: Part 3 Example

From p-Values to Decisions: A Consolidated Group Position on Analyzing and Reporting Business Research

Introduction

The final forum asked our group two questions about the last stages of a research project: what a statistically significant result tells a manager, and how researchers should report findings so that managers can use them. In Part 1, our threads explained hypothesis testing correctly but treated significance as the main thing a manager needed to know. Classmates' replies in Part 2 persuaded us that this emphasis was misplaced. Our consolidated thesis is that a business research report should lead with the size of an effect, its precision and what it means for the decision, and should present significance tests in a supporting role. The sections below explain how four recommendations led us to that position.

What a Significant Result Does and Does Not Show

Our Part 1 thread used the example of a composite home goods retailer that tested a redesigned product page. With 200,000 visitors in each version, the new page produced a conversion rate of 3.12% compared with 3.00%, and a test of the difference yielded p = .03. The thread concluded that the new page was better and should be adopted.

A classmate asked the question our thread skipped: would a gain of 0.12 percentage points pay for the change? Schindler (2019) presents hypothesis testing as a way of judging whether a difference in a sample probably reflects a difference in the population, and that judgment is silent on magnitude. The American Statistical Association's statement, written by Wasserstein and Lazar (2016), lists six principles, and our original thread broke several of them. In the statement's terms, the p-value describes how poorly the observed data fit one particular model, the model of no difference; it is not the chance that the redesign truly works, and a small p-value can sit beside an effect too small to care about. The authors also cautioned that a decision in business or policy should never hinge on a single bright line such as .05, and that full reporting of every analysis performed is part of proper inference.

Cohen (1994) made the same argument earlier and more sharply, criticizing the ritual of testing a null hypothesis of no effect, which in large samples will almost always be rejected, and urging researchers to report effect sizes with confidence intervals instead. Kirk (1996) distinguished statistical from practical significance and argued that researchers have an obligation to tell readers whether an effect is large enough to be useful. With 400,000 visitors, even a very small difference will reach significance; the question a manager must answer is whether it is worth acting on.

We therefore revised our example. The report now gives the estimated lift, 0.12 percentage points, and the range within which the true lift plausibly lies, about 0.01 to 0.23 points at 95% confidence, and it presents what that range means in revenue, as discussed below.

Planning Analyses Before Seeing the Data

A second classmate raised a problem none of us had considered. Our retailer example mentioned that the team had also examined results by device, by region and by new versus returning customers, and that the effect was strongest among returning mobile customers. The classmate asked how many comparisons had been run before that one was highlighted.

Simmons et al. (2011) demonstrated how flexibility in analysis inflates false-positive results. Using simulations and two experiments, they showed that routine choices, such as when to end data collection, which measures to report, whether to include covariates and which subgroups to examine, could raise the chance of finding a significant effect where none exists far above the nominal 5%. A business analytics team that examines dozens of segments and reports the one that reached significance is exposed to the same problem. We adopted the recommendation that our process require the primary comparison and any planned subgroup analyses to be written down before a test begins, with all other findings labeled as exploratory and treated as hypotheses for the next test.

Association, Causation and the Language of the Report

The third recommendation concerned our second Part 1 thread, which used a correlation from a customer survey, r = .41 between satisfaction with delivery speed and repeat purchase, to recommend investing in faster shipping. A classmate noted that the thread treated the correlation as a cause. Antonakis et al. (2010) reviewed causal claims in management research and found that many studies drew causal conclusions from correlational data without addressing omitted variables, simultaneity or measurement error. In our example, customers who buy more often may simply be more satisfied with everything, including delivery, or they may live in urban areas where delivery is both faster and more convenient.

Schindler (2019) distinguishes measures of association from evidence of causation, and we now apply that distinction in the language of our reports. Correlational findings are described as relationships, with plausible alternative explanations named, and recommendations based on them are framed as candidates for a controlled test. In the shipping example, the report recommends a test in which a random set of orders receives expedited shipping at no charge, so that the effect on repeat purchases can be measured directly.

Translating Findings Into Business Terms

The final recommendation was to express results in units managers use. Aguinis et al. (2010) argue that researchers should report findings with their practical impact in mind, translating effect sizes into outcomes that decision makers value. For the retailer, an increase of 0.12 percentage points on 1.5 million monthly visitors means about 1,800 additional orders a month. At an average order value of $85, that is roughly $153,000 in monthly revenue, though the confidence interval implies a plausible range from about $17,000 to $289,000. Set beside a redesign cost of $60,000, the central estimate repays the investment quickly in revenue terms, and the real decision turns on whether the margin earned near the low end of the range would still cover it, which is exactly the conversation a report should prompt. Presented this way, the result supports a decision; presented as p = .03, it supported only a belief that the page was better.

How the Recommendations Changed Our Position

Biblical Integration

Proverbs 12:22 sets lying speech against faithful dealing and says which one God takes pleasure in. A research report can mislead without containing a single false number, by presenting a trivial effect as important, by highlighting the one significant subgroup out of many or by implying a cause the data cannot show. Truthful reporting means giving decision makers the size, the uncertainty and the limits of a finding, so that they can act wisely. The recommendations our group adopted are, in that sense, practices of honesty as much as of statistics.

Conclusion

Our group began this forum treating statistical significance as the main finding of a study. The Part 2 replies moved us to a position that we believe better serves managers: lead with the effect size and its uncertainty, plan analyses before seeing the data, describe associations as associations and translate results into the units of the decision. These practices make research reports less impressive in appearance and more useful in practice, which is the purpose of business research. They also change the relationship between researchers and managers. A report that shows its uncertainty invites the manager into the judgment instead of handing down a verdict, and a team that labels exploratory findings as exploratory earns trust that lasts beyond a single project. Our group will carry these habits into the case assignments and into any research we commission in our own organizations.

References

Aguinis, H., Werner, S., Abbott, J. L., Angert, C., Park, J. H., & Kohlhausen, D. (2010). Customer-centric science: Reporting significant research results with rigor, relevance, and practical impact in mind. Organizational Research Methods, 13(3), 515-539. https://doi.org/10.1177/1094428109333339

Antonakis, J., Bendahan, S., Jacquart, P., & Lalive, R. (2010). On making causal claims: A review and recommendations. The Leadership Quarterly, 21(6), 1086-1120. https://doi.org/10.1016/j.leaqua.2010.10.010

Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49(12), 997-1003. https://doi.org/10.1037/0003-066X.49.12.997

Kirk, R. E. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56(5), 746-759. https://doi.org/10.1177/0013164496056005002

Schindler, P. S. (2019). Business research methods (13th ed.). McGraw-Hill Education.

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359-1366. https://doi.org/10.1177/0956797611417632

Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129-133. https://doi.org/10.1080/00031305.2016.1154108