BUSI 600 Week 4 Group DB Forum 2: Part 3 Example

Fitting Research Design to the Business Question: A Consolidated Group Position

Introduction

Business research is useful only when its design matches the decision it is meant to inform. In Part 1 of this forum our group answered two questions: when a researcher should choose qualitative methods over a survey, and what an experiment in a real workplace can prove that questionnaire data cannot. Our threads agreed that no design is best in general. The replies in Part 2, however, pressed us to be more specific about how designs combine, what to do when an experiment is impractical and how to protect survey findings from bias. This paper presents our consolidated position, organized around the questions a manager actually brings to a researcher, and shows how each recommendation changed it.

Our thesis is that the decision a study serves should determine its design. If the manager needs to learn what is happening and why, the study must be exploratory; if the manager needs to know how common a condition is, it must be descriptive; and if the manager needs to know whether an action will cause a result, the study must be causal (Schindler, 2019). The sections below follow that sequence.

Discovering and Measuring: Qualitative Work and Surveys in Sequence

Our original thread on qualitative research argued that interviews and observation are the right choice when concepts are not yet understood, drawing on the principle of methodological fit set out by Edmondson and McManus (2007). In nascent areas, open-ended data collected in the field are needed to identify constructs, while mature areas call for established measures and statistical tests. We applied this to a software company whose trial users stopped logging in after a week, reasoning that no one at the firm yet knew which of many possible causes applied.

Two classmates recommended that we stop presenting qualitative work and surveys as alternatives. One noted that a manager cannot act on twenty interviews alone, because the interviews do not show how widespread each reason is. We agree and have adopted a sequential design. Johnson and Onwuegbuzie (2004) describe mixed methods research as a practical approach that combines qualitative and quantitative elements so that each compensates for the other's weaknesses. In our revised design, interviews with users who abandoned the trial generate the list of reasons, coded with the systematic procedure Gioia et al. (2013) describe, from informants' own terms to broader themes. A survey of several hundred trial users then estimates how often each reason occurs and which segments it affects. The manager receives both an explanation and a priority order.

The revision changed our answer from "choose qualitative or quantitative" to "decide what each stage must establish." This framing also clarifies sample design. The interview stage needs variety, users who left at different points and for different apparent reasons, while the survey stage needs a sample large enough and representative enough to support estimates.

Establishing Cause: Field Experiments and Their Alternatives

Our second thread used Bloom et al. (2015) to show what random assignment adds. At Ctrip, whether a volunteering call-center agent spent nine months at home or at a desk depended on whether the agent's birth date was odd or even; home workers' performance rose by 13% and their attrition fell by about half. Because assignment was random, these differences could be attributed to the arrangement rather than to who chose it.

A classmate agreed but observed that most managers cannot run such a trial: randomizing employees can be disruptive, unfair or simply impossible when a decision applies to everyone at once. In response we added a feasibility test. A field experiment is appropriate when the firm can assign the treatment to some units and not others, when the outcome can be measured from records rather than opinions, when the trial can run long enough for effects to appear and when withholding the treatment from a control group raises no serious ethical problem. Eden (2017) reviews field experiments in organizations and argues that they are underused relative to their value, in part because researchers overestimate how hard they are to arrange; staggered rollouts, in which units adopt a change at different times, can supply a comparison group without denying anyone the change permanently.

A second classmate recommended experimental vignettes for questions about judgment, such as how customers would respond to a price change or how managers would rate candidates. Aguinis and Bradley (2014) explain that vignette studies present participants with carefully constructed scenarios whose features are varied systematically, allowing causal inferences about how those features affect decisions. They also caution that realism must be built into the scenarios so that results transfer to real choices. We now present vignette experiments as a middle option between a correlational survey and a full field experiment.

Protecting Survey Findings From Method Bias

The final recommendation concerned surveys that must stand alone. A classmate noted that our thread criticized self-reports without saying what to do when a survey is the only feasible design. Podsakoff et al. (2003) recommend both procedural and statistical remedies for common method bias. The procedural remedies are the most practical for business researchers: obtain the predictor and the outcome from different sources, such as employee surveys paired with company performance records; separate the measurement of the two in time; protect respondent anonymity; and word items to reduce ambiguity and social desirability. We have added these remedies to our recommendations for any survey-only study. In the software example, that would mean measuring trial users' satisfaction by survey but measuring their continued use from the product's own login records, so the two variables come from different sources and different points in time. Where a single questionnaire cannot be avoided, the researcher should at least place predictor and outcome items in different sections, vary response formats and report the limitation plainly, so that the manager knows how much confidence the result deserves. Our classmate's point was that acknowledging a weakness is not enough; the design should reduce it wherever the budget allows.

How the Recommendations Changed Our Position

Biblical Integration

Proverbs 18:17 observes that the first person to present a case seems right until another comes forward and examines him. Our group's process followed that pattern: each Part 1 thread seemed complete until classmates questioned it, and the paper is stronger for the questioning. The verse is about hearing both sides before judging a dispute, but the principle carries over to a research team. Members who hold their first answers loosely, and who treat a classmate's challenge as help rather than criticism, produce better work and practice the humility Scripture commends. Research design applies the same humility to evidence itself, by building in comparison groups, second sources and later tests so that a first impression is examined before an organization acts on it.

Conclusion

Our consolidated position is that a manager's decision should choose the design: exploratory work when causes are unknown, descriptive surveys when frequency matters and experiments when the question is whether an action causes a result. The Part 2 recommendations made that position usable by showing how designs combine, what to do when a field experiment is impractical and how to strengthen surveys that must stand alone. For a manager, the practical result is a short set of questions to ask before approving any study: what decision will this inform, what must the study establish to inform it, and does the proposed design establish that at a cost the decision justifies?

References

Aguinis, H., & Bradley, K. J. (2014). Best practice recommendations for designing and implementing experimental vignette methodology studies. Organizational Research Methods, 17(4), 351-371. https://doi.org/10.1177/1094428114547952

Bloom, N., Liang, J., Roberts, J., & Ying, Z. J. (2015). Does working from home work? Evidence from a Chinese experiment. The Quarterly Journal of Economics, 130(1), 165-218. https://doi.org/10.1093/qje/qju032

Eden, D. (2017). Field experiments in organizations. Annual Review of Organizational Psychology and Organizational Behavior, 4, 91-122. https://doi.org/10.1146/annurev-orgpsych-041015-062400

Edmondson, A. C., & McManus, S. E. (2007). Methodological fit in management field research. Academy of Management Review, 32(4), 1246-1264. https://doi.org/10.5465/amr.2007.26586086

Gioia, D. A., Corley, K. G., & Hamilton, A. L. (2013). Seeking qualitative rigor in inductive research: Notes on the Gioia methodology. Organizational Research Methods, 16(1), 15-31. https://doi.org/10.1177/1094428112452151

Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33(7), 14-26. https://doi.org/10.3102/0013189X033007014

Podsakoff, P. M., MacKenzie, S. B., Lee, J.-Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879-903. https://doi.org/10.1037/0021-9010.88.5.879

Schindler, P. S. (2019). Business research methods (13th ed.). McGraw-Hill Education.