BUSI 600 Week 6 Group DB Forum 3: Part 2 Example

Group Discussion Board Forum 3: Part 2

BUSI 600 Business Research Methods, Group Discussion Board Forum 3: Part 2 (replies)

Thread by [Student Name]

The two classmates and their threads are composites; the research and Scripture cited are real, and the requirements follow Liberty's public course guide for BUSI 600.

Reply to Derek

Derek, your thread did something many survey reports skip: it reported the engagement scale's internal consistency, an alpha of .89, and explained what the number means. I want to question one conclusion, though, the statement that an alpha above .70 shows the survey is valid. The research on alpha suggests it shows something narrower.

Cortina (1993) showed that coefficient alpha is driven both by how strongly the items correlate on average and by how many items the scale contains. Adding items pushes alpha upward even when those items relate to one another only modestly, and an impressive value tells us nothing about whether the scale taps one concept or several. A scale combining two related but distinct sets of items, for example questions about pride in the company and questions about workload, can still produce an impressive alpha. Your company's twelve-item scale may be consistent largely because it is long, and it may be measuring several things at once.

Even a consistent, single-factor scale may not measure what its name claims. Cronbach and Meehl (1955) introduced construct validity as the question of whether a test measures a theoretical attribute, and they argued that it is established by showing that scores behave as theory predicts within a network of related concepts and outcomes. For engagement, that means checking whether scores relate to the outcomes engagement should affect, such as discretionary effort, absenteeism and voluntary turnover, and whether they are distinct from related concepts like job satisfaction. If engagement scores predict turnover no better than a single satisfaction item, the company has an expensive satisfaction survey with a new name.

Alpha answers whether the items agree with one another; validity answers whether they are right. Proverbs 14:15 contrasts the simple person, who believes everything, with the prudent person, who gives thought to each step. A manager who accepts a scale because one statistic is high is believing everything. Prudence here means asking for the second and third kinds of evidence before tying bonuses or promotions to the scores.

The stakes are practical. Many companies now rank managers by their teams' engagement scores, and a manager whose department scores low may lose a bonus or a promotion. If the scale is partly measuring workload, which managers often cannot control, the company could be penalizing managers of understaffed units for a problem created elsewhere. Validity is therefore a fairness question as well as a technical one. In our group's Part 3 paper, two changes would address this. First, the paper should describe alpha accurately, as evidence of consistency rather than validity. Second, we should add a validation step to our example: correlate the engagement scores with the following year's voluntary turnover by department, and run a factor analysis to see whether the twelve items form one dimension or several. Both can be done with data the company already has, and together they would tell managers whether the survey deserves the weight they give it.

Reply to Alicia

Alicia, I agree that an 18% response rate on your company's customer survey should make anyone cautious, and your thread was right to ask whether the results could be trusted. Where I would differ is the conclusion that the survey should be discarded and repeated. The rate alone cannot tell us whether the results are biased, and a repeat survey could produce the same rate without telling us any more.

Baruch and Holtom (2008) analyzed response rates in 1,607 studies published in 17 refereed management journals and found an average of about 53% for surveys of individuals and about 36% for surveys of organizations, with standard deviations of roughly 20 points around each average, so many published studies fell far below them. By those benchmarks your survey is low, but customer surveys sent by email routinely fall well below the averages for academic studies, so the low rate is not by itself unusual.

The more useful question is whether the 18% who replied answer differently, on the topics the survey covers, from the 82% who stayed silent. A low rate creates room for bias, but bias only appears if the decision to respond is connected to the opinions being measured. If customers who had a bad experience were more eager to respond, for example, the results would look worse than reality; if satisfied customers were more willing to help, they would look better. Neither pattern can be assumed in advance, which is why the evidence has to be checked. Rogelberg and Stanton (2007) describe practical ways to assess this. A researcher can compare respondents with the full sample on characteristics already known, such as purchase history, region and account age; compare early and late respondents, since late respondents tend to resemble nonrespondents; or follow up with a small random sample of nonrespondents by phone. If respondents look like the whole customer list on known characteristics and early and late respondents give similar answers, the results deserve more confidence than the rate suggests.

Jesus said that a tree is recognized by its fruit (Matthew 7:16-20). A survey should be judged in the same way, by examining what it produced and whether its respondents resemble the people it claims to describe, rather than by a single number on the cover of the report. That approach is also fairer to the customers who did take the time to answer.

My suggestion for the consolidated paper is to replace the statement that response rates below 30% invalidate a survey with a short procedure: report the rate, compare respondents with the full customer list on known characteristics, and compare early and late respondents on the key questions. Only if those checks reveal differences should the company spend money on a second survey, and even then a targeted follow-up with nonrespondents is likely to be cheaper and more informative.

References

Baruch, Y., & Holtom, B. C. (2008). Survey response rate levels and trends in organizational research. Human Relations, 61(8), 1139-1160. https://doi.org/10.1177/0018726708094863

Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78(1), 98-104. https://doi.org/10.1037/0021-9010.78.1.98

Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281-302. https://doi.org/10.1037/h0040957

Rogelberg, S. G., & Stanton, J. M. (2007). Introduction: Understanding and dealing with organizational survey nonresponse. Organizational Research Methods, 10(2), 195-209. https://doi.org/10.1177/1094428106294693