BUSI 453 Week 5 Discussion Board Forum 3 Example

Discussion Board Forum 3: Sampling Design and Procedures

BUSI 453 Applied Marketing Analysis, Discussion Board Forum 3 Thread and Replies

Thread by [Student Name]

The forum prompt and the classmates are composites; the textbook, the research and the Scripture cited are real.

Thread

Many managers believe that a bigger sample is a better sample. Suppose a regional restaurant chain emails a satisfaction survey to all 40,000 members of its rewards program and receives 3,200 responses. That sounds impressive, but the number alone says little about whether the results describe the chain's customers.

Malhotra (2015) defines the target population as the collection of elements that possess the information sought, and the sampling frame as the list from which the sample is drawn. The chain's frame, rewards members with email addresses, already excludes customers who do not join the program, who may be less loyal. Then only 8 percent responded. Those who did respond chose themselves, which makes the final sample a self-selected one, closer to a convenience sample than to the probability sample its size suggests. A larger sample reduces sampling error, the random difference between the sample and the population. It does nothing to reduce nonresponse error, which arises when the people who reply are unlike the people who stay silent.

A natural fix is to push the response rate higher. But Groves and Peytcheva (2008) pooled 59 studies in which the true bias could be checked against outside records, and the response rate turned out to say little about how biased an estimate was. Surveys with high response rates sometimes had large bias, and some with low rates had little. Bias appears when the urge to answer is itself tied to the thing the survey measures. If the chain's most satisfied customers are the most likely to answer, its satisfaction scores will be too high no matter how many responses arrive.

The opposite lesson also holds: an unrepresentative sample can be useful if its makeup is known. Wang et al. (2015) forecast the 2012 U.S. presidential election from a poll of Xbox users, a sample heavily skewed toward young men. After statistically adjusting the results to match the electorate's demographic makeup, their forecasts were close to those of the leading polls. The adjustment worked because the researchers measured the characteristics on which the sample differed.

Malhotra (2015) offers several ways to reduce or adjust for nonresponse: callbacks and reminders, incentives, subsampling nonrespondents for intensive follow-up, and weighting. A probability sample drawn from all customers, not only rewards members, would also fix the frame problem, for example by handing survey invitations to every fifth customer at the register over two weeks, a systematic sample that includes customers outside the program.

For the restaurant chain, the practical steps follow: compare respondents' visit frequency and spending, which the chain already knows from its records, with those of nonrespondents; weight the results if they differ; and report satisfaction separately for frequent and occasional customers. None of these steps requires more responses; all of them require knowing more about who responded.

In the parable of the lost sheep, the shepherd leaves the ninety-nine to look for the one that is missing (English Standard Version Bible, 2001/2016, Luke 15:4). The shepherd's attention goes to the one who did not come back. Good researchers share that instinct. The customers who never answer the survey may be the ones the business is losing, and finding out who they are matters more than collecting a few hundred more responses from those already content.

Reply to Kayla

Kayla, your plan to buy responses from an online panel for your QMR study is practical, since your target population, adults who buy plant-based foods, is hard to reach through a frame of your own. Online samples have real strengths, but they need checks.

Goodman et al. (2013) compared Mechanical Turk workers with community and student samples and found that the workers produced results similar to traditional samples on many classic decision-making effects, but were less likely to pay attention to experimental materials and more likely to search for answers online. Couper (2000) reviewed web surveys and distinguished coverage error, from people without access or not on the panel, from nonresponse and measurement error, and warned that a large web sample does not by itself make inferences to the general population sound.

I would add two steps to the plan: include an attention check item and screen out speeders who finish far faster than the median time, and describe your sample's demographics in Draft 3 so readers can judge how it differs from plant-based food buyers in general. Proverbs 11:3 pictures integrity as a guide that keeps upright people on course (English Standard Version Bible, 2001/2016). Reporting honestly what kind of sample you have is part of that integrity. Which screening questions will your panel use? A question confirming that the respondent bought a plant-based product in the past month, placed among other product questions so the right answer is not obvious, would keep out people who only want the panel's payment. Reporting how many respondents failed each check would also show readers how much screening the data needed.

Reply to Marcus

Marcus, you worried that the 22 percent response rate in your employee survey makes the results useless. I would not discard them yet. Groves (2006) showed that low response rates do not necessarily produce biased estimates; the damage depends on how closely a person's chance of answering tracks the answers the survey is after. In an employee survey, the worry would be that unhappy employees skipped it, or that they were the most eager to answer, and either pattern is possible.

There is also a simple check you can run. Armstrong and Overton (1977) proposed extrapolation: compare those who responded early with those who responded late, on the theory that late respondents resemble nonrespondents more closely. If early and late respondents give similar answers on your key measures, nonresponse bias is less likely to be serious. You could also compare respondents with the whole staff on department and tenure, which human resources records show. If one department responded at twice the rate of another, weighting the results by department would keep that group from dominating the overall satisfaction score.

Gideon's army was reduced from ten thousand to three hundred by watching how the men drank at the water (English Standard Version Bible, 2001/2016, Judges 7:5-7). That was a purposeful selection with a clear criterion, not a random one, and the account shows that who is selected, and why, matters more than how many. Do you have the date each response arrived, so you could split early and late respondents? If the late group looks more negative, you could report results separately for the two groups and treat the late group's figures as a closer estimate of how the whole staff feels.

References

Armstrong, J. S., & Overton, T. S. (1977). Estimating nonresponse bias in mail surveys. Journal of Marketing Research, 14(3), 396-402. https://doi.org/10.1177/002224377701400320

Couper, M. P. (2000). Web surveys: A review of issues and approaches. Public Opinion Quarterly, 64(4), 464-494. https://doi.org/10.1086/318641

English Standard Version Bible. (2016). Crossway. (Original work published 2001)

Goodman, J. K., Cryder, C. E., & Cheema, A. (2013). Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26(3), 213-224. https://doi.org/10.1002/bdm.1753

Groves, R. M. (2006). Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly, 70(5), 646-675. https://doi.org/10.1093/poq/nfl033

Groves, R. M., & Peytcheva, E. (2008). The impact of nonresponse rates on nonresponse bias: A meta-analysis. Public Opinion Quarterly, 72(2), 167-189. https://doi.org/10.1093/poq/nfn011

Malhotra, N. K. (2015). Essentials of marketing research: A hands-on orientation. Pearson.

Wang, W., Rothschild, D., Goel, S., & Gelman, A. (2015). Forecasting elections with non-representative polls. International Journal of Forecasting, 31(3), 980-991. https://doi.org/10.1016/j.ijforecast.2014.06.001