BUSI 614 Week 4 Qualitative Assignment 3 Example

From Panel to Clean File: Fieldwork and Data Preparation for an Online Pet Owner Survey

Choosing the Source of Respondents

Northfield Pet Foods needs about 1,200 completed questionnaires from dog owners aged 25 to 44 across the United States, with enough recent switchers to fresh food to compare them with other owners. A mail or telephone survey would be slow and costly for a group that is hard to reach by phone. An online panel is the practical choice, but panels differ in quality.

Peer et al. (2017) compared several crowdsourcing platforms used for behavioral research and found that Amazon Mechanical Turk and Prolific produced higher-quality data than CrowdFlower, with Prolific participants less familiar with common research tasks and less likely to behave dishonestly than Mechanical Turk workers. Hulland and Miller (2018), writing as journal editors in marketing, cautioned that such platforms raise concerns about whether participants represent the populations studied, about experienced respondents who take many surveys and about attention, and they urged researchers to justify their use and report the steps taken to protect data quality.

Because Northfield needs a sample that resembles American dog owners rather than the population of crowd workers, I recommend a consumer research panel with demographic profiling and quotas, rather than a crowdsourcing platform. The panel provider would be asked to document recruitment methods, the share of members who are new and how often members are invited, and to remove duplicate identities. Paying more for a well-managed panel is cheaper than making a product decision from bad data.

Fieldwork Plan and Controls

The field period would last two weeks, with invitations released in waves so that the sample is not dominated by the most eager panelists. Quotas would be set for age, region and household income based on the profile of dog-owning households in that age range, and for recent switchers, defined as owners who changed their main food in the past 24 months, to guarantee at least 300 for analysis. Incidence, the share of invited panelists who qualify, is likely to be low, since only some adults own dogs and fewer have switched, so the budget must cover many screened-out invitations.

Controls during fieldwork would include monitoring completion rates and median completion time daily, checking the distribution of quota groups, blocking repeat entries from the same device and pausing a quota group once it fills. The soft launch from the pretest would set the expected median time, about 12 minutes.

The panel provider's own controls would be checked rather than assumed. Northfield's analyst would review a daily file of partial and complete responses, compare the open-ended answers for signs of copied or automated text and confirm that the reward offered is modest enough not to attract respondents who take surveys mainly for payment. Any change to the questionnaire after launch, even a typo correction, would be dated in the field log so that responses before and after it can be compared.

Screening Careless and Fraudulent Responses

Careless responding is common enough to matter. Meade and Craig (2012), studying a large sample of undergraduate survey respondents, estimated that roughly 10 to 12% responded carelessly, and they found that instructed response items, which tell respondents which answer to select, and statistical indices, such as Mahalanobis distance and checks of consistency between similar items, were useful for detecting it. They recommended building detection into the survey design rather than relying only on cleaning afterward.

I expect these rules to remove 8 to 12% of completes, and the panel would be asked to replace them so that quotas are still met. Rules that combine two signals before removing a case, such as straightlining with speed, reduce the risk of discarding respondents who simply hold consistent views. The removed cases would be kept in a separate file and compared with the retained sample on age, region and switching status. If screening removed one group disproportionately, younger men answering on phones for instance, the rules would be reviewed, since overly strict screening can trade one bias for another.

Estimating Nonresponse Bias

Even a clean sample may misstate the population when the people who answer hold different views from the people who ignore the invitation. Armstrong and Overton (1977) proposed extrapolation methods for estimating nonresponse bias in mail surveys, including comparing respondents who answer early with those who answer late, on the reasoning that late respondents, who needed more prompting, resemble nonrespondents. Northfield would compare the first and last waves on key measures, such as attitudes toward fresh food and purchase intentions. If late respondents differ in a consistent direction, the estimates would be adjusted or reported with that caution. The sample's demographic profile would also be compared with external figures on dog-owning households. Neither check is conclusive, since nonrespondents may differ in ways no survey measure captures, but together they show whether the most likely sources of bias are present and how large they might be.

Editing, Coding and Cleaning

Data preparation would follow the textbook's sequence. Editing would review incomplete questionnaires and illogical answers. Coding would assign numeric codes to every response, including codes for "don't know" and for skipped questions, recorded in a codebook that defines each variable, its values and its source question. Open-ended answers about reasons for switching would be coded into categories drawn from the interview themes, with two coders and a check of their agreement, with disagreements resolved by discussion and the final code list added to the codebook. Cleaning would run range checks, so that no seven-point item holds an eight, and consistency checks, such as respondents who report never buying fresh food but rate a fresh brand they claim to use.

Missing Values and Weighting

Item nonresponse is likely to be low because most questions require an answer, but income and spending questions are often skipped. Cases missing one item in a multi-item scale would receive the mean of their remaining items on that scale; cases missing outcome variables would be excluded from analyses using those variables, a pairwise approach that preserves sample size, and the analysis would report how many cases each test uses. If the final sample departs from the population profile on age or region despite quotas, post-stratification weights would bring it into line, with the weighting scheme documented so that results can be reproduced. Analyses would be run on both the weighted and unweighted data at first; if the key results differ substantially, the report would show both and explain why. Each of these steps would be logged, giving Northfield a clean dataset and a record of how it was produced.

References

Armstrong, J. S., & Overton, T. S. (1977). Estimating nonresponse bias in mail surveys. Journal of Marketing Research, 14(3), 396-402. https://doi.org/10.1177/002224377701400320

Hulland, J., & Miller, J. (2018). "Keep on Turkin'"? Journal of the Academy of Marketing Science, 46(5), 789-794. https://doi.org/10.1007/s11747-018-0587-4

Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437-455. https://doi.org/10.1037/a0028085

Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017). Beyond the Turk: Alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology, 70, 153-163. https://doi.org/10.1016/j.jesp.2017.01.006