PSYC 421 Week 4 Item Development and Analysis Worksheet Example

Item Development and Analysis Worksheet: Writing, Scoring and Evaluating Five Multiple-Choice Items on Research Methods

Part 1: Purpose and Items

The five items belong to a short quiz on research methods for an introductory psychology course. The blueprint calls for two items on recall of definitions, two on application to a described study and one on evaluating a design. Cohen and Swerdlik (2018) describe test development as a sequence of conceptualization, construction, tryout, item analysis and revision, and this worksheet follows that sequence. The items, with their keyed answers, were:

• 1. Which variable does a researcher manipulate in an experiment? A. Dependent B. Independent (key) C. Confounding D. Extraneous
• 2. A survey finds that students reporting longer nightly sleep tend to have higher GPAs. What can be concluded? A. Sleep causes higher grades B. Grades cause more sleep C. The two variables are related (key) D. Nothing can be concluded
• 3. Which threat to internal validity is most likely when participants are tested twice with the same quiz? A. Maturation B. Testing (key) C. Attrition D. Instrumentation
• 4. In a study comparing a new teaching method with a lecture, what is the control group? A. Students taught by the new method B. Students taught by lecture (key) C. Students who could choose either method D. Students not tested
• 5. Why does random assignment strengthen an experiment? A. It ensures a large sample B. It makes the sample representative C. It equalizes groups on other variables on average (key) D. It removes measurement error

Part 2: Item Statistics

The quiz was given to 40 students. The upper and lower groups, set by the rule Kelley (1939) proposed, were defined as the top and bottom 27% of total scores, 11 students each. Difficulty (p) is the proportion of all 40 students answering correctly. The discrimination index (D) is the number correct in the upper group minus the number correct in the lower group, divided by 11.

Part 3: Interpretation

For four-option items, chance success is .25, and a common benchmark for optimal difficulty is about .63, halfway between chance and 1.00. Item 2, at .55 with D = .64, is close to optimal and discriminates very well. Item 5, at .45 with D = .55, is somewhat hard but highly effective. Item 1 is easy at .85 but still discriminates adequately at .36, which is acceptable for a definition item meant to confirm basic knowledge. Item 3 is a problem: only 23% answered correctly, below chance, and it did not separate strong from weak students at all. Item 4 is the most serious problem. Its negative discrimination index means that more students in the lower group than in the upper group chose the keyed answer, so the item penalized knowledge instead of rewarding it.

Part 4: Distractor Analysis

For item 4, the response counts showed that 4 of the 11 upper-group students chose option C, "students who could choose either method," and 1 chose A, while only 1 lower-group student chose C. Stronger students appear to have read C as a reasonable comparison condition, perhaps reasoning that a self-selected group shows what happens without the new method. The item's stem asks for "the control group" without stating that students were randomly assigned, so knowledgeable students may have seen the design as unclear. For item 3, answers were spread evenly across all four options in both groups, which suggests guessing; the term "testing" as a threat may not yet have been taught clearly.

Part 5: Qualitative Review

Haladyna et al. (2002) reviewed published item-writing guidelines and identified widely endorsed rules, such as stating the central idea in the stem, keeping options similar in length and structure and avoiding options that give cues to the answer. Items 1, 2 and 5 follow these rules. Item 3's correct option repeats a word from the stem, "tested" and "testing," a grammatical cue that should have made it easier, which suggests the concept itself was unfamiliar. Item 4 violates the rule of a clear stem. No item contains language or examples that would favor one cultural or gender group, although item 2's sleep example assumes students know what a GPA is in an American context, which is reasonable for this population.

Part 6: Revisions

Item 3 revised: "Participants take the same memory quiz before and after a training program, and scores improve partly because they remember the questions. Which threat to internal validity does this illustrate?" The scenario removes the word cue and teaches the concept through an example.

Item 4 revised: "Students are randomly assigned either to a new teaching method or to a standard lecture. Which group serves as the control group?" Option C is replaced with "Students who were randomly assigned to both methods," which is clearly incorrect.

After the next administration, the same statistics should be computed to confirm that item 3's difficulty rises toward the optimal range and that item 4's discrimination becomes positive.

References

Cohen, R. J., & Swerdlik, M. E. (2018). Psychological testing and assessment: An introduction to tests and measurement (9th ed.). McGraw-Hill Education.

Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309-333. https://doi.org/10.1207/S15324818AME1503_5

Kelley, T. L. (1939). The selection of upper and lower groups for the validation of test items. Journal of Educational Psychology, 30(1), 17-24. https://doi.org/10.1037/h0057123