5 vs 4: A Quantitative Investigation into the Quality Metrics of Different Multiple-Choice Test Formats

Abstract

This study employed quantitative methods to address two primary objectives: 1) to compare the quality of 5-choice and 4-choice multiple-choice tests, and 2) to evaluate the discriminant power of these formats using test response theory with kernel smoothing. Data were collected from 1,966 students at Sukhothai Thammathirat Open University who took a 120-question multiple-choice exam during the second semester of 2019. Four test configurations were analyzed: the Initial Case utilized the original 5-choice format; Case 1 randomly omitted one option from the 5-choice test, excluding the correct answer; Case 2 randomly omitted one option, including the correct answer; and Case 3 adapted the options based on the test-taker’s proficiency level. The study employed Cronbach’s Alpha (denoted as raw_alpha) as a reliability metric, discovering varying levels of reliability across the four cases. The highest reliability was observed in Case 3 with a raw_alpha value of 0.87. There were no differences in the difficulty values or discriminatory power across all cases. The mean scores indicated that students generally performed better on the 4-choice tests in Cases 1-3 than on the original 5-choice format, referred to as the Initial Case. These findings have significant implications for test design, suggesting that 4-choice tests can achieve comparable reliability and discriminatory power to traditional 5-choice tests.