Balaha Magdy H, El-Ibiary Mona T, El-Dorf Ayman A, El-Shewaikh Shereef L, Balaha Hossam M
Department of Obstetrics and Gynecology, College of Medicine, King Faisal University, Kingdom of Saudi Arabia.
Department of Obstetrics and Gynecology, Faculty of Medicine, Tanta University, Tanta, Egypt.
Avicenna J Med. 2022 Aug 31;12(3):138-147. doi: 10.1055/s-0042-1755332. eCollection 2022 Jul.
The item-writing flaws (IWFs) in multiple-choice questions (MCQs) can affect test validity. The purpose of this study was to explore the IWFs in the published resources, estimate their frequency and pattern, rank, and compare the current study resources, and propose a possible impact for teachers and test writers. This cross-sectional study was conducted from September 2017 to December 2020. MCQs from the published MCQ books in Obstetrics and Gynecology was the target resources. They were stratified into four clusters (study-book related, review books, self-assessment books, and online-shared test banks). The sample size was estimated and 2,300 out of 11,195 eligible MCQs were randomly selected. The MCQs (items) were judged on a 20-element compiled checklist that is organized under three sections as follows: (1) structural flaws (seven elements), (2) test-wiseness flaws (five elements), and (3) irrelevant difficulty flaws (eight elements). Rating was done dichotomously, 0 = violating and 1 = not violating. Item flaws were recorded and analyzed using the Excel spreadsheets and IBM SPSS. Twenty three percent of the items ( = 537) were free from any violations, whereas 30% ( = 690) contained one violation, and 47% ( = 1073) contained more than one violation. The most commonly reported IWFs were "Options are Not in Order (61%)." The best questions with the least flaws (75th percentiles) were obtained from the self-assessment books followed by study-related MCQ books. The average scores of good-quality items in the cluster of self-assessment books were significantly higher than other book clusters. There were variable presentations and percentages of item violations. Lower quality questions were observed in review-related MCQ books and the online-shared test banks. Using questions from these resources needs a caution or avoidance strategy. Relative higher quality questions were reported for the self-assessment followed by the study-related MCQ books. An adoption strategy may be applied with mitigation if needed.
多项选择题（MCQs）中的试题编写缺陷（IWFs）会影响测试效度。本研究旨在探究已发表资源中的试题编写缺陷，估计其出现频率和模式、进行排序，并比较当前的研究资源，同时为教师和试题编写者提出可能的影响。 这项横断面研究于2017年9月至2020年12月进行。妇产科已出版的MCQ书籍中的题目为目标资源。这些题目被分为四类（与学习书籍相关、复习书籍、自我评估书籍和在线共享题库）。估计了样本量，并从11195道符合条件的MCQ中随机选取了2300道。这些MCQ（题目）根据一份由20个要素组成的编制清单进行评判，该清单分为三个部分：（1）结构缺陷（7个要素），（2）应试技巧缺陷（5个要素），以及（3）无关难度缺陷（8个要素）。评分采用二分法，0 = 违反，1 = 未违反。使用Excel电子表格和IBM SPSS记录并分析题目缺陷。 23%的题目（ = 537）没有任何违反情况，而30%（ = 690）包含一处违反，47%（ = 1073）包含多处违反。最常报告的试题编写缺陷是“选项未按顺序排列（61%）”。缺陷最少的最佳题目（第75百分位数）来自自我评估书籍，其次是与学习相关的MCQ书籍。自我评估书籍类别中高质量题目的平均得分显著高于其他书籍类别。 题目违规的呈现方式和百分比各不相同。在与复习相关的MCQ书籍和在线共享题库中观察到质量较低的题目。使用这些资源中的题目需要谨慎或采取回避策略。自我评估之后报告的题目质量相对较高，其次是与学习相关的MCQ书籍。如有需要，可以采取采用策略并进行缓解。
