Asli Nurul Fariena, Mohd Matore Mohd Effendi Ewan, Md Yunus Melor
Faculty of Education, The National University of Malaysia, 43600, Bangi, Selangor, Malaysia.
Heliyon. 2024 Nov 4;10(22):e40053. doi: 10.1016/j.heliyon.2024.e40053. eCollection 2024 Nov 30.
In assessing performance-based language assessment, the use of a valid and reliable scoring rubric is crucial to minimize measurement errors that become threats in the rating process. The validation process of rubrics which previously was based on qualitative data is not satisfying since empirical evidence is not present. Thus, drawing on the Assessment Use Argument (AUA) Validation Framework, this study aims to search for evidence to prove a claim which is Primary Trait Writing (PTW) rubrics for students self assessment activities are relevant to the construct being measured. Based on that, two warrants and one rebuttal were derived to prove the claim. The participants consisted of 149 secondary school students in a state in Malaysia and three facets identified in the study were the examinee (149 students), the rater (149 students and 3 teachers), and the trait (Content, Format, Cohesive Device, and Sentence Fluency). Many Facet Rasch Model was employed to search for evidence in approving the warrants and rejecting the rebuttal. Based on the statistical results, evidence has shown that PTW rubrics successfully discriminated between students' writing ability, and fulfilled the six basic conditions of rating scale effectiveness to a certain extent where the Cohesive Device trait became the primary concern. In addition, the fit statistics for all traits demonstrated internal consistency and the high-reliability index portrayed the criteria were well differentiated in terms of difficulty level. Thus, all evidence had shown that PTW rubrics obtained construct validity where the warrants were supported, and the rebuttal was rejected which led to the acceptance of the claim. The implication of this study highlights the importance of validating assessment rubrics to ensure their internal validity and the use of MRFM in providing comprehensive analyses of the strengths and weaknesses of the developed rubrics. The use of a primary trait type scoring rubric as a tool in students' self-assessment activities must be highlighted as more current studies are focusing on holistic and analytic scoring. Therefore, it is suggested for future research to expand the use of the Primary Trait rubrics in other type of essays and to be used in peer-assessment activities. Therefore, it is suggested for future research to expand the use of the Primary Trait rubrics in other types of essay and to be used in peer-assessment activities.
在评估基于表现的语言评估时，使用有效且可靠的评分标准对于最大限度地减少在评分过程中成为威胁的测量误差至关重要。以前基于定性数据的评分标准验证过程并不令人满意，因为缺乏实证证据。因此，本研究借鉴评估使用论证（AUA）验证框架，旨在寻找证据来证明一项主张，即用于学生自我评估活动的主要特征写作（PTW）评分标准与所测量的结构相关。基于此，得出了两个论据和一个反例来证明该主张。参与者包括马来西亚一个州的149名中学生，研究中确定的三个方面是考生（149名学生）、评分者（149名学生和3名教师）以及特征（内容、格式、衔接手段和句子流畅性）。采用多面Rasch模型来寻找证据以支持论据并反驳反例。基于统计结果，证据表明PTW评分标准成功地区分了学生的写作能力，并在一定程度上满足了评分量表有效性的六个基本条件，其中衔接手段特征成为主要关注点。此外，所有特征的拟合统计显示出内部一致性，高可靠性指数表明标准在难度水平方面有很好的区分度。因此，所有证据都表明PTW评分标准具有结构效度，论据得到支持，反例被驳回，从而该主张被接受。本研究的意义在于强调验证评估评分标准以确保其内部效度的重要性，以及使用MRFM对所开发评分标准的优缺点进行全面分析。必须强调将主要特征类型评分标准用作学生自我评估活动工具的重要性，因为目前更多的研究集中在整体评分和分析评分上。因此，建议未来的研究扩大主要特征评分标准在其他类型文章中的应用，并用于同伴评估活动。因此，建议未来的研究扩大主要特征评分标准在其他类型文章中的应用，并用于同伴评估活动。
