PASAA
Publication Date
2023-10-01
Abstract
Differential rater severity (DRS), one prevalent case of differential rater functioning (aka rater bias or rater interaction) effects, manifests itself when a rater assigns unusually severe or lenient ratings, threatening the validity and fairness of ratermediated assessment. Building on a many-facets Rasch measurement (MFRM) approach, this study aimed to detect whether teachers exercised DRS towards rating criteria and student subgroups (classroom, proficiency, and gender) in a high-stakes EFL classroom writing examination. Data were collected from three teachers who applied a four-point fivecriteria analytic rubric to rate opinion essays written by 42 English-major undergraduates during the examination. Main findings revealed that the teachers were not uniform in their severity levels, with the most experienced teacher exhibiting the highest severity and the least experienced teacher exercising the lowest severity. Whilst the most experienced and most severe teacher exposed slight DRS towards student genders, the less experienced and less severe teachers exerted substantial DRS in reverse pattern towards rating criteria and student classrooms. Surprisingly, the less experienced teachers scored their own classroom students less severely but marked each other’s classroom students more severely than expected. The current findings raise the attention and awareness of teachers, educators, and policymakers concerning the impact of rater effects on the validity and fairness of rater-mediated assessment in the classroom context.
DOI
10.58837/CHULA.PASAA.66.1.1
First Page
5
Last Page
36
Recommended Citation
Khamboonruang, Apichat
(2023)
"Detecting Differential Rater Severity in a High-Stakes EFL Classroom Writing Assessment: A Many-Facets Rasch Measurement Approach,"
PASAA: Vol. 66:
Iss.
1, Article 1.
DOI: 10.58837/CHULA.PASAA.66.1.1
Available at:
https://digital.car.chula.ac.th/pasaa/vol66/iss1/1