•  
  •  
 

PASAA

Publication Date

2023-01-01

Abstract

This study aimed to revise an academic English writing rubric for a university graduate admission test, following the Triangulation Scale Revision Process (Banerjee et al., 2015) and the Mixed-methods Conceptual Design Model (Janssen et al., 2015). The current scale revision was informed by a corpus analysis of writing responses, multi-facet RASCH model (MFRM) analysis, and a rater discussion. The samples of the study were 134 scored writing responses. Each writing response was rated by two raters. Seven raters were recruited in the rating process, and four of them participated in the subsequent raters’ discussion process. Corpus analysis was employed to investigate the lexical profiles—i.e., lexical diversity, lexical density, and response length—and the grammatical profile, using the online Web Vocabulary Profilers Program and Grammarly, respectively. The MFRM analysis was used for identifying the in-depth correlations among three facets in rating: test takers’ abilities, raters’ severity, and scales. Finally, a three-hour rater discussion was conducted to identify significant features in rating and to examine the extent to which the rubric could be revised. The corpus analyses revealed that the grammatical and lexical profiles were not significantly correlated (p<0.05). The language use trait was then separated into grammar and vocabulary traits. The results of the MFRM showed that the original rubric could be revised in terms of score weighting, use of decimal scores, and raters’ severity. Through discussion, consensus was research to weight each trait equally and to only allow integer scores. The rubric was revised following the results of the corpus analyses by separating language use traits into grammar and vocabulary, and the descriptors were revised according to the significant features retrieved from the raters’ discussion. The results of the MFRM also suggested the need for rater training.

DOI

10.58837/CHULA.PASAA.65.1.9

First Page

234

Last Page

262

Download

Share

COinS
 
 

To view the content in your browser, please download Adobe Reader or, alternately,
you may Download the file to your hard drive.

NOTE: The latest versions of Adobe Reader do not support viewing PDF files within Firefox on Mac OS and if you are using a modern (Intel) Mac, there is no official plugin for viewing PDF files within the browser window.