Burak Senel, a graduate student in applied linguistics and technology in the Department of English, has been selected as the winner of the 2024 MwALT Best Student Paper Award for his paper, “Lexical Bundles as Predictors in AI-Powered Scoring of Non-Native Academic Writing.”
The Midwest Association of Language Testers is a community of graduate students, faculty, and professionals that supports research and practice in language testing and assessment.
Abstract
This study investigates the incorporation of lexical bundle (LB) frequency features into automated essay scoring systems like the e-rater used for TOEFL independent writing tasks. Analyzing the TOEFL11 corpus, consisting of essays written by test takers from diverse L1 backgrounds and scored by ETS-trained raters, the study compares a baseline scoring model with an extended model that includes LB frequencies, distinguishing between prompt and non-prompt types across various lengths. Results reveal a significant relationship between LB usage and proficiency levels, particularly with non-prompt bundles. Notably, the extended model shows enhanced performance with a 5.6% increase in Cohen’s Kappa score. The study highlights the potential of integrating LB analysis into automated scoring (AS) for a more refined assessment of language proficiency and for better construct representation. This research contributes to the ongoing enhancement of AS, aiming for more accurate and linguistically informed assessments in high-stakes language testing.