Volume 4 • Issue 1 • PP: 16–24 • 2025
Analytic Trait Scoring of English Language Learner Essays Using Fine-Tuned DeBERTa-v3 on the ELLIPSE Corpus
Open Access & Copyright
© 2025 The Author(s). Published by ASPG. This article is licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0).
View PDF Abstract
Automated Essay Scoring (AES) technologies have been extensively researched for holistic, topic-specific scoring, but their use to predict multiple analytic writing quality traits of English Language Learner (ELL) student essays has received less attention. This research contributes to this knowledge gap by systematically investigating multi-trait AES on the ELLIPSE corpus (Learning Agency Lab, 2022), a publicly accessible dataset of 6,482 argumentative essays written by grades 8–12 ELLs and rated by human raters on six analytic traits: cohesion, syntax, vocabulary, phraseology, grammar, and conventions. We experiment with five models: Ridge regression, Support Vector Regression (SVR) with a radial basis function (RBF) kernel, Random Forest, fine-tuned BERT-base-uncased, and fine-tuned DeBERTa-v3-base. The mean Quadratic Weighted Kappa (QWK) across six traits is highest for DeBERTa-v3 (0.726), a 26.5-point improvement over the Ridge baseline (0.461) and a 6-point improvement over BERT (0.666). Phraseology is the most difficult trait to score automatically (DeBERTa QWK = 0.701) and cohesion the easiest (DeBERTa QWK = 0.742). Analysis of inter-trait correlations reveals high co-variation between vocabulary and phraseology (r = 0.79), which may reflect common linguistic skills that can be leveraged by multi-task learning. This research sets a replicable baseline for multi-trait AES on the ELLIPSE corpus and suggests that phraseology scoring is the most urgent area for future architectural innovation.
Keywords
References
[1] D. Ramesh and S. K. Sanampudi, “An automated essay scoring systems: A systematic literature review,” Artifi-cial Intelligence Review, vol. 55, no. 3, pp. 2495–2527, 2022.
[2] Y. Wang, C. Wang, R. Li, and H. Lin, “On the use of BERT for automated essay scoring: Joint learning of multi-scale essay representation,” in Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan-guage Technologies. Seattle, WA, USA: Association for Computational Linguistics, 2022, pp. 3416–3425.
[3] R. Yang, J. Cao, Z. Wen, Y. Wu, and X. He, “Enhancing automated essay scoring performance via fine-tuning pre-trained language models with combination of re-gression and ranking,” in Findings of the Association for Computational Linguistics: EMNLP 2020. Online: Association for Computational Linguistics, 2020, pp. 1560–1569.
[4] Learning Agency Lab, “Feedback prize – english lan-guage learning (ellipse) corpus,” Kaggle competition dataset, 2022.
[5] M. Uto, “A review of deep-neural automated essay scor-ing models,” Behaviormetrika, vol. 48, no. 2, pp. 459–484, 2021.
[6] E. Mayfield and A. W. Black, “Should you fine-tune BERT for automated essay scoring?” in Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications. Seattle, WA, USA: Association for Computational Linguistics, 2020, pp. 151–162.
[7] J. Xie, K. Cai, L. Kong, J. Zhou, and W. Qu, “Auto-mated essay scoring via pairwise contrastive regression,” in Proceedings of the 29th International Conference on Computational Linguistics. Gyeongju, Republic of Korea: International Committee on Computational Linguistics, 2022, pp. 2724–2733.
[8] W. Song, K. Zhang, R. Fu, L. Liu, T. Liu, and M. Cheng, “Multi-stage pre-training for automated chinese essay scoring,” in Proceedings of the 2020 Conference on Em-pirical Methods in Natural Language Processing. On-line: Association for Computational Linguistics, 2020, pp. 6723–6733.
[9] R. Ridley, L. He, X. Dai, S. Huang, and J. Chen, “Auto-mated cross-prompt scoring of essay traits,” in Proceed-ings of the AAAI Conference on Artificial Intelligence, vol. 35, 2021, pp. 13 745–13 753.
[10] R. Kumar, S. Mathias, S. Saha, and P. Bhattacharyya, “Many hands make light work: Using essay traits to automatically score essays,” in Proceedings of the 2022 Conference of the North American Chapter of the As-sociation for Computational Linguistics: Human Lan-guage Technologies. Seattle, WA, USA: Association for Computational Linguistics, 2022, pp. 1485–1495.
[11] M. Cho, J.-X. Huang, and O.-W. Kwon, “Dual-scale BERT using multi-trait representations for holistic and trait-specific essay grading,” ETRI Journal, vol. 46, no. 1, pp. 82–95, 2024.
[12] B. Alnasyan, M. Basheri, and M. O. Alassafi, “The power of deep learning techniques for predicting stu-dent performance in virtual learning environments: A systematic literature review,” Computers and Education: Artificial Intelligence, vol. 6, p. 100231, 2024.
[13] R. S. Baker and A. Hawn, “Algorithmic bias in educa-tion,” International Journal of Artificial Intelligence in Education, vol. 32, no. 4, pp. 1052–1092, 2022.
[14] A. Gonzalez-Nucamendi, J. Noguez, L. Neri, V. Robledo-Rella, and R. M. G. Garcia-Castelan, “Predictive analytics study to determine undergraduate students at risk of dropout,” Frontiers in Education, vol. 8, p. 1244686, 2023.
[15] M. Hlosta, C. Herodotou, T. Papathoma, A. Gillespie, and P. Bergamin, “Predictive learning analytics in online education: A deeper understanding through explaining algorithmic errors,” Computers and Education: Artifi-cial Intelligence, vol. 3, p. 100108, 2022.
[16] M.-R. Borna, H. Saadat, A. T. Hojjati, and E. Akbari, “Analyzing click data with ai: Implications for student performance prediction and learning assessment,” Fron-tiers in Education, vol. 9, p. 1421479, 2024.
Cite This Article
Choose your preferred format
Publisher's Note
The statements, opinions, and data presented in this article are solely those of the author(s) and do not necessarily represent those of ASPG, the journal, or its editors. ASPG and the editors disclaim responsibility for any harm arising from the use of any ideas, methods, instructions, or products described in this article, to the fullest extent permitted by applicable law.