Educational Assessment, Evaluation and Accountability Образование

Reliable yet different: Human-AI alignment and severity in assessing writing

Atasoy, Arzu, Çetin, Bayram, İlhan, Mustafa, Moslemi Nezhad Arani, Saieed
Springer Nature · 2026-09-29 · DOI 10.1007/s11092-026-09512-0

Аннотация

This study investigates the comparability of human and artificial intelligence (AI) scoring in evaluating middle school students’ writing performance in Türkiye. Seventy-three student compositions on six familiar topics were assessed using a revised analytic rubric with three dimensions: Planning, Spelling-Punctuation, and Language-Expression. Four human raters and four AI raters (ChatGPT 3.5, Copilot Free, Gemini 3 Flash, and DeepSeek-V3.2) provided scores, which were analyzed through many-facet Rasch model (MFRM). Three models were tested: human-only, AI-only, and combined human-AI scoring. Results showed that both human and AI raters employed the rubric effectively, though AI raters demonstrated clearer category distinctions and stronger differentiation of writing dimensions. While both groups distinguished examinees with varying ability levels, AI scorings yielded higher separation and reliability indices. A statistically significant difference emerged in severity: human raters were more lenient overall than AI raters. Despite this difference, intraclass correlation analyses indicated a very strong relationship ( r = 0.85) between human- and AI-based ability estimates, suggesting substantial alignment in ranking students. Findings support the potential of AI-based scoring as a complement or even substitute for human evaluation, while also highlighting the need for attention to rater-type effects. This research contributes context-specific evidence from Turkish writing assessment to ongoing debates about the responsible use of AI in educational measurement.

Читать у издателя →