Educational Assessment, Evaluation and Accountability Образование
Beyond accuracy: differential scoring patterns in generative AI writing assessment across cultural and linguistic diversity
Аннотация
Generative artificial intelligence (GenAI) systems are increasingly used in writing assessment; however, evidence regarding their validity and fairness across culturally and linguistically diverse contexts remains limited. This study examined whether GenAI-based writing assessments exhibit scoring differences associated with cultural content familiarity and writers’ first language (L1) backgrounds. A 2 × 2 mixed-design experiment was conducted with 54 EFL learners (Turkish and Arabic L1 speakers at lower-intermediate proficiency). Participants produced essays on culturally proximal and Western-familiar topics, evaluated by three GenAI models (GPT-4, Claude 3.5 Sonnet, and Gemini 1.5 Pro) and three experienced human raters using an analytic rubric. GenAI models assigned substantially lower scores to culturally proximal essays than to Western-familiar essays, whereas context-familiar human raters showed no comparable difference. GenAI scores also differed by L1 background, with moderately lower scores for Arabic L1 writers than for proficiency-matched Turkish L1 writers. The Topic × L1 interaction did not reach statistical significance in either the GPT-4/Claude composite or the Gemini subsample, although the lowest mean GenAI scores were observed descriptively for Arabic L1 writers on culturally proximal topics. Because the topic manipulation involved differences in experiential versus mediated knowledge access, and human ratings were provided by context-familiar raters, the findings should not be interpreted as definitive evidence of systematic GenAI scoring bias. Rather, they identify conditions under which GenAI scoring diverged from local expert judgment and raise validity questions requiring further investigation in multilingual assessment contexts.