Education and Information Technologies Library and Information Sciences
Zero-shot prompting with GenAI for EFL cloze text generation: a mixed-methods evaluation for language assessment in China
Abstract
For many English teachers in non-English-speaking countries, locating appropriate texts for developing language tests is a key challenge. Generative artificial intelligence (GenAI) may help address this practical bottleneck, yet little attention has been paid to how AI-generated texts compare with human-written texts when evaluated against teacher-informed and context-specific quality criteria. This study, therefore, examines the use of GPT-4o with zero-shot prompting to generate cloze source texts for China’s Zhongkao , a high-stakes junior high school achievement test. An exploratory sequential mixed-methods design, involving 117 junior high English teachers, was employed across four phases: prompt design, text generation, framework development and human-AI text comparison. Qualitative and quantitative data were collected through questionnaires, text analysis, and stimulated recall interviews, and analyzed using inductive thematic coding, descriptive statistics, linear mixed-effects modelling, two one-sided tests (TOST), and McNemar-Bowker test. Results showed equivalence with human-written texts in narrativity and structure, while the evidence for educational value and relevance remained inconclusive. GenAI-generated texts received slightly lower ratings for difficulty appropriateness, which emerged as the main area of concern. Teachers were generally willing to use the GenAI-generated texts, although their qualitative responses reflected cautious optimism and a continued need for professional review. The study provides preliminary support for standards-aligned zero-shot GenAI for assessment-oriented text generation in this context, but GenAI should be used within a human-in-the-loop process rather than as an autonomous test development tool. Future research should incorporate student-level evidence and larger text samples to further validate GenAI-generated assessment materials.