Software Quality Journal Software

AI-as-a-Judge for software quality assessment: Consistency, attitude effects, and complexity sensitivity in multi-agent code evaluation

Chou, Sherlinford
Springer Nature · 2026-08-25 · DOI 10.1007/s11219-026-09785-0

Abstract

The rapid adoption of large language models (LLMs) for code generation has led to increased interest in using LLMs as automated judges for “software quality assessment”. However, the reliability of such LLM-as-a-Judge systems remains unclear due to potential inconsistencies among different judge models and sensitivity to prompt design. In this paper, a systematic empirical study is conducted on the consistency, attitude effects, and complexity sensitivity of LLM judges in code evaluation. Using four state-of-the-art LLMs (Gemini, Deepseek, Kimi, and Doubao) under three evaluation attitudes (strict, objective, lenient), 3,000 eight-dimensional scoring records were collected on 250 code samples. Principal component analysis (PCA) was used to derive objective dimension weights. Statistical analyses including intraclass correlation (ICC), Spearman correlation, paired t-tests with Cohen’s d, and ANOVA reveal three main findings: (1) inter-judge consistency is moderate ( $$ICC = 0.457-0.565$$ ), indicating that different LLM judges cannot be used interchangeably; (2) role-prompt attitudes have very large effects on scores (Cohen’s $$d > 0.9$$ for most comparisons), demonstrating that simple prompt engineering can effectively control judge strictness; (3) code complexity significantly affects scores in 11 out of 12 judge–attitude combinations, though effect sizes are small to medium ( $$\eta ^2 = 0.03-0.15$$ ). A notable exception is identified where Doubao under strict attitude becomes insensitive to complexity, likely due to a floor effect. These findings provide practical guidance for selecting LLM judges, designing role prompts, and employing multi-judge voting strategies in “automated software quality assurance”.

Read at publisher →