Validation practices in bibliometric topic modeling: an empirical assessment of marketing research

Echchakoui, Saïd
Journal of Marketing Analytics · Springer Nature · 2026-10-31 · DOI: 10.1057/s41270-026-00526-7

Abstract

Bibliometric topic modeling has become a standard way of mapping the intellectual structure of marketing research. Whether the themes it produces reflect genuine intellectual organization or merely convenient algorithmic artifacts has rarely been examined. This paper addresses that question through two connected studies: a systematic audit of validation practice in 244 bibliometric topic-modeling studies published in fifteen ABS-ranked marketing and business journals between 2013 and 2025, and an AI-assisted auditing methodology in which two independent large language model agents code validation practices against a six-dimensional, 44-metric taxonomy. The audit's central finding is that the validation deficit is dissociated, not uniform. Most studies report no canonical validation of their topic or cluster solutions, but the gap sits almost entirely in the co-word and bibliometric-mapping tradition, which borrows the vocabulary of "topics" without the validation apparatus that probabilistic topic modeling provides; stability testing is rare in every method family. The methodological study establishes that inter-agent coding reliability depends far more on the design of the coding instrument than on the capability of the model, and that human oversight is needed only at definitional boundary cases. The paper makes three contributions: the first discipline-specific, method-conditional baseline of the validation gap in marketing bibliometrics; a reusable validation framework for bibliometric research, together with a hybrid human-AI auditing protocol calibrated against expert judgment; and a minimum-viable-validation checklist that translates the findings into method-specific standards for authors, reviewers, and editors. The deficit this paper documents is largely invisible in current practice; the framework and the checklist make it measurable and correctable.

Read at publisher →

Similar articles