Journal of Intelligent Information Systems Software
Robust evaluation of lexical, transformer, and retrieval-grounded language models for health condition identification
Abstract
Health condition identification from short symptom narratives remains a demanding information-systems task because useful solutions must combine representation quality, transfer robustness, retrieval support, and operational efficiency. This paper compares sparse lexical models, BERT-family encoders, and retrieval-grounded large language models within one harmonised evaluation framework built from multiple symptom-text sources. The internal benchmark favours biomedical transformers, with BlueBERT achieving the strongest in-distribution performance, but the ranking changes once the models are tested on external datasets, cross-source transfer, and leave-one-source-out (LOSO) validation. Retrieval grounding improves some instruction-tuned LLMs substantially, especially Meta-Llama-3-8B-Instruct and Llama3-Med42-8B, while helping other checkpoints only selectively. The results show that model choice should be guided by source shift, evidence format, shortlist quality, and inference cost rather than by one internal score alone.