Performance and concordance of contemporary large language models on Royal Australasian College of Physicians examination questions: an exploratory analysis

Rohan Joshua Krishnaswamy, Joseph Julian Suttie
Internal Medicine Journal · Wiley · 2026-10-09 · DOI: 10.1111/imj.70664

Abstract

Abstract We assessed six publicly available large language models (LLM) on Royal Australasian College of Physicians (RACP) adult medicine written examination questions. In a 30‐question set with answers, models achieved 93% correct responses overall, and accuracy was higher for text‐based than visual‐media questions. In a 170‐question exploratory set without an answer key, complete inter‐model agreement occurred in 86% of questions but was lower for visual‐media than text‐based questions and for scenario‐based than knowledge‐based questions. Contemporary LLM performed strongly on RACP‐style written questions, but interpretation of visual media and complex clinical vignettes remains a limitation.

Read at publisher →

Similar articles