Medical Reasoning in Large Language Models: Development and Evaluation of the MedEvalArena Benchmark

Journal of Medical Internet Research ·

Background: Large language models (LLMs) perform strongly on medical specialty board multiple-choice questions (MCQs) but underperform in complex medical reasoning. This gap indicates a need to improve LLM medical reasoning and evaluation paradigms. Objective: This study aimed to develop and evaluate MedEvalArena, a dynamically regenerable benchmark for assessing medical reasoning in LLMs. Methods: In a symmetric round-robin format, 6 frontier LLMs from the OpenAI, Grok, Gemini, Claude, Kimi, and DeepSeek families generated challenging board-style medical MCQs, adjudicated question validity on an ensemble LLM-as-judge panel, and answered the validated examination. Candidate items were generated until each model contributed 50 validated questions according to Association of American Medical Colleges (AAMC)/American Board of Medical Specialties (ABMS) physician specialty distributions, yielding a 300-question examination. Judges adjudicated candidate questions’ logical validity and medical accuracy. We compared question-generation validity rates, assessed leave-one-out judge agreement, and tested for self-judging preference. Questions were audited for overlap with public MedQA (medical question-answering) benchmarks. We tested question-answering (QA) accuracy across LLMs using a binomial-logit generalized linear mixed model likelihood-ratio test. A generator-stratified 90-question subset was completed by 5 board-certified physicians for paired human-LLM analysis. We evaluated inference cost-accuracy tradeoffs. Multiple comparisons were adjusted by the Benjamini-Hochberg (BH) procedure. Results: Judge-panel validity rates were significantly higher for questions generated by OpenAI (94.8%, Wilson 95% CI 85.9%-98.2%), Gemini (92.3%, 95% CI 83.2%-96.7%), and Claude (83.3%, 95% CI 72.6%-90.4%) models than those generated by Kimi (63.8%, 95% CI 54.3%-72.4%), Grok (56%, 95% CI 45.8%-65.8%), and DeepSeek (46.0%, 95% CI 37.4%-54.7%). Kimi also demonstrated significantly higher validity than DeepSeek (2-sample tests of proportions; BH-adjusted =.01). The public-source similarity audit found no exact or near-duplicate matches. Leave-one-out judge agreement was moderate. Generator-specific same-family judging coefficients were nonnegative (0 to 11.0 percentage points); however, none remained significant after BH adjustment. There was no significant difference between the 6 frontier LLM QA accuracies, which ranged from 85.7% (257/300) to 91.7% (275/300), on the 300-question examination (generalized linear mixed model likelihood-ratio test =.07). The study had 80% power to detect LLM accuracy differences of 9.2 percentage points; nonsignificance did not establish equivalence. On the 90-question subset, physician accuracy was 85.6% (95% CI 79.4%‐91.7%; stratified normal with finite-population correction), compared with aggregate LLM performance of 94.4% (85/90; Wilson 95% CI 87.6%‐97.6%). This difference was not significant (McNemar test, =.08); however, the study was powered to detect a human vs LLM accuracy difference of 14% to 15%. Answering-only costs varied by nearly 2 orders of magnitude, with multiple models on the cost-accuracy Pareto frontier. Conclusions: MedEvalArena enables scalable, regenerable, peer-judged benchmarking of LLM medical reasoning. Question-generation validity differentiated model families more clearly than answering accuracy, suggesting that valid MCQ generation may complement QA benchmarks for assessing LLM medical reasoning.

Источник: Journal of Medical Internet Research