Diagnostic Accuracy and Fine-Grained Image-Interpretation Performance of Multimodal Large Language Models on Bilingual Radiology Questions: Cross-Sectional Benchmark Study
Journal of Medical Internet Research ·
Background: Multimodal large language models (LLMs) are rapidly expanding in radiology and digital health; however, their diagnostic performance across diverse clinical scenarios and fine-grained visual reasoning tasks requires systematic evaluation. Objective: The study aims to evaluate the diagnostic accuracy and fine-grained visual reasoning performance of 5 multimodal LLMs with bilingual medical imaging questions. Methods: This cross-sectional benchmark study evaluated bilingual medical imaging questions using zero-shot prompting on 5 LLMs: GPT-5.2, Gemini 3 Pro, Doubao, Yuanbao, and Llama 4 Maverick. Subgroup analyses by language (English/Chinese), question type (image/text), imaging modality (computed tomography, radiography, magnetic resonance imaging, ultrasound, and nuclear medicine), and anatomical system (neurologic, respiratory, cardiovascular, digestive, genitourinary, and musculoskeletal) evaluated intermodel accuracy and intramodel consistency. Image-based visual reasoning performance was assessed across modality, anatomy, lesion detection, and localization. Results: A total of 298 questions (198 image-based, 100 text-based) were evaluated. Gemini 3 Pro achieved the highest overall accuracy (n=268, 89.9%, 95% CI 86.0%‐92.9%), significantly outperforming Doubao (n=235, 78.9%; <.001), GPT-5.2 (n=233, 78.2%; <.001), Yuanbao (n=216, 72.5%; <.001), and Llama 4 Maverick (n=216, 72.5%; <.001). All models performed better on text-based than image-based questions (Yuanbao, <.001; Doubao, =.02; GPT-5.2, <.001; Gemini 3 Pro, =.002; Llama 4 Maverick, <.001). GPT-5.2 (<.001), Llama 4 Maverick (=.03), and Yuanbao (=.03) favored English questions, while intramodel performance remained stable across modalities (Yuanbao, =.46; Doubao, =.21; GPT-5.2, =.26; Gemini 3 Pro, =.96; Llama 4 Maverick, =.45) and anatomical systems (Yuanbao, =.82; Doubao, =.52; GPT-5.2, =.35; Gemini 3 Pro, =.13; Llama 4 Maverick, =.97). Despite near-perfect macroscopic visual recognition (modality and anatomy: 99.0%‐100.0%), accuracy dropped in complex tasks. Gemini 3 Pro outperformed Yuanbao (<.001), GPT-5.2 (=.002), and Llama 4 Maverick (<.001) in lesion detection (175/198, 88.4%) and outperformed Yuanbao (<.001), Doubao (=.004), and Llama 4 Maverick (=.004) in lesion localization (189/198, 95.5%). Conclusions: Gemini 3 Pro demonstrated optimal, comprehensive performance and cross-lingual stability. However, current multimodal LLMs face a technical bottleneck in complex visual interpretation—particularly in fine-grained lesion detection and localization—emphasizing the need to prioritize local feature perception in future model development.