Large Language Model–Assisted Radiology Report Quality Assurance: Real-World Multicenter Evaluation
Journal of Medical Internet Research ·
Background: Radiology reports serve as the definitive clinical record of imaging diagnoses, with their accuracy being directly critical to patient safety. However, amid escalating workloads, manual review remains inadequate for capturing subtle errors in reports on a large scale. Although large language models (LLMs) have demonstrated considerable promise in medical text processing, previous studies have predominantly been confined to radiology report text from a single imaging modality or datasets with synthetic errors. Objective: This study aimed to evaluate the feasibility, accuracy, and generalizability of general-purpose LLMs for quality assurance of radiology report text from multiple imaging modalities (x-ray, computed tomography [CT], and magnetic resonance imaging [MRI]), multi-anatomical regions, and multicenter clinical settings, with head-to-head comparisons conducted against radiologists of varying seniority. Methods: Radiology report texts from 3 aforementioned imaging modalities and 5 anatomical regions (head, chest, abdomen, spine, and knee) were retrospectively collected from center 1 (March 2025-May 2025) and reviewed using a clinician-annotated 5-category error taxonomy (type 1: interpretive; type 2: factual). Five LLMs (Qwen3, GPT-5.4 Pro, Gemini 3.1 Pro, Llama 4, and DeepSeek-V3.1) were benchmarked on 10,000 randomly selected reports, and the best-performing model was evaluated on the full cohort. A blinded head-to-head comparison with 3 radiologists (3-10 years of experience) used a modality-stratified, error-enriched challenge set. The radiologists subsequently rated the model-generated explanations using a 5-point Likert scale. Cross-center performance was evaluated in an independent external cohort. The same 2 senior radiologists labeled reports from both centers using the same protocol, resolving disagreements by consensus. Results: In total, 4.6% (4431/95,582) of the reports in center 1 contained at least one confirmed text internal error. Qwen3, the selected model, achieved a report-level sensitivity, specificity, positive predictive value, and -score of 0.99, 0.99, 0.90, and 0.94, respectively, with high performance across reports from the 3 imaging modalities and multiple anatomical regions and better detection of factual than interpretive errors. In the error-enriched challenge set, its error instance exact category recall exceeded that of the radiologists (0.96 vs 0.45-0.67; <.001), with a shorter mean task completion time (5.80 vs 55.80-67.80 seconds per report). Explanations had a mean Likert score of 4.57, and all inter- and intraobserver intraclass correlation coefficients exceeded 0.89. In center 2, balanced accuracy was 0.94 (95% CI 0.92‐0.95). Before consensus, Cohen κ for report-level error presence was 0.783 (center 1) and 0.850 (center 2). Conclusions: This study suggests that LLMs can serve as efficient adjunctive tools for text-level quality control of radiology reports, particularly for detecting explicit factual errors. Performance remained stable across centers. In the text-only review task, the model achieved higher exact category recall and shorter task completion times than the participating radiologists. These findings support the potential value of LLM-assisted radiology report text review in real-world clinical environments.