Evaluating DeepSeek in Real-World Clinical Scenarios on Performance and Safeguards: Multicenter Comparative Study
Journal of Medical Internet Research ·
Background: DeepSeek, an open-source large language model (LLM) extensively adopted in hospitals across China, presents a locally deployable alternative to proprietary models such as GPT. However, its real-world clinical performance and safeguards remain underexplored. Objective: The aim of this study is to evaluate the performance of the general-purpose and reasoning versions of DeepSeek (DeepSeek AI) and GPT models in diagnosis, medical record analysis, and safeguards against adversarial vulnerabilities. Methods: This cross-sectional study included 50 real-world medical records from 10 hospitals spanning 14 specialties. A total of 7 LLMs were evaluated, including DeepSeek-V3, DeepSeek-R1, GPT-4o (via ChatGPT; OpenAI), o3-mini (via ChatGPT), o3-mini-high (via ChatGPT), GPT-4o (via Copilot; Microsoft), and GPT-4o-reasoning (via Copilot). Models were tasked with generating diagnoses based on admission notes, answering 12 questions related to extraction, summarization, and inference using discharge summaries, and undergoing 4 rounds of simulated adversarial prompt attacks designed to elicit misinformation. The diagnostic evaluation was performed on 47 records after the exclusion of 3 flawed admission notes, while the other tasks used the full set of 50 records. Diagnostic performance was evaluated based on accuracy, with the causes of misdiagnosis further categorized. Medical record analysis was assessed by scores in extraction, summarization, and inference tasks. Safeguard evaluation included identifying the earliest round of successful prompt attack and verifying reference authenticity and accuracy. Results: Among the 7 models included in this study, DeepSeek-R1 and GPT-4o-reasoning (via Copilot) achieved the highest diagnostic accuracy at 64% (30/47). In medical record analysis, both models also excelled across the 3 subdimensions. GPT-4o-reasoning (via Copilot) achieved accuracies of 97.7% (293/300) in extraction, 87.5% (175/200) in summarization, and 87% in inference, while DeepSeek-R1 reached 97.7% (293/300), 83.5% (167/200), and 85% (85/100), respectively. In simulated adversarial safeguard evaluation involving 4 rounds of prompt attacks, all evaluated models showed high vulnerability. DeepSeek-R1 generated medical disinformation in 98% (49/50) of cases, with only a 2% refusal rate. GPT-4o-reasoning (via Copilot) showed the highest refusal rate at 30%, but it still generated disinformation in 70% (35/50) of cases. True hallucination rates ranged from 38.6% (27/70; GPT-4o-reasoning via Copilot) to 73% (73/100; DeepSeek-V3). Conclusions: These findings underscore the clinical application potential of DeepSeek models, while also emphasizing the critical need to strengthen safeguards against adversarial vulnerabilities prior to their clinical deployment.