Generative Large Language Models for Psychiatric Concept Detection in Spanish-Language Electronic Health Records: Development and Evaluation Study
Journal of Medical Internet Research ·
Background: The accurate detection of clinical symptoms in electronic health records (EHRs) is pivotal for advancing large-scale genetic and longitudinal studies in psychiatry. Free-text clinical notes are an essential source of symptom-level information, particularly in psychiatry; however, the automated extraction of symptoms from clinical text remains challenging. Objective: We aimed to evaluate the ability of open-source generative large language models (LLMs) to detect psychiatric symptom-level concepts from clinical text in Spanish and to develop a new fine-tuned, open-source LLM that can be shared without risk of exposing patient information. Methods: We tested 11 open-source generative LLMs for their ability to detect 109 psychiatric concepts from clinical text, using annotated EHR notes from a psychiatric hospital in Colombia. The LLMs were evaluated both “out-of-the-box” and after fine-tuning and compared against a traditional natural language processing (tNLP) method developed from the same data. To develop a fine-tuned LLM that can be shared without risk of exposing patient information, we created a novel synthetic dataset, leveraging real concept annotations while generating fully synthetic clinical text, and fine-tuned a top-performing LLM on these data. Fine-tuned model generalizability was assessed through cross-site evaluation using data from a second independent hospital in Colombia. Results: We show that while base LLM performance was poor to moderate (zero-shot: 0.2-0.6 macro- F 1 ; few-shot: 0.2-0.74 macro- F 1 ), it improved significantly after fine-tuning (0.75-0.86 macro- F 1 ), with several fine-tuned LLMs outperforming the tNLP method. In total, 100 psychiatric concepts could be reliably detected ( F 1 >0.8) using either a fine-tuned LLM or tNLP. Using the synthetic dataset, we developed Mistral-small-psych, a new LLM capable of detecting psychiatric concepts from Spanish clinical text with performance comparable to models trained on real EHR data (macro- F 1 =0.79) while remaining fully distributable. In cross-site evaluations, most LLMs generalized well, with small out-of-domain performance losses (macro- F 1 decrease of 0.02-0.16). Conclusions: This work provides the first large-scale, cross-institutional evaluation of state-of-the-art LLMs for psychiatric concept detection in Spanish EHRs, introduces a privacy-preserving pathway for building shareable clinical natural language processing models, and presents a new open model for extracting symptom-level psychiatric information from Spanish clinical text.