Natural Language Processing Supported by Annotation Process Artifacts to Detect Risky Alcohol Use in Electronic Health Records: Algorithm Development and Validation Study

Journal of Medical Internet Research ·

Background: High-risk alcohol use and alcohol use disorder (AUD) are significant contributors to surgical risk and complications yet remain underidentified in surgical populations. While electronic health records (EHRs) contain relevant information about patients’ alcohol use in clinical notes, manual chart review is time-intensive and impractical at scale. Natural language processing (NLP) offers a path to automated identification, but alcohol-related risk classes are highly imbalanced in clinical populations, making multiclass classification particularly challenging. Objective: This study aims to develop an automated NLP pipeline for 4-class alcohol risk classification using clinical notes in a surgical population, addressing class imbalance by leveraging intermediate annotation evidence generated during the human labeling process. Methods: We manually labeled 1000 surgical patients drawn from a large academic health system into ordinal alcohol risk classes (no-alcohol, low-risk, high-risk, and probable-dependence) and a fifth class indicating no mention of alcohol use. We used a structured annotation process with 6 trained annotators and expert-developed guidelines. We developed a multistage NLP pipeline operating on 3 years of clinical notes per patient. First, a rules-based preprocessing stage filtered notes to clinically relevant sections containing alcohol-related content. No-mention cases were excluded from further modeling. Second, a BioClinicalBERT-based multilabel classifier (“auto-annotator”) was fine-tuned to reproduce the annotations created by human labelers at the section level, producing a structured 26-dimensional probability vector for each section. Finally, a bidirectional long short-term memory (LSTM) network classified patients into the 4 risk categories by processing each patient’s ordered sequence of auto-annotation vectors, allowing the model to capture temporally accumulating evidence across clinical encounters. Results: Human annotators achieved 93.7% first-pass agreement overall (Fleiss κ=0.8891), but agreement was as low as 67.5% for probable-dependence cases. The 4-class LSTM classifier achieved a weighted-average F 1 -score of 0.86 and a macroaverage F 1 -score of 0.77 on a held-out validation set of 195 patients. One-vs-rest area under the curve (AUC) scores were 0.97 for no-alcohol, 0.88 for low-risk, 0.90 for probable-dependence, and 0.71 for high-risk. Collapsed to a binary high/low risk classification, the model achieved an accuracy of 0.92 and a macro- F 1 -score of 0.80. Performance was comparable across biological sex without explicit sex-specific training. Conclusions: We used a 2-stage NLP pipeline to identify 4 ordinal classes of alcohol use risk in surgical patients. Four-class alcohol-use classification presents a highly imbalanced problem that constrains the availability of high-importance but rare labels in the dataset. We addressed this data scarcity by using intermediate annotation evidence as well as final labels to enrich the feature space for the classification task. The pipeline was able to separate patient records into 4 strata of alcohol-use risk, with “high-risk” patients being most likely to be misclassified into an adjacent class.

Источник: Journal of Medical Internet Research