American Society of Anesthesiologists Physical Status Classification: Hybrid Machine Learning–Large Language Model Ensemble Retrospective Validation Study
Journal of Medical Internet Research ·
Background: The American Society of Anesthesiologists Physical Status (ASA-PS) classification system fundamentally shapes perioperative care delivery but suffers from poor interrater reliability (0.4‐0.6). Machine learning (ML) models and large language models (LLMs) offer complementary remedies: ML processes structured data consistently but cannot interpret the nuance of clinical narratives, while LLMs parse narrative text and generate explanations but adhere inconsistently to diagnostic criteria. Objective: This study aimed to develop and evaluate a parallel ML-LLM ensemble that combines the complementary strengths of both approaches for automated ASA-PS classification. Methods: We retrospectively analyzed 3000 adult surgical encounters from the University of Arkansas for Medical Sciences (UAMS) between August 2024 and May 2025. Cases were randomly allocated to training (n=2000), validation (N=500), and held-out test sets (N=500). The validation set was used exclusively for architecture comparison and hyperparameter tuning, and the held-out test set was reserved for final model performance validation. We developed multiple architectures, including traditional ML models (Extreme Gradient Boosting [XGBoost], Light Gradient Boosting Machine [LightGBM], ExtraTrees), an LLM-only baseline (Generative Pretrained Transformer-4o [GPT-4o]), and hybrid approaches. The parallel ensemble processed structured data through XGBoost and unstructured clinical notes through GPT-4o independently, with outputs combined via weighted averaging. Model performance was evaluated using macro--score, exact match accuracy, and ±1-class accuracy. Results: A theoretical performance ceiling of macro--score=0.59 was determined through evaluation by an expert panel of 2 board-certified anesthesiologists (AF and GF) who independently rated 50 patient charts for ASA-PS scores. On the true held-out test set (N=500), the parallel ensemble (=0.30) achieved a macro--score of 0.530 (95% CI 0.350‐0.600), exact match accuracy of 0.630 (95% CI 0.580‐0.670), and ±1-class accuracy of 0.980 (95% CI 0.968‐0.992), exceeding the frozen LLM-only baseline evaluated on the same set (macro--score 0.493, exact match accuracy 0.621,±1-class accuracy 0.981). On the validation set used for architecture selection, the ensemble outperformed structured-only ML (XGBoost macro--score=0.333, <.001), and the LLM-only model reached a macro--score of 0.642 (95% CI 0.525‐0.691), nominally exceeding the expert panel ceiling. Per-class analysis attributed this to a perfect score on the 2 validation ASA-PS 5 cases. Excluding that class, the adjusted macro--score was approximately 0.55, within the range of the parallel ensemble. The LLM component also generated explanations detailing comorbidities, severity descriptors, and functional status indicators. Conclusions: The parallel ML-LLM ensemble achieved performance approaching the theoretical ceiling established by the expert panel benchmark while providing interpretable clinical explanations. The ±1-class accuracy of 0.980 (95% CI 0.968‐0.992) may enhance operational safety by minimizing extreme misclassifications. This approach demonstrates how complementary AI architectures can enhance perioperative risk assessment, which is particularly valuable given current health care workforce shortages.