Evaluating Model Resilience in Predicting Surgical Site Infections
$3.5B+
Annual US Healthcare Costs
Surgical Site Infections (SSIs) represent a massive financial burden on the healthcare system.
7-10
Extra Hospital Days
Patients with SSIs face longer, more complicated recoveries, increasing morbidity and mortality risk.
2-11x
Increased Mortality Risk
SSIs increase the risk of mortality and 75% of SSI-related deaths are directly attributable to the SSI itself.
SSIs are proportionally rare, relative to the total number of surgeries, creating datasets where non-infection cases vastly outnumber infection cases. Traditional models become biased towards predicting the majority (no infection), failing to identify the patients most at risk.
Clinical data is not static. Patient populations, protocols, and coding practices change over time. A model trained on past or different data can fail dramatically when deployed for inference on new or "shifted" data, rendering models with good train-test-validation performance unreliable.
Class imbalance and distribution shift are typically handled by applying synthetic data augmentation or sampling methods to rebalance the training data. SMOTE, or Synthetic Minority Over-sampling Technique, is a popular method that tries to balance data and class distributions by generating new examples along data element line segments connecting existing samples and their nearest neighbors.
To ensure a realistic assessment, the study used real-world 2022-2023 data for training-test-validation and 2024 data for separate evaluation. A explicit "dataset classifier" was built to see if it could tell the two datasets apart. Its near-perfect accuracy proved a significant distribution shift occurred, creating a genuine stress test for the models.
Standard models (like CatBoost) using the common technique of generating synthetic data (SMOTE) to balance the classes.
Combining multiple classical models, each trained on a different class balance, to vote on a final prediction.
A complex architecture with specialized "expert" networks and a "gating" network to route data to the right expert.
A Large Language Model (ModernBERT) that treats each patient record as a sentence, leveraging its deep language understanding.
While the standard CatBoost + SMOTE model looked good on the real-world 2022-2023 hold out test and validation data, its performance collapsed on the distribution-shifted 2024 data. This highlights the danger of relying on synthetic data and older architectures.
When tested on the 2024 evaluation data, the differences between models became clear. This chart shows the performance profile of each model. A larger, more balanced shape indicates better overall performance. The LLM's profile is clearly the most robust, perfoming better in Precision and MCC, without sacrificing Recall.
By converting structured data into sentences, the LLM leverages its vast pre-trained knowledge of context and relationships. It doesn't just see numbers; it understands the clinical narrative. This is a powerful form of transfer learning.
The study reveals that SMOTE can have negative effects. It can create a false sense of security by helping models learn artificial patterns that don't exist in the real world, leading to fragile models that can have less-than-advertised perfromance in production.
Use SMOTE to create synthetic data to "fix" the imbalance for simpler models.
Use powerful, pre-trained models like LLMs that can learn robustly from imbalanced, real-world data without synthetic fixes.