Validation of the MySurgeryRisk Algorithm for Predicting Complications and Death after Major Surgery: A Retrospective Multicenter Study Using OneFlorida Data Trust

Despite advances in surgical techniques and care, postoperative complications are prevalent and effects up to 15% of the patients who underwent a major surgery. The objective of this study is to develop and validate models for predicting postoperative complications and death after major surgery on a large and multicenter dataset, following the previously validated MySurgeryRisk algorithm. This retrospective, longitudinal and multicenter cohort analysis included 508,097 encounters from 366,875 adult inpatients who underwent major surgeries and were admitted to healthcare institutions within the OneFlorida+ network between 01/01/2012 and 04/29/2023. We applied the validated feature selection and transformation approach in MySurgeryRisk models and redeveloped eXtreme Gradient Boosting (XGBoost) models for predicting risk of postoperative acute kidney injury (AKI), need for intensive care unit (ICU) admission, need for mechanical ventilation (MV) therapy and in-hospital mortality on a development set and evaluated the model performance on a validation set. Area under the receiver operating characteristics curve values were obtained for need for ICU admission, 0.93 (95% Confidence Interval [CI], 0.93-0.93); need for MV, 0.94 (95% CI, 0.94-0.94); AKI, 0.92 (95% CI, 0.92-0.92); and in-hospital mortality, 0.95 (95% CI, 0.94-0.95). Area under the precision-recall curve values were computed for need for ICU admission, 0.62 (95% CI, 0.62-0.63); need for MV, 0.51 (95% CI, 0.49-0.52); AKI, 0.53 (95% CI, 0.53-0.54); and in-hospital mortality, 0.26 (95% CI, 0.24-0.29). The performance of these models is comparable to that of the previously validated MySurgeryRisk models, suggesting the enhanced generalizability of the models. Primary procedure code and provider specialty consistently appeared as the top influential variables, providing valuable insights into the factors influencing surgical outcomes.