Skip to main content
Download PDF

Development and validation of a score to predict mortality in ICU patients with sepsis: a multicenter retrospective study

Journal of Translational Medicine volume 19, Article number: 322 (2021) Cite this article

Abstract

Background

Early and accurate identification of septic patients at high risk for ICU mortality can help clinicians make optimal clinical decisions and improve the patients’ outcomes. This study aimed to develop and validate (internally and externally) a mortality prediction score for sepsis following admission in the ICU.

Methods

We extracted data retrospectively regarding adult septic patients from one teaching hospital in Wenzhou, China and a large multi-center critical care database from the USA. Demographic data, vital signs, laboratory values, comorbidities, and clinical outcomes were collected. The primary outcome was ICU mortality. Through multivariable logistic regression, a mortality prediction score for sepsis was developed and validated.

Results

Four thousand two hundred and thirty six patients in the development cohort and 8359 patients in three validation cohorts. The Prediction of Sepsis Mortality in ICU (POSMI) score included age ≥ 50 years, temperature < 37 °C, Respiratory rate > 35 breaths/min, MAP ≤ 50 mmHg, SpO2 < 90%, albumin ≤ 2 g/dL, bilirubin ≥ 0.8 mg/dL, lactate ≥ 4.2 mmol/L, BUN ≥ 21 mg/dL, mechanical ventilation, hepatic failure and metastatic cancer. In addition, the area under the receiver operating characteristic curve (AUC) for the development cohort was 0.831 (95% CI, 0.813–0.850) while the AUCs ranged from 0.798 to 0.829 in the three validation cohorts. Moreover, the POSMI score had a higher AUC than both the SOFA and APACHE IV scores. Notably, the Hosmer–Lemeshow (H–L) goodness-of-fit test results and calibration curves suggested good calibration in the development and validation cohorts. Additionally, the POSMI score still exhibited excellent discrimination and calibration following sensitivity analysis. With regard to clinical usefulness, the decision curve analysis (DCA) of POSMI showed a higher net benefit than SOFA and APACHE IV in the development cohort.

Conclusion

POSMI was validated to be an effective tool for predicting mortality in ICU patients with sepsis.

Background

Sepsis remains the leading cause of death in patients admitted to the Intensive Care Unit (ICU), worldwide and is therefore a great challenge to clinicians [1,2,3]. According to guidelines by the Surviving Sepsis Campaign, timely diagnosis and identification of patients at risk is recommended to provide aggressive early intervention and improve the prognosis of septic patients [4]. Therefore, early recognition of septic patients at high risk is essential for reducing fatality. Additionally, it is necessary to build a reliable predictive system for clinicians to improve the patients’ outcomes given that sepsis is a complex, heterogeneous disease associated with high morbidity and mortality. Moreover, trustworthy clinical management strategies can be provided for critically ill patients, with the assistance of various predictive systems.

Prediction of mortality in sepsis continues to be the main focus of critical care medicine. In addition, accurate clinical risk prediction models can objectively estimate disease severity and stratify patients according to the risk of death. They may also alert clinicians and allow for timely identification of high-risk populations that require aggressive management and intervention. Consequently, several predictive systems have been developed to date, including the Acute Physiology and Chronic Health Evaluation (APACHE) II and IV scores [5, 6], Simplified Acute Physiology Score (SAPS) II and III [7, 8], Sequential Organ Failure Assessment (SOFA) score [4], Predisposition, Insult/Infection, Response and Organ Dysfunction (PIRO) [9] and the Charlson comorbidity index [10]. Although these scores have been applied widely in a variety of patient groups, especially those who are critically ill, they may exhibit poor sensitivity or/and specificity and low reproducibility when applied to some specific diseases [11]. Notably, there are currently no risk prediction systems available specifically for septic patients.

Therefore, the present study attempted to develop and validate (both internally and externally) a mortality prediction score based on clinical and laboratory data from more than 200 hospitals, to estimate mortality in ICU patients with sepsis. The study also compared the prediction performance of the score with that of the APACHE IV and SOFA scores. It was hypothesized that the risk prediction score developed by the study would more accurately predict the risk for sepsis than both the APACHE IV and SOFA scores, hence being more beneficial to septic patients.

Materials and methods

Data sources

This was a multicenter, retrospective, observational study that was conducted using data from the eICU Collaborative Research Database which is a large multi-center critical care database containing information on 139,367 patients from 335 ICUs in 208 hospitals across the USA, in 2014 and 2015 [12, 13]. The study also obtained patient clinical data from the Second Affiliated Hospital and Yuying Children’s Hospital of Wenzhou Medical University, Wenzhou, China, which had over 2000 beds. Data on septic patients admitted to the hospital between January 1, 2010 and September 31, 2020 was obtained through the electronic medical record management system. The study was approved by the Ethics Committee of the Second Affiliated Hospital and Yuying Children’s Hospital of Wenzhou Medical University.

Participants

Participants were enrolled based on the definition of sepsis-3 i.e. a known or suspected infection plus SOFA > 2 points for organ dysfunction [14, 15]. In addition, the first ICU admission was selected for septic patients admitted to the ICU more than once. The study however excluded patients who were younger than 18 years of age. Considering the different medical care levels in different regions, we divided the patients into three groups (Midwest, West and South) according to the hospital locations in the USA. Septic patients from the Midwest were used as the development cohort because of the largest sample size. Patients from West and South were used as external validation sets. And, data from the Chinese ICU was acted as another external validation set.

Variables

The Structured Query Language (SQL) with pgAdmin 4 (version 4.30) was used to extract data from the eICU database. The study retrospectively collected the following data: (1) demographic information including age, sex, race, height and weight; (2) site of infection, including pulmonary, renal/urinary tract infection (UTI), cutaneous/soft tissue, Gastrointestinal (GI), gynecologic, others, and unknown; (3) APACHE IV and SOFA scores on the day of ICU admission; (4) vital signs including temperature, heart rate, respiratory rate, systolic pressure, diastolic pressure, mean arterial pressure (MAP) and oxygen saturation levels at the first records after ICU admission; (5) laboratory data, including albumin, bicarbonate, bilirubin, creatinine, glucose, hematocrit, hemoglobin, lactate, platelet, blood urea nitrogen (BUN), white blood cell (WBC) and alanine transaminase (ALT) within 24 h of ICU admission; (6) comorbidities including Acquired Immunodeficiency Syndrome (AIDS), hepatic failure, lymphoma, metastatic cancer, leukemia, immunosuppression, and cirrhosis. For laboratory data recorded more than once, values associated with the most severe form of sepsis were employed. The proportion of missing values was less than 10% across all the variables.

Endpoints

The main outcome of the present study was ICU mortality. Survival following admission to the ICU was clearly recorded in eICU database. On the other hand, electronic medical records were used to retrieve information on patients who survived following admission in the ICU of the Second Affiliated Hospital and Yuying Children’s Hospital of Wenzhou Medical University.

Statistical analysis

The Shapiro–Wilk test was used to examine whether the data was normally distributed. Categorical variables were described by frequency (percentages) and mean (SD) or median (interquartile range) for continuous variables, as appropriate. In addition, non-normal continuous variables were compared using the Wilcoxon rank-sum test while the Student’s t test was employed for the normally distributed data. Moreover, categorical variables were analyzed using the chi-squared test or Fisher’s exact test, accordingly.

The primary outcome for the study was ICU mortality. Therefore, univariate logistic regression analyses were conducted to identify the unadjusted association between potential predictors and ICU mortality. For the ICU mortality model, the backward stepdown logistic regression based on the smallest Akaike Information Criterion (AIC) value was selected to confirm the independent risk variables for ICU mortality [16]. Additionally, multicollinearity of variables was examined using the Variance Inflation Factor (VIF) for each predictive variable and a VIF ≥ 5 indicated multicollinearity among variables.

Thereafter, the above continuous independent predictor variables were transformed into categorical variables based on quartiles then all the categorical variables (including AIDS, hepatic failure and metastatic cancer) were subjected to multivariable logistic regression to identify the final predictor variables in the prediction scoring system. The study developed this scoring system to predict mortality in septic patients and named it, Prediction of Sepsis Mortality in the ICU (POSMI). POSMI was developed by allocating an integer or half an integer score, which was calculated by dividing the regression coefficient of each predictor variable with the smallest regression coefficient. The sum of each predictor variable score yielded a total score for each individual and this total score was included in the final regression model. In addition, the model’s discrimination for ICU mortality was examined using the area under the receiver operating characteristic curve (AUC) and calibration was conducted using calibration curves and the Hosmer–Lemeshow (H–L) goodness-of-fit test. Moreover, the DeLong’s non-parametric method was used to compare the two AUC values with an equal sample size [17]. Following recommendations by Hosmer and Lemeshow, an AUC ≥ 0.7 indicated an acceptable discrimination while an AUC ≥ 0.8 showed excellent discrimination. Furthermore, Integrated Discrimination Improvement (IDI) was used to evaluate improvement in model performance [18] and the 95% Confidence Intervals (CIs) were calculated using non-parametric bootstrapping. In order to assess the clinical utility of the POSMI score, Decision Curve Analysis (DCA) was performed to compare the net benefit of the POSMI, APACHE IV and SOFA scores in the prediction of ICU mortality, at different threshold probabilities.

Given that missing data could have influenced the results to some extent, the multiple imputation technique using chained equations was employed in order to minimize bias and maintain the power of the study before data analysis. The “mice” package in R was used to implement this method. Additionally, sensitivity analyses were conducted to evaluate the influence of missing value filling. All the statistical analyses were conducted using R (version 3.6.1) and a p value < 0.05 was considered to be statistically significant.

Results

Populations

Ten thousand seven hundred and fifty four septic patients from the eICU database met the inclusion criteria. Based on geographical locations in the USA, 4236 patients were from the Midwest, 3185 from the West, 2934 from the South, 386 from the Northeast and 13 were from unknown locations. In addition, the study consecutively collected a total of 1878 sepsis cases from the Second Affiliated Hospital and Yuying Children’s Hospital of Wenzhou Medical University, Wenzhou, China, between January 2010 and September 2020. Notably, septic patients from the Midwest, who had the largest sample size, were used as the development cohort. The median age in the development cohort was 69 years (range, 57 to 80 years) and 53% of the patients were male. Additionally, the infection sites most frequently associated with sepsis were pulmonary (41%), renal/UTI (23%), GI (13%), unknown (10%), cutaneous/soft tissue (8%) and others (4%). The ICU mortality and hospital mortality rates were 11.8%, and 19.1%, respectively. On the other hand, patients from the West of USA and Wenzhou, China were used as the validation set, named as the West and Wenzhou validation cohorts. Moreover, patients from the Northeast and unknown regions were grouped into the South region due to the small sample sizes and used as another validation set, named as the South validation cohort. Table 1 shows the demographic and clinical characteristics of the development and validation cohorts.

Table 1 Demographic and clinical characteristics of the patients with sepsis following ICU admission
Full size table

Predictors of ICU mortality

Univariate logistic regression was used to test for the potential risk factors that would predict ICU mortality. Most of the variables were associated with ICU mortality (Additional file 1: Table S1). However, the study performed the backward stepdown multivariate logistic regression analysis based on the smallest AIC value in order to determine the independent risk factors. The results revealed fourteen independent risk factors including; age, temperature, heart rate, respiratory rate, MAP, SpO2, mechanical ventilation, albumin, bilirubin, lactate, BUN, AIDS, hepatic failure and metastatic cancer (Additional file 1: Table S2). Thereafter, continuous variables were converted into categorical variables based on the quartiles in order to further validate the above continuous independent predictor variables and for practical purposes. Multivariable logistic regression analysis identified age ≥ 50 years, temperature < 37 °C, respiratory rate > 35 breaths/min, MAP ≤ 50 mmHg, SpO2 < 90%, albumin ≤ 2 g/dL, bilirubin ≥ 0.8 mg/dL, lactate ≥ 4.2 mmol/L, BUN ≥ 21 mg/dL, mechanical ventilation, hepatic failure and metastatic cancer (Additional file 1: Table S3).

The POSMI score

Twelve variables were used to create the POSMI score and each prognostic variable was assigned a score (Table 2). The POSMI score for each patient was derived by obtaining a sum of the points corresponding to prognostic factors, whose scores ranged from 0 to 25. In addition, septic patients in the Midwest development cohort were divided into four categories according to each POSMI score distribution. These included the low risk (0–6 points) category which had 1.2% ICU mortality, moderate risk (7–10 points) which had 6.3% ICU mortality, high risk (11–15 points) which had 23.3% ICU mortality and very high risk (> 7 points) which had 66.9% ICU mortality (Table 3).

Table 2 Risk factors for predictive model for ICU mortality in the midwest development cohort (n = 4236)
Full size table
Table 3 Risk of ICU mortality in the development and validation cohorts according to risk stratification
Full size table

Risk stratification

Classification of the Midwest development cohort based on the POSMI score resulted in 1126 (26.6%) patients in the low-risk class, 1,836 (43.3%) in the moderate-risk group, 1105 (26.1%) in the high-risk category and 169 (4.0%) in the very high-risk class (Table 3). Classification results for the West, South and Wenzhou validation cohorts were similar to those of the development cohort. In the West, South and Wenzhou validation cohorts, 24.8%, 24.5%, and 24.6% of the patients, respectively, were assigned to the low-risk class; 40.9%, 42.7% and 39.7%, respectively, fell under the moderate-risk group; 28.5%, 27.5% and 30.0%, respectively, were classified into the high-risk category and 5.8%, 5.3% and 5.7%, respectively, were assigned to the very high-risk class (Table 3). Moreover, patients in the three validation cohorts showed ICU mortality rates similar to those in the development cohort, in the four risk classifications (Table 3). Furthermore, the ICU mortality predicted by the POSMI score was very close to that of actual ICU mortality in the four different risk levels (Table 3).

Validation of the POSMI score

Performance of the POSMI score was compared to that of the SOFA and APACHE IV scores in predicting ICU mortality in septic patients. The AUC value for the POSMI score was 0.831 (95% CI, 0.813–0.850) and was significantly higher than that of the SOFA score which was 0.728 (95 CI, 0.703–0.754) and the APACHE IV score which was 0.773 (95% CI, 0.752–0.795), in the development cohort (Fig. 1A; Table 4). This indicated that the POSMI score had better discrimination than both the SOFA and APACHE IV scores. Similarly, the AUC values for the POSMI score in the West and South validation cohorts were more than 0.8 and were also significantly higher than those of both the SOFA and APACHE IV scores. The results therefore showed that the POSMI score had excellent discrimination in predicting mortality in ICU patients with sepsis (Fig. 1B, C; Table 4). In the Wenzhou validation cohort, the AUC value of the POSMI score was 0.798 (95% CI, 0.769–0.826) and was higher than that of the SOFA score which was 0.747 (95% CI, 0.714–0.780) and the APACHE IV score which was 0.777 (95% CI, 0.747–0.807). However, no significant differences in AUC were obtained between POSMI and APACHE IV (Fig. 1D; Table 4).

Fig. 1
figure 1

Receiver operating characteristic curves of POSMI, SOFA and APACHE IV scores in predicting ICU mortality in the development and validation cohorts. Receiver operating characteristic curves of the three scores in predicting mortality in the (A) Midwest development cohort, (B) West validation cohort, (C) South validation cohort and (D) Wenzhou validation cohort

Full size image
Table 4 Comparison of models in predicting the ICU mortality of sepsis
Full size table

Additionally, the accuracy of the POSMI score was assessed using calibration curves and the H–L Chi-square test, in the development and validation cohorts. The bias-corrected curve, generated through a bootstrap method, showed a slight deviation from the reference line although the predicted ICU mortality was still in good agreement with the actual ICU mortality (Fig. 2). Moreover, the H–L Chi-square test showed that the POSMI score had good calibration in the Midwest development cohort (HL Chi-square = 10.963; p = 0.204). Good calibration was also confirmed in the West (HL Chi-square = 3.092; p = 0.929), South (HL Chi-square = 10.888; p = 0.208) and Wenzhou validation cohorts (HL Chi-square = 13.135; p = 0.107) as shown in Table 4. In addition, there was a significant increase in the IDI of the POSMI score compared to that of the SOFA and APACHE IV scores in the development and validation cohorts. This suggested that the POSMI score could improve significantly in prediction performance (Table 4). Furthermore, excellent discrimination and calibration were still observed in the sensitivity analyses when missing values were excluded, in the development and validation cohorts. In addition, the AUCs for predicting ICU mortality in the Midwest, West, South and Wenzhou cohorts were 0.794 (95% CI, 0.767–0.822), 0.818 (95% CI, 0.789–0.847), 0.835 (95% CI, 0.811–0.859) and 0.826 (95% CI, 0.790–0.862), respectively. On the other hand, the H–L goodness-of-fit test results were 13.761 (p = 0.088), 2.657 (p = 0.954), 1.008 (p = 0.201) and 7.405(p = 0.493), respectively.

Fig. 2
figure 2

Calibration curves constructed through the bootstrap approach in the development and validation cohorts. The predicted ICU mortality was in agreement with the actual ICU mortality in the (A) Midwest development cohort, (B) West validation cohort, (C) South validation cohort and (D) Wenzhou validation cohort (D)

Full size image

Net benefit of using the POSMI score

Decision curve analysis (Fig. 3) showed that POSMI had a positive net benefit at a predicted threshold probability between 1 and 80% compared to treating septic patients as if they would all have died or they would all have survived (i.e., treat-all or treat-none strategies). The estimates of net benefits from using the POSMI score at different probability thresholds are provided in Table 5 (more estimates of net benefits are shown in Table S4). When the predicted threshold probability was 1% to 60% for the SOFA score and 1–80% for the APACHE IV score, the net benefits were positive in both scores (Table 5). With regard to clinical use, medical treatment aided by POSMI had more net benefit than using both the SOFA and APACHE IV scores when the predicted threshold probability was between 1 and 80% (Fig. 3; Table 5).

Fig. 3
figure 3

The DCA curve of medical intervention in patients with the POSMI, SOFA, and APACHE IV scores in the Midwest development cohort

Full size image
Table 5 Net benefit of using the POSMI score, APACHE IV and SOFA compared to treating sepsis assuming all of them will die during the ICU stay
Full size table

Discussion

The present study involved 12,631 patients admitted with sepsis to more than 300 ICUs in over 200 hospitals. The study developed and validated (both internally and externally) a POSMI score for predicting the risk of ICU mortality. Although some of the predictor variables in the risk score have been reported previously, there is a limited number of tools for predicting the risk of mortality in septic patients [19,20,21]. Notably, the novel POSMI score developed by the study had a number of advantages. The score could easily be implemented based on the available common variables and had good calibration as well as discrimination for ICU mortality in septic patients in both the development and validation cohorts. Additionally, the discrimination and IDI of the POSMI score were significantly higher than those of the APACHE IV and SOFA scores (discrimination of the POSMI score was similar to that of the APACHE IV score in Wenzhou validation cohort). The score may therefore be ideal for guiding decision-making in clinical practice for the management of septic patients. Moreover, the POSMI score showed comparable or better discrimination for predicting ICU mortality in sepsis, compared to other predictive scoring systems in sepsis and critically ill patients. Such include prediction of mortality in sepsis (AUC, 0.68–0.75) [20, 22,23,24], prediction of ICU mortality in surgical patients (AUC, 0.72) [25], prediction of mortality in the critically ill with sepsis using the SOFA score (AUC, 0.77) [26] and prediction of mortality in an academic cardiac intensive care unit using the APACHE IV score (AUC, 0.82) [27]. Considering the high morbidity and mortality rates associated with sepsis, it is necessary to establish a risk score for clinicians to accurately predict and evaluate the outcomes of septic patients. This will also be important in clinical decision-making.

Body temperature is a main area of focus in studies on sepsis [28]. For instance, two recent studies on body temperature and sepsis showed that hyperpyrexia was associated with poor prognosis in septic patients [29, 30]. In addition, a randomized controlled trial demonstrated that fever control by external cooling, significantly reduced early mortality in septic shock [31]. Additionally, most studies showed that hypothermia was associated with a higher mortality in septic patients [32,33,34]. According to a previous study, the occurrence of fever in sepsis may be associated with better survival [35]. However, the present study found an association between an admission body temperature below 37 °C and the risk of ICU mortality although body temperature alone was not sufficiently predictive of the severity of illness. In addition to body temperature, the study showed that respiratory rate and blood pressure were also predictors of poor outcomes in patients with sepsis. It is noteworthy that the two have been adopted as predictors in many critical illness prediction scoring systems, such as qSOFA [20] as well as the APACHE II and IV scores [5, 6]. Moreover, heart rate was not independently associated with mortality from sepsis in the study. Nonetheless, variability in heart rate was associated with mortality from sepsis in some studies previously reported [36,37,38]. Consequently, heart rate at admission was not incorporated in the prediction model for convenience of implementation and practical implications. The POSMI score also included age, SpO2, mechanical ventilation, hepatic failure, metastatic cancer and clinical laboratory values. Patients with low SpO2 or mechanical ventilation may have already developed sepsis-induced acute respiratory failure/Acute Respiratory Distress Syndrome (ARDS) which is associated with high case-fatality [2, 39, 40]. Furthermore, the clinical laboratory variables incorporated in the present prediction model are plausible risk factors for sepsis. Notably, low albumin, high bilirubin and BUN reflect acute and/or chronic damage of the liver and kidney, which are both strong and independent risk factors of prognosis in critical illness [41]. Additionally, high serum lactate was proven to be significantly associated with mortality in patients with sepsis [42,43,44]. All these factors could therefore provide important prognostic information for the prediction model.

Performance of the model in this study was evaluated based on discrimination and calibration through statistical analysis and graphical methods. The AUC for the development and validation cohorts ranged from 0.798 to 0.831, reflecting the excellent ability of the model to discriminate ICU mortality in patients. Additionally, the H–L goodness-of-fit test results and calibration curves suggested that the predicted ICU mortality was similar to the actual ICU mortality, indicating that the prediction model was well calibrated. Moreover, the study validated the calibration of the prediction model at four risk levels (low-, moderate-, high- and very high risk). Expectedly, the ICU mortality predicted by the model was almost consistent with the actual ICU mortality. In addition, the IDI of POSMI in the development and validation cohorts were all significantly higher than those of the APACHE IV and SOFA score, suggesting that the prediction model was superior to both the APACHE IV and SOFA scores. The study also used cohorts with no missing values to conduct sensitivity analysis. Although the multiple imputation approach was used, the POSMI score still maintained excellent discrimination and calibration. With regard to clinical benefit, patients could get more net benefit from using the POSMI score.

Although a high- or a very high-risk score does not directly influence treatment decision-making, it may be useful in making objective prognoses and recommendations for clinicians as well as patients and their families. Nevertheless, further studies are required to confirm the clinical application of the POSMI score. In addition, clinical trials on sepsis may benefit from using the POSMI score as an inclusion and exclusion criterion. For instance, very high-risk patients, where therapeutic measures may not bring clinical benefits because of the severity of disease and low-risk patients whose event rate may be too low to warrant inclusion, may be excluded to optimize the study design. Furthermore, the POSMI score could facilitate patient stratification in clinical studies.

The present study had a number of strengths. First, the POSMI score had excellent model performance in the development and external validation cohorts. The POSMI score was also relatively easy to calculate and all the variables could easily be obtained. In addition, the development and validation cohorts were from hospitals of different sizes (most hospitals from the eICU database are small and medium-sized while the Wenzhou validation cohort came from a large-sized hospital), making it possible to use the model in other hospitals or countries. Nonetheless, the study had a few limitations. First, this was a retrospective cohort study and although we adjusted for many potential confounders, the possibility of residual confounders remains, and the POSMI score was only validated in USA and China, further validation is needed to determine whether our prediction model is applied to other locations or countries. Second, information on the time from onset of illness to hospital admission was missing. Third, the reported ICU mortality was all-cause mortality, the cause of death was not available in the cohorts. Finally, there was no information about treatment in preventing the ICU mortality. Most notably, the baseline differences between the study populations (different continents), ICU practices, study dates (2 years vs. 10 years, one of which includes the COVID-19 pandemic which is likely to have influenced the ICU data collected during that time) were not addressed in present study. Hence, more prospective studies are therefore needed to validate these findings.

Conclusions

In conclusion, the present study developed and validated a simple risk score, POSMI score, which is valuable in predicting mortality in septic patients admitted to the ICU. The POSMI score is superior to the SOFA and APACHE IV scores in present study. We anticipate it will be most useful for risk stratification and decision-making.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

ICU:

Intensive care unit

APHACHE:

Acute physiology and chronic health evaluation

SAPS:

Simplified acute physiology score

SOFA:

Sequential organ failure assessment

PIRO:

Predisposition, insult/infection, response and organ dysfunction

SQL:

Structured query language

UTI:

Urinary tract infection

MAP:

Mean arterial pressure

BUN:

Blood urea nitrogen

WBC:

White blood cell

ALT:

Alanine transaminase

ARDS:

Acquired immunodeficiency syndrome

AIC:

Akaike information criterion

VIF:

Variance inflation factor

POSMI:

Prediction of sepsis mortality in the ICU

ROC:

Receiver operating characteristic curve

H–L:

Hosmer–Lemeshow

IDI:

Integrated discrimination improvement

CI:

Confidence interval

DCA:

Decision curve analysis

SpO2:

Oxyhemoglobin saturation

ARDS:

Acute respiratory distress syndrome

References

  1. Vincent JL, Marshall JC, Namendys-Silva SA, et al. Assessment of the worldwide burden of critical illness: the intensive care over nations (ICON) audit. Lancet Respir Med. 2014;2(5):380–6.

    Article  PubMed  Google Scholar 

  2. Lelubre C, Vincent JL. Mechanisms and treatment of organ failure in sepsis. Nat Rev Nephrol. 2018;14(7):417–27.

    Article  PubMed  Google Scholar 

  3. Yébenes JC, Ruiz-Rodriguez JC, Ferrer R, et al. Epidemiology of sepsis in Catalonia: analysis of incidence and outcomes in a European setting. Ann Intensive Care. 2017;7(1):19.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Singer M, Deutschman CS, Seymour CW, et al. The third international consensus definitions for sepsis and septic shock (Sepsis-3). JAMA. 2016;315(8):801–10.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  5. Knaus WA, Draper EA, Wagner DP, et al. APACHE II: a severity of disease classification system. Crit Care Med. 1985;13(10):818–29.

    Article  CAS  PubMed  Google Scholar 

  6. Zimmerman JE, Kramer AA, McNair DS, et al. Acute physiology and chronic health evaluation (APACHE) IV: hospital mortality assessment for today’s critically ill patients. Crit Care Med. 2006;34(5):1297–310.

    Article  PubMed  Google Scholar 

  7. Le Gall JR, Lemeshow S, Saulnier F. A new simplified acute physiology score (SAPS II) based on a European/North American multicenter study. JAMA. 1993;270(24):2957–63.

    Article  PubMed  Google Scholar 

  8. Moreno RP, Metnitz PG, Almeida E, et al. SAPS 3—from evaluation of the patient to evaluation of the intensive care unit. Part 2: development of a prognostic model for hospital mortality at ICU admission. Intensive Care Med. 2005;31(10):1345–55.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Cardoso T, Teixeira-Pinto A, Rodrigues PP, et al. Predisposition, insult/infection, response and organ dysfunction (PIRO): a pilot clinical staging system for hospital mortality in patients with infection. PLoS ONE. 2013;8(7):e70806.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  10. Charlson ME, Pompei P, Ales KL, et al. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis. 1987;40(5):373–83.

    Article  CAS  PubMed  Google Scholar 

  11. Ghorbani M, Ghaem H, Rezaianzadeh A, et al. A study on the efficacy of APACHE-IV for predicting mortality and length of stay in an intensive care unit in Iran. F1000Res. 2017;6:2032.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Pollard TJ, Johnson AEW, Raffa JD, et al. The eICU Collaborative Research Database, a freely available multi-center database for critical care research. Sci Data. 2018;5:180178.

    Article  PubMed  PubMed Central  Google Scholar 

  13. O’Halloran HM, Kwong K, Veldhoen RA, Maslove DM. Characterizing the patients, hospitals, and data quality of the eICU collaborative research database. Crit Care Med. 2020;48(12):1737–43.

    Article  PubMed  Google Scholar 

  14. Shankar-Hari M, Phillips GS, Levy ML, et al. Developing a new definition and assessing new clinical criteria for septic shock: for the third international consensus definitions for sepsis and septic shock (Sepsis-3). JAMA. 2016;315(8):775–87.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  15. Levy MM, Fink MP, Marshall JC, et al. 2001 SCCM/ESICM/ACCP/ATS/SIS international sepsis definitions conference. Crit Care Med. 2003;31(4):1250–6.

    Article  PubMed  Google Scholar 

  16. Harrell FE Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996;15(4):361–87.

    Article  PubMed  Google Scholar 

  17. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44(3):837–45.

    Article  CAS  PubMed  Google Scholar 

  18. Pencina MJ, D’Agostino RB Sr, D’Agostino RB, et al. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. 2008;27(2):157–72.

    Article  PubMed  Google Scholar 

  19. Serafim R, Gomes JA, Salluh J, et al. A comparison of the Quick-SOFA and systemic inflammatory response syndrome criteria for the diagnosis of sepsis and prediction of mortality: a systematic review and meta-analysis. Chest. 2018;153(3):646–55.

    Article  PubMed  Google Scholar 

  20. Seymour CW, Liu VX, Iwashyna TJ, et al. Assessment of clinical criteria for sepsis: for the third international consensus definitions for sepsis and septic shock (Sepsis-3). JAMA. 2016;315(8):762–74.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  21. Ford DW, Goodwin AJ, Simpson AN, et al. A severe sepsis mortality prediction model and score for use with administrative data. Crit Care Med. 2016;44(2):319–27.

    Article  PubMed  PubMed Central  Google Scholar 

  22. Liu Z, Meng Z, Li Y, et al. Prognostic accuracy of the serum lactate level, the SOFA score and the qSOFA score for mortality among adults with Sepsis. Scand J Trauma Resusc Emerg Med. 2019;27(1):51.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Wernly B, Mamandipoor B, Baldia P, et al. Machine learning predicts mortality in septic patients using only routinely available ABG variables: a multi-centre evaluation. Int J Med Inform. 2021;145:104312.

    Article  PubMed  Google Scholar 

  24. Raith EP, Udy AA, Bailey M, et al. Prognostic accuracy of the SOFA score, SIRS criteria, and qSOFA score for in-hospital mortality among adults with suspected infection admitted to the intensive care unit. JAMA. 2017;317(3):290–300.

    Article  PubMed  Google Scholar 

  25. Zhang XM, Zhang WW, Yu XZ, et al. Comparing the performance of SOFA, TPA combined with SOFA and APACHE-II for predicting ICU mortality in critically ill surgical patients: a secondary analysis. Clin Nutr. 2020;39(9):2902–9.

    Article  PubMed  Google Scholar 

  26. Gao X, Liu Y, Xu F, et al. Assessment of apoptosis inhibitor of macrophage/CD5L as a biomarker to predict mortality in the critically ill with sepsis. Chest. 2019;156(4):696–705.

    Article  PubMed  Google Scholar 

  27. Bennett CE, Wright RS, Jentzer J, et al. Severity of illness assessment with application of the APACHE IV predicted mortality and outcome trends analysis in an academic cardiac intensive care unit. J Crit Care. 2019;50:242–6.

    Article  PubMed  Google Scholar 

  28. Annane D. Body temperature in sepsis: a hot topic. Lancet Respir Med. 2018;6(3):162–3.

    Article  PubMed  Google Scholar 

  29. Wu DY, Lu SQ. The effects of abnormal body temperature on the prognosis of patients with septic shock. Ther Hypothermia Temp Manag. 2020;10(3):148–52.

    Article  PubMed  Google Scholar 

  30. Shen Y, Lou Y, Zhu S. Hyperthermia is a predictor of high mortality in patients with sepsis. Crit Care. 2020;24(1):543.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Schortgen F, Clabault K, Katsahian S, et al. Fever control using external cooling in septic shock: a randomized controlled trial. Am J Respir Crit Care Med. 2012;185(10):1088–95.

    Article  PubMed  Google Scholar 

  32. Rumbus Z, Matics R, Hegyi P, et al. Fever is associated with reduced, hypothermia with increased mortality in septic patients: a meta-analysis of clinical trials. PLoS ONE. 2017;12(1):e0170152.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  33. Shimazui T, Nakada TA, Walley KR, et al. Significance of body temperature in elderly patients with sepsis. Crit Care. 2020;24(1):387.

    Article  PubMed  PubMed Central  Google Scholar 

  34. Kushimoto S, Abe T, Ogura H, et al. Impact of body temperature abnormalities on the implementation of sepsis bundles and outcomes in patients with severe sepsis: a retrospective sub-analysis of the focused outcome research on emergency care for acute respiratory distress syndrome, sepsis and trauma study. Crit Care Med. 2019;47(5):691–9.

    Article  PubMed  Google Scholar 

  35. Walter EJ, Hanna-Jumma S, Carraretto M, et al. The pathophysiological basis and consequences of fever. Crit Care. 2016;20(1):200.

    Article  PubMed  PubMed Central  Google Scholar 

  36. Arbo JE, Lessing JK, Ford WJH, et al. Heart rate variability measures for prediction of severity of illness and poor outcome in ED patients with sepsis. Am J Emerg Med. 2020;38(12):2607–13.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Jarkovska D, Valesova L, Chvojka J, et al. Heart-rate variability depression in porcine peritonitis-induced sepsis without organ failure. Exp Biol Med (Maywood). 2017;242(9):1005–12.

    Article  CAS  Google Scholar 

  38. Barnaby DP, Fernando SM, Herry CL, et al. Heart rate variability, clinical and laboratory measures to predict future deterioration in patients presenting with sepsis. Shock. 2019;51(4):416–22.

    Article  PubMed  Google Scholar 

  39. von Bahr V, Hultman J, Eksborg S, et al. Long-term survival in adults treated with extracorporeal membrane oxygenation for respiratory failure and Sepsis. Crit Care Med. 2017;45(2):164–70.

    Article  Google Scholar 

  40. Sheu CC, Gong MN, Zhai R, et al. Clinical characteristics and outcomes of sepsis-related vs non-sepsis-related ARDS. Chest. 2010;138(3):559–67.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  41. Jensen JS, Peters L, Itenov TS, et al. Biomarker-assisted identification of sepsis-related acute liver impairment: a frequent and deadly condition in critically ill patients. Clin Chem Lab Med. 2019;57(9):1422–31.

    Article  CAS  PubMed  Google Scholar 

  42. Ryoo SM, Lee J, Lee YS, et al. Lactate level versus lactate clearance for predicting mortality in patients with septic shock defined by sepsis-3. Crit Care Med. 2018;46(6):e489–95.

    Article  CAS  PubMed  Google Scholar 

  43. Mikkelsen ME, Miltiades AN, Gaieski DF, et al. Serum lactate is associated with mortality in severe sepsis independent of organ failure and shock. Crit Care Med. 2009;37(5):1670–7.

    Article  CAS  PubMed  Google Scholar 

  44. Woolum JA, Abner EL, Kelly A, Thompson Bastin ML, Morris PE, Flannery AH. Effect of thiamine administration on lactate clearance and mortality in patients with septic shock. Crit Care Med. 2018;46(11):1747–52.

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgements

We thank the Dr. Cheng Lei for supporting this study (department of Neurosurgery, Xuanwu Hospital, China International Neuroscience Institute, Capital Medical University, Beijing, China). We also thank the intensivists, data managers, and other staff in the participating eICU Collaborative Research Database.

Funding

This study was supported by National Natural Science Foundation of China, No. 81772054.

Author information

Author notes

  1. Jie Weng and Ruonan Hou are equal contributors

Authors and Affiliations

  1. Department of General Practice, The Second Affiliated Hospital and Yuying Children’s Hospital of Wenzhou Medical University, Wenzhou, 325027, China

    Jie Weng, Ruonan Hou, Xiaoming Zhou, Zhiliang Zhou, Peng Wang & Zhiyi Wang

  2. Department of Emergency Intensive Care Unit, The Second Affiliated Hospital and Yuying Children’s Hospital of Wenzhou Medical University, Wenzhou, 325027, China

    Zhe Xu

  3. Department of Public Health, Robbins College of Health and Human Sciences, Baylor University, Waco, TX, USA

    Liang Wang

  4. Department of Geriatric Medicine, The First Affiliated Hospital, Wenzhou Medical University, Wenzhou, 325000, China

    Chan Chen

  5. Institute of Genomic Medicine, Wenzhou Medical University, Wenzhou, 325000, China

    Jinyu Wu

  6. Institute of Bioscaffold Transplantation and Immunology, School of Basic Medical Sciences, Wenzhou Medical University, Wenzhou, 325035, China

    Zhiyi Wang

  7. Center for Health Assessment, Wenzhou Medical University, Wenzhou, China

    Zhiyi Wang

Authors
  1. Jie Weng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ruonan Hou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaoming Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhe Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhiliang Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Peng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Liang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Chan Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Jinyu Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Zhiyi Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

JW, RH, JW and ZW designed the study and drafted the manuscript; XM and ZX helped interpret the results and write some discussion; ZZ, PW and LW helped in the statistical analysis and result interpretation; CC prepared the figures and interpret the results. JW and ZW are identified as the guarantors of the paper, taking responsibility for the integrity of the work as a whole, from inception to published article. The author(s) read and approved the final manuscript.

Corresponding authors

Correspondence to Jinyu Wu or Zhiyi Wang.

Ethics declarations

Ethics approval and consent to participate

The study was based on existing dataset and was approved by the Ethics Committee of the Second Affiliated Hospital and Yuying Children's Hospital of Wenzhou Medical University.

Consent for publication

Not applicable.

Competing interests

We declared no conflicts of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: Table S1

. The prognostic factors of ICU mortality in univariate analysis. Table S2. The prognostic factors of ICU mortality in multivariate logistic analysis. Table S3. The independent prognostic factors of ICU mortality. Table S4. Net benefit of using the POSMI score, APACHE IV and SOFA compared to managing sepsis assuming all of them will die during the ICU stay.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Weng, J., Hou, R., Zhou, X. et al. Development and validation of a score to predict mortality in ICU patients with sepsis: a multicenter retrospective study. J Transl Med 19, 322 (2021). https://doi.org/10.1186/s12967-021-03005-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12967-021-03005-y

Keyword

Journal of Translational Medicine

ISSN: 1479-5876

Contact us