Revalidating the prognostic COVID-19 severity assessment (COSA) score for variants of concern

Dear Editor,

Many prediction models have been published for Coronavirus disease 2019 (COVID-19) to support clinical decision-making for either diagnosis, prediction of mortality risk, or disease progression. An ongoing review concluded that most of these were poorly reported and had a high risk of bias, casting doubt over their real-world predictive value [1]. Additionally, dominant strains are no longer the ancestral type but variants of concern (VOCs) with different pathogenicity, and vaccinations and herd immunity can influence individual outcomes. We aimed to assess the performance of our previously published COSA (COVID-19 Severity Assessment) score on admitted patients in the twelve months following its development, a time during which the score was used clinically for risk stratification.

The COSA score was developed using data from patients who tested positive for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) by reverse-transcriptase polymerase chain reaction (RT-PCR) between February 1st and November 16th, 2020 (1st and 2nd waves in Switzerland, i.e. the original cohort) [2]. The revalidation cohort consisted of patients who tested positive for SARS-CoV-2 between November 17th, 2020 and November 16th, 2021. The primary outcome was disease severity, as determined by the worst outcome at any point after diagnosis:

• Non-severe outcome: no intensive care unit (ICU) admission or death of any cause during the observational period

• Severe outcome: ICU admission at any stage and / or death of any cause

All patients were discharged or had died by the time the revalidation was performed.

The COSA score (Table 1) was calculated for each patient using the most extreme laboratory values within 3 days prior to 1 day after the positive SARS-CoV-2 test. The results of the original and the revalidation cohort were then compared.

Table 1 COVID-19 severity assessment (COSA) score (from Ref. [2])

Data wrangling, analysis, and visualization were performed in GNU R (version 4.0.2, [3]). Statistical significance levels were defined at a p-value of < 0.05, and determined with the Wilcoxon rank sum test for continuous variables, and the Chi-square test for categorical variables using the stats package (version 4.0.2). The area under the receiver operating characteristic curve (AUROC) and the 95% confidence intervals (CI) were calculated to assess the discriminatory power of the COSA score to predict a severe outcome. The COSA score was considered validated if the 95% CI of the AUROC from fitting the original and the revalidation data set were overlapping.

The original cohort consisted of 626 patients (457 non-severe and 169 severe outcomes), and 508 patients were included in the revalidation cohort (301 non-severe and 208 severe outcomes). A comparison of the demographics and laboratory parameters of both cohorts is provided in Table 2. The patients in the original non-severe cohort were significantly younger with a lower share of inpatients than in the non-severe revalidation cohort. The latter had significantly higher peak C-reactive protein (CRP) and glucose, and lower minimal hemoglobin and estimated glomerular filtration rate values. For severe patients, we noted a higher proportion of inpatients and deaths, and significantly lower body weights and body mass index in the revalidation cohort. There was no significant difference in laboratory parameters.

Table 2 General demographics and laboratory parameters of the original and revalidation cohort

The AUROC was 0.85 (95% CI 0.82–0.88) and 0.80 (95% CI 0.76–0.84) for the original and revalidation cohort, respectively (Fig. 1). A closer look at the severity distribution per score value (Fig. 2) revealed a greater share of severe cases with low score values (0–4 points) in the revalidation cohort than in the original cohort. No major differences are noticeable for score points greater than 4.

Fig. 1

Area under the receiver operating characteristic (AUROC) of the COVID-19 severity assessment (COSA) score. Original (red) and revalidation cohort (blue)

Fig. 2

Percentage amount of patients with severe and non-severe COVID-19 in relation to score points in the original (A) and revalidation (B) cohort

Patients in the revalidation cohort presented with overall worse laboratory markers and a greater likelihood of severe outcomes compared to the original cohort (69.1% vs. 37.4%). This could be due to more selective laboratory testing for hospitalized patients only and differences in circulating variants. Although no sequencing data is available for either cohort, national surveys indicate that the dominant strain during the 1st and 2nd waves (original cohort) was the ancestral type, whereas the 3rd and 4th waves (revalidation cohort) were driven by the more transmissible and virulent VOCs Alpha (B.1.1.7) and Delta (B.1.617.2) [4]. Vaccination campaigns and an emerging population-level immunity likely mitigated the individual disease severity [5].

While the performance of the score remains robust with strong positive discriminative ability we did note a decrease in specificity towards the lower end of the scale (Fig. 2). This suggests that readjustment of the cut-offs might be beneficial for a better separation of the severity classes as new variants emerge.

Despite changes in the viral landscape and population immunity, the COSA score still delivered good predictions of disease progression one year after its development. We attribute this to the simple set of covariates and the rigorous internal and external validation in the original model-building process. Adaptations to the score could become necessary in the near future as vaccination effects begin to wane and if drastically different VOCs appear.