Skip to main content

Revalidating the prognostic COVID-19 severity assessment (COSA) score for variants of concern

Dear Editor,

Many prediction models have been published for Coronavirus disease 2019 (COVID-19) to support clinical decision-making for either diagnosis, prediction of mortality risk, or disease progression. An ongoing review concluded that most of these were poorly reported and had a high risk of bias, casting doubt over their real-world predictive value [1]. Additionally, dominant strains are no longer the ancestral type but variants of concern (VOCs) with different pathogenicity, and vaccinations and herd immunity can influence individual outcomes. We aimed to assess the performance of our previously published COSA (COVID-19 Severity Assessment) score on admitted patients in the twelve months following its development, a time during which the score was used clinically for risk stratification.

The COSA score was developed using data from patients who tested positive for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) by reverse-transcriptase polymerase chain reaction (RT-PCR) between February 1st and November 16th, 2020 (1st and 2nd waves in Switzerland, i.e. the original cohort) [2]. The revalidation cohort consisted of patients who tested positive for SARS-CoV-2 between November 17th, 2020 and November 16th, 2021. The primary outcome was disease severity, as determined by the worst outcome at any point after diagnosis:

  • Non-severe outcome: no intensive care unit (ICU) admission or death of any cause during the observational period

  • Severe outcome: ICU admission at any stage and / or death of any cause

All patients were discharged or had died by the time the revalidation was performed.

The COSA score (Table 1) was calculated for each patient using the most extreme laboratory values within 3 days prior to 1 day after the positive SARS-CoV-2 test. The results of the original and the revalidation cohort were then compared.

Table 1 COVID-19 severity assessment (COSA) score (from Ref. [2])

Data wrangling, analysis, and visualization were performed in GNU R (version 4.0.2, [3]). Statistical significance levels were defined at a p-value of < 0.05, and determined with the Wilcoxon rank sum test for continuous variables, and the Chi-square test for categorical variables using the stats package (version 4.0.2). The area under the receiver operating characteristic curve (AUROC) and the 95% confidence intervals (CI) were calculated to assess the discriminatory power of the COSA score to predict a severe outcome. The COSA score was considered validated if the 95% CI of the AUROC from fitting the original and the revalidation data set were overlapping.

The original cohort consisted of 626 patients (457 non-severe and 169 severe outcomes), and 508 patients were included in the revalidation cohort (301 non-severe and 208 severe outcomes). A comparison of the demographics and laboratory parameters of both cohorts is provided in Table 2. The patients in the original non-severe cohort were significantly younger with a lower share of inpatients than in the non-severe revalidation cohort. The latter had significantly higher peak C-reactive protein (CRP) and glucose, and lower minimal hemoglobin and estimated glomerular filtration rate values. For severe patients, we noted a higher proportion of inpatients and deaths, and significantly lower body weights and body mass index in the revalidation cohort. There was no significant difference in laboratory parameters.

Table 2 General demographics and laboratory parameters of the original and revalidation cohort

The AUROC was 0.85 (95% CI 0.82–0.88) and 0.80 (95% CI 0.76–0.84) for the original and revalidation cohort, respectively (Fig. 1). A closer look at the severity distribution per score value (Fig. 2) revealed a greater share of severe cases with low score values (0–4 points) in the revalidation cohort than in the original cohort. No major differences are noticeable for score points greater than 4.

Fig. 1
figure 1

Area under the receiver operating characteristic (AUROC) of the COVID-19 severity assessment (COSA) score. Original (red) and revalidation cohort (blue)

Fig. 2
figure 2

Percentage amount of patients with severe and non-severe COVID-19 in relation to score points in the original (A) and revalidation (B) cohort

Patients in the revalidation cohort presented with overall worse laboratory markers and a greater likelihood of severe outcomes compared to the original cohort (69.1% vs. 37.4%). This could be due to more selective laboratory testing for hospitalized patients only and differences in circulating variants. Although no sequencing data is available for either cohort, national surveys indicate that the dominant strain during the 1st and 2nd waves (original cohort) was the ancestral type, whereas the 3rd and 4th waves (revalidation cohort) were driven by the more transmissible and virulent VOCs Alpha (B.1.1.7) and Delta (B.1.617.2) [4]. Vaccination campaigns and an emerging population-level immunity likely mitigated the individual disease severity [5].

While the performance of the score remains robust with strong positive discriminative ability we did note a decrease in specificity towards the lower end of the scale (Fig. 2). This suggests that readjustment of the cut-offs might be beneficial for a better separation of the severity classes as new variants emerge.

Despite changes in the viral landscape and population immunity, the COSA score still delivered good predictions of disease progression one year after its development. We attribute this to the simple set of covariates and the rigorous internal and external validation in the original model-building process. Adaptations to the score could become necessary in the near future as vaccination effects begin to wane and if drastically different VOCs appear.

Availability of data and materials

The source code and corresponding input file are available on GitHub: https://github.com/cptbern/COSAscore. The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

AUROC:

Area under the receiver operating characteristic

BMI:

Body mass index

COSA:

COVID-19 severity assessment

COVID-19:

Coronavirus disease 19

CRP:

C-reactive protein

eGFR:

Estimated glomerular filtration rate

ICU:

Intensive Care Unit

RT-PCR:

Reverse-transcriptase polymerase chain reaction

SARS-CoV-2:

Severe acute respiratory syndrome coronavirus 2

VOC:

Variants of concern

References

  1. Wynants L, Van Calster B, Collins GS, Riley RD, Heinze G, Schuit E, Bonten MMJ, Dahly DL, Damen JA, Debray TPA, et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ. 2020;369: m1328.

    Article  Google Scholar 

  2. Schöning V, Liakoni E, Baumgartner C, Exadaktylos AK, Hautz WE, Atkinson A, Hammann F. Development and validation of a prognostic COVID-19 severity assessment (COSA) score and machine learning models for patient triage at a tertiary hospital. J Trans Med. 2021;19:56.

    Article  Google Scholar 

  3. R Core Team. R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing; 2022.

    Google Scholar 

  4. Covid-19 Schweiz: Epidemiologischer Verlauf, Schweiz und Liechtenstein, Virusvarianten. https://www.covid19.admin.ch/de/epidemiologic/virus-variants?time=total&variants=VariantB11529,VariantB16172,VariantP1,VariantB1351,VariantB117,VariantC37,VariantB16171,VariantP2,VariantB1525,VariantB1526. Accessed 5 July 2022.

  5. Zeng B, Gao L, Zhou Q, Yu K, Sun F. Effectiveness of COVID-19 vaccines against SARS-CoV-2 variants of concern: a systematic review and meta-analysis. BMC Med. 2022;20:200.

    CAS  Article  Google Scholar 

Download references

Acknowledgements

We thank Noel Frey, Myoori Wijayasingham, and the Insel Data Coordination Lab for database and infrastructure support.

Funding

None to declare.

Author information

Authors and Affiliations

Authors

Contributions

FH and VS conceptualized this study. VS and FH performed the data analysis and drafted the manuscript. AA assessed the machine learning models and statistics, EL, CB, AE, and WH assessed the clinical score. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Felix Hammann.

Ethics declarations

Ethics approval and consent to participate

The study was approved by the Cantonal Ethics Committee of Bern (Project-ID 2020–00973). Participants either agreed to a general research consent or, for participants with no registered general research consent status (neither agreement nor rejection), a waiver of consent was granted by the ethics committee.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Schöning, V., Liakoni, E., Baumgartner, C. et al. Revalidating the prognostic COVID-19 severity assessment (COSA) score for variants of concern. J Transl Med 20, 427 (2022). https://doi.org/10.1186/s12967-022-03634-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12967-022-03634-x