Abstract
There is a need to discriminate which COVID-19 inpatients are at higher risk for venous thromboembolism (VTE) to inform prophylaxis strategies. The IMPROVE-DD VTE risk assessment model (RAM) has previously demonstrated good discrimination in non-COVID populations. We aimed to externally validate the IMPROVE-DD VTE RAM in medical patients hospitalized with COVID-19. This retrospective cohort study evaluated the IMPROVE-DD VTE RAM in adult patients with COVID-19 admitted to one of thirteen Northwell Health hospitals in the New York metropolitan area between March 1, 2020 and April 27, 2020. VTE was defined as new-onset symptomatic deep venous thrombosis or pulmonary embolism. To assess the predictive value of the RAM, the receiver operating characteristic (ROC) curve was plotted and the area under the curve (AUC) was calculated. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated. Of 9407 patients who met study criteria, 274 patients developed VTE with a prevalence of 2.91%. The VTE rate was 0.41% for IMPROVE-DD score 0–1 (low risk), 1.21% for score 2–3 (moderate risk), and 5.30% for score ≥ 4 (high risk). Approximately 45.7% of patients were classified as high VTE risk, 33.3% moderate risk, and 21.0% low risk. Discrimination of low versus moderate-high VTE risk demonstrated sensitivity 0.971, specificity 0.215, PPV 0.036, and NPV 0.996. ROC AUC was 0.703. In this external validation study, the IMPROVE-DD VTE RAM demonstrated very good discrimination to identify hospitalized COVID-19 patients at low, moderate, and high VTE risk.
Highlights
-
In this external validation study, the IMPROVE-DD VTE RAM demonstrated a very good degree of discrimination to identify hospitalized COVID-19 patients at low, moderate, and high risk of developing VTE.
-
Using the IMPROVE-DD VTE RAM in hospitalized COVID-19 patients, 45.7% of patients were classified as high VTE risk, 33.3% moderate risk, and 21.0% low risk.
-
This validation of the IMPROVE-DD VTE RAM in COVID-19 inpatients will assist healthcare providers in individualizing thromboprophylaxis strategies based on VTE risk category, and thus, has the potential to decrease VTE-associated morbidity and mortality in this medically ill population.
Introduction
Venous thromboembolism (VTE), including deep vein thrombosis (DVT) and pulmonary embolism (PE), is very common in acutely ill medical patients with coronavirus disease-2019 (COVID-19), especially in patients with elevated D-dimer (Dd). Early data in critically ill patients suggested extraordinarily high rates of VTE [1]. More recent data in medical ward patients show VTE rates up to 6.2% [2], which is still substantially higher than historical rates of VTE in hospitalized non-COVID medical populations. Indeed, VTE risk in hospitalized COVID-19 patients remains a serious concern [2, 3]. Antithrombotic guidelines recommend a universal thromboprophylaxis strategy for COVID-19 inpatients, and suggest intermediate-dose anticoagulation may be beneficial in subsets of high-risk patients [4, 5]. Randomized trials assessing escalated or treatment dose anticoagulants for thromboprophylaxis are ongoing [6]. Effective stratification of patients and identification of those at high risk of VTE remains challenging.
The International Medical Prevention Registry on Venous Thromboembolism (IMPROVE) risk assessment model (RAM) has been extensively validated and shown excellent discrimination between low- and at-risk hospitalized medical patients [7]. The IMPROVE VTE RAM assigns 1 to 3 points to seven VTE risk factors: age > 60, previous VTE, known thrombophilia, current lower-limb paralysis, active cancer, immobilization, and intensive care (ICU)/cardiac care unit (CCU) stay. An 8-factor model (IMPROVE-DD) incorporating Dd > 2 times the upper limit of normal (ULN) enhances area under the curve (AUC) for the receiver operating characteristic curve (ROC) by 0.06 [8]. Noting the association of elevated Dd with increased risk of thrombosis and poor outcomes in our COVID-19 inpatient population [8, 9] we sought to externally validate the IMPROVE-DD VTE RAM in a hospitalized, COVID-19 medical population.
Methods
This retrospective cohort study included medical patients aged 18 years or older who were admitted to 1 of 13 Northwell Health hospitals in the New York metropolitan area between March 1, 2020 and April 27, 2020, and who were confirmed to be COVID-19 positive by polymerase chain reaction. Patients were excluded if they were on the obstetrics service, experienced a VTE event within 8 h of admission, experienced a length of stay less than 8 h, or if key variables were missing. Data and outcomes were extracted from Sunrise Clinical Manager electronic health record (EHR) (Allscripts, Chicago, IL) and tracked until April 30, 2020. This study was approved by the Institutional Review Board of Northwell Health, and the need for consent of individual patients was waived.
VTE was defined as new-onset symptomatic DVT or PE, as diagnosed by imaging studies performed by the Radiology Department or by point-of-care lower extremity ultrasound. Events were manually adjudicated by two attending radiologists.
Points were assigned to patient characteristics corresponding to the variables of the IMPROVE-DD RAM [8]. Previous VTE, thrombophilia, and cancer were extracted from corresponding ICD-9 or ICD-10 codes. In light of standard room isolation precautions for COVID-19, all patients were considered relatively immobile and assigned 1 point for this factor. For the risk factor of ICU stay, a proxy of admission to a named ICU, use of mechanical ventilation, or administration of vasopressors was used. Considering common use of paralytic agents in our system, mechanical ventilation was also used as a proxy for lower limb paralysis. For Dd value, 2 points were assigned if the maximum Dd value throughout the hospitalization or prior to VTE event was ≥ 2 times the upper limit of normal per local laboratory.
To assess the predictive value of the RAM, the ROC curve was plotted, and AUC was calculated. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated using standard methods, using the observed VTE prevalence in the calculation of PPV and NPV. Analyses were performed with SAS version 9.4 (SAS institute, Cary, North Carolina).
To assess the predictive value of the RAM, the ROC curve was plotted, and area under the curve (AUC) was calculated. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated using standard methods, using the observed VTE prevalence in the calculation of PPV and NPV. Further exploratory analyses were done comparing the AUC for both the IMPROVE-DD RAM and the original 7-factor IMPROVE RAM without D-dimers [10]. The curves were compared using the method of DeLong, DeLong and Clarke-Pearson [11]. Upon finding a significant difference, a comparison using a Bonferroni adjustment was used, such that p < 0.0083 was considered significant. Analyses were performed with SAS version 9.4 (SAS institute, Cary, North Carolina).
Results
Of 9407 patients that met study criteria, 274 patients developed VTE, with a prevalence of 2.91%. The VTE rate was 0.41% for IMPROVE-DD score 0–1 (low risk), 1.21% for score 2–3 (moderate risk), and 5.30% for score ≥ 4 (high risk) (Table 1). Approximately 45.7% of patients were classified as high VTE risk, 33.3% as moderate risk, and 21.0% as low risk.
The IMPROVE-DD RAM discrimination of low versus moderate-high VTE risk demonstrated a sensitivity of 0.971, specificity of 0.215, PPV of 0.036, and NPV of 0.996. The AUC of the ROC was calculated to be 0.703 (Fig. 1). The IMPROVE RAM without D-dimers discrimination of low versus moderate-high VTE risk demonstrated a sensitivity of 0.839, specificity of 0.292, PPV of 0.034, and NPV of 0.984. The AUC of the ROC was calculated to be 0.635. The 6.8% difference of the AUC (delta AUC) comparing the IMPROVE-DD and IMPROVE RAMs was significant (p < 0.0001).
Discussion
With an AUC of ROC of 0.703, the IMPROVE-DD VTE RAM demonstrated very good model discrimination and negative predictive value to predict the risk of VTE in this cohort of COVID-19 inpatients, similar to the discrimination seen in prior validations of the original IMPROVE and IMPROVE-DD RAMs in non-COVID cohorts [7]. In comparison with the original 7-factor IMPROVE RAM, addition of elevated D-dimers with the IMPROVE-DD RAM improved model discrimination of VTE in COVID-19 inpatients by 6.8%, a difference that was statistically significant. The IMPROVE-DD was also recently validated in the hospitalized COVID-19 population, which achieved an AUC of 0.702, though the study was limited by the absence of thrombophilia data [12]. In comparison, our cohort included 123 (1.31%) patients with thrombophilia and achieved similar AUC. This confirms the expectation of good model discrimination with the addition of risk factors that are strongly weighted, yet relatively uncommon, such as inherited thrombophilia.
As additional data on thromboprophylaxis in COVID-19 continues to surface, international guidelines suggest a need for a risk-adapted approach to thromboprophylaxis [3,4,5]. Notably, while 21.0% of our study population was identified as low VTE risk, nearly half (45.5%) of patients were identified as high VTE risk. These patients represent a subgroup of hospitalized COVID-19 patients who may potentially benefit from increased intensity thromboprophylaxis [3]. Though this approach is not currently supported by guidelines, high quality randomized trials data is forthcoming [6]. Conversely, our data suggests potential harm from a universal thromboprophylaxis strategy in approximately 20% of COVID-19 inpatients identified as low VTE risk by our validation study.
As a retrospective study, our validation has several limitations. Although VTE events were adjudicated by two attending radiologists, the VTE rate may be underreported as a result of avoidance of confirmatory studies due to concerns of virus exposure. We relied on surrogate markers for lower limb paralysis and immobility. In addition, the highest Dd value during hospital stay was used, which may alter the accuracy of the RAM when used to stratify risk on admission.
Conclusion
This large, external validation study of the IMPROVE-DD VTE RAM demonstrated very good discrimination to identify hospitalized COVID-19 patients at low, moderate, or high risk of VTE. The use of this model led to the stratification of 45.7% of patients as high VTE risk, 33.3% moderate risk, and 21.0% low risk, which can be used to create individualized, risk-adapted strategies for VTE thromboprophylaxis in hospitalized patients with COVID-19.
Data availability
The data that support the findings of this study are available on request from http://COVID19@northwell.edu. The data are not publicly available due to restrictions as it could compromise the privacy of research participants. This was performed at Northwell Health.
References
Tang N, Bai H, Chen X, Gong J, Li D, Sun Z (2020) Anticoagulant treatment is associated with decreased mortality in severe coronavirus disease 2019 patients with coagulopathy. J Thrombos Haemost 18(5):1094–1099
Bilaloglu S, Aphinyanaphongs Y, Jones S, Iturrate E, Hochman J, Berger JS (2020) Thrombosis in hospitalized patients with COVID-19 in a New York City Health System. JAMA 324(8):799–801. https://doi.org/10.1001/jama.2020.13372
Al-Samkari H, Gupta S, Leaf RK, Wang W, Rosovsky RP, Brenner SK, Hayek SS, Berlin H, Kapoor R, Shaefi S, Melamed ML, Sutherland A, Radbel J, Green A, Garibaldi BT, Srivastava A, Leonberg-Yoo A, Shehata AM, Flythe JE, Rashidi A, Goyal N, Chan L, Mathews KS, Hedayati SS, Dy R, Toth-Manikowski SM, Zhang J, Mallappallil M, Redfern RE, Bansal AD, Short SAP, Vangel MG, Admon AJ, Semler MW, Bauer KA, Hernan MA, Leaf DE, S-C Investigators (2021) Thrombosis, bleeding, and the observational effect of early therapeutic anticoagulation on survival in critically ill patients with COVID-19. Ann Intern Med. https://doi.org/10.7326/M20-6739
Cuker A, Tseng EK, Nieuwlaat R, Angchaisuksiri P, Blair C, Dane K, Davila J, DeSancho MT, Diuguid D, Griffin DO, Kahn SR, Klok FA, Lee AI, Neumann I, Pai A, Pai M, Righini M, Sanfilippo KM, Siegal D, Skara M, Touri K, Akl EA, Bou Akl I, Boulos M, Brignardello-Petersen R, Charide R, Chan M, Dearness K, Darzi AJ, Kolb P, Colunga-Lozano LE, Mansour R, Morgano GP, Morsi RZ, Noori A, Piggott T, Qiu Y, Roldan Y, Schünemann F, Stevens A, Solo K, Ventresca M, Wiercioch W, Mustafa RA, Schünemann HJ (2021) American Society of Hematology 2021 guidelines on the use of anticoagulation for thromboprophylaxis in patients with COVID-19. Blood Adv. 5(3):872–888. https://doi.org/10.1182/bloodadvances.2020003763
Spyropoulos AC, Levy JH, Ageno W, Connors JM, Hunt BJ, Iba T, Levi M, Samama CM, Thachil J, Giannis D, Douketis JD, Subcommittee on Perioperative CCTHotSSCotISoT, Haemostasis (2020) Scientific and Standardization Committee communication: Clinical guidance on the diagnosis, prevention, and treatment of venous thromboembolism in hospitalized patients with COVID-19. J Thromb Haemost 18(8):1859–1865. https://doi.org/10.1111/jth.14929
Full dose blood thinners decreased need for life support and improved outcome in hospitalized COVID-19 patients (2021) [Internet]. Available from: https://www.nih.gov/news-events/news-releases/full-dose-blood-thinners-decreased-need-life-support-improved-outcome-hospitalized-covid-19-patients. Accessed 9 Mar 2021
Rosenberg D, Eichorn A, Alarcon M, McCullagh L, McGinn T, Spyropoulos AC (2014) External validation of the risk assessment model of the International Medical Prevention Registry on Venous Thromboembolism (IMPROVE) for medical patients in a tertiary health system. J Am Heart Assoc 3(6):e001152. https://doi.org/10.1161/JAHA.114.001152
Gibson CM, Spyropoulos AC, Cohen AT, Hull RD, Goldhaber SZ, Yusen RD, Hernandez AF, Korjian S, Daaboul Y, Gold A, Harrington RA, Chi G (2017) The IMPROVEDD VTE risk score: incorporation of D-dimer into the IMPROVE Score to improve venous thromboembolism risk stratification. TH Open 1(1):e56–e65. https://doi.org/10.1055/s-0037-1603929
Cohen SL, Gianos E, Barish MA, Chatterjee S, Kohn N, Lesser M, Giannis D, Coppa K, Hirsch J, McGinn T, Goldin M, Spyropoulos A (2021) Prevalence and predictors of venous thromboembolism or mortality in hospitalized COVID-19 patients. Thromb Haemost. https://doi.org/10.1055/a-1366-9656
Spyropoulos AC, Anderson FA Jr, FitzGerald G, Decousus H, Pini M, Chong BH, Zotz RB, Bergmann JF, Tapson V, Froehlich JB, Monreal M, Merli GJ, Pavanello R, Turpie AGG, Nakamura M, Piovella F, Kakkar AK, Spencer FA, I Investigators (2011) Predictive and associative models to identify hospitalized medical patients at risk for VTE. Chest 140(3):706–714. https://doi.org/10.1378/chest.10-1944
DeLong ER, DeLong DM, Clarke-Pearson DL (1988) Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44(3):837–845
Spyropoulos ACCS, Gianos E, Kohn N, Giannis D, Chatterjee S, Goldin M, Lesser M, Coppa K, Hirsch JS, McGinn TG, Barish MA on behalf of the COVID-19 Consortium Group (2021) Validation of the IMPROVE-DD risk assessment model for venous thromboembolism among hospitalized patients with COVID-19. Res Pract Thromb Haemost 5:1–5
Funding
This study was partially funded by the Broxmeyer Fellowship in Clinical Thrombosis, as well as awards from the National Institutes of Health (5R18HS026196-03; 3R18HS026196-02S1). SLC received funding support from the Association of University Radiologists GE Radiology Research Academy Fellowship (GERRAF) and Siemens Healthinners and was a consultant for Infervision 2019.
Author information
Authors and Affiliations
Contributions
Authors contributed to the study conception and design. NK and MQ contributed to data collection and analysis. The first draft of the manuscript was written by MG, SL, and AS. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Conflict of interest
ACS is a consultant for Boehringer Ingelheim, Janssen, Bayer, Bristol Myers Squibb, and Portola. No other authors have stated conflicts of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Goldin, M., Lin, S.K., Kohn, N. et al. External validation of the IMPROVE-DD risk assessment model for venous thromboembolism among inpatients with COVID-19. J Thromb Thrombolysis 52, 1032–1035 (2021). https://doi.org/10.1007/s11239-021-02504-5
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11239-021-02504-5