INTRODUCTION

Health-care managers often seek changes of clinical care delivery to increase safety, improve clinical effectiveness, or comply with new laws or accreditation requirements. While practice change occurs within clinical settings, the responsibility for ensuring implementation often falls on higher levels of the health-care organization, where non-compliance may affect funding, accreditation, quality measures, hiring, and reputation.1 Therefore, improving the efficiency and effectiveness of organizational policy is of broad interest to managers of health systems.

Randomized policy evaluations offer a unique opportunity to rigorously examine the uptake and effectiveness of policy changes. The Veterans Health Administration (VHA) is an ideal system within which to conduct such evaluations because it is the largest health-care system in the U.S.A. and has centralized oversight. Although VHA routinely guides care through policy documents, policies are rarely tested with randomized trials, leaving policy leaders with little evidence-based guidance.2 Moreover, the U.S. Government Accountability Office has criticized VHA for its ambiguous policies and inconsistent processes.3 Thus, there is a need for randomized evaluations to inform how to improve VHA policy development.

In 2016, the Comprehensive Addiction and Recovery Act (CARA) required VHA to improve strategies in treating patients prescribed opioid analgesics and to ensure responsible prescribing (Subtitle A Sec 911). Fulfilling the requirements of CARA, the VHA Office of Mental Health and Suicide Prevention (OMHSP) developed the Stratification Tool for Opioid Risk Mitigation (STORM), a novel clinical decision support dashboard that facilitates implementation of clinical practice guideline–recommended interventions relevant to patients receiving opioid analgesics within VHA.4 The dashboard is based on the STORM risk model that estimates the risk that each patient receiving an opioid analgesic will experience a serious adverse event. Prior to CARA and STORM, VHA launched the Opioid Safety Initiative (OSI) in 2013, the first of several system-wide initiatives to address opioid overuse by reducing the use of opioid medications and improving the safety of opioid prescribing, while expanding alternative pain therapies. The VHA saw a reduction in the number of veterans dispensed an opioid each quarter by 25% between 2013 and 2016.5

VHA’s implementation of STORM provided a unique opportunity to examine the effectiveness of how policy mandates are communicated. In 2018, VHA developed a policy mandating that patients estimated to be at “very high” risk of an adverse event based on the STORM risk model receive a case review by an interdisciplinary team of clinicians with expertise spanning chronic pain, mental health and substance use disorders, pharmacy, and rehabilitation. This policy was promulgated in a randomized fashion such that half of VHA facilities (aka medical centers) had an additional oversight component (see Appendix C).6,7 We exploit this randomization to test whether patients in the oversight arm had a lower risk of opioid-related serious adverse events (SAEs) or death compared to those in the non-oversight arm. We hypothesized that patients cared for by VHA facilities in the oversight arm would achieve lower opioid-related SAEs or all-cause mortality. Findings from this study could illuminate how to write more effective policy mandates to improve patient care.

SETTING

STORM

The STORM dashboard prioritizes and prominently displays very-high-risk patients, which include those having a STORM risk score ≥ 0.166. This value corresponds to the lower bound of the top 1st percentile of risk scores of VHA patients analyzed in 2010.4 The dashboard, updated daily, includes patients with active opioid prescriptions. Likewise, risk scores are updated daily to reflect changes in contributing factors.

VHA Policy Notices

On April 17, 2018, the VHA released policy notices describing the STORM dashboard and the significance of deploying risk mitigation strategies and mandating case reviews (Appendix A-B). Two notices were released, each to a randomly selected half of VHA facilities, with differing language. Each facility point of contact received an email with a link to their version of the policy notice.

The policy notices specified that in the first quarter of fiscal year 2019 (October 1, 2018–December 31, 2018), at least 97% of patients displayed in the STORM dashboard as very high risk be case reviewed if they have not been in the past 12 months. One version of the policy notice included an extra paragraph stating that facilities not meeting the 97% target in the first quarter of fiscal year 2019 will receive technical assistance and have to submit an action plan focused on improving their case review rate. The facility point(s) of contact must report to the VHA OMHSP quarterly to update their progress to meet the target. This oversight paragraph was designed to increase adherence to the new policy, encourage providers to conduct more case reviews, and reduce negative health outcomes.7

Case reviews included assessing and recording patients’ estimated risk of adverse events (including overdose and suicide risks), events that led to the opioid prescription, appropriateness of prescriptions, use of risk mitigation strategies, need for increased monitoring or support for treatment adherence or recovery, need for prompt outreach or receipt of referrals for additional services, and attention to mental health needs.

PROGRAM DESCRIPTION

Randomization

One hundred and forty VHA facilities were randomly assigned using permuted block randomization (n = 70 treatment and n = 70 control). Patients’ risk scores were not displayed on the STORM dashboard, and providers were blinded to the risk score threshold that defined “very high risk.” A protocol describing the study design and analysis plan has been published7 and was registered in ISRCTN (http://isrctn.com/ISRCTN16012111).

This study was approved by VA Boston Healthcare System IRB & R&D Committees (Protocol #3069). An additional level of randomization which was part of the research design (though not relevant to the analysis in this paper) is explained briefly in Appendix D.

Cohort

The study cohort was VHA patients who were prescribed opioids and classified as very high risk in the STORM dashboard between April 18, 2018, and November 8, 2019. The final cohort size was 16,272 patients. Of these, 8734 patients were in the oversight (i.e., treatment) arm and 7538 were in the non-oversight (i.e., control) arm. The unit of analysis was at the person-month, and a patient entered the study on the month they first appeared on the STORM dashboard.

Each patient in the cohort was assigned to a VHA facility based on where they received the majority of their primary care and opioid prescriptions. Patient characteristics, baseline comorbidities, and health outcomes were collected from VHA databases.

Outcomes

The primary outcome of interest was opioid-related SAEs, such as opioid-related overdose or falls, defined using ICD10 codes (Appendix E). We also included death as a secondary outcome. In addition, case reviews were intended to reduce the risk of adverse events. Therefore, we tested the completion of case reviews, as documented in the VHA electronic medical record, as an intermediate outcome. Our definition of case review follows that of the VHA medical record. We created binary indicators for each person, where 1 indicated the first SAE, death, or first case review completion.

Analysis

Successful randomization was evaluated by the balance of patient characteristics and baseline STORM variables between the oversight and non-oversight arms. For patient characteristics, we compared age, sex, race, and prevalence of Elixhauser comorbidities. We also compared baseline STORM risk score on study entry date and the proportion of patients who had at least one case review prior to entering the study. We used standardized differences to check balance, for which > 10% indicates a meaningful difference.8 Patients were followed for a minimum of 120 days (4 months) and up to 690 days (23 months). For the primary outcome, opioid-related SAEs, patients were followed until their first incident or censored at death or end of study. We explored the difference in case review, opioid-related SAEs, and all-cause mortality between oversight and non-oversight using Cox proportional hazard models.

PROGRAM EVALUATION

Patient characteristics including age, sex, race, and STORM–related variables such as baseline STORM risk scores and case review rate between the oversight and non-oversight arms were compared using standardized difference (Table 1 and Appendix F). In both groups, the majority of patients were white males with an average age of 56 years old. The non-oversight arm included a greater proportion of patients with Hispanic ethnicity than the oversight arm (9.3%, 5.1%, respectively). Rates of Elixhauser comorbidities from the baseline period (1 year prior to study entry) were balanced between patients in the oversight and non-oversight arms. Nearly all (90%) of the cohort was diagnosed with depression, and over 60% of patients were diagnosed with alcohol abuse, drug abuse, and/or uncomplicated hypertension. On average, the patients’ baseline STORM risk score was 0.26, and more than 6% of patients had a case review prior to entering into the study.

Table 1 Baseline Patient Characteristic Means (SDs)
Full size table

On average, patients were in the study for 13 months, 57.0% of patients were case reviewed during the study period, 34.9% of patients experienced any SAE, and 3.8% of patients died during the study period (results not shown). Between April 18, 2018 (FY18Q3) and November 8, 2019 (FY20Q1), the number of very-high-risk patients entering the study declined over time (Fig. 1). Using Cox proportion hazard analysis, we did not observe evidence of an effect of the presence of the oversight and accountability language in the policy notice on SAEs or death (Table 2). Patients in the oversight arm were significantly less likely to receive case reviews than patients in the non-oversight arm (HR 0.91, 95% CI 0.87–0.95).

Fig. 1
figure 1

Number of patients entering the study each fiscal quarter.

Full size image
Table 2 Main Results of the Cox Proportion Hazard Model for Case Review, Any SAE, and All-Cause Mortality (N = 16,272)
Full size table

DISCUSSION

Our analysis found that patients in the VHA facility in the non-oversight arm were significantly more likely to receive case review than patients in VHA facilities in the oversight arm. Rogal and colleagues analyzed the case review completion rate at the VHA facility level (our analysis was at the patient level) and found that between the non-oversight and oversight arms it did not significantly differ.9 However, their study did find that VHA facilities in the non-oversight arm were significantly more likely to meet the 97% target (specified in the policy) than those in the oversight arm (30% vs. 11%, respectively), explained by personnel (i.e., age and tenure) and strategy differences among VHA facilities.

One possible explanation of our findings is managers at facilities in the oversight arm felt less directly responsible for outcomes.10 The VHA is a dynamic environment where health-care administrators and quality improvement specialists receive frequent mandates for new program implementation, quality improvement, and performance goals from accrediting bodies, oversight agencies, and both regional and national offices. In fact, STORM was released a few years after OSI. The VHA’s effort to enhance opioid safety over multiple initiatives may have caused confusion and fatigue among VHA providers. Also, by indicating that the national office would initiate oversight and support activities if implementation targets were not met, the policy may have reduced perceived local responsibility for achieving goals.

The lack of correspondence between case review rates and outcomes could be explained by the fact that the difference in case review rates between the oversight and non-oversight arms may have been too small to produce any discernably different impact on outcomes. Alternatively, sites that took more responsibility and implemented without oversight may have utilized a less effective case review program than those that took more time and waited for VHA to intervene. As a result, the non-oversight facilities may have been more likely to do case reviews, but each case review was less effective. It is also possible that oversight actions themselves may not have been effective in translating case reviews to outcomes. Future research is needed to unpack some of these potential explanations.

Despite the unexpected findings in our study, the STORM policy overall may have been successful in focusing the provider’s attention on very-high-risk patients. Approximately 60% of patients received a case review (compared to 6.6% during the baseline period), and the number of very-high-risk patients entering the study declined over time, likely the result of the overall VHA’s effort to increase opioid safety. Consistent with this hypothesis, another study observed a strong association between patients with a case review having a lower risk of experiencing negative health outcomes.11 Future studies could focus on policy implementation strategies and other system-level characteristics to explore the contributing factors between contrasting policy language and SAEs or mortality.