The efficiency of COVID cases to COVID policies: a robust conditional approach

Abstract

This paper examines the efficiency of four major COVID-19 social distancing policies: (i) shelter-in-place orders (SIPO), (ii) non-essential business closures, (iii) mandatory quarantine for travelers, and (iv) bans on large gatherings. Results suggest that the average US state is highly inefficient in producing the fraction of the population that does not have COVID-19 without social distancing policies put in place. We find that having any of the four major social distancing policies increases conditional efficiency by 9.7 (9.5) percentage points in the first 100 days (full sample). This corresponds to 57 (172) fewer total COVID-19 cases per 100,000 population in the first 100 days (full sample). We also find that population density accounts for a majority of unconditional state inefficiency. Evidence suggests considerable heterogeneity in conditional efficiency improvement, indicating that no uniform national social distancing policy would have been more effective; more effective strategies would have been to target more densely populated areas. Conditional efficiency regressions suggest that bans on large gatherings were the most effective policies, with SIPOs and non-essential business closures having smaller impacts. States that implemented social distancing policies except mandatory quarantine for traveler policies were highly effective for the first 100 days, but had less effectiveness over the full sample. There is also preliminary evidence that premature revocations of social distancing policies reduced conditional efficiency, leading to COVID-19 case spikes.

1 Introduction

The COVID-19 pandemic has led to unprecedented losses of life, and one of the worst global recessions in decades, with the USA taking center stage as the country with the largest number of deaths, and one of the highest per capita cumulative death rates (Bilinski and Emanuel, 2020). The early onset of the pandemic gave rise to reactionary mitigation measures across the world, ranging from outright international border closures to various types of social distancing orders at local levels. In mid-April, to reduce the burden of COVID-19 on the healthcare system as well as to mitigate its spread, the USA began implementing various social distancing policies, including shelter-in-place orders, school closures, bans on large gatherings, seating limits in bars and restaurants, mask mandates, and more, with nearly 95 percent of the American public being affected by state at home orders (Baek et al. 2020). The burgeoning literature on social distancing policies and their effectiveness in curbing the spread of COVID-19 is not clear. Some studies (Gibson, 2020; Homburg, 2020; Williams et al., 2021) find that lockdowns had no impact on COVID-19 mortality in a variety of countries, while others have found that social distancing policies have generally been credited with a reduction in the spread of COVID-19 (Abouk and Heydari, 2020; Courtemanche et al., 2020; Dave et al., 2020a; Friedson et al., 2020), these containment measures had detrimental economic impacts. Hence, the role of policy makers is to strike a delicate balance between mitigating the loss of life versus the economic cost of shutdowns (Acemoglu et al. 2020), as well as to establish the most efficient measures of controlling the pandemic (Ibrahim et al., 2020).

Without the availability of vaccines and pharmacological treatments at the start of the pandemic, social distancing policies became the first line of response. Since it was up to states regarding which policies to implement, there was substantial heterogeneity in the timing and type of policies, with shelter-in-place orders being the most common containment measure. California was the first state to implement a shelter-in-place order (SIPO) on March 19, 2020 and within a month, another 39 states followed suit (Dave et al., 2020a). Additional complimentary policies adopted by states included school closures, bans on large gatherings, entertainment venue closures, restricting dine-in in restaurants, etc. (Courtemanche et al. 2020). Recent evidence suggests that some of these policies, such as shelter-in-place orders, have been more successful than others (e.g. bans on large gatherings and school closures) (Abouk and Heydari, 2020; Andersen, 2020; Courtemanche et al., 2020). Some of these policies, and most specifically the shutdowns, however, have been linked to negative economic effects (Baek et al., 2020; Baker et al., 2020; Coibion et al., 2020; Mulligan, 2020).

While recent literature has primarily focused on examining the casual effect of social distancing policies, emerging research has recognized the need to examine these policies in the context of efficiency analysis. For example, works by Ibrahim et al. (2020), Shirouyehzad et al. (2020), and Breitenbach et al. (2020) rely on a non-parametric data envelopment analysis (DEA) methodology to analyze the cross-country efficiency of contagion control, pointing out that many nations were not efficient in utilizing their resources to “flatten the curve,” including the USA.

We are not aware of any studies within the US that analyze the efficiency of state-level social distancing policies. Hence, the goal of this study is to contribute to the ongoing discussion on designing the most efficient policies. We focus on four of the most commonly analyzed social distancing policies: shelter-in-place orders (SIPO), non-essential business closures, mandatory quarantines for travelers, and bans on large gatherings. SIPO’s, also known as stay-at-home orders, require residents to stay at home for activities that are deemed non-essential. Residents, however, can engage in essential activities such as buying food, caring for others, travelling to work, exercising, etc. (Castaneda and Saygili 2020; Dave et al. 2020a, 2020b). SIPO policies tend to be American-specific. The term “lockdowns” has been used by the media to describe the restriction on movement of people in attempts to mitigate COVID-19 in Europe and China. While the terms are not technical they generally have the same meaning, but might vary based on their severity or scope. For example, a major difference is that shelter-in-place orders in the USA were regionally based, determined by localities or states. Lockdown orders in non-US countries were (for the most part) determined at the national level, and had significantly higher penalties for violators.

We rely on newer non-parametric order-m efficiency estimators developed by Cazals et al. (2002), and estimate the efficiency of US states, both unconditionally and conditionally, based on these four social distancing policies. Our estimation technique allows us to circumvent technical issues pertaining to both parametric estimators and non-parametric DEA estimator used in previous research. Our technique also allows us to address pertinent and relevant policy issues as to the best and most efficient course of action to take when designing policies to battle future pandemics. We estimate efficiency over the first 100 days, as well as a full 185 day sample.¹

Our results extend the literature by providing an insight as to which policies may be more effective in combatting future pandemics. We first establish that the average US state was considerably inefficient in preventing the COVID-19 epidemic.² Unconditionally, we see that in the first 100 days (full 185 day sample) of the COVID-19 pandemic, the average US state was 38.6 (36.8) percent inefficient in the output orientation. This suggests that there were 750,000 (over 2 million) more COVID-19 cases than there should have been in the first 100 days (full 185 day sample). This also suggests that there is considerable inefficiency, even after controlling for population density. Importantly, even if we use methods to adjust for uncounted COVID-19 cases, the average US state was 32.6 (33.9) percent inefficient in the first 100 days (full 185 day sample).

Second, we find that there exists considerable state-level heterogeneity in inefficiency. In the first 100 days (full sample), California was the most inefficient state in the output direction, being 43.3 (42.0) percent inefficient. Other early hard-hit states, including New York and Washington, were comparably inefficient in the output direction. On the other hand, relatively less populated states like South Dakota and North Dakota were the most efficient states unconditionally, even though both were around 34 (30) percent inefficient unconditionally in the output direction in the first 100 days (full sample).

Third, we find that a significant amount of the unconditional inefficiency of states due to COVID-19 is related to population density.³ For instance, California’s efficiency improves by 18.3 (21.0) percentage points after controlling for population density in the first 100 days (full sample). Given that California had 135,000 (718,000) total COVID-19 cases 100 days (full sample) after the start of the pandemic, if California had a population density more similar to North or South Dakota, there would have been nearly 25,000 (100,000) fewer total COVID-19 cases by this date. We also see that the impact of population density is much less in less densely populated states.

Fourth, we find that in the first 100 days (full sample), having any social distancing policy led to conditional efficiency improving by 9.7 (9.5) percentage points, using our estimates where we do not control for population density.⁴ Having a SIPO policy alone led to conditional efficiency improving by 7.9 (4.8) percentage points. Having any social distancing policy, therefore, led to 57 (172) fewer total COVID-19 cases per 100,000 population in the first 100 days (full sample). Given that there were 588.7 (1,184.2) total COVID-19 cases in the first 100 days (full sample), this suggests considerable benefits in bending the COVID-19 curve from social distancing policies. Our results also hold if we utilize COVID-19 test rates per million population, suggesting that increased testing would not have mitigated the pandemic. Fifth, we find that conditional efficiency improvements reduced over time; the likeliest explanations were either behavioral noncompliance, where individuals and businesses chose to flaunt rules the longer they were in place, or states were too premature in revoking their policies, leading to explosions in case rates. Sixth, using a quasi difference-in-difference methodology, we find evidence that premature revocation of COVID-19 social distancing policies led to reductions in conditional efficiency, suggesting that states relaxed COVID-19 policies too soon.

Seventh, partial regression plots suggest that conditional efficiency improved having a SIPO policy, a ban on large gatherings, and/or non-essential business closures; on the other hand, mandatory quarantines for travelers were ineffective in combatting COVID-19, with productivity worsening in states that implemented these social distancing policies. In fact, bans on large gatherings were the most effective social distancing policy put into place. SIPO policies alone were effective in curbing the spread of COVID-19; this suggests that having these policies in combination with others reduced their efficacy. Lastly, we find that our results are qualitatively similar if we use the unadjusted COVID-19 case data from our main efficiency estimates, or the methods used by Millimet and Parmeter (2021) to adjust for uncounted cases. We find that conditional efficiency improvements using the “adjusted” data are about 70-percent that of the unadjusted data, suggesting that slightly over a quarter of inefficiency is related to noise from uncounted COVID-19 cases. However, the similarity in results using the unadjusted or adjusted COVID-19 case data is analogous to the results from Orea and Alvarez (2022) and Orea et al. (2021), further providing proof for our analysis.

The reminder of the paper is organized as follows. Section 2 compiles a list of previous literature, while Sect. 3 presents the data and methods used in the paper. Section 4 presents the empirical results, while Sect. 5 concludes the study.

2 Literature review

This study relates to several strands of literature examining the effectiveness of COVID-19 containment policies, as well the growing body of literature focusing on those social distancing policies that are deemed most efficient.

The growing literature on the effectiveness of social distancing policies on curbing the spread of COVID-19 (or reducing its mortality rate) is contradictory. Several studies have found that these economic “lockdowns” were highly ineffective. Williams et al. (2021) find that lockdown in England and Wales had no significant impact on COVID-19 related mortality. Under their preferred specification, the lockdown was associated with a positive increase in net mortalities. The authors explain that increasing mortality as a potential case of a Peltzman offsetting effect, where people change their behavior in response to changes in perceived level of risk. For instance, lockdowns might have altered people’s perception of risk associated COVID-19, consequently leading them not to seek care for non-COVID-19 illness, and potentially resulting in additional deaths. Overall, their results indicate that the effectiveness of the COVID-19 lockdown in England and Wales was limited. Gibson (2020) examines policy response to COVID-19 in New Zealand, which implemented the world’s most stringent lockdown. Focusing on variation in lockdown policies across US counties, the study finds that lockdowns do not reduce COVID-19-related deaths. Drawing conclusions from the US data, Gibson (2020) concludes that due to its stringent lockdown, New Zealand suffered substantial output losses of around 10 billion dollars. Similarly, Homburg (2020), using cross section data from several countries, finds that lockdowns were “superfluous and ineffective.”

Other studies have found that “lockdown” policies were effective, though at an economic cost. Within the context of the effects of SIPO polices, Friedson et al. (2020) focuses on the effects of SIPO policies in California over 29 days; they find that SIPOs led to a reduction in COVID-19 cases by 125.5 to 219.7 positive cases per 100,000 population, and led to 1,661 fewer COVID-19 related deaths. Their analysis also suggests that there were 400 jobs lost per each one life saved. Abouk and Heydari (2020) find that SIPOs were effective at keeping people at home as well as reducing mobility outside of homes, and thus led to a steady decline in COVID-19 cases. On the other hand, less restrictive policies had no statistically significant effect on reducing the spread of the infection. Similarly, Dave et al. (2020a) find substantial heterogeneity in the effect of SIPOs across US states. More specifically, their results indicate that statewide SIPOs were associated with a 5 to 10 percent increase in the fraction of population that sheltered in place on a given day. This does imply that these policies were considerably effective in reducing social mobility. Along with this, even with substantial state-level heterogeneity, SIPOs were associated with a 44 percent decline in COVID-19 cases, with the early adopter states enjoying the largest declines in positive COVID-19 cases.

Though COVID-19 social distancing policies were effective in reducing social mobility and, on average, the spread of the disease, the same containment measures had detrimental economic ramifications: these include declines in employment, GDP, consumer spending, debt and loan payments (Baker et al. 2020; Bick and Blandin, 2020; Coibion et al., 2020). For instance, Coibion et al. (2020) find that declines in spending and employment can be largely attributed to lockdowns rather than to COVID-19 infections, with lockdowns accounting for a nearly sixty percent decline in the employment-to-population ratio. Baek et al. (2020) find that each additional week of being exposed to SIPOs leads to a 1.9 percentage point increase in a state’s unemployment claims. Walmsley et al. (2020) estimate that, for a three-month closure, GDP losses are $4.3 trillion, with an employment decline of 35.2 million workers. Baker et al. (2020) find sizable declines in consumer spending, with the declines being twice as large in those states that implemented SIPOs. Acemoglu et al. (2020) indicate that keeping the COVID-19 mortality rate below 0.2 percent would necessitate full or partial lockdowns for one and a half years, with economic costs of upwards of 38 percent of GDP; while keeping economic damages to less than 10 percent of annual GDP would mean accepting a mortality rate of over 1 percent. These highlight the quite substantial economic costs that may negate the potential health benefits from COVID-19 social distancing policies.

Given the costs of social distancing policies, additional research has focused on optimal policies to improve the tradeoff been losses to the economy and saving lives within the framework of a Susceptible-Infected-Recovered (SIR) epidemiological model (Acemoglu et al., 2020; Alvarez et al., 2020; Berger et al., 2020; Farboodi et al., 2020; Eichenbaum et al., 2020). Within the context of COVID-19, the effects of social distancing policies enter the model through the case transmission rate; the goal of optimal policy design is to achieve a given case transmission rate while minimizing economic costs (Atkenson, 2020; Stock, 2020;). Alvarez et al. (2020), using a variation of the SIR model, find that an optimal policy prescribes a severe lockdown two weeks after the beginning of an outbreak, covering 60 percent of the population after one month, and then eased to 20 percent of the population after three months. Berger et al. (2020), using a similar SEIR (Susceptible-Exposed-Infectious-Recovered) model, show that testing at a higher rate alongside targeted quarantine policies dampens the economic consequences of COVID-19, while simultaneously reducing peak symptomatic infections. Unfortunately, these highly targeted COVID-19 social distancing policies suffer from two notable flaws: (i) they require participation of nearly all of the covered population; and (ii) the effectiveness will be limited if there exists a substantial fraction of the population that is asymptomatic or do not receive regular testing. The second point is notable, as many papers have attempted to adjust reported COVID-19 numbers to account for cases that are not captured (Flaxman et al., 2020; Gibbons et al., 2014; Hortascu et al., 2020; Li et al., 2020; Millimet and Parmeter, 2021).

These worries are codified by Eichenbaum et al. (2020), who note that relaxing a containment policy prematurely will reduce the burden on the economy but lead to higher infection rates, consequently resulting in subsequent future economic declines with the need for additional containment policies. Acemoglu et al. (2020), using a variation of the epidemiological SIR model, find that optimal non-uniform policies such as strict lockdowns for the most vulnerable groups, can be more effective than uniform policies in minimizing economic loss and the loss of life; compared with a GDP loss of 38 percent annually from a full lockdown, targeted lockdowns would keep the mortality rate at 0.2 percent, while limiting economic losses to 24.8 percent of annual GDP.

Another strand of literature has been focusing on the efficiency of COVID-19 response policies, relying on non-parametric data envelopment analysis (DEA) methodology. DEA is a performance evaluation methodology that transforms multiple input and output variables into a single measure of productive efficiency (Akazili et al. 2008). These studies, conducted at the international level, find considerable inefficiency in combatting the COVID-19 pandemic (Breitenbach et al., 2020; Ibrahim et al., 2020; Shirouyehzad et al., 2020). However, there are considerable limitations to these papers. For example, DEA estimators suffer from well-known statistical issues, including a curse of dimensionality and sensitivity to outliers, both of which will be present in an analysis of COVID-19 social distancing policies.

The sensitivity of the DEA estimator to outliers, is likely to be problematic in studies at the international level. The variation in access to healthcare facilities, “experimental” treatments for severe illnesses, expectations for vaccine adoption, as well as a country’s governance structure make it likely that there will be outliers that will bias the results. With respect to the curse of dimensionality, the DEA non-parametric estimator has less than root-n convergence, suggesting that it needs more datapoints than parametric investigations. For instance, using all 194 internationally recognized countries with daily data from March 1, 2020 to May 1, 2021 would lead to 82,644 unique datapoints. However, if the DEA specification in these papers had 1 output and just 4 inputs, this would be analogous to a regression equation with just 1,898 unique datapoints suggesting considerable data loss. Therefore, it is unlikely that these papers provide meaningful insight. To address these non-parametric issues, we use newer nonparametric estimators, which have none of the canonical DEA limitations.

3 Data and theory

Data on COVID-19, including cumulative (or daily) cases, deaths, and testing at the state level are accumulated by the Center for Systems Science and Engineering at Johns Hopkins University, as a free repository.⁵ These data are corroborated by the Centers for Disease Control (CDC) in the USA. We also utilize the Kaiser Family Foundation database on state social distancing action effective and rollback dates. The four main social distancing policies that we use are: (i) SIPOs; (ii) non-essential business closures; (iii) mandatory quarantine for travelers; and (iv) bans on large gatherings.⁶ Data are collected from March 1, 2020 to September 1, 2020.

Efficiency estimation utilizes three types of variables: (i) inputs; (ii) outputs; and (iii) environmental variables. Our main output measure of interest is the percent of the population that is not considered a COVID-19 case. We utilize a variety of mobility measures as inputs. Social mobility inputs are provided from SafeGraph, Inc. SafeGraph has anonymized population movement data from cellphones of nearly 45 million devices. We use two state-by-day measures of mobility: (i) the percent of the state-day population that remains at home for the entire day; and (ii) the average distance, in meters, the average person moves outside the home in a state-day, standardized so that bigger values are associated with traveling less outside of the home.⁷ We include two additional proxies for work mobility. The first is state-by-week initial unemployment claims; the second is state-by-week continued unemployment claims. Given the considerable unemployment impact caused by both COVID-19 and social distancing policies, we believe that the transmission of COVID-19 due to work could be impacted; therefore, the inclusion of these inputs. Later, as a robustness check, we include the number of COVID-19 tests performed per million population. We do not include this as a main input because not all states reliably report these numbers each day. For instance, only 38 states (out of 51 states plus Washington D.C.) have full observations.

For our environmental variables, we utilize a variety of state-level policies aimed at reducing the spread of COVID-19, largely through decreasing social contact (White House, 2020). These policies have been studied on the heterogeneity of state policies between early COVID-19 policy adopters and high population density states (Dave et al., 2020a); their effectiveness (Abouk and Heydari, 2020; Friedson et al., 2020); and their impact on health outcomes (Dave et al., , 2020a, 2020b; Friedson et al., 2020). We use a variety of state-level social distancing policy sets to investigate the impact of these policies on COVID-19 outcomes: (i) having any of the four major social distancing policies in place (Any 4); (ii) having all of the four major social distancing policies (All 4); (iii) a SIPO policy, non-essential business closure policy, and ban on large gatherings (S/B/L); or (iv) a SIPO policy (SIPO only).

We conduct an analysis on more policies than a simple SIPO as many states had multiple policies occurring at the same time. For instance, California had the nation’s earliest SIPO on March 19, 2020, but also instituted essential business closures on March 19, 2020 and placed a ban on large gatherings on March 16, 2020. We plan on investigating efficiency in the output orientation, as policymakers are worried about bending the COVID-19 curve by reducing COVID-19 cases.

In addition to using these social mobility measures as environmental variables, we include state-level population density measures, then compare estimates to models where population density measures are not included. The inclusion of population density is contested in the literature. According to Meijer et al. (2021) there is a positive association between population density and mortality. This is especially relevant in the context of virus transmission since population density facilitates virus spread via close person-to-person contact. Prior research showed that COVID-19 transmission is more likely to occur in higher density cities. Wong and Li (2020) find that county level population density was an important predictor of COVID-19 cases across counties, and should be explicitly included in COVID-19 models that predict transmission. Similarly, Sy et al. (2021) show that dense areas are more susceptible to disease transmission, with a substantial increase in case rates and COVID-19 transmission rates. This is corroborated by Rubin et al. (2020).

Other studies have found the opposite. Hamidi et al. (2020) finds that COVID-19 death rates were lower in more dense counties possibly due to better access to healthcare and easier management of social distancing policies. Carozzi et al. (2020) find that while population density impacts the timing of an outbreak, it has no impact on COVID-19 cases or deaths. The authors note that there are mediating factors; more dense areas are younger and more likely to engage in social distancing. Similarly, Hashim et al. (2020) find that population density is not related to cumulative mortality rates, while Dreher et al. (2021) find that population density was not a factor in the COVID-19 reproductive rate.

The work by Althoff et al. (2021) finds that areas with higher population density have a higher share of jobs that can (and were) performed remotely, and that these jobs tended to be higher paid. Therefore, it is possible that population density could be associated with worse COVID-19 outcomes, better COVID-19 outcomes, or no difference in COVID-19 outcomes.

These suggest the use of two specification s: one with and another without the inclusion of a population density variable, given the considerable disparity in the literature. If population density does matter, then public health authorities can target areas with higher levels of density for higher levels of testing and contact tracing, while reducing efforts in less dense areas. One other concern with the inclusion of population density would be that it is a highly imperfect proxy for a number of other related variables that could influence social mobility: use of public transportation, degree of urbanization, connectivity to other states, and employment patterns in certain industries (among others). Mattson (2020) provides evidence that population density has relationships to broader environmental variables. Though population density is not a perfectly all-encompassing variable to measure population spread and contact, Mattson (2020) notes that higher population density is associated with increased demand for public transit, more land use mix, and better accessibility.

One area of concern is that the social distancing policies influence COVID-19 spread indirectly, via social mobility. We run a regression of state-level demographic variables (age, race, gender), state-level socioeconomic variables (median income, unemployment rate), four social distancing policy indicators, day and state fixed effects, with regressions weighted by state-level population, on our measures of social mobility. Our results are reported in the Appendix Table 9. We find that none of our major social distancing policies have any influence on travel distance from home, while the SIPO policy increases the median time spent at home (while population density decreases it). Therefore, it appears that the impact of these SIPO policies on time spent at home accounts for about 17.4 percent of our estimated impact of SIPO policies on COVID-19 case rates.

3.1 Estimators

This paper uses non-parametric estimators for a variety of reasons. Parametric estimators require potentially untenable specification assumptions or have other serious drawbacks. For instance, the distribution of the composite error term must be specified, often by a half-normal or truncated normal distribution, for use in stochastic frontier analysis (SFA).⁸ Simar and Wilson (2013) provides a comprehensive analysis on data envelopment analysis (DEA) estimators, while Parmeter and Kumbhakar (2014) provides one for SFA estimators.⁹

Non-parametric estimators are often used by researchers because they do not require a priori specification of the functional relationship that is being estimated. Similarly, because of the lack of distributional assumption, incorporating multiple outputs is seamless. However, certain non-parametric estimators, such as DEA estimator suffer from well-known shortcomings that make validity and inference a problem. The shortcomings include the DEA estimator having less than root-n convergence due to the curse of dimensionality, where the number of observations required to obtain meaningful estimates increases with the number of production inputs and outputs used in the estimation, and the estimator being sensitive to outliers (Kneip et al. 1998).

Two newer non-parametric estimators have been developed in recent years: the order-α and order-m estimators. Both estimators eliminate many of the issues found in other non-parametric estimators, like the DEA estimator, and are not sensitive to outliers and have the classical, parametric, root-n rate of convergence (Cazals et al., 2002; Simar and Wilson, 2008; Wheelock and Wilson, 2009). Thus, the order-α and order-m estimators offer the distributional flexibility of non-parametric estimators, while simultaneously providing traditional statistical features found in parametric estimators. We utilize the order-m estimator. We next briefly discuss the statistical features of the order-m estimator, noting that we have a set $\chi$ of n state-day observations, characterized by p inputs $x\left({x}_{1}\dots {x}_{p}\right)$ and q outputs $y\left({y}_{1}\dots {y}_{q}\right)$.

3.1.1 Unconditional order-m estimator

The order-m estimator was developed by Cazals et al. (2002), and denotes the best production set as the free disposal hull (FDH) of undominated input–output combinations

$${\Psi }_{FDH}=\left\{\left(x,y\right)\in {\mathfrak{R}}_{+}^{p+q}|x\le {X}_{i},y\le {Y}_{i}\in \chi \right\}$$

(1)

We choose an output-orientation for the order-m estimator; that is, given a fixed level of inputs, what is the output shortfall for a county relative to the best practice frontier.¹⁰ We choose an output orientation since policymakers have attempted to minimize the total number of COVID-19 cases, which is the reciprocal of our output measure.

As in Fried et al. (2008), the output-oriented efficiency estimates measure the distance to the best practice frontier

$$\lambda \left({x}_{0},{y}_{0}\right)=sup\left\{\lambda |{x}_{o},\lambda {y}_{o}\in {\Psi }_{FDH}\right\}$$

(2)

In the output-orientation, an inefficient observation has an efficiency score, $\lambda$, larger than 1. The value $\left(\lambda -1\right)$ indicates the potential percentage increase in output if the observation would produce as efficient as its reference partner. Unfortunately, the model in (2) is deterministic, and may be influenced by outliers. Cazals et al. (2002) mitigate these outlier issues by drawing from the sample population, with replacement, subsamples of size $m<n$ among those ${Y}_{i}$ such that ${x}_{0}\ge {X}_{i}$ (observations with fewer inputs than the evaluated observation).

Details of this method are shown in Cazals et al. (2002), who note that the order-m estimator achieves the parametric root-n rate of convergence. This partial sample size, m, is determined as the value for which the number of super-efficient observations is constant. The sampling and efficiency estimations are done B times (where B is sufficiently large), and the order-m efficiency estimates, ${\lambda }^{m}\left({x}_{0},{y}_{0}\right)$ are obtained as the arithmetic average of the B inefficiencies or, conversely, an integral formulation of this bootstrap.

3.1.2 Conditional order-m efficiency measures

Order-m efficiency estimates can be adapted to take into account exogenous variation, denoted by a vector of exogenous variables z, between states, that do not require the separability assumption that is found in Simar and Wilson (2007, 2011). These methods were developed by Daraio and Simar (2005) who proposed drawing subsamples of size m by a given probability, which is determined by a Kernel function around the variables Z, drawn B times with replacement. The B efficiency estimates are averaged to obtain the conditional order-m estimates ${\lambda }^{m}\left({x}_{0},{y}_{0}|{z}_{0}\right)$, where interpretations are similar to the unconditional order-m estimator. However, the estimation allowed only continuous exogenous variables to be included in the estimation. As shown in Daraio and Simar (2005), the conditional order-m estimator may not have root-n convergence, as the convergence rate depends on the dimension of Z, necessitating a parsimonious specification of the vector of exogenous variables that influence efficiency scores.

De Witte and Kortelainen (2013) extend the conditional order-m estimator to include discrete exogenous variables, which do not influence the convergence rate of the conditional order-m estimator any further, since econometric theory states that the convergence rate of non-parametric estimators for conditional density and distribution functions involving mixed variables only depend on the number of continuous variables. Similar to De Witte and Kortelainen (2013), we construct a ratio of unconditional to conditional estimates, $\frac{{\lambda }^{m}\left({x}_{0},{y}_{0}\right)}{{\lambda }^{m}\left({x}_{0},{y}_{0}|{z}_{0}\right)}$, which are regressed on the Z exogenous variables.

Since the regression of exogenous variables Z is on a ratio, the marginal coefficient is less meaningful than a standard regression. If Z is multivariate, one can utilize partial regression plots for the visualization of the effect to determine how a single exogenous variable in Z affects the production process, holding the other variables in Z constant. In the output-orientation, a horizontal line implies no effect, and an increasing (decreasing) smoothed regression curve shows that Z improves (decreases) efficiency in the production process. De Witte and Kortelainen (2013) obtain p-values of the significance of the influence of Z on the efficiency scores, based on the work of Li and Racine (2007), by utilizing a local linear regression estimation and a non-parametric naive bootstrap procedure. Unlike the second-stage efficiency regression proposed by Simar and Wilson (2007), this model does not need a full separability assumption for proper inference to be determined.

4 Results

For the output-oriented order-m estimator, the choice of m is found where the percentage of super-efficient observations decreases smoothly with m, as explained in Daraio and Simar (2007a).¹¹ In our analysis, we choose a value of m = 500 for the first 100 days, and m = 1,000 for our full 185 day sample.¹² To interpret our efficiency estimates, a value greater than 1 indicates inefficiency in the output direction; in other words, it is the percent increase in output that is feasible given the input levels, if there was perfect efficiency. To calculate the percent potential increase in output, subtract one from the efficiency value and multiply by 100-percent.

Table 1 presents our unconditional efficiency findings where we estimate how effectively social mobility measures provided by SafeGraph Inc. are in limiting COVID-19 cases, in the first 100 days. Table 1 also provides conditional efficiency estimates, where our unconditional efficiency estimates are conditioned on state-level social distancing policies (environmental variables) that are put into place: (i) Any 4, which is having any social distancing policy on day j; (ii) All 4, which is having all four social distancing policies on day j; (iii) S/B/L, which is having all of the SIPO, closure of non-essential business, and bans on large gathering policies on day j; and (iv) SIPO, which is having a SIPO on day j. We provide estimates where population density is not included as an environmental variable, and estimates where it is.

Table 1 Efficiency Estimates, Unconditional and Conditional, First 100 Days

In the first 100 days, the average state is 38.6-percent inefficient unconditionally.¹³ This would suggest that there were nearly 750,000 more COVID-19 cases than there should have been in that first 100 days, given the level of social mobility found in the USA. We believe that behavioral noncompliance is one of the leading explanations for this phenomenon. Given the importance of “superspreader” events in COVID-19 transmission (Dave et al. 2020b) and the evidence that post-April, movement patterns increased (O’Donoghue et al., 2020), it does suggest that the reopening of America in late April, 2020 and early May, 2020 led to significant rises in COVID-19 cases.

We see that a variety of states that are unconditionally inefficient, including California, Washington D.C., Hawaii, Massachusetts, New Jersey, and Pennsylvania, have some of the highest population densities in the USA, with a minimum density of over 200 people per square mile. On the other hand, the least inefficient states, including Alaska, North Dakota, and South Dakota, have some of the lowest population densities in the USA, with Alaska (North Dakota; South Dakota) having 1 (10; 11) people per square mile. If we focus on conditional efficiency estimates that do not take into account population density, having any of the four social distancing policies (Any 4) leads to California (Washington D.C.) being 26.1 (28.2) percent inefficient; Alaska (North Dakota; South Dakota) would be 26.5 (30.7; 32.7) percent inefficient. This suggests that, in California, there should have been 26.1 percent more of the population not having ever been a positive COVID-19 case. This also does suggest that population density may play a role in high levels of unconditional inefficiency.

Focusing on efficiency estimates that condition for population density, California (Washington D.C.) is now 7.8 (7.5) percent inefficient. Alaska, North Dakota, and South Dakota, some of the least dense states, are now 19.5, 25.5, and 25.2 percent inefficient, respectively. In California, there should have been 7.8 percent more of the population not having ever been a positive COVD-19 case. Importantly, we see that even conditioning for having any social distancing policy (Any4) and population density, the average US state is nearly 12-percent inefficient. This amounts to the average state having about 69.8 more COVID-19 cases per 100,000 population than it should have, given how socially mobile the population was. These results also accord with Dave et al. (2021), who found that states with higher population densities reaped more of the benefits of social distancing policies. In fact, it appears that conditioning for population density improves conditional efficiency more than social distancing policies. Results also suggest that weather could play a role in these findings, where colder states are more likely to benefit from conditioning for population density.

We see that conditioning our efficiency estimates on various social distancing policies leads to an elimination of 9.6 to 18.9 percent of the inefficiency gap, depending on the type of social distancing policy implemented; this would correspond to 187,000 to 368,000 fewer COVID-19 cases in the first 100 days in total. Having any of the four social distancing policies (Any 4) leads to the largest efficiency improvement of nearly 19 percentage points. Having all four of the social distancing policies (All 4) leads to the smallest efficiency improvement of under 10 percentage points. In case values, this means that having any of the four social distancing policies led to over 54 fewer total COVID-19 cases per 100,000 population in the first 100 days. Having all four of the social distancing policies, the least effective policy set, led to nearly 31 fewer total COVID-19 cases per 100,000 population in the first 100 days. This compares to a total COVID-19 case count of 588.7 cumulative COVID-19 cases per 100,000 population on the 100^th day of the pandemic.

These results suggest the incredibly effective nature of the social distancing policies in limiting the COVID-19 pandemic in the first 100 days. It also suggests that there was not a national policy set that would have uniformly reduced COVID-19 cases from the level found in the first 100 days, and that the piecemeal approach adopted by states was likely the superior option. We also find that having multiple policies may have actually harmed attempts to curb the COVID-19 pandemic. This last point has interesting implications; the two likeliest explanations are: (i) behavioral noncompliance, where individuals and/or businesses chose to purposefully ignore laws (if they felt that they were not legal and/or the economic costs were too high); or (ii) confusion about what was allowed with the restrictions due to different laws being put into place at different times. Lastly, we do see that the impact of population density on conditional efficiency suggests that a national social distancing policy set would not have been effective, especially in less densely populated areas.

One area of potential concern is that we have transformed a “bad” output, the COVID-19 case rate, to a “good” output. Similarly, we have transformed “bad” inputs to “good” inputs. To ensure that our results are not spurious due to our transformation, we utilize methods that incorporate “bad” inputs and outputs in the analysis, similar to Cordero et al. (2015) and Seiford and Zhu (2002). In the latter, a “bad” output is multiplied by -1, where a translation vector of sufficient size is added to ensure the output is now positive (with a similar transformation for a “bad” input). Our results for our transformed estimates, as well as the Seiford and Zhu (2002) transformed estimates, for the first 100 days are presented in Appendix Table 10. Importantly, we see that the efficiency results are qualitatively the same. Due to ease of interpreting “good” inputs and outputs, we therefore focus on our transformation.

When we focus on our full sample, which includes data through September 1, 2020, our results are qualitatively similar to both Dave et al. (2020a) and Friedson et al. (2020), though lower in magnitude.¹⁴ These results are presented in Table 2. This suggests that the impacts of social distancing policies were immediate and moderately long-lasting, but that the effects declined over time. In fact, we do see that for SIPO policies, consistent with Dave et al. (2020a) and Friedson et al. (2020), the earliest adopters of SIPOs saw the biggest conditional efficiency gains.

Table 2 Efficiency Estimates, Unconditional and Conditional, Full 185 Days

In unconditional terms, the average state is 36.8 percent inefficient; this means that there were 2 million more COVID-19 positive cases than there should have been, given levels of social mobility. This would again suggest that the average state was unprepared for the COVID-19 pandemic, and hints that even if early policies were effective in curbing the COVID-19 pandemic, this unpreparedness was a significant source of COVID-19. Though lower in magnitude than our estimates in Table 1, we see that conditioning our efficiency estimates on a variety of social distancing policies, without taking into account population density, eliminates a not-insubstantial amount of inefficiency. For instance, having Any 4 (All4) of the social distancing policies leads to conditional efficiency seeing improvements of 9.5 (2.9) percentage points. In terms of cases, having Any 4 (All 4) of the social distancing policies leads to 172.3 (52.6) fewer COVID-19 cases per 100,000 population on September 1, 2020; on this date, the total COVID-19 case rate was 1,814.2 total cases per 100,000 population. Similar to our conclusions from Table 1, not accounting for population density makes certain states look inefficient in combatting COVID-19; it also suggests again that implementing national social distancing policies would have been ineffective in less densely populated states, which could have further heightened resistance to other national mandates, including mask mandates.

When we condition for population density, similar findings emerge; COVID-19 social distancing policies were effective in curbing the COVID-19 pandemic, though not as effective as in the first 100 days. Having any of the four social distancing policies (Any 4) led to conditional efficiency improving by 10.1 percentage points; this corresponds to nearly 187 fewer COVID-19 cases per 100,000 population on September 1, 2020. Having all of the social distancing policies (All 4) leads to nearly 65 fewer COVID-19 cases per 100,000 population. Intriguingly, we see that the efficiency gains from having social distancing policies falls, the longer sample we estimate. For instance, in the first 100 days, having Any 4 (All 4) of the social distancing policies improves conditional efficiency by 9.7 (6.0) percentage points. For the full 185 day sample, conditional efficiency improves by 9.5 (2.9) percentage points. This again suggests that long-lasting COVID-19 social distancing policies may have had lowered effectiveness as time wore on, as individuals grew weary of the mandates and perhaps became more behaviorally noncompliant.

One potential source of unconditional efficiency present in both Tables 1 and 2 could be a lack of testing; Table 3, therefore, uses the number of COVID-19 tests performed per million population as an additional input.

Table 3 Efficiency Estimates, Conditional and Unconditional, Tests per Million Population as Additional Input

Compared to efficiency estimates where we do not control for COVID-19 tests per million population, we see that unconditional efficiency in the first 100 days improves by 0.6 percentage points when we add in tests per million population as an additional input, and unconditional efficiency over the full sample improves by 2.3 percentage points. Given the lack of reliable (and available) testing in the first 100 days of the COVID-19 pandemic, this suggests that testing was woefully inadequate during this time. In the first 100 days of the pandemic, cumulative testing was 4,351 per 100,000 population. By September 1, 2020, cumulative testing was 16,203 per 100,000 population. This nearly four-fold increase suggests that unconditional efficiency should improve. Focusing on our full-sample estimates that condition for population density, even with inadequate testing rates, inefficiency after conditioning for any of the four major social distancing policies is only about 11.6-percent, compared to an unconditional value of nearly 35-percent inefficiency. This corresponds to having nearly 1.4 million fewer individuals with COVID-19.

Next, we provide a basic test of the hypothesis that states ended social distancing policies too early, which led to higher than expected COVID-19 cases given input levels and relative to states that did not end the policies (worsened conditional efficiency) weeks after, in Table 4.

Table 4 Estimates of impact of loosening social distancing restrictions on efficiency estimates

To do this, we measure 7, 14, or 21 day averages of conditional efficiency after ending a social distancing policy. From this, we subtract the corresponding 7, 14, or 21 day average of conditional efficiency before ending the policy. To net out any pre-existing trends, we also subtract from this the difference between the 7 (14; 21) day average of the unconditional efficiency estimates after the policy change, minus the 7 (14; 21) day average of the unconditional efficiency estimates before the policy change. A positive value indicates productivity regression, which would indicate higher than expected COVID-19 cases given input levels, relative to states that did not relax social distancing policies.

Ex-ante, we would expect that relaxing social distancing measures during the pandemic would worsen efficiency. In part, relaxing social distancing measures may increase social mobility, which would potentially increase virus spread, reducing efficiency. Similarly, there have been mentions that relaxation of social distancing measures were done more based on political considerations than pandemic considerations, which would also reduce efficiency. At best, relaxing social distancing measures would lead to no change in personal behavior or virus spread, which would lead to no change in efficiency.

For the 7 (14; 21) day rolling average, we see that 5 (5; 5) states did not have conditional efficiency regression after revoking (or allowing to expire) a social distancing policy. On average, the 7 (14; 21) day rolling average has a 9.0 (10.0; 10.5) percentage point reduction in conditional efficiency after the revocation or expiration of any social distancing policy. This means that the average state had 9.0 (10.0; 10.5) percent more COVID-19 cases than expected, given input levels and relative to states that did not relax social distancing policies, in the 7 (14; 21) days after revocation. In fact, there is a moderate correlation between the date at which the first social distancing policy expired (or was revoked) in a state, and the conditional efficiency regression from the expiration or revocation of these policies. Though not causal, this is suggestive evidence that premature revocation or expiration likely led to increases in COVID-19 transmissibility and COVID-19 rates.

While we can explore the impact of social distancing policies on efficiency measurements, we also utilize the regression of unconditional to conditional efficiency scores on exogenous social distancing variables, shown by Cazals et al. (2002), Daraio and Simar (2005, 2007a, 2007b), and De Witte and Kortelainen (2013) for a variety of exogenous variables. Point estimates in this type of regression analysis are less meaningful than in traditional regressions, so we provide p-values. We set up partial regression plots, where an upward sloping diagram represents improvements in efficiency. Improvements in efficiency means having fewer COVID-19 cases than expected, given input levels and compared to states without the social distancing policies; declines in efficiency (or productivity regression) means there were too many COVID-19 cases, given the input levels and compared to states without the social distancing policies.

Table 5 examines the impact of our different social distancing specifications for our full sample, where we include population density as an environmental variable.

Table 5 Conditional efficiency regressions, various social distancing policies, full sample

Panel A explores the impact of having any social distancing policy (Any 4). We see that the presence of any policy except a mandatory quarantine for travelers improves conditional efficiency which means having fewer COVID-19 cases than expected in a population, relative to an area with no social distancing policies in place. One explanation for the finding that having a mandatory quarantine for travelers leads to having more COVID-19 cases than expected is the reduced travel traffic during the height of the COVID-19 pandemic, suggesting that these policies were ineffective. Another explanation is that the quarantines largely depended on behavioral compliance of travelers self-isolating; this perhaps suggests that individuals who did travel likely did not follow the full quarantine guideline.

Figure 1 provides the partial regression plot for Panel A.

Fig. 1

Regression of Ratio of Unconditional to Conditional Efficiency Estimates, Full Sample, Any Social Distancing Policy. Note: effratio refers to the ratio of unconditional efficiency to conditional efficiency. This figure analyzes the impact of having any of the social distancing policies in place on day j. We have set up the figures so that an upward-sloping diagram indicates productivity improvement, while a downward-sloping diagram indicates productivity regression

The illustration on Fig. 1 implies that the most effective policy was bans on large gatherings, with SIPOs and the closure of non-essential businesses being less effective in curbing the spread of COVID-19. We do see that the partial regression plot indicates that mandatory quarantine for traveler policies leads to a decrease in conditional efficiency, which again suggests that there were more COVID-19 cases than expected in states that adopted these policies, given social mobility levels and compared to states with no mandatory quarantine for traveler’s policy in place. The decline in conditional efficiency values was small, however, suggesting that any behavioral noncompliance on the part of travelers was not significant.

We do see that efficiency was highest around 40 days into the pandemic, with conditional efficiency declining after this point, which suggests that social distancing policies were most effective in curbing COVID-19 cases over this time span. This correlates with our findings from Tables 1 and 2, where conditional efficiency improvements were highest in the first 100 days. This does indicate pandemic fatigue, where individuals either unconsciously relaxed behaviors, or willingly started to operate in violation of social distancing policies. Lastly, we see that considerable inefficiency can be explained by population density. We see that conditional efficiency “improves”, or COVID-19 cases were not as high as expected, the higher the population density of a state. This result highlights the fact that more densely populated states had higher COVID-19 transmission and case rates because of the density of their populations.

Panels B, C, and D of Table 5 provide our regression estimates for states with all social distancing policies (All 4), states with SIPO, non-essential business closure, and bans on large gatherings (S/B/L), and states with SIPO policies only (SIPO only). We see that efficiency increased in each case, where having these social distancing policies means fewer COVID-19 cases than expected, given input levels compared to states with none of these policies. We again see that efficiency worsened over time, or we had more COVID-19 cases than expected given levels of social mobility, and that states with higher population densities had fewer COVID-19 cases than expected, relative to their less densely populated counterparts. Figure 2 provides the partial regression plots.¹⁵

Fig. 2

Regression of Ratio of Unconditional to Conditional Efficiency Estimates, Full Sample, a All Four Policies, b SIPO/Business/Large, (c) SIPO Alone. Note: effratio refers to the ratio of unconditional efficiency to conditional efficiency. This figure analyzes the impact of having any of the social distancing policies in place on day j. We have set up the figures so that an upward-sloping diagram indicates productivity improvement, while a downward-sloping diagram indicates productivity regression

Table 6 restricts our sample to the first 100 days, when conditional efficiency improvements were higher than in our full sample. The results are qualitatively similar to those found in Table 5.

Table 6 Conditional efficiency regressions, various social distancing policies, first 100 days

In Panel A, when analyzing the impacts of having any social distancing policy (Any 4), having a mandatory quarantine for travelers leads to having more COVID-19 cases than expected relative to states that did not have this policy, while having any of the other three policies (non-essential business closures, bans on large gatherings, SIPOs) leads to fewer COVID-19 cases than expected and relative to states that did not have these social distancing policies. Figure 3 provides the partial regression plot for Panel A. We also see that conditional efficiency worsened over time, and that having higher population densities were associated with improved conditional efficiency, meaning having fewer COVID-19 cases than expected, given input levels and relative to less densely populated states.

Fig. 3

Regression of Ratio of Unconditional to Conditional Efficiency Estimates, First 100 Days, Any Social Distancing Policy. Note: effratio refers to the ratio of unconditional efficiency to conditional efficiency. This figure analyzes the impact of having any of the social distancing policies in place on day j. We have set up the figures so that an upward-sloping diagram indicates productivity improvement, while a downward-sloping diagram indicates productivity regression

It also shows that, similar to our full sample, having a SIPO leads to fewer COVID-19 cases than expected, relative to states with no SIPO policy in place. Non-essential business closures and bans on large gatherings had significant improvements in conditional efficiency, improving by 4 and 30 percentage points, respectively. Again, in the first 100 days, the most effective policy was a ban on large gatherings. These again suggest that having these policies led to fewer COVID-19 cases than expected, relative to states that did not implement them. In Panels B, C, and D, we see that, again, having all four policies, having a SIPO, closure of non-essential business, and bans on large gathering policies, as well as having a SIPO policy by itself, were all highly effective in reducing COVID-19 cases. These are confirmed by the partial regression plots in Fig. 4.

Fig. 4

Regression of Ratio of Unconditional to Conditional Efficiency Estimates, First 100 Days, a All Four Policies, b SIPO/Business/Large, c SIPO Alone. Note: effratio refers to the ratio of unconditional efficiency to conditional efficiency. This figure analyzes the impact of having any of the social distancing policies in place on day j. We have set up the figures so that an upward-sloping diagram indicates productivity improvement, while a downward-sloping diagram indicates productivity regression

4.1 Stochastic Frontier analysis and COVID-19 case undercounting

As noted previously, it is likely that there is considerable undercounting of COVID-19 cases and deaths, which is a known issue in epidemiology (Gibbons et al., 2014). For instance, Millimet and Parmeter (2021) determine that there are 2.6 times more COVID-19 cases than reported, along with 2.0 times more COVID-19 deaths.¹⁶

Therefore, one of the main goals is to calculate a “multiplication factor”, which is an area-time construct that determines the multiple of actual cases or deaths that there are from reported cases and deaths. There have been a number of epidemiological attempts to model uncounted cases and deaths in different areas of the world, including China (Li et al., 2020), Europe (Flaxman et al., 2020), and the USA (Hortascu et al., 2020).

It is important to determine whether the inefficiency that we estimated in our sample is due to inefficiency, or “noise” related to uncounted COVID-19 cases, which has diametrically opposed policy conclusions. Similar to our analysis, both Millimet and Parmeter (2021, 2022) and Orea et al. (2021) have modeled this multiplication factor in terms of a stochastic frontier analysis (SFA) model, where the error term is a composite term. One of the components is a traditional noise term, while the other captures unobserved variables that measure the “inefficiency” in COVID-19 case or death counting (in other words, it will capture undercounting of cases and deaths). After taking into account these “uncounted” cases and deaths, Millimet and Parmeter (2021) find that non-pharmaceutical interventions (NPIs), which would include our four major social distancing policies, led to fewer COVID-19 cases and deaths. The association goes away in Fall 2022 (and may even be positive), in part due to pandemic fatigue. Orea and Alvarez (2022) found that Spanish NPI’s reduced the spread of COVID-19, therefore limiting cases.

As a robustness check, we replicate the analyses performed in Millimet and Parmeter (2021) and Orea and Alvarez (2022), utilizing stochastic frontier analysis and our state-level analysis, then calculate the marginal effects of the social distancing policies.¹⁷ Estimates are shown in Table 7. For the first 100 days, having a SIPO policy is associated with 8.4 percent fewer COVID-19 cases. Having any four of the social distancing policies reduces COVID-19 cases by 15.7 percent. For the full sample, our SFA estimates indicate that having a SIPO policy only reduces COVID-19 case rates by 7.4 percent. Having any four of the social distancing policies reduces COVID-19 cases by 12.5 percent. Though large, our estimates are consistent with our mainline efficiency estimates.

Table 7 Marginal effects of social distancing policies on Frontier, SFA analysis

As a further robustness check, we “adjust” our COVID-19 case totals by state-specific multiplication factors calculated via the method of Millimet and Parmeter (2021), and then run our main conditional order-m analysis on this. Using our adjusted COVID-19 case total, we estimate state average unconditional and conditional efficiency estimates, for having Any 4 policies, controlling for population density. We then compare these with our unadjusted efficiency estimates from Tables 1 and 2. These estimates are presented in Table 8. The last column provides the state-specific multiplication factors, while Appendix Fig. 5 shows the time-varying multiplication factors.

Table 8 Unconditional and conditional efficiency estimates, adjusted for uncounted COVID-19 cases and state-specific multiplication factors

We see that, in the first 100 days, our unadjusted conditional efficiency improves by 26.8 percentage points when accounting for Any 4 social distancing policies. Our adjusted conditional efficiency estimates show improvement of 18.8 percentage points, which is in line with the estimates provided in Table 7. Similar analysis holds for the full sample. Therefore, even though the conditional efficiency improvements using the adjusted COVID-19 case data is about 70-percent of the unadjusted conditional efficiency improvements, which would suggest that a little less than a third of inefficiency measures is from uncounted cases, we have confidence that our results are correct.

5 Conclusion

Broadly, we see that, regardless of the time span, our results confirm that social distancing policies were important in reducing cumulative COVID-19 cases, by improving conditional efficiency. However, based on our unconditional estimates, states were broadly unprepared for the pandemic, which led to considerable inefficiency. For instance, in the first 100 days of the COVID-19 pandemic, the average US state was considered 38.6 percent inefficient, which suggested 750,000 more COVID-19 cases than should have been. Conditioning our results on the types of social distancing policies implemented, either the full sample or the first 100 days, eliminated a considerable amount of the inefficiency. For instance, in the full sample, having any of the four social distancing policies led to 172 fewer total COVID-19 cases per 100,000 population on September 1, 2020; there were 1,814.2 cumulative cases per 100,000 population on that date. However, even after controlling for population density, there is still considerable state-level inefficiency in combatting COVID-19; in the full sample, the average state is still 15.6-percent inefficient relative to a hypothetically fully efficient state. Given inputs, there could be 283.7 fewer total COVID-19 cases per 100,000 population. This suggests that the social distancing policies that were put into place, while effective in curbing the COVID-19 pandemic, were not fully effective. This may have been due to lax enforcement of the policies, behavioral noncompliance on the parts of citizens and/or businesses, or the infectious nature of the disease itself.

Our results suggest that ex-ante policies, such as better emergency preparedness, bans on international travel, and more aggressive mask enforcement early on in the pandemic may have been able to reduce a significant fraction of the COVID-19 inefficiency, which means having fewer COVID-19 cases than expected, relative to states that did not enact these policies (or states that enacted them later). While a majority of states adopted various, and often different, sets of social distancing policies, there was considerable heterogeneity in state-level conditional efficiency improvements. This finding suggests that federal mandates, with regards to social distancing policies, would have been broadly ineffective in combatting COVID-19, and that considerable regional differences led to different policies needed in different parts of the country. In fact, states that were more densely populated had fewer COVID-19 cases than expected relative to states that are less densely populated, shown via conditional efficiency improvements. The policy implication is that targeted policies to combat pandemics should focus on highly dense urban centers first; population density had a larger impact on improving conditional efficiency than having any of the four major social distancing policies.

Our finding that conditional efficiency improvements were higher in the first 100 days, indicating fewer COVID-19 cases than expected during this time period given social mobility, suggests three potential explanations: (i) confusion about the variety of laws and what was allowed or not; (ii) behavioral noncompliance, where the longer social distancing policies lasted, the less likely that individuals complied with the mandates; and/or (iii) premature revocation or expiration of social distancing policies, which caused spikes in COVID-19 cases. Though we are unable to formally test (i) and (ii), our results are indicative that both likely played a part.

Though not a formal test for (iii) above, we look at 7, 14, or 21 day conditional efficiency averages after a state has eliminated a social distancing policy. From this, we subtract the corresponding 7, 14, or 21 day conditional efficiency averages before a state has eliminated a social distancing policy. We then subtract the unconditional estimates, creating quasi difference-in-difference estimates. We see find evidence that premature revocation of social distancing policies led to reductions in conditional efficiency, which means higher levels of COVID-19 cases than expected, given inputs and relative to states that kept policies in place. 14 (21) days after the revocation of a social distancing policy, 5 (5) states did not see reductions in COVID-19 conditional efficiency. In fact, 14 (21) days after the revocation of a policy, average conditional efficiency decreased by 10.0 (10.5) percentage points. Though not causal, our results are indicative of the considerable efficiency losses, which means higher than expected COVID-19 cases given input levels, from dropping social distancing policies too early (relative to those that did not).

While there has been considerable research in the economic literature on the impact of COVID-19, our results suggest investigation in four areas that may help inform and provide improved policy prescriptions. The first area is emergency preparedness; our results indicate considerable inefficiencies in this area, suggesting that it was unlikely that much could have been done once the pandemic started. The second area is the optimal length of public health policies; there is suggestive evidence that having these policies for both too short and too long of time spans reduced their efficacy. In the former case, individuals chose to engage in riskier behaviors once restrictions were relaxed. In the latter, it was likely that behavioral noncompliance became a bigger factor the longer policies lasted, with both citizens and businesses feeling that the economic costs of the shutdowns outweighed the public health benefits. The third is to provide further evidence that combating a pandemic may depend on the population density of an areas. Efforts and money should flow first to more densely populated urban areas, which can have considerably more of an impact than enforcing and instituting social distancing policies. In fact, more specifically targeted social distancing policies, such as SIPOs in major urban centers, may have reduced later resistance to further (and continued) social distancing policies in many parts of the country, perhaps reducing the cumulative impact of COVID-19 in America. Finally, the role of weather and timing on the effectiveness of COVID-19 policies can be studied further. Populations in colder regions such as North Dakota, South Dakota and Alaska will respond differently to SIPOS enacted in cold months (such as February) than those in California or Florida.

Appendix

See Tables 9 and 10, Fig. 5.

Table 9 Output, Input, and Hyperbolic Unconditional Order-m Estimates, Full Sample and 1st 100 Days

Table 10 Comparison of Author’s Estimates with Seiford and Zhu (2002) “Bad Output” and “Bad Input” Transformations

Fig. 5

Multiplication Factor by State and Day