Themed Section: COVID-19| Volume 25, ISSUE 5, P699-708, May 2022

Ok

# Quantifying the Effect of Public Activity Intervention Policies on COVID-19 Pandemic Containment Using Epidemiologic Data From 145 Countries

• Author Footnotes
∗ Jichao Sun, Yefeng Zheng, Wenhua Liang, Zifeng Yang, and Zhiqi Zeng contributed equally to this work.
Published:December 07, 2021

## Highlights

• The evidence regarding the effectiveness of different public activity intervention policies on coronavirus disease 2019 (COVID-19) containment is currently inconsistent.
• Earlier implementation, longer durations, and more strictness of intervention policies at the early but not middle stage were associated with reduced infections of COVID-19.
• A novel counterfactual estimator proved to be valid and reliable in estimating the quantitative effects of policy intervention on COVID-19 containment.

## Abstract

### Objectives

Most countries have adopted public activity intervention policies to control the coronavirus disease 2019 (COVID-19) pandemic. Nevertheless, empirical evidence of the effectiveness of different interventions on the containment of the epidemic was inconsistent.

### Methods

We retrieved time-series intervention policy data for 145 countries from the Oxford COVID-19 Government Response Tracker from December 31, 2019, to July 1, 2020, which included 8 containment and closure policies. We investigated the association of timeliness, stringency, and duration of intervention with cumulative infections per million population on July 1, 2020. We introduced a novel counterfactual estimator to estimate the effects of these interventions on COVID-19 time-varying reproduction number (Rt).

### Results

There is some evidence that earlier implementation, longer durations, and more strictness of intervention policies at the early but not middle stage were associated with reduced infections of COVID-19. The counterfactual model proved to have controlled for unobserved time-varying confounders and established a valid causal relationship between policy intervention and Rt reduction. The average intervention effect revealed that all interventions significantly decrease Rt after their implementation. Rt decreased by 30% (22%-41%) in 25 to 32 days after policy intervention. Among the 8 interventions, school closing, workplace closing, and public events cancellation demonstrated the strongest and most consistent evidence of associations.

### Conclusions

Our study provides more reliable evidence of the quantitative effects of policy interventions on the COVID-19 epidemic and suggested that stricter public activity interventions should be implemented at the early stage of the epidemic for improved containment.

## Introduction

As of July 2021, the severe acute respiratory syndrome coronavirus 2, causing the coronavirus disease 2019 (COVID-19), is still spreading globally.
Coronavirus COVID-19 global cases by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University (JHU).
The nonpharmaceutical interventions appear to be an important way that could reduce virus transmission until effective treatment regimens or mass immunizations are available.
• Fong M.W.
• Gao H.
• Wong J.Y.
• et al.
Nonpharmaceutical measures for pandemic influenza in nonhealthcare settings-social distancing measures.
,
• Ryu S.
• Gao H.
• Wong J.Y.
• et al.
Nonpharmaceutical measures for pandemic influenza in nonhealthcare settings-international travel-related measures.
These intervention strategies include swift surveillance, quarantine, and physical distancing measures such as school and workplace closing, internal and external travel restrictions, and stay at home requirements.
• Chu D.K.
• Akl E.A.
• Duda S.
• et al.
Physical distancing, face masks, and eye protection to prevent person-to-person transmission of SARS-CoV-2 and COVID-19: a systematic review and meta-analysis.
• Nussbaumer-Streit B.
• Mayr V.
• Dobrescu A.
• et al.
Quarantine alone or in combination with other public health measures to control COVID-19: a rapid review.
• Davies N.G.
• Kucharski A.J.
• Eggo R.M.
• Gimma A.
• Edmunds W.J.
Centre for the Mathematical Modelling of Infectious Diseases COVID-19 Working Group. Effects of non-pharmaceutical interventions on COVID-19 cases, deaths, and demand for hospital services in the UK: a modelling study.
• Flaxman S.
• Mishra S.
• Gandy A.
• et al.
Estimating the effects of non-pharmaceutical interventions on COVID-19 in Europe.
Almost all countries have adopted a series of containment and closure policies at different time points, and some seem to have curbed the virus transmission with varying degrees of success.
• Chowdhury R.
• Heng K.
• Shawon M.S.R.
• et al.
Dynamic interventions to control COVID-19 pandemic: a multivariate prediction modelling study comparing 16 worldwide countries.
• Jarvis C.I.
• Van Zandvoort K.
• Gimma A.
• et al.
Quantifying the impact of physical distance measures on the transmission of COVID-19 in the UK.
• Ngonghala C.N.
• Iboi E.
• Eikenberry S.
• et al.
Mathematical assessment of the impact of non-pharmaceutical interventions on curtailing the 2019 novel coronavirus.
Nevertheless, the quantitative evidence on the effectiveness of different intervention policies has been inconsistent.
Most early studies applied modeling assumptions using data within a single country to examine the effectiveness of interventions.
• Davies N.G.
• Kucharski A.J.
• Eggo R.M.
• Gimma A.
• Edmunds W.J.
Centre for the Mathematical Modelling of Infectious Diseases COVID-19 Working Group. Effects of non-pharmaceutical interventions on COVID-19 cases, deaths, and demand for hospital services in the UK: a modelling study.
,
• Jarvis C.I.
• Van Zandvoort K.
• Gimma A.
• et al.
Quantifying the impact of physical distance measures on the transmission of COVID-19 in the UK.
• Ngonghala C.N.
• Iboi E.
• Eikenberry S.
• et al.
Mathematical assessment of the impact of non-pharmaceutical interventions on curtailing the 2019 novel coronavirus.

Milne GJ, Xie S. The Effectiveness of Social Distancing in Mitigating COVID-19 Spread: a modelling analysis. Preprint. Posted online March 23, 2020. medRxiv 2020.03.20.20040055. https://doi.org/10.1101/2020.03.20.20040055.

• Lai S.
• Ruktanonchai N.W.
• Zhou L.
• et al.
Effect of non-pharmaceutical interventions to contain COVID-19 in China.
• Matrajt L.
• Leung T.
Evaluating the effectiveness of social distancing interventions to delay or flatten the epidemic curve of coronavirus disease.
For example, Lai et al
• Lai S.
• Ruktanonchai N.W.
• Zhou L.
• et al.
Effect of non-pharmaceutical interventions to contain COVID-19 in China.
developed a simulation framework using daily travel networks across China. They estimated that without nonpharmaceutical interventions, the COVID-19 cases would likely have increased 67-fold. Another research group developed an age-structured susceptible-exposed-infectious-removed model with data from medium-sized cities in the United States, suggesting that interventions that started earlier in the epidemic delayed the epidemic curve and interventions that started later flattened the epidemic curve.
• Matrajt L.
• Leung T.
Evaluating the effectiveness of social distancing interventions to delay or flatten the epidemic curve of coronavirus disease.
Nevertheless, those results were derived from mathematical assumptions under presumptive scenarios and thus could not be verified.
Data from individual countries suffered from its intrinsic incapability in quantifying and comparing the effects of different interventions. By now, evidence from comparative analysis using data from multiple countries is still inconsistent. The major issue in analyzing the causal relationships is that there exist unobserved confounders such as different testing capacities over time and heterogeneity across countries. Previous empirical studies leveraging straightforward statistical methods failed to address this problem.
• Liu Y.
• Morgenstern C.
• Kelly J.
• Lowe R.
• Jit M.
CMMID COVID-19 Working Group
The impact of non-pharmaceutical interventions on SARS-CoV-2 transmission across 130 countries and territories.
,
• Islam N.
• Sharp S.J.
• Chowell G.
• et al.
Physical distancing interventions and incidence of coronavirus disease 2019: natural experiment in 149 countries.
By early July 2020, with most Asian and European countries reaching the late stage of the first epidemic wave and the availability of detailed intervention information, we were able to retrospectively scrutinize the effects of interventions using more sophisticated statistical methods. In this study, we performed a comprehensive analysis of the effectiveness of the timeliness, stringency, and duration of 8 public intervention policies on COVID-19 containment using data from worldwide countries. Moreover, we introduced a novel counterfactual estimator based on the time-series COVID-19 epidemic data to quantify the effects of different interventions with less bias.

## Methods

### Data Sources and Selection

The daily confirmed cases for COVID-19 of each country were retrieved from https://ourworldindata.org from December 31, 2019, to July 1, 2020. The data were collected and reported by the health authority of each country. The country-based time-series data for the containment and closure policies were retrieved from the Oxford COVID-19 Government Response Tracker (https://github.com/OxCGRT/covid-policy-tracker) during the same time period. Details of the data collection and annotation have been described in a working article.
• Hale T.
• Noam A.
• Beatriz K.
• et al.
Variation in government responses to COVID-19. Blavatnik School of Government Working Paper.
In brief, a group of policy and government experts routinely collected information on public policies worldwide, including containment and closure interventions, and economic and healthcare supports. The policies of our interest were containment and closure interventions including 8 regimens, namely, school closing, workplace closing, public events cancellation, restrictions on gatherings, public transport closing, stay at home requirements, restrictions on internal movement, and international travel controlling. Each of the 8 interventions was recorded on an ordinal scale representing the level of strictness of the policy. Take workplace closing for example, 0 represents no measures; 1 represents recommending closing; 2 represents requiring closing for some sectors or categories of workers; and 3 represents requiring closing (or work from home) for all-but-essential workplaces. We selected countries with a time series longer than 90 days and the number of cumulative infections > 100 to reduce uncertainties. We coded each of the 8 interventions into 3 independent variables (Start-Date, Stringency, and Duration). Start-Date was recorded as the days of intervention commencement relative to the first 100 cases occurrence in that country, which served as an indicator for timely response to the pandemic. We selected the date of the first 100 cases instead of the first case as the start point because the detection of the first case was subject to more randomness that might introduce substantial noise in the following analysis. Stringency was measured as the average level of strictness across all days during a certain epidemic phase. Duration was measured as the number of days under intervention divided by the number of total days during a certain epidemic phase.

### Outcomes

The first outcome was the estimated cumulative infections per million population for each country on July 1, 2020. This outcome was used to investigate the correlations with different intervention variables. We chose July 1, 2020, as the endpoint because most Asian and European countries had reached the end of their first epidemic waves by late June or early July 2020 (see Appendix Fig. 1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.10.007). During this period, some governments lifted policy restrictions and the public loosened precautions, which in certain cases led to the resurgence of cases and deaths subsequently. Then, the governments reimposed policies in a policy seesaw as the epidemic waxed and waned.
• Hale T.
• Angrist N.
• Goldszmidt R.
• et al.
A global panel database of pandemic policies (Oxford COVID-19 Government Response Tracker).
,
• Looi M.K.
COVID-19: is a second wave hitting Europe?.
The lift and reimposition of policies and resurgence of the epidemic would affect the estimation of intervention effects. We aim to study the effect of policies within a single epidemic wave, with the consideration of reducing confounding and uncertainty. Nevertheless, there is no precise and consistent cutoff date for all countries. Therefore, we performed additional sensitivity analyses, by altering the cutoff date 1 month earlier (June 1, 2020) or later (August 1, 2020), to test the robustness of the results. The number of reported cases represents a straightforward metric reflecting the severity of the pandemic and has been widely used in previous articles for comparison between countries.
• Chaudhry R.
• Dranitsaris G.
• Mubashir T.
• Bartoszko J.
• Riazi S.
A country level analysis measuring the impact of government actions, country preparedness and socioeconomic factors on COVID-19 mortality and related health outcomes.
,
• Zweig S.A.
• Zapf A.J.
• Xu H.
• et al.
Impact of public health and social measures on the COVID-19 pandemic in the United States and other countries: descriptive analysis.
Nevertheless, the metric has limitations because the testing and reporting strategies are different across countries. In the current study, we chose the number of true infections rather than the number of reported cases as the metric for comparison. Given that the true infections were not known, we referred to the age-structured susceptible-exposed-infectious-removed model (https://github.com/mrc-ide/squire) developed by Imperial College London to estimate true infections. This Imperial College London model is among the most widely used approaches for estimations of true infections and has been recommended by Our World in Data for studying policy effects. In brief, the model fit data on confirmed deaths by using an estimated infection fatality rate to “back-calculate” how many infections would have occurred over the previous weeks to produce that number of deaths. It also accounted for mobility and testing rates data by country if available under a range of assumptions and epidemiological knowledge to generate a less biased infection estimate.
The second outcome was the time-varying reproduction number (Rt) for each country on each day. Rt is a measurement that represents the mean number of secondary cases that were infected by 1 index case. We used the median Rt estimates from the widely used EpiForecasts model (https://epiforecasts.io). The process of estimation was based on confirmed cases and deaths while accounting for uncertainties of the incubation period, the infection-to-confirmation delays, and the infection-to-death delays. The method of calculating Rt has been detailed in Cori et al.
• Cori A.
• Ferguson N.M.
• Fraser C.
• Cauchemez S.
A new framework and software to estimate time-varying reproduction numbers during epidemics.
In brief, the transmission rate of COVID-19 can be estimated by the ratio between new infections or deaths at time t and the infectious people at time t - d where d is the previous infection-to-confirmation delays or the infection-to-death delays as appropriate.
The missing data (ie, confirmed cases or policy intervention) in the middle of time series were linearly interpolated using nonmissing observations.

### Correlation Analysis

We conducted Spearman rank correlation analysis between Start-Date variable of each intervention and cumulative infection numbers using paired data from countries, to test whether the delayed policy implementation was associated with more infected cases. We chose Spearman rank correlation because the Start-Date variable (also the Stringency and Duration variables) has a highly skewed distribution. The correlation coefficient ranging from −1 to 1 is a statistical measure of the strength of a monotonic relationship between 2 variables. We also draw boxplots of the outcome by tertiles of Start-Date to examine the monotonic relationship. Because countries hit by the epidemic later may implement interventions timelier in view of the outbreaks in other countries, we performed an additional partial correlation analysis that was adjusted for the absolute date of first case occurrence.
We investigated the Spearman rank correlations of cumulative infection numbers with Stringency and Duration of interventions, separately at the early and middle stages of the epidemic. We adopted an approach proposed in a previous article

Batista M. Estimation of the final size of coronavirus epidemic by the logistic model. Preprint. Posted online February 28, 2020. medRxiv 2020.02.16.20023606. https://doi.org/10.1101/2020.02.16.20023606.

that divided the progression curve of COVID-19 into the early slow growth phase, the middle fast growth phase, and the late steady phase, using a data-driven phenomenological logistic model. The slope of the curve represents the rate of epidemic growth, which is used to divide different phases. Examples of the fitted curves and phase cutoffs are shown in Appendix Fig. 2 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.10.007. For sensitivity analysis, we alternatively defined the early phase as the first month since the first confirmed case occurrence for each country.

### Counterfactual Effect Estimates

First, we compared the trends in the Rt before and after the implementation of each intervention for descriptive purpose. We calculated the mean value (95% confidence intervals [CIs]) of the Rt for all countries on different days relative to intervention implementation. As a summarization, we defined the commence date of any intervention as the median date of different intervention initiation dates if available and thus generated a new variable named “any intervention.” This simple “averaging method” provided a direct way for us to inspect the effect of interventions on Rt.
We introduced a new counterfactual estimator to infer causal relationships between interventions and Rt using time-series cross-sectional data (see Appendix Fig. 3 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.10.007). Counterfactual estimator compares the observed outcomes with those one would expect if the intervention had not been implemented. In brief, the counterfactual estimator first constructs a model using time-series observations in the preintervention period and then takes observations under intervention as missing data and directly estimates their counterfactuals. This method has been detailed in previous articles and shown to provide more reliable causal effects than the conventional linear 2-way fixed effect models when the intervention effect is heterogeneous among units or there exist unobserved time-varying confounders.

Liu L, Wang Y, Xu Y. A practical guide to counterfactual estimators for causal inference with time-series cross-sectional data. Preprint. Posted online April 10, 2020. SSRN 3555463. https://doi.org/10.2139/ssrn.3555463.

Therefore, the improved model was able to take account of the influences of time factor and country (unit) factor (such as population and Gross Domestic Product) on outcomes.
The original article

Liu L, Wang Y, Xu Y. A practical guide to counterfactual estimators for causal inference with time-series cross-sectional data. Preprint. Posted online April 10, 2020. SSRN 3555463. https://doi.org/10.2139/ssrn.3555463.

provided 3 sets of counterfactual estimators and we chose the improved interactive 2-way fixed-effects model because of its ability to deal with time-varying confounders, as suggested by the author.
The model is as follows:
For any i = 1, 2, …, N and t = 1, 2, …, T,
$Yit=δitDit+Xit′β+λi′ft+αi+ξt+εit$

where Yit is the outcome (Rt) for country i at time t; Dit is intervention indicator that equals 1 if country i is under intervention at time t and equals 0 otherwise; δit is the intervention effect on country i at time t; Xit is a (p×1) vector of exogenous covariates; β is a (p×1) vector of unknown parameters; ft is an (r × 1) vector of unobserved common factors; and λi is an (r × 1) vector of unknown factor loadings. Intuitively, factors can be understood as time-varying trends that affect each country differently, and factor loadings capture their heterogeneous impacts caused by each country’s various unobserved characteristics. Here, the interactive component $λi′ft$ implicitly captures the effects of unobserved time-varying confounders and the effects of other policies, through which the influence of other policies is eliminated or controlled when performing estimations of δit. αi and ξt are additive country and time fixed effects, respectively, and εit represents unobserved idiosyncratic shocks for country i at time t and has 0 mean.
The primary causal quantity of interest is the average intervention effect, which is an approximation of the estimated effects of the intervention on the outcome after policy implementation over time. Details about the calculation of average intervention effect were shown in Appendix Method 1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.10.007.

## Results

Overall, time-series data of the intervention policies were retrieved for 178 countries on July 1, 2020, from the Oxford COVID-19 Government Response Tracker. After exclusion, a total of 145 countries were included in the current study. The number of countries in different continents is as follows: 36 in Europe, 36 in Asia, 47 in Africa, 13 in North America, 11 in South America, and 2 in Oceania. Cumulative infections per million population on July 1 ranges from 46 (Burundi) to 212 154 (Peru), with a median value of 9867 (interquartile range 2655-30 581).

### Association of Intervention Start-Date With Cumulative Infections Per Million Population

Correlations between the Start-Date of 8 interventions and cumulative infections per million population are shown in Table 1. Start-Dates of all 8 interventions were significantly and positively associated with the outcome, suggesting that the later intervention was commenced, the more infected cases that would be expected in that country. Start-Dates of public events cancellation (correlation coefficient [r] = 0.45), school closing (r = 0.43), and international travel controls (r = 0.43) showed the most pronounced associations. We displayed the boxplots of the outcome by tertiles of Start-Date in Figure 1.
Table 1Correlations of the Start-Date for 8 interventions with cumulative infections per million population on July 1, 2020.
InterventionsCorrelation coefficients95% CI
School closing0.430.28-0.55
Workplace closing0.280.12-0.42
Public events cancellation0.450.31-0.57
Restrictions on gatherings0.320.17-0.46
Public transport closing0.220.06-0.37
Stay at home requirements0.270.12-0.42
Restrictions on internal movement0.270.12-0.42
International travel controls0.430.29-0.55
Note. The correlation coefficients were calculated using Spearman rank correlation analysis. Start-Date for each intervention was the days of intervention initiation relative to the date of first cumulative 100 cases occurrence.
CI indicates confidence interval.
The distributions of cumulative infections by tertitles of Start-Date for 8 interventions are presented in Figure 1. A similar monotonic increasing trend of cumulative infections along with Start-Date tertiles was observed for all interventions. Countries in tertile 3 demonstrated notably more infections and wider distributions than those in tertiles 1 and 2.
Additional partial correlation analysis that was adjusted for the absolute date of first case occurrence did not change the results substantially (see Appendix Table 1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.10.007). Sensitivity analysis by changing the definition of Start-Date variable to days relative to the first 10 cases occurrence showed that the positive associations still persist in most cases, although some are not significant (see Appendix Table 2 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.10.007).

### Association of Intervention Stringency and Duration With Cumulative Infections Per Million Population

The associations of intervention Stringency and Duration with cumulative infections per million population at different epidemic phases are presented in Table 2 and Appendix Table 3 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.10.007. Most of the Stringency and Duration variables for the 8 interventions in the early phase (slow growth period) were negatively correlated with the outcome, with some of them showing significance (Table 2). Nevertheless, during the middle phase (the fast growth phase), the Stringency and Duration were mostly positively correlated with the outcome. The average duration of the early phase for all countries was 61 days. We conducted a further analysis by calculating Stringency and Duration in the first month and the second month of the epidemic, respectively. Similar results were found that the Stringency and Duration variables in the first month were mostly negatively, whereas in the second month were mostly positively, associated with cumulative infections (see Appendix Table 3 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.10.007).
Table 2Correlations of the Stringency and Duration for 8 interventions with cumulative infections per million population on July 1, 2020.
InterventionsStringencyDuration
Correlation coefficient95% CICorrelation coefficient95% CI
Early phase
School closing−0.14−0.30 to 0.02−0.15−0.31 to 0.01
Workplace closing−0.05−0.11 to 0.21−0.02−0.18 to 0.15
Public events cancellation−0.16−0.31 to 0.00−0.18−0.33 to −0.02
Restrictions on gatherings−0.02−0.18 to 0.14−0.10−0.26 to 0.06
Public transport closing0.04−0.13 to 0.200.04−0.13 to 0.20
Stay at home requirements−0.05−0.12 to 0.21−0.01−0.18 to 0.15
Restrictions on internal movement−0.03−0.14 to 0.19−0.02−0.14 to 0.18
International travel controls−0.18−0.33 to −0.02−0.23−0.38 to −0.07
Middle phase
School closing0.11−0.07 to 0.270.170.00-0.34
Workplace closing0.340.17-0.480.340.18-0.48
Public events cancellation0.15−0.02 to 0.310.15−0.02 to 0.32
Restrictions on gatherings0.340.18-0.480.220.05-0.38
Public transport closing0.15−0.02 to 0.320.250.08-0.41
Stay at home requirements0.180.01-0.340.240.07-0.39
Restrictions on internal movement0.260.10-0.420.330.17-0.47
International travel controls−0.03−0.20 to 0.150.03−0.15 to 0.20
Note. The correlation coefficients were calculated using Spearman correlation analysis, separately in the early phase and middle phase.
CI indicates confidence interval.
The results suggested some evidence that the longer and stricter implementations of some interventions in the very early phase but not the middle phase were associated with reductions in infected cases at the end.

### Rt Before and After the Interventions

The COVID-19 Rt before and after the implementation of 8 interventions is presented in Figure 2. Overall, a consistent similar pattern was observed for all interventions that Rt decreased slowly before the intervention, yet after the intervention, Rt decreased rapidly in 7 to 14 days, and the decreasing trend attenuated afterward. Rt converged to around 1 in approximately 30 days after the intervention. Overall, the average Rt decreased by 6.7% (95% CI 4.8-12.4) at 7 days and by 17.0% (95% CI 7.8-29.1) at 14 days after any of the interventions (see Appendix Fig. 4 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.10.007).

### Counterfactual Estimates for the Effects of Interventions on Rt

With counterfactual estimators, the average effects of different interventions by the time are presented in Figure 3, and the average values for all periods after intervention are presented in Table 3. All interventions give average estimates significantly < 0, among which the estimate of international travel controls is marginally significant. The test for no pretrend results is shown in Appendix Figure 5 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.10.007. All 8 interventions have passed the equivalence test, suggesting the model successfully controlled for the effects of time-varying confounders and other interventions. In most cases, the average effect estimates surround zero in the preintervention period and decrease rapidly to below zero in 7 to 14 days after the intervention, to its minimum values (ranging from −0.52 to −0.08 with a median of −0.30) in 25 to 32 days (Fig. 3). This corresponds to a maximum 22% to 41% reduction in Rt. Among the 8 interventions, school closing, workplace closing, and public events cancellation demonstrated the strongest and most consistent evidence of associations.
Table 3Counterfactual estimates for the average effects of 8 interventions on Rt.
InterventionAIE on Rt95% CIP value
School closing−0.29−0.40 to −0.195.28E-08
Workplace closing−0.29−0.38 to −0.205.28E-11
Public events cancellation−0.39−0.52 to −0.277.12E-10
Restrictions on gatherings−0.24−0.35 to −0.147.63E-06
Public transport closing−0.11−0.20 to −0.033.55E-03
Stay at home requirements−0.17−0.25 to −0.081.02E-04
Restrictions on internal movement−0.21−0.28 to −0.147.58E-09
International travel controls−0.20−0.39 to −0.023.16E-02
Note. The data show the average effects of intervention policies on Rt for all countries and across postintervention periods.
AIE indicates average intervention effect; CI, confidence interval.

### Robustness Analyses by Altering the Endpoint Date

Appendix Tables 4 to 7 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.10.007 provide the robustness analyses for correlation results between variables of interventions and cumulative infections, and Appendix Tables 8 and 9 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.10.007 provide the robustness analyses for counterfactual effect estimates on Rt, by altering the cutoff date 1 month earlier (June 1, 2020) or later (August 1, 2020). The results demonstrate that the abovementioned findings are roughly unchanged, although the estimates differ to some extent.

## Discussion

In this study, we found some evidence that earlier implementation, longer durations, and more strictness of containment policies at the early stage but not middle stage were associated with reduced infections of COVID-19. With a novel counterfactual estimator, we were able to control for the unobserved time-varying confounders, generating more reliable causal relationships. Our results showed that the government intervention policies were associated with a 22% to 41% reduction in COVID-19 transmission in approximately 25 to 32 days after their implementations.

### Comparison With Previous Studies

The findings from our work align with those from previous studies,
• Chowdhury R.
• Heng K.
• Shawon M.S.R.
• et al.
Dynamic interventions to control COVID-19 pandemic: a multivariate prediction modelling study comparing 16 worldwide countries.
• Matrajt L.
• Leung T.
Evaluating the effectiveness of social distancing interventions to delay or flatten the epidemic curve of coronavirus disease.
,
• Islam N.
• Sharp S.J.
• Chowell G.
• et al.
Physical distancing interventions and incidence of coronavirus disease 2019: natural experiment in 149 countries.
,
• Kraemer M.U.G.
• Yang C.H.
• Gutierrez B.
• et al.
The effect of human mobility and control measures on the COVID-19 epidemic in China.
• Viner R.M.
• Russell S.J.
• Croker H.
• et al.
School closure and management practices during coronavirus outbreaks including COVID-19: a rapid systematic review.
• Wong C.K.H.
• Wong J.Y.H.
• Tang E.H.M.
• Au C.H.
• Lau K.T.K.
• Wai A.K.C.
Effects of national containment measures on decelerating the increase in daily new cases of COVID-19 in 54 countries and 4 epicenters of pandemic: Comparative Observational Study.
• Cowling B.J.
• Ali S.T.
• Ng T.W.Y.
• et al.
Impact assessment of non-pharmaceutical interventions against coronavirus disease 2019 and influenza in Hong Kong: an observational study.
• Tian H.
• Liu Y.
• Li Y.
• et al.
An investigation of transmission control measures during the first 50 days of the COVID-19 epidemic in China.
except that previous results mostly depended on modeling assumptions under presumptive scenarios or used data within a single country. Only a few studies assessed the impact of intervention policies for different countries using comparative methods.
• Liu Y.
• Morgenstern C.
• Kelly J.
• Lowe R.
• Jit M.
CMMID COVID-19 Working Group
The impact of non-pharmaceutical interventions on SARS-CoV-2 transmission across 130 countries and territories.
,
• Islam N.
• Sharp S.J.
• Chowell G.
• et al.
Physical distancing interventions and incidence of coronavirus disease 2019: natural experiment in 149 countries.
,
• Wong C.K.H.
• Wong J.Y.H.
• Tang E.H.M.
• Au C.H.
• Lau K.T.K.
• Wai A.K.C.
Effects of national containment measures on decelerating the increase in daily new cases of COVID-19 in 54 countries and 4 epicenters of pandemic: Comparative Observational Study.
Nevertheless, these studies mainly depended on straightforward statistical methods, simply relating intervention policies to COVID-19 growth rate or Rt directly, which failed to account for time-varying confounders that affected the effect estimates. One study comparing the COVID-19 curve trends before and after interventions using data from 54 countries suggested that stay at home orders, curfews, and lockdowns curbed the increase in daily new case to < 5% within a month.
• Wong C.K.H.
• Wong J.Y.H.
• Tang E.H.M.
• Au C.H.
• Lau K.T.K.
• Wai A.K.C.
Effects of national containment measures on decelerating the increase in daily new cases of COVID-19 in 54 countries and 4 epicenters of pandemic: Comparative Observational Study.
Another study including 149 countries leveraged a simple meta-analysis method and synthesized the incidence rate ratios of COVID-19 before and after the implementation of physical distancing, concluding that physical distancing was associated with a 13% reduction in COVID-19 incidence.
• Islam N.
• Sharp S.J.
• Chowell G.
• et al.
Physical distancing interventions and incidence of coronavirus disease 2019: natural experiment in 149 countries.
This study had less focus on the timeliness, strictness, and durations of interventions and was thus not able to conclude causal relationships. To the best of our knowledge, our study is the first study that addressed the issue of confounding using a novel counterfactual estimator based on an interactive 2-way fixed-effects model. The results from our study provided more reliable evidence and could better assist policy making.

### Interpretation of Our Findings

We found that the early implementation of all containment policies was associated with reduced infection cases. This finding was as expected and in concert with most previous studies.
• Matrajt L.
• Leung T.
Evaluating the effectiveness of social distancing interventions to delay or flatten the epidemic curve of coronavirus disease.
,
• Ainslie K.E.C.
• Walters C.E.
• Fu H.
• et al.
Evidence of initial success for China exiting COVID-19 social distancing policy after achieving containment.
• Lee V.J.
• Chiew C.J.
• Khong W.X.
Interrupting transmission of COVID-19: lessons from containment efforts in Singapore.
• Koo J.R.
• Cook A.R.
• Park M.
• et al.
Interventions to mitigate early spread of SARS-CoV-2 in Singapore: a modelling study [published correction appears in Lancet Infect Dis. 2020;20(5):e79].
Alongside this finding, we also found some evidence that the higher stringency and longer duration of some containment policies at the early or slow growth stage were correlated with reduced infection cases. Nevertheless, results from the middle or fast growth stage suggested evidence of positive associations. This is a novel finding that previous studies did not address. The positive associations between Stringency and Duration of intervention in the middle stage and total infections were probably attributed to reverse causality, which means that some countries strengthened and prolonged the interventions in face of more severe situations.
Recently, the new variants of severe acute respiratory syndrome coronavirus 2, especially the Delta variant that is more contagious than the original strain, spread rapidly in some countries such as India and the United States. Evidence regarding whether government intervention policies are effective on containing the new variants is scarce because most countries have loosened restrictions on public activities. Nevertheless, a recent regional outbreak of the Delta variant attacked Guangzhou and Shenzhen in China from May 2021 to June 2021. The local governments immediately enforced strict control measures since the identification of the first new case, including public events cancellation, unnecessary workplace closing, and contact tracing. The regional outbreak was successfully controlled within a month, preventing virus spillover and large-scale spreading.
• Zhang M.
• Xiao J.
• Deng A.
• et al.
Transmission dynamics of an outbreak of the COVID-19 Delta variant B.1.617.2 - Guangdong Province, China, May-June 2021.
On the contrary, countries that did not implement strict containment measures at the very beginning are experiencing an uncontrollable domestic outbreak.

Salvatore M, Bhattacharyya R, Purkayastha S, et al. Resurgence of SARS-CoV-2 in India: potential role of the B.1.617.2 (Delta) variant and delayed interventions. Preprint. Posted online June 30, 2021. medRxiv 2021.06.23.21259405. https://doi.org/10.1101/2021.06.23.21259405.

This provides us some preliminary evidence that the early and stringent interventions are effective in controlling the outbreak of new variants. Given the strong transmissibility of the new variants, governments should enforce aggressive control measures as early as possible, even though the growth rate might be very slow in their countries at the early period. It seems too late to remedy when arriving at the fast growth stage.
The descriptive results from Figure 2 show that Rt demonstrated a decreasing trend before the intervention, which suggested that apart from the 8 interventions, some other unobserved factors such as public self-protective measures also had an effect on transmission reduction. Nevertheless, the preintervention period decreasing trend disappeared in Figure 3 with our counterfactual estimator, suggesting that our methods had successfully eliminated the effects of unobserved confounding factors and generated less biased effect estimates. Notably, we observed the strongest and consistent effects for school closing, workplace closing, and public events cancellation. All 3 containment policies were mandatory policies and more likely to take effects because it is easier to close public facilities.
Quantitatively, we found most interventions took their effects on reducing Rt rapidly about 7 to 14 days after implementation. The effects were strengthened by time to a maximum effect of around 30% reduction for Rt in 25 to 32 days. The estimates were similar to a previous study,
• Lauer S.A.
• Grantz K.H.
• Bi Q.
• et al.
The incubation period of coronavirus Disease 2019 (COVID-19) from publicly reported confirmed cases: estimation and application.
except that our results provided the effect trends over time.

### Limitations

Our study does have several limitations. First, the coding of intervention variables from Oxford COVID-19 Government Response Tracker relied on government announcements. Nevertheless, announcements did not guarantee mandatory implementation and people adherence varied because of the cultural and legal system differences. Second, because of the relatively small sample size for the number of countries, not all correlation analyses are significant, especially for Stringency and Duration; hence, those findings need to be interpreted with caution. Third, in addition to the public containment and closure policies, other personal protection strategies including wearing masks, quarantine, and hand hygiene also played an important role in epidemic mitigation. Those strategies were not the focus of our current study and have been addressed in previous researches.
• Chu D.K.
• Akl E.A.
• Duda S.
• et al.
Physical distancing, face masks, and eye protection to prevent person-to-person transmission of SARS-CoV-2 and COVID-19: a systematic review and meta-analysis.
,
• Nussbaumer-Streit B.
• Mayr V.
• Dobrescu A.
• et al.
Quarantine alone or in combination with other public health measures to control COVID-19: a rapid review.
,
• Hellewell J.
• Abbott S.
• Gimma A.
• et al.
Feasibility of controlling COVID-19 outbreaks by isolation of cases and contacts [published correction appears in Lancet Glob Health. 2020;9(5):e597].
• Ferretti L.
• Wymant C.
• Kendall M.
• et al.
Quantifying SARS-CoV-2 transmission suggests epidemic control with digital contact tracing.
• Meng S.
• Wu Y.J.
• et al.
Epidemiology, causes, clinical manifestation and diagnosis, prevention and control of coronavirus disease (COVID-19) during the early outbreak period: a scoping review.
Moreover, some intervention policies were often introduced in close temporal sequences. It has been difficult to untangle the individual effects. Although our counterfactual approach proved to have been largely controlled for the effects of time-varying confounders and other policies, statistical models dealing with confounders might be not perfect. Hence, results of interventions that are temporally correlated need to be interpreted with caution. Fourth, a large proportion of confirmed cases and deaths has been recorded from nursing homes including both residents and care workers.
• Abrams H.R.
• Loomer L.
• Gandhi A.
• Grabowski D.C.
Characteristics of U.S. Nursing homes with COVID-19 cases.
Nevertheless, data regarding the fraction of cases and deaths emerging from nursing homes were not available for most countries. We were not able to investigate the effects of the general policy interventions on nursing home epidemics at this point.

## Conclusions

Using epidemiological data from 145 countries, we found some evidence that earlier, stricter, and longer implementation of containment policies at the early stage was associated with a reduction in infected cases. Moreover, the novel counterfactual estimator proved to have generated more reliable intervention effect estimates of policies. Our results provided evidence of the quantitative effect of different policy intervention over time. Those findings shall have important implications for governments to enact or lift containment policies in fighting against the current and future waves of the COVID-19 outbreak. Future studies should emphasize on how adding and removing intervention policies affect the transmission of the virus, especially its new mutants such as the Delta variant, for decision making on lifting containment policies.

## Article and Author Information

Author Contributions: Concept and design: Sun, Zheng, Liang, Yang, Li, Luo, He, Zhong
Acquisition of data: Sun, Luo
Analysis and interpretation of data: Sun, Zheng, Liang, Yang, Zeng, Li, Luo
Drafting of the manuscript: Sun, Zheng, Zeng, Luo
Critical revision of the paper for important intellectual content: Sun, Zheng, Liang, Yang, Zeng, Li, Luo, Alexander Ng, He
Statistical analysis: Sun, Luo
Administrative, technical, or logistic support: Alexander Ng, Zhong
Supervision: Alexander Ng, He, Zhong
Conflict of Interest Disclosures: The authors reported no conflicts of interest.
Funding/Support: This work was funded by Key-Area Research and Development Program of Guangdong Province, China (No. 2018B010111001).

## Supplemental Material

• Supplementary information

Appendix Method 1. Calculation of the average intervention effects (AIEs).

Appendix Table 1. Partial correlations (adjusted for the date of the first case) between the Start-Date (days relative to date of the first 100 cases occurrence) for 8 interventions and cumulative infections per million population on July 1, 2020.

Appendix Table 2. Correlations between the Start-Date (days relative to date of the first 10 cases occurrence) for 8 interventions and cumulative infections per million population on July 1, 2020.

Appendix Table 3. Correlations between the Stringency and Duration for 8 interventions (for the first month and second month) and cumulative infections per million population on July 1, 2020.

Appendix Table 4. Robustness analysis: correlations between the Start-Date (relative to date of the first 100 cases occurrence) for 8 interventions and cumulative infections per million population on June 1, 2020.

Appendix Table 5. Robustness analysis: correlations between the Start-Date (relative to date of the first 100 cases occurrence) for 8 interventions and cumulative infections per million population on August 1, 2020.

Appendix Table 6. Robustness analysis: correlations between the Stringency and Duration for 8 interventions (for the early phase and middle phase) and cumulative infections per million population on June 1, 2020.

Appendix Table 7. Robustness analysis: correlations between the Stringency and Duration for 8 interventions (for the early phase and middle phase) and cumulative infections per million population on August 1, 2020.

Appendix Table 8. Robustness analysis: counterfactual estimates for the average effects of 8 interventions on Rt through June 1, 2020.

Appendix Table 9. Robustness analysis: counterfactual estimates for the average effects of 8 interventions on Rt through August 1, 2020.

Appendix Figure 1. Country examples of trajectories for COVID-19 pandemic.

Appendix Figure 2. Illustrations of COVID-19 growth curve with different phases.

Appendix Figure 3. An illustration of time-series cross-sectional data for School closing.

Appendix Figure 4. Rt before and after the implementation of any intervention.

Appendix Figure 5. Tests for no pretrend analysis.

## References

1. Coronavirus COVID-19 global cases by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University (JHU).
(Available at:)
https://coronavirus.jhu.edu/map.html
Date accessed: July 10, 2021
• Fong M.W.
• Gao H.
• Wong J.Y.
• et al.
Nonpharmaceutical measures for pandemic influenza in nonhealthcare settings-social distancing measures.
Emerg Infect Dis. 2020; 26: 976-984
• Ryu S.
• Gao H.
• Wong J.Y.
• et al.
Nonpharmaceutical measures for pandemic influenza in nonhealthcare settings-international travel-related measures.
Emerg Infect Dis. 2020; 26: 961-966
• Chu D.K.
• Akl E.A.
• Duda S.
• et al.
Physical distancing, face masks, and eye protection to prevent person-to-person transmission of SARS-CoV-2 and COVID-19: a systematic review and meta-analysis.
Lancet. 2020; 395: 1973-1987
• Nussbaumer-Streit B.
• Mayr V.
• Dobrescu A.
• et al.
Quarantine alone or in combination with other public health measures to control COVID-19: a rapid review.
Cochrane Database Syst Rev. 2020; 9CD013574
• Davies N.G.
• Kucharski A.J.
• Eggo R.M.
• Gimma A.
• Edmunds W.J.
Centre for the Mathematical Modelling of Infectious Diseases COVID-19 Working Group. Effects of non-pharmaceutical interventions on COVID-19 cases, deaths, and demand for hospital services in the UK: a modelling study.
Lancet Public Health. 2020; 5: e375-e385
• Flaxman S.
• Mishra S.
• Gandy A.
• et al.
Estimating the effects of non-pharmaceutical interventions on COVID-19 in Europe.
Nature. 2020; 584: 257-261
• Chowdhury R.
• Heng K.
• Shawon M.S.R.
• et al.
Dynamic interventions to control COVID-19 pandemic: a multivariate prediction modelling study comparing 16 worldwide countries.
Eur J Epidemiol. 2020; 35: 389-399
• Jarvis C.I.
• Van Zandvoort K.
• Gimma A.
• et al.
Quantifying the impact of physical distance measures on the transmission of COVID-19 in the UK.
BMC Med. 2020; 18: 124
• Ngonghala C.N.
• Iboi E.
• Eikenberry S.
• et al.
Mathematical assessment of the impact of non-pharmaceutical interventions on curtailing the 2019 novel coronavirus.
Math Biosci. 2020; 325108364
2. Milne GJ, Xie S. The Effectiveness of Social Distancing in Mitigating COVID-19 Spread: a modelling analysis. Preprint. Posted online March 23, 2020. medRxiv 2020.03.20.20040055. https://doi.org/10.1101/2020.03.20.20040055.

• Lai S.
• Ruktanonchai N.W.
• Zhou L.
• et al.
Effect of non-pharmaceutical interventions to contain COVID-19 in China.
Nature. 2020; 585: 410-413
• Matrajt L.
• Leung T.
Evaluating the effectiveness of social distancing interventions to delay or flatten the epidemic curve of coronavirus disease.
Emerg Infect Dis. 2020; 26: 1740-1748
• Liu Y.
• Morgenstern C.
• Kelly J.
• Lowe R.
• Jit M.
• CMMID COVID-19 Working Group
The impact of non-pharmaceutical interventions on SARS-CoV-2 transmission across 130 countries and territories.
BMC Med. 2021; 19: 40
• Islam N.
• Sharp S.J.
• Chowell G.
• et al.
Physical distancing interventions and incidence of coronavirus disease 2019: natural experiment in 149 countries.
BMJ. 2020; 370: m2743
• Hale T.
• Noam A.
• Beatriz K.
• et al.
Variation in government responses to COVID-19. Blavatnik School of Government Working Paper.
• Hale T.
• Angrist N.
• Goldszmidt R.
• et al.
A global panel database of pandemic policies (Oxford COVID-19 Government Response Tracker).
Nat Hum Behav. 2021; 5: 529-538
• Looi M.K.
COVID-19: is a second wave hitting Europe?.
BMJ. 2020; 371: m4113
• Chaudhry R.
• Dranitsaris G.
• Mubashir T.
• Bartoszko J.
• Riazi S.
A country level analysis measuring the impact of government actions, country preparedness and socioeconomic factors on COVID-19 mortality and related health outcomes.
EClinicalMedicine. 2020; 25100464
• Zweig S.A.
• Zapf A.J.
• Xu H.
• et al.
Impact of public health and social measures on the COVID-19 pandemic in the United States and other countries: descriptive analysis.
JMIR Public Health Surveill. 2021; 7e27917
• Cori A.
• Ferguson N.M.
• Fraser C.
• Cauchemez S.
A new framework and software to estimate time-varying reproduction numbers during epidemics.
Am J Epidemiol. 2013; 178: 1505-1512
3. Batista M. Estimation of the final size of coronavirus epidemic by the logistic model. Preprint. Posted online February 28, 2020. medRxiv 2020.02.16.20023606. https://doi.org/10.1101/2020.02.16.20023606.

4. Liu L, Wang Y, Xu Y. A practical guide to counterfactual estimators for causal inference with time-series cross-sectional data. Preprint. Posted online April 10, 2020. SSRN 3555463. https://doi.org/10.2139/ssrn.3555463.

• Kraemer M.U.G.
• Yang C.H.
• Gutierrez B.
• et al.
The effect of human mobility and control measures on the COVID-19 epidemic in China.
Science. 2020; 368: 493-497
• Viner R.M.
• Russell S.J.
• Croker H.
• et al.
School closure and management practices during coronavirus outbreaks including COVID-19: a rapid systematic review.
Lancet Child Adolesc. 2020; 4: 397-404
• Wong C.K.H.
• Wong J.Y.H.
• Tang E.H.M.
• Au C.H.
• Lau K.T.K.
• Wai A.K.C.
Effects of national containment measures on decelerating the increase in daily new cases of COVID-19 in 54 countries and 4 epicenters of pandemic: Comparative Observational Study.
J Med Internet Res. 2020; 22e19904
• Cowling B.J.
• Ali S.T.
• Ng T.W.Y.
• et al.
Impact assessment of non-pharmaceutical interventions against coronavirus disease 2019 and influenza in Hong Kong: an observational study.
Lancet Public Health. 2020; 5: e279-e288
• Tian H.
• Liu Y.
• Li Y.
• et al.
An investigation of transmission control measures during the first 50 days of the COVID-19 epidemic in China.
Science. 2020; 368: 638-642
• Ainslie K.E.C.
• Walters C.E.
• Fu H.
• et al.
Evidence of initial success for China exiting COVID-19 social distancing policy after achieving containment.
Wellcome Open Res. 2020; 5: 8
• Lee V.J.
• Chiew C.J.
• Khong W.X.
Interrupting transmission of COVID-19: lessons from containment efforts in Singapore.
J Travel Med. 2020; 27taaa039
• Koo J.R.
• Cook A.R.
• Park M.
• et al.
Interventions to mitigate early spread of SARS-CoV-2 in Singapore: a modelling study [published correction appears in Lancet Infect Dis. 2020;20(5):e79].
Lancet Infect Dis. 2020; 20: 678-688
• Zhang M.
• Xiao J.
• Deng A.
• et al.
Transmission dynamics of an outbreak of the COVID-19 Delta variant B.1.617.2 - Guangdong Province, China, May-June 2021.
China CDC Wkly. 2021; 3: 584-586
5. Salvatore M, Bhattacharyya R, Purkayastha S, et al. Resurgence of SARS-CoV-2 in India: potential role of the B.1.617.2 (Delta) variant and delayed interventions. Preprint. Posted online June 30, 2021. medRxiv 2021.06.23.21259405. https://doi.org/10.1101/2021.06.23.21259405.

• Lauer S.A.
• Grantz K.H.
• Bi Q.
• et al.
The incubation period of coronavirus Disease 2019 (COVID-19) from publicly reported confirmed cases: estimation and application.
Ann Intern Med. 2020; 172: 577-582
• Hellewell J.
• Abbott S.
• Gimma A.
• et al.
Feasibility of controlling COVID-19 outbreaks by isolation of cases and contacts [published correction appears in Lancet Glob Health. 2020;9(5):e597].
Lancet Glob Health. 2020; 8: e488-e496
• Ferretti L.
• Wymant C.
• Kendall M.
• et al.
Quantifying SARS-CoV-2 transmission suggests epidemic control with digital contact tracing.
Science. 2020; 368eabb6936
• Meng S.
• Wu Y.J.
• et al.
Epidemiology, causes, clinical manifestation and diagnosis, prevention and control of coronavirus disease (COVID-19) during the early outbreak period: a scoping review.
Infect Dis Pover. 2020; 9: 29
• Abrams H.R.
• Loomer L.
• Gandhi A.
• Grabowski D.C.
Characteristics of U.S. Nursing homes with COVID-19 cases.
J Am Geriatr Soc. 2020; 68: 1653-1656