Quantifying Vaccine Effectiveness in Preventing Severe COVID-19

The effectiveness of the three vaccines against SARS-CoV-2 authorized in the US were gauged in their prevention of COVID-19 admissions to the hospital, intensive care unit (ICU) and emergency department, using records from the VISION Network of health care systems and research centers established by the CDC.

Mark Thompson, PhD, deputy chief, Science, Epidemiology and Prevention Branch, and COVID-19 Response Team, CDC, and colleagues point out that while the phase 3 trials evidenced high effectiveness of vaccines in preventing symptomatic COVID-19, less was known about how well the vaccines protect against more severe illness that leads to hospitalization, admission to an ICU, or to an emergency department or urgent care visit.

In an editorial accompanying the study, Natalie Dean, PhD, Department of Bioinformatics, Rollins School of Public Health, Emory University, Atlanta, and colleagues agree with the need to evaluate these 3 distinct outcome measures, as they reflect, "the combined benefit of vaccines to prevent infection with SARS-CoV-2 and reduce subsequent progression to medically attended disease."

The investigators designed the current study to also better account for patients who are disproportionately affected by COVID-19, including older adults, persons with chronic medical conditions, and Black or Hispanic populations."Data are lacking from real-world estimates of effectiveness of all three COVID-19 vaccines authorized in the United States," Thompson and colleagues observed.

The investigators accessed the records of over 41,000 hospital admissions for symptoms consistent with COVID-19 in 187 hospitals and over 21,000 visits to 221 emergency departments or urgent care clinics during the period from January 1 through June 22, 2021. In the test-negative design, the cases were those with a positive molecular test for SARS CoV-2 and the control group were those with negative tests.The vaccine effectiveness emerged in comparing the odds of vaccination between cases and controls.

"The test-negative design has been routinely used to estimate vaccine effectiveness against seasonal influenza, but its application in studies of COVID-19, although increasingly common, is new," Dean and colleagues point out.

The investigators report finding effectiveness of full mRNA vaccination, ≥14 days after 2nd dose, to be 89% (95% CI, 87-91) against infection leading to hospitalization; 90% (85-93) against an ICU admission; and 91% (89-93) against an emergency department or urgent care clinic.The Pfizer/BioNTech and Moderna mRNA vaccines were similarly effective in preventing these outcomes; with range of 81% to 95% among adults 85 years of age or older, persons with chronic medical conditions, and Black or Hispanic adults.

The Johnson & Johnson (Janssen) vaccine was found 68% (95%CI, 50-79) effective against hospitalization and 73% (59-82) against infection leading to an emergency department of urgent care clinic visit.The vaccine was used in 5 of the network health care systems, with insufficient data to assess effectiveness in preventing transfer to ICU.

Thompson and colleagues indicate that this study expanded knowledge of the clinical protection offered by the vaccines in 4 important areas: for the understudied populations of persons 85 years of age or older; and for Black adults and Hispanic adults; for finding durations of vaccine effectiveness at least the 112 days after the second dose, which was the longest interval after vaccination studied; and for deriving estimates of effectiveness of all 3 vaccines authorized in the US across the same large population and time period.

Dean and colleagues also offered 4 points to consider in the study, but in the areas of interpreting the test-negative design results.First, they note the possibility that unmeasured differences between vaccinated and unvaccinated persons that could influence the occurrence of COVID-19.They point out, however, that Thompson and colleagues used case weighting and logistic regression to adjust for several potential confounders.Second, case and control sample bias could occur, particularly in retrospective ascertainment of infection status as occurred in the present study.

A third potential source of bias is that infection or vaccination status might be misclassified, through several circumstances including timing of tests.Here again, however, they note that Thompson and colleagues have addressed the timing issue by broadening the testing period to include tests to detect infection taken within 14 days before, to less than 72 hours after hospital admission or emergency department and urgent care visits.Their fourth area of concern is that, however broad based the population study, there may still be limits on generalizing to other populations, particularly those with different access to medical care.

This study set a high bar in addressing these areas, however, according to Dean and colleagues."Researchers who use test-negative designs to investigate COVID-19 vaccine effectiveness can look to the article by Thompson et al for examples of how to report primary findings and assess the sensitivity of these finding to potential biases that are specific to the test-negative design," they conclude.