Taking It As It Comes: Creating Evidence-Based Guidelines During a Rapidly Changing Pandemic
With few exceptions, COVID-19 therapies have not been studied comparatively, so choosing therapies to preferentially recommend is difficult and is done, instead, by strength of evidence and quality of data.
COVID-19 has been quite an educator. One of its lessons has been a work in progress on how clinicians keep up-to-date on best practices while dealing with the sheer volume of studies about the disease. Fortunately, many groups have created guidelines to treat COVID-19, including individual hospitals, government organizations like the National Institutes of Health,1 and professional organizations such as the Infectious Diseases Society of America (IDSA).2 No organization can claim a “best” guideline, but because I have had the experience of serving on the IDSA guideline panel for the past 2 years, I would like to explain how these guidelines have been developed “on the fly” and how the recommendations are formulated.
HOW THEY ARE DONE
The IDSA guideline panel consists of a mix of clinicians and methodologists. The clinicians are a mix of physicians and pharmacists, with expertise in infectious diseases, microbiology, pediatrics, and critical care. The methodologists serve as experts in research design and analysis and construct the evidence summaries for clinical questions as they develop. Some of these questions are natural, such as when novel therapies are made available by emergency-use authorization or when arms or analyses of landmark trials are completed. Others are prompted by panel members themselves. Meetings occur on a generally biweekly basis using video conferencing software.
The literature evaluated is primarily randomized controlled trials (RCTs) when they are available, but other types of studies are considered when RCTs are not available or when the alternative design appropriately supplements RCTs, such as real-world safety studies. Peer-reviewed publications are, unsurprisingly, preferred, but the panel also evaluates data in preprints. When press releases are the only data available, preliminary assessments are made, sometimes with the benefit of additional data.
The methodologists create evidence summaries and synthesize clinical questions by PICO format (population, intervention, comparison, and outcomes) using GRADE (grading of recommendations assessment, development, and evaluation working group) methodology.2 If data from more than 1 trial are available with comparable outcomes, the data are analyzed together. This evidence is reviewed by panelists before meetings and discussed during them, often leading to a recommendation.
Only clinical panelists vote on recommendations, and the language is specific (Table). The term “recommend” is a strong one, made only when the panel believes that both the evidence is of sufficient quality and that the recommendation could be reasonably adapted into policies. The term “suggest” is used when the evidence is not sufficiently strong to justify a recommend (such as due to fragility of the effect found) or when the intervention is one that patients may reasonably refuse. Thus far, the high threshold for a recommend has limited this level to only one positive recommendation—the use of glucocorticoids for patients with critical COVID-19 disease.
Although the most-read statements from the panel are the recommendations themselves, effort is also directed into clarifying the recommendations with the remarks that accompany them, which may direct clinicians to specific subpopulations in whom benefits appear most substantial or may clarify other elements of the recommendation, such as dosing. The evidence summary is also posted for each recommendation, containing the data evaluated from the included studies. Also, a narrative is written for each recommendation to put it into context.
HOW TO READ THEM
Admittedly, the guidelines are not the most user friendly. Because COVID-19 is a novel disease and each therapeutic has been studied compared with standard of care, each PICO is written as a comparison between the agent and not receiving the agent. This makes many of the recommendations read as binary, without the nuances that clinicians consider in their patient evaluations. Clinical questions are more along the lines of “How should I treat this patient with COVID-19 who requires supplemental oxygen?” than “Should remdesivir (Veklury) or not remdesivir be given to patients who require supplemental oxygen?” With few exceptions, COVID-19 therapies have not been studied comparatively, so choosing therapies to preferentially recommend is difficult and is done, instead, by strength of evidence and quality of data.
Another area in which miscommunications occur is in the use of therapies in combination. As the science of studying therapies during the pandemic has progressed, combination therapies have been studied more frequently as comparisons with standard-of-care regimens, strengthening the ability to recommend them. However, important deficits in the knowledge base about combination therapies remain, and the guidelines largely eschew recommendations about them where those deficits exist. Because the guidelines take an evidence-based approach, recommendations are dependent on the evidence generated.
The guidelines are quite long, keep growing, and will hopefully continue to grow with the evaluation of successful therapies. Most people look no further than the recommendations and possibly the remarks, but reading the narrative is important when these recommendations do not line up with preexisting expectations or are not sufficiently clear. The evidence summary shows the data the panel used to determine its recommendation and is included near the end of each section, showing not only the data from the studies themselves, but the determination of the strength of those data, the risk of bias from the study design(s), and the certainty that the effects seen are real.
HOW TO USE THEM
Clinicians commonly note that “guidelines are not rules” but that people often treat them that way. The oft-mentioned phrase in guidelines that they do not substitute for clinical judgment is important though, in ways that I did not appreciate before serving on the guideline panel. First and most obviously, not every patient’s nuance will make it into a clinical trial. The recommendations made are for the broad base of patients with COVID-19—and plenty of exceptions exist. For example, patients with deficient antibody response may benefit from antiviral and antibody therapies in later stages of disease than the majority of patients enrolled in studies.
It is also important for guideline readers to realize that therapies that are not explicitly recommended in guidelines may have utility for some patients. For a therapy to reach the level of a guideline recommendation, it needs a sufficient evidence base for a judgment to be rendered. Now that effective therapeutics exist, if a promising but incompletely studied therapy is recommended before sufficient evidence justifies its use, then people may utilize it in lieu of one with greater evidence, potentially causing harm. However, that does not mean those therapies are not good choices for individual patients, and the lack of a recommendation for them does not mean they should be avoided.
Finally, the panel is careful to avoid making recommendations against therapies when clinical trials are continuing to explore their utility. Negative recommendations are made when the totality of evidence leads to a conclusion that a therapy is ineffective. However, new evidence can change a recommendation, particularly when populations studied are refined to those most likely to benefit.
DIFFICULTIES WITH THE GUIDELINES PROCESS
Creating guidelines during a pandemic has been a continuous work in progress. Although the guidelines are a living document, parts can inevitably grow stale as the pace of SARS-CoV-2 and COVID-19 therapies change. A notable example of this is the recommendation for neutralizing antibody therapies, for which literature evaluated becomes out-of-date quickly as variants evolve and become less (or more) susceptible to various monoclonal antibodies. To date, the panel has taken the position of recommending that local variants are considered when these therapies are chosen instead of trying to keep up with the rapid changes being seen. Another major challenge is keeping up with the sheer volume of literature generated about COVID-19 therapy. Fortunately, the methodologists consistently review for updates, and most major studies and therapies receive the attention of multiple panel members as well.
SUMMARY
The guidelines are a document that, although not without flaws, have hopefully helped guide treatment decisions that have aided numerous patients with COVID-19. The panelists are some of the most dedicated people with whom I have had the pleasure of working, and they are committed to the improvement of our collective work.
References
- Coronavirus disease 2019 (COVID-19) treatment guidelines. National Institutes of Health. Updated April 8, 2022. Accessed April 18, 2022. https://www.covid19treatmentguidelines.nih.gov
- IDSA guidelines on the treatment and management of patients with COVID-19. Infectious Diseases Society of America. April 11, 2020. Updated March 23, 2022. Accessed April 18, 2022.https://www.idsociety.org/practice-guideline/covid-19-guideline-treatment-and-management/
