How Do You Measure the Success of Your Stewardship Program?

Antimicrobial use data are essential for informing and assessing the progress of a stewardship program, but there are multiple challenges to collecting and analyzing the data in a meaningful, useful way. Data can be obtained from electronic health records, internal pharmacy documentation, or, most tediously, manual chart review.

Regulatory bodies such as the Centers for Medicare & Medicaid Services (CMS), Centers for Disease Control and Prevention (CDC), and the Joint Commission have developed several standards to guide the development and implementation of antimicrobial stewardship programs (ASPs). These standards define what elements of a stewardship program, including the tracking and reporting of metrics, must be met for reimbursement. Although antibiotic use data must be collected by each facility to receive CMS funding, reporting of facility-specific data to the CDC’s National Healthcare Safety Network (NHSN) remains optional, in contrast to hospital-acquired infection data, for which reporting is mandatory.¹

Current metrics are typically categorized broadly as either a process-of-care measure or an outcome measure. Process measures such as duration of therapy or proportion of patients compliant with facility-based guidelines are used to assess the impact of ASP interventions on antimicrobial use, whereas outcome measures like hospital length of stay are centered around the end effect on patients (or hospitals).^2,3

Examining Current Metrics

The 2016 Infectious Diseases Society of America (IDSA), Society for Healthcare Epidemiology of America (SHEA), and Pediatric Infectious Diseases Society (PIDS) ASP implementation guidelines suggest measurement of antibiotic use as the best overall reflection of ASP impact versus antimicrobial cost (Table 1).² ASP interventions may incidentally lead to cost savings (eg, by decreasing unnecessary antimicrobial use) and help garner administrative support for the program, but antibiotic costs are not always congruent with patient outcomes or antibiotic appropriateness. In fact, appropriate ASP interventions can sometimes increase antimicrobial costs.^2,4

Antimicrobial consumption data are available to hospitals and allow for internal (at the same institution over time) and external (across institutions) comparison. External comparison is limited, however, by the variation in reported metrics.¹ Defined daily doses (DDDs) is a consumption metric determined by dividing the amount of each antimicrobial used by its corresponding World Health Organization (WHO) standard daily dose; the sum of these amounts is then typically normalized by dividing by a volume measure such as 1000 patient-days. Although measurement of DDDs is recommended by the WHO and widely used outside the United States, it is heavily reliant on the WHO definition of standard daily dose.^1,3 This can lead to contrasting results when indication-specific, weight-based, or renally adjusted doses are used.^2,4

Days of therapy (DOT) is the preferred metric in the United States and endorsed by the IDSA/SHEA/PIDS ASP guidelines.² It is a count of the number of individual antimicrobial agents given to a patient on each calendar day multiplied by the total number of days of therapy, which is then standardized to account for volume by dividing by 1000 patient-days. Length of therapy (LOT) is a similar metric but considers the duration of overall antimicrobial therapy without accounting for the number of antimicrobials administered each day.^1,3 DOT can be considered to account for the total burden of antimicrobial use while LOT accounts for the days of antimicrobial exposure. Both metrics are most useful as a tool for comparing an ASP’s performance against its own historical performance, but comparison to other facilities is limited because DOT and LOT do not account for differences in patient populations. Notably, measures of consumption should be used cautiously to avoid confusing the goal of therapy optimization with unjustified antimicrobial reduction. Although quantitative measures such as DOT and antimicrobial costs are commonly utilized performance metrics, they are only peripherally related to the impact of ASP interventions on patient quality of care and outcomes.

The most recently developed ASP metric is the NHSN SAAR, or standardized antimicrobial administration ratio. This is a standardized risk–adjusted metric of antimicrobial use that compares observed to predicted days of antimicrobial use and is useful for comparing an ASP’s performance to its expected performance based on the given patient population. The predicted number of antimicrobial days are calculated using statistical models based on nationally aggregated data from the CDC. A SAAR ratio less than 1 indicates antimicrobial use was less than predicted, a SAAR of 1 indicates use was equivalent to predicted use, and a SAAR of more than 1 indicates use was greater than expected.

In 2017, SAAR baselines were established for 13 adult and pediatric medical, surgical, medical-surgical intensive care units (ICUs), wards, step-down units, and oncology units, and there are 15 antimicrobial categories for which an institution could receive SAAR information back from the NHSN. For example, one could choose to see how their gram-positive–targeted antimicrobial use (eg, vancomycin, daptomycin, linezolid, etc) compared with the expected use for a certain unit type such as a medical ICU. NHSN allows for reporting of 89 specific antimicrobials and includes new drugs as they become FDA approved. However, because the system does not allow for manual data input, significant information technology infrastructure and investment are required to submit reportable data to NHSN. In addition, certain specialized unit types such as burn ICUs are not yet validated by NHSN and do not have a SAAR baseline to provide predicted antimicrobial days.⁵

The CDC also provides guidance on suggested metrics for non–hospital-based stewardship programs (Table 2^6-8). Irrespective of site, recommendations emphasize that even though savings in drug costs after implementation of a stewardship program may be used to justify the cost of the ASP, antimicrobial expenditures should not be used to track the effectiveness of stewardship efforts.^6-8

Certain settings allow for unique metrics. For example, informing outpatient clinicians that they prescribe more antimicrobials than 80% of their peers or comparing them to peers who perform in the top 10% on quality measures can result in overall reductions in antimicrobial prescribing.⁷ Smaller settings may be better able to track prescribing at the clinician level; however, they may also make tracking the complications of antimicrobial use (eg, Clostridium difficile infection, drug interactions, and adverse drug reactions) less reliable or useful due to small sample sizes.⁶ Nursing homes or long-term care (LTC) facilities may choose to do point prevalence surveys of antimicrobial use, which track the proportion of residents receiving antimicrobials during a given period, to get a snapshot of the burden of antimicrobial use in the facility.⁸ A unique challenge for nursing homes is that antimicrobials are often provided by LTC pharmacy vendors, which may create an additional layer of complexity when trying to assess antimicrobial use and limit the reporting of antimicrobial use available to a facility (eg, antimicrobial start dates or indications).⁸

Consider Goals to Determine Benchmark

The ideal benchmark would be centered around the primary goal(s) of ASPs: to improve patient outcomes via appropriate use of antimicrobials and to optimize antimicrobial regimens, including dosing, duration of therapy, and route of administration.²

Although CMS and the Joint Commission similarly call for evidence-based antimicrobial use as well as minimization of adverse events, resistance, and hospital-acquired infections, it is difficult to measure these outcomes, and little data connect them to our most commonly collected metrics.

Key stakeholders also want metrics that correlate more with patient care. A survey of physicians and pharmacists identified that they favored patient-centered outcomes such as infection-related mortality and considered these disparate from the metrics reportedly used at their institution.⁹

There is no perfect metric by which to judge the success of one’s stewardship program. More and more facilities are, however, opting to share their antimicrobial use data with the CDC NHSN. In turn, this provides better, higher quality data for future SAAR baseline validations and a way for stewardship programs to benchmark their institutions against others with a similar patient population.

Jason Chao, PharmD, is a postgraduate year 2 infectious diseases pharmacy resident at the University of California Davis Medical Center in Sacramento.

Nicola Clayton, PharmD, BCIDP, is an infectious diseases clinical pharmacist and the pharmacy leader of the Antimicrobial Stewardship Program at the University of California Davis Medical Center.

References

1. ^{Bennett N, Schulz L, Boyd S, Newland JG. Understanding inpatient antimicrobial stewardship metrics. Am J Health Syst Pharm. 2018;75(4):230-238. doi:10.2146/ajhp160335}
2. ^{Barlam TF, Cosgrove SE, Abbo LM, et al. Implementing an antibiotic stewardship program: guidelines by the Infectious Diseases Society of America and the Society for Healthcare Epidemiology of America. Clin Infect Dis. 2016;62(10):e51-e77. doi:10.1093/cid/ciw118}
3. ^{Ibrahim OM, Polk RE. Antimicrobial use metrics and benchmarking to improve stewardship outcomes: methodology, opportunities, and challenges. Infect Dis Clin North Am. 2014;28(2):195-214. doi:10.1016/j.idc.2014.01.006}
4. ^{Al-Hasan MN, Winders HR, Bookstaver PB, Justo JA. Direct measurement of performance: a new era in antimicrobial stewardship. Antibiotics (Basel). 2019;8(3):127. doi:10.3390/antibiotics8030127}
5. ^{Keys to Success with the SAAR. Centers for Disease Control and Prevention. Published March 2020. Accessed April 12, 2021. https://www.cdc.gov/nhsn/pdfs/ps-analysis-resources/aur/keys-success-saar-h.pdf}
6. ^{Centers for Disease Control and Prevention (CDC). Implementation of antibiotic stewardship core elements at small and critical access hospitals. Accessed April 2, 2021. https://www.cdc.gov/antibiotic-use/healthcare/pdfs/core-elements-small-critical.pdf}
7. ^{CDC. Core elements of outpatient antibiotic stewardship. Accessed April 2, 2021. https://www.cdc.gov/antibiotic-use/community/pdfs/16_268900-A_CoreElementsOutpatient_508.pdf}
8. ^{Centers for Disease Control and Prevention. Core Elements of Antibiotic Stewardship for Nursing Homes: APPENDIX B: Measures of Antibiotic Prescribing, Use and Outcome. Centers for Disease Control and Prevention. Accessed April 2, 2021. https://www.cdc.gov/longtermcare/pdfs/core-elements-antibiotic-stewardship-appendix-b.pdf}
9. ^{Bumpass JB, McDaneld PM, DePestel DD, et al. Outcomes and metrics for antimicrobial stewardship: survey of physicians and pharmacists. Clin Infect Dis. 2014;59(suppl 3):S108-S111. doi:10.1093/cid/ciu545}