Session: Estimating Hospital Quality
Room: Hollister 110
Time: Wed 08:30-10:00

Presenter: Andrew Ryan (Weill Cornell Medical College. Public Health)

Discussant: Zeynal KaracaAvalere Health

Abstract

For public quality and pay-for-performance programs, the main obstacle to using outcomes as hospital performance measures is the large random variation in these measures. From an agency theory perspective, incentivizing outcome measures is desirable given that outcome performance is closer to the principal’s (i.e. the payer) objective than process or structure measures.

Several approaches have been developed to estimate outcome performance in the presence of large random variation in outcomes, including the observed-over-expected estimator (OE), Risk Standardized Mortality Rate (RSMR), Dimick and Staiger (DS) estimator, and the moving average (MA) estimator. CMS currently uses the RSMR with three years of previous data for its public quality reporting program. However, no research has compared empirically the accuracy of alternative methods for estimating hospital outcomes.

We perform a Monte Carlo simulation experiment to test the accuracy of these estimators using the following process: 50 simulated hospitals are created and randomly assigned a sample size, sample size trajectory, "true" mortality rate (correlated with sample size), and a “true” mortality rate trajectory. Then, a random shock is added to each hospital's true mortality rate to create an “observed” mortality rate. Using only the observed mortality rates from year t-n through year t-1, we use the previously described estimators to estimate hospitals' "unknown" true mortality rate in year t. The RSMR and DS estimators are calculated using both one and three years of data, the MA estimator is calculated using three years of data, and the OE is calculated using only one year of data. The performance of these estimators is captured for two criteria: the mean absolute deviation (MAD) (the difference between the estimated and true mortality rates) and the proportion of hospitals correctly classified into quintiles of true mortality performance. This process is repeated for 3000 iterations. We then test the accuracy of each estimator on the performance criteria for three simulated conditions: pneumonia, heart failure, and CABG.

Overall, the MA estimator using three years of data had the best performance based on the MAD, with values lower than other estimators for each simulated condition, and significantly (p < .05) lower for pneumonia and heart failure. The MA estimator also had the best performance on the proportion correctly classified criterion, although performance was not significantly better than the RSMR and DS estimators using three years of data. For both criteria, the OE and RSMR estimators using one year of data were by far the least accurate estimators.

Results from this study suggest that CMS’s change, in 2009, from using the RSMR with one year of data to using the RSMR with three years of data has likely substantially improved the accuracy of publicly reported mortality rates. However, for the conditions examined, results indicate that the MA estimator is more accurate than the RSMR. Implications are discussed.

Key Terms

hospital outcomes, measurement, simulation

Authors:

Andrew Ryan (Weill Cornell Medical College. Public Health) , James F. Burgess (Boston University. Public Health) and Justin Dimick (University of Michigan School of Medicine. Surgery)