Comparison of administrative data and the American College of Surgeons National Surgical Quality Improvement Program data in a New South Wales Hospital.

Background: The National Surgical Quality Improvement Program (NSQIP) is widely used in North America for benchmarking. In 2015, NSQIP was introduced to four New South Wales public hospitals. The aim of this study is to investigate the agreement between NSQIP and administrative data in the Australian setting; to compare the performance of models derived from each data set to predict 30-day outcomes.

Methods: The NSQIP and administrative data variables were mapped to select variables available in both data sets where coding may be influenced by interpretation of the clinical information. These were compared for agreement. Logistic regression models were fitted to estimate the probability of adverse outcomes within 30 days. Models derived from NSQIP and administrative data were compared by receiver operating characteristic curve analysis.

Results: A total of 2240 procedures over 21 months had matching records. Functional status demonstrated poor agreement (kappa 0.02): administrative data recorded only one (1%) patient with partial- or total-dependence as recorded by NSQIP data. The American Society of Anesthesiologists class demonstrated excellent agreement (kappa 0.91). Other perioperative variables demonstrated poor to fair agreement (kappa 0.12-0.61). Predictive model based on NSQIP data was excellent at predicting mortality but was less accurate for complications and readmissions. The NSQIP model was better in predicting mortality and complications (receiver operating characteristic curve 0.93 versus 0.87; P = 0.029 and 0.71 versus 0.64; P = 0.027).

Conclusions: There is poor agreement between NSQIP data and administrative data. Predictive models associated with NSQIP data were more accurate at predicting surgical outcomes than those from administrative data. To drive quality improvement in surgery, high-quality clinical data are required and we believe that NSQIP fulfils this function.