Treating metastatic disease: Which survival model is best suited for the clinic?

Background: To avoid complications associated with under- or overtreatment of patients with skeletal metastases, doctors need accurate survival estimates. Unfortunately, prognostic models for patients with skeletal metastases of the extremities are lacking, and physician-based estimates are generally inaccurate.

Questions/purposes: We developed three types of prognostic models and compared them using calibration plots, receiver operating characteristic (ROC) curves, and decision curve analysis to determine which one is best suited for clinical use.

Methods: A training set consisted of 189 patients who underwent surgery for skeletal metastases. We created models designed to predict 3- and 12-month survival using three methods: an Artificial Neural Network (ANN), a Bayesian Belief Network (BBN), and logistic regression. We then performed crossvalidation and compared the models in three ways: calibration plots plotting predicted against actual risk; area under the ROC curve (AUC) to discriminate the probability that a patient who died has a higher predicted probability of death compared to a patient who did not die; and decision curve analysis to quantify the clinical consequences of over- or undertreatment.

Results: All models appeared to be well calibrated, with the exception of the BBN, which underestimated 3-month survival at lower probability estimates. The ANN models had the highest discrimination, with an AUC of 0.89 and 0.93, respectively, for the 3- and 12-month models. Decision analysis revealed all models could be used clinically, but the ANN models consistently resulted in the highest net benefit, outperforming the BBN and logistic regression models.

Conclusions: Our observations suggest use of the ANN model to aid decisions about surgery would lead to better patient outcomes than other alternative approaches to decision making.

Level Of Evidence: Level II, prognostic study. See Instructions for Authors for a complete description of levels of evidence.