Recommendation to use exact P-values in biomarker discovery research in place of approximate P-values.

Background: Biomarker candidates are often ranked using P-values. Standard P-value calculations use normal or logit-normal approximations, which may not be correct for small P-values and small sample sizes common in discovery research.

Methods: We compared exact P-values, correct by definition, with logit-normal approximations in a simulated study of 40 cases and 160 controls. The key measure of biomarker performance was sensitivity at 90% specificity. Data for 3000 uninformative false markers and 30 informative true markers were generated randomly. We also analyzed real data for 2371 plasma protein markers measured in 121 breast cancer cases and 121 controls.

Results: In our simulation, using the same discovery criterion, exact P-values led to discovery of 24 true and 82 false biomarkers, while logit-normal approximate P-values yielded 20 true and 106 false biomarkers. The estimated true discovery rate was substantially off for approximate P-values: logit-normal estimated 42 but found 20. The exact method estimated 22, very close to 24, which was the actual number of true discoveries. Although these results are based on one specific simulation, qualitatively similar results were obtained from 10 random repetitions. With real data, ranking candidate biomarkers by exact P-values, versus approximate P-values, resulted in a very different ordering of these markers.

Conclusions: Exact P-values, which correspond to permutation tests with non-parametric rank statistics such as empirical ROC statistics, are preferred over approximate P-values. Approximate P-values can lead to inappropriate biomarker selection rules and incorrect conclusions.

Impact: Exact P-values in place of approximate P-values in discovery research may improve the yield of biomarkers that validate clinically.