Using Artificial Intelligence to Support Informed Decision-Making on Mutation Testing.

Purpose: Precision oncology relies on accurate and interpretable reporting of testing and mutation rates. Focusing on the mutations in advanced colorectal carcinoma, non-small-cell lung carcinoma, and cutaneous melanoma, we developed a platform displaying testing and mutation rates reported in the literature, which we annotated using an artificial intelligence (AI) and natural language processing (NLP) pipeline.

Methods: Using AI, we identified publications that likely reported a testing or mutation rate, filtered publications for cancer type, and identified sentences that likely reported rates. Rates and covariates were subsequently manually curated by three experts. The AI performance was evaluated using precision and recall metrics. We used an interactive platform to explore and present the annotated testing and mutation rates by certain study characteristics.

Results: The interactive dashboard, accessible at the BRAF dimensions website, enables users to filter mutation and testing rates with relevant options (eg, country of study, study type, mutation type) and to visualize annotated rates. The AI pipeline demonstrated excellent filtering performance (>90% precision and recall for all target cancer types) and moderate performance for sentence classification (53%-99% precision; ≥75% recall). The manual annotation of testing and mutation rates revealed inter-rater disagreement (testing rate, 19%; mutation rate, 70%), indicating unclear or nonstandard reporting of rates in some publications.

Conclusion: Our AI-driven NLP pipeline demonstrated the potential for annotating biomarker testing and mutation rates. The difficulties we encountered highlight the need for more advanced AI-powered literature searching and data extraction, and more consistent reporting of testing rates. These improvements would reduce the risk of misinterpretation or misunderstanding of testing and mutation rates by AI-based technologies and the health care community, with beneficial impacts on clinical decision-making, research, and trial design.