Purpose: Precision oncology in non-small cell lung cancer (NSCLC) relies on biomarker testing for clinical decision making. Despite its importance, challenges like the lack of genomic oncology training, nonstandardized biomarker reporting, and a rapidly evolving treatment landscape hinder its practice. Generative artificial intelligence (AI), such as ChatGPT, offers promise for enhancing clinical decision support. Effective performance metrics are crucial to evaluate these models' accuracy and their propensity for producing incorrect or hallucinated information. We assessed various ChatGPT versions' ability to generate accurate next-generation sequencing reports and treatment recommendations for NSCLC, using a novel Generative AI Performance Score (G-PS), which considers accuracy, relevancy, and hallucinations.
Methods: We queried ChatGPT versions for first-line NSCLC treatment recommendations with an Food and Drug Administration-approved targeted therapy, using a zero-shot prompt approach for eight oncogenes. Responses were assessed against National Comprehensive Cancer Network (NCCN) guidelines for accuracy, relevance, and hallucinations, with G-PS calculating scores from -1 (all hallucinations) to 1 (fully NCCN-compliant recommendations). G-PS was designed as a composite measure with a base score for correct recommendations (weighted for preferred treatments) and a penalty for hallucinations.
Results: Analyzing 160 responses, generative pre-trained transformer (GPT)-4 outperformed GPT-3.5, showing higher base score (90% 60%; < .01) and fewer hallucinations (34% 53%; < .01). GPT-4's overall G-PS was significantly higher (0.34 -0.15; < .01), indicating superior performance.
Conclusion: This study highlights the rapid improvement of generative AI in matching treatment recommendations with biomarkers in precision oncology. Although the rate of hallucinations improved in the GPT-4 model, future generative AI use in clinical care requires high levels of accuracy with minimal to no room for hallucinations. The GP-S represents a novel metric quantifying generative AI utility in health care compared with national guidelines, with potential adaptation beyond precision oncology.
Download full-text PDF |
Source |
---|---|
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC11634130 | PMC |
http://dx.doi.org/10.1200/CCI.24.00123 | DOI Listing |
Targeted therapy has emerged as a promising option in cancer treatment, driven by advances in the understanding of DNA changes and the molecular basis of cancer. This article provides an overview of next-generation sequencing and types of genetic alterations, common cancer biomarkers, a review of circulating tumor DNA testing and its applications for oncology treatments, how to read a genomic testing report, examples of targeted therapy for cancer pathologic variants and tumor markers, and the implications for nursing practice in this emerging field.
View Article and Find Full Text PDFBr J Clin Pharmacol
January 2025
Parexel International, Durham, North Carolina, USA.
Early approval mechanisms, such as conditional approval in the EU, have been used extensively to provide timely access to therapeutic innovations to cancer patients with unmet medical needs. While based on promising early evidence, such approvals are challenging from many perspectives due to the lack of comprehensive data. The limitation typically relates to data that demonstrates clinical benefit via early endpoints and is only acceptable when the early evidence is particularly convincing to assume that the benefits of early access are greater than the potential harms.
View Article and Find Full Text PDFMed Int (Lond)
December 2024
Department of Clinical and Experimental Medicine, Endocrine Unit 2, University of Pisa, I-56122 Pisa, Italy.
The limitations of two-dimensional (2D) models in cancer research have hindered progress in fully understanding the complexities of drug resistance and therapeutic failures. However, three-dimensional (3D) models provide a more accurate representation of environments, capturing critical cellular interactions and dynamics that are essential in evaluating the efficacy and toxicity of tyrosine kinase inhibitors (TKIs). These advanced models enable researchers to explore drug resistance mechanisms with greater precision, optimizing treatment strategies and improving the predictive accuracy of clinical outcomes.
View Article and Find Full Text PDFiScience
January 2025
Department of Biology, University of Copenhagen, 2100 Copenhagen, Denmark.
Chromothripsis, a hallmark of cancer, is characterized by extensive and localized DNA rearrangements involving one or a few chromosomes. However, its genome-wide frequency and characteristics in urothelial carcinoma (UC) remain largely unknown. Here, by analyzing single-regional and multi-regional whole-genome sequencing (WGS), we present the chromothripsis blueprint in 488 UC patients.
View Article and Find Full Text PDFJTO Clin Res Rep
January 2025
Icahn School of Medicine at Mount Sinai, New York, New York.
Lung cancer remains a leading cause of cancer-related mortality globally and presents significant challenges in Egypt. In 2023, the first annual meeting of the Thoracic Oncology Multidisciplinary Faculty, organized by the Egyptian Cancer Research Network and the Egyptian Society of Respiratory Neoplasms, was held in Cairo, Egypt. The meeting aimed to address gaps in lung cancer management across Egypt and the broader Middle East and North Africa region.
View Article and Find Full Text PDFEnter search terms and have AI summaries delivered each week - change queries or unsubscribe any time!