Accuracy and Safety of AI-Enabled Scribe Technology: Instrument Validation Study.

Artificial intelligence-enabled ambient digital scribes may have many potential benefits, yet results from our study indicate that there are errors that must be evaluated to mitigate safety risks.

Development of a Preliminary Patient Safety Classification System for Generative AI.

Generative artificial intelligence (AI) technologies have the potential to revolutionise healthcare delivery but require classification and monitoring of patient safety risks. To address this need, we developed and evaluated a preliminary classification system for categorising generative AI patient safety errors. Our classification system is organised around two AI system stages (input and output) with specific error types by stage.

Cross entropy and log likelihood ratio cost as performance measures for multi-conclusion categorical outcomes scales.

The inconclusive category in forensics reporting is the appropriate response in many cases, but it poses challenges in estimating an "error rate". We discuss the use of a class of information-theoretic measures related to cross entropy as an alternative set of metrics that allows for performance evaluation of results presented using multi-category reporting scales. This paper shows how this class of performance metrics, and in particular the log likelihood ratio cost, which is already in use with likelihood ratio forensic reporting methods and in machine learning communities, can be readily adapted for use with the widely used multiple category conclusions scales.

Artificial intelligence related safety issues associated with FDA medical device reports.

The Biden 2023 Artificial Intelligence (AI) Executive Order calls for the creation of a patient safety program. Patient safety reports are a natural starting point for identifying issues. We examined the feasibility of this approach by analyzing reports associated with AI/Machine Learning (ML)-enabled medical devices.