Uncertainty estimation in diagnosis generation from large language models: next-word probability is not pre-test probability.

Objective: To evaluate large language models (LLMs) for pre-test diagnostic probability estimation and compare their uncertainty estimation performance with a traditional machine learning classifier.

Materials And Methods: We assessed 2 instruction-tuned LLMs, Mistral-7B-Instruct and Llama3-70B-chat-hf, on predicting binary outcomes for Sepsis, Arrhythmia, and Congestive Heart Failure (CHF) using electronic health record (EHR) data from 660 patients. Three uncertainty estimation methods-Verbalized Confidence, Token Logits, and LLM Embedding+XGB-were compared against an eXtreme Gradient Boosting (XGB) classifier trained on raw EHR data.

The TRIPOD-LLM reporting guideline for studies using large language models.

Large language models (LLMs) are rapidly being adopted in healthcare, necessitating standardized reporting guidelines. We present transparent reporting of a multivariable model for individual prognosis or diagnosis (TRIPOD)-LLM, an extension of the TRIPOD + artificial intelligence statement, addressing the unique challenges of LLMs in biomedical applications. TRIPOD-LLM provides a comprehensive checklist of 19 main items and 50 subitems, covering key aspects from title to discussion.

Lessons learned on information retrieval in electronic health records: a comparison of embedding models and pooling strategies.

Objectives: Applying large language models (LLMs) to the clinical domain is challenging due to the context-heavy nature of processing medical records. Retrieval-augmented generation (RAG) offers a solution by facilitating reasoning over large text sources. However, there are many parameters to optimize in just the retrieval system alone.

LCD benchmark: long clinical document benchmark on mortality prediction for language models.

Objectives: The application of natural language processing (NLP) in the clinical domain is important due to the rich unstructured information in clinical documents, which often remains inaccessible in structured data. When applying NLP methods to a certain domain, the role of benchmark datasets is crucial as benchmark datasets not only guide the selection of best-performing models but also enable the assessment of the reliability of the generated outputs. Despite the recent availability of language models capable of longer context, benchmark datasets targeting long clinical document classification tasks are absent.

Outcomes and Cost-Effectiveness of an EHR-Embedded AI Screener for Identifying Hospitalized Adults at Risk for Opioid Use Disorder.

Unlabelled: Hospitalized adults with opioid use disorder (OUD) are at high risk for adverse events and rehospitalizations. This pre-post quasi-experimental study evaluated whether an AI-driven OUD screener embedded in the electronic health record (EHR) was non-inferior to usual care in identifying patients for Addiction Medicine consults, aiming to provide a similarly effective but more scalable alternative to human-led ad hoc consultations. The AI screener analyzed EHR notes in real-time with a convolutional neural network to identify patients at risk and recommend consultation.