Multimodal Machine Learning in Image-Based and Clinical Biomedicine: Survey and Prospects.

Machine learning (ML) applications in medical artificial intelligence (AI) systems have shifted from traditional and statistical methods to increasing application of deep learning models. This survey navigates the current landscape of multimodal ML, focusing on its profound impact on medical image analysis and clinical decision support systems. Emphasizing challenges and innovations in addressing multimodal representation, fusion, translation, alignment, and co-learning, the paper explores the transformative potential of multimodal models for clinical predictions.

Unsupervised ensemble-based phenotyping enhances discoverability of genes related to left-ventricular morphology.

Recent genome-wide association studies have successfully identified associations between genetic variants and simple cardiac morphological parameters derived from cardiac magnetic resonance images. However, the emergence of large databases, including genetic data linked to cardiac magnetic resonance facilitates the investigation of more nuanced patterns of cardiac shape variability than those studied so far. Here we propose a framework for gene discovery coined unsupervised phenotype ensembles.

Digital twins for health: a scoping review.

The use of digital twins (DTs) has proliferated across various fields and industries, with a recent surge in the healthcare sector. The concept of digital twin for health (DT4H) holds great promise to revolutionize the entire healthcare system, including management and delivery, disease treatment and prevention, and health well-being maintenance, ultimately improving human life. The rapid growth of big data and continuous advancement in data science (DS) and artificial intelligence (AI) have the potential to significantly expedite DT research and development by providing scientific expertise, essential data, and robust cybertechnology infrastructure.

Modelling-based joint embedding of histology and genomics using canonical correlation analysis for breast cancer survival prediction.

Traditional approaches to predicting breast cancer patients' survival outcomes were based on clinical subgroups, the PAM50 genes, or the histological tissue's evaluation. With the growth of multi-modality datasets capturing diverse information (such as genomics, histology, radiology and clinical data) about the same cancer, information can be integrated using advanced tools and have improved survival prediction. These methods implicitly exploit the key observation that different modalities originate from the same cancer source and jointly provide a complete picture of the cancer.

Multimodal Pediatric Lymphoma Detection using PET and MRI.

Lymphoma is one of the most common types of cancer for children (ages 0 to 19). Due to the reduced radiation exposure, PET/MR systems that allow simultaneous PET and MR imaging have become the standard of care for diagnosing cancers and monitoring tumor response to therapy in the pediatric population. In this work, we developed a multimodal deep learning algorithm for automatic pediatric lymphoma detection using PET and MRI.

Fusing modalities by multiplexed graph neural networks for outcome prediction from medical data and beyond.

With the emergence of multimodal electronic health records, the evidence for diseases, events, or findings may be present across multiple modalities ranging from clinical to imaging and genomic data. Developing effective patient-tailored therapeutic guidance and outcome prediction will require fusing evidence across these modalities. Developing general-purpose frameworks capable of modeling fine-grained and multi-faceted complex interactions, both within and across modalities is an important open problem in multimodal fusion.

Methods for improving colorectal cancer annotation efficiency for artificial intelligence-observer training.

Background: Missing occult cancer lesions accounts for the most diagnostic errors in retrospective radiology reviews as early cancer can be small or subtle, making the lesions difficult to detect. Second-observer is the most effective technique for reducing these events and can be economically implemented with the advent of artificial intelligence (AI).

Aim: To achieve appropriate AI model training, a large annotated dataset is necessary to train the AI models.

Towards Automatic Prediction of Outcome in Treatment of Cerebral Aneurysms.

Intrasaccular flow disruptors treat cerebral aneurysms by diverting the blood flow from the aneurysm sac. Residual flow into the sac after the intervention is a failure that could be due to the use of an undersized device, or to vascular anatomy and clinical condition of the patient. We report a machine learning model based on over 100 clinical and imaging features that predict the outcome of wide-neck bifurcation aneurysm treatment with an intrasaccular embolization device.

Exploring approaches for predictive cancer patient digital twins: Opportunities for collaboration and innovation.

We are rapidly approaching a future in which cancer patient digital twins will reach their potential to predict cancer prevention, diagnosis, and treatment in individual patients. This will be realized based on advances in high performance computing, computational modeling, and an expanding repertoire of observational data across multiple scales and modalities. In 2020, the US National Cancer Institute, and the US Department of Energy, through a trans-disciplinary research community at the intersection of advanced computing and cancer research, initiated team science collaborative projects to explore the development and implementation of predictive Cancer Patient Digital Twins.

Semantic Expansion of Clinician Generated Data Preferences for Automatic Patient Data Summarization.

Patient Electronic Health Records (EHRs) typically contain a substantial amount of data, which can lead to information overload for clinicians, especially in high-throughput fields like radiology. Thus, it would be beneficial to have a mechanism for summarizing the most clinically relevant patient information pertinent to the needs of clinicians. This study presents a novel approach for the curation of clinician EHR data preference information towards the ultimate goal of providing robust EHR summarization.

AI Accelerated Human-in-the-loop Structuring of Radiology Reports.

Rule-based Natural Language Processing (NLP) pipelines depend on robust domain knowledge. Given the long tail of important terminology in radiology reports, it is not uncommon for standard approaches to miss items critical for understanding the image. AI techniques can accelerate the concept expansion and phrasal grouping tasks to efficiently create a domain specific lexicon ontology for structuring reports.

Extracting and Learning Fine-Grained Labels from Chest Radiographs.

Chest radiographs are the most common diagnostic exam in emergency rooms and intensive care units today. Recently, a number of researchers have begun working on large chest X-ray datasets to develop deep learning models for recognition of a handful of coarse finding classes such as opacities, masses and nodules. In this paper, we focus on extracting and learning fine-grained labels for chest X-ray images.

Combining Deep Learning and Knowledge-driven Reasoning for Chest X-Ray Findings Detection.

The application of deep learning algorithms in medical imaging analysis is a steadily growing research area. While deep learning methods are thriving in the medical domain, they seldom utilize the rich knowledge associated with connected radiology reports. The knowledge derived from these reports can be utilized to enhance the performance of deep learning models.

Comparison of Chest Radiograph Interpretations by Artificial Intelligence Algorithm vs Radiology Residents.

Importance: Chest radiography is the most common diagnostic imaging examination performed in emergency departments (EDs). Augmenting clinicians with automated preliminary read assistants could help expedite their workflows, improve accuracy, and reduce the cost of care.

Objective: To assess the performance of artificial intelligence (AI) algorithms in realistic radiology workflows by performing an objective comparative evaluation of the preliminary reads of anteroposterior (AP) frontal chest radiographs performed by an AI algorithm and radiology residents.

Improving the Path from Diagnoses to Documentation: A Cognitive Review Tool for Clinical Notes and Administrative Records.

EMR systems are intended to improve patient-centered care management and hospital administrative processing. However, the information stored in EMRs can be disorganized, incomplete, or inconsistent, creating problems at the patient and system level. We present a technology that reconciles inconsistencies between clinical diagnoses and administrative records by analyzing free-text notes, problem lists and recorded diagnoses in real time.

Generalized Extraction and Classification of Span-Level Clinical Phrases.

Much of the critical information in a patient's electronic health record (EHR) is hidden in unstructured text. As such, there is an increasing role for automated text extraction and summarization to make this information available in a way that can be quickly and easily understood. While many clinical note text extraction techniques have been examined, most existing techniques are either narrowly targeted or focus primarily on concept-level extraction, potentially missing important contextual information.

Building medical image classifiers with very limited data using segmentation networks.

Deep learning has shown promising results in medical image analysis, however, the lack of very large annotated datasets confines its full potential. Although transfer learning with ImageNet pre-trained classification models can alleviate the problem, constrained image sizes and model complexities can lead to unnecessary increase in computational cost and decrease in performance. As many common morphological features are usually shared by different classification tasks of an organ, it is greatly beneficial if we can extract such features to improve classification with limited samples.

Fast anatomy segmentation by combining coarse scale multi-atlas label fusion with fine scale corrective learning.

Deformable registration based multi-atlas segmentation has been successfully applied in a broad range of anatomy segmentation applications. However, the excellent performance comes with a high computational burden due to the requirement for deformable image registration and voxel-wise label fusion. To address this problem, we investigate the role of corrective learning (Wang et al.

Role of Big Data and Machine Learning in Diagnostic Decision Support in Radiology.

The field of diagnostic decision support in radiology is undergoing rapid transformation with the availability of large amounts of patient data and the development of new artificial intelligence methods of machine learning such as deep learning. They hold the promise of providing imaging specialists with tools for improving the accuracy and efficiency of diagnosis and treatment. In this article, we will describe the growth of this field for radiology and outline general trends highlighting progress in the field of diagnostic decision support from the early days of rule-based expert systems to cognitive assistants of the modern era.

Automatic Generation of Conditional Diagnostic Guidelines.

The diagnostic workup for many diseases can be extraordinarily nuanced, and as such reference material text often contains extensive information regarding when it is appropriate to have a patient undergo a given procedure. In this work we employ a three task pipeline for the extraction of statements indicating the conditions under which a procedure should be performed, given a suspected diagnosis. First, we identify each instance in the text where a procedure is being recommended.

Metric hashing forests.

In this paper, we propose metric Hashing Forests (mHF) which is a supervised variant of random forests tailored for the task of nearest neighbor retrieval through hashing. This is achieved by training independent hashing trees that parse and encode the feature space such that local class neighborhoods are preserved and encoded with similar compact binary codes. At the level of each internal node, locality preserving projections are employed to project data to a latent subspace, where separability between dissimilar points is enhanced.

Mining echocardiography workflows for disease discriminative patterns.

Objective: To provide quick diagnostic insights to medical practitioners into echocardiograms by only analyzing the echocardiogram workflows (defined as the sequence of modalities examined).

Methods: We define a dictionary of workflows, called subflows, which are commonly encountered in echocardiography workflows but are mutually exclusive. We represent each workflow as a mixture of dictionary subflows and learn discriminative models for various cardiac diseases using Support Vector Machines.

Automatic detection of dilated cardiomyopathy in cardiac ultrasound videos.

In this paper we address the problem of automatic detection of dilated cardiomyopathy from cardiac ultrasound videos. Specifically, we present a new method of robustly locating the left ventricle by using the key idea that the region closest to the apex in a 4-chamber view is the left ventricular region. For this, we locate a region of interest containing the heart in an echocardiogram image using the bounding lines of the viewing sector to locate the apex of the heart.

Finding similar 2D X-ray coronary angiograms.

In clinical practice, physicians often exploit previously observed patterns in coronary angiograms from similar patients to quickly assess the state of the disease in a current patient. These assessments involve visually observed features such as the distance of a junction from the root and the tortuosity of the arteries. In this paper, we show how these visual features can be automatically extracted from coronary artery images and used for finding similar coronary angiograms from a database.