Metrics reloaded: recommendations for image analysis validation.

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics.

Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers.

Removing non-nuclei information from histopathological images: A preprocessing step towards improving nuclei segmentation methods.

Disease interpretation by computer-aided diagnosis systems in digital pathology depends on reliable detection and segmentation of nuclei in hematoxylin and eosin (HE) images. These 2 tasks are challenging since appearance of both cell nuclei and background structures are very variable. This paper presents a method to improve nuclei detection and segmentation in HE images by removing tiles that only contain background information.

ROOD-MRI: Benchmarking the robustness of deep learning segmentation models to out-of-distribution and corrupted data in MRI.

Deep artificial neural networks (DNNs) have moved to the forefront of medical image analysis due to their success in classification, segmentation, and detection challenges. A principal challenge in large-scale deployment of DNNs in neuroimage analysis is the potential for shifts in signal-to-noise ratio, contrast, resolution, and presence of artifacts from site to site due to variances in scanners and acquisition protocols. DNNs are famously susceptible to these distribution shifts in computer vision.

Targeting tumour-associated macrophages in hodgkin lymphoma using engineered extracellular matrix-mimicking cryogels.

Tumour-associated macrophages are linked with poor prognosis and resistance to therapy in Hodgkin lymphoma; however, there are no suitable preclinical models to identify macrophage-targeting therapeutics. We used primary human tumours to guide the development of a mimetic cryogel, wherein Hodgkin (but not Non-Hodgkin) lymphoma cells promoted primary human macrophage invasion. In an invasion inhibitor screen, we identified five drug hits that significantly reduced tumour-associated macrophage invasion: marimastat, batimastat, AS1517499, ruxolitinib, and PD-169316.

Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers.

A radiomic biomarker for prognosis of resected colorectal cancer liver metastases generalizes across MRI contrast agents.

Introduction: Contrast-enhanced MRI is routinely performed as part of preoperative work-up for patients with Colorectal Cancer Liver Metastases (CRLM). Radiomic biomarkers depicting the characteristics of CRLMs in MRI have been associated with overall survival (OS) of patients, but the reproducibility and clinical applicability of these biomarkers are limited due to the variations in MRI protocols between hospitals.

Methods: In this work, we propose a generalizable radiomic model for predicting OS of CRLM patients who received preoperative chemotherapy and delayed-phase contrast enhanced (DPCE) MRIs prior to hepatic resection.

BI-RADS BERT and Using Section Segmentation to Understand Radiology Reports.

Radiology reports are one of the main forms of communication between radiologists and other clinicians, and contain important information for patient care. In order to use this information for research and automated patient care programs, it is necessary to convert the raw text into structured data suitable for analysis. State-of-the-art natural language processing (NLP) domain-specific contextual word embeddings have been shown to achieve impressive accuracy for these tasks in medicine, but have yet to be utilized for section structure segmentation.

Cell segmentation for immunofluorescence multiplexed images using two-stage domain adaptation and weakly labeled data for pre-training.

Cellular profiling with multiplexed immunofluorescence (MxIF) images can contribute to a more accurate patient stratification for immunotherapy. Accurate cell segmentation of the MxIF images is an essential step. We propose a deep learning pipeline to train a Mask R-CNN model (deep network) for cell segmentation using nuclear (DAPI) and membrane (NaKATPase) stained images.

Missed Breast Cancers on MRI in High-Risk Patients: A Retrospective Case-Control Study.

: To determine if MRI features and molecular subtype influence the detectability of breast cancers on MRI in high-risk patients. : Breast cancers in a high-risk population of 104 patients were diagnosed following MRI describing a BI-RADS 4-5 lesion. MRI characteristics at the time of diagnosis were compared with previous MRI, where a BI-RADS 1-2-3 lesion was described.

Quantitative single-cell analysis of immunofluorescence protein multiplex images illustrates biomarker spatial heterogeneity within breast cancer subtypes.

Background: The extent of cellular heterogeneity in breast cancer could have potential impact on diagnosis and long-term outcome. However, pathology evaluation is limited to biomarker immunohistochemical staining and morphology of the bulk cancer. Inter-cellular heterogeneity of biomarkers is not usually assessed.

Self-supervised driven consistency training for annotation efficient histopathology image analysis.

Training a neural network with a large labeled dataset is still a dominant paradigm in computational histopathology. However, obtaining such exhaustive manual annotations is often expensive, laborious, and prone to inter and intra-observer variability. While recent self-supervised and semi-supervised methods can alleviate this need by learning unsupervised feature representations, they still struggle to generalize well to downstream tasks when the number of labeled instances is small.

SPIE-AAPM-NCI BreastPathQ challenge: an image analysis challenge for quantitative tumor cellularity assessment in breast cancer histology images following neoadjuvant treatment.

: The breast pathology quantitative biomarkers (BreastPathQ) challenge was a grand challenge organized jointly by the International Society for Optics and Photonics (SPIE), the American Association of Physicists in Medicine (AAPM), the U.S. National Cancer Institute (NCI), and the U.

Overcoming the limitations of patch-based learning to detect cancer in whole slide images.

Whole slide images (WSIs) pose unique challenges when training deep learning models. They are very large which makes it necessary to break each image down into smaller patches for analysis, image features have to be extracted at multiple scales in order to capture both detail and context, and extreme class imbalances may exist. Significant progress has been made in the analysis of these images, thanks largely due to the availability of public annotated datasets.

Loss odyssey in medical image segmentation.

The loss function is an important component in deep learning-based segmentation methods. Over the past five years, many loss functions have been proposed for various segmentation tasks. However, a systematic study of the utility of these loss functions is missing.

Learning to segment images with classification labels.

Two of the most common tasks in medical imaging are classification and segmentation. Either task requires labeled data annotated by experts, which is scarce and expensive to collect. Annotating data for segmentation is generally considered to be more laborious as the annotator has to draw around the boundaries of regions of interest, as opposed to assigning image patches a class label.

Deep neural network models for computational histopathology: A survey.

Histopathological images contain rich phenotypic information that can be used to monitor underlying mechanisms contributing to disease progression and patient survival outcomes. Recently, deep learning has become the mainstream methodological choice for analyzing and interpreting histology images. In this paper, we present a comprehensive review of state-of-the-art deep learning approaches that have been used in the context of histopathological image analysis.

Convolutional neural networks versus radiologists in characterization of small hypoattenuating hepatic nodules on CT: a critical diagnostic challenge in staging of colorectal carcinoma.

Our objective was to compare the diagnostic performance and diagnostic confidence of convolutional neural networks (CNN) to radiologists in characterizing small hypoattenuating hepatic nodules (SHHN) in colorectal carcinoma (CRC) on CT scans. Retrospective review of CRC CT scans over 6-years yielded 199 patients (550 SHHN) defined as < 1 cm in diameter. The reference standard was established through 1-year stability/MRI for benign or nodule evolution for malignant nodules.

BIAS: Transparent reporting of biomedical image analysis challenges.

The number of biomedical image analysis challenges organized per year is steadily increasing. These international competitions have the purpose of benchmarking algorithms on common data sets, typically to identify the best method for a given problem. Recent research, however, revealed that common practice related to challenge reporting does not allow for adequate interpretation and reproducibility of results.

Intensity Augmentation to Improve Generalizability of Breast Segmentation Across Different MRI Scan Protocols.

Objective: The segmentation of the breast from the chest wall is an important first step in the analysis of breast magnetic resonance images. 3D U-Nets have been shown to obtain high segmentation accuracy and appear to generalize well when trained on one scanner type and tested on another scanner, provided that a very similar MR protocol is used. There has, however, been little work addressing the problem of domain adaptation when image intensities or patient orientation differ markedly between the training set and an unseen test set.

Use of radiomics for the prediction of local control of brain metastases after stereotactic radiosurgery.

Background: Local response prediction for brain metastases (BM) after stereotactic radiosurgery (SRS) is challenging, particularly for smaller BM, as existing criteria are based solely on unidimensional measurements. This investigation sought to determine whether radiomic features provide additional value to routinely available clinical and dosimetric variables to predict local recurrence following SRS.

Methods: Analyzed were 408 BM in 87 patients treated with SRS.

Automated and Manual Quantification of Tumour Cellularity in Digital Slides for Tumour Burden Assessment.

The residual cancer burden index is an important quantitative measure used for assessing treatment response following neoadjuvant therapy for breast cancer. It has shown to be predictive of overall survival and is composed of two key metrics: qualitative assessment of lymph nodes and the percentage of invasive or in situ tumour cellularity (TC) in the tumour bed (TB). Currently, TC is assessed through eye-balling of routine histopathology slides estimating the proportion of tumour cells within the TB.

Sample-Size Determination Methodologies for Machine Learning in Medical Imaging Research: A Systematic Review.

Purpose: The required training sample size for a particular machine learning (ML) model applied to medical imaging data is often unknown. The purpose of this study was to provide a descriptive review of current sample-size determination methodologies in ML applied to medical imaging and to propose recommendations for future work in the field.

Methods: We conducted a systematic literature search of articles using Medline and Embase with keywords including "machine learning," "image," and "sample size.

An investigation of the effect of fat suppression and dimensionality on the accuracy of breast MRI segmentation using U-nets.

Purpose: Accurate segmentation of the breast is required for breast density estimation and the assessment of background parenchymal enhancement, both of which have been shown to be related to breast cancer risk. The MRI breast segmentation task is challenging, and recent work has demonstrated that convolutional neural networks perform well for this task. In this study, we have investigated the performance of several two-dimensional (2D) U-Net and three-dimensional (3D) U-Net configurations using both fat-suppressed and nonfat-suppressed images.

A graph-based lesion characterization and deep embedding approach for improved computer-aided diagnosis of nonmass breast MRI lesions.

Nonmass-like enhancements are a common but diagnostically challenging finding in breast MRI. Nonmass-like lesions can be described as clusters of spatially and temporally inter-connected regions of enhancements, so they can be modeled as networks and their properties characterized via network-based connectivity. In this work, we represented nonmass lesions as graphs using a link formation energy model that favors linkages between regions of similar enhancement and closer spatial proximity.