Metrics reloaded: recommendations for image analysis validation.

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics.

Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers.

Possibilities of Using Multi-b-value Diffusion Magnetic Resonance Imaging for Classification of Brain Lesions.

In contrast to conventional diffusion magnetic resonance imaging (MRI), multi-b-value diffusion MRI methods are able to separate the signal from free water, pseudo-diffusion, and non-Gaussian components of water molecule diffusion. These approaches can then be utilised in so-called intravoxel incoherent motion imaging and diffusion kurtosis imaging. Various parameters provided by these methods can describe additional characteristics of the tissue microstructure and potentially help in the diagnosis and classification of various pathological processes.

The Cell Tracking Challenge: 10 years of objective benchmarking.

The Cell Tracking Challenge is an ongoing benchmarking initiative that has become a reference in cell segmentation and tracking algorithm development. Here, we present a significant number of improvements introduced in the challenge since our 2017 report. These include the creation of a new segmentation-only benchmark, the enrichment of the dataset repository with new datasets that increase its diversity and complexity, and the creation of a silver standard reference corpus based on the most competitive results, which will be of particular interest for data-hungry deep learning-based strategies.

Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers.

Lactic Acidosis Interferes With Toxicity of Perifosine to Colorectal Cancer Spheroids: Multimodal Imaging Analysis.

Colorectal cancer (CRC) is a disease with constantly increasing incidence and high mortality. The treatment efficacy could be curtailed by drug resistance resulting from poor drug penetration into tumor tissue and the tumor-specific microenvironment, such as hypoxia and acidosis. Furthermore, CRC tumors can be exposed to different pH depending on the position in the intestinal tract.

BIAS: Transparent reporting of biomedical image analysis challenges.

The number of biomedical image analysis challenges organized per year is steadily increasing. These international competitions have the purpose of benchmarking algorithms on common data sets, typically to identify the best method for a given problem. Recent research, however, revealed that common practice related to challenge reporting does not allow for adequate interpretation and reproducibility of results.

Drug Penetration Analysis in 3D Cell Cultures Using Fiducial-Based Semiautomatic Coregistration of MALDI MSI and Immunofluorescence Images.

In this paper, we present an easy-to-follow procedure for the analysis of tissue sections from 3D cell cultures (spheroids) by matrix-assisted laser desorption/ionization mass spectrometry imaging (MALDI MSI) and laser scanning confocal microscopy (LSCM). MALDI MSI was chosen to detect the distribution of the drug of interest, while fluorescence immunohistochemistry (IHC) followed by LSCM was used to localize the cells featuring specific markers of viability, proliferation, apoptosis and metastasis. The overlay of the mass spectrometry (MS) and IHC spheroid images, typically without any morphological features, required fiducial-based coregistration.

Quantitative Assessment of Anti-Cancer Drug Efficacy From Coregistered Mass Spectrometry and Fluorescence Microscopy Images of Multicellular Tumor Spheroids.

Spheroids-three-dimensional aggregates of cells grown from a cancer cell line-represent a model of living tissue for chemotherapy investigation. Distribution of chemotherapeutics in spheroid sections was determined using the matrix-assisted laser desorption/ionization mass spectrometry imaging (MALDI MSI). Proliferating or apoptotic cells were immunohistochemically labeled and visualized by laser scanning confocal fluorescence microscopy (LSCM).

CytoPacq: a web-interface for simulating multi-dimensional cell imaging.

Motivation: Objective assessment of bioimage analysis methods is an essential step towards understanding their robustness and parameter sensitivity, calling for the availability of heterogeneous bioimage datasets accompanied by their reference annotations. Because manual annotations are known to be arduous, highly subjective and barely reproducible, numerous simulators have emerged over past decades, generating synthetic bioimage datasets complemented with inherent reference annotations. However, the installation and configuration of these tools generally constitutes a barrier to their widespread use.

Author Correction: Why rankings of biomedical image analysis competitions should be interpreted with care.

In the original version of this Article the values in the rightmost column of Table 1 were inadvertently shifted relative to the other columns. This has now been corrected in the PDF and HTML versions of the Article.

Why rankings of biomedical image analysis competitions should be interpreted with care.

International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now.

An objective comparison of cell-tracking algorithms.

We present a combined report on the results of three editions of the Cell Tracking Challenge, an ongoing initiative aimed at promoting the development and objective evaluation of cell segmentation and tracking algorithms. With 21 participating algorithms and a data repository consisting of 13 data sets from various microscopy modalities, the challenge displays today's state-of-the-art methodology in the field. We analyzed the challenge results using performance measures for segmentation and tracking that rank all participating methods.

Image-based surrogate biomarkers for molecular subtypes of colorectal cancer.

Motivation: Whole genome expression profiling of large cohorts of different types of cancer led to the identification of distinct molecular subcategories (subtypes) that may partially explain the observed inter-tumoral heterogeneity. This is also the case of colorectal cancer (CRC) where several such categorizations have been proposed. Despite recent developments, the problem of subtype definition and recognition remains open, one of the causes being the intrinsic heterogeneity of each tumor, which is difficult to estimate from gene expression profiles.

Virtual cell imaging: A review on simulation methods employed in image cytometry.

The simulations of cells and microscope images thereof have been used to facilitate the development, selection, and validation of image analysis algorithms employed in cytometry as well as for modeling and understanding cell structure and dynamics beyond what is visible in the eyepiece. The simulation approaches vary from simple parametric models of specific cell components-especially shapes of cells and cell nuclei-to learning-based synthesis and multi-stage simulation models for complex scenes that simultaneously visualize multiple object types and incorporate various properties of the imaged objects and laws of image formation. This review covers advances in artificial digital cell generation at scales ranging from particles up to tissue synthesis and microscope image simulation methods, provides examples of the use of simulated images for various purposes ranging from subcellular object detection to cell tracking, and discusses how such simulators have been validated.

Challenges and Benchmarks in Bioimage Analysis.

Similar to the medical imaging community, the bioimaging community has recently realized the need to benchmark various image analysis methods to compare their performance and assess their suitability for specific applications. Challenges sponsored by prestigious conferences have proven to be an effective means of encouraging benchmarking and new algorithm development for a particular type of image data. Bioimage analysis challenges have recently complemented medical image analysis challenges, especially in the case of the International Symposium on Biomedical Imaging (ISBI).

Characterization of the role of collagen network structure and composition in cancer cell migration.

The geometry of 3D collagen networks is a key factor that influences the behavior of live cells within extra-cellular matrices. This paper presents a method for automatic quantification of the 3D collagen network geometry with fiber resolution in confocal reflection microscopy images. The proposed method is based on a smoothing filter and binarization of the collagen network followed by a fiber reconstruction algorithm.

Cell Tracking Accuracy Measurement Based on Comparison of Acyclic Oriented Graphs.

Tracking motile cells in time-lapse series is challenging and is required in many biomedical applications. Cell tracks can be mathematically represented as acyclic oriented graphs. Their vertices describe the spatio-temporal locations of individual cells, whereas the edges represent temporal relationships between them.

Performance and sensitivity evaluation of 3D spot detection methods in confocal microscopy.

Reliable 3D detection of diffraction-limited spots in fluorescence microscopy images is an important task in subcellular observation. Generally, fluorescence microscopy images are heavily degraded by noise and non-specifically stained background, making reliable detection a challenging task. In this work, we have studied the performance and parameter sensitivity of eight recent methods for 3D spot detection.

A benchmark for comparison of cell tracking algorithms.

Motivation: Automatic tracking of cells in multidimensional time-lapse fluorescence microscopy is an important task in many biomedical applications. A novel framework for objective evaluation of cell tracking algorithms has been established under the auspices of the IEEE International Symposium on Biomedical Imaging 2013 Cell Tracking Challenge. In this article, we present the logistics, datasets, methods and results of the challenge and lay down the principles for future uses of this benchmark.

Knockdown of apoptosis-inducing factor disrupts function of respiratory complex I.

Recent findings suggest that apoptotic protein apoptosis-inducing factor (AIF) may also play an important non-apoptotic function inside mitochondria. AIF was proposed to be an important component of respiratory chain complex I that is the major producer of superoxide radical. The possible role of AIF is still controversial.