Virtual birefringence imaging and histological staining of amyloid deposits in label-free tissue using autofluorescence microscopy and deep learning.

Systemic amyloidosis involves the deposition of misfolded proteins in organs/tissues, leading to progressive organ dysfunction and failure. Congo red is the gold-standard chemical stain for visualizing amyloid deposits in tissue, showing birefringence under polarization microscopy. However, Congo red staining is tedious and costly to perform, and prone to false diagnoses due to variations in amyloid amount, staining quality and manual examination of tissue under a polarization microscope.

Roadmap on computational methods in optical imaging and holography [invited].

Unlabelled: Computational methods have been established as cornerstones in optical imaging and holography in recent years. Every year, the dependence of optical imaging and holography on computational methods is increasing significantly to the extent that optical methods and components are being completely and efficiently replaced with computational methods at low cost. This roadmap reviews the current scenario in four major areas namely incoherent digital holography, quantitative phase imaging, imaging through scattering layers, and super-resolution imaging.

Pyramid diffractive optical networks for unidirectional image magnification and demagnification.

Diffractive deep neural networks (DNNs) are composed of successive transmissive layers optimized using supervised deep learning to all-optically implement various computational tasks between an input and output field-of-view. Here, we present a pyramid-structured diffractive optical network design (which we term P-DNN), optimized specifically for unidirectional image magnification and demagnification. In this design, the diffractive layers are pyramidally scaled in alignment with the direction of the image magnification or demagnification.

Automated HER2 Scoring in Breast Cancer Images Using Deep Learning and Pyramid Sampling.

Human epidermal growth factor receptor 2 (HER2) is a critical protein in cancer cell growth that signifies the aggressiveness of breast cancer (BC) and helps predict its prognosis. Here, we introduce a deep learning-based approach utilizing pyramid sampling for the automated classification of HER2 status in immunohistochemically (IHC) stained BC tissue images. Accurate assessment of IHC-stained tissue slides for HER2 expression levels is essential for both treatment guidance and understanding of cancer mechanisms.

Information-hiding cameras: Optical concealment of object information into ordinary images.

We introduce an information-hiding camera integrated with an electronic decoder that is jointly optimized through deep learning. This system uses a diffractive optical processor, which transforms and hides input images into ordinary-looking patterns that deceive/mislead observers. This information-hiding transformation is valid for infinitely many combinations of secret messages, transformed into ordinary-looking output images through passive light-matter interactions within the diffractive processor.

Broadband nonlinear modulation of incoherent light using a transparent optoelectronic neuron array.

Nonlinear optical processing of ambient natural light is highly desired for computational imaging and sensing. Strong optical nonlinear response under weak broadband incoherent light is essential for this purpose. By merging 2D transparent phototransistors (TPTs) with liquid crystal (LC) modulators, we create an optoelectronic neuron array that allows self-amplitude modulation of spatially incoherent light, achieving a large nonlinear contrast over a broad spectrum at orders-of-magnitude lower intensity than achievable in most optical nonlinear materials.

Label-Free Virtual HER2 Immunohistochemical Staining of Breast Tissue using Deep Learning.

The immunohistochemical (IHC) staining of the human epidermal growth factor receptor 2 (HER2) biomarker is widely practiced in breast tissue analysis, preclinical studies, and diagnostic decisions, guiding cancer treatment and investigation of pathogenesis. HER2 staining demands laborious tissue treatment and chemical processing performed by a histotechnologist, which typically takes one day to prepare in a laboratory, increasing analysis time and associated costs. Here, we describe a deep learning-based virtual HER2 IHC staining method using a conditional generative adversarial network that is trained to rapidly transform autofluorescence microscopic images of unlabeled/label-free breast tissue sections into bright-field equivalent microscopic images, matching the standard HER2 IHC staining that is chemically performed on the same tissue sections.

Universal linear intensity transformations using spatially incoherent diffractive processors.

Under spatially coherent light, a diffractive optical network composed of structured surfaces can be designed to perform any arbitrary complex-valued linear transformation between its input and output fields-of-view (FOVs) if the total number (N) of optimizable phase-only diffractive features is ≥~2NN, where N and N refer to the number of useful pixels at the input and the output FOVs, respectively. Here we report the design of a spatially incoherent diffractive optical processor that can approximate any arbitrary linear transformation in time-averaged intensity between its input and output FOVs. Under spatially incoherent monochromatic light, the spatially varying intensity point spread function (H) of a diffractive network, corresponding to a given, arbitrarily-selected linear intensity transformation, can be written as H(m, n; m', n') = |h(m, n; m', n')|, where h is the spatially coherent point spread function of the same diffractive network, and (m, n) and (m', n') define the coordinates of the output and input FOVs, respectively.

Rapid and stain-free quantification of viral plaque via lens-free holography and deep learning.

A plaque assay-the gold-standard method for measuring the concentration of replication-competent lytic virions-requires staining and usually more than 48 h of runtime. Here we show that lens-free holographic imaging and deep learning can be combined to expedite and automate the assay. The compact imaging device captures phase information label-free at a rate of approximately 0.

Data-Class-Specific All-Optical Transformations and Encryption.

Diffractive optical networks provide rich opportunities for visual computing tasks. Here, data-class-specific transformations that are all-optically performed between the input and output fields-of-view (FOVs) of a diffractive network are presented. The visual information of the objects is encoded into the amplitude (A), phase (P), or intensity (I) of the optical field at the input, which is all-optically processed by a data-class-specific diffractive network.

Unidirectional imaging using deep learning-designed materials.

A unidirectional imager would only permit image formation along one direction, from an input field-of-view (FOV) A to an output FOV B, and in the reverse path, B → A, the image formation would be blocked. We report the first demonstration of unidirectional imagers, presenting polarization-insensitive and broadband unidirectional imaging based on successive diffractive layers that are linear and isotropic. After their deep learning-based training, the resulting diffractive layers are fabricated to form a unidirectional imager.

Roadmap on Deep Learning for Microscopy.

Through digital imaging, microscopy has evolved from primarily being a means for visual observation of life at the micro- and nano-scale, to a quantitative tool with ever-increasing resolution and throughput. Artificial intelligence, deep neural networks, and machine learning are all niche terms describing computational methods that have gained a pivotal role in microscopy-based research over the past decade. This Roadmap is written collectively by prominent researchers and encompasses selected aspects of how machine learning is applied to microscopy image data, with the aim of gaining scientific knowledge by improved image quality, automated detection, segmentation, classification and tracking of objects, and efficient merging of information from multiple imaging modalities.

All-optical image classification through unknown random diffusers using a single-pixel diffractive network.

Classification of an object behind a random and unknown scattering medium sets a challenging task for computational imaging and machine vision fields. Recent deep learning-based approaches demonstrated the classification of objects using diffuser-distorted patterns collected by an image sensor. These methods demand relatively large-scale computing using deep neural networks running on digital computers.

Deep learning-enabled virtual histological staining of biological samples.

Histological staining is the gold standard for tissue examination in clinical pathology and life-science research, which visualizes the tissue and cellular structures using chromatic dyes or fluorescence labels to aid the microscopic assessment of tissue. However, the current histological staining workflow requires tedious sample preparation steps, specialized laboratory infrastructure, and trained histotechnologists, making it expensive, time-consuming, and not accessible in resource-limited settings. Deep learning techniques created new opportunities to revolutionize staining methods by digitally generating histological stains using trained neural networks, providing rapid, cost-effective, and accurate alternatives to standard chemical staining methods.

Biopsy-free in vivo virtual histology of skin using deep learning.

An invasive biopsy followed by histological staining is the benchmark for pathological diagnosis of skin tumors. The process is cumbersome and time-consuming, often leading to unnecessary biopsies and scars. Emerging noninvasive optical technologies such as reflectance confocal microscopy (RCM) can provide label-free, cellular-level resolution, in vivo images of skin without performing a biopsy.

Calcium pyrophosphate crystal size and characteristics.

Objective: To describe the characteristics of calcium pyrophosphate (CPP) crystal size and morphology under compensated polarized light microscopy (CPLM). Secondarily, to describe CPP crystals seen only with digital enhancement of CPLM images, confirmed with advanced imaging techniques.

Methods: Clinical lab-identified CPP-positive synovial fluid samples were collected from 16 joint aspirates.

Deep Learning-Based Holographic Polarization Microscopy.

Polarized light microscopy provides high contrast to birefringent specimen and is widely used as a diagnostic tool in pathology. However, polarization microscopy systems typically operate by analyzing images collected from two or more light paths in different states of polarization, which lead to relatively complex optical designs, high system costs, or experienced technicians being required. Here, we present a deep learning-based holographic polarization microscope that is capable of obtaining quantitative birefringence retardance and orientation information of specimen from a phase-recovered hologram, while only requiring the addition of one polarizer/analyzer pair to an inline lensfree holographic imaging system.

Computational cytometer based on magnetically modulated coherent imaging and deep learning.

Detecting rare cells within blood has numerous applications in disease diagnostics. Existing rare cell detection techniques are typically hindered by their high cost and low throughput. Here, we present a computational cytometer based on magnetically modulated lensless speckle imaging, which introduces oscillatory motion to the magnetic-bead-conjugated rare cells of interest through a periodic magnetic force and uses lensless time-resolved holographic speckle imaging to rapidly detect the target cells in three dimensions (3D).

Pathological crystal imaging with single-shot computational polarized light microscopy.

Pathological crystal identification is routinely practiced in rheumatology for diagnosing arthritis disease such as gout, and relies on polarized light microscopy as the gold standard method used by medical professionals. Here, we present a single-shot computational polarized light microscopy method that reconstructs the transmittance, retardance and slow-axis orientation of a birefringent sample using a single image captured with a pixelated-polarizer camera. This method is fast, simple-to-operate and compatible with all the existing standard microscopes without extensive or costly modifications.