EXSCLAIM!: Harnessing materials science literature for self-labeled microscopy datasets.

This work introduces the EXSCLAIM! toolkit for the automatic extraction, separation, and caption-based natural language annotation of images from scientific literature. EXSCLAIM! is used to show how rule-based natural language processing and image recognition can be leveraged to construct an electron microscopy dataset containing thousands of keyword-annotated nanostructure images. Moreover, it is demonstrated how a combination of statistical topic modeling and semantic word similarity comparisons can be used to increase the number and variety of keyword annotations on top of the standard annotations from EXSCLAIM! With large-scale imaging datasets constructed from scientific literature, users are well positioned to train neural networks for classification and recognition tasks specific to microscopy-tasks often otherwise inhibited by a lack of sufficient annotated training data.

Boost event-driven tactile learning with location spiking neurons.

Tactile sensing is essential for a variety of daily tasks. Inspired by the event-driven nature and sparse spiking communication of the biological systems, recent advances in event-driven tactile sensors and Spiking Neural Networks (SNNs) spur the research in related fields. However, SNN-enabled event-driven tactile learning is still in its infancy due to the limited representation abilities of existing spiking neurons and high spatio-temporal complexity in the event-driven tactile data.

Whole-brain imaging of freely-moving zebrafish.

One of the holy grails of neuroscience is to record the activity of every neuron in the brain while an animal moves freely and performs complex behavioral tasks. While important steps forward have been taken recently in large-scale neural recording in rodent models, single neuron resolution across the entire mammalian brain remains elusive. In contrast the larval zebrafish offers great promise in this regard.

A Joint Intensity-Neuromorphic Event Imaging System With Bandwidth-Limited Communication Channel.

We present a novel adaptive multimodal intensity-event algorithm to optimize an overall objective of object tracking under bit rate constraints for a host-chip architecture. The chip is a computationally resource-constrained device acquiring high-resolution intensity frames and events, while the host is capable of performing computationally expensive tasks. We develop a joint intensity-neuromorphic event rate-distortion compression framework with a quadtree (QT)-based compression of intensity and events scheme.

Fast non-line-of-sight imaging with high-resolution and wide field of view using synthetic wavelength holography.

The presence of a scattering medium in the imaging path between an object and an observer is known to severely limit the visual acuity of the imaging system. We present an approach to circumvent the deleterious effects of scattering, by exploiting spectral correlations in scattered wavefronts. Our Synthetic Wavelength Holography (SWH) method is able to recover a holographic representation of hidden targets with sub-mm resolution over a nearly hemispheric angular field of view.

Guided Event Filtering: Synergy Between Intensity Images and Neuromorphic Events for High Performance Imaging.

Many visual and robotics tasks in real-world scenarios rely on robust handling of high speed motion and high dynamic range (HDR) with effectively high spatial resolution and low noise. Such stringent requirements, however, cannot be directly satisfied by a single imager or imaging modality, rather by multi-modal sensors with complementary advantages. In this paper, we address high performance imaging by exploring the synergy between traditional frame-based sensors with high spatial resolution and low sensor noise, and emerging event-based sensors with high speed and high dynamic range.

Exploiting Wavelength Diversity for High Resolution Time-of-Flight 3D Imaging.

The poor lateral and depth resolution of state-of-the-art 3D sensors based on the time-of-flight (ToF) principle has limited widespread adoption to a few niche applications. In this work, we introduce a novel sensor concept that provides ToF-based 3D measurements of real world objects and surfaces with depth precision up to 35 μm and point cloud densities commensurate with the native sensor resolution of standard CMOS/CCD detectors (up to several megapixels). Such capabilities are realized by combining the best attributes of continuous wave ToF sensing, multi-wavelength interferometry, and heterodyne interferometry into a single approach.

Intensity interferometry-based 3D imaging.

The development of single-photon counting detectors and arrays has made tremendous steps in recent years, not the least because of various new applications, e.g., LIDAR devices.

Bayesian analysis of depth resolved OCT attenuation coefficients.

Optical coherence tomography (OCT) is an optical technique which allows for volumetric visualization of the internal structures of translucent materials. Additional information can be gained by measuring the rate of signal attenuation in depth. Techniques have been developed to estimate the rate of attenuation on a voxel by voxel basis.

Broadband X-ray ptychography using multi-wavelength algorithm.

Ptychography is a rapidly developing scanning microscopy which is able to view the internal structures of samples at a high resolution beyond the illumination size. The achieved spatial resolution is theoretically dose-limited. A broadband source can provide much higher flux compared with a monochromatic source; however, it conflicts with the necessary coherence requirements of this coherent diffraction imaging technique.

Rapid reconstruction of highly undersampled, non-Cartesian real-time cine k-space data using a perceptual complex neural network (PCNN).

Highly accelerated real-time cine MRI using compressed sensing (CS) is a promising approach to achieve high spatio-temporal resolution and clinically acceptable image quality in patients with arrhythmia and/or dyspnea. However, its lengthy image reconstruction time may hinder its clinical translation. The purpose of this study was to develop a neural network for reconstruction of non-Cartesian real-time cine MRI k-space data faster (<1 min per slice with 80 frames) than graphics processing unit (GPU)-accelerated CS reconstruction, without significant loss in image quality or accuracy in left ventricular (LV) functional parameters.

Computational Optical Sensing and Imaging: feature issue introduction.

This Feature Issue includes 19 articles that highlight advances in the field of Computational Optical Sensing and Imaging. Many of the articles were presented at the 2019 OSA Topical Meeting on Computational Optical Sensing and Imaging held in Munich, Germany, on June 24-27. Articles featured in the issue cover a broad array of topics ranging from imaging through scattering media, imaging round corners and compressive imaging to machine learning for recovery of images.

Compressive ghost imaging through scattering media with deep learning.

Imaging through scattering media is challenging since the signal to noise ratio (SNR) of the reflection can be heavily reduced by scatterers. Single-pixel detectors (SPD) with high sensitivities offer compelling advantages for sensing such weak signals. In this paper, we focus on the use of ghost imaging to resolve 2D spatial information using just an SPD.

Rapid Reconstruction of Four-dimensional MR Angiography of the Thoracic Aorta Using a Convolutional Neural Network.

Purpose: To implement an integrated reconstruction pipeline including a graphics processing unit (GPU)-based convolutional neural network (CNN) architecture and test whether it reconstructs four-dimensional non-Cartesian, non-contrast material-enhanced MR angiographic k-space data faster than a central processing unit (CPU)-based compressed sensing (CS) reconstruction pipeline, without significant losses in data fidelity, summed visual score (SVS), or arterial vessel-diameter measurements.

Materials And Methods: Raw k-space data of 24 patients (18 men and six women; mean age, 56.8 years ± 11.

Snapshot multifocal light field microscopy.

Light field microscopy (LFM) is an emerging technology for high-speed wide-field 3D imaging by capturing 4D light field of 3D volumes. However, its 3D imaging capability comes at a cost of lateral resolution. In addition, the lateral resolution is not uniform across depth in the light field dconvolution reconstructions.

Hand-guided qualitative deflectometry with a mobile device.

We introduce a system that exploits the screen and front-facing camera of a mobile device to perform three-dimensional deflectometry-based surface measurements. In contrast to current mobile deflectometry systems, our method can capture surfaces with large normal variation and wide field of view (FoV). We achieve this by applying automated multi-view panoramic stitching algorithms to produce a large FoV normal map from a hand-guided capture process without the need for external tracking systems, like robot arms or fiducials.

Computational multifocal microscopy.

Despite recent advances, high performance single-shot 3D microscopy remains an elusive task. By introducing designed diffractive optical elements (DOEs), one is capable of converting a microscope into a 3D "kaleidoscope," in which case the snapshot image consists of an array of tiles and each tile focuses on different depths. However, the acquired multifocal microscopic (MFM) image suffers from multiple sources of degradation, which prevents MFM from further applications.

Design and simulation of a snapshot multi-focal interferometric microscope.

Realizing both high temporal and spatial resolution across a large volume is a key challenge for 3D fluorescent imaging. Towards achieving this objective, we introduce an interferometric multifocus microscopy (iMFM) system, a combination of multifocus microscopy (MFM) with two opposing objective lenses. We show that the proposed iMFM is capable of simultaneously producing multiple focal plane interferometry that provides axial super-resolution and hence isotropic 3D resolution with a single exposure.

Nonlinear Unmixing of Hyperspectral Datasets for the Study of Painted Works of Art.

Nonlinear unmixing of hyperspectral reflectance data is one of the key problems in quantitative imaging of painted works of art. The approach presented is to interrogate a hyperspectral image cube by first decomposing it into a set of reflectance curves representing pure basis pigments and second to estimate the scattering and absorption coefficients of each pigment in a given pixel to produce estimates of the component fractions. This two-step algorithm uses a deep neural network to qualitatively identify the constituent pigments in any unknown spectrum and, based on the pigment(s) present and Kubelka-Munk theory to estimate the pigment concentration on a per-pixel basis.

CS-ToF: High-resolution compressive time-of-flight imaging.

Three-dimensional imaging using Time-of-flight (ToF) sensors is rapidly gaining widespread adoption in many applications due to their cost effectiveness, simplicity, and compact size. However, the current generation of ToF cameras suffers from low spatial resolution due to physical fabrication limitations. In this paper, we propose CS-ToF, an imaging architecture to achieve high spatial resolution ToF imaging via optical multiplexing and compressive sensing.

High-depth-resolution range imaging with multiple-wavelength superheterodyne interferometry using 1550-nm lasers.

Lasers and laser diodes are widely used as illumination sources for optical imaging techniques. Time-of-flight (ToF) cameras with laser diodes and range imaging based on optical interferometry systems using lasers are among these techniques, with various applications in fields such as metrology and machine vision. ToF cameras can have imaging ranges of several meters, but offer only centimeter-level depth resolution.

Rapid alignment of nanotomography data using joint iterative reconstruction and reprojection.

As x-ray and electron tomography is pushed further into the nanoscale, the limitations of rotation stages become more apparent, leading to challenges in the alignment of the acquired projection images. Here we present an approach for rapid post-acquisition alignment of these projections to obtain high quality three-dimensional images. Our approach is based on a joint estimation of alignment errors, and the object, using an iterative refinement procedure.

Subsampled phase retrieval for temporal resolution enhancement in lensless on-chip holographic video.

On-chip holographic video is a convenient way to monitor biological samples simultaneously at high spatial resolution and over a wide field-of-view. However, due to the limited readout rate of digital detector arrays, one often faces a tradeoff between the per-frame pixel count and frame rate of the captured video. In this report, we propose a subsampled phase retrieval (SPR) algorithm to overcome the spatial-temporal trade-off in holographic video.

SAVI: Synthetic apertures for long-range, subdiffraction-limited visible imaging using Fourier ptychography.

Synthetic aperture radar is a well-known technique for improving resolution in radio imaging. Extending these synthetic aperture techniques to the visible light domain is not straightforward because optical receivers cannot measure phase information. We propose to use macroscopic Fourier ptychography (FP) as a practical means of creating a synthetic aperture for visible imaging to achieve subdiffraction-limited resolution.

Compressive holographic video.

Compressed sensing has been discussed separately in spatial and temporal domains. Compressive holography has been introduced as a method that allows 3D tomographic reconstruction at different depths from a single 2D image. Coded exposure is a temporal compressed sensing method for high speed video acquisition.