Deep learning at the edge enables real-time streaming ptychographic imaging.

Coherent imaging techniques provide an unparalleled multi-scale view of materials across scientific and technological fields, from structural materials to quantum devices, from integrated circuits to biological cells. Driven by the construction of brighter sources and high-rate detectors, coherent imaging methods like ptychography are poised to revolutionize nanoscale materials characterization. However, these advancements are accompanied by significant increase in data and compute needs, which precludes real-time imaging, feedback and decision-making capabilities with conventional approaches.

20 µs-resolved high-throughput X-ray photon correlation spectroscopy on a 500k pixel detector enabled by data-management workflow.

The performance of the new 52 kHz frame rate Rigaku XSPA-500k detector was characterized on beamline 8-ID-I at the Advanced Photon Source at Argonne for X-ray photon correlation spectroscopy (XPCS) applications. Due to the large data flow produced by this detector (0.2 PB of data per 24 h of continuous operation), a workflow system was deployed that uses the Advanced Photon Source data-management (DM) system and high-performance software to rapidly reduce area-detector data to multi-tau and two-time correlation functions in near real time, providing human-in-the-loop feedback to experimenters.

Data management and processing workflow for the Materials Physics and Engineering group beamlines at the Advanced Photon Source.

The ability to store, organize, process and distribute experimental data effectively, efficiently and securely is particularly important for large user facilities like the Advanced Photon Source. In this article, the deployment of the APS Data Management System (DM) at the 1-ID and 6-BM beamlines of the APS is described. These two beamlines support a wide range of experimental techniques and generate data at relatively high rates, making them ideal candidates to illustrate the deployment and customization of the DM system and its tools.

APS Data Management System.

As the capabilities of modern X-ray detectors and acquisition technologies increase, so do the data rates and volumes produced at synchrotron beamlines. This brings into focus a number of challenges related to managing data at such facilities, including data transfer, near real-time data processing, automated processing pipelines, data storage, handling metadata and remote user access to data. The Advanced Photon Source Data Management System software is designed to help beamlines deal with these issues.