radiation sensitivity assessment for individual head and neck cancer patients using deep learning-based automated nuclei and DNA damage foci detection.

Introduction: Tumor biopsy tissue response to irradiation is potentially an interesting biomarker for tumor response, therefore, for treatment personalization. Tumor response can be characterized by DNA damage response, expressed by the large-scale presence of DNA damage foci in tumor nuclei. Currently, characterizing tumor nuclei and DNA damage foci is a manual process that takes hours per patient and is subjective to inter-observer variability, which is not feasible in for clinical decision making. Therefore, our goal was to develop a method to automatically segment nuclei and DNA damage foci in tumor tissue samples treated with radiation to characterize the DNA damage response, as potential biomarker for radio-sensitivity.

Methods: Oral cavity tumor tissue of 21 patients was irradiated (5 or 0 Gy), fixated 2 h post-radiation, and used to develop our method for automated nuclei and 53BP1 foci segmentation. The segmentation model used both deep learning and conventional image-analysis techniques. The training (22 %), validation (22 %), and test set (56 %) consisted of thousands of manually segmented nuclei and foci. The segmentations and number of foci per nucleus in the test set were compared to their ground truths.

Results: The automatic nuclei and foci segmentations were highly accurate (Dice = 0.901 and Dice = 0.749, respectively). An excellent correlation (R = 0.802) was observed for the foci per nucleus that outperformed reported inter-observation variation. The analysis took ∼ 8 s per image.

Conclusion: This model can replace manual foci analysis for irradiation of head-and-neck squamous cell carcinoma tissue, reduces the image-analysis time from hours to minutes, avoids the problem of inter-observer variability, enables assessment of multiple images or conditions, and provides additional information about the foci size. Thereby, it allows for reliable and rapid radio-sensitivity assessment, as potential biomarker for response and treatment personalization.