TC-Net: A joint learning framework based on CNN and vision transformer for multi-lesion medical images segmentation.

Background: With the rapid advancement of medical imaging technology, the demand for accurate segmentation of medical images is increasing. However, most existing methods are unable to capture locality and long-range dependency information in integrated ways for medical images.

Method: In this paper, we propose an elegant segmentation framework for medical images named TC-Net, which can utilize both the locality-aware and long-range dependencies in the medical images. As for the locality-aware perspective, we employ a CNN-based encoder and decoder structure. The CNN branch uses the locality of convolution operations to dig out local information in medical images. As for the long-range dependencies, we construct a Transformer branch to focus on the global context. Additionally, we proposed a locality-aware and long-range dependency concatenation strategy (LLCS) to aggregate the feature maps obtained from the two subbranches. Finally, we present a dynamic cyclical focal loss (DCFL) to address the class imbalance problem in multi-lesion segmentation.

Results: Comprehensive experiments were conducted on lesion segmentation tasks using two fundus image databases and a skin image database. The TC-Net achieves scores of 0.6985 and 0.5171 in the metric of mean pixel accuracy on the IDRiD and DDR databases, respectively. Moreover, on the skin image database, the TC-Net reached mean pixel accuracy of 0.8886. The experiment results demonstrate that the proposed method achieves better performance than other deep learning segmentation schemes. Furthermore, the proposed DCFL achieves higher performance than other loss functions in multi-lesion segmentation.

Significance: The proposed TC-Net is a promising new framework for multi-lesion medical image segmentation and many other challenging image segmentation tasks. © 2001 Elsevier Science. All rights reserved.