In object detection, current methods require voluminous manually-annotated region of interest (ROI) labels and class labels to train both the lesion detection and diagnosis models. In clinical practice, the ROI labels, i.e. ground truths, may not always be optimal for the classification task due to individual experience of sonologists, resulting in the issue of coarse annotation that limits the diagnosis performance of a deep-learning model. To solve problems above, a novel Two-Stage Detection and Diagnosis Network (TSDDNet) is proposed based on weakly supervised learning to enhance diagnostic accuracy of the ultrasound-based method for breast cancers.
Original Title:
Weakly Supervised Lesion Detection and Diagnosis for Breast Cancers with Partially Annotated Ultrasound Images.
Original Link:
https://ieeexplore.ieee.org/document/10439278
Original authors and their affiliations:
Jian Wang1, Liang Qiao1, Shichong Zhou2, Jin Zhou2, Jun Wang1, Juncheng Li1, Shihui Ying1，Cai Chang2, and Jun Shi1
1Shanghai University
2Fudan University Shanghai Cancer Center
Conference/Journal Information:
IEEE Transactions on Medical Imaging
I Contributions
1.A WSL-based ROI refinement method is developed to not only improve detection accuracy of the ROI-level pseudo labels predicted by the detection network, but also refine the manually annotated ground truth ROIs.
2.A novel TSDDNet with a two-stage training strategy is proposed for automatically detecting breast lesion and then diagnosing in a unified framework.
3.A self-distillation based joint optimization is proposed to incorporate the D-Net and C-Net into a unified framework in the second training stage.
II Data and Method
The experiments conducted on a B-mode breast ultrasound image (BBUI) dataset acquired from Fudan University Shanghai Cancer Center. There were totally 890 benign images and 870 malignant samples. All samples were scanned by the Mindray Resona7 ultrasound scanner (Shenzhen Mindray BioMedical Electronics Co., Ltd., Shenzhen, China) with the L11- 3 linear-array probe. For each BUS image, a rectangle ROI was annotated by an experienced sonologist to indicate the lesion area.
Fig 1. Framework of TSDDNet
Fig 1 described the framework of TSDDNet, which consists of 3 sub-networks, a detection network D-Net, a classification network C-Net, and a fusion network F-Net. And the training process consists of two stages.
The training samples was divided into two groups. One includes fully annotated images that have both ROI-level and image-level labels, and the other one includes partially annotated images that only have image-level labels. In the first training stage, both D-Net and C-Net are trained using the fully annotated images with both ROI-level and image-level labels, respectively. After that, the partially annotated images with only image-level labels are fed into the trained D-Net to generate ROI-level pseudo labels. Moreover, a candidate selection mechanism is designed to refine the ground truth ROI labels in the fully annotated images and the pseudo-ROI labels in the partially annotated images.
Fig 2. Joint training with self-distillation strategy.
In the second training stage, a self-distillation method is adopted to further finetune the DNet and C-Net by squeezing knowledge from F-net into the DNet and C-Net. As illustrated in Fig 2, the three sub-networks are cascaded in the second stage of TSDDNet. Meanwhile, the fully connection layer in C-Net is removed and two classifiers are added after the last layer of the D-Net and C-Net, respectively, during the joint training stage. Specifically, the classification information in the deep portion (F-Net) is squeezed into the shallow ones (C-Net and D-Net) to improve the lesion localization and discrimination performance of C-Net and D-Net, respectively.
III Results
Table 1 compares classification results of different methods. The weakly supervised approaches used a p-value of 0.2, while the supervised algorithms used a p-value of 1.0. The proposed TSDDNet achievesthe best average accuracy of 89.62 ± 1.24%, sensitivity of 90.73±1.15%, specificity of 87.59±1.27%, and YI of 78.32±1.95%. It also gets the improvements by at least 3.53%, 3.47%, 2.69%, and 6.59% on the corresponding indices over other compared algorithms.
Table 1. Classification results of different methods
Fig. 3 also presented the ROC curves and AUC values for all algorithms. It is seen that the ROC curve of TSDDNet outperforms all other algorithms with the highest AUC value of 0.928, indicating the best classification performance.
Fig 3. ROC curves and AUC values of different methods
As shown in Fig. 6, TSDDNet achieves the best detection results on breast lesions with the highest classification probability in BUS images, indicating that the ROIs predicted by TSDDNet are more suitable for the subsequent classification task in deep-learning. This benefits from the candidate selection mechanism used in TSDDNet.
Fig 4. Visualization results of manual and predicted ROIs. PP means Predicted Probability
Table 2 shows the results of ablation experiments. TSDDNet-B adopted the same network structure as TSDDNet, but removed both the candidate selection mechanism and self-distillation strategy. TSDDNet-B+CS utilized the same network as TSDDNet-B, but only applied the candidate selection mechanism in the first training stage to refine the ROI-level labels, and did not conduct self-distillation strategy in the second training stage. TSDDNet-B+SD adopted the same network as TSDDNet-B, but only performed the selfdistillation strategy in the second training stage, and did not conduct the candidate selection mechanism in the first training stage.
It can be observed that the proposed TSDDNet improves by at least 1.70%, 2.45%, 1.58%, and 4.04% on classification accuracy, sensitivity, specificity, YI, respectively, indicating the effectiveness of distinguishing the benign and malignant nature of lesions. Moreover, the TSDDNet-B+CS improves by 4.23%, 4.52%, 4.84%, 3.65%, and 8.16% on the corresponding indices over TSDDNet-B, which suggests that the candidate selection mechanism can effectively improve the classification performance by refining ROI bounding boxes in BUS images. On the other hand, compared to TSDDNet-B, TSDDNet-B+SD achieves improvements of 1.36%, 3.03%, 2.04%, and 5.07% on accuracy, sensitivity, specificity, YI, respectively. This fully demonstrates the effectiveness of F-Net, which transfers knowledge to D-Net and C-Net via self-distillation, helping both networks to learn more discriminant features for classification.
Table 2. Results of ablation Experiments
IV Conclusion
In conclusion, a novel TSDDNet is proposed to automatically detect and diagnose breast cancers, which is trained using only coarsely and partially annotated ROIs in BUS images. It integrates lesion detection network and classification network into a unified deep-learning model, and a two-stage training strategy is developed to solve the issue of coarse annotation together with the SSS problem. Specifically, the WSL-based ROI refinement method can effectively refine manually annotated ground truth ROIs, which is beneficial to the training of detection and classification models. Extensive experiments indicate that the proposed TSDDNet outperforms all compared algorithms, indicating its potential applications.