"External validation of deep learning models for mouse thorax autocontouring"

Justin Malimban, Danny Lathouwers, Haibin Qian, Frank Verhaegen, Julia Wiedemann, Sytze Brandenburg and Marius Staring

Abstract

Introduction: Organ contouring is one of the most laborious and time-consuming stages in the preclinical irradiation workflow. Since deep learning algorithms have shown excellent performance on human organ segmentation, their application for animals have also been recently explored. However, previously developed deep learning-based animal autocontouring models were mostly trained on one type of dataset, and their predictive performance was also evaluated on the same distribution as the training data. In a preclinical facility wherein studies involving various strains of animals and image acquisition protocols are performed, it is important to demonstrate the robustness of these tools across different populations and settings. Therefore, in this work, we externally validated two deep learning pipelines for mouse thorax segmentation to assess their usability and portability when implemented on a larger scale.

Materials & Methods: We trained the 2D and 3D models of nnU-Net (i.e., one of the best performing algorithms for clinical segmentation) and compared them with the state-of-the-art AIMOS pipeline for segmentation of the mouse thorax. We allotted 105 native micro-CT scans of mice for the training and initially performed internal validation using 35 native micro-CT scans not included in the model development to determine the best nnU-Net model. Then, the proposed nnU-Net model and AIMOS were externally validated using 35 contrast-enhanced micro-CTs, which comprise of scans with a different mouse strain and imaging parameters than the training data. The predictive performance was evaluated in terms of the Dice score (DSC), mean surface distance (MSD), and 95% Hausdorff distance (95% HD).

Results: When tested against native micro-CTs, all models of nnU-Net (3d_fullres, 3d_cascade, 3d_lowres, 2d) and AIMOS generated accurate contours, achieving average DSC greater than 0.94, 0.90, 0.97 and 0.95 for the heart, spinal cord, right and left lungs, respectively. The average MSD was less than the in-plane voxel size of 0.14 mm while the average 95% HD was below 0.60 mm except for the right lung results of nnU-Net 2d. Among the nnU-Net models, 3d_fullres was considered the superior model, producing the most accurate contours at a reasonable speed. The nnU-Net 3d_fullres model and AIMOS were then evaluated against the external dataset. The nnU-Net 3d_fullres model achieved average DSC of 0.92, 0.85, 0.96 and 0.95 whereas AIMOS showed inferior results: 0.83, 0.82, 0.87 and 0.77. Consistent for all organs, AIMOS recorded unacceptably large 95% HD > 1 mm and produced incomplete contours. Moreover, AIMOS failed to distinguish the right lung from left lung. These entail that AIMOS requires more labor-intensive corrections than nnU-Net 3d_fullres for data on which the model was not trained on.

Conclusion: We have shown that the nnU-Net 3d_fullres model is more robust and generalizable than the current best performing algorithm for mouse segmentation (AIMOS). Our findings also demonstrate the importance of thoroughly evaluating the performance of autocontouring tools before implementation in routine preclinical practice.

BibTeX entry