Abstract
Separating overlapped nuclei is a significant challenge in histopathology image analysis. Recently published approaches have achieved promising overall performance on nuclei segmentation; however, their performance on separating overlapped nuclei is limited. To address this issue, we propose a novel multitask learning network with a bending loss regularizer to separate overlapped nuclei accurately. The newly proposed multitask learning architecture enhances generalization by learning shared representation from the following three tasks: instance segmentation, nuclei distance map prediction, and overlapped nuclei distance map prediction. The proposed bending loss defines high penalties to concave contour points with large curvatures, and small penalties are applied to convex contour points with small curvatures. Minimizing the bending loss avoids generating contours that encompass multiple nuclei. In addition, two new quantitative metrics, the Aggregated Jaccard Index of overlapped nuclei (AJIO) and the accuracy of overlapped nuclei (ACCO), have been designed to evaluate overlapped nuclei segmentation. We validate the proposed approach on the CoNSeP and MoNuSegv1 data sets using the following seven quantitative metrics: Aggregate Jaccard Index, Dice, Segmentation Quality, Recognition Quality, Panoptic Quality, AJIO, and ACCO. Extensive experiments demonstrate that the proposed Bend-Net outperforms eight state-of-the-art approaches.