Deep Learning For Nuclei Segmentation and Classification of Histopathology Images

Abstract

To improve the process of diagnosis and treatment of cancer disease, automatic segmentation and classification of haematoxylin and eosin (H & E) stained histopathology images are important steps in digital pathology. The advent of new computation systems like GPU, fast digital scanners, and the availability of lots of data, Deep Learning (DL) techniques have shown superior perfor- mance in different applications of medical image analysis. The potential and applicability of deep learning models for the analysis of histopathology images have been demonstrated by many researchers. Due to variations in the appear- ance and complex clinical structure of histopathology slides, reported results still needed to be improved for accurate diagnosis of disease. An accurate and efficient classification algorithm that exactly resembles the clinical feature of cancer disease is still open-ended research. This thesis investigates a detailed methodology for the design and implementation of deep learning architectures which includes nuclei detection and segmentation, characterization of subtypes of cancer, and grading of histopathological tissues. In the first part of the thesis, the analysis of histopathology images by us- ing efficient segmentation algorithms is presented. In this study, an effective encoder-decoder architecture with a separable convolution pyramid pooling network (SCPP-Net) is designed and implemented for automatically segment- ing complex nuclei present in digital histopathology images. The SCPP unit focuses on two aspects: first, it increases the receptive field by varying four different dilation rates, keeping the kernel size fixed, and second, it reduces the trainable parameter by using depth-wise separable convolution. For multi- organ histopathology analysis, a new deep learning framework is proposed, that consists of a high-resolution encoder path, an atrous spatial pyramid pooling (ASPP) bottleneck module, and a powerful decoder. The proposed network is wide and deep that effectively leverages the strength of residual learning as well as encoder-decoder architecture. The problem of the vanished bound- ary of detected nuclei is addressed by proposing an efficient loss function that better trains the proposed deep structured residual encoder-decoder network (DSREDN) and reduces the false prediction. The obtained score of nuclei segmentation indicated that the proposed architectures achieved a considerable margin over state-of-the-art deep learning models on three different publicly iavailable histopathology image datasets. Next, in the thesis, a novel dataset and an efficient deep-learning framework for the classification of subtypes of renal cell carcinoma (RCC) from kidney histopathological images are proposed. The proposed RenalNet is intended to capture cross-channel and inter-spatial features at three different scales paral- lelly and held them together. The proposed model contains a new convolu- tional neural network (CNN) block called multiple channel residual transfor- mation (MCRT), to focus on the most relevant morphological features of RCC by fusing the information of multiple paths. Further, to improve the network’s representation power, a novel block called group convolution deep localization (GCDL) is introduced that effectively integrates three different feature descrip- tors. A new benchmark dataset for the classification of subtypes of RCC from kidney histopathology images is also introduced as a part of this study. The re- sults of the proposed model are compared with the existing DL models trained from scratch as well as networks leveraged by transfer learning of pre-trained weights. During the experimentation, the proposed network achieved an accu- racy 91.67%, and F1-Score 91.65% on the proposed kidney dataset that is the highest among all competitive models. The experimental results show that the proposed RenalNet architecture is best in terms of training and prediction time, classification accuracy, F1 score, and computational complexity. A pathologist report affirmed that the stage and grade of diagnosis is the most important prognostic factor. In these cases, continuous staging and grading evaluation is extremely important for the clinical management of patients. This study proposed a robust and computationally efficient fully automated Renal Cell Carcinoma Grading Network (RCCGNet) from kidney histopathology im- ages. The proposed shared channel residual (SCR) block shares the information between two different layers and operates the shared data separately by provid- ing beneficial supplements to each other. As a part of this study, a new dataset also has been introduced for the grading of RCC with five different grades. The simulation results include deep learning models trained from scratch as well as transfer learning techniques using pre-trained weights of the ImageNet. The performance of the proposed RCCGNet is evaluated by the most preferred quality metrics and achieved 90.14% of accuracy, and 89.06% F1-score on the introduced kidney dataset. iiAnother proposed architecture is called Robust CNN (RoCNN) for grading (Normal, Grade-1, Grade-2, Grade-3, and Grade-4) and classification (Normal, KIRC, KIRP, KICH) in kidney cancer tissue. To demonstrate that the proposed model is generalized and independent of the dataset, it has experimented on two well-known datasets, the KMC kidney dataset of five different grades and the TCGA dataset of four classes. The RoCNN is capable of learning features at varying convolutional filter sizes because of the inception modules employed in it. Squeeze and Extract (SE) blocks are used to remove unnecessary contri- butions from noisy channels and improve model accuracy. Regarding the com- putational complexities the proposed RoCNN is extremely efficient compared to the reference models. Due to a substantial reduction in the computational complexities, incorporation of the proposed method into FPGA board process- ing for next-generation histopathological image analysis is a significant step in the right direction. Compared to the best-performing state-of-the-art model the accuracy of RoCNN shows significant improvement of about 4.22% and 3.01% for two different datasets. All proposed deep learning algorithms evolved to be the most promising, stable, and computationally efficient for the analysis of histopathological images.