Atrous Spatial Pyramid Pooling Research Articles

A semantic segmentation method for remote sensing images based on multiple contextual feature extraction

SummarySemantic segmentation of remote sensing images plays a significant role in many applications such as urban planning and ecological protection, but its semantic segmentation suffers from large intra‐category variation and large differences in the scale of objects, so it is prone to misclassification. To cope with this issue, an embedded channel's categorical attention module (ECCA) is proposed to extract contextual information from the perspective of categories, and a channel attention module is embedded in it to achieve multiple contextual information extraction. Combined with the remote sensing atrous spatial pyramid pooling module (RSASPP), which is composed of atrous convolution with different expansion rates, feature fusion of objects at different scales is achieved. The refinement module (RM) is added for boundary refinement to achieve finer segmentation. Experiments are conducted on the WHDLD dataset to prove the effectiveness of the method.

Multi-Branch Adaptive Hard Region Mining Network for Urban Scene Parsing of High-Resolution Remote-Sensing Images

Scene parsing of high-resolution remote-sensing images (HRRSIs) refers to parsing different semantic regions from the images, which is an important fundamental task in image understanding. However, due to the inherent complexity of urban scenes, HRRSIs contain numerous object classes. These objects present large-scale variation and irregular morphological structures. Furthermore, their spatial distribution is uneven and contains substantial spatial details. All these features make it difficult to parse urban scenes accurately. To deal with these dilemmas, in this paper, we propose a multi-branch adaptive hard region mining network (MBANet) for urban scene parsing of HRRSIs. MBANet consists of three branches, namely, a multi-scale semantic branch, an adaptive hard region mining (AHRM) branch, and an edge branch. First, the multi-scale semantic branch is constructed based on a feature pyramid network (FPN). To reduce the memory footprint, ResNet50 is chosen as the backbone, which, combined with the atrous spatial pyramid pooling module, can extract rich multi-scale contextual information effectively, thereby enhancing object representation at various scales. Second, an AHRM branch is proposed to enhance feature representation of hard regions with a complex distribution, which would be difficult to parse otherwise. Third, the edge-extraction branch is introduced to supervise boundary perception training so that the contours of objects can be better captured. In our experiments, the three branches complemented each other in feature extraction and demonstrated state-of-the-art performance for urban scene parsing of HRRSIs. We also performed ablation studies on two HRRSI datasets from ISPRS and compared them with other methods.

MFPA-Net: An efficient deep learning network for automatic ground fissures extraction in UAV images of the coal mining area

Ground fissures caused by high-intensity underground coal mining activities will damage the ecological environment and endanger mine safety. The complicated surface conditions in coal mining areas make fissure detection labor-intensive work, and manual fissure extraction is inefficient, restricting the surface monitoring work. In this study, an efficient Deep Learning (DL) network named MFPA-Net (Multi-scale Feature Pyramids and Attention Network) is proposed to extract ground fissures in Unmanned Aerial Vehicle (UAV) images from the coal mining area automatically and accurately. In MFPA-Net, the Dilated Residual Networks (DRN) are used to extract diverse context information, the Dual Attention Mechanism (DAM) is introduced to integrate the dependence of pixels' spatial location and feature channels to generate high-level features, the Atrous Spatial Pyramid Pooling (ASPP) is utilized for mining multi-scale context information from high-level features, and the Multi-scale Feature Pyramid Network (MFPN) is designed to combine high-level and low-level features. Moreover, the Focal Tversky Loss Function is adopted to handle the unbalanced samples. To promote DL technologies application in the fissure monitoring of mining areas, the GFCMA (Ground Fissures of the Coal Mining Area) dataset is constructed. Experiments on GFCMA show that MFPA-Net can achieve high Precision (69.4%), Recall (70.7%), F1-Score (70.0%), and Mean Intersection over Union (MIoU) (75.1%) simultaneously, which significantly outperform traditional image processing methods and recently DL networks. Experiments on public pavement datasets Crack500 and DeepCrack prove MFPA-Net's high reliability and widespread applicability. The performance of the trained MFPA-Net on real large-scale scenarios demonstrates its practical value, strong robustness, and high efficiency. This study provides a solution for rapid monitoring of ground fissures under complicated surface conditions, which can serve the safe production and ecological restoration high-efficiently in mining areas.

Frequency Spectrum Intensity Attention Network for Building Detection from High-Resolution Imagery

Computational intelligence techniques have been widely used for automatic building detection from high-resolution remote sensing imagery and especially the methods based on neural networks. However, existing methods do not pay attention to the value of high-frequency and low-frequency information in the frequency domain for feature extraction of buildings in remote sensing images. To overcome these limitations, this paper proposes a frequency spectrum intensity attention network (FSIANet) with an encoder–decoder structure for automatic building detection. The proposed FSIANet mainly involves two innovations. One, a novel and plug-and-play frequency spectrum intensity attention (FSIA) mechanism is devised to enhance feature representation by evaluating the informative abundance of the feature maps. The FSIA is deployed after each convolutional block in the proposed FSIANet. Two, an atrous frequency spectrum attention pyramid (AFSAP) is constructed by introducing FSIA in widely used atrous spatial pyramid pooling. The AFSAP is able to select the features with high response to building semantic features at each scale and weaken the features with low response, thus enhancing the feature representation of buildings. The proposed FSIANet is evaluated on two large public datasets (East Asia and Inria Aerial Image Dataset), which demonstrates that the proposed method can achieve the state-of-the-art performance in terms of F1-score and intersection-over-union.

Improved YOLOX Foreign Object Detection Algorithm for Transmission Lines

It is quite simple for foreign objects to attach themselves to transmission line corridors because of the wide variety of laying and the complex, changing environment. If these foreign objects are not found and removed in a timely manner, they can have a significant impact on the transmission lines’ ability to operate safely. Due to the problem of poor accuracy of foreign object identification in transmission line image inspection, we provide an improved YOLOX technique for detection of foreign objects in transmission lines. The method improves the YOLOX target detection network by first using Atrous Spatial Pyramid Pooling to increase sensitivity to foreign objects of different scales, then by embedding Convolutional Block Attention Module to increase model recognition accuracy, and finally by using GIoU loss to further optimize. The testing findings show that the enhanced YOLOX network has a mAP improvement of around 4.24% over the baseline YOLOX network. The target detection SSD, Faster R-CNN, YOLOv5, and YOLOV7 networks have improved less than this. The effectiveness and superiority of the algorithm are proven.

Multi-scale feature similarity-based weakly supervised lymphoma segmentation in PET/CT images

Accurate lymphoma segmentation in PET/CT images is important for evaluating Diffuse Large B-Cell Lymphoma (DLBCL) prognosis. As systemic multiple lymphomas, DLBCL lesions vary in number and size for different patients, which makes DLBCL labeling labor-intensive and time-consuming. To reduce the reliance on accurately labeled datasets, a weakly supervised deep learning method based on multi-scale feature similarity is proposed for automatic lymphoma segmentation. Weak labeling was performed by randomly dawning a small and salient lymphoma volume for the patient without accurate labels. A 3D V-Net is used as the backbone of the segmentation network and image features extracted in different convolutional layers are fused with the Atrous Spatial Pyramid Pooling (ASPP) module to generate multi-scale feature representations of input images. By imposing multi-scale feature consistency constraints on the predicted tumor regions as well as the labeled tumor regions, weakly labeled data can also be effectively used for network training. The cosine similarity, which has strong generalization, is exploited here to measure feature distances. The proposed method is evaluated with a PET/CT dataset of 147 lymphoma patients. Experimental results show that when using data, half of which have accurate labels and the other half have weak labels, the proposed method performed similarly to a fully supervised segmentation network and achieved an average Dice Similarity Coefficient (DSC) of 71.47%. The proposed method is able to reduce the requirement for expert annotations in deep learning-based lymphoma segmentation.

Multitask Image Splicing Tampering Detection Based on Attention Mechanism

In today’s modern communication society, the authenticity of digital media has never been of such importance as it is now. In this aspect, the reliability of digital images is of paramount importance because images can be easily manipulated by means of sophisticated software, such as Photoshop. Splicing tampering is a commonly used photographic manipulation for modifying images. Detecting splicing tampering remains a challenging task in the area of image forensics. A new multitask model based on attention mechanism, densely connected network, Atrous Spatial Pyramid Pooling (ASPP) and U-Net for locating splicing tampering in an image, AttDAU-Net, was proposed. The proposed AttDAU-Net is basically a U-Net that incorporates the spatial rich model filtering, an attention mechanism, an ASPP module and a multitask learning framework, in order to capture more multi-scale information while enlarging the receptive field and improving the detection precision of image splicing tampering. The experimental results on the datasets of CASIA1 and CASIA2 showed promising performance metrics for the proposed model (F1-scores of 0.7736 and 0.6937, respectively), which were better than other state-of-the-art methods for comparison, demonstrating the feasibility and effectiveness of the proposed AttDAU-Net in locating image splicing tampering.

A Super-resolution Guided Network for Improving Automated Thyroid Nodule Segmentation

Background and Objective: A thyroid nodule is an abnormal lump that grows in the thyroid gland, which is the early symptom of thyroid cancer. In order to diagnose and treat thyroid cancer at the earliest stage, it is desired to characterize the nodule accurately. Ultrasound thyroid nodules segmentation is a challenging task due to the speckle noise, intensity heterogeneity, low contrast and low resolution. In this paper, we propose a novel framework to improve the accuracy of thyroid nodules segmentation. Methods: Different from previous work, a super-resolution reconstruction network is firstly constructed to upscale the resolution of the input ultrasound image. After that, our proposed N-shape network is utilized to perform the segmentation task. The guidance of super-resolution reconstruction network can make the high-frequency information of the input thyroid ultrasound image richer and more comprehensive than the original image. Our N-shape network consists of several atrous spatial pyramid pooling blocks, a multi-scale input layer, a U-shape convolutional network with attention blocks and a proposed parallel atrous convolution(PAC) module. These modules are conducive to capture context information at multiple scales so that semantic features can be fully utilized for lesion segmentation. Especially, our proposed PAC module is beneficial to further improve the segmentation by extracting high-level semantic features from different receptive fields. We use the UTNI-2021 dataset for model training, validating and testing. Results: The experimental results show that our proposed method achieve a Dice value of 91.9%, a mIoU value of 87.0%, a Precision value of 88.0%, a Recall value 83.7% and a F1-score value of 84.3%, which outperforms most state-of-the-art methods. Conclusions: Our method achieves the best performance on the UTNI-2021 dataset and provides a new way of ultrasound image segmentation. We believe that our method can provide doctors with reliable auxiliary diagnosis information in clinical practice.

Deep attentive convolutional neural network for automatic grading of imbalanced diabetic retinopathy in retinal fundus images.

Automated fine-grained diabetic retinopathy (DR) grading was of great significance for assisting ophthalmologists in monitoring DR and designing tailored treatments for patients. Nevertheless, it is a challenging task as a result of high intra-class variations, high inter-class similarities, small lesions, and imbalanced data distributions. The pivotal factor for the success in fine-grained DR grading is to discern more subtle associated lesion features, such as microaneurysms (MA), Hemorrhages (HM), soft exudates (SE), and hard exudates (HE). In this paper, we constructed a simple yet effective deep attentive convolutional neural network (DACNN) for DR grading and lesion discovery with only image-wise supervision. Designed as a top-down architecture, our model incorporated stochastic atrous spatial pyramid pooling (sASPP), global attention mechanism (GAM), category attention mechanism (CAM), and learnable connected module (LCM) to better extract lesion-related features and maximize the DR grading performance. To be concrete, we devised sASPP combining randomness with atrous spatial pyramid pooling (ASPP) to accommodate the various scales of the lesions and struggle against the co-adaptation of multiple atrous convolutions. Then, GAM was introduced to extract class-agnostic global attention feature details, whilst CAM was explored for seeking class-specific distinctive region-level lesion feature information and regarding each DR severity grade in an equal way, which tackled the problem of imbalance DR data distributions. Further, the LCM was designed to automatically and adaptively search the optimal connections among layers for better extracting detailed small lesion feature representations. The proposed approach obtained high accuracy of 88.0% and kappa score of 88.6% for multi-class DR grading task on the EyePACS dataset, respectively, while 98.5% AUC, 93.8% accuracy, 87.9% kappa, 90.7% recall, 94.6% precision, and 92.6% F1-score for referral and non-referral classification on the Messidor dataset. Extensive experimental results on three challenging benchmarks demonstrated that the proposed approach achieved competitive performance in DR grading and lesion discovery using retinal fundus images compared with existing cutting-edge methods, and had good generalization capacity for unseen DR datasets. These promising results highlighted its potential as an efficient and reliable tool to assist ophthalmologists in large-scale DR screening.

Glacier extraction based on high-spatial-resolution remote-sensing images using a deep-learning approach with attention mechanism

Abstract. The accurate and rapid extraction of glacier boundaries plays an important role in the study of glacier inventory, glacier change and glacier movement. With the successive launches of high-resolution remote-sensing satellites and the increasing abundance of available remote-sensing data, great opportunities and challenges now exist. In this study, we improved the DeepLab V3+ as Attention DeepLab V3+ and designed a complete solution based on the improved network to automatically extract glacier outlines from Gaofen-6 panchromatic and multispectral (PMS) images with a spatial resolution of 2 m. In the solution, test-time augmentation (TTA) was adopted to increase model robustness, and the convolutional block attention module (CBAM) was added into the atrous spatial pyramid pooling (ASPP) structure in DeepLab V3+ to enhance the weight of the target pixels and reduce the impact of superfluous features. The results show that the improved model effectively increases the robustness of the model, enhances the weight of target image elements and reduces the influence of non-target elements. Compared with deep-learning models, such as full convolutional network (FCN), U-Net and DeepLab V3+, the improved model performs better in the test dataset. Moreover, our method achieves superior performance for glacier boundary extraction in parts of the Tanggula Mountains, the Kunlun Mountains and the Qilian Mountains based on Gaofen-6 PMS images. It could distinguish glaciers from terminal moraine lakes, thin snow and clouds, thus demonstrating excellent performance and great potential for rapid and precise extraction of glacier boundaries.

Encoder–Decoder Network with Depthwise Atrous Spatial Pyramid Pooling for Automatic Brain Tumor Segmentation

Encoder–Decoder Network with Depthwise Atrous Spatial Pyramid Pooling for Automatic Brain Tumor Segmentation

Tea Sprout Picking Point Identification Based on Improved DeepLabV3+

Tea sprout segmentation and picking point localization via machine vision are the core technologies of automatic tea picking. This study proposes a method of tea segmentation and picking point location based on a lightweight convolutional neural network named MC-DM (Multi-Class DeepLabV3+ MobileNetV2 (Mobile Networks Vision 2)) to solve the problem of tea shoot picking point in a natural environment. In the MC-DM architecture, an optimized MobileNetV2 is used to reduce the number of parameters and calculations. Then, the densely connected atrous spatial pyramid pooling module is introduced into the MC-DM to obtain denser pixel sampling and a larger receptive field. Finally, an image dataset of high-quality tea sprout picking points is established to train and test the MC-DM network. Experimental results show that the MIoU of MC-DM reached 91.85%, which is improved by 8.35% compared with those of several state-of-the-art methods. The optimal improvements of model parameters and detection speed were 89.19% and 16.05 f/s, respectively. After the segmentation results of the MC-DM were applied to the picking point identification, the accuracy of picking point identification reached 82.52%, 90.07%, and 84.78% for single bud, one bud with one leaf, and one bud with two leaves, respectively. This research provides a theoretical reference for fast segmentation and visual localization of automatically picked tea sprouts.

Multi-class motion-based semantic segmentation for ureteroscopy and laser lithotripsy.

Ureteroscopy with laser lithotripsy has evolved as the most commonly used technique for the treatment of kidney stones. Automated segmentation of kidney stones and the laser fiber is an essential initial step to performing any automated quantitative analysis, particularly stone-size estimation, that can be used by the surgeon to decide if the stone requires further fragmentation. However, factors such as turbid fluid inside the cavity, specularities, motion blur due to kidney movements and camera motion, bleeding, and stone debris impact the quality of vision within the kidney, leading to extended operative times. To the best of our knowledge, this is the first attempt made towards multi-class segmentation in ureteroscopy and laser lithotripsy data. We propose an end-to-end convolution neural network (CNN) based learning framework for the segmentation of stones and laser fiber. The proposed approach utilizes two sub-networks: (I) HybResUNet, a hybrid version of residual U-Net, that uses residual connections in the encoder path of the U-Net to improve semantic predictions, and (II) a DVFNet that generates deformation vector field (DVF) predictions by leveraging motion differences between the adjacent video frames which is then used to prune the prediction maps. We also present ablation studies that combine different dilated convolutions, recurrent and residual connections, atrous spatial pyramid pooling, and attention gate models. Further, we propose a compound loss function that significantly boosts the segmentation performance in our data. We have also provided an ablation study to determine the optimal data augmentation strategy for our dataset. Our qualitative and quantitative results illustrate that our proposed method outperforms state-of-the-art methods such as UNet and DeepLabv3+ showing a DSC improvement of 4.15% and 13.34%, respectively, in our in vivo test dataset. We further show that our proposed model outperforms state-of-the-art methods on an unseen out-of-sample clinical dataset with a DSC improvement of 9.61%, 11%, and 5.24% over UNet, HybResUNet, and DeepLabv3+, respectively in the case of the stone class and an improvement of 31.79%, 22.15%, and 10.42% over UNet, HybResUNet, and DeepLabv3+, respectively, in case of the laser class.

Oestrus detection in dairy cows by using atrous spatial pyramid and attention mechanism

The mounting behaviour of dairy cows is driven by oestrus and is also related to their welfare. To improve the accuracy and speed of the detection of mounting of dairy cows in dense scenes, a detection model for the mounting behaviour of dairy cows was developed in this study based on the feature enhancement module. Firstly, an improved attention module (C3GC-3) based on global context information and convolution was proposed to capture long-distance dependence. Then, a feature enhancement module based on atrous spatial pyramid pooling and the C3GC-3 was designed to realise the multiscale fusion of high-level semantic information and improve the feature extraction ability of the model for dense dairy cows images. Finally, the K-means clustering algorithm was used to obtain new anchors for the mounting behaviour of dairy cows, and CIoU was used to optimise the loss function. A camera was installed on a farm containing 200 dairy cows for data collection. The mosaic method was used to expand the 2668 images of the mounting behaviour of dairy cows to train the model, and the remaining 675 images were used to test the model. The experimental results showed that the proposed model had a high inference speed of 61 fps, and the detection time for each image was 16.3 ms, which met the real-time performance requirements for detecting mounting in dairy cows. The mean average precision of the model was 94.3%, which was 5.9% higher than that of YOLOv5l. The proposed model showed promising results regarding the mounting of dairy cows and might be used in any weather. • A mounting behaviour dataset of group-raised dairy cows in dense scenes was proposed.. • ASPP can improve the accuracy of multi-scale object detection. • Attention mechanism can improve the feature extraction ability of the model. • Mean average precision model was 94.3%, which is 5.9% higher than that of YOLOv5l.

A novel MCF-Net: Multi-level context fusion network for 2D medical image segmentation

Medical image segmentation is a crucial step in the clinical applications for diagnosis and analysis of some diseases. U-Net-based convolution neural networks have achieved impressive performance in medical image segmentation tasks. However, the multi-level contextual information integration capability and the feature extraction ability are often insufficient. In this paper, we present a novel multi-level context fusion network (MCF-Net) to improve the performance of U-Net on various segmentation tasks by designing three modules, hybrid attention-based residual atrous convolution (HARA) module, multi-scale feature memory (MSFM) module, and multi-receptive field fusion (MRFF) module, to fuse multi-scale contextual information. HARA module was proposed to effectively extract multi-receptive field features by combing atrous spatial pyramid pooling and attention mechanism. We further design the MSFM and MRFF modules to fuse features of different levels and effectively extract contextual information. The proposed MCF-Net was evaluated on the ISIC 2018, DRIVE, BUSI, and Kvasir-SEG datasets, which have challenging images of many sizes and widely varying anatomy. The experimental results show that MCF-Net is very competitive with other U-Net models, and it offers tremendous potential as a general-purpose deep learning model for 2D medical image segmentation.

Environment Understanding Algorithm for Substation Inspection Robot Based on Improved DeepLab V3.

Compared with traditional manual inspection, inspection robots can not only meet the all-weather, real-time, and accurate inspection needs of substation inspection, they also reduce the work intensity of operation and maintenance personnel and decrease the probability of safety accidents. For the urgent demand of substation inspection robot intelligence enhancement, an environment understanding algorithm is proposed in this paper, which is an improved DeepLab V3+ neural network. The improved neural network replaces the original dilate rate combination in the ASPP (atrous spatial pyramid pooling) module with a new dilate rate combination with better segmentation accuracy of object edges and adds a CBAM (convolutional block attention module) in the two up-samplings, respectively. In order to be transplanted to the embedded platform with limited computing resources, the improved neural network is compressed. Multiple sets of comparative experiments on the standard dataset PASCAL VOC 2012 and the substation dataset have been made. Experimental results show that, compared with the DeepLab V3+, the improved DeepLab V3+ has a mean intersection-over-union (mIoU) of eight categories of 57.65% on the substation dataset, with an improvement of 6.39%, and the model size of 13.9 M, with a decrease of 147.1 M.

MBTFCN: A novel modular fully convolutional network for MRI brain tumor multi-classification

Brain tumors represent one of the most challenging tumors that affect the human body due to the nonlinear characteristics of their morphological and textural appearance. Automated brain tumor diagnosis systems based on magnetic resonance images (MRI) help surgeons select suitable clinical practices for the patient. Therefore, increasing the performance of such systems plays a vital role in saving human life. A new modular deep fully convolutional neural network is designed to address this problem. The proposed network consists of four modules namely, Feature Extraction (FE), Residual Strip Pooling Attention (RSPA), Atrous Spatial Pyramid Pooling (ASPP), and classification module. First, discriminative brain tumor features using multiple residual convolutional blocks are extracted by the FE module, and then prominent tumor regions relevant to brain tumor classification are strengthened by the RSPA module. Multi-scale features which carry informative context information are captured by the ASPP module. Finally, the classification module is adopted using convolutional layers with adjusted stride values to classify the extracted multiscale features. The combination of these modules helps to extract both local and contextual information appropriate for brain tumor classification. Four public benchmark datasets containing 9581 brain magnetic resonance images are utilized. The datasets contain different classification tasks, number of samples, image sizes, contrast, and planes. The experimental results show that the proposed method surpasses the performance of other state-of-the-art methods. The proposed method encourages the diagnosis of medical imaging and can solve the problem of large intra-class variations and small-size datasets in medical image classification.

Multi-scale dense and attention mechanism for image semantic segmentation based on improved DeepLabv3+

Semantic scene segmentation has become an important application in computer vision and is an essential part of intelligent transportation systems for complete scene understanding of the surrounding environment. Several methods based on convolutional neural networks have emerged, but they have some problems, including small-scale target loss, inaccurate detailed region segmentation, and boundary category confusion. Using shallow features, we exploit the capabilities of global context information according to the theory of pyramids. A weighted pyramid feature fusion module is constructed to fuse the feature maps of different scales generated by the backbone network, and the proportion of feature fusion is dynamically updated by trainable parameters. After that, a self-attention mechanism is introduced to discover information about spatial channel interdependencies. Finally, the atrous spatial pyramid pooling module of the DeepLabv3+ network is improved by connecting the atrous convolution with different dilation rates at the receptive field. The experimental results show 4.1% mean pixel accuracy and 3.92% mean intersection over union improvements in the proposed method compared with the DeepLabv3+, and the result of semantic segmentation is more accurate.

AMM-FuseNet: Attention-Based Multi-Modal Image Fusion Network for Land Cover Mapping

Land cover mapping provides spatial information on the physical properties of the Earth’s surface for various classes of wetlands, artificial surface and constructions, vineyards, water bodies, etc. Having reliable information on land cover is crucial to developing solutions to a variety of environmental problems, such as the destruction of important wetlands/forests, and loss of fish and wildlife habitats. This has made land cover mapping become one of the most widespread applications in remote sensing computational imaging. However, due to the differences between modalities in terms of resolutions, content, and sensors, integrating complementary information that multi-modal remote sensing imagery exhibits into a robust and accurate system still remains challenging, and classical segmentation approaches generally do not give satisfactory results for land cover mapping. In this paper, we propose a novel dynamic deep network architecture, AMM-FuseNet that promotes the use of multi-modal remote sensing images for the purpose of land cover mapping. The proposed network exploits the hybrid approach of the channel attention mechanism and densely connected atrous spatial pyramid pooling (DenseASPP). In the experimental analysis, in order to verify the validity of the proposed method, we test AMM-FuseNet with three datasets whilst comparing it to the six state-of-the-art models of DeepLabV3+, PSPNet, UNet, SegNet, DenseASPP, and DANet. In addition, we demonstrate the capability of AMM-FuseNet under minimal training supervision (reduced number of training samples) compared to the state of the art, achieving less accuracy loss, even for the case with 1/20 of the training samples.

Tongue image segmentation and tongue color classification based on deep learning

ObjectiveTo propose two novel methods based on deep learning for computer-aided tongue diagnosis, including tongue image segmentation and tongue color classification, improving their diagnostic accuracy. MethodsLabelMe was used to label the tongue mask and Snake model to optimize the labeling results. A new dataset was constructed for tongue image segmentation. Tongue color was marked to build a classified dataset for network training. In this research, the Inception + Atrous Spatial Pyramid Pooling (ASPP) + UNet (IAUNet) method was proposed for tongue image segmentation, based on the existing UNet, Inception, and atrous convolution. Moreover, the Tongue Color Classification Net (TCCNet) was constructed with reference to ResNet, Inception, and Triple-Loss. Several important measurement indexes were selected to evaluate and compare the effects of the novel and existing methods for tongue segmentation and tongue color classification. IAUNet was compared with existing mainstream methods such as UNet and DeepLabV3+ for tongue segmentation. TCCNet for tongue color classification was compared with VGG16 and GoogLeNet. ResultsIAUNet can accurately segment the tongue from original images. The results showed that the Mean Intersection over Union (MIoU) of IAUNet reached 96.30%, and its Mean Pixel Accuracy (MPA), mean Average Precision (mAP), F1-Score, G-Score, and Area Under Curve (AUC) reached 97.86%, 99.18%, 96.71%, 96.82%, and 99.71%, respectively, suggesting IAUNet produced better segmentation than other methods, with fewer parameters. Triplet-Loss was applied in the proposed TCCNet to separate different embedded colors. The experiment yielded ideal results, with F1-Score and mAP of the TCCNet reached 88.86% and 93.49%, respectively. ConclusionIAUNet based on deep learning for tongue segmentation is better than traditional ones. IAUNet can not only produce ideal tongue segmentation, but have better effects than those of PSPNet, SegNet, UNet, and DeepLabV3+, the traditional networks. As for tongue color classification, the proposed network, TCCNet, had better F1-Score and mAP values as compared with other neural networks such as VGG16 and GoogLeNet.