Small Object Segmentation Research Articles

CaraNet: context axial reverse attention network for segmentation of small medical objects.

Segmenting medical images accurately and reliably is important for disease diagnosis and treatment. It is a challenging task because of the wide variety of objects' sizes, shapes, and scanning modalities. Recently, many convolutional neural networks have been designed for segmentation tasks and have achieved great success. Few studies, however, have fully considered the sizes of objects; thus, most demonstrate poor performance for small object segmentation. This can have a significant impact on the early detection of diseases. We propose a context axial reverse attention network (CaraNet) to improve the segmentation performance on small objects compared with several recent state-of-the-art models. CaraNet applies axial reserve attention and channel-wise feature pyramid modules to dig the feature information of small medical objects. We evaluate our model by six different measurement metrics. We test our CaraNet on segmentation datasets for brain tumor (BraTS 2018) and polyp (Kvasir-SEG, CVC-ColonDB, CVC-ClinicDB, CVC-300, and ETIS-LaribPolypDB). Our CaraNet achieves the top-rank mean Dice segmentation accuracy, and results show a distinct advantage of CaraNet in the segmentation of small medical objects. We proposed CaraNet to segment small medical objects and outperform state-of-the-art methods.

Semantics recalibration and detail enhancement network for real‐time semantic segmentation

AbstractReal‐time semantic segmentation is a crucial technology in automatic driving scenarios, which needs to meet both high precision and real‐time. The authors observe that learning complex correlations between object categories is vital in the real‐time semantic segmentation task. Moreover, image spatial detail information plays an important role in small object segmentation and preserving edges and textures. A Semantics Recalibration and Detail Enhancement Network for real‐time semantic segmentation based on BiSeNet V2 is proposed. On the one hand, a lightweight Semantics Recalibration module is designed to effectively extract global semantic contextual information, which combines pyramid segmentation and adaptive recalibration operations to learn the correlations between object categories. On the other hand, a Detail Enhancement module takes the feature maps of the shallow layers in the semantics branch as input and refines the feature maps to highlight the detail information. Finally, quantitative and qualitative analyses on Cityscapes and CamVid datasets demonstrate the effectiveness and generalisation of the proposed method.

Adjacent Bi-Hierarchical Network for Scene Parsing of Remote Sensing Images

Driven by the rapid development and application of earth observation sensors, the scene parsing of remote sensing images (RSIs) has attracted extensive research attention in recent years. Restricted by the limited local receptive field of successive convolution layers, traditional models of scene parsing cannot effectively and interactively utilize the local-global information and digital surface model (DSM) of RSIs. Comparatively, accurate scene parsing faces more challenges because of unbalanced categories, small targets, and more complex scenes. To address these challenges, herein, we propose a novel adjacent bi-hierarchical network (ABHNet). Specifically, we introduce a DSM-enhanced (DSE) module to excavate characteristic DSM information from DSM images and enhance the red, green, and blue (RGB) features by exploiting informative cues between RGB and DSM modalities. In addition, an adjacent context exploration (ACE) module is proposed, which contains current and adjacent branches. The branches first exploit multiscale complementary characteristics of multilevel features and then integrate these features by applying adjacent exploration. Our model includes five ACE modules—three are deployed to activate detailed features and two obtain deep-guided features. The mutual collaboration of deep and detailed features is more beneficial to the segmentation of small objects. Extensive experiments on two remote sensing benchmark datasets (ISPRS Potsdam and Vaihingen) showed that the proposed ABHNet qualitatively and quantitatively outperformed other methods.

Event‐based motion segmentation of small objects in the wild

Event-based cameras are sensitive to brightness changes and can capture rich temporal information with very high temporal resolution, which has great potential for motion segmentation of moving objects. Under static background, events are only triggered by motion of objects, thereby moving objects can be easily segmented. However, in many real-world applications, events are also triggered by the motion of camera or background and submerge the ones corresponding to moving objects. In this letter, an event-based motion segmentation method to segment moving small objects in events obtained from the wild is proposed. First, motion estimation is performed to align the events triggered by the background. Then, candidate events corresponding to moving objects or moving backgrounds are detected. Finally, motion information is adopted to segment the events of moving small objects from the ones triggered by the background. In addition, the first dataset for event-based motion segmentation of small objects is developed. Experimental results demonstrate the effectiveness of this method and show that this method can achieve robust motion segmentation of small moving objects in the wild.

Segmentation of Small Objects in Satellite Imagery Using Dense U-Net in Massachusetts Buildings Dataset

Class imbalance is a serious problem that disrupts the process of semantic segmentation of satellite imagery in urban areas in Earth remote sensing. Due to the large objects dominating the segmentation process, small object are consequently limited, so solutions based on optimizing overall accuracy are often unsatisfactory. Due to the class imbalance of semantic segmentation in Earth remote sensing images in urban areas, we developed the concept of Down-Sampling Block (DownBlock) to obtain contextual information and Up-Sampling Block (UpBlock) to restore the original resolution. We proposed an end-to-end deep convolutional neural network (DenseU-Net) architecture for pixel-wise urban remote sensing image segmentation. this method to segmentation the small object in satellite imagery.The accuracy of the small object class in this study was further improved using our proposed method. This study used data from the Massachusetts Buildings dataset using Dense U-Net method and obtained an overall accuracy of 84.34%.

An Attention-Preserving Network-Based Method for Assisted Segmentation of Osteosarcoma MRI Images

Osteosarcoma is a malignant bone tumor that is extremely dangerous to human health. Not only does it require a large amount of work, it is also a complicated task to outline the lesion area in an image manually, using traditional methods. With the development of computer-aided diagnostic techniques, more and more researchers are focusing on automatic segmentation techniques for osteosarcoma analysis. However, existing methods ignore the size of osteosarcomas, making it difficult to identify and segment smaller tumors. This is very detrimental to the early diagnosis of osteosarcoma. Therefore, this paper proposes a Contextual Axial-Preserving Attention Network (CaPaN)-based MRI image-assisted segmentation method for osteosarcoma detection. Based on the use of Res2Net, a parallel decoder is added to aggregate high-level features which effectively combines the local and global features of osteosarcoma. In addition, channel feature pyramid (CFP) and axial attention (A-RA) mechanisms are used. A lightweight CFP can extract feature mapping and contextual information of different sizes. A-RA uses axial attention to distinguish tumor tissues by mining, which reduces computational costs and thus improves the generalization performance of the model. We conducted experiments using a real dataset provided by the Second Xiangya Affiliated Hospital and the results showed that our proposed method achieves better segmentation results than alternative models. In particular, our method shows significant advantages with respect to small target segmentation. Its precision is about 2% higher than the average values of other models. For the segmentation of small objects, the DSC value of CaPaN is 0.021 higher than that of the commonly used U-Net method.

A U-snake based deep learning network for right ventricle segmentation.

Ventricular segmentation is of great importance for the heart condition monitoring. However, manual segmentation is time-consuming, cumbersome, and subjective. Many segmentation methods perform poorly due to the complex structure and uncertain shape of the right ventricle, so we combine deep learning to achieve automatic segmentation. This paper proposed a method named U-Snake network which is based on the improvement of deep snake together with level set to segment the right ventricular in the MR images. U-snake aggregates the information of each receptive field which is learned by circular convolution of multiple dilation rates. At the same time, we also added dice loss functions and transferred the result of U-Snake to the level set so as to further enhance the effect of small object segmentation. Our method is tested on the test 1 and test 2 datasets in the right ventricular segmentation challenge (RVSC), which shows the effectiveness. The experiment showed that we have obtained good result in the RVSC. The highest segmentation accuracy on the right ventricular test set 2 reached a dice coefficient of 0.911, and the segmentation speed reached 5 fps. Our method, a new deep learning network named U-snake, has surpassed the previous excellent ventricular segmentation method based on mathematical theory and other classical deep learning methods, such as Residual U-net, Inception CNN, and Dilated CNN. However, it can only be used as an auxiliary tool instead of replacing the work of human beings.

Real-Time Tracking of Object Melting Based on Enhanced DeepLab v3+ Network.

In order to reveal the dissolution behavior of iron tailings in blast furnace slag, the main component of iron tailings, SiO2, was used for research. Aiming at the problem of information loss and inaccurate extraction of tracking molten SiO2 particles in high temperature, a method based on the improved DeepLab v3+ network was proposed to track, segment, and extract small object particles in real time. First, by improving the decoding layer of the DeepLab v3+ network, construct dense ASPP (atrous spatial pyramid pooling) modules with different dilation rates to optimize feature extraction, increase the shallow convolution of the backbone network, and merge it into the upper convolution decoding part to increase detailed capture. Secondly, integrate the lightweight network MobileNet v3 to reduce network parameters, further speed up image detection, and reduce the memory usage to achieve real-time image segmentation and adapt to low-level configuration hardware. Finally, improve the expression of the loss function for the binary classification model of small object in this paper, combining the advantages of the Dice Loss binary classification segmentation and the Focal Loss balance of positive and negative samples, solving the problem of unbalanced dataset caused by the small proportion of positive samples. Experimental results show that MIoU (mean intersection over union) of the proposed model for small object segmentation is 6% higher than that of the original model, the overall MIoU is increased by 3%, and the execution time and memory consumption are only half of the original model, which can be well applied to real-time tracking and segmentation of small particles.

Explored Normalized Cut With Random Walk Refining Term for Image Segmentation.

The Normalized Cut (NCut) model is a popular graph-based model for image segmentation. But it suffers from the excessive normalization problem and weakens the small object and twig segmentation. In this paper, we propose an Explored Normalized Cut (ENCut) model that establishes a balance graph model by adopting a meaningful-loop and a k-step random walk, which reduces the energy of small salient region, so as to enhance the small object segmentation. To improve the twig segmentation, our ENCut model is further enhanced by a new Random Walk Refining Term (RWRT) that adds local attention to our model with the help of an un-supervising random walk. Finally, a move-making based strategy is developed to efficiently solve the ENCut model with RWRT. Experiments on three standard datasets indicate that our model can achieve state-of-the-art results among the NCut-based segmentation models.

LFG-Net: Low-Level Feature Guided Network for Precise Ship Instance Segmentation in SAR Images

Ship instance segmentation of high-resolution SAR images is a valuable and challenging task due to the complex scattering and noise properties. In this article, we pioneered the construction of the low-level feature to discriminate the ships and complemented the super-resolution denoising techniques in the network modules, termed low-level feature guided network (LFG-Net), for precise ship instance segmentation in SAR images. LFG-Net consists of the low-level feature concerned pyramid (LFCP), the high-resolution interaction module (HR-FIM), and the compression recovery segmentation branch (CRSB). LFCP extends vanilla FPN with the P₁ layer and complements super-resolution techniques to capture the regional and texture information at the image level for small object segmentation. HR-FIM interacts the bounding box region of interest (RoI) feature and mask RoI feature at the instance level with high-resolution techniques to enhance the mask RoI feature. CRSB aims at recovering the high-resolution mask predictions to improve the ship segmentation performance. Comprehensive experiments on HRSID, PSeg-SSDD, and AirSARShip indicate that LFG-Net* achieves 11.7%, 6.3%, and 12.7% AP increments compared with the Mask R-CNN baseline, respectively. Besides, it receives 9.5%, 4.9%, and 7.3% AP increments compared with state-of-the-art method, respectively, which bridges the gap of instance segmentation precision in SAR images. In terms of the visualized instance segmentation results, LFG-Net* is capable of segmenting the complex scenes, e.g, the adjacent distributed ships and ships with strong reflection noise interference, in SAR images. Code is available at: https://github.com/Evarray/LFG-Net.

Holistic indoor scene understanding by context-supported instance segmentation

We propose a new method flow that utilizes pixel-level labeling information for instance-level object detection in indoor scenes from RGB-D data. Semantic labeling and instance segmentation are two different paradigms for indoor scene understanding that are usually accomplished separately and independently. We are interested in integrating the two tasks in a synergistic way in order to take advantage of their complementary nature for comprehensive understanding. Our work can capitalize on any deep learning networks used for semantic labeling by treating the intermediate layer as the category-wise local detection output, from which instance segmentation is optimized by jointly considering both the spatial fitness and the relational context encoded by three graphical models, namely, the vertical placement model (VPM), horizontal placement model (HPM) and non-placement model (NPM). VPM, HPM and NPM represent three common but distinct indoor object placement configurations: vertical, horizontal and hanging relationships, respectively. Experimental results on two standard RGB-D datasets show that our method can significantly improve small object segmentation with promising overall performance that is competitive with the state-of-the-art methods.

Point Cloud Semantic Segmentation Network Based on Multi-Scale Feature Fusion.

The semantic segmentation of small objects in point clouds is currently one of the most demanding tasks in photogrammetry and remote sensing applications. Multi-resolution feature extraction and fusion can significantly enhance the ability of object classification and segmentation, so it is widely used in the image field. For this motivation, we propose a point cloud semantic segmentation network based on multi-scale feature fusion (MSSCN) to aggregate the feature of a point cloud with different densities and improve the performance of semantic segmentation. In our method, random downsampling is first applied to obtain point clouds of different densities. A Spatial Aggregation Net (SAN) is then employed as the backbone network to extract local features from these point clouds, followed by concatenation of the extracted feature descriptors at different scales. Finally, a loss function is used to combine the different semantic information from point clouds of different densities for network optimization. Experiments were conducted on the S3DIS and ScanNet datasets, and our MSSCN achieved accuracies of 89.80% and 86.3%, respectively, on these datasets. Our method showed better performance than the recent methods PointNet, PointNet++, PointCNN, PointSIFT, and SAN.

AVILNet: A New Pliable Network with a Novel Metric for Small-Object Segmentation and Detection in Infrared Images

Infrared small-object segmentation (ISOS) has a persistent trade-off problem—that is, which came first, recall or precision? Constructing a fine balance between of them is, au fond, of vital importance to obtain the best performance in real applications, such as surveillance, tracking, and many fields related to infrared searching and tracking. F1-score may be a good evaluation metric for this problem. However, since the F1-score only depends upon a specific threshold value, it cannot reflect the user’s requirements according to the various application environment. Therefore, several metrics are commonly used together. Now we introduce F-area, a novel metric for a panoptic evaluation of average precision and F1-score. It can simultaneously consider the performance in terms of real application and the potential capability of a model. Furthermore, we propose a new network, called the Amorphous Variable Inter-located Network (AVILNet), which is of pliable structure based on GridNet, and it is also an ensemble network consisting of the main and its sub-network. Compared with the state-of-the-art ISOS methods, our model achieved an AP of 51.69%, F1-score of 63.03%, and F-area of 32.58% on the International Conference on Computer Vision 2019 ISOS Single dataset by using one generator. In addition, an AP of 53.6%, an F1-score of 60.99%, and F-area of 32.69% by using dual generators, with beating the existing best record (AP, 51.42%; F1-score, 57.04%; and F-area, 29.33%).

Investigation on the Effect of the Feature Extraction Backbone for Small Object Segmentation using Fully Convolutional Neural Network in Traffic Signs Application

Small objects are widely found in different applications such as traffic signs and to segment those objects make it difficult to extract features due to the small number of pixels. Previous research has been done to show how error prone the semantic segmentation networks to small objects in variance of application such as medical images and remote sensing and how it leads to class imbalance. However, small object segmentation seems to be tricky and making the network struggle. Recently there are small amount of research has been done in the effect of the feature extraction backbone to the small object datasets. In this paper we investigate the effect of different backbone feature extraction such as AlexNet, VGGNet, GoogleNet on an imbalanced small objects dataset after grouping them by shape and colour in the Fully Convolutional Networks (FCN). We measure the performance on PASCAL VOC and Malaysian Traffic Sign Dataset (MTSD) showing the pixel accuracy, mean accuracy per class, mean IoU and frequency weighted IoU for each backbone and FCN. The results show that VGGNet as a backbone with Cross Entropy (CE) combined with Dice Loss (DL) achieves the highest score in mean IoU for imbalanced dataset but not for balanced dataset. However, in the imbalanced dataset major classes have a higher probability to confuse with minor classes due to the class imbalance. In conclusion we investigate different backbone networks with grouped labels dataset in shape and colour and we recommend using VGGNet FCN with CE combined with DL for imbalanced datasets.

Morphological Cross Entropy Loss for Improved Semantic Segmentation of Small and Thin Objects

Image segmentation often faces a trade-off between using higher resolution images to detect fine details, such as the edges of objects or thin structures, and lower resolution images which is more suitable for accurate segmentation of massive objects. Because low resolution images require less resources, accurate detection of small objects is often less prioritized in trying to achieve the highest accuracy. In this paper, we propose to improve the segmentation of small and thin objects by convolutional neural networks by adding a morphological element to the loss function used for training the segmentation network. The approach is tested on a traffic sign segmentation problem using the Cityscapes dataset with a training set of 2 979 images and is shown to have an advantage over the popular cross-entropy (CE) and a ground truth (GT) affinity map weighted CE as it yields higher global IoU and, more importantly, an IoU gain among the smaller traffic signs with no additional computational resources.

CentroidNetV2: A hybrid deep neural network for small-object segmentation and counting

This paper presents CentroidNetV2, a novel hybrid Convolutional Neural Network (CNN) that has been specifically designed to segment and count many small and connected object instances. This complete redesign of the original CentroidNet uses a CNN backbone to regress a field of centroid-voting vectors and border-voting vectors. The segmentation masks of the individual object instances are produced by decoding centroid votes and border votes. A loss function that combines cross-entropy loss and Euclidean-distance loss achieves high quality centroids and borders of object instances. Several backbones and loss functions are tested on three different datasets ranging from precision agriculture to microbiology and pathology. CentroidNetV2 is compared to the state-of-the art networks You Only Look Once Version 3 (YOLOv3) and Mask Recurrent Convolutional Neural Network (MRCNN). On two out of three datasets CentroidNetV2 achieves the highest F1 score and on all three datasets CentroidNetV2 achieves the highest recall. CentroidNetV2 demonstrates the best ability to detect small objects although the best segmentation masks for larger objects are produced by MRCNN.

High-resolution SAR images segmentation using NSCT denoising and QIGA based parameters selection of PCNN model

The high-resolution synthetic aperture radar (SAR) images usually contain inhomogeneous coherent speckle noises. For the high-resolution SAR image segmentation with such noises, the conventional methods based on pulse coupled neural networks (PCNN) have to face heavy parameters with a low efficiency. In order to solve the problems, this paper proposes a novel SAR image segmentation algorithm based on non-subsampling Contourlet transform (NSCT) denoising and quantum immune genetic algorithm (QIGA) improved PCNN models. The proposed method first denoising the SAR images for a pre-processing based on NSCT. Then, by using the QIGA to select parameters for the PCNN models, such models self-adaptively select the suitable parameters for segmentation of SAR images with different scenes. This method decreases the number of parameters in the PCNN models and improves the efficiency of PCNN models. At last, by using the optimal threshold to binary the segmented SAR images, the small objects and large scales from the original SAR images will be segmented. To validate the feasibility and effectiveness of the proposed algorithm, four different comparable experiments are applied to validate the proposed algorithm. Experimental results have shown that NSCT pre-processing has a better performance for coherent speckle noises suppression, and QIGA-PCNN model based on denoised SAR images has an obvious segmentation performance improvement on region consistency and region contrast than state-of-the-arts methods. Besides, the segmentation efficiency is also improved than conventional PCNN model, and the level of time complexity meets the state-of-the-arts methods. Our proposed NSCT+QIGA-PCNN model can be used for small object segmentation and large scale segmentation in high-resolution SAR images. The segmented results will be further used for object classification and recognition, regions of interest extraction, and moving object detection and tracking.

L-Seg: An end-to-end unified framework for multi-lesion segmentation of fundus images

Diabetic retinopathy and diabetic macular edema are the two leading causes for blindness in working-age people, and the quantitative and qualitative diagnosis of these two diseases usually depends on the presence and areas of lesions in fundus images. The main related lesions include soft exudates, hard exudates, microaneurysms, and haemorrhages. However, segmentation of these four kinds of lesions is difficult due to their uncertainty in size, contrast, and high interclass similarity. Therefore, we aim to design a multi-lesion segmentation model. We have designed the first small object segmentation network (L-Seg) that can segment the four kinds of lesions simultaneously. Taking into account that small lesion regions could not response at high level of network, we propose a multi-scale feature fusion method to handle this problem. In addition, when considering the cases of both class-imbalance and loss-imbalance problems, we propose a multi-channel bin loss. We have evaluated L-Seg on three fundus datasets including two publicly available datasets - IDRiD and e-ophtha and one private dataset - DDR. Extensive experiments have demonstrated that L-Seg achieves better performance in small lesion segmentation than other deep learning models and traditional methods. Specially, the mAUC score of L-Seg is over 16.8%, 1.51% and 3.11% higher than that of DeepLab v3+ on IDRiD, e-ophtha and DDR datasets, respectively. Moreover, our framework shows competitive performance compared with top-3 teams in IDRiD challenge. The source code of L-Seg is available at: https://github.com/guomugong/L-Seg.

Small object segmentation with fully convolutional network based on overlapping domain decomposition

We propose a new segmentation algorithm based on deep learning. To segment ice hockey players, a fully convolutional network (FCN) is adopted and fine-tuned with our augmented training data. The original FCN has difficulty segmenting small-size objects. To solve this problem, our method divides an input image into four overlapping sub-images and each image is fed into the deep learning network. After obtaining segmentation results from all sub-images, we combine them into a single result. The segmentation results should be consistent over time in video. Thus, our method tracks segments over time and removes false positives that appear for brief periods. Mathematically, we show that our overlapping subdivision process can be interpreted as overlapping domain decomposition methods, which enable the FCN to regularize over consecutive sub-images in training time. Experimental results demonstrate that our method accurately segments ice hockey players when they appear small and when there exists severe background clutter. Our method shows real-time performance.

An end-to-end differential network learning method for semantic segmentation

Deep convolution neural network has become the primary framework for semantic image segmentation in recent years, and most existing methods using deep learning have achieved a great improvement on the performance compared with traditional methods. Although most methods using fully convolutional networks are concerned about the segmentation of small objects or small/fine parts of objects, the small object segmentation is still a challenging problem. To the best of our knowledge, the main reason is that several pooling or convolution operations with two or more stride size cause the features of small objects to vanish in later layers, even if taking different kinds of multi-scale measures. In the paper, we design a novel differential network which addresses the small object segmentation. Specifically, our networks include two pipelines: the first pipeline is served as the primary segmentation network using existing methods, and the second one is a refine network that we propose. The score maps of two networks are merged by calculating the sum of corresponding channels in their last layers. We first learn the primary segmentation network to get a coarse segmentation model, and then train the two networks jointly in an end-to-end fashion. Experiments show that our method can deal with small objects effectively. The segmentation performance of our method on PASCAL VOC 2012 dataset is superior to the state-of-the-art methods using only the primary segmentation model without applying a differential network.