Real-time Segmentation Research Articles

Towards Real-Time Segmentation on the Edge

The research in real-time segmentation mainly focuses on desktop GPUs. However, autonomous driving and many other applications rely on real-time segmentation on the edge, and current arts are far from the goal. In addition, recent advances in vision transformers also inspire us to re-design the network architecture for dense prediction task. In this work, we propose to combine the self attention block with lightweight convolutions to form new building blocks, and employ latency constraints to search an efficient sub-network. We train an MLP latency model based on generated architecture configurations and their latency measured on mobile devices, so that we can predict the latency of subnets during search phase. To the best of our knowledge, we are the first to achieve over 74% mIoU on Cityscapes with semi-real-time inference (over 15 FPS) on mobile GPU from an off-the-shelf phone.

LcmUNet: A Lightweight Network Combining CNN and MLP for Real-Time Medical Image Segmentation.

In recent years, UNet and its improved variants have become the main methods for medical image segmentation. Although these models have achieved excellent results in segmentation accuracy, their large number of network parameters and high computational complexity make it difficult to achieve medical image segmentation in real-time therapy and diagnosis rapidly. To address this problem, we introduce a lightweight medical image segmentation network (LcmUNet) based on CNN and MLP. We designed LcmUNet's structure in terms of model performance, parameters, and computational complexity. The first three layers are convolutional layers, and the last two layers are MLP layers. In the convolution part, we propose an LDA module that combines asymmetric convolution, depth-wise separable convolution, and an attention mechanism to reduce the number of network parameters while maintaining a strong feature-extraction capability. In the MLP part, we propose an LMLP module that helps enhance contextual information while focusing on local information and improves segmentation accuracy while maintaining high inference speed. This network also covers skip connections between the encoder and decoder at various levels. Our network achieves real-time segmentation results accurately in extensive experiments. With only 1.49 million model parameters and without pre-training, LcmUNet demonstrated impressive performance on different datasets. On the ISIC2018 dataset, it achieved an IoU of 85.19%, 92.07% recall, and 92.99% precision. On the BUSI dataset, it achieved an IoU of 63.99%, 79.96% recall, and 76.69% precision. Lastly, on the Kvasir-SEG dataset, LcmUNet achieved an IoU of 81.89%, 88.93% recall, and 91.79% precision.

Refined Feature-based Multi-frame and Multi-scale Fusing Gate network for accurate segmentation of plaques in ultrasound videos

The accurate segmentation of carotid plaques in ultrasound videos will provide evidence for clinicians to evaluate the properties of plaques and treat patients effectively. However, the confusing background, blurry boundaries and plaque movement in ultrasound videos make accurate plaque segmentation challenging. To address the above challenges, we propose the Refined Feature-based Multi-frame and Multi-scale Fusing Gate Network (RMFG_Net), which captures spatial and temporal features in consecutive video frames for high-quality segmentation results and no manual annotation of the first frame. A spatial–temporal feature filter is proposed to suppress the noise of low-level CNN features and promote the detailed target area. To obtain a more accurate plaque position, we propose a transformer-based cross-scale spatial location algorithm, which models the relationship between adjacent layers of consecutive video frames to achieve stable positioning. To make full use of more detailed and semantic information, multi-layer gated computing is applied to fuse features of different layers, ensuring sufficient useful feature map aggregation for segmentation. Experiments on two clinical datasets demonstrate that the proposed method outperforms other state-of-the-art methods under different evaluation metrics, and it processes images with a speed of 68 frames per second which is suitable for real-time segmentation. A large number of ablation experiments were conducted to demonstrate the effectiveness of each component and experimental setting, as well as the potential of the proposed method in ultrasound video plaque segmentation tasks. The codes can be publicly available from https://github.com/xifengHuu/RMFG_Net.git.

Real-Time Location, Correction and Segmentation Algorithm based on Tilted License Plate

Background: Vehicles have become one of the most important means of transportation, and license plate is the only identifying mark of vehicles. License plate recognition technologies are being applied to a series of occasions, such as supervising road traffic violations, recovering stolen vehicles, monitoring wanted vehicles, and dispatching special vehicles. The license plate tilt phenomenon caused by various reasons has brought great trouble to its own recognition. Objective: How to identify tilted license plates efficiently and accurately becomes the key to the automatic management of a large number of vehicles. Therefore, a real-time location, correction and segmentation algorithm based on the tilted license plate is proposed in this paper. Methods: Firstly, an end-to-end deep convolutional neural network (CNN) is proposed for the location detection of the tilted license plate. The developed CNN is optimized by the most advanced RefineDet algorithm. By improving the object detection module, the representation ability of the CNN for small features and the accuracy of the detector are improved, so as to make the location regression and label prediction of the detected object more accurate. Secondly, the optimized perspective transformation algorithm is applied to correct the tilted license plate. According to the vertices coordinates of the bounding box detected by our CNN, the license plate area cropped out from the original image has a certain tilt angle, and the perspective transformation algorithm achieves the correction. Finally, digital image processing technology is used to segment the characters of license plates. Results: The experimental results in the Chinese City Parking (CCP) dataset show that the proposed algorithm exhibits location average precisions improvements of 2.4-5.4% over the other algorithm. Conclusion: The proposed algorithm achieves high-accuracy correction and real-time segmentation.

URTSegNet: A real-time segmentation network of unstructured road at night based on thermal infrared images for autonomous robot system

When it comes to poor lighting conditions such as night, backlight, haze, etc., the detection method of unstructured roads based on visible light cameras often fails. Therefore, this paper aims to improve the detection performance of unstructured roads based on thermal infrared images for autonomous robot systems. For this purpose, we propose a real-time network for unstructured road segmentation at night based on thermal infrared images (URTSegNet) in this paper. Like BiSeNet V2, we adopt a two-branch structure, including Detail Branch and Semantic Branch, to extract spatial detail and contextual features. The Detail Branch maintains a small receptive field and high capacity and adjusts the computational cost by employing depth-wise separable convolutions. In the Semantic Branch, we modify Stem Block to obtain more information because of the characteristics of infrared images. Dilated convolution expands the receptive field while keeping the number of parameters unchanged. Finally, we design a more concise and efficient Aggregation Block using a gate mechanism. An infrared unstructured road dataset, including total darkness at night, low light, backlight, haze, and other scenes, is collected to get the real-world dataset and effectively evaluate performance. We got 97.5% Intersection over Union (IoU) and reached an inference speed of 129 FPS on a single NVIDIA GTX 1660S card on our dataset. In addition, extensive evaluation experiments are conducted on two public datasets to verify the generalization ability of the model and the role of each module. This research addresses the problem of road detection for unmanned platforms in unstructured environments at night or under poor lighting conditions. The algorithm has high accuracy and a fast inference speed so that it can be implemented on various embedded devices.

Travel Time Distribution Estimation by Learning Representations Over Temporal Attributed Graphs

Travel time estimation is a crucial task in practical transportation applications, while providing the reliability of estimation is important in many working scenarios. Most existing studies do not consider the dynamics of traffic status for different road segments in real time, thus yielding unsatisfactory results. To address the problem, we propose to formulate the traffic network as a temporal attributed graph and perform node representation learning on it. The learned representation is capable of jointly exploiting the dynamic traffic conditions and the topology of the road network, which is then fed into a route-based spatio-temporal dependence learning module to estimate the travel time. By incorporating a distribution loss function, our proposed model is able to predict the distribution of travel time. In the meantime, we design an auxiliary local task of predicting the congestion status of each road segment, which further enhances the generalization performance of the representation learning. Extensive experiments on real-world large-scale datasets demonstrated the superiority of our method compared with the state-of-the-arts.

Lightweight Deep Learning Model for Real-Time Colorectal Polyp Segmentation

In colonoscopy, computer vision and artificial intelligence technology have enabled the automatic detection of the location of polyps and their visualization. These advancements have facilitated considerable research in the field. However, deep learning models used in the segmentation problem for capturing various patterns of polyps are becoming increasingly complex, which has rendered their operation in real time difficult. To identify and overcome this problem, a study was conducted on a model capable of precise polyp segmentation while increasing its processing speed. First, an efficient, high-performance, and lightweight model suitable for the segmentation of polyps was sought; the performance of existing segmentation models was compared and combined to obtain a learning model that exhibited good accuracy and speed. Next, hyperparameters were found for the MobileNetV3-encoder-based DeepLabV3+ model and, after tuning the hyperparameters, quantitative and qualitative results were compared, and the final model was selected. The experimental results showed that this model achieved high accuracy, with a Dice coefficient of 93.79%, while using a limited number of parameters and computational resources. Specifically, the model used 6.18 million parameters and 1.623 giga floating point operations for the CVC-ClinicDB dataset. This study revealed that increasing the amount of computation and parameters of the model did not guarantee unconditional performance. Furthermore, for the search and removal of polyps in cases in which morphological information is critical, an efficient model with low model complexity and high accuracy was proposed for real-time segmentation.

Medical image segmentation method based on multi-feature interaction and fusion over cloud computing

Medical image segmentation is a crucial task in computer-aided diagnosis. While deep learning has significantly improved this field, relying solely on local computing power makes it challenging to achieve real-time segmentation results. Furthermore, traditional convolutional neural networks (CNNs) lack the ability to extract global features. To address these issues, this paper proposes a cloud-based medical image segmentation method that leverages multi-feature extraction and interactive fusion. Specifically, this method employs cloud computing to process a large number of medical images and overcome local computing power limitations. It also combines Transformer and CNNs to extract global and local features, respectively, and introduces an interactive fusion attention module to improve segmentation accuracy. The proposed approach is validated on multiple medical image datasets, and experimental results demonstrate its effectiveness and progress.

Real-time segmentation network for compact camera module assembly adhesives based on improved U-Net

Real-time segmentation network for compact camera module assembly adhesives based on improved U-Net

Automatic Detection of Small Sample Apple Surface Defects Using ASDINet.

The appearance quality of apples directly affects their price. To realize apple grading automatically, it is necessary to find an effective method for detecting apple surface defects. Aiming at the problem of a low recognition rate in apple surface defect detection under small sample conditions, we designed an apple surface defect detection network (ASDINet) suitable for small sample learning. The self-developed apple sorting system collected RGB images of 50 apple samples for model verification, including non-defective and defective apples (rot, disease, lacerations, and mechanical damage). First, a segmentation network (AU-Net) with a stronger ability to capture small details was designed, and a Dep-conv module that could expand the feature capacity of the receptive field was inserted in its down-sampling path. Among them, the number of convolutional layers in the single-layer convolutional module was positively correlated with the network depth. Next, to achieve real-time segmentation, we replaced the flooding of feature maps with mask output in the 13th layer of the network. Finally, we designed a global decision module (GDM) with global properties, which inserted the global spatial domain attention mechanism (GSAM) and performed fast prediction on abnormal images through the input of masks. In the comparison experiment with state-of-the-art models, our network achieved an AP of 98.8%, and a 97.75% F1-score, which were higher than those of most of the state-of-the-art networks; the detection speed reached 39ms per frame, achieving accuracy-easy deployment and substantial trade-offs that are in line with actual production needs. In the data sensitivity experiment, the ASDINet achieved results that met the production needs under the training of 42 defective pictures. In addition, we also discussed the effect of the ASDINet in actual production, and the test results showed that our proposed network demonstrated excellent performance consistent with the theory in actual production.

AMDNet: A Modern UAV RGB Remote-Sensing Tree Species Image Segmentation Model Based on Dual-Attention Residual and Structure Re-Parameterization

Tree species classification is an important and challenging task in image recognition and the management of forest resources. Moreover, the task of tree species classification based on remote sensing images can significantly improve the efficiency of the tree species survey and save costs. In recent years, many large models have achieved high accuracy in the task of tree species classification in an airborne remote-sensing manner, but due to their fixed geometric structure, traditional convolutional neural networks are inherently limited to the local receptive field and can only provide segmental context information. The limitation of insufficient context information greatly affects the segmentation accuracy. In this paper, a dual-attention residual network (AMDNet) and a re-parameterized model approach are proposed to capture the global context information, fuse the weight, reduce the model volume, and maintain the computational efficiency. Firstly, we propose MobileNetV2 as the backbone network for feature extraction to further improve the feature identification by modeling semantic dependencies in the spatial dimension and channel dimension and adding the output of the two attention modules. Then, the attention perception features are generated by stacking the attention modules, and the in-depth residual attention network is trained using attention residual learning, through which more accurate segmentation results can be obtained. Secondly, we adopt the approach of structure re-parameterization, use a multi-branch topology for training, carry out weighted averaging on multiple trained models, and fuse multiple branch modules into a completely equivalent module in inference. The proposed approach results in a reduction in the number of parameters and an accelerated inference speed while also achieving improved classification accuracy. In addition, the model training strategy is optimized based on Transformer to enhance the accuracy of segmentation. The model was used to conduct classification experiments on aerial orthophotos of Hongya Forest Farm in Sichuan, China, and the MIOU of tree species recognition using the test equipment reached 93.8%. Compared with current models such as UNet, our model exhibits a better performance in terms of both speed and accuracy, in addition to its enhanced deployment capacity, and its speed advantage is more conducive to real-time segmentation, thereby representing a novel approach for the classification of tree species in remote sensing imagery with significant potential for practical applications.

Bi-Unet: A Dual Stream Network for Real-Time Highway Surface Segmentation

Highway surface segmentation consists of extracting road surface at pixel-level from the surveillance camera view. Since the intelligent traffic event detection task does not require the detection of off-road scene, the segmentation of highway surface is of great demand. However, it is challenging to accurately extract road surface in real time scenarios. To cope with the above issues, Bi-Unet, a dual stream lightweight network is proposed. Firstly, the dual stream structure enhances segmentation performance on the narrow remote end of the highway and preserves the detailed border information. Then, to perform real-time segmentation, a novel lightweight module (LSM) is introduced to lighten the model and provide higher segmentation accuracy. Moreover, to ensure road segmentation for complex scenes, a Road Attention Network (RAN) module is proposed. Lastly, due to the lack of a suitable benchmark dataset serve for the highway segmentation problem, a new large and high-quality segmentation dataset named Highway-Surface-Free (HSF) is proposed in this paper, which is collected from the perspective of highway surveillance cameras under all-day and all-weather conditions. Compared with the state of arts, the extensive experimental results verify that our proposed Bi-Unet achieves the best overall performance on our proposed HSF dataset.

Spatial-Temporal Aggregated Shuffle Attention for Video Instance Segmentation of Traffic Scene

Aiming at the problem of spatial focus regions distribution dispersion and dislocation in feature pyramid networks and insufficient feature dependency acquisition in both spatial and channel dimensions, this paper proposes a spatial-temporal aggregated shuffle attention for video instance segmentation (STASA-VIS). First, an mixed subsampling (MS) module to embed activating features from the low-level target area of feature pyramid into the high-level is designed, so as to aggregate spatial information on target area. Taking advantage of the coherent information in video frames, STASA-VIS uses the first ones of every 5 video frames as the key-frames and then propagates the keyframe feature maps of the pyramid layers forward in the time domain, and fuses with the non-keyframe mixed subsampled features to achieve time-domain consistent feature aggregation. Finally, STASA-VIS embeds shuffle attention in the backbone to capture the pixel-level pairwise relationship and dimensional dependencies among the channels and reduce the computation. Experimental results show that the segmentation accuracy of STASA-VIS reaches 41.2%, and the test speed reaches 34FPS, which is better than the state-of-the-art one stage video instance segmentation (VIS) methods in accuracy and achieves real-time segmentation.

Real-time gastric intestinal metaplasia diagnosis tailored for bias and noisy-labeled data with multiple endoscopic imaging

This work presents real-time segmentation viz. gastric intestinal metaplasia (GIM). Recently, GIM segmentation of endoscopic images has been carried out to differentiate GIM from a healthy stomach. However, real-time detection is difficult to achieve. Conditions are challenging, and include multiple color modes (white light endoscopy and narrow-band imaging), other abnormal lesions (erosion and ulcer), noisy labels etc. Herein, our model is based on BiSeNet and can overcome the many issues regarding GIM. Application of auxiliary head and additional loss are seen to improve performance as well as enhance multiple color modes accurately. Further, multiple pre-processing techniques are utilized for leveraging detection performance: namely, location-wise negative sampling, jigsaw augmentation, and label smoothing. Finally, the decision threshold can be adjusted separately for each color mode. Work undertaken at King Chulalongkorn Memorial Hospital examined 940 histologically proven GIM images and 1239 non-GIM images, obtained over 173 frames per second (FPS). In terms of accuracy, our model is seen to outperform all baselines. Our results demonstrate sensitivity, specificity, positive predictive, negative predictive, accuracy, and mean intersection over union (IoU), achieving GIM segmentation values of 91%, 96%, 91%, 91%, 96%, and 55%, respectively.

CFPNet-M: A light-weight encoder-decoder based network for multimodal biomedical image real-time segmentation

—Deep learning techniques are proving instrumental in identifying, classifying, and quantifying patterns in medical images. Segmentation is one of the important applications in medical image analysis. The U-Net has become the predominant deep-learning approach to medical image segmentation tasks. Existing U-Net based models have limitations in several respects, however, including: the requirement for millions of parameters in the U-Net, which consumes considerable computational resources and memory; the lack of global information; and incomplete segmentation in difficult cases. To remove some of those limitations, we built on our previous work and applied two modifications to improve the U-Net model: 1) we designed and added the dilated channel-wise CNN module and 2) we simplified the U-shape network. We then proposed a novel light-weight architecture, the Channel-wise Feature Pyramid Network for Medicine (CFPNet-M). To evaluate our method, we selected five datasets from different imaging modalities: thermography, electron microscopy, endoscopy, dermoscopy, and digital retinal images. We compared its performance with several models having a variety of complexities. We used the Tanimoto similarity instead of the Jaccard index for gray-level image comparisons. The CFPNet-M achieves segmentation results on all five medical datasets that are comparable to existing methods, yet require only 8.8 MB memory, and just 0.65 million parameters, which is about 2% of U-Net. Unlike other deep-learning segmentation methods, this new approach is suitable for real-time application: its inference speed can reach 80 frames per second when implemented on a single RTX 2070Ti GPU with an input image size of 256 × 192 pixels.

Accurate Crack Detection Based on Distributed Deep Learning for IoT Environment.

Defects or cracks in roads, building walls, floors, and product surfaces can degrade the completeness of the product and become an impediment to quality control. Machine learning can be a solution for detecting defects effectively without human experts; however, the low-power computing device cannot afford that. In this paper, we suggest a crack detection system accelerated by edge computing. Our system consists of two: Rsef and Rsef-Edge. Rsef is a real-time segmentation method based on effective feature extraction that can perform crack image segmentation by optimizing conventional deep learning models. Then, we construct the edge-based system, named Rsef-Edge, to significantly decrease the inference time of Rsef, even in low-power IoT devices. As a result, we show both a fast inference time and good accuracy even in a low-powered computing environment.

Global video object segmentation with spatial constraint module

We present a lightweight and efficient semi-supervised video object segmentation network based on the space-time memory framework. To some extent, our method solves the two difficulties encountered in traditional video object segmentation: one is that the single frame calculation time is too long, and the other is that the current frame’s segmentation should use more information from past frames. The algorithm uses a global context (GC) module to achieve high-performance, real-time segmentation. The GC module can effectively integrate multi-frame image information without increased memory and can process each frame in real time. Moreover, the prediction mask of the previous frame is helpful for the segmentation of the current frame, so we input it into a spatial constraint module (SCM), which constrains the areas of segments in the current frame. The SCM effectively alleviates mismatching of similar targets yet consumes few additional resources. We added a refinement module to the decoder to improve boundary segmentation. Our model achieves state-of-the-art results on various datasets, scoring 80.1% on YouTube-VOS 2018 and a {cal J}{rm{& }}{cal F} score of 78.0% on DAVIS 2017, while taking 0.05 s per frame on the DAVIS 2016 validation dataset.

Wildfire aerial thermal image segmentation using unsupervised methods: a multilayer level set approach

Background and aims Infrared thermal images of a propagating wildfire taken by manned or unmanned aerial vehicles can help firefighting authorities with combat planning. Segmenting these images into regions of different fire temperatures is a necessary step to measure the fire perimeter and determine the location of the fire front. Methods This work proposes a multilayer segmentation method based on level sets, which have the property of handling topology, making them suitable to segment images that contain scattered fire areas. The experimental results were compared using hand-drawn labels over a set of images provided by the Portuguese Air Force as ground truth. These labels were carefully drawn by the authors to ensure that they complied with the requirements indicated by the Portuguese National Authority for Emergency and Civil Protection. The proposed method was optimised to ensure contour smoothness and reliability, as well as reduce computation time. Key results The proposed method can surpass other common unsupervised methods in terms of intersection over union, although it has not yet been able to perform real-time segmentation. Conclusions Although falling out of use in relation to supervised and deep learning methods, unsupervised segmentation can still be very useful when annotated datasets are unavailable.

Volumetric evaluation of the sphenoid sinus among different races in the Southeast Asian (SEA) population: a computerized tomography study.

Introduction: The fundament of forensic science lies in identifying a body. The morphological complexity of the paranasal sinus (PNS), which varies greatly amongst individual, possess a discriminatory value that potentially contributes to the radiological identification. The sphenoid bone represents the keystone of the skull and forms part of the cranial vault. It is intimately associated with vital neurovascular structures. The sphenoid sinus, located within the body of the sphenoid bone, has variable morphology. The sphenoid septum's inconsistent position and the degree, as well as the direction disparities of sinus pneumatization, have indeed accorded it a unique structure in providing invaluable information in forensic personnel identification. Additionally, the sphenoid sinus is situated deep within the sphenoid bone. Therefore, it is well protected from traumatic degradation from external causes and can be potentially utilized in forensic studies. The authors aim to study the possibility of variation among the race, and gender in the Southeast Asian (SEA) population, using volumetric measurements of the sphenoid sinus. Materials and methods: This is a retrospective cross-sectional analysis of computerized tomographic (CT) imaging of the PNS of 304 patients (167 males, 137 females) in a single centre. The volume of the sphenoid sinus was reconstructed and measured using commercial real-time segmentation software. Result: The total volume of sphenoid sinus of male gender had shown to be larger, 12.22 (4.93 - 21.09) cm3 compared to the counterpart of 10.19 (3.75 - 18.72) cm3 (p = .0090). The Chinese possessed a larger total sphenoid sinus volume, 12.96 (4.62 - 22.21) cm3) than the Malays, 10.68 (4.13 - 19.25) cm3 (p = .0057). No correlation was identified between the age and volume of the sinus (cc= -.026, p = .6559). Conclusion: The sphenoid sinus volume in males was found to be larger than those of females. It was also shown that race influences sinus volume. Volumetric analysis of the sphenoid sinus can potentially be utilized in gender and race determination. The current study provided normative data on the sphenoid sinus volume in the SEA region, which can be helpful for future studies.

Semi-supervised Learning for Real-time Segmentation of Ultrasound Video Objects: A Review

Semi-supervised Learning for Real-time Segmentation of Ultrasound Video Objects: A Review