Real-time Segmentation Research Articles

RoboSeg: Real-Time Semantic Segmentation on Computationally Constrained Robots

Real-time and high-performance segmentation is a crucial but challenging perception task for computationally constrained robots, such as the humanoid NAO robot used in the RoboCup Soccer Standard Platform League. However, most existing convolutional neural network (CNN)-based models for semantic segmentation suffer from massive computational costs, which prevents them from being applied to performing real-time inference with a NAO. In this article, we first publish meticulously annotated datasets for training and evaluating semantic segmentation models. Then, we propose a fast downsampling module that downsamples the image while maintaining the spatial information and a novel dense learning module that learns high-level semantic information while recovering the spatial details. Based on these operations, by using a multiscale fusion method to recover the resolution, we propose a more efficient and real-time segmentation model called RoboSeg primarily aimed at offering better speed and accuracy tradeoffs. Finally, to accommodate practical engineering applications, we offer a promising deployment guideline for the CNN model describing how to deploy it on computational resource-limited robots and achieve real-time performance. The experimental results show that the RoboSeg exceeds the state-of-the-art networks in RoboCup scene segmentation: we attain a mean IoU of 87.35% and a pixel accuracy of 96.88% on our dataset using a model that contains only 0.29M parameters and performs just 0.73 GFLOPs. Under the proposed deployment strategies, the network can run at above 30 FPS on NAO robots with downsampled frames.

Content Estimation of Foreign Fibers in Cotton Based on Deep Learning

Cotton foreign fibers directly affect the quality of a textile product; the less foreign fibers in raw cotton, the higher the quality grade of the textile product. Based on the foreign fiber clean machine, this paper proposed an evaluation method of foreign fiber content using deep learning. First of all, a large number of images of foreign fibers were collected from different production lines and annotated to obtain the mask image dataset of foreign fibers. Secondly, by comparing the image segmentation algorithm based on deep learning, tests showed that U-Net has a better performance on different segment metrics evaluations, and U-Net is improved to realize the real-time segmentation of foreign fiber images. The actual size of the foreign fiber could be calculated through the combination of the segment result and the mechanical parameters of the machine. Finally, the test results showed that the relative error between the estimated size and the actual size was less than 4%. After the prototype test, the algorithm was deployed on the actual production line and, by comparing the algorithm data in a random time with the actual foreign fiber statistical data, the overall error was less than 2%. The test showed that the new evaluation method can fully reflect the content of foreign fiber in raw cotton.

Fast body part segmentation and tracking of neonatal video data using deep learning

Photoplethysmography imaging (PPGI) for non-contact monitoring of preterm infants in the neonatal intensive care unit (NICU) is a promising technology, as it could reduce medical adhesive-related skin injuries and associated complications. For practical implementations of PPGI, a region of interest has to be detected automatically in real time. As the neonates’ body proportions differ significantly from adults, existing approaches may not be used in a straightforward way, and color-based skin detection requires RGB data, thus prohibiting the use of less-intrusive near-infrared (NIR) acquisition. In this paper, we present a deep learning-based method for segmentation of neonatal video data. We augmented an existing encoder-decoder semantic segmentation method with a modified version of the ResNet-50 encoder. This reduced the computational time by a factor of 7.5, so that 30 frames per second can be processed at 960 × 576 pixels. The method was developed and optimized on publicly available databases with segmentation data from adults. For evaluation, a comprehensive dataset consisting of RGB and NIR video recordings from 29 neonates with various skin tones recorded in two NICUs in Germany and India was used. From all recordings, 643 frames were manually segmented. After pre-training the model on the public adult data, parts of the neonatal data were used for additional learning and left-out neonates are used for cross-validated evaluation. On the RGB data, the head is segmented well (82% intersection over union, 88% accuracy), and performance is comparable with those achieved on large, public, non-neonatal datasets. On the other hand, performance on the NIR data was inferior. By employing data augmentation to generate additional virtual NIR data for training, results could be improved and the head could be segmented with 62% intersection over union and 65% accuracy. The method is in theory capable of performing segmentation in real time and thus it may provide a useful tool for future PPGI applications.Graphical This work presents the development of a customized, real-time capable Deep Learning architecture for segmenting of neonatal videos recorded in the intensive care unit. In addition to hand-annotated data, transfer learning is exploited to improve performance.

Automatic Detection and Segmentation on Gas Plumes from Multibeam Water Column Images

The detection of gas plumes from multibeam water column (MWC) data is the most direct way to discover gas hydrate reservoirs, but current methods often have low reliability, leading to inefficient detections. Therefore, this paper proposes an automatic method for gas plume detection and segmentation by analyzing the characteristics of gas plumes in MWC images. This method is based on the AdaBoost cascade classifier, combining the Haar-like feature and Local Binary Patterns (LBP) feature. After obtaining the detected result from the above algorithm, a target localization algorithm, based on a histogram similarity calculation, is given to exactly localize the detected target boxes, by considering the differences in gas plume and background noise in the backscatter strength. On this basis, a real-time segmentation method is put forward to get the size of the detected gas plumes, by integration of the image intersection and subtraction operation. Through the shallow-water and deep-water experiment verification, the detection accuracy of this method reaches 95.8%, the precision reaches 99.35% and the recall rate reaches 82.7%. Integrated with principles and experiments, the performance of the proposed method is analyzed and discussed, and finally some conclusions are drawn.

Efficient Semantic Segmentation Using Multi-Path Decoder

Benefiting from the booming of deep learning, the state-of-the-art models achieved great progress. But they are huge in terms of parameters and floating point operations, which makes it hard to apply them to real-time applications. In this paper, we propose a novel deep neural network architecture, named MPDNet, for fast and efficient semantic segmentation under resource constraints. First, we use a light-weight classification model pretrained on ImageNet as the encoder. Second, we use a cost-effective upsampling datapath to restore prediction resolution and convert features for classification into features for segmentation. Finally, we propose to use a multi-path decoder to extract different types of features, which are not ideal to process inside only one convolutional neural network. The experimental results of our model outperform other models aiming at real-time semantic segmentation on Cityscapes. Based on our proposed MPDNet, we achieve 76.7% mean IoU on Cityscapes test set with only 118.84GFLOPs and achieves 37.6 Hz on 768 × 1536 images on a standard GPU.

DENS-ECG: A deep learning approach for ECG signal delineation

Objectives With the technological advancements in the field of tele-health monitoring, it is now possible to gather huge amount of electro-physiological signals such as the electrocardiogram (ECG). It is therefore necessary to develop models/algorithms that are capable of analysing these massive amount of data in real-time. This paper proposes a deep learning model for real-time segmentation of heartbeats. Methods The proposed DENS-ECG algorithm, combines convolutional neural network (CNN) and long short-term memory (LSTM) model to detect onset, peak, and offset of different heartbeat waveforms such as the P-waves, QRS complexes, T-waves, and No waves (NW). Using ECG as the inputs, the model learns to extract high level features through the training process, which, unlike other classical machine learning based methods, eliminates the feature engineering step. Results The proposed DENS-ECG model was trained and validated on a dataset with 105 ECG records of length 15 min each and achieved an average sensitivity and precision of 97.95% and 95.68%, respectively, using a stratified 5-fold cross validation. Additionally, the model was evaluated on an unseen dataset to examine its robustness in QRS detection, which resulted in a sensitivity of 99.61% and precision of 99.52%. Conclusion The empirical results show the flexibility and accuracy of the combined CNN-LSTM model for ECG signal delineation. Significance This paper proposes an efficient and easy to use approach using deep learning for heartbeat segmentation, which could potentially be used in real-time tele-health monitoring systems.

Current Challenges and Barriers to Real-World Artificial Intelligence Adoption for the Healthcare System, Provider, and the Patient.

Artificial intelligence (AI), or the use of automated systems that display the ability to correctly interpret, to learn from, and to achieve specific goals by use of external data, is an emerging technology that has myriad implications for changing the way we interact with the world. Although this technology is already being used in many fields such as banking, retail, and education, AI has the potential to transform other fields including healthcare. Within healthcare, ophthalmology is uniquely positioned to benefit from AI not only through clinical decision support technology but also through improved image processing innovations such as real-time segmentation, automated image quality improvements, and assisted or autonomous disease screening tools.1,2 Although there are now Food and Drug Administration–approved technologies within ophthalmology, such as IDx-DR (Coralville, IA, USA) for early diagnosis of diabetic retinopathy and diabetic macular edema, numerous challenges still exist to realize the potentially transformative impact of these technologies in day to day practice. The need for the ophthalmology community to take a thoughtful approach to AI innovation and implementation is accentuated by the high stakes involved. The impact of misleading patients and clinicians on a health condition is much greater than a retail store misinterpreting the next book you may like to buy. As a result, we need to increase discussion about the issues surrounding who, what, when, how, and why we might use AI in practice, including ethical and liability considerations, to determine how best to implement AI for all stakeholders including practitioners, patients, practices/hospitals, and industry. This article aims to highlight the challenges and barriers to real-world AI adoption that impact the technology's utility. We examine these specific challenges that will face health care organizations, providers, and patients.

Real-time segmentation and tracking of excised corneal contour by deep neural networks for DALK surgical navigation

Objective: Corneal disease is one of the main causes of blindness for humans globally nowadays, and deep anterior lamellar keratoplasty (DALK) is a widely applied technique for corneal transplantation. However, the position of stitch points highly influences the success rate of such surgery, which would require accurate control and manipulation of surgical instruments.Methods: In this paper, we present a deep learning framework for augmented reality (AR) based surgery navigation to guide the suturing in DALK. It can robustly track the excised corneal contour by semantic segmentation and the reconstruction of occlusion. We propose a novel optical flow inpainting network to recover the missing motion caused by occlusion. The occluded regions are detected by weakly supervised segmentation of surgical instruments and reconstructed by key frame warping along the completed optical flow. Then we introduce two types of loss function to adapt the inpainting network in the optical flow space.Results: Our techniques are tested and evaluated by a number of real surgery videos from Shandong Eye Hospital in China. We compare our approaches with other typical methods in the corneal contour segmentation, optical flow inpainting and occlusion regions reconstruction. The tracking accuracy reachs 99.2% in average and PSNR reaches 25.52 for the reconstruction of the occluded frames.Conclusion: From the experimental evaluations and user study, both the qualitative and quantitative results indicate that our techniques can achieve accurate detection and tracking of corneal contour under complex disturbance in real-time surgical scenes. Our prototype AR navigation system would be highly useful in clinical practice.

Efficient use of recent progresses for Real-time Semantic segmentation

Different approaches were proposed to design deep CNNs for semantic segmentation. Usually, they are built upon an encoder–decoder architecture and require computationally expensive operations on high-resolution activation maps. Since for real-time segmentation the costs are critical, efficient approaches compromise spatial information to achieve real-time segmentation but with a considerable drop in accuracy. We introduce a new module based on depthwise separable, shuffled and grouped convolutions that optimize up-sampling operations by using a sizeable receptive field and preserving spatial information. Then, we designed an efficient network based on dense connectivity to achieve a remarkable trade-off accuracy and speed. We show through set of experiments that even by up-sampling with a lightweight decoder, our applied architecture scores on Cityscape 69.5% Mean IoU with $$1024\times 512$$ inputs and 95.2 FPS on the test set.

Real-time retinal layer segmentation of OCT volumes with GPU accelerated inferencing using a compressed, low-latency neural network.

Segmentation of retinal layers in optical coherence tomography (OCT) is an essential step in OCT image analysis for screening, diagnosis, and assessment of retinal disease progression. Real-time segmentation together with high-speed OCT volume acquisition allows rendering of en face OCT of arbitrary retinal layers, which can be used to increase the yield rate of high-quality scans, provide real-time feedback during image-guided surgeries, and compensate aberrations in adaptive optics (AO) OCT without using wavefront sensors. We demonstrate here unprecedented real-time OCT segmentation of eight retinal layer boundaries achieved by 3 levels of optimization: 1) a modified, low complexity, neural network structure, 2) an innovative scheme of neural network compression with TensorRT, and 3) specialized GPU hardware to accelerate computation. Inferencing with the compressed network U-NetRT took 3.5 ms, improving by 21 times the speed of conventional U-Net inference without reducing the accuracy. The latency of the entire pipeline from data acquisition to inferencing was only 41 ms, enabled by parallelized batch processing. The system and method allow real-time updating of en face OCT and OCTA visualizations of arbitrary retinal layers and plexuses in continuous mode scanning. To the best our knowledge, our work is the first demonstration of an ophthalmic imager with embedded artificial intelligence (AI) providing real-time feedback.

Sensorless adaptive-optics optical coherence tomographic angiography

Optical coherence tomographic angiography (OCTA) can image the retinal blood flow but visualization of the capillary caliber is limited by the low lateral resolution. Adaptive optics (AO) can be used to compensate ocular aberrations when using high numerical aperture (NA), and thus improve image resolution. However, previously reported AO-OCTA instruments were large and complex, and have a small sub-millimeter field of view (FOV) that hinders the extraction of biomarkers with clinical relevance. In this manuscript, we developed a sensorless AO-OCTA prototype with an intermediate numerical aperture to produce depth-resolved angiograms with high resolution and signal-to-noise ratio over a 2 × 2 mm FOV, with a focal spot diameter of 6 µm, which is about 3 times finer than typical commercial OCT systems. We believe these parameters may represent a better tradeoff between resolution and FOV compared to large-NA AO systems, since the spot size matches better that of capillaries. The prototype corrects defocus, astigmatism, and coma using a figure of merit based on the mean reflectance projection of a slab defined with real-time segmentation of retinal layers. AO correction with the ability to optimize focusing in arbitrary retinal depths - particularly the plexuses in the inner retina - could be achieved in 1.35 seconds. The AO-OCTA images showed greater flow signal, signal-to-noise ratio, and finer capillary caliber compared to commercial OCTA. Projection artifacts were also reduced in the intermediate and deep capillary plexuses. The instrument reported here improves OCTA image quality without excessive sacrifice in FOV and device complexity, and thus may have potential for clinical translation.

Real-time segmentation of remote sensing images with a combination of clustering and Bayesian approaches

In the area of remote sensing image processing, accurate segmentation of high-resolution remote sensing images in real time remains a challenging problem and numerous approaches have been developed for the problem. This paper proposes a new unsupervised approach that can efficiently analyze a remote sensing image and provide accurate segmentation results. The approach performs segmentation in three stages. In the first stage, an image is partitioned into blocks of equal sizes. The mean values of the R, G and B components of the pixels in each block are computed to form a feature vector of the block. A preliminary segmentation result is obtained by clustering the feature vectors with a simple clustering algorithm. In the second stage, a Bayesian approach is applied to refine the preliminary segmentation result. In the final stage, a graph-based method is utilized to recognize regions with complex texture structures. The performance of this approach has been tested on a few benchmark datasets, and its segmentation accuracy is compared with that of many state-of-the-art segmentation tools for remote sensing images. The testing results show that the overall segmentation accuracy of the proposed approach is higher than that of the other tools, and real-time analysis suggests that the approach is promising for real-time applications. An implementation of the approach in MATLAB is freely available upon request.

Measuring Efficiency of Semi-automated Brain Tumor Segmentation by Simulating User Interaction

Traditionally, radiologists have crudely quantified tumor extent by measuring the longest and shortest dimension by dragging a cursor between opposite boundary points across a single image rather than full segmentation of the volumetric extent. For algorithmic-based volumetric segmentation, the degree of radiologist experiential involvement varies from confirming a fully automated segmentation, to making a single drag on an image to initiate semi-automated segmentation, to making multiple drags and clicks on multiple images during interactive segmentation. An experiment was designed to test an algorithm that allows various levels of interaction. Given the ground-truth of the BraTS training data, which delimits the brain tumors of 285 patients on multi-spectral MR, a computer simulation mimicked the process that a radiologist would follow to perform segmentation with real-time interaction. Clicks and drags were placed only where needed in response to the deviation between real-time segmentation results and assumed radiologist's goal, as provided by the ground-truth. Results of accuracy for various levels of interaction are presented along with estimated elapsed time, in order to measure efficiency. Average total elapsed time, including loading the study through confirming 3D contours, was 46 s.

Combining Real-Time Segmentation and Classification of Rehabilitation Exercises with LSTM Networks and Pointwise Boosting

Autonomous biofeedback tools in support of rehabilitation patients are commonly built as multi-tier pipelines, where a segmentation algorithm is first responsible for isolating motion primitives, and then classification can be performed on each primitive. In this paper, we present a novel segmentation technique that integrates on-the-fly qualitative classification of physical movements in the process. We adopt Long Short-Term Memory (LSTM) networks to model the temporal patterns of a streaming multivariate time series, obtained by sampling acceleration and angular velocity of the limb in motion, and then we aggregate the pointwise predictions of each isolated movement using different boosting methods. We tested our technique against a dataset composed of four common lower-limb rehabilitation exercises, collected from heterogeneous populations (clinical and healthy). Experimental results are promising and show that combining segmentation and classification of orthopaedic movements is a valid method with many potential real-world applications.

Catheter segmentation in X-ray fluoroscopy using synthetic data and transfer learning with light U-nets

Background and objectivesAutomated segmentation and tracking of surgical instruments and catheters under X-ray fluoroscopy hold the potential for enhanced image guidance in catheter-based endovascular procedures. This article presents a novel method for real-time segmentation of catheters and guidewires in 2d X-ray images. We employ Convolutional Neural Networks (CNNs) and propose a transfer learning approach, using synthetic fluoroscopic images, to develop a lightweight version of the U-Net architecture. Our strategy, requiring a small amount of manually annotated data, streamlines the training process and results in a U-Net model, which achieves comparable performance to the state-of-the-art segmentation, with a decreased number of trainable parameters.MethodsThe proposed transfer learning approach exploits high-fidelity synthetic images generated from real fluroscopic backgrounds. We implement a two-stage process, initial end-to-end training and fine-tuning, to develop two versions of our model, using synthetic and phantom fluoroscopic images independently. A small number of manually annotated in-vivo images is employed to fine-tune the deepest 7 layers of the U-Net architecture, producing a network specialized for pixel-wise catheter/guidewire segmentation. The network takes as input a single grayscale image and outputs the segmentation result as a binary mask against the background.ResultsEvaluation is carried out with images from in-vivo fluoroscopic video sequences from six endovascular procedures, with different surgical setups. We validate the effectiveness of developing the U-Net models using synthetic data, in tests where fine-tuning and testing in-vivo takes place both by dividing data from all procedures into independent fine-tuning/testing subsets as well as by using different in-vivo sequences. Accurate catheter/guidewire segmentation (average Dice coefficient of ~ 0.55, ~ 0.26 and ~ 0.17) is obtained with both U-Net models. Compared to the state-of-the-art CNN models, the proposed U-Net achieves comparable performance ( ± 5% average Dice coefficients) in terms of segmentation accuracy, while yielding a 84% reduction of the testing time. This adds flexibility for real-time operation and makes our network adaptable to increased input resolution.ConclusionsThis work presents a new approach in the development of CNN models for pixel-wise segmentation of surgical catheters in X-ray fluoroscopy, exploiting synthetic images and transfer learning. Our methodology reduces the need for manually annotating large volumes of data for training. This represents an important advantage, given that manual pixel-wise annotations is a key bottleneck in developing CNN segmentation models. Combined with a simplified U-Net model, our work yields significant advantages compared to current state-of-the-art solutions.

Real time Fully Automated Internal Layer Segmentation of Human Retina in Optical Coherence Tomography Images

In the field of ophthalmology, optical coherence tomography (OCT) has proven to be a powerful imaging technique when it comes to diagnosing various eye-related diseases. This research article introduces a real-time automatic retinal layer segmentation algorithm based on intensity variation in the OCT images. The built algorithm is capable of detecting internal retinal layers like the internal limiting membrane (ILM), the retinal pigment epithelium (RPE) and the retinal nerve fiber layer (RNFL) with micrometer level precision, the algorithm uses openMP for parallelized computation for real-time visualization of the segmented retinal layers. The total execution time of the algorithm was evaluated using various image sizes and compared with the OCT frame rate to demonstrate the efficiency of real-time segmentation.

MFENet: Multi-level feature enhancement network for real-time semantic segmentation

Accuracy is of vital importance for real-time semantic segmentation. However, modern methods often weaken high-level or low-level feature extraction to promote inference speed, thereby resulting in poor accuracy. In this paper, we present a Multi-level Feature Enhancement Network (MFENet) to enhance the feature extraction of each level in backbone. This approach can achieve high performance while maintaining high inference speed. We first rely on a Spatial and Edge Extraction Module with the Laplace Operator to improve the edge information extraction of low-level features. Next, we design a Context Boost Module to increase the context information inside each object of high-level features. Finally, we introduce the Selective Refinement Module to selectively combine the information from these two modules. Our network attained precise real-time segmentation results on Cityscapes, CamVid and COCO-Stuff datasets. More specifically, the architecture achieved 76.7% Mean IoU on the Cityscapes test dataset with 12.5 GFLOPS and a speed of 47 FPS on one NVIDIA Titan Xp card, which is more accurate than existing real-time methods.

Visually Guided Acquisition of Contact Dynamics and Case Study in Data-Driven Haptic Texture Modeling.

Data-driven modeling of human hand contact dynamics starts with a tedious process of data collection. The data of contact dynamics consist of an input describing an applied action and response stimuli from the environment. The quality and stability of the model mainly depend on how well data points cover the model space. Thus, in order to build a reliable data-driven model, a user usually collects data dozens of times. In this article, we aim to build an interactive system that assists a user in data collection. We develop an online segmentation framework that partitions a multivariate streaming signal. Real-time segmentation allows for tracking the process of how the model space is being populated. We applied the proposed framework for a haptic texture modeling use-case. In order to guide a user in data collection, we designed a user interface mapping applied input to alternative visual modalities based on the theory of direct perception. A combination of the segmentation framework and user interface implements a human-in-loop system, where the user interface assigns the target combination of input variables and the user tries to acquire them. Experimental results show that the proposed data collection schema considerably increases the approximation quality of the model, whereas the proposed user interface considerably reduces mental workload experienced during data collection.

Real-time segmentation of weeds in cornfields based on depthwise separable convolution residual network

Traditional artificial spraying of pesticides leads not only to lower utilisation of pesticides but also to environmental pollution. However, intelligent weeding devices can identify weeds and crops through sensing devices for selective spraying, which will effectively reduce the use of pesticides. The accurate and efficient identification method of crops and weeds is very crucial to the development of the mechanised weeding model. To improve the segmentation exactitude and real-time performance of crops and weeds, we propose a lightweight network based on the encoder-decoder architecture, namely, SResNet. The shuffle-split-separable-residual block was employed to compress the model and increase the number of network layers at the same time, thereby extracting more abundant pixel category information. Besides, the model was optimised by a weighted cross-entropy loss function due to the imbalance of pixel ratios of background, crops and weeds. The results of the experiment prove that the method presented can greatly improve the segmentation accuracy and real-time segmentation speed on the crops and weeds dataset.

基于膨胀卷积平滑及轻型上采样的实时语义分割

基于膨胀卷积平滑及轻型上采样的实时语义分割