Real-time Segmentation Research Articles

DNS: A multi-scale deconvolution semantic segmentation network for joint detection and segmentation

Real-time semantic segmentation has become crucial in many applications such as medical image analysis and autonomous driving. In this paper, we introduce a single semantic segmentation network, called DNS, for joint object detection and segmentation task. We take advantage of multi-scale deconvolution mechanism to perform real time computations. To this goal, down-scale and up-scale streams are utilized to combine the multi-scale features for the final detection and segmentation task. By using the proposed DNS, not only the tradeoff between accuracy and cost but also the balance of detection and segmentation performance are settled. Experimental results for PASCAL VOC datasets show competitive performance for joint object detection and segmentation task.

3D transesophageal echocardiography valve model: bedside, real time segmentation, with automated analysis of anatomic landmarks

3D transesophageal echocardiography valve model: bedside, real time segmentation, with automated analysis of anatomic landmarks

Semi-autonomous real-time programmable fluorescence lifetime segmentation with a digital micromirror device.

Time-correlated single-photon counting (TCSPC) is the gold standard for performing lifetime spectroscopy in biological assays. Traditional fluorescence lifetime imaging (FLIM) using laser scanning microscopes are inherently slow due to point scanning all pixels in the field-of-view. Wide-field implementations of TCSPC spectroscopy using microchannel plates benefit from particularly fast acquisition times at the expense of temporal resolution, and are fundamentally limited by photon counting rates. Here, we introduce programmable lifetime imaging (PLI), combining the advantages of wide-field imaging using total internal reflection excitation with state-of-the-art TCSPC detector technology for accurate lifetime determination in an object-oriented manner using a digital micromirror device (DMD). The fluorescent emission is projected onto the DMD to facilitate the sequential segmentation of fluorescence from individual objects in the field-of-view, allowing for both image acquisition and fluorescence lifetime determination of the assay. The sensitivity of PLI is demonstrated by manually segmenting fluorescence from fixed cell assays. We also demonstrate an automated implementation of PLI, using a camera as a feedback mechanism to segment fluorescence produced by emitting objects of interest in the imaging field-of-view, highlighting the advantages of measurement only in areas where valuable information exists. As a result, PLI is able to reduce acquisition time of fluorescence lifetime data by at least an order of magnitude compared to laser scanning implementations.

A low complexity hardware architecture of K-means algorithm for real-time satellite image segmentation

The Real time monitoring of forest area, coastal regions, sea, river basins, nation borders etc. helps in quick determination of devastations caused by natural or man-made catastrophes, which can lead to emergency situations. Real-time Segmentation of satellite images is essential to detect fire, floods, volcanoes, earthquakes etc. of a specific geographical zone. Effective determination of such calamities and taking proactive measures can save a lot of lives and natural resources. In this paper, an efficient FSM based architecture is proposed for performing segmentation of satellite Images procured from NASA’s Operational Land Imager (OLI) on Landsat 8 and Landsat 5. The K-means clustering algorithm even after five decades of it’s existence is quite effective and mostly used for big data applications, owing to its lower complexity. However, it’s effective implementation has started only in the last couple of years. Here a FSM-based reconfigurable architecture is proposed for K-means algorithm taking into consideration the real-time segmentation of satellite Images. Compared to other architectures in the literature, the proposed one has reduced hardware usage, lower area, power consumption with a satisfactory clock frequency, most importantly it is reconfigurable and can be used with the on-board circuitry within satellites. The testing is carried out on eight latest satellite images of natural and human devastations, taken from NASA’s OLI. The number of possible clusters within the geographical image is studied with the help of histogram analysis based on Otsu’s thresholding method. The performance analysis is carried out by computing the Peak Signal to Noise (PSNR) and Mean Structural Similarity Index (M-SSIM) of the resultant clusters. It is observed that by increasing the number of clusters, the detailed classes were effectively segmented and the quality of clustering is ensured by the improvement in PSNR and M-SSIM values. Comparison of the obtained clusters with their true land features also yielded satisfactory results.

Automated and real-time segmentation of suspicious breast masses using convolutional neural network.

In this work, a computer-aided tool for detection was developed to segment breast masses from clinical ultrasound (US) scans. The underlying Multi U-net algorithm is based on convolutional neural networks. Under the Mayo Clinic Institutional Review Board protocol, a prospective study of the automatic segmentation of suspicious breast masses was performed. The cohort consisted of 258 female patients who were clinically identified with suspicious breast masses and underwent clinical US scan and breast biopsy. The computer-aided detection tool effectively segmented the breast masses, achieving a mean Dice coefficient of 0.82, a true positive fraction (TPF) of 0.84, and a false positive fraction (FPF) of 0.01. By avoiding positioning of an initial seed, the algorithm is able to segment images in real time (13–55 ms per image), and can have potential clinical applications. The algorithm is at par with a conventional seeded algorithm, which had a mean Dice coefficient of 0.84 and performs significantly better (P< 0.0001) than the original U-net algorithm.

Color sensors and their applications based on real-time color image segmentation for cyber physical systems

Color information plays an important role in the color image segmentation and real-time color sensor, which affects the result of video image segmentation and correct real-time temperature value. In this paper, a novel real-time color image segmentation method is proposed, which is based on color similarity in RGB color space. According to the color and luminance information in RGB color space, the dominant color is determined at first, and then color similarity can be calculated with the proposed calculation method of color component, which creates a color-class map. Next, the information of the corresponding color-class map is utilized to classify the pixels. Due to the characteristic that thermal inks feature color values that change in real time as the temperature changes, the segmentation results of thermal ink can be used as a real-time color sensor. Then, we also propose a method of color correction and light source compensation for the sake of potential inaccuracy of its measures. We discuss the proposed segmentation method application combining with color sensor (thermal ink) in real-time color image segmentation for Cyber physical system (CPS) by the application in fire detection and summarize a new method in identifying fire in a video based on these characteristics. The experiments showed that the proposed method in vision-based fire detection and identification in videos was effective; the results were accurate and can be used in real-time analysis.

Toward a real-time system for temporal enhanced ultrasound-guided prostate biopsy.

We have previously proposed temporal enhanced ultrasound (TeUS) as a new paradigm for tissue characterization. TeUS is based on analyzing a sequence of ultrasound data with deep learning and has been demonstrated to be successful for detection of cancer in ultrasound-guided prostate biopsy. Our aim is to enable the dissemination of this technology to the community for large-scale clinical validation. In this paper, we present a unified software framework demonstrating near-real-time analysis of ultrasound data stream using a deep learning solution. The system integrates ultrasound imaging hardware, visualization and a deep learning back-end to build an accessible, flexible and robust platform. A client-server approach is used in order to run computationally expensive algorithms in parallel. We demonstrate the efficacy of the framework using two applications as case studies. First, we show that prostate cancer detection using near-real-time analysis of RF and B-mode TeUS data and deep learning is feasible. Second, we present real-time segmentation of ultrasound prostate data using an integrated deep learning solution. The system is evaluated for cancer detection accuracy on ultrasound data obtained from a large clinical study with 255 biopsy cores from 157 subjects. It is further assessed with an independent dataset with 21 biopsy targets from six subjects. In the first study, we achieve area under the curve, sensitivity, specificity and accuracy of 0.94, 0.77, 0.94 and 0.92, respectively, for the detection of prostate cancer. In the second study, we achieve an AUC of 0.85. Our results suggest that TeUS-guided biopsy can be potentially effective for the detection of prostate cancer.

Grey three index prediction method based on data assimilation preprocessing

In order to improve the effectiveness of analysis on human capital of migrant worker, a kind of analysis method on human capital of migrant worker based on the gray-level cubic exponential prediction preprocessed by data assimilation is proposed. Firstly, by analyzing the action mechanism of human capital of migrant workers to economic growth, a kind of analysis model on human capital of migrant workers with opening degree and government intervention is designed; secondly, in order to enhance the performance of Kalman filtering algorithm, combining with output features predicted in the model for human capital of migrant workers, the Kalman data assimilation algorithm with local set transformation is designed. And then, the way of division point is set to build the sliding window, and the cubic exponential smoothing algorithm is combined to make real-time segmentation for the data in grey model, so as to acquire the real-time statistic feature of data. Finally, by empirical analysis on human capital of Chinese migrant workers, the algorithm effectiveness is verified.

스테리오 비젼에 기반한 어류 표면적과 부피 측정을 위한 실시간 영상 분리와 3D 좌표의 결정

This paper proposes the real-time image segmentation and determination of 3D coordinates for fish surface area and volume measurements based on stereo vision. The segmentation process is performed in real-time on the video frame. Complexity and computation time are reduced by segmenting hue and value feature spaces separately before combining result. Morphological erosion and morphological dilation are applied to remove any object pixels that are clustered as background pixels. Finally, image segmentation is obtained in the binary image of an object. A step-by-step approach to computing a binary image is applied to obtain segmentation image. The object is considered a fish on a platform with a known background color, a known geometric shape for the fish, and the stereo camera is in fixed position to capture an image. The objective of this paper is to obtain a simple but accurate 3D image that is capable of capturing the 3D coordinates values of the fish object. The followings are done for this task: First, the stereo camera is calibrated to collect its intrinsic parameters and distortion parameters correctly. Second, an image is captured by the camera, that is segmented in real-time to distinguish between the object’s destination and its surroundings. Finally, the object coordinates, surface area, and volume of the fish are measured. Experiment results show that the proposed method has good results in real-time segmentation, and shows different surface areas and volumes within about 6% and 5.3%, respectively.

Construction of employee training program evaluation system of three exponential forecast based on sliding window

One construction method for employee training program evaluation system of three exponential forecast based on sliding window has been proposed to improve the effectiveness of construction process for employee training program evaluation system. Firstly, construct employee training program evaluation forecast model with gray model and set up regression forecast objective function; secondly, construct sliding window by setting segmentation points and make real-time segmentation for gray model data by combining three smoothness exponential algorithm, attain real-time statistical feature of data, construct the function relation between sequence error forecast and compression ratio, make use of error forecast sequence to make segmentation point judgment and improve robustness of test. Lastly, the effectiveness of the algorithm has been verified through simulation experiment.

Real-Time Segmentation and Feature Extraction of Electromyography: Towards Control of a Prosthetic Hand

Surface electromyographic (sEMG) signals can be used as inputs to control a myoelectric prosthetic hand. This requires the discrimination of sEMG associated with different hand movements. The signals for key unilateral hand movements such as wrist extension, wrist flexion, and power grip are similar, making the classification and control of these hand movements challenging. This preliminary study explores the viability of classifying the sEMG signals of these hand movements in real-time, in order to control a software simulation of a prosthetic hand. sEMG data was recorded from two bipolar electrodes for offline classifier training purposes. A novel segmentation technique was used to separate the muscle contraction and rest periods of the sEMG time-series data. A time-frequency algorithm was then applied for the first time to extract sEMG-based features from the segmented data. Features were used to train four support vector machines offline, in a one-versus-all architecture. The classification system was tested offline and in real-time. The system yielded accuracies of 89.39% and 84.93% for offline and real-time testing respectively. Real-time classification was done in 10.2 ms when processing 0.5 s of input sEMG data. This shows that the system is accurate and computationally efficient, even with limited electrodes. Thus, it can serve as a foundation for further work in the implemented segmentation and feature extraction methods, for an inexpensive alternative to commercial myoelectric prosthetic devices.

Nighttime Foreground Pedestrian Detection Based on Three-Dimensional Voxel Surface Model.

Pedestrian detection is among the most frequently-used preprocessing tasks in many surveillance application fields, from low-level people counting to high-level scene understanding. Even though many approaches perform well in the daytime with sufficient illumination, pedestrian detection at night is still a critical and challenging problem for video surveillance systems. To respond to this need, in this paper, we provide an affordable solution with a near-infrared stereo network camera, as well as a novel three-dimensional foreground pedestrian detection model. Specifically, instead of using an expensive thermal camera, we build a near-infrared stereo vision system with two calibrated network cameras and near-infrared lamps. The core of the system is a novel voxel surface model, which is able to estimate the dynamic changes of three-dimensional geometric information of the surveillance scene and to segment and locate foreground pedestrians in real time. A free update policy for unknown points is designed for model updating, and the extracted shadow of the pedestrian is adopted to remove foreground false alarms. To evaluate the performance of the proposed model, the system is deployed in several nighttime surveillance scenes. Experimental results demonstrate that our method is capable of nighttime pedestrian segmentation and detection in real time under heavy occlusion. In addition, the qualitative and quantitative comparison results show that our work outperforms classical background subtraction approaches and a recent RGB-D method, as well as achieving comparable performance with the state-of-the-art deep learning pedestrian detection method even with a much lower hardware cost.

Real-time segmentation of strawberry flesh and calyx from images of singulated strawberries during postharvest processing

This paper presents an image processing algorithm that automatically extracts the flesh and calyx areas from strawberry images. Images are captured by a camera included in a strawberry de-capping machine. Lighting is controlled and the background is known, conditions that are typical of postharvest processing. The goal is to extract as many flesh and calyx pixels as possible while rejecting any pixels belonging to the background. The proposed approach relies on image color segmentation in a two-dimensional color space, followed by a blob detection and selection stage. A set of 250 images is used to analyze the sensitivity of the algorithm with respect to user-defined parameters, and evaluate the performance of the approach. The algorithm appears to be easy to tune and allows accurate extraction of the areas of interest despite natural variation in strawberry shape and visual appearance. More than 98% of the flesh area was successfully extracted by the algorithm with less than 1% of the background pixels falsely included. Moreover, up to 79% of the calyx area could be extracted with less than 0.25% erroneous background pixels. Finally, the algorithm has been implemented using the C++ and Cuda languages and can be executed in real-time.

Real-time action detection and temporal segmentation in continuous video

ABSTRACTTemporal segmentation of actions has been under intensive focus in the field of computer vision for a prolonged period. The present study proposed a template-based framework to resolve the issues concerning timeliness and real-time performance in the temporal segmentation in a continuous video. A complete action can be detected, based on the previous frames, and the action can be segmented immediately without waiting for the follow-up frames. Herein, characteristic frames are selected by a martingale-based method, followed by the formation of the corresponding motion history through backtracking along the characteristic frames, and the final segmentation is determined according to the recognition model trained by the extreme learning machine. In the experiment on the IXMAS database, the average rate of the detection of action reached 91%, and the accuracy in the frame level reached 83.5%. In the experiment on the 3D skeleton data based on Kinect, the detection rate reached 94%.

EDTriangles: a high-speed triangle detection algorithm with a false detection control

We present a high-speed method for triangular object detection. The proposed method utilizes the recently developed, real-time edge segment detection algorithm, Edge Drawing; hence, the name EDTriangles, which consists of a detection stage and a validation stage. In the detection stage, EDTriangles extracts edge segments from the image using Edge Drawing and converts these edge segments into line segments, which are then converted into line pairs according to the angles between the line segments and the distance between their endpoints. Next, the line pairs are combined together using some heuristics to generate many triangle candidates, some of which are valid detections and some invalid. Finally, in the validation stage the candidate triangles are validated using the Helmholtz principle and number of false alarms computation to eliminate false detections. Experimental results show that EDTriangles runs very fast, detects various types of triangular objects ranging from narrow to wide-angled triangles and offers a higher detection performance compared to some of the well-known triangle detection algorithms found in the literature.

SoC Hardware Implementation of Real-Time Video Segmentation based on the Mixture of Gaussian Algorithm

Video segmentation based on the Mixture of Gaussian (MoG) algorithm is widely used in video processing systems, and hardware implementations have been proposed in the past years. Most previous work focused on high-performance custom design of the MoG algorithm to meet real-time requirement of high-frame-rate high-resolution video segmentation tasks. This paper focuses on the System-on-Chip (SoC) design and the priority is SoC integration of the system for flexibility/adaptability, while at the same time, custom design of the original MoG algorithm is included. To maximally retain the accuracy of the MoG algorithm for best segmentation performance, we minimally modified the MoG algorithm for hardware implementation at the cost of hardware resources. The MoG algorithm is custom-implemented as a hardware IP (Intellectual Property), which is then integrated within an SoC platform together with other video processing components, so that some key control parameters can be configured on-line, which makes the video segmentation system most suitable for different scenarios. The proposed implementation has been demonstrated and tested on a Xilinx Spartan-3A DSP Video Starter Board. Experiment results show that under a clock frequency of 25 MHz, this design meets the real-time requirement for VGA resolution (640 × 480) at 30 fps (frame-per-second).

A Dual Image Sensor Approach for Automated, High Resolution, Region-of-Interest Imaging in a 96-Well Plate

Phenotypic screens are performed on model organisms to understand how chemical compounds affect biological systems. High-throughput devices are needed to efficiently gather large volumes of data for these screens. Recently we developed a low cost high-throughput imaging solution to capture the effects of chemicals on the egg-laying behaviour of C. elegans. We present a second-generation prototype capable of high-resolution (0.2 µm/px) imaging of individual worms and their internal structures. It features an automatic real time segmentation of C. elegans worms and is compatible with 96-well plates. In order to maximize efficiency in data collection, we developed an approach for simultaneous dual imaging through a single objective. A low-resolution (15 µm/px) arm continuously collects images of an entire well using a CCD camera (1280 px by 1024 px). The images are continuously fed into a Python image algorithm that tracks the movement of the worms in regions of interest (ROIs). Using this information, high resolution images of the worms are captured using a linescan camera (8000 px). This solution provides automated high-resolution imaging minimizing acquisition time by eliminating the need for multiple objectives, allowing image collection with a continuously moving system (high speed linescan camera) and minimizing the amount of stitching (large linescan sensor). The scan time of a 96-well plate is proportional to the number of ROIs, which is significantly less than imaging an entire plate. The high-resolution images can be fed into an image processing pipeline that will be able to classify the effects of chemicals on the internal structure of the worms. Simultaneous acquisition of high and low resolution data can be used to correlate deformations of internal structure with changes in mobility. This low cost solution provides users with an automated approach to obtain high-resolution images and mobility statistics of C. elegans.

A New Method for Automatic Vehicle License Plate Detection

License plate recognition has been widely studied, and the advance in image capture technology helps enhance or create new methods to achieve this objective. In this work is presented a method for real time detection and segmentation of car license plates based on image analyzing and processing techniques. The results show that the computational cost and accuracy rate considering the proposed approach are acceptable to real time applications, with an execution time under 1 second. The proposed method was validated using two datasets (A and B). It was obtained over 92% detection success for dataset A, 88% in digit segmentation for datasets A and B, and 95% digits classification accuracy rate for dataset B.

An intelligent system for mineral identification in thin sections based on a cascade approach

In this study, an intelligent system for mineral identification in thin sections is proposed based on RGB and HSI color spaces and texture features in plane and cross polarized light. The proposed system has two phases for mineral identification. In phase#1, which is the segmentation phase, 12 color components are extracted for each pixel, and using an incremental clustering algorithm, several mineral clusters including index mineral are produced. Afterwards, in phase#2 which is the identification phase, the produced mineral clusters are identified based on a cascade classification approach. The first level of the cascade includes a set of artificial neural networks (ANNs) corresponding to the number of input minerals which are trained based on color components. In the first level, those minerals exhibiting different colors in plane or cross polarized light are identified. The second level of the cascade includes one ANN which is trained based on texture features in plane and cross polarized light images. In the second level, those minerals which are indistinguishable based on color components in both plane and cross polarized light are identified (are rejected in the first level of the cascade). The final output of the system is the name and number of minerals, boundary and percentage of each mineral in thin section, and eventually the name of probable target rock. The proposed system is able to recognize 23 test igneous minerals with the overall accuracy of 93.81%. The proposed system can be applied in important applications which require a real time segmentation and identification map such as petrography, and NASA Mars Explorations.

Fast Threshold Image Segmentation Based on 2D Fuzzy Fisher and Random Local Optimized QPSO.

In the paper, a real-time segmentation method that separates the target signal from the navigation image is proposed. In the approaching docking stage, the navigation image is composed of target and non-target signal, which are separately bright spot and space vehicle itself. Since the non-target signals is the main part of the navigation image, the traditional entropy-related criterions and Ostu-related criterions will bring inadequate segmentation, while the mere 2D Fisher criterion will causes over-segmentation, all the methods show their shortages in dealing with this kind of case. To guarantee a precise image segmentation, a revised 2D fuzzy Fisher is proposed in the paper to make a trade-off between positioning target regions and retaining target fuzzy boundaries. First, to reduce redundant computations in finding the threshold pair, a 2D fuzzy Fisher criterion-based integral image is established by way of simplifying the corresponding fuzzy domains. Then, to quicken the convergence, a random orthogonal component is added in its quasi-optimum particle to enhance its local searching capacity in each iteration. Experimental results show its competence of quick segmentation.