Real-time Segmentation Research Articles

Real-time echosounder based acoustic seabed segmentation with two first echo parameters

Seabeds with different physical and biological properties (for example, mud, sand, gravel, rock, seagrass, shell beds) generally produce differently shaped echoes in response to echosounder pings. After allowance for propagation losses and for artefacts caused by sampling at equal time intervals, instead of equi-angular spacing, geometrical (i.e. shape based) estimates of the similarity of echo envelopes may be used to geographically segment the seabed into areas with similar acoustic responses. Post-processing of entire echoes by direct clustering provides a statistically and geometrically optimal method of segmentation which in itself does not need ground truthing, since it is based on the actual seabed acoustic response. However, real-time processing allows sampling strategies to be adapted in response to findings during survey. A real-time segmentation method is presented which uses only two simple echo parameters. Classes formed by direct clustering of actual echoes map to unique areas of the 2-parameter space in a very simple and regular fashion, fully validating its use as a segmentation scheme. Groundtruth (seabed samples or video) must be taken if descriptions (labels) are to be assigned to the 2-parameter segmentations (the segmentation becomes a classification). However, the relation to direct clustering results means that in principle groundtruth is not needed to validate the segmentations. In contrast it has never been shown that echo features used by existing real-time schemes contain sufficient information to adequately characterize seabeds.

Real-time segmentation and recognition of surgical tasks in cataract surgery videos.

In ophthalmology, it is now common practice to record every surgical procedure and to archive the resulting videos for documentation purposes. In this paper, we present a solution to automatically segment and categorize surgical tasks in real-time during the surgery, using the video recording. The goal would be to communicate information to the surgeon in due time, such as recommendations to the less experienced surgeons. The proposed solution relies on the content-based video retrieval paradigm: it reuses previously archived videos to automatically analyze the current surgery, by analogy reasoning. Each video is segmented, in real-time, into an alternating sequence of idle phases, during which no clinically-relevant motions are visible, and action phases. As soon as an idle phase is detected, the previous action phase is categorized and the next action phase is predicted. A conditional random field is used for categorization and prediction. The proposed system was applied to the automatic segmentation and categorization of cataract surgery tasks. A dataset of 186 surgeries, performed by ten different surgeons, was manually annotated: ten possibly overlapping surgical tasks were delimited in each surgery. Using the content of action phases and the duration of idle phases as sources of evidence, an average recognition performance of Az = 0.832 ± 0.070 was achieved.

Adequation and hardware implementation of the color structure descriptor for real-time temporal video segmentation

In this paper, we focused on the study of the MPEG-7 color structure descriptor (CSD) for real-time temporal video segmentation. A new hardware architecture of the CSD was proposed. In the aim of optimizing this implementation in terms of hardware resources and execution time, different algorithm transformations have been tested for the considered application. The CSD was applied for different quantization levels in the HMMD color space and for different grey levels, with and without a frame skip. We have demonstrated that the use of a low number of quantization levels and a frame skip can significantly reduce the complexity and assure a better computing performance while preserving a satisfactory level of accuracy in terms of shot boundary detection rate. This is useful for implementation on resource-constrained hardware platform and multiprocessing applications. The performances of the proposed architecture were evaluated for different quantization levels to show the effect on occupied hardware resources and execution time. The comparative study demonstrates the effectiveness of the proposed architecture, which can operate in real-time video and without restriction on image size. In this work we have also designed a system on chip (SOC) for real-time video summarization based on the CSD. The proposed SOC integrating the CSD module was developed using a platform based on a Xilinx Virtex5 FPGA. A complete demonstration, including CSD extraction, shot boundary detection and key-frames visualization, was realized.

Structured Time Series Analysis for Human Action Segmentation and Recognition.

We address the problem of structure learning of human motion in order to recognize actions from a continuous monocular motion sequence of an arbitrary person from an arbitrary viewpoint. Human motion sequences are represented by multivariate time series in the joint-trajectories space. Under this structured time series framework, we first propose Kernelized Temporal Cut (KTC), an extension of previous works on change-point detection by incorporating Hilbert space embedding of distributions, to handle the nonparametric and high dimensionality issues of human motions. Experimental results demonstrate the effectiveness of our approach, which yields realtime segmentation, and produces high action segmentation accuracy. Second, a spatio-temporal manifold framework is proposed to model the latent structure of time series data. Then an efficient spatio-temporal alignment algorithm Dynamic Manifold Warping (DMW) is proposed for multivariate time series to calculate motion similarity between action sequences (segments). Furthermore, by combining the temporal segmentation algorithm and the alignment algorithm, online human action recognition can be performed by associating a few labeled examples from motion capture data. The results on human motion capture data and 3D depth sensor data demonstrate the effectiveness of the proposed approach in automatically segmenting and recognizing motion sequences, and its ability to handle noisy and partially occluded data, in the transfer learning module.

Refinement-cut: user-guided segmentation algorithm for translational science.

In this contribution, a semi-automatic segmentation algorithm for (medical) image analysis is presented. More precise, the approach belongs to the category of interactive contouring algorithms, which provide real-time feedback of the segmentation result. However, even with interactive real-time contouring approaches there are always cases where the user cannot find a satisfying segmentation, e.g. due to homogeneous appearances between the object and the background, or noise inside the object. For these difficult cases the algorithm still needs additional user support. However, this additional user support should be intuitive and rapid integrated into the segmentation process, without breaking the interactive real-time segmentation feedback. I propose a solution where the user can support the algorithm by an easy and fast placement of one or more seed points to guide the algorithm to a satisfying segmentation result also in difficult cases. These additional seed(s) restrict(s) the calculation of the segmentation for the algorithm, but at the same time, still enable to continue with the interactive real-time feedback segmentation. For a practical and genuine application in translational science, the approach has been tested on medical data from the clinical routine in 2D and 3D.

Real-time segmentation of multiple implanted cylindrical liver markers in kilovoltage and megavoltage x-ray images

Gold markers implanted in or near a tumor can be used as x-ray visible landmarks for image based tumor localization. The aim of this study was to develop and demonstrate fast and reliable real-time segmentation of multiple liver tumor markers in intra-treatment kV and MV images and in cone-beam CT (CBCT) projections, for real-time motion management. Thirteen patients treated with conformal stereotactic body radiation therapy in three fractions had 2–3 cylindrical gold markers implanted in the liver prior to treatment. At each fraction, the projection images of a pre-treatment CBCT scan were used for automatic generation of a 3D marker model that consisted of the size, orientation, and estimated 3D trajectory of each marker during the CBCT scan. The 3D marker model was used for real-time template based segmentation in subsequent x-ray images by projecting each marker's 3D shape and likely 3D motion range onto the imager plane. The segmentation was performed in intra-treatment kV images (526 marker traces, 92 097 marker projections) and MV images (88 marker traces, 22 382 marker projections), and in post-treatment CBCT projections (42 CBCT scans, 71 381 marker projections). 227 kV marker traces with low mean contrast-to-noise ratio were excluded as markers were not visible due to MV scatter. Online segmentation times measured for a limited dataset were used for estimating real-time segmentation times for all images. The percentage of detected markers was 94.8% (kV), 96.1% (MV), and 98.6% (CBCT). For the detected markers, the real-time segmentation was erroneous in 0.2–0.31% of the cases. The mean segmentation time per marker was 5.6 ms [2.1–12 ms] (kV), 5.5 ms [1.6–13 ms] (MV), and 6.5 ms [1.8–15 ms] (CBCT). Fast and reliable real-time segmentation of multiple liver tumor markers in intra-treatment kV and MV images and in CBCT projections was demonstrated for a large dataset.

Traversable Ground Surface Segmentation and Modeling for Real-Time Mobile Mapping

Remote vehicle operator must quickly decide on the motion and path. Thus, rapid and intuitive feedback of the real environment is vital for effective control. This paper presents a real-time traversable ground surface segmentation and intuitive representation system for remote operation of mobile robot. Firstly, a terrain model using voxel-based flag map is proposed for incrementally registering large-scale point clouds in real time. Subsequently, a ground segmentation method with Gibbs-Markov random field (Gibbs-MRF) model is applied to detect ground data in the reconstructed terrain. Finally, we generate a texture mesh for ground surface representation by mapping the triangles in the terrain mesh onto the captured video images. To speed up the computation, we program a graphics processing unit (GPU) to implement the proposed system for large-scale datasets in parallel. Our proposed methods were tested in an outdoor environment. The results show that ground data is segmented effectively and the ground surface is represented intuitively.

Multi-Bernoulli sample consensus for simultaneous robust fitting of multiple structures in machine vision

In many image processing applications, such as parametric range and motion segmentation, multiple instances of a model are fitted to data points. The most common robust fitting method, RANSAC, and its extensions are normally devised to segment the structures sequentially, treating the points belonging to other structures as outliers. Thus, the ratio of inliers is small and successful fitting requires a very large number of random samples, incurring cumbrous computation. This paper presents a new method to simultaneously fit multiple structures to data points in a single run. We model the parameters of multiple structures as a random finite set with multi-Bernoulli distribution. Simultaneous search for all structure parameters is performed by Bayesian update of the multi-Bernoulli parameters. Experiments involving segmentation of numerous structures show that our method outperforms well-known methods in terms of estimation error and computational cost. The fast convergence and high accuracy of our method make it an excellent choice for real-time estimation and segmentation of multiple structures in image processing applications.

ASIC and FPGA Implementation of the Gaussian Mixture Model Algorithm for Real-Time Segmentation of High Definition Video

Background identification is a common feature in many video processing systems. This paper proposes two hardware implementations of the OpenCV version of the Gaussian mixture model (GMM), a background identification algorithm. The implemented version of the algorithm allows a fast initialization of the background model while an innovative, hardware-oriented, formulation of the GMM equations makes the proposed circuits able to perform real-time background identification on high definition (HD) video sequences with frame size 1920 × 1080. The first of the two circuits is designed with commercial field-programmable gate-array (FPGA) devices as target. When implemented on Virtex6 vlx75t, the proposed circuit process 91 HD fps (frames per second) and uses 3% of FPGA logic resources. The second circuit is oriented to the implementation in UMC-90 nm CMOS standard cell technology, and is proposed in two versions. Both versions can process at a frame rate higher than 60 HD fps. The first version uses the constant voltage scaling technique to provide a low power implementation. It provides silicon area occupation of 28847 μm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> and energy dissipation per pixel of 15.3 pJ/pixel. The second version is designed to reduce silicon area utilization and occupies 21847 μm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> with an energy dissipation of 49.4 pJ/pixel.

Adaptive real-time motion segmentation technique based on statistical background model

Motion segmentation is a crucial step for video analysis and has many applications. This paper proposes a method for motion segmentation, which is based on construction of statistical background model. Variance and Covariance of pixels are computed to construct the model for scene background. We perform average frame differencing with this model to extract the objects of interest from the video frames. Morphological operations are used to smooth the object segmentation results. The proposed technique is adaptive to the dynamically changing background because of change in the lighting conditions and in scene background. The method has the capability to relearn the background to adapt these variations. The immediate advantage of the proposed method is its high processing speed of 30 frames per second on large sized (high resolution) videos. We compared the proposed method with other five popular methods of object segmentation in order to prove the effectiveness of the proposed technique. Experimental results demonstrate the novelty of the proposed method in terms of various performance parameters. The method can segment the video stream in real-time, when background changes, lighting conditions vary, and even in the presence of clutter and occlusion

Sloped terrain segmentation for autonomous drive using sparse 3D point cloud.

A ubiquitous environment for road travel that uses wireless networks requires the minimization of data exchange between vehicles. An algorithm that can segment the ground in real time is necessary to obtain location data between vehicles simultaneously executing autonomous drive. This paper proposes a framework for segmenting the ground in real time using a sparse three-dimensional (3D) point cloud acquired from undulating terrain. A sparse 3D point cloud can be acquired by scanning the geography using light detection and ranging (LiDAR) sensors. For efficient ground segmentation, 3D point clouds are quantized in units of volume pixels (voxels) and overlapping data is eliminated. We reduce nonoverlapping voxels to two dimensions by implementing a lowermost heightmap. The ground area is determined on the basis of the number of voxels in each voxel group. We execute ground segmentation in real time by proposing an approach to minimize the comparison between neighboring voxels. Furthermore, we experimentally verify that ground segmentation can be executed at about 19.31 ms per frame.

Quality Assessment of Preclassification Maps Generated From Spaceborne/Airborne Multispectral Images by the \textit{Satellite Image Automatic Mapper} and \textit{Atmospheric/Topographic Correction-Spectral Classification} Software Products: Part 1—Theory

In compliance with the Quality Assurance Framework for Earth Observation (QA4EO) guidelines, the goal of this paper is to provide a theoretical comparison and an experimental quality assessment of two operational (ready-for-use) expert systems (prior knowledge-based nonadaptive decision trees) for automatic near real-time preattentional classification and segmentation of spaceborne/airborne multispectral (MS) images: the Satellite Image Automatic Mapper™ (SIAM™) software product and the Spectral Classification of surface reflectance signatures (SPECL) secondary product of the Atmospheric/Topographic Correction™ (ATCOR™) commercial software toolbox. For the sake of simplicity, this paper is split into two: Part 1—Theory, presented herein, and Part 2—Experimental results, already published elsewhere. The main theoretical contribution of the present Part 1 is threefold. First, it provides the published Part 2 with an interdisciplinary terminology and a theoretical background encompassing multiple disciplines, such as philosophical hermeneutics, machine learning, artificial intelligence, computer vision, human vision, and remote sensing (RS). Second, it highlights the several degrees of novelty of the ATCOR-SPECL and SIAM deductive preliminary classifiers (preclassifiers) at the four levels of abstraction of an information processing system, namely, system design, knowledge/information representation, algorithms, and implementation. Third, the present Part 1 requires the experimental Part 2 to collect a minimum set of complementary statistically independent metrological quality indicators (QIs) of operativeness (QIOs), in compliance with the QA4EO guidelines and the principles of statistics. In particular, sample QIs are required to be: 1) statistically significant, i.e., provided with a degree of uncertainty in measurement; and 2) statistically valid (consistent), i.e., representative of the entire population being sampled, which requires the implementation of a probability sampling protocol. Largely overlooked by the RS community, these sample QI requirements are almost never satisfied in the RS common practice. As a consequence, to date, QIOs of existing RS image understanding systems (RS-IUSs), including thematic map accuracy, remain largely unknown in statistical terms. The conclusion of the present Part 1 is that the proposed comparison of the two alternative ATCOR-SPECL and SIAM prior knowledge-based preclassifiers in operating mode, accomplished in the Part 2, can be considered appropriate, well-timed, and of potential interest to a large portion of the RS readership.

An improved particle swarm optimisation for video image segmentation

Video image segmentation is the key step of objects classification and recognition. Threshold-based image segmentation algorithms, especially those based on the two-dimensional maximum entropy, have been studied by many scholars now. Although there are many algorithms that can get the entropy, they have their own weaknesses. For instances, the exhaustive search algorithm is time-consuming to implement the real-time image segmentation, the convergence rate of the traditional genetic algorithm is relatively slow, the standard particle swarm algorithm is prone to premature and difficult to get the optimal solution in the high-dimensional data space. In this paper, we proposed an improved particle swarm optimisation algorithm based on the law of universal gravitation to obtain the optimal entropy. Experimental results show that the improved algorithm is robust and obtain fast convergence rate.

Real-time 3D curved needle segmentation using combined B-mode and power Doppler ultrasound.

This paper presents a real-time segmentation method for curved needles in biological tissue based on analysis of B-mode and power Doppler images from a tracked 2D ultrasound transducer. Mechanical vibration induced by an external voice coil results in a Doppler response along the needle shaft, which is centered around the needle section in the ultrasound image. First, B-mode image analysis is performed within regions of interest indicated by the Doppler response to create a segmentation of the needle section in the ultrasound image. Next, each needle section is decomposed into a sequence of points and transformed into a global coordinate system using the tracked transducer pose. Finally, the 3D shape is reconstructed from these points. The results of this method differ from manual segmentation by 0.71 ± 0.55 mm in needle tip location and 0.38 ± 0.27 mm along the needle shaft. This method is also fast, taking 5-10 ms to run on a standard PC, and is particularly advantageous in robotic needle steering, which involves thin, curved needles with poor echogenicity.

Real-time segmentation for baggage tracking on a cost effective embedded platform

This paper describes segmentation and tracking parts of a machine vision based airport baggage tracking system. A simplified codebook based background subtraction method is used to segment the bag from a semi-static background. Morphological processing using an integral image is used to filter the foreground mask and the bag location is found using statistical methods. The system was implemented on a cost effective embedded processor and runs in real time at 30 fps. Five ARM based embedded platforms are evaluated and it is shown that all of them are capable of the required performance.

Background subtraction using finite mixtures of asymmetric Gaussian distributions and shadow detection

Foreground segmentation of moving regions in image sequences is a fundamental step in many vision systems including automated video surveillance, human-machine interface, and optical motion capture. Many models have been introduced to deal with the problems of modeling the background and detecting the moving objects in the scene. One of the successful solutions to these problems is the use of the well-known adaptive Gaussian mixture model. However, this method suffers from some drawbacks. Modeling the background using the Gaussian mixture implies the assumption that the background and foreground distributions are Gaussians which is not always the case for most environments. In addition, it is unable to distinguish between moving shadows and moving objects. In this paper, we try to overcome these problem using a mixture of asymmetric Gaussians to enhance the robustness and flexibility of mixture modeling, and a shadow detection scheme to remove unwanted shadows from the scene. Furthermore, we apply this method to real image sequences of both indoor and outdoor scenes. The results of comparing our method to different state of the art background subtraction methods show the efficiency of our model for real-time segmentation.

Real-time 3D interactive segmentation of echocardiographic data through user-based deformation of B-spline explicit active surfaces

Image segmentation is an ubiquitous task in medical image analysis, which is required to estimate morphological or functional properties of given anatomical targets. While automatic processing is highly desirable, image segmentation remains to date a supervised process in daily clinical practice. Indeed, challenging data often requires user interaction to capture the required level of anatomical detail. To optimize the analysis of 3D images, the user should be able to efficiently interact with the result of any segmentation algorithm to correct any possible disagreement. Building on a previously developed real-time 3D segmentation algorithm, we propose in the present work an extension towards an interactive application where user information can be used online to steer the segmentation result. This enables a synergistic collaboration between the operator and the underlying segmentation algorithm, thus contributing to higher segmentation accuracy, while keeping total analysis time competitive. To this end, we formalize the user interaction paradigm using a geometrical approach, where the user input is mapped to a non-cartesian space while this information is used to drive the boundary towards the position provided by the user. Additionally, we propose a shape regularization term which improves the interaction with the segmented surface, thereby making the interactive segmentation process less cumbersome. The resulting algorithm offers competitive performance both in terms of segmentation accuracy, as well as in terms of total analysis time. This contributes to a more efficient use of the existing segmentation tools in daily clinical practice. Furthermore, it compares favorably to state-of-the-art interactive segmentation software based on a 3D livewire-based algorithm.

Performance of dynamic texture segmentation using GPU

This work is focused on the assessment of the use of GPU computation in dynamic texture segmentation under the mixture of dynamic textures (MDT) model. In this generative video model, the observed texture is a time-varying process commanded by a hidden state process. The use of mixtures in this model allows simultaneously handling of different visual processes. Nowadays, the use of GPU computing is growing in high-performance applications, but the adaptation of existing algorithms in such a way as to obtain a benefit from its use is not an easy task. In this paper, we made two implementations, one in CPU and the other in GPU, of a known segmentation algorithm based on MDT. In the MDT algorithm, there is a matrix inversion process that is highly demanding in terms of computing power. We make a comparison between the gain in performance obtained by porting to GPU this matrix inversion process and the gain obtained by porting to GPU the whole MDT segmentation process. We also study real-time motion segmentation performance by separating the learning part of the algorithm from the segmentation part, leaving the learning stage as an off-line process and keeping the segmentation as an online process. The results of performance analyses allow us to decide the cases in which the full GPU implementation of the motion segmentation process is worthwhile.

Real-time segmentation of burst suppression patterns in critical care EEG monitoring

ObjectiveDevelop a real-time algorithm to automatically discriminate suppressions from non-suppressions (bursts) in electroencephalograms of critically ill adult patients. MethodsA real-time method for segmenting adult ICU EEG data into bursts and suppressions is presented based on thresholding local voltage variance. Results are validated against manual segmentations by two experienced human electroencephalographers. We compare inter-rater agreement between manual EEG segmentations by experts with inter-rater agreement between human vs automatic segmentations, and investigate the robustness of segmentation quality to variations in algorithm parameter settings. We further compare the results of using these segmentations as input for calculating the burst suppression probability (BSP), a continuous measure of depth-of-suppression. ResultsAutomated segmentation was comparable to manual segmentation, i.e. algorithm-vs-human agreement was comparable to human-vs-human agreement, as judged by comparing raw EEG segmentations or the derived BSP signals. Results were robust to modest variations in algorithm parameter settings. ConclusionsOur automated method satisfactorily segments burst suppression data across a wide range adult ICU EEG patterns. Performance is comparable to or exceeds that of manual segmentation by human electroencephalographers. SignificanceAutomated segmentation of burst suppression EEG patterns is an essential component of quantitative brain activity monitoring in critically ill and anesthetized adults. The segmentations produced by our algorithm provide a basis for accurate tracking of suppression depth.

Real-time segmentation and tracking of brain metabolic state in ICU EEG recordings of burst suppression.

We provide a method for estimating brain metabolic state based on a reduced-order model of EEG burst suppression. The model, derived from previously suggested biophysical mechanisms of burst suppression, describes important electrophysiological features and provides a direct link to cerebral metabolic rate. We design and fit the estimation method from EEG recordings of burst suppression from a neurological intensive care unit and test it on real and synthetic data.