Estimated Motion Information Research Articles

Self-supervised human semantic parsing for video-based person re-identification

Video-based person re-identification is an important research topic in computer vision that entails associating a pedestrian’s identity with non-overlapping cameras. It suffers from severe temporal appearance misalignment and visual ambiguity problems. We propose a novel self-supervised human semantic parsing approach (SS-HSP) for video-based person re-identification in this work. It employs self-supervised learning to adaptively segment the human body at pixel-level by estimating motion information of each body part between consecutive frames and explores complementary temporal relations for pursuing reinforced appearance and motion representations. Specifically, a semantic segmentation network within SS-HSP is designed, which exploits self-supervised learning by constructing a pretext task of predicting future frames. The network learns precise human semantic parsing together with the motion field of each body part between consecutive frames, which permits the reconstruction of future frames with the aid of several customized loss functions. Local aligned features of body parts are obtained according to the estimated human parsing. Moreover, an aggregation network is proposed to explore the correlation information across video frames for refining the appearance and motion representations. Extensive experiments on two video datasets have demonstrated the effectiveness of the proposed approach.

Motion estimation in hazy videos

• A novel end-to-end deep network is proposed for haze removal from hazy video frames. • Failure of conventional approaches in estimating motion information in hazy videos has been analyzed. • A novel haze removal → motion estimation approach is proposed for motion estimation from hazy videos. Motion estimation is the basic need for the success of many video analysis algorithms such as moving object detection, human activity recognition, etc. Most of the motion estimation algorithms are prone to weather conditions and thus, they fail to estimate the motion in degraded weather. Severe weather situations like snow, rain, haze, smog, etc., degrades the performance and reliability of video analysis algorithms. In this paper, we have analyzed the effect of the haze on motion estimation in hazy videos. We propose a cascaded architecture i.e. haze removal followed by optical flow for motion estimation in hazy videos. The proposed image de-hazing network is build upon the Res idual and I nception module concepts and named as ResINet. Further, an optical flow is utilized to estimate the motion information. We have carried out the visual analysis to validate the proposed approach for motion estimation in hazy videos. Also, to validate the proposed ResINet for de-hazing, we carried out the quantitative analysis on two benchmark image de-hazing datasets.

Autonomous collision detection and avoidance for ARAGON USV: Development and field tests

AbstractThis study addresses the development of algorithms for multiple target detection and tracking in the framework of sensor fusion and its application to autonomous navigation and collision avoidance systems for the unmanned surface vehicle (USV) Aragon. To provide autonomous navigation capabilities, various perception sensors such as radar, lidar, and cameras have been mounted on the USV platform and automatic ship detection algorithms are applied to the sensor measurements. The relative position information between the USV and nearby objects is obtained to estimate the motion of the target objects in a sensor‐level tracking filter. The estimated motion information from the individual tracking filters is then combined in a central‐level fusion tracker to achieve persistent and reliable target tracking performance. For automatic ship collision avoidance, the combined track data are used as obstacle information, and appropriate collision avoidance maneuvers are designed and executed in accordance with the international regulations for preventing collisions at sea (COLREGs). In this paper, the development processes of the vehicle platform and the autonomous navigation algorithms are described, and the results of field experiments are presented and discussed.

Superpixel Labeling Priors and MRF for Aerial Video Segmentation

Video segmentation is a task of partitioning pixels that exhibit homogeneous appearance and motion into coherent spatial-temporal groups, which is still challenging for aerial applications. In this paper, a principled combination of superpixel labeling priors and Markov random field (S-MRF) is proposed for aerial video segmentation. The proposed approach has several contributions: 1) we develop a metadata-based global projection model with coordinate transformation to estimate motion information between frames; 2) the superpixel labeling priors from previous frames are incorporated into the segmentation of the current frame, leading to a highly efficient probabilistic label propagation algorithm; and 3) we perform an MRF optimization on the initial segments with propagated labeling priors to improve the temporal coherency. In addition, a new video dataset is collected and will be made publicly available to evaluate the performance of aerial video segmentation algorithms. The experimental results show that the proposed approach outperforms the state-of-the-art video segmentation methods.

COLREGS‐compliant path planning considering time‐varying trajectory uncertainty of autonomous surface vehicle

A path planning approach is proposed that is in accordance with Collision Regulations (COLREGS) for avoidance of a potential collision and considers the time-varying trajectory uncertainties of an autonomous surface vehicle (ASV) and an obstacle. Unfortunately, as the motion information for each object includes various uncertainties caused by characteristics of the navigation and perception systems, the trajectory uncertainties increase linearly during the pre-specified time in a trajectory prediction process. This trajectory uncertainty is preferentially modelled using the error covariance of a tracking filter designed for estimating motion information. Subsequently, collision risk is computed using a probabilistic method, and a collision risk zone (CRZ) is configured on the predicted trajectory. Finally, a COLREGS-compliant path is planned with consideration of the dynamic characteristics of the ASV to avoid the CRZ. Simulation results that demonstrate the feasibility of the proposed approach with marine traffic examples are presented and discussed.

Head motion correction based on filtered backprojection for x-ray CT imaging.

For head x-ray CT imaging, the head needs to remain motionless during the scan. In clinical practice, however, head motion is sometimes unavoidable depending on the patient. The motion can occur abruptly during the scan and can be unpredictable. It thereby causes motion artifacts such as tissue blurring or doubled edges around the skull area. To mitigate this problem, we propose a 3D head motion estimation (ME) and compensation algorithm based on filtered backprojection. If a patient moves his or her head during the scan, a motion-corrupted sinogram is obtained. Modeling the head motion as a 3D rigid transformation, we develop a motion-compensated (MC) reconstruction algorithm based on the FDK algorithm. To determine the head motion of a rigid transformation, we propose two optimization-based ME schemes depending on the degree of head motion, both of which are performed by updating motion parameters and the corresponding MC reconstructed image alternatively until the proposed cost function is minimized for the MC reconstructed image. In particular, to improve the robustness in the case of large motion, we propose attaching a fiducial marker to the head so that more reliable motion parameters can be initialized by determining the marker position, before the optimization. To evaluate the proposed algorithm, a numerical phantom with realistic, continuous, and smoothly varying motion, and a moving physical phantom are used with a gantry rotation time of 1s. In the simulation using a numerical phantom and in the experiment using a physical phantom, the proposed algorithm provides well-restored 3D motion-compensated images in both cases of small and large motion. In particular, in the case of large motion of the physical phantom, using a fiducial marker, we obtain remarkable improvement of image quality in cerebral arteries and a lesion as well as the skull. Quantitative evaluations using the image sharpness and root-mean-square error also show noticeable improvement of image quality in both simulations and experiments. We propose a framework for head motion correction in an axial CT scan, which consists of motion estimation and compensation steps. Two image-based ME algorithms for rigid motion tracking are developed according to the degree of head motion. The estimated motion information is then used for MC image reconstruction. Both motion estimation and compensation algorithms are based on computationally efficient filtered backprojection. Excellent performance of the proposed framework is illustrated by means of simulations using a numerical phantom and experiments using a physical phantom.

Adaptive Kalman Filter Applied to Vision Based Head Gesture Tracking for Playing Video Games

This paper proposes an adaptive Kalman filter (AKF) to improve the performance of a vision-based human machine interface (HMI) applied to a video game. The HMI identifies head gestures and decodes them into corresponding commands. Face detection and feature tracking algorithms are used to detect optical flow produced by head gestures. Such approaches often fail due to changes in head posture, occlusion and varying illumination. The adaptive Kalman filter is applied to estimate motion information and reduce the effect of missing frames in a real-time application. Failure in head gesture tracking eventually leads to malfunctioning game control, reducing the scores achieved, so the performance of the proposed vision-based HMI is examined using a game scoring mechanism. The experimental results show that the proposed interface has a good response time, and the adaptive Kalman filter improves the game scores by ten percent.

An Efficient Motion Detection and Tracking Scheme for Encrypted Surveillance Videos

Performing detection on surveillance videos contributes significantly to the goals of safety and security. However, performing detection on unprotected surveillance video may reveal the privacy of innocent people in the video. Therefore, striking a proper balance between maintaining personal privacy while enhancing the feasibility of detection is an important issue. One promising solution to this problem is to encrypt the surveillance videos and perform detection on the encrypted videos. Most existing encrypted signal processing methods focus on still images or small data volumes; however, because videos are typically much larger, investigating how to process encrypted videos is a significant challenge. In this article, we propose an efficient motion detection and tracking scheme for encrypted H.264/AVC video bitstreams, which does not require the previous decryption on the encrypted video. The main idea is to first estimate motion information from the bitstream structure and codeword length and, then, propose a region update (RU) algorithm to deal with the loss and error drifting of motion caused by the video encryption. The RU algorithm is designed based on the prior knowledge that the object motion in the video is continuous in space and time. Compared to the existing scheme, which is based on video encryption that occurs at the pixel level, the proposed scheme has the advantages of requiring only a small storage of the encrypted video and has a low computational cost for both encryption and detection. Experimental results show that our scheme performs better regarding detection accuracy and execution speed. Moreover, the proposed scheme can work with more than one format-compliant video encryption method, provided that the positions of the macroblocks can be extracted from the encrypted video bitstream. Due to the coupling of video stream encryption and detection algorithms, our scheme can be directly connected to the video stream output (e.g., surveillance cameras) without requiring any camera modifications.

Motion Estimation-based Human Fall Detection for Visual Surveillance

Currently, the world’s elderly population continues to grow at a dramatic rate. As the number of senior citizens increases, detection of someone falling has attracted increasing attention for visual surveillance systems. This paper presents a novel fall-detection algorithm using motion estimation and an integrated spatiotemporal energy map of the object region. The proposed method first extracts a human region using a background subtraction method. Next, we applied an optical flow algorithm to estimate motion vectors, and an energy map is generated by accumulating the detected human region for a certain period of time. We can then detect a fall using k-nearest neighbor (kNN) classification with the previously estimated motion information and energy map. The experimental results show that the proposed algorithm can effectively detect someone falling in any direction, including at an angle parallel to the camera’s optical axis.

상지 편마비 환자의 능동형 재활운동을 위한 양측성 훈련 인터페이스 기법에 대한 연구

For the self-directed rehabilitation of upper extremity hemiplegia patients, in this paper we propose an interface method capable of doing bilateral exercises in rehabilitation robotics. This is a method for estimating information of movements from the unaffected-side, and projects it to the affected-side in order. That the affected-side is followed the movements of the unaffected-side. For estimation of the unaffected-side movements information, gyro sensor data and acceleration sensor data were fused. In order to improve the measurement error in data fusion, a HDR filter and a complementary filter were applied. Estimated motion information is derived the one side of the drive input of rehabilitation robot. In order to validate the proposed method, experimental equipment is designed to be similar to the body's joints. The verification was performed by comparing the estimation angle data from inertial sensors and the encoder data which were attached to the mechanism.

Multi-image high dynamic range algorithm using a hybrid camera

This paper proposes a multi-image high dynamic range (HDR) algorithm using a hybrid camera, plus a tone mapping algorithm based on sub-band decomposed multi-scale retinex (SD-MSR). Conventional multi-image HDR algorithms, which produce a single HDR image from multiple low dynamic range (LDR) images, usually perform registration by exploiting homography derived from scale-invariant feature transform matching between adjacent images to overcome severe differences in luminance. However, such a registration can cause ghost artifacts, because motion information for moving objects can be inaccurate. In order to solve this problem, we employ a hybrid camera that simultaneously produces low-resolution (LR) video sequences and high-resolution (HR) still images. So, we can estimate accurate motion from the LR video because of homogeneous luminance. Then, we apply the estimated motion information to the registration between the corresponding HR images prior to HDR image synthesis. Finally, we present a tone mapping algorithm based on SD-MSR to enhance details in the synthesized HDR image. Experimental results show that the proposed algorithms outperform state-of-the-art HDR algorithms.

Occupancy grid mapping in urban environments from a moving on-board stereo-vision system.

Occupancy grid map is a popular tool for representing the surrounding environments of mobile robots/intelligent vehicles. Its applications can be dated back to the 1980s, when researchers utilized sonar or LiDAR to illustrate environments by occupancy grids. However, in the literature, research on vision-based occupancy grid mapping is scant. Furthermore, when moving in a real dynamic world, traditional occupancy grid mapping is required not only with the ability to detect occupied areas, but also with the capability to understand dynamic environments. The paper addresses this issue by presenting a stereo-vision-based framework to create a dynamic occupancy grid map, which is applied in an intelligent vehicle driving in an urban scenario. Besides representing the surroundings as occupancy grids, dynamic occupancy grid mapping could provide the motion information of the grids. The proposed framework consists of two components. The first is motion estimation for the moving vehicle itself and independent moving objects. The second is dynamic occupancy grid mapping, which is based on the estimated motion information and the dense disparity map. The main benefit of the proposed framework is the ability of mapping occupied areas and moving objects at the same time. This is very practical in real applications. The proposed method is evaluated using real data acquired by our intelligent vehicle platform “SeTCar” in urban environments.

Motion based prediction and development of response to an "on the way" stimulus

Prior information on regularities of the world is essential for optimal performance of sensory processing. In the particular case of the detection of visual motion, the prior knowledge on the temporal coherency of motion facilitates estimation of predictable trajectories. Based on such prior information, we have proposed a generic bayesian modeling framework to implement anisotropic diffusion of estimated motion information. By applying particle filtering (Sequential Monte Carlo) we were able to study the emergence of a global neural response through a probabilistic pool of local sensory information. To explore if such diffusive mechanisms could explain neurophysiological recordings from the responses of neural populations to a predictable moving stimulus that we have recorded from V1 neurons in macaque monkeys, we simulated such response in the model. Our main experimental protocol consisted of a bar moving to a fixed direction but with different trajectory lengths before reaching the receptive fields of the recorded neurons. Experimental results show that a bar moving towards a long trajectory generates an anticipatory response that build-up gradually with time before the bar actually enters into the receptive field of the cell. In the modeling part we aim to asses the role of prediction in the dynamical development of neural activity at, or before, the arrival of the sensory stimulus. We simulated the response of the model in target receptive fields to a smoothly moving stimulus whose trajectory crosses the receptive field. This was done under various conditions for two classic stimuli: a dot and a slanted bar. We found that, as expected, a smoothly moving stimulus triggers a transient response as it crosses the receptive field and that he peak time of this response matches the arrival of the stimulus to the it’s center. We found that the development of a response to an ‘on the way stimulus’ is enhanced by increasing the reliability of sensory information along the trajectory of motion. Also we observed an advance of the peak time with respect to the degree of predictability of the trajectory. This is in agreement with the neural activity that we have recorded from V1 neurons in response to a smoothly moving slanted bar with different trajectory lengths. This results may be explained with the definition of classical and extra classical receptive fields while former is activated only by having a stimulus inside it and later covers bigger area and collects information from a wider neighborhood [1]. Motion based prediction by highlighting the anisotropic diffusion of estimated motion information can develop another perspective on lateral spread of activity in primary visual cortex and better understanding of modulatory effect of cells on each other.

De-Interlacing Algorithm Using Spatial-Temporal Correlation-Assisted Motion Estimation

De-interlacing algorithms are used to convert interlaced video into progressive scanning format. The motion-adaptive technique provides acceptable picture quality, but the quality of the motion area still needs to be improved. Among the various de-interlacing techniques, the motion-compensated de-interlacing technique provides the best performance if the estimated motion information is reliable. However, it suffers from inaccurate motion estimation, and the weak error protection thus deteriorates the visual quality. This paper presents a motion-compensated de-interlacing algorithm with highly accurate motion estimation and robust error detection. In order to obtain more accurate motion information, spatial-temporal correlation-assisted motion estimation is proposed. The spatial and temporal correlations among the motion vectors (MVs) are exploited to find the true motion of the object. In order to reject incorrect temporal information, a hierarchical MV reliability verification is provided. The possible defects in both large and small areas can be detected effectively. The experimental results show that the proposed algorithm outperforms existing algorithms and produces high quality de-interlaced results in various video sequences.

Restoration of motion-blurred iris images on mobile iris recognition devices

With the widespread use of biometric technology in many applications, iris recognition techniques on mobile devices have been more commercialized recently. Some of these techniques include the Pier 2.3 system and the HIIDE Series 4. When working with mobile devices, the shaking movements of a given user's hand can sometimes cause motion-blurred iris images, which can reduce iris recognition accuracy. We propose a new method of restoring these kinds of motion-blurred iris images. We present three improvements over previous works: (1) We estimated the direction and magnitude of motion, based on the corneal specular reflection of the pupil. (2) The point spread function (PSF) was adaptively modeled based on the estimated motion information. (3) The restoration of motion-blurred iris images was performed with the filter parameters that were determined in terms of increasing the accuracy of iris recognition. Experimental results showed that the equal error rate (EER) without the proposed motion-deblurring method was 1.538% and that with the proposed method was 0.962%. Consequently, the EER was reduced as much as 0.576% (=1.538−0.962%), and results showed that the accuracy of iris recognition with the proposed method was superior to results with conventional motion-deblurring methods.

Robust motion from space curves and 3D reconstruction from multiviews using perpendicular double stereo rigs

This paper proposes a robust curve based method to reconstruct 3D model of an object from image sequences captured by two perpendicular stereo rigs. First, corresponding points and the geometry of points are computed in stereo images by extracting unique space curves. A new algorithm is proposed to extract unique space curves from plane curves in stereo images based on curvature and torsion consistency. The proposed method provides accurate geometry of the curve points with extremely reduced number of outliers. Contrarily to the standard sparse approaches that need sub-pixel accuracy to compute structure and motion, the proposed curve matching method deals with pixel accuracy information. More importantly, it finds the correspondence based on curve shape and does not use any photometric information. This property makes the matching process very robust against the color and intensity maladjustment of stereo rigs. Second, the recovered space curves are employed to estimate robust motion by minimizing the curve distance in the next sequence of stereo images. An efficient structure of stereo rigs – perpendicular double stereo – is proposed to improve accuracy of motion estimation. We discuss and prove its properties mathematically. Third, a set of calibrated virtual cameras are constructed from estimated motion information to take advantage of the shape-from-silhouette using multiple views and extract the object’s visual hull as fine as possible. As a whole, a complete automatic and practical system of three-dimensional modeling from raw images captured by calibrated perpendicular double stereo rigs to surface representation is proposed. Fine motion estimation is the main advantage of the proposed method using perpendicular stereo rigs, which makes use of the space curves in a large base line camera setup. While the previous methods of motion estimation suffer from the statistical bias due to quantization noise, measurement error, and outliers in the input data set, the proposed method overcomes the bias problem even in pixel-level information. Experimental results demonstrate the privileged performance of the proposed method for a variety of object shapes and textures.

Tracking a partially occluded target with a cluster of Kalman filters

Tracking moving objects is one of the most important techniques in motion analysis and understanding, and it has many difficult problems to solve. Estimating and identifying moving objects, when the background and moving objects vary dynamically, are especially difficult. It is possible under such a complex environment that targets might disappear totally or partially due to occlusion by other objects. The Kalman filter has been used to estimate motion information and use the information in predicting the appearance of targets in succeeding frames. In this article, we propose another version of the Kalman filter, to be called the structural Kalman filter, which can successfully accomplish its role of estimating motion information under such a deteriorating condition as occlusion. Experimental results show that the suggested approach is very effective in estimating and tracking non-rigid moving objects reliably. © 2002 Wiley Periodicals, Inc.

2D human body tracking with Structural Kalman filter

Tracking moving objects is one of the most important techniques in motion analysis and understanding, and it has many difficult problems to solve. Especially, estimating and identifying moving objects, when the background and moving objects vary dynamically, are very difficult. It is possible under such a complex environment that targets may disappear totally or partially due to occlusion by other objects. The Kalman filter has been used to estimate motion information and use the information in predicting the appearance of targets in the succeeding frames. In this paper, we propose another version of the Kalman filter, to be called Structural Kalman filter, which can successfully work its role of estimating motion information under such a deteriorating condition as occlusion. Experimental results show that the suggested approach is very effective in estimating and tracking non-rigid moving objects reliably.

Hybrid mapping parameter estimation using hierarchical structure in object-oriented coding

This paper investigates motion estimation and compensation in object-oriented analysis-synthesis coding. Object-oriented coding employs a mapping parameter technique for estimating motion information in each object. The mapping parameter technique using gradient operators requires high computational complexity. The main objective of this paper is to propose a hybrid mapping parameter estimation method using the hierarchical structure in object-oriented coding. The hierarchical structure employed constructs a low-resolution image. Then six mapping parameters for each object are estimated from the low-resolution image and these parameter values are verified based on the displaced frame difference (DFD). If the verification test succeeds, the parameters and object boundaries are coded. Otherwise, eight mapping parameters are estimated in a low-resolution image and the verification test is again applied to an image reconstructed by estimated parameters. If it succeeds, the parameters and object boundaries are coded, otherwise, the regions are coded by second-order polynomial approximation. Theoretical analysis and computer simulation show that the peak signal to noise ratio (PSNR) of the image reconstructed by the proposed method lies between those of images reconstructed by the conventional 6- and 8-parameter estimation methods with reduction of the computation time by a factor of about four.

Mapping parameter estimation using integral projections and segmented moving objects in object‐oriented analysis‐synthesis coding

Object-oriented analysis-synthesis coding subdivides each image of a sequence into a number of moving objects and compensates the motion of each object. Thus it can reconstruct real motion better than conventional motion-compensated coding techniques at very low bit rates. It uses a mapping parameter technique for estimating motion information of each object. Since a mapping parameter technique employs gradient operators, the estimated parameters are sensitive to redundant details and noise. To determine mapping parameters accurately, we propose a new analysis method using integral projections for estimation of gradient values. Also, to reconstruct the local motion correctly, the proposed algorithm divides an image into segmented objects, each having uniform motion information, whereas the conventional one assumes a large object having the same motion information. Computer simulation results with several test sequences show that the proposed image analysis method in object-oriented analysis-synthesis coding gives better performance than the conventional one.