Depth Motion Research Articles

An Enhanced Depth Motion Map and Deep Learning Method to Identify Human Actions from Depth Action Sequences

An Enhanced Depth Motion Map and Deep Learning Method to Identify Human Actions from Depth Action Sequences

Enhanced Depth Motion Map and Deep Learning Assisted Human Action Recognition from Depth Action Sequences

Enhanced Depth Motion Map and Deep Learning Assisted Human Action Recognition from Depth Action Sequences

Motion-in-depth effects on interceptive timing errors in an immersive environment

We often need to interact with targets that move along arbitrary trajectories in the 3D scene. In these situations, information of parameters like speed, time-to-contact, or motion direction is required to solve a broad class of timing tasks (e.g., shooting, or interception). There is a large body of literature addressing how we estimate different parameters when objects move both in the fronto-parallel plane and in depth. However, we do not know to which extent the timing of interceptive actions is affected when motion-in-depth (MID) is involved. Unlike previous studies that have looked at the timing of interceptive actions using constant distances and fronto-parallel motion, we here use immersive virtual reality to look at how differences in the above-mentioned variables influence timing errors in a shooting task performed in a 3D environment. Participants had to shoot at targets that moved following different angles of approach with respect to the observer when those reached designated shooting locations. We recorded the shooting time, the temporal and spatial errors and the head’s position and orientation in two conditions that differed in the interval between the shot and the interception of the target’s path. Results show a consistent change in the temporal error across approaching angles: the larger the angle, the earlier the error. Interestingly, we also found different error patterns within a given angle that depended on whether participants tracked the whole target’s trajectory or only its end-point. These differences had larger impact when the target moved in depth and are consistent with underestimating motion-in-depth in the periphery. We conclude that the strategy participants use to track the target’s trajectory interacts with MID and affects timing performance.

The effect of eccentricity on the linear-radial speed bias: Testing the motion-in-depth model

Radial motion is perceived as faster than linear motion when local spatiotemporal properties are matched. This radial speed bias (RSB) is thought to occur because radial motion is partly interpreted as motion-in-depth. Geometry dictates that a fixed amount of radial expansion at increasing eccentricities is consistent with smaller motion in depth, so it is perhaps surprising that the impact of eccentricity on RSB has not been examined. With this issue in mind, across 3 experiments we investigated the RSB as a function of eccentricity. In a 2IFC task, participants judged which of a linear (test – variable speed) or radial (reference – 2 or 4°/s) stimulus appeared to move faster. Linear and radial stimuli comprised 4 Gabor patches arranged left, right, above and below fixation at varying eccentricities (3.5°–14°). For linear stimuli, Gabors all drifted left or right, whereas for radial stimuli Gabors drifted towards or away from the centre. The RSB (difference in perceived speeds between matched linear and radial stimuli) was recovered from fitted psychometric functions. Across all 3 experiments we found that the RSB decreased with eccentricity but this tendency was less marked beyond 7° - i.e. at odds with the geometry, the effect did not continue to decrease as a function of eccentricity. This was true irrespective of whether stimuli were fixed in size (Experiment 1) or varied in size to account for changes in spatial scale across the retina (Experiment 2). It was also true when we removed conflicting stereo cues via monocular viewing (Experiment 3). To further investigate our data, we extended a previous model of speed perception, which suggests perceived motion for such stimuli reflects a balance between two opposing perceptual interpretations, one for motion in depth and the other for object deformation. We propose, in the context of this model, that our data are consistent with placing greater weight on the motion in depth interpretation with increasing eccentricity and this is why the RSB does not continue to reduce in line with purely geometric constraints.

Surge and heave hydrodynamic coefficients for a combination of a porous and a rigid cylinder in motion in finite ocean depth

Radiation of ocean water waves for (i) a system consisting of a hollow porous cylinder at the top, (ii) another system comprising a solid porous cylinder at the top; with a rigid solid cylinder considered at the bottom, is studied. Translational motions in the x-, and z-directions, surge and heave, are considered. Subsequently, the associated added mass and damping coefficients are investigated. The present configuration can be observed to be a wave energy device which taps ocean wave energy. Continuity conditions for pressure and velocity across the linear interfaces are used for setting up a system of linear equations. To validate the model, result of damping coefficients is compared with a result of Miloh [Wave loads on a floating solar pond. In: Proceedings of the international workshop on ship and platform motions. Berkeley: Department of Naval Architecture and Ocean Engineering; 1983] for the first case and with that of Yeung [Added mass and damping of a vertical cylinder in finite-depth waters. Appl Ocean Res. 1981;3(3):119–133] for the second case, both showing a good agreement. It is found that the variation of porous coefficient, radii, and depth has immense influence on the coefficients for such a cylindrical system.

Real-Time Action Recognition System for Elderly People Using Stereo Depth Camera.

Smart technologies are necessary for ambient assisted living (AAL) to help family members, caregivers, and health-care professionals in providing care for elderly people independently. Among these technologies, the current work is proposed as a computer vision-based solution that can monitor the elderly by recognizing actions using a stereo depth camera. In this work, we introduce a system that fuses together feature extraction methods from previous works in a novel combination of action recognition. Using depth frame sequences provided by the depth camera, the system localizes people by extracting different regions of interest (ROI) from UV-disparity maps. As for feature vectors, the spatial-temporal features of two action representation maps (depth motion appearance (DMA) and depth motion history (DMH) with a histogram of oriented gradients (HOG) descriptor) are used in combination with the distance-based features, and fused together with the automatic rounding method for action recognition of continuous long frame sequences. The experimental results are tested using random frame sequences from a dataset that was collected at an elder care center, demonstrating that the proposed system can detect various actions in real-time with reasonable recognition rates, regardless of the length of the image sequences.

Human action recognition based on multi-scale feature maps from depth video sequences

Human action recognition is an active research area in computer vision. Although great progress has been made, previous methods mostly recognize actions from depth video sequences at only one scale, and thus they often neglect multi-scale spatial changes that provide additional information in practical applications. In this paper, we present a novel framework with a multi-scale mechanism to improve scale diversity of motion features. We propose a multi-scale feature map called Laplacian pyramid depth motion images(LP-DMI). First, We employ depth motion images (DMI) as the templates to generate the multi-scale static representation of actions. Then, we caculate LP-DMI to enhance multi-scale dynamic information of motions and reduce redundant static information in human bodies. We further extract the multi-granularity descriptor called LP-DMI-HOG to provide more discriminative features. Finally, we utilize extreme learning machine (ELM) for action classification. The proposed method yeilds the recognition accuracy of 93.41%, 85.12%, 91.94% on the public MSRAction3D, UTD-MHAD and DHA dataset. Through extensive experiments, we prove that our method outperforms the state-of-the-art benchmarks.

A covered eye fails to follow an object moving in depth

To clearly view approaching objects, the eyes rotate inward (vergence), and the intraocular lenses focus (accommodation). Current ocular control models assume both eyes are driven by unitary vergence and unitary accommodation commands that causally interact. The models typically describe discrete gaze shifts to non-accommodative targets performed under laboratory conditions. We probe these unitary signals using a physical stimulus moving in depth on the midline while recording vergence and accommodation simultaneously from both eyes in normal observers. Using monocular viewing, retinal disparity is removed, leaving only monocular cues for interpreting the object’s motion in depth. The viewing eye always followed the target’s motion. However, the occluded eye did not follow the target, and surprisingly, rotated out of phase with it. In contrast, accommodation in both eyes was synchronized with the target under monocular viewing. The results challenge existing unitary vergence command theories, and causal accommodation-vergence linkage.

Exploring 3D Human Action Recognition Using STACOG on Multi-View Depth Motion Maps Sequences.

This paper proposes an action recognition framework for depth map sequences using the 3D Space-Time Auto-Correlation of Gradients (STACOG) algorithm. First, each depth map sequence is split into two sets of sub-sequences of two different frame lengths individually. Second, a number of Depth Motion Maps (DMMs) sequences from every set are generated and are fed into STACOG to find an auto-correlation feature vector. For two distinct sets of sub-sequences, two auto-correlation feature vectors are obtained and applied gradually to -regularized Collaborative Representation Classifier (-CRC) for computing a pair of sets of residual values. Next, the Logarithmic Opinion Pool (LOGP) rule is used to combine the two different outcomes of -CRC and to allocate an action label of the depth map sequence. Finally, our proposed framework is evaluated on three benchmark datasets named MSR-action 3D dataset, DHA dataset, and UTD-MHAD dataset. We compare the experimental results of our proposed framework with state-of-the-art approaches to prove the effectiveness of the proposed framework. The computational efficiency of the framework is also analyzed for all the datasets to check whether it is suitable for real-time operation or not.

Blind image deblurring for a close scene under a 6-DOF motion path

PurposeHow to model blind image deblurring that arises when a camera undergoes ego-motion while observing a static and close scene. In particular, this paper aims to detail how the blurry image can be restored under a sequence of the linear model of the point spread function (PSF) that are derived from the 6-degree of freedom (DOF) camera’s accurate path during the long exposure time.Design/methodology/approachThere are two existing techniques, namely, an estimation of the PSF and a blind image deconvolution. Based on online and short-period inertial measurement unit (IMU) self-calibration, this motion path has discretized a sequence of the uniform speed of 3-DOF rectilinear motion, which unites with a 3-DOF rotational motion to form a discrete 6-DOF camera’s path. These PSFs are evaluated through the discrete path, then combine with a blurry image to restoration through deconvolution.FindingsThis paper describes to build a hardware attachment, which is composed of a consumer camera, an inexpensive IMU and a 3-DOF motion mechanism to the best of the knowledge, together with experimental results demonstrating its overall effectiveness.Originality/valueFirst, the paper proposes that a high-precision 6-DOF motion platform periodically adjusts the speed of a three-axis rotational motion and a three-axis rectilinear motion in a short time to compensate the bias of the gyroscope and the accelerometer. Second, this paper establishes a model of 6-DOF motion and emphasizes on rotational motion, translational motion and scene depth motion. Third, this paper addresses a novel model of the discrete path that the motion during long exposure time is discretized at a uniform speed, then to estimate a sequence of PSFs.

The Effect of Motion Direction and Eccentricity on Vection, VR Sickness and Head Movements in Virtual Reality.

Virtual Reality (VR) experienced through head-mounted displays often leads to vection, discomfort and sway in the user. This study investigated the effect of motion direction and eccentricity on these three phenomena using optic flow patterns displayed using the Valve Index. Visual motion stimuli were presented in the centre, periphery or far periphery and moved either in depth (back and forth) or laterally (left and right). Overall vection was stronger for motion in depth compared to lateral motion. Additionally, eccentricity primarily affected stimuli moving in depth with stronger vection for more peripherally presented motion patterns compared to more central ones. Motion direction affected the various aspects of VR sickness differently and modulated the effect of eccentricity on VR sickness. For stimuli moving in depth far peripheral presentation caused more discomfort, whereas for lateral motion the central stimuli caused more discomfort. Stimuli moving in depth led to more head movements in the anterior-posterior direction when the entire visual field was stimulated. Observers demonstrated more head movements in the anterior-posterior direction compared to the medio-lateral direction throughout the entire experiment independent of motion direction or eccentricity of the presented moving stimulus. Head movements were elicited on the same plane as the moving stimulus only for stimuli moving in depth covering the entire visual field. Acorrelation showed a positive relationship between dizziness and vection duration and between general discomfort and sway. Identifying where in the visual field motion presented to an individual causes the least amount of VR sickness without losing vection and presence can guide development for Virtual Reality games, training and treatment programmes.

Design, simulation of AUV (VIAM-AUV1000) for research and rescue

Autonomous Underwater Vehicles have gained popularity for the last decades, especially a lot of AUVs were considered as the most suitable tool for the purpose of reducing risks of people in dangerous marine operations. This paper presents the preliminary results of the research on hardware design, the controller of an autonomous underwater vehicle model for the task of survey, search and rescue ... With a compact design, AUV can operate in limited spaces. Through a unique ducted propeller and rudder located at the aft, the AUV can perform horizontal motion. It can also control pitch angle and depth motion by an inside mass shifter mechanism (MSM) which changes the vehicle center of gravity. In addition, The AUV is integrated with powerful eletronic system, highprecision sensors helping it carries on missions from simple to complex. The use of Sliding Mode Control (SMC) to independently design the heading and depth controllers for AUV demonstrates the steady stability of the controllers with the nonlinear model, uncertainty parameters and disturbances. Finally, the simulation results show that the SMC controllers can control the AUV nonlinear model to track the desired steering angle and depth with high accuracy and stability.

Research on Human Motion Recognition Based on Data Redundancy Technology

Aiming at the problems of low recognition rate and slow recognition speed of traditional body action recognition methods, a human action recognition method based on data deduplication technology is proposed. Firstly, the data redundancy technology and perceptual hashing technology are combined to form an index, and the image is filtered from the structure, color, and texture features of human action image to achieve image redundancy processing. Then, the depth feature of processed image is extracted by depth motion map; finally, feature recognition is carried out by convolution neural network so as to achieve the purpose of human action recognition. The simulation results show that the proposed method can obtain the optimal recognition results and has strong robustness. At the same time, it also fully proves the importance of human motion recognition.

Controller design for underwater robotic vehicle based on improved whale optimization algorithm

This paper presents the impact of introducing a two-controller on the linearized autonomous underwater vehicle (AUV) for vertical motion control. These controllers are presented to overcome the sensor noise of the AUV control model that effect on the tolerance and stability of the depth motion control. Linear quadratic Gaussian (LQG) controller is cascaded with AUV model to adapt the tolerance and the stability of the system and compared with FOPID established by the improved whale optimization algorithm (IWOA) to identify which controlling method can make the system more harmonize and tolerable. The developed algorithm is based on improving the original WOA by reshaping a specific detail on WOA to earn a warranty that the new IWOA will have values for the update position lower than the identified lower-bound (LB), and upper-bound (UB). Furthermore, the algorithm is examined by a set of test functions that include (unimodal, multimodal and fixed dimension multimodal functions). The privileges of applying IWOA are reducing the executing time and obtaining the semi-optimal objective function as compared with the original WOA algorithm and other popular swarm-intelligence optimization algorithms.

Bio-inspired visual neural network on spatio-temporal depth rotation perception

In primates’ cerebral cortex, depth rotation sensitive (DRS) neurons have the property of preferential selectivity for depth rotation motion, whereas such a property is rarely adopted to create computational models for depth rotation motion detection. To fill this gap, a novel feedforward visual neural network is developed to execute depth rotation object detection, based on the recent neurophysiologic achievements on the mammalian vision system. The proposed neural network consists of two parts, i.e., presynaptic and postsynaptic neural networks. The former comprises multiple lateral inhibition neural sub-networks for the capture of visual motion information, and the latter extracts the cues of translational and depth motion and later, synthesizes such clues to perceive the process of depth rotation of an object. Experimentally, the neural network is sufficiently examined by different types of depth rotation under multiple conditions and settings. Numerical experiments show that not only it can effectively detect the spatio-temporal energy change of depth rotation of a moving object, but also its output excitation curve is a quasi-sinusoidal one, which is compatible with the hypothesis suggested by Johansson and Jansson in projective geometry. This research is a critical step toward the construction of artificial vision system for depth rotation object recognition.

FRAME SAMPLING ASSISTED DEPTH MOTION MAPS FOR DEPTH BASED HUMAN ACTION RECOGNITION

The discovery of depth sensors has brought new opportunities in the Human Action Research by providing depth image data. Compared to the conventional RGB image data, the depth image data has additional benefits like color, illumination invariant, and provides clues about the shape of body. Inspired with these benefits, we present a new Human Action Recognition model from depth images. For a given action video, the consideration of an entire frames constitutes less detailed information about the shape and movements of body. Hence we have proposed a new method called Frame Sampling to reduce the frame count and chooses only key frames. After key frames extraction, they are processed through Depth Motion Map for action representation followed by Support Vector Machine for classification. The developed model is evaluated on a standard public dataset captured by depth cameras. The experimental results demonstrate the superior performance compared with state-of-art methods

Depth Trajectory-Aware Stereoscopic Video Retargeting

The image/video retargeting (resolution adaptation) problem—adaption of source data (image/video) to different screen resolution—has garnered wide attention due to compelling applications in intelligent adaptive display. Most of the current image/video displaying application achieves the adaption through scaling source uniformly. Compared with traditional image/video retargeting, stereoscopic video retargeting poses new challenges that should preserve shape and depth of salient objects with temporal coherence. In this paper, we propose a depth trajectory-aware stereoscopic video retargeting method by imposing constraints to optimize the coordinates and depths simultaneously. To reduce visual discomfort induced by fast depth motion, our method uses a temporal depth distortion energy to optimize the depth trajectory in temporal direction. As a result, our solution can preserve shape, depth and temporal fidelity of salient objects simultaneously with comfortable visual experience.

Fusing appearance and motion information for action recognition on depth sequences

With the advent of cost-efficient depth cameras, many effective feature descriptors have been proposed for action recognition from depth sequences. However, most of them are based on single feature and thus unable to extract the action information comprehensively, e.g., some kinds of feature descriptors can represent the area where the motion occurs while they lack the ability of describing the order in which the action is performed. In this paper, a new feature representation scheme combining different feature descriptors is proposed to capture various aspects of action cues simultaneously. First of all, a depth sequence is divided into a series of sub-sequences using motion energy based spatial-temporal pyramid. For each sub-sequence, on the one hand, the depth motion maps (DMMs) based completed local binary pattern (CLBP) descriptors are calculated through a patch-based strategy. On the other hand, each sub-sequence is partitioned into spatial grids and the polynormals descriptors are obtained for each of the grid sequences. Then, the sparse representation vectors of the DMMs based CLBP and the polynormals are calculated separately. After pooling, the ultimate representation vector of the sample is generated as the input of the classifier. Finally, two different fusion strategies are applied to conduct fusion. Through extensive experiments on two benchmark datasets, the performance of the proposed method is proved better than that of each single feature based recognition method.

Decoding Neural Responses to Motion-in-Depth Using EEG.

Two stereoscopic cues that underlie the perception of motion-in-depth (MID) are changes in retinal disparity over time (CD) and interocular velocity differences (IOVD). These cues have independent spatiotemporal sensitivity profiles, depend upon different low-level stimulus properties, and are potentially processed along separate cortical pathways. Here, we ask whether these MID cues code for different motion directions: do they give rise to discriminable patterns of neural signals, and is there evidence for their convergence onto a single “motion-in-depth” pathway? To answer this, we use a decoding algorithm to test whether, and when, patterns of electroencephalogram (EEG) signals measured from across the full scalp, generated in response to CD- and IOVD-isolating stimuli moving toward or away in depth can be distinguished. We find that both MID cue type and 3D-motion direction can be decoded at different points in the EEG timecourse and that direction decoding cannot be accounted for by static disparity information. Remarkably, we find evidence for late processing convergence: IOVD motion direction can be decoded relatively late in the timecourse based on a decoder trained on CD stimuli, and vice versa. We conclude that early CD and IOVD direction decoding performance is dependent upon fundamentally different low-level stimulus features, but that later stages of decoding performance may be driven by a central, shared pathway that is agnostic to these features. Overall, these data are the first to show that neural responses to CD and IOVD cues that move toward and away in depth can be decoded from EEG signals, and that different aspects of MID-cues contribute to decoding performance at different points along the EEG timecourse.

Kinematic joint descriptor and depth motion descriptor with convolutional neural networks for human action recognition

Human Action Recognition has gained a huge research interest due to its widespread applications in various fields. However, due to several challenges like noisy and occluded data, view-point variations, body sizes etc., still the action recognition remains a challenging task. Most of the existing action recognition methods focused on the single data type thereby the recognition system has limited performance. To improve the recognition performance, we have modeled a new approach for human action recognition from two different data types; they are depth images and skeleton joints. Two different descriptors are developed for action representation; they are Differential Depth Motion History Image for depth maps and Motion Kinematic Joint Descriptor for skeleton joints. To attain a discriminative feature set, we have trained three different Convolutional Neural Network Models and the results are fused for final action classification. Simulation is carried out over two public datasets and the obtained results indicate that the proposed approach outperforms state-of-art methods.