Multi-camera Video Research Articles

Space–time computational analysis of bio-inspired flapping-wing aerodynamics of a micro aerial vehicle

We present a detailed computational analysis of bio-inspired flapping-wing aerodynamics of a micro aerial vehicle (MAV). The computational techniques used include the Deforming-Spatial-Domain/Stabilized Space---Time (DSD/SST) formulation, which serves as the core computational technique. The DSD/SST formulation is a moving-mesh technique, and in the computations reported here we use the space---time version of the residual-based variational multiscale (VMS) method, which is called "DSD/ SST-VMST." The motion and deformation of the wings are based on data extracted from the high-speed, multi-camera video recordings of a locust in a wind tunnel. A set of special space---time techniques are also used in the computations in conjunction with the DSD/SST method. The special techniques are based on using, in the space---time flow computations, NURBS basis functions for the temporal representation of the motion and deformation of the wings and for the mesh moving and remeshing. The computational analysis starts with the computation of the base case, and includes computations with increased temporal and spatial resolutions compared to the base case. In increasing the temporal resolution, we separately test increasing the temporal order, the number of temporal subdivisions, and the frequency of remeshing. In terms of the spatial resolution, we separately test increasing the wing-mesh refinement in the normal and tangential directions and changing the way node connectivities are handled at the wingtips. The computational analysis also includes using different combinations of wing configurations for the MAV and investigating the beneficial and disruptive interactions between the wings and the role of wing camber and twist.

Space–time techniques for computational aerodynamics modeling of flapping wings of an actual locust

We present the special space---time computational techniques we have introduced recently for computational aerodynamics modeling of flapping wings of an actual locust. These techniques have been designed to be used with the deforming-spatial-domain/stabilized space---time (DSD/SST) formulation, which is the core computational technique. The DSD/SST formulation was developed for flow problems with moving interfaces and was elevated to newer versions over the years, including the space---time version of the residual-based variational multiscale (VMS) method, which is called "DSD/SST-VMST" and used in the computations reported here. The special space---time techniques are based on using, in the space---time flow computations, NURBS basis functions for the temporal representation of the motion and deformation of the locust wings. The motion and deformation data is extracted from the high-speed, multi-camera video recordings of a locust in a wind tunnel. In addition, temporal NURBS basis functions are used in representation of the motion and deformation of the volume meshes computed and also in remeshing. These ingredients provide an accurate and efficient way of dealing with the wind tunnel data and the mesh. The computations demonstrate the effectiveness of the core and special space---time techniques in modeling the aerodynamics of flapping wings, with the wing motion and deformation coming from an actual locust.

Intelligent multi-camera video surveillance: A review

Intelligent multi-camera video surveillance is a multidisciplinary field related to computer vision, pattern recognition, signal processing, communication, embedded computing and image sensors. This paper reviews the recent development of relevant technologies from the perspectives of computer vision and pattern recognition. The covered topics include multi-camera calibration, computing the topology of camera networks, multi-camera tracking, object re-identification, multi-camera activity analysis and cooperative video surveillance both with active and static cameras. Detailed descriptions of their technical challenges and comparison of different solutions are provided. It emphasizes the connection and integration of different modules in various environments and application scenarios. According to the most recent works, some problems can be jointly solved in order to improve the efficiency and accuracy. With the fast development of surveillance systems, the scales and complexities of camera networks are increasing and the monitored environments are becoming more and more complicated and crowded. This paper discusses how to face these emerging challenges.

Multimodal Semantics Extraction from User-Generated Videos

User-generated video content has grown tremendously fast to the point of outpacing professional content creation. In this work we develop methods that analyze contextual information of multiple user-generated videos in order to obtain semantic information about public happenings (e.g., sport and live music events) being recorded in these videos. One of the key contributions of this work is a joint utilization of different data modalities, including such captured by auxiliary sensors during the video recording performed by each user. In particular, we analyze GPS data, magnetometer data, accelerometer data, video- and audio-content data. We use these data modalities to infer information about the event being recorded, in terms of layout (e.g., stadium), genre, indoor versus outdoor scene, and the main area of interest of the event. Furthermore we propose a method that automatically identifies the optimal set of cameras to be used in a multicamera video production. Finally, we detect the camera users which fall within the field of view of other cameras recording at the same public happening. We show that the proposed multimodal analysis methods perform well on various recordings obtained in real sport events and live music performances.

Falling-incident detection and throughput enhancement in a multi-camera video-surveillance system

For most elderly, unpredictable falling incidents may occur at the corner of stairs or a long corridor due to body frailty. If we delay to rescue a falling elder who is likely fainting, more serious consequent injury may occur. Traditional secure or video surveillance systems need caregivers to monitor a centralized screen continuously, or need an elder to wear sensors to detect falling incidents, which explicitly waste much human power or cause inconvenience for elders. In this paper, we propose an automatic falling-detection algorithm and implement this algorithm in a multi-camera video surveillance system. The algorithm uses each camera to fetch the images from the regions required to be monitored. It then uses a falling-pattern recognition algorithm to determine if a falling incident has occurred. If yes, system will send short messages to someone needs to be noticed. The algorithm has been implemented in a DSP-based hardware acceleration board for functionality proof. Simulation results show that the accuracy of falling detection can achieve at least 90% and the throughput of a four-camera surveillance system can be improved by about 2.1 times.

Reception beamforming in horseshoe bats is a dynamic process

Horseshoe bats execute conspicuous pinna movements as part of their biosonar behaviors. These movements contain rigid rotations as well as deformation components that can result in considerable changes to the pinna shapes. Since pinna shape determines the biosonar system's sensitivity as a function of direction and frequency, these non-rigid deformations could be functionally relevant. In the work reported here, it has been investigated how the non-rigid deformations of pinna shape relate to the arrival of echoes. For this purpose, multi-camera high-speed video recordings of both pinna of a bat were synchronized with ultrasonic recordings of the emitted pulses and the returning echoes from a microphone placed next to the head of the animals. Landmark dots were painted on the pinna to enhance the determination of position and velocity from stereo pairs of video recordings. It was found that the pinnae were deformed between as well as during echo receptions. Hence, the echoes were received by different static each pinna shapes (e.g., bent or upright) as well as by pinna shapes that underwent non-rigid deformation within the duration of the echo. As a consequence, echo reception by horseshoe bats involves dynamic beampatterns that can change between as well as during echoes.

H.264-based Wireless Surveillance Sensors in Application to Target Identification and Tracking

In this paper, the author propose a hardware-based m icro structure inside the architecture of H.264 encoder, called Vector Bank (VB), which can dram atically reduce the bandwidth, mem ory, and com putation tim e of a multi-cam era video surveillance system . VB is a m em ory-based structure that contains the vector inform ation for every single Macro Block (MB) of the video fram e from the Motion Estim ation m odule of the H.264 encoder. After extraction of the several vectors from several fram es, a dedicated Digital Signal Processor (DSP) can be assigned to analyze and predict the trajectories of the objects in m otion. With the application of 2-D edge detectors, such as Sobel or Laplacian of Gaussian (LoG) operators, the objects in m otion can be isolated from the background and thus be easily identified. In addition, a m iddleware-based technique derived from Directional Discrete Cosine Transform (DDCT) is introduced to im prove the im age quality without sacrificing the system perform ance. We propose four new m odes of operations in DDCT for better im age com pression ratio. The experim ental results show that we can im prove the im age quality of the targets, isolate them from the background, and track them easily in both linear and exponential trajectories of the m otion.

Multi-camera video surveillance for real-time analysis and reconstruction of soccer games

Soccer analysis and reconstruction is one of the most interesting challenges for wide-area video surveillance applications. Techniques and system implementation for tracking the ball and players with multiple stationary cameras are discussed. With video data captured from a football stadium, the real-world, real-time positions of the ball and players can be generated. The whole system contains a two-stage workflow, i.e., single view and multi-view processing. The first stage includes categorizing of players and filtering of the ball after changing detection against an adaptive background and image-plane tracking. Occlusion reasoning and tracking-back is applied for robust ball filtering. In the multi-view stage, multiple observations from overlapped single views are fused to refine players’ positions and to estimate 3-D ball positions using geometric constraints. Experimental results on real data from long sequences are demonstrated.

The neurophysiological correlates of motor tics following focal striatal disinhibition

The cortico-basal ganglia pathway is involved in normal motor control and implicated in multiple movement disorders. Brief repetitive muscle contractions known as motor tics are a common symptom in several basal ganglia related motor disorders. We used focal micro-injections of the GABA-A antagonist bicuculline to the sensorimotor putamen of behaving primates to induce stereotyped tics similar to those observed in human disorders. This focal disruption of GABA transmission in the putamen led to motor tics confined to a single or a few muscles. The temporal and structural properties of the tics were identified using electromyogram and frame-by-frame analysis of multi-camera video recordings. During experimental sessions the tics would wax and wane, but their size and shape remained highly stereotyped within the session. Neuronal spiking activity and local field potentials were recorded simultaneously from multiple locations along the cortico-basal ganglia pathway: motor cortex, putamen and globus pallidus external and internal segments. The local field potentials displayed stereotyped tic-related voltage transients lasting several hundred milliseconds. These 'local field potential spikes', which appeared throughout the cortico-basal ganglia pathway, were consistently observed in close temporal association to the motor tics. During tic expression, neuronal activity was altered in most of the recorded neurons in a temporally focal manner, displaying phasic firing rate modulations time locked to the tics. Consistent with theoretical models of tic generation, transient inhibition of the basal ganglia output nucleus prior to and during tic expression was observed. The phasic reduction of basal ganglia output was correlated with a disinhibition of cortical activity, manifesting as short bursts of activity in motor cortex. The results demonstrate that the basal ganglia provide a finely timed disinhibition in the output nuclei of the basal ganglia. However, a large fraction of the neurons were simultaneously inhibited during tics, although tics were only manifested in a small confined muscle group. This suggests that rather than representing a specific action within the basal ganglia itself, these nuclei provide a temporally exact but spatially distributed release signal. The tics induced by striatal disinhibition bear a striking resemblance to motor tics recognized in human pathologies associated with basal ganglia dysfunction. The neuronal changes observed during tic formation may provide valuable insights into the underlying mechanism of tic disorders, as well as into basic information processing in the cortico-basal ganglia loop.

Twin Gaussian Processes for Structured Prediction

We describe twin Gaussian processes (TGP), a generic structured prediction method that uses Gaussian process (GP) priors on both covariates and responses, both multivariate, and estimates outputs by minimizing the Kullback-Leibler divergence between two GP modeled as normal distributions over finite index sets of training and testing examples, emphasizing the goal that similar inputs should produce similar percepts and this should hold, on average, between their marginal distributions. TGP captures not only the interdependencies between covariates, as in a typical GP, but also those between responses, so correlations among both inputs and outputs are accounted for. TGP is exemplified, with promising results, for the reconstruction of 3d human poses from monocular and multicamera video sequences in the recently introduced HumanEva benchmark, where we achieve 5 cm error on average per 3d marker for models trained jointly, using data from multiple people and multiple activities. The method is fast and automatic: it requires no hand-crafting of the initial pose, camera calibration parameters, or the availability of a 3d body model associated with human subjects used for training or testing.

An audio-driven dancing avatar

We present a framework for training and synthesis of an audio-driven dancing avatar. The avatar is trained for a given musical genre using the multicamera video recordings of a dance performance. The video is analyzed to capture the time-varying posture of the dancer’s body whereas the musical audio signal is processed to extract the beat information. We consider two different marker-based schemes for the motion capture problem. The first scheme uses 3D joint positions to represent the body motion whereas the second uses joint angles. Body movements of the dancer are characterized by a set of recurring semantic motion patterns, i.e., dance figures. Each dance figure is modeled in a supervised manner with a set of HMM (Hidden Markov Model) structures and the associated beat frequency. In the synthesis phase, an audio signal of unknown musical type is first classified, within a time interval, into one of the genres that have been learnt in the analysis phase, based on mel frequency cepstral coefficients (MFCC). The motion parameters of the corresponding dance figures are then synthesized via the trained HMM structures in synchrony with the audio signal based on the estimated tempo information. Finally, the generated motion parameters, either the joint angles or the 3D joint positions of the body, are animated along with the musical audio using two different animation tools that we have developed. Experimental results demonstrate the effectiveness of the proposed framework.

Euclidean path modeling for video surveillance

In this paper, we address the issue of Euclidean path modeling in a single camera for activity monitoring in a multi-camera video surveillance system. The method consists of a path building training phase and a testing phase. During the unsupervised training phase, after auto-calibrating a camera and thereafter metric rectifying the input trajectories, a weighted graph is constructed with trajectories represented by the nodes, and weights determined by a similarity measure. Normalized-cuts are recursively used to partition the graph into prototype paths. Each path, consisting of a partitioned group of trajectories, is represented by a path envelope and an average trajectory. For every prototype path, features such as spatial proximity, motion characteristics, curvature, and absolute world velocity are then recovered directly in the rectified images or by registering to aerial views. During the testing phase, using our simple yet efficient similarity measures for these features, we seek a relation between the trajectories of an incoming sequence and the prototype path models to identify anomalous and unusual behaviors. Real-world pedestrian sequences are used to evaluate the steps, and demonstrate the practicality of the proposed approach.

Multi-scale 3D scene flow from binocular stereo sequences

Scene flow methods estimate the three-dimensional motion field for points in the world, using multi-camera video data. Such methods combine multi-view reconstruction with motion estimation. This paper describes an alternative formulation for dense scene flow estimation that provides reliable results using only two cameras by fusing stereo and optical flow estimation into a single coherent framework. Internally, the proposed algorithm generates probability distributions for optical flow and disparity. Taking into account the uncertainty in the intermediate stages allows for more reliable estimation of the 3D scene flow than previous methods allow. To handle the aperture problems inherent in the estimation of optical flow and disparity, a multi-scale method along with a novel region-based technique is used within a regularized solution. This combined approach both preserves discontinuities and prevents over-regularization—two problems commonly associated with the basic multi-scale approaches. Experiments with synthetic and real test data demonstrate the strength of the proposed approach.

Distributed Interactive Video Arrays for Event Capture and Enhanced Situational Awareness

Video surveillance activity has dramatically increased over the past few years. Earlier work dealt mostly with single stationary cameras, but the recent trend is toward active multicamera systems. Such systems offer several advantages over single camera systems - multiple overlapping views for obtaining 3D information and handling occlusions, multiple nonoverlapping cameras for covering wide areas, and active pan-tilt-zoom (PTZ) cameras for observing object details. To address these issues, we have developed a multicamera video surveillance approach, called distributed interactive video array. The DIVA framework provides multiple levels of semantically meaningful information ("situational" awareness) to match the needs of multiple remote observers. We have designed DIVA-based systems that can track and identify vehicles and people, monitor perimeters and bridges, and analyze activities. A new video surveillance approach employing a large-scale cluster of video sensors demonstrates the promise of multicamera arrays for homeland security.

Anatomy of a multicamera video surveillance system

We present a framework for multicamera video surveillance. The framework consists of three phases: detection, representation, and recognition. The detection phase handles multisource spatiotemporal data fusion for efficiently and reliably extracting motion trajectories from video. The representation phase summarizes raw trajectory data to construct hierarchical, invariant, and content-rich descriptions of the motion events. Finally, the recognition phase deals with event classification and identification on the data descriptors. Through empirical study in a parking-lot-surveillance setting, we show that our spatiotemporal fusion scheme and biased sequence-data learning method are highly effective in identifying suspicious events.

Appearance-based virtual view generation from multicamera videos captured in the 3-d room

We present an appearance-based virtual view generation method that allows viewers to fly through a real dynamic scene. The scene is captured by multiple synchronized cameras. Arbitrary views are generated by interpolating two original camera-views near the given viewpoint. The quality of the generated synthetic view is determined by the precision, consistency and density of correspondences between the two images. All or most of previous work that uses interpolation extracts the correspondences from these two images. However, not only is it difficult to do so reliably (the task requires a good stereo algorithm), but also the two images alone sometimes do not have enough information, due to problems such as occlusion. Instead, we take advantage of the fact that we have many views, from which we can extract much more reliable and comprehensive 3D geometry of the scene as a 3D model. Dense and precise correspondences between the two images, to be used for interpolation, are obtained using this constructed 3D model.

Operation of the Single-Grip Harvester: Motor-Sensory and Cognitive Work

This article describes the interface between a one-grip harvester and its operator as it relates to motor-sensory and cognitive activities of the operator when carrying out thinning and regeneration harvesting in Sweden. Work tasks are described, as are operator data gathering, cognitive and motor-sensory processes. Descriptions of motor- sensory inputs and control manipulation are based on digital data gathering from machine controls combined with parallel, multi-camera video taping of work activities. Work element descriptions, data gathering and cognitive processes were inferred based on interviews with 20 operators and operator instructors. Work features found to limit an operator’s efficiency were: few breaks in the work; very intensive handling of controls (4000 control inputs per hour in our study) due to lack of automatic functions in the boom and harvester head; restricted view from the cab; lack of information about the stand and log; and skewed and twisted work postures. The main part of the operator’s cognitive work was found to be done through automated skills. The experienced operator acts upon the global situation and seldom analyses it explicitly. Their chunks of information (memory aggregates recognized in toto) guide them in what to do, and usually they see only one course of action to follow in a particular situation. Key future technical development needs and opportunities include additional automation to reduce operator control inputs, developing means of improving visibility such as 3-D laser scanning and additional use of stand inventory and historical data about log properties to improve automation of cross-cutting. One of the key future research areas regarding training and robotization is to determine the data contained in the ‘chunks’ of information used by operators in making machine location, harvesting and processing decisions.