Abstract
It is often advantageous to track objects in a scene using multimodal information when such information is available. We use audio as a complementary modality to video data, which, in comparison to vision, can provide faster localization over a wider field of view. We present a particle-filter based tracking framework for performing multimodal sensor fusion for tracking people in a videoconferencing environment using multiple cameras and multiple microphone arrays. One advantage of our proposed tracker is its ability to seamlessly handle temporary absence of some measurements (e.g., camera occlusion or silence). Another advantage is the possibility of self-calibration of the joint system to compensate for imprecision in the knowledge of array or camera parameters by treating them as containing an unknown statistical component that can be determined using the particle filter framework during tracking. We implement the algorithm in the context of a videoconferencing and meeting recording system. The system also performs high-level semantic analysis of the scene by keeping participant tracks, recognizing turn-taking events and recording an annotated transcript of the meeting. Experimental results are presented. Our system operates in real time and is shown to be robust and reliable.
Highlights
The goal of most machine perception systems is to mimic the performance of human and animal systems
We present a probabilistic framework for combining results from the two modes and develop a particle filter based joint audio-video tracking algorithm
We present experimental results showing the potential of the developed algorithm
Summary
The goal of most machine perception systems is to mimic the performance of human and animal systems. Capabilities of computers have reached such a level that it is possible to build and develop systems that can combine multiple audio and video sensors and perform meaningful joint-analysis of a scene, such as joint audiovisual speaker localization, tracking, speaker change detection and remote speech acquisition using beamforming techniques, which is necessary for the development of natural, robust and environmentally-independent applications. Applications of such systems include novel human-computer interfaces, robots that sense and perceive their environment, perceptive spaces for applications in immersive virtual or augmented reality, and so forth. We present experimental results showing the potential of the developed algorithm
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.