Pan-tilt-zoom Research Articles

Accuracy Analysis of a Dual Camera System with an Asymmetric Photogrammetric Configuration

AbstractThis paper presents a dual camera system combining a wide field-of-view (FOV) surveillance camera and a pan-tiltzoom (PTZ) camera with an asymmetric photogrammetric configuration, and focuses on the analysis of its attainable measurement accuracy. First, we discuss the geometric modeling of the asymmetric photogrammetric configuration and analyze the accuracy of measurement based on error propagation for different baseline lengths, different focal lengths, and different pan angles of the PTZ camera. Second, we performed a comprehensive accuracy analysis based on Monte Carlo simulation, which incorporated artificial noise into the input data. Third, we conducted actual experiments in indoor and outdoor environments to verify the theoretical and simulation results. We found that the baseline length between the dual cameras was the main factor influencing measurement accuracy. Increase of the PTZ camera focal length could improve the measurement accuracy, but this trend was not significant when its focal length was relatively long. The pan angle of the PTZ camera also influenced the measurement accuracy, but this influence was not significant at short ranges. From these discoveries, we present an optimum configuration of the dual camera system for better than 1 percent measurement accuracy of the range within a normal observation range (e.g., 60 m). This proposed dual camera system provides enhanced machine vision capabilities that can be used in various applications.

A simplified two-view geometry based external calibration method for omnidirectional and PTZ camera pairs

The external calibration of a camera system is essential for most of the applications that involve an omnidirectional and a pan-tilt-zoom (PTZ) camera. The methods in the literature fall into two major categories; (1) a complete external calibration of the system which allows all degrees of freedom but highly time consuming, (2) spatial mapping between the pixel coordinates in omnidirectional camera and pan/tilt angles of the PTZ camera instead of explicitly computing the rotation and translation. Most methods in this category make restrictive assumptions about the camera setup such as optical axes of the cameras coincide. We propose an external calibration method that is effective and practical. Using the two-view geometry principles and making reasonable assumptions about the camera setup, calibration is performed with just two scene points. We extract rotation using the point correspondences in images. Locating the PTZ camera in the omnidirectional image is used to find the translation parameters and the real distance between the two scene points lets us compute the translation in correct scale. Results of the simulated and real image experiments show that our method works effectively in real world cases and its accuracy is comparable to the state-of-the-art methods.

Foreground detection for moving cameras with stochastic approximation

Most foreground detection algorithms do not perform well with pan-tilt-zoom (PTZ) cameras for video surveillance and static cameras that experience vibration, since they rely on the assumption that the background does not move. Here a novel approach based on stochastic approximation learning of probabilistic mixtures is proposed. It assumes that the camera can zoom and move in both horizontal and vertical planes, and it is also adequate for egomotion sequences without abrupt changes. In other words it is a non panoramic model for moving cameras, where the camera movement allows to reuse enough background information from the previous frame. Two pixel models are used, one to follow the camera movement and the other to detect foreground objects. A procedure is developed to transform and interpolate the covariance matrices of the Gaussian mixture components as the camera moves and zooms. Moreover, a background extrapolation method is presented in order to generate new mixture models for previously unseen regions. The proposal is compared with some state-of-the-art alternatives, with competitive results both quantitatively and qualitatively.

Configurable audio/video/physiological data telehealth platform designed for physical medicine and rehabilitation

Telehealth is defined as the use of information and communication technologies (ICT) to extend health care service delivery across distance. Remote vital sign monitoring is a common example for home telehealth and one promising market, but lacks the breadth, comforting and subtleties of face-to-face assessments usually done by having either the patient go to the clinic or the clinician going to see the patient at home. To conduct more lively and interactive sessions, virtual visits involve the use of video and audio for live and remote consultations between clinicians and patients [1] . These consultations are done through fixed workstations or mobile devices using web videoconferencing systems, which may not always be sufficient to the practice of clinicians [2] . To provide flexibility to the interactions that would normally take place in face-to-face consultations, we are developing a multi-point, multi-view and multi-data (video, audio, medical devices) videoconferencing system, called VIGIL, to conduct live sessions between health caregivers (e.g., physiotherapists, occupational therapists, doctors, nurses) and patients at home. Like Skype and OmniJoin/Nefsis, VIGIL is a cloud-based videoconferencing system working on mobile devices, laptops and workstations. VIGIL has customized and simple interfaces to improve usability for specific telehealth needs. For instance, it can transmit images from one or multiple cameras, control pan-tilt-zoom (PTZ) cameras from a distance, and interface a mobile communication badge and a mobile PTZ camera for interaction away from the workstation. It can also receive data from wireless medical devices. VIGIL is currently hosted at Sherbrooke Techno-Centre and exploited in trials are underway at the Centre de réadaptation de l’Estrie. Technologically, it is possible to provide a diversity of interaction modalities in support of telehealth practice. It is now more a matter of finding out what is necessary to address those needs.

Ray Saliency: Bottom-Up Visual Saliency for a Rotating and Zooming Camera

We extend the classical notion of computational visual saliency to multi-image data collected using a stationary pan-tilt-zoom (PTZ) camera by introducing the concept of consistency: the requirement that the set of generated saliency maps should each assign the same saliency value to unique regions of the environment that appear in more than one image. We show that processing each image independently will often fail to provide a consistent measure of saliency, and that using an image mosaic to quantify saliency suffers from several drawbacks. We then propose ray saliency and an immediate extension, approximate ray saliency: a mosaic-free method for calculating a consistent measure of bottom-up saliency. Experimental results demonstrating the effectiveness of the proposed approach are presented.

Moving object detection using unstable camera for video surveillance systems

This paper presents a robust moving object detection method by compensating for motion of an unstable camera. Assuming that global camera motion results in affine transform between two successive frames, local affine motions are separately estimated in multiple pre-specified regions for fast, robust estimation of the global motion. The global camera motion is then estimated by the least squares method using the pre-estimated multiple local affine motions. Given a current frame as the reference, the subsequent frame is registered to the current frame using the estimated global motion. The moving objects are finally detected using difference of Gaussian and non-parametric kernel density estimation from the set of registered three frames. Experimental results show that the proposed method can robustly detect moving objects in unstable imaging environment for intelligent surveillance systems using various types of cameras including pan-tilt-zoom (PTZ) and unmanned aerial vehicle (UAV) cameras.

Intrusion Detection and Cooperative Tracking Using PTZ Network Thermal Imagers

Nowadays more and more network thermal imaging cameras are working over distributed networks, offering the capability of online remote intelligent video surveillance. In this paper, we propose an original intrusion detection and cooperative tracking approach applied for PTZ (Pan/Tilt/Zoom) network thermal imagers. It consists of three modules. The key module is the real-time FOV (Field of View) matching module, which is realized in a parallel way. The intrusion detection module first eliminates the offset between current frame and prior frame through FOV matching and then handles intrusion detection by motion detection in the preset surveillance zone. The cooperative tracking module shifts the priority of tracking by imager pose estimation, which is also based on FOV matching, avoiding transferring the local features from one imager to another. Experiments are conducted to demonstrate that the proposed approach is of high accuracy for intrusion detection and cooperative tracking and keeps the frame rate over 20 fps.

SmartCamera: a low-cost and intelligent camera management system

Intelligent camera management systems were developed to automatically record meetings for videoconferencing. These systems provided many benefits, such as reducing the production cost and conveniently documenting events. However, automatically recorded videos in general were not visually engaging. This paper presents a novel approach that intelligently controls camera shots and angles to improve the visual interest. We use 3D infrared images captured by a Kinect sensor to recognize active speakers and their positions in a meeting. A movable camera, constructed by placing a wireless PTZ (pan-tilt-zoom) camera on top of a motorized rail, can automatically move its position to frame an active speaker in the center of the screen. Without interrupting the meeting, a speaker can seamlessly switch video sources through gesture-based commands. We have summarized and implemented a set of heuristic rules to simulate a human director. These rules can be visually edited through a graphical user interface. The customization of a virtual director makes our system applicable in various scenarios. We conducted a user study, and the evaluation results justified the quality of an automated video.

EAGLE-EYE: A DUAL-PTZ-CAMERA SYSTEM FOR TARGET TRACKING IN A LARGE OPEN AREA

An active camera, i.e., pan-tilt-zoom (PTZ) camera, can be used either to monitor a wide area or capture a high resolution image of a specific object by adjusting the zoom value. In order to achieve the above two goals simultaneously just like an eagle’s eye, a novel dual–PTZ-camera system, called Eagle-Eye, is proposed in this paper. The system can keep monitoring the whole area while tracking and focusing on the details of an object. Two techniques, moving object detection and fuzzy matching, are used alternatively for target tracking. According to the experimental results obtained with the implemented prototype, the success rates of tracking tasks for various moving speeds during daytime and nighttime are about 90 percent. The success rate with occlusion condition is also more than 80 percent. Furthermore, the average success rates with four special moving paths are 83.8 percent. These results show that Eagle-Eye system is feasible and achieves good tracking performance under various conditions.

A Real-Time Active Pedestrian Tracking System Inspired by the Human Visual System

Pedestrian detection and tracking play a significant role in surveillance. Despite the numerous detection and tracking methods proposed in the literature, when the pedestrian is too small to recognize, which is a common case in modern surveillance systems, all methods fail. In order to deal with such case, we propose an active pedestrian tracking system inspired by the human visual system. A coarse-to-fine pedestrian detection algorithm is proposed for the small pedestrian detection by combining the Gaussian mixture model background subtraction with the histogram of oriented gradient detection. In addition, a three-dimensional pan–tilt–zoom control model is presented, which requires no calibration and is more accurate than other control models. In order to actively track a pedestrian in real time, we utilize an active control algorithm and a tracking–learning–detection tracker. Experimental results demonstrate that our active tracking system is both efficient and effective.

Non-myopic information theoretic sensor management of a single pan–tilt–zoom camera for multiple object detection and tracking

Automatic multiple object tracking with a single pan–tilt–zoom (PTZ) cameras is a hard task, with few approaches in the literature, most of them proposing simplistic scenarios. In this paper, we present a novel PTZ camera management framework in which at each time step, the next camera pose (pan, tilt, focal length) is chosen to support multiple object tracking. The policy can be myopic or non-myopic, where the former analyzes exclusively the current frame for deciding the next camera pose, while the latter takes into account plausible future target displacements and camera poses, through a multiple look-ahead optimization. In both cases, occlusions, a variable number of subjects and genuine pedestrian detectors are taken into account, for the first time in the literature. Convincing comparative results on synthetic data, realistic simulations and real trials validate our proposal, showing that non-myopic strategies are particularly suited for a PTZ camera management.

Dynamic 3D surface reconstruction and motion modeling from a pan–tilt–zoom camera

3D surface reconstruction and motion modeling has been integrated in several industrial applications. Using a pan–tilt–zoom (PTZ) camera, we present an efficient method called dynamic 3D reconstruction (D3DR) for recovering the 3D motion and structure of a freely moving target. The proposed method estimates the PTZ measurements to keep the target in the center of the field of view (FoV) of the camera with the same size. Feature extraction and tracking approach are used in the imaging framework to estimate the target's translation, position, and distance. A selection strategy is used to select keyframes that show significant changes in target movement and directly update the recovered 3D information. The proposed D3DR method is designed to work in a real-time environment, not requiring all frames captured to be used to update the recovered 3D motion and structure of the target. Using fewer frames minimizes the time and space complexity required. Experimental results conducted on real-time video streams using different targets to prove the efficiency of the proposed method. The proposed D3DR has been compared to existing offline and online 3D reconstruction methods, showing that it uses less execution time than the offline method and uses an average of 49.6% of the total number of frames captured.

An Automatic Lecture Recording System Using Pan-Tilt-Zoom Camera to Track Lecturer and Handwritten Data

For automatic recording of the traditional lecture with a wide blackboard by using a Pan-Tilt-Zoom (PTZ) camera, a system, called ACRS (After Class Reviewing System), was proposed. From the point of view of students reviewing the lecture, the handwritten data is more important than the video of the lecturer. So, the handwritten data and the lecturer were considered together to record a lecture most suitable for student review. In the experimental study, the video recorded by ACRS was compared with that recorded manually by a cameraman. About 83% of the video time was close to the video recorded manually and suitable for review by students.

Consistent Labeling Approach for a PTZ Camera Based on Template Cache and Least Recently Used Replacement Strategy

Object tracking is an important function of video surveillance system. For the same object in a multi-camera environment, how to assign the same label to this object is so-called consistent labeling problem. Many consistent labeling approaches proposed by previous studies mainly based on the environment with multiple fixed cameras. In this study, a consistent labeling approach is proposed for a PTZ (Pan-Tilt-Zoom) camera. The same object is assigned the same label while the object in the FOV (Field-of-View) of a PTZ camera without influencing by the pan/tilt rotation. In order to achieve the above goal, the proposed approach using several methods, such as temporal differencing, template matching, mean-shift tracking, Kalman filter, and so on. A template cache is also designed for preserving the templates of an object with various angles and a least recently used (LRU) replacement mechanism is used to update the cache. The experimental results show that accuracy of the proposed approach for the consistent labeling of a PTZ camera can reach about 83 percentage.

Cooperative object tracking using dual‐pan–tilt–zoom cameras based on planar ground assumption

Pan–tilt–zoom (PTZ) cameras play an important role in visual surveillance system. Dual-PTZ camera system is the simplest and most typical one. The superiority of this system lies in that it can obtain both large-view information and high-resolution local-view information of the tracked object at the same time. One method to achieve such task is to use master–slave configuration. One camera (master) tracks moving objects at low resolution and provides the positional information to another camera (slave). Then the slave camera can point towards the object at high resolution and track it dynamically. In this paper, we propose a novel framework exploiting planar ground assumption to achieve cooperative tracking. The approach differs from conventional methods in that we exploit planar geometric constraint to solve the camera collaboration problem. Compared with the existing approach, the proposed framework can be used in the case of wide baseline, and allows the depth change of the tracked object. The proposed method can also adapt to the dynamic change of the surveillance scene. Besides, we also describe a self-calibration method of homography matrix which is induced by the ground plane between two cameras. We demonstrate the effectiveness of the proposed method by testing it with a tracking system for surveillance applications.

Large-Area, Multilayered, and High-Resolution Visual Monitoring Using a Dual-Camera System

Large-area, high-resolution visual monitoring systems are indispensable in surveillance applications. To construct such systems, high-quality image capture and display devices are required. Whereas high-quality displays have rapidly developed, as exemplified by the announcement of the 85-inch 4K ultrahigh-definition TV by Samsung at the 2013 Consumer Electronics Show (CES), high-resolution surveillance cameras have progressed slowly and remain not widely used compared with displays. In this study, we designed an innovative framework, using a dual-camera system comprising a wide-angle fixed camera and a high-resolution pan-tilt-zoom (PTZ) camera to construct a large-area, multilayered, and high-resolution visual monitoring system that features multiresolution monitoring of moving objects. First, we developed a novel calibration approach to estimate the relationship between the two cameras and calibrate the PTZ camera. The PTZ camera was calibrated based on the consistent property of distinct pan-tilt angle at various zooming factors, accelerating the calibration process without affecting accuracy; this calibration process has not been reported previously. After calibrating the dual-camera system, we used the PTZ camera and synthesized a large-area and high-resolution background image. When foreground targets were detected in the images captured by the wide-angle camera, the PTZ camera was controlled to continuously track the user-selected target. Last, we integrated preconstructed high-resolution background and low-resolution foreground images captured using the wide-angle camera and the high-resolution foreground image captured using the PTZ camera to generate a large-area, multilayered, and high-resolution view of the scene.

A Master-Slave Calibration Algorithm with Fish-Eye Correction

Surveillance systems capable of autonomously monitoring vast areas are an emerging trend, particularly when wide-angle cameras are combined with pan-tilt-zoom (PTZ) cameras in a master-slave configuration. The use of fish-eye lenses allows the master camera to maximize the coverage area while the PTZ acts as a foveal sensor, providing high-resolution images of regions of interest. Despite the advantages of this architecture, the mapping between image coordinates and pan-tilt values is the major bottleneck in such systems, since it depends on depth information and fish-eye effect correction. In this paper, we address these problems by exploiting geometric cues to perform height estimation. This information is used both for inferring 3D information from a single static camera deployed on an arbitrary position and for determining lens parameters to remove fish-eye distortion. When compared with the previous approaches, our method has the following advantages: (1) fish-eye distortion is corrected without relying on calibration patterns; (2) 3D information is inferred from a single static camera disposed on an arbitrary location of the scene.

Dual Dynamic PTZ Tracking Using Cooperating Cameras

This paper presents a real-time, dynamic system that uses high resolution gimbals and motorized lenses with position encoders on their zoom and focus elements to “recalibrate” the system as needed to track a target. Systems that initially calibrate for a mapping between pixels of a wide field of view (FOV) master camera and the pan-tilt (PT) settings of a steerable narrow FOV slave camera assume that the target is travelling on a plane. As the target travels through the FOV of the master camera, the slave cameras PT settings are then adjusted to keep the target centered within its FOV. In this paper, we describe a system we have developed that allows both cameras to move and extract the 3D coordinates of the target. This is done with only a single initial calibration between pairs of cameras and high-resolution pan-tilt-zoom (PTZ) platforms. Using the information from the PT settings of the PTZ platform as well as the precalibrated settings from a preset zoom lens, the 3D coordinates of the target are extracted and compared to those of a laser range finder and static-dynamic camera pair accuracies.

Scale-invariant optical flow in tracking using a pan-tilt-zoom camera

SUMMARYAn efficient method for tracking a target using a single Pan-Tilt-Zoom (PTZ) camera is proposed. The proposed Scale-Invariant Optical Flow (SIOF) method estimates the motion of the target and rotates the camera accordingly to keep the target at the center of the image. Also, SIOF estimates the scale of the target and changes the focal length relatively to adjust the Field of View (FoV) and keep the target appear in the same size in all captured frames. SIOF is a feature-based tracking method. Feature points used are extracted and tracked using Optical Flow (OF) and Scale-Invariant Feature Transform (SIFT). They are combined in groups and used to achieve robust tracking. The feature points in these groups are used within a twist model to recover the 3D free motion of the target. The merits of this proposed method are (i) building an efficient scale-invariant tracking method that tracks the target and keep it in the FoV of the camera with the same size, and (ii) using tracking with prediction and correction to speed up the PTZ control and achieve smooth camera control. Experimental results were performed on online video streams and validated the efficiency of the proposed method SIOF, comparing with OF, SIFT, and other tracking methods. The proposed SIOF has around 36% less average tracking error and around 70% less tracking overshoot than OF.

Distributed Constrained Optimization for Bayesian Opportunistic Visual Sensing

We propose a control mechanism to obtain opportunistic high resolution facial imagery, via distributed constrained optimization of the pan, tilt, zoom (PTZ) parameters for each camera in a sensor network. The objective function of the optimization problem quantifies the per camera per target image quality. The tracking constraints, which are a lower bound on the information about the estimated position for each target, define the feasible PTZ parameter space. Each camera alters its own PTZ settings. All cameras use information broadcast by neighboring cameras such that the PTZ parameters of all cameras are simultaneously optimized relative to the global objective. At certain times of opportunity, due to the configuration of the targets relative to the cameras, and the fact that each camera may track many targets, the camera network may be able to reconfigure itself to achieve the tracking specification for all targets with remaining degrees of freedom that can be used to obtain high-res facial images from desirable aspect angles for certain targets. The challenge is to define algorithms to automatically find these time instants, the appropriate imaging camera, and the appropriate parameter settings for all cameras to capitalize on these opportunities. The solution proposed herein involves a Bayesian formulation in a game theoretic setting. The Bayesian formulation automatically trades off objective maximization versus the risk of losing track of any target. This paper describes the problem and solution formulations, design of aligned local and global objective functions and the inequality constraint set, and development of a distributed Lagrangian consensus algorithm that allows cameras to exchange information and asymptotically converge on a pair of primal-dual optimal solutions. This paper presents the theoretical solution along with the simulation results.