Pan-tilt-zoom Camera Research Articles

Data-driven hierarchical learning approach for multi-point servo control of Pan–Tilt–Zoom cameras

Pan–Tilt–Zoom (PTZ) cameras, with their significant features of free rotation and zoom, are widely used in areas such as border security, ecological conservation, emergency management, and the military. PTZ cameras can achieve automatic monitoring of a selected area through multiple servo operations, known as multi-point servo control. However, due to the deficiencies in the servo control Software Development Kit (SDK), hardware wear, and interference from complex external environments, the multi-point servo control process generates significant errors. This paper proposes a precise multi-point servo control framework based on Deep Reinforcement Learning (DRL) to address this issue. The complexity of real-world environments necessitates a reward function design that fully considers various factors, for which we propose the directional gravity reward function. Due to the instability during the training process, prolonged trial-and-error interactions between the agent and the equipment can cause irreversible damage to the devices. This framework employs a phased training approach, where the agent learns sequentially from offline, off-policy, and on-policy data, reducing direct interaction with the equipment while enhancing the agent’s overall performance. Additionally, real-time and accurate device status can be obtained by performing feature matching on images from adjacent time frames, which is crucial for the system’s operation. Evaluation results indicate that our proposed precise multi-point servo control framework significantly outperforms other methods in 4-point servo control tasks in both virtual and real-world scenarios. Additionally, the operational process fully considers safety issues.

Towards Fully Autonomous Drone Tracking by a Reinforcement Learning Agent Controlling a Pan–Tilt–Zoom Camera

Towards Fully Autonomous Drone Tracking by a Reinforcement Learning Agent Controlling a Pan–Tilt–Zoom Camera

Active Visual Perception Enhancement Method Based on Deep Reinforcement Learning

Traditional object detection methods using static cameras are constrained by their limited perspectives, hampering the effective detection of low-confidence targets. To address this challenge, this study introduces a deep reinforcement learning-based visual perception enhancement technique. This approach leverages pan–tilt–zoom (PTZ) cameras to achieve active vision, enabling them to autonomously make decisions and actions tailored to the current scene and object detection outcomes. This optimization enhances both the object detection process and information acquisition, significantly boosting the intelligent perception capabilities of PTZ cameras. Experimental findings demonstrate the robust generalization capabilities of this method across various object detection algorithms, resulting in an average confidence level improvement of 23.80%.

Real-time risk assessment of road vehicles based on inverse perspective mapping

Pan/Tilt/Zoom (PTZ) cameras play an important role in traffic scenes due to their wide monitoring fields and high flexibility. However, since the focal length and angle of PTZ cameras change irregularly with the monitoring needs, it is difficult to obtain accurate physical information about the real world from the image information of PTZ cameras. Aiming to address the need for real-time risk assessment of road vehicles in traffic monitoring scenarios, a vehicle position and velocity measurement scheme based on camera inverse perspective transformation is proposed, along with a method for real-time risk assessment based on the position and velocity. Specifically, Firstly, the vehicle target in the video is detected and tracked by deep learning YOLO detection algorithm and optical flow tracking algorithm. According to the obtained trajectory set, the vanishing points in the road direction are calculated by Cascade Hough Transform and the road marking lines are detected. Then, according to the vanishing point and marking line, the camera calibration task is accomplished via exploratory focal length. After camera calibration, the camera-to-road inverse perspective transformation is applied to project the image plane onto the road surface and obtain, the actual position information of vehicles. Finally, the vehicle speed measurement and real-time road risk assessment are achieved by calculating the average of instantaneous velocities across multiple frames. Simulation experiment results in a traffic monitoring scenario demonstrate that this perspective-based method for real-time road vehicle risk assessment achieves good stability and practicality, which meets the requirements for vehicle speed measurement and real-time road risk assessment.

Crack inspection in tunnel structures by fusing information from a 3D light detection and ranging and pan-tilt-zoom camera system

This study proposes a practical crack inspection method for tunnel structures by fusing measurements from two novel sensors: 3D light detection and ranging (LiDAR) and pan-tilt-zoom (PTZ) camera. The proposed method comprises three phases: prerequisite phase, real-time measurement phase, and inspection phase. First, the 3D LiDAR and the PTZ camera is combined to generate 3D point cloud data (PCD) with RGB data prior to extract the plane contained the crack. Second, the PTZ camera tracked the crack of interest at the pre-registered location. Specifically, several signal processing methods including the Hough transform and random sample consensus are addressed to effectively recognize the crack of interest. Then, the pan and tilt of the camera are controlled to track the crack of interest and record the crack at several magnifications of the zoom automatically because the optimal magnification value of the zoom is unknown in practical field applications. Third, crack propagation rate is estimated by calculating the length of the crack using information on the crack recorded in previous measurements. Cracks in a recorded image are extracted by addressing a multiscale multilevel mask deep convolutional neural network (MSML Mask DCNN) and pixel-based clustering method. The MSML Mask DCNN constructs distinct feature maps to effectively extract features of cracks and the pixel-based clustering method effectively eliminate cracks not of interest to accurately calculate the crack propagation rate for the crack of interest. Finally, the crack propagation rate was calculated through statistical analysis with several images at different zoom magnifications to secure high accuracy. Field experiments demonstrate that the proposed method is effective to inspect cracks. Systematic analysis on experiments also revealed that all methods deployed in the crack inspection enable ensures high accuracy and robustness in field applications. Consequently, the proposed crack inspection system and method provide a cost-effective and autonomous solution for crack inspection of tunnel structures.

Marker-Based Localization System Using an Active PTZ Camera and CNN-Based Ellipse Detection

Localization in GPS-denied environments is challenging, and many existing solutions have infrastructural and on-site calibration requirements. This article tackles these challenges by proposing a localization system that is infrastructure free and does not require on-site calibration, using a single active pan–tilt–zoom camera to detect, track, and localize a circular LED marker. We propose to use a convolutional neural network (CNN) trained using only synthetic images to detect the LED marker as an ellipse and show that our approach is more robust than using traditional ellipse detection without requiring tuning of parameters for feature extraction. We also propose to leverage the predicted elliptical angle as a measure of uncertainty of the CNN's predictions and show how it can be used in a filter to improve marker range estimation and 3-D localization. We evaluate our system's performance through localization of a unmanned aerial vehicle in real-world flight experiments and show that it can outperform alternative methods for localization in GPS-denied environments. We also demonstrate our system's performance in indoor and outdoor environments.

WLTOG: An optimization approach for wild large‐range target omnidirectional geolocation based on monocular PTZ camera

AbstractThe pan‐tilt‐zoom (PTZ) cameras are increasingly popular in urban management and resource protection due to their omnidirectional coverage and reasonable cost. Traditionally, the research of monocular camera geolocation focuses on small‐range ones with a central projection camera under ideal conditions, making it challenging to meet the requirements of omnidirectional coverage and large‐range geolocation for PTZ cameras. In this article, a wild large‐range target omnidirectional geolocation (WLTOG) algorithm was proposed. Firstly, the WLTOG algorithm is used to transform the target in a single image to the camera's optical axis, thereby obtaining the camera posture value. After that, by using a high‐order overdetermined equation, it fits the nonlinear mapping relationship between multiple pairs of PTZ camera postures and the geospatial coordinates. At last, it realizes the bidirectional calculation between pixel coordinates and geospatial coordinates. The experimental PTZ camera is mounted on a communication tower in a wild environment. Therefore, this method can be utilized to construct an omnidirectional geolocation relationship that meets the requirements of omnidirectional geolocation within 945 m based on unknown accurate internal and external orientation elements. The WLTOG algorithm is conducive to the geometric measurement of video and GIS‐augmented video surveillance.

Step-by-step optimisation of robotic-assisted radical prostatectomy using augmented reality

ABSTRACTIntroduction:Surgical training will be complemented by digitalisation, as the COVID 19 pandemic continues (1). Proximie is an augmented reality (AR) platform that can display up to 4 native camera views, with live or semi live telementoring. It can optimise ergonomics of the surgeon at the console (2), and robotic instrument orientation. We describe the utilisation of Proximie as a step-by-step guide in a robotic assisted radical prostatectomy (RARP).Surgical Technique:Author V. P. performed a transperitoneal multiport da Vinci Xi RARP with the Proximie platform: a laptop computer, multiple HD webcams, microphones and speakers. Using an HDMI cable to the Intuitive Surgical tower, output display from the console and an additional laparoscopic tower is shown. Each webcam was mounted to the side armrests of the console, directed at the surgeon's hands. An independent ‘drop in’ laparoscope via an additional 5mm left upper quadrant port was utilised. Observers can visualise the AR platform's recordings on a laptop and/or smartphone. A PTZ (pan-tilt-zoom) camera can capture the operating room, bedside assistant, ports and patient position. Our video demonstrates three of four camera views for posture, forearm, wrist, hand, and finger orientation, relative to the translated robotic steps. A pincer grasp of the endowrist manipulator during anastomosis allows optimal robotic wrist rotation. The second laparoscopic camera view demonstrated intracorporeal angles of robotic arm and bedside assistant's instrument position for critical steps such as nerve sparing and anastomosis (3). The console time was 100 minutes, no intraoperative complications, or delay in image transmission occurred with utilising the platform.Considerations:An AR platform can create deeper learning for RARP in real time or recorded sessions. Two-way verbal and visual communication with ability to annotate on screen, allows long distance mentoring. The platform's utility can be accessed in anywhere, to project surgeons beyond their immediate environment. This allows for democratisation of access to high volume institutions and their evolution of techniques (4), to assist patients globally. Potential developments are artificial intelligence (AI) networks analysing repository of such recorded data, to identify intraoperative hand motion and robotic instrument tracking. AR is a pertinent building block to enhance robotic training, skill dissemination, precision medicine (5) and surgery overall.

An Autonomous System for Efficient Control of PTZ Cameras

This article addresses the research problem of how to autonomously control Pan/Tilt/Zoom (PTZ) cameras in a manner that seeks to optimize the face recognition accuracy or the overall threat detection and proposes an overall system. The article presents two alternative schemes for camera scheduling: Grid-Based Grouping (GBG) and Elevator-Based Planning (EBP). The camera control works with realistic 3D environments and considers many factors, including the direction of the subject’s movement and its location, distances from the cameras, occlusion, overall recognition probability so far, and the expected time to leave the site, as well as the movements of cameras and their capabilities and limitations. In addition, the article utilizes clustering to group subjects, thereby enabling the system to focus on the areas that are more densely populated. Moreover, it proposes a dynamic mechanism for controlling the pre-recording time spent on running the solution. Furthermore, it develops a parallel algorithm, allowing the most time-consuming phases to be parallelized, and thus run efficiently by the centralized parallel processing subsystem. We analyze through simulation the effectiveness of the overall solution, including the clustering approach, scheduling alternatives, dynamic mechanism, and parallel implementation in terms of overall recognition probability and the running time of the solution, considering the impacts of numerous parameters.

Anomalous object detection by active search with PTZ cameras

Due to the large amount of visual information generated daily, proposals that automatically analyze and process data are becoming increasingly necessary. This work focuses on the detection of anomalous objects in video sequences captured by PTZ (pan-tilt-zoom) cameras, considering as anomalies the objects which belong to categories that should not appear in a specific scene (e.g. pedestrians on a highway). There is a lack in the previous literature of a principled approach for the control of PTZ cameras that takes advantage of the recent developments in deep learning-based object detection. Our proposal aims to fill this gap by offering a probabilistic framework where the guidance of PTZ cameras is accommodated. The proposed methodology involves three different modules. An object detection stage, where deep learning networks are used to detect the objects that appear in the scene; an anomalous detection module, where a mixture of Dirichlet distributions is considered to detect automatically, and without supervised training, those detected objects which are likely to be anomalous; and finally, a PTZ camera controller which allows to follow and focus on the object considered as the most probably anomalous in the scene. The experimental results show the performance and viability of our proposal, which outperforms several competitors from qualitative and quantitative points of view.

Hospital El Salvador: a novel paradigm of intensive care in response to COVID-19 in central America

Hospital El Salvador: a novel paradigm of intensive care in response to COVID-19 in central America

Use of Pan Tilt Zoom camera in Lattismus dorsi flap monitoring – Our experience

Tele-monitoring is an emerging concept in the field of medicine, thereby avoiding various complications that the patients face due to repeated visits like cross-infections and circumvents disadvantages that are faced by the medical person due to requirement of repeated rounds. At the present state of a pandemic, safe distancing is essential. In such a situation, tele-monitoring comes into play, bridging the gap between the patient and doctor at this tough time.

Automation of Recording in Smart Classrooms via Deep Learning and Bayesian Maximum a Posteriori Estimation of Instructor's Pose

Internet of Things is making objects smarter and more autonomous. At the other side, online education is gaining momentum and many universities are now offering online degrees. Content preparation for such programs usually involves recording the classes. In this article, we intend to introduce a deep learning-based camera management system as a substitute for the academic filming crew. The solution mainly consists of two cameras and a wearable gadget for the instructor. The fixed camera is used for the instructor's position and pose detection and the pan-tilt-zoom (PTZ) camera does the filming. In the proposed solution, image processing and deep learning techniques are merged together. Face recognition and skeleton detection algorithms are used to detect the position of instructor. But the main contribution lies in the application of deep learning for instructor's skeleton detection and postprocessing of the deep network output for correction of the pose detection results using a Bayesian Maximum A Posteriori (MAP) estimator. This estimator is defined on a Markov state machine. The pose detection result along with the position info is then used by the PTZ camera controller for filming purposes. The proposed solution is implemented by using OpenPose which is a convolutional neural network for detection of body parts. Feeding a neural network pose classifier with 12 features extracted from the output of the deep network yields an accuracy of 89%. However, as we show, the accuracy can be improved by the Markov model and MAP estimator to reach as high as 95.5%.

Deep Features Homography Transformation Fusion Network-A Universal Foreground Segmentation Algorithm for PTZ Cameras and a Comparative Study.

The foreground segmentation method is a crucial first step for many video analysis methods such as action recognition and object tracking. In the past five years, convolutional neural network based foreground segmentation methods have made a great breakthrough. However, most of them pay more attention to stationary cameras and have constrained performance on the pan–tilt–zoom (PTZ) cameras. In this paper, an end-to-end deep features homography transformation and fusion network based foreground segmentation method (HTFnetSeg) is proposed for surveillance videos recorded by PTZ cameras. In the kernel of HTFnetSeg, there is the combination of an unsupervised semantic attention homography estimation network (SAHnet) for frames alignment and a spatial transformed deep features fusion network (STDFFnet) for segmentation. The semantic attention mask in SAHnet reinforces the network to focus on background alignment by reducing the noise that comes from the foreground. STDFFnet is designed to reuse the deep features extracted during the semantic attention mask generation step by aligning the features rather than only the frames, with a spatial transformation technique in order to reduce the algorithm complexity. Additionally, a conservative strategy is proposed for the motion map based post-processing step to further reduce the false positives that are brought by semantic noise. The experiments on both CDnet2014 and Lasiesta show that our method outperforms many state-of-the-art methods, quantitively and qualitatively.

Geometric Recognition of Moving Objects in Monocular Rotating Imagery Using Faster R-CNN

Moving object detection and tracking from image sequences has been extensively studied in a variety of fields. Nevertheless, observing geometric attributes and identifying the detected objects for further investigation of moving behavior has drawn less attention. The focus of this study is to determine moving trajectories, object heights, and object recognition using a monocular camera configuration. This paper presents a scheme to conduct moving object recognition with three-dimensional (3D) observation using faster region-based convolutional neural network (Faster R-CNN) with a stationary and rotating Pan Tilt Zoom (PTZ) camera and close-range photogrammetry. The camera motion effects are first eliminated to detect objects that contain actual movement, and a moving object recognition process is employed to recognize the object classes and to facilitate the estimation of their geometric attributes. Thus, this information can further contribute to the investigation of object moving behavior. To evaluate the effectiveness of the proposed scheme quantitatively, first, an experiment with indoor synthetic configuration is conducted, then, outdoor real-life data are used to verify the feasibility based on recall, precision, and F1 index. The experiments have shown promising results and have verified the effectiveness of the proposed method in both laboratory and real environments. The proposed approach calculates the height and speed estimates of the recognized moving objects, including pedestrians and vehicles, and shows promising results with acceptable errors and application potential through existing PTZ camera images at a very low cost.

Foveation Pipeline for 360° Video-Based Telemedicine.

Pan-tilt-zoom (PTZ) and omnidirectional cameras serve as a video-mediated communication interface for telemedicine. Most cases use either PTZ or omnidirectional cameras exclusively; even when used together, images from the two are shown separately on 2D displays. Conventional foveated imaging techniques may offer a solution for exploiting the benefits of both cameras, i.e., the high resolution of the PTZ camera and the wide field-of-view of the omnidirectional camera, but displaying the unified image on a 2D display would reduce the benefit of “omni-” directionality. In this paper, we introduce a foveated imaging pipeline designed to support virtual reality head-mounted displays (HMDs). The pipeline consists of two parallel processes: one for estimating parameters for the integration of the two images and another for rendering images in real time. A control mechanism for placing the foveal region (i.e., high-resolution area) in the scene and zooming is also proposed. Our evaluations showed that the proposed pipeline achieved, on average, 17 frames per second when rendering the foveated view on an HMD, and showed angular resolution improvement on the foveal region compared with the omnidirectional camera view. However, the improvement was less significant when the zoom level was 8× and more. We discuss possible improvement points and future research directions.

Video-Based Vehicle Counting for Expressway: A Novel Approach Based on Vehicle Detection and Correlation-Matched Tracking Using Image Data from PTZ Cameras

Vehicle counting plays a significant role in vehicle behavior analysis and traffic incident detection for established video surveillance systems on expressway. Since the existing sensor method and the traditional image processing method have the problems of difficulty in installation, high cost, and low precision, a novel vehicle counting method is proposed, which realizes efficient counting based on multivehicle detection and multivehicle tracking. For multivehicle detection tasks, a construction of the new expressway dataset consists of a large number of sample images with a high resolution (1920 × 1080) captured from real-world expressway scenes (including the diversity climatic conditions and visual angles) by Pan-Tilt-Zoom (PTZ) cameras, in which vehicle categories and annotation rules are defined. Moreover, a correlation-matched algorithm for multivehicle tracking is proposed, which solves the problem of occlusion and vehicle scale change in the tracking process. Due to the discontinuity and unsmooth of the trajectories that occurred during the tracking process, we designed a trajectory optimization algorithm based on least square method. Finally, a new vehicle counting method is designed based on the tracking results, in which the driving direction information of the vehicle is added in the counting process. The experimental results show that the proposed counting method in this research can achieve more than 93% accuracy and an average speed of 25 frames per second in expressway video sequence.

Fusing Self-Organized Neural Network and Keypoint Clustering for Localized Real-Time Background Subtraction.

Moving object detection in video streams plays a key role in many computer vision applications. In particular, separation between background and foreground items represents a main prerequisite to carry out more complex tasks, such as object classification, vehicle tracking, and person re-identification. Despite the progress made in recent years, a main challenge of moving object detection still regards the management of dynamic aspects, including bootstrapping and illumination changes. In addition, the recent widespread of Pan-Tilt-Zoom (PTZ) cameras has made the management of these aspects even more complex in terms of performance due to their mixed movements (i.e. pan, tilt, and zoom). In this paper, a combined keypoint clustering and neural background subtraction method, based on Self-Organized Neural Network (SONN), for real-time moving object detection in video sequences acquired by PTZ cameras is proposed. Initially, the method performs a spatio-temporal tracking of the sets of moving keypoints to recognize the foreground areas and to establish the background. Then, it adopts a neural background subtraction, localized in these areas, to accomplish a foreground detection able to manage bootstrapping and gradual illumination changes. Experimental results on three well-known public datasets, and comparisons with different key works of the current literature, show the efficiency of the proposed method in terms of modeling and background subtraction.

An Automatic Analog Instrument Reading System Using Computer Vision and Inspection Robot

Since the manual inspection of analog instruments is inefficient, many computer vision-based automatic reading systems have been proposed recently. However, most of them use fixed cameras that are costly due to a large number of used cameras. Although some other systems adopting the pan–tilt–zoom camera and the movable inspection robot can avoid using many cameras, they have to overcome high computational cost in aligning the camera to the tested instrument. Meanwhile, most existing systems are instrument type-dependent and, hence, cannot handle multiple types of instruments simultaneously. In this article, first, based on an inspection robot, an automatic reading system equipped with a pan–tilt–zoom camera is designed for different types of round-shape analog instruments. Then, a fast camera alignment algorithm based on visual servo is proposed, in which YOLOv3 is applied and improved to locate the instrument and guide the camera to iteratively align with the instrument. Finally, a monocular-vision pointer reconstruction algorithm is proposed to accurately read the instrument. Experimental results demonstrated that our proposed system is fast and reliable in the camera-alignment process and is effective in reading different types of analog instruments during the robot-based inspection.

Accurate and Practical Calibration of Multiple Pan-Tilt-Zoom Cameras for Live Broadcasts

We aim to realize novel services for live broadcasts, including visualizing the 3D position of a subject, using multiple pan-tilt-zoom (PTZ) cameras. For this purpose, a camera calibration technique with high accuracy over a wide pan-tilt-zoom range and high operability is required. Herein, we propose an accurate and practical method to calibrate multiple PTZ cameras for live broadcasts. This method models the PTZ camera system and accurately pre-estimates unknown parameters in the model. After the pre-estimation, accurate camera parameters can be calculated in real time using the pan-tilt angles and zoom values acquired from the sensors in the camera system. The proposed method accurately estimates the unknown parameters for a high-magnification zoom lens by making repeated initial guesses and nonlinear optimizations for high accuracy, and consists of a two-stage camera calibration for operability. We demonstrated the effectiveness of the proposed method by conducting experiments on computer graphics data and real data. We also applied it to a live sports graphics system.