Camera Motion Compensation Research Articles

A robust method for multi object tracking in autonomous ship navigation systems

A novel Maritime Multi-Object Tracking method is proposed, combining a deep learning-based object detector with target association algorithms to achieve robust sea-surface multi-object tracking. Specifically, the Multi-Object Tracking employs You Only Look Once version 7 detector for object detection. In the data association part, a module for onboard camera motion compensation is developed, a maritime dynamic spatial information-based intersection-over-union is presented as a similarity metric, and a progressive refinement cascade matching strategy is designed to enhance the tracker's sea-surface multi-object tracking capabilities. The Jari Maritime Tracking Dataset is utilised to validate the effectiveness performance of the proposed method. Experimental results demonstrate that compared to the earlier process, the proposed method exhibits a significant enhancement in multiple object tracking accuracy, with an increase of 27.8% and an achieved score of 81.3. In particular, it reduces the number of identifications switching and missed targets, achieving holistically preferable performance. Meanwhile, the speed of the proposed Multi-Object Tracking fulfils the engineering application requirements for an autonomous ship navigation system.

Computer Vision and Augmented Reality for Human-Centered Fatigue Crack Inspection.

A significant percentage of bridges in the United States are serving beyond their 50-year design life, and many of them are in poor condition, making them vulnerable to fatigue cracks that can result in catastrophic failure. However, current fatigue crack inspection practice based on human vision is time-consuming, labor intensive, and prone to error. We present a novel human-centered bridge inspection methodology to enhance the efficiency and accuracy of fatigue crack detection by employing advanced technologies including computer vision and augmented reality (AR). In particular, a computer vision-based algorithm is developed to enable near-real-time fatigue crack detection by analyzing structural surface motion in a short video recorded by a moving camera of the AR headset. The approach monitors structural surfaces by tracking feature points and measuring variations in distances between feature point pairs to recognize the motion pattern associated with the crack opening and closing. Measuring distance changes between feature points, as opposed to their displacement changes before this improvement, eliminates the need of camera motion compensation and enables reliable and computationally efficient fatigue crack detection using the nonstationary AR headset. In addition, an AR environment is created and integrated with the computer vision algorithm. The crack detection results are transmitted to the AR headset worn by the bridge inspector, where they are converted into holograms and anchored on the bridge surface in the 3D real-world environment. The AR environment also provides virtual menus to support human-in-the-loop decision-making to determine optimal crack detection parameters. This human-centered approach with improved visualization and human-machine collaboration aids the inspector in making well-informed decisions in the field in a near-real-time fashion. The proposed crack detection method is comprehensively assessed using two laboratory test setups for both in-plane and out-of-plane fatigue cracks. Finally, using the integrated AR environment, a human-centered bridge inspection is conducted to demonstrate the efficacy and potential of the proposed methodology.

Multi-Object Tracking Model Based on Detection Tracking Paradigm in Panoramic Scenes

Multi-Object Tracking (MOT) technology is dedicated to continuously tracking multiple targets of interest in a sequence of images and accurately identifying their specific positions at different times. This technology is crucial in key application areas such as autonomous driving and security surveillance. However, the application process often requires the coordination of cameras from multiple angles for tracking. Directly studying Multi-Object Tracking algorithms in panoramic scenes is an effective way to address this issue. The uniqueness of panoramic scenes causes target position changes at the boundaries and tracking difficulties due to continuous changes in target scales. To ensure the accuracy of target tracking, this study explores a detection-based tracking method using the newly improved YOLOx detector and the adjusted DeepSORT algorithm. Firstly, YOLOx_s was chosen as the detector because its simple network structure ensures a fast computational speed. During the feature extraction stage, we used the Polarized Self-Attention (PSA) mechanism to capture more feature information, thereby improving the tracking performance on small-scale targets. Secondly, the tracker was improved by adding a camera motion compensation module before predicting the target’s position to mitigate the impact of camera shake on tracking. Finally, to address the difficulty of continuously tracking targets in specific areas of panoramic scenes, this study proposes specific tracking strategies. These strategies effectively resolve the problem of tracking failure caused by target position changes at the boundaries. Experimental results show that improved algorithms have a superior performance on multiple evaluation metrics compared to other algorithms in the field. Compared to the original algorithm, the improved algorithm exhibits a 6% increase in the quantitative metric MOTA, a 7% increase in IDF1, and a 40% decrease in IDSWs, demonstrating its leading performance.

BoostTrack: boosting the similarity measure and detection confidence for improved multiple object tracking

Handling unreliable detections and avoiding identity switches are crucial for the success of multiple object tracking (MOT). Ideally, MOT algorithm should use true positive detections only, work in real-time and produce no identity switches. To approach the described ideal solution, we present the BoostTrack, a simple yet effective tracing-by-detection MOT method that utilizes several lightweight plug and play additions to improve MOT performance. We design a detection-tracklet confidence score and use it to scale the similarity measure and implicitly favour high detection confidence and high tracklet confidence pairs in one-stage association. To reduce the ambiguity arising from using intersection over union (IoU), we propose a novel Mahalanobis distance and shape similarity additions to boost the overall similarity measure. To utilize low-detection score bounding boxes in one-stage association, we propose to boost the confidence scores of two groups of detections: the detections we assume to correspond to the existing tracked object, and the detections we assume to correspond to a previously undetected object. The proposed additions are orthogonal to the existing approaches, and we combine them with interpolation and camera motion compensation to achieve results comparable to the standard benchmark solutions while retaining real-time execution speed. When combined with appearance similarity, our method outperforms all standard benchmark solutions on MOT17 and MOT20 datasets. It ranks first among online methods in HOTA metric in the MOT Challenge on MOT17 and MOT20 test sets. We make our code available at https://github.com/vukasin-stanojevic/BoostTrack.

UCMCTrack: Multi-Object Tracking with Uniform Camera Motion Compensation

Multi-object tracking (MOT) in video sequences remains a challenging task, especially in scenarios with significant camera movements. This is because targets can drift considerably on the image plane, leading to erroneous tracking outcomes. Addressing such challenges typically requires supplementary appearance cues or Camera Motion Compensation (CMC). While these strategies are effective, they also introduce a considerable computational burden, posing challenges for real-time MOT. In response to this, we introduce UCMCTrack, a novel motion model-based tracker robust to camera movements. Unlike conventional CMC that computes compensation parameters frame-by-frame, UCMCTrack consistently applies the same compensation parameters throughout a video sequence. It employs a Kalman filter on the ground plane and introduces the Mapped Mahalanobis Distance (MMD) as an alternative to the traditional Intersection over Union (IoU) distance measure. By leveraging projected probability distributions on the ground plane, our approach efficiently captures motion patterns and adeptly manages uncertainties introduced by homography projections. Remarkably, UCMCTrack, relying solely on motion cues, achieves state-of-the-art performance across a variety of challenging datasets, including MOT17, MOT20, DanceTrack and KITTI. More details and code are available at https://github.com/corfyi/UCMCTrack.

JDECMC: Improving JDE based multi-object tracking with Camera Motion Compensation

Jointly Detection and Embedding (JDE) based approaches estimate bounding boxes and embedding features of objects with a single network in multi-object tracking. Due to the crowded scenarios or non-rigid camera motion, the bounding box location in the image can shift dramatically, which affects the performance of the trackers. In addition, JDE-based approaches fuse the target appearance information and location information by applying the same rule, which could fail to associate when the target is lost or occluded. To address these issues, we propose a simple yet effective architecture based on JDE, called JDECMC. Our approach incorporates the embedding and location distance of the object, which combines embedding cosine distance and location distance of objects, as well as Camera Motion Compensation (CMC) to predict the correct location of the bounding box. This is especially useful in scenarios where dynamic camera motion can cause the bounding box to shift significantly. By enhancing the accuracy of the tracking system, JDECMC achieves a MOTA of 74.2%, 72.0%, and 90.5% on the MOT17, MOT20 and DanceTrack benchmark datasets, respectively, while also scoring an IDF1 of 72.5%, 72.4%, and 79.8% on those benchmarks. The experimental results demonstrate our effectiveness and advantages over the other one-shot trackers. The code is available at https://github.com/Melikamuliyih/JDECMC.

Robust multiple obstacle tracking method based on depth aware OCSORT for agricultural robots

Multiple Object Tracking (MOT) of dynamic obstacles in field is an important prerequisite for agricultural robots to achieve dynamic obstacle avoidance. The complex and unpredictable road environment in the rural areas will cause severe vibration to the robot, affecting the camera pose and consequently leading to object matching errors. Therefore, we propose an improved method named Depth Aware Observation Centric Simple Online and Realtime Tracking (DA-OCSORT) with two new modules named Inertial Measurement Unit (IMU)-based Camera Motion Compensation (ICMC) and Depth Aware (DA). This method can use IMU information to compensate for camera ego-motion and perform multi-dimensional matching through object depth information, thereby minimizing the influence of camera motion on tracking process. We created an open-source MOT dataset, named MOTorchard. Building upon the pure motion-based method OCSORT, we achieve SOTA on this MOTorchard with HOTA 54.1, MOTA 83.3, IDF1 52.9 and FPS 22.7. Moreover, the proposed two modules prove to have good real-time performance and strong adaptability to other published MOT algorithms.

Development and Validation of a Framework for Smart Wireless Strain and Acceleration Sensing

Civil infrastructure worldwide is subject to factors such as aging and deterioration. Structural health monitoring (SHM) can be used to assess the impact of these processes on structural performance. SHM demands have evolved from routine monitoring to real-time and autonomous assessment. One of the frontiers in achieving effective SHM systems has been the use of wireless smart sensors (WSSs), which are attractive compared to wired sensors, due to their flexibility of use, lower costs, and ease of long-term deployment. Most WSSs use accelerometers to collect global dynamic vibration data. However, obtaining local behaviors in a structure using measurands such as strain may also be desirable. While wireless strain sensors have previously been developed by some researchers, there is still a need for a high sensitivity wireless strain sensor that fully meets the general demands for monitoring large-scale civil infrastructure. In this paper, a framework for synchronized wireless high-fidelity acceleration and strain sensing, which is commonly termed multimetric sensing in the literature, is proposed. The framework is implemented on the Xnode, a next-generation wireless smart sensor platform, and integrates with the strain sensor for strain acquisition. An application of the multimetric sensing framework is illustrated for total displacement estimation. Finally, the potential of the proposed framework integrated with vision-based measurement systems for multi-point displacement estimation with camera-motion compensation is demonstrated. The proposed approach is verified experimentally, showing the potential of the developed framework for various SHM applications.

Multi-modal feature fusion for better understanding of human personality traits in social human–robot interaction

Since the dynamic nature of human–robot interaction becomes increasingly prevalent in our daily life, there is a great demand for enabling the robot to better understand human personality traits and inspiring humans to be more engaged in the interaction with the robot. Therefore, in this work, as we design the paradigm of human–robot interaction as close to the real situation as possible, the following three main problems are addressed: (1) fusion of visual and audio features of human interaction modalities, (2) integration of variable length feature vectors, and (3) compensation of shaky camera motion caused by movements of the robot’s communicative gesture. Specifically, the three most important visual features of humans including head motion, gaze, and body motion were extracted from a camera mounted on the robot performing verbal and body gestures during the interaction. Then, our system was geared to fuse the aforementioned visual features and different types of vocal features, such as voice pitch, voice energy, and Mel-Frequency Cepstral Coefficient, dealing with variable length multiple feature vectors. Lastly, considering unknown patterns and sequential characteristics of human communicative behavior, we proposed a multi-layer Hidden Markov Model that improved the classification accuracy of personality traits and offered notable advantages of fusing the multiple features. The results were thoroughly analyzed and supported by psychological studies. The proposed multi-modal fusion approach is expected to deepen the communicative competence of social robots interacting with humans from different cultures and backgrounds.

Infrared video image stabilization algorithm based on joint camera path

To solve the problem of infrared video image stabilization caused by parallax variation, the infrared video image stabilization algorithm based on joint camera path was proposed, which included four steps: preprocessing, joint camera path solving, multi-path optimization and motion compensation. Firstly, histogram equalization, feature point extraction, matching processing and pre-mapping were needed for infrared image. Then, each frame of image was divided into m×n grid cells, and local homography matrixs obtained from the corresponding grid in each frame were multiplied by the mapping motion representation based on the grid to obtain the joint camera path. Then, the path of the joint camera was smoothed by the strategy of ‘multi-path optimization’. Finally, the smooth path was used to stabilize the video. The experimental results show that this method can effectively deal with the nonlinear motion caused by parallax, which is better than the traditional image stabilization algorithm, and can achieve good image stabilization even when the feature points have partial occlusion.

Efficient development of vision-based dense three-dimensional displacement measurement algorithms using physics-based graphics models

This research investigates a framework for the efficient development of vision-based dense three-dimensional displacement measurement algorithms to support reliable structural health monitoring of civil structures. The framework exploits the use of a photo-realistic synthetic model, termed a physics-based graphics model, to simulate the entire process of vision-based measurement. At the same time, the synthetic environment is used to evaluate the performance of different post-processing algorithms quantitatively for a given measurement scenario, such as camera selection and camera placement. The effectiveness of the framework is demonstrated by optimizing the algorithms for the three-dimensional displacement measurement of a 14-bay laboratory truss structure. The vision-based dense three-dimensional displacement estimation algorithms optimized in this study consist of four steps: (1) camera parameter estimation, (2) camera motion estimation and compensation, (3) vision-based two-dimensional tracking, and (4) projection of two-dimensional tracking results to three-dimensional space. The algorithms use the knowledge from the finite element model to facilitate the implementation and maximize the measurement outcome, that is, model-informed approach. To test and evaluate the model-informed approach, synthetic videos are rendered for two measurement scenarios, that is, using a Digital Single Lens Reflex camera mounted on a tripod and using an Unmanned Aerial Vehicle camera. Then, the performance of the model-informed approach is evaluated by comparing the estimated displacement with the ground truth values. Based on the performance evaluation, an algorithm with the highest expected performance is selected for each of the two measurement scenarios. Finally, the selected algorithm is tested in a laboratory experiment. In contrast to the existing literature that investigates fixed individual measurement scenarios, the proposed framework can be used to test different measurement scenarios and estimate the outcome of each scenario before performing actual tests, facilitating the implementation of vision-based measurement for the structural health monitoring of civil structures.

Mixed Road User Trajectory Extraction From Moving Aerial Videos Based on Convolution Neural Network Detection

Vehicle trajectory data under mixed traffic conditions provides critical information for urban traffic flow modeling and analysis. Recently, the application of unmanned aerial vehicles (UAV) creates a potential of reducing traffic video collection cost and enhances flexibility at the spatial-temporal coverage, supporting trajectory extraction in diverse environments. However, accurate vehicle detection is a challenge due to facts such as small vehicle size and inconspicuous object features in UAV videos. In addition, camera motion in UAV videos hardens the trajectory construction procedure. This research aims at proposing a novel framework for accurate vehicle trajectory construction from UAV videos under mixed traffic conditions. Firstly, a Convolution Neural Network (CNN)-based detection algorithm, named You Only Look Once (YOLO) v3, is applied to detect vehicles globally. Then an image registration method based on Shi-Tomasi corner detection is applied for camera motion compensation. Trajectory construction methods are proposed to obtain accurate vehicle trajectories based on data correlation and trajectory compensation. At last, the ensemble empirical mode decomposition (EEMD) is applied for trajectory data denoising. Our framework is tested on three aerial videos taken by an UAV on urban roads with one including intersection. The extracted vehicle trajectories are compared with manual counts. The results show that the proposed framework achieves an average Recall of 91.91% for motor vehicles, 81.98% for non-motorized vehicles and 78.13% for pedestrians in three videos.

Intelligent English classroom video clarity improvement based on motion compensation and grid flow

For the English classroom teaching video denoising algorithm, it is not only necessary to consider whether the noise removal of the output video is thorough, but also to consider the actual operating efficiency and robustness of the algorithm. In the process of the thesis research, after reading a large number of internal and external documents on video denoising algorithms and analyzing the pros and cons of various denoising algorithms, this paper proposes a new video denoising algorithm, which uses the recently proposed grid flow motion model based on camera motion compensation to generate denoised video. Compared with the current advanced video denoising schemes, our method processes noisy frames faster and has good robustness. In addition, this article improves the algorithm framework so that the algorithm can not only deal with offline video denoising, but also deal with online video denoising.

A Hazard Detection and Tracking System for People with Peripheral Vision Loss using Smart Glasses and Augmented Reality

Peripheral vision loss is the lack of ability to recognise objects and shapes in the outer area of the visual field. This condition can affect people’s daily activities and reduces their quality of life. In this work, a smart technology that implements computer vision algorithms in real-time to detect and track moving hazards around people with peripheral vision loss is presented. Using smart glasses, the system processes real-time captured video and produces warning notifications based on predefined hazard danger levels. Unlike other obstacle avoidance systems, this system can track moving objects in real-time and classify them based on their motion features (such as speed, direction, and size) to display early warning notification. A moving camera motion compensation method was used to overcome artificial motions caused by camera movement before an object detection phase. The detected moving objects were tracked to extract motion features which were used to check if the moving object is a hazard or not. A detection system for camera motion states was implemented and tested on real street videos as the first step before an object detection phase. This system shows promising results in motion detection, motion tracking, and camera motion detection phases. Initial tests have been carried out on Epson’s smart glasses to evaluate the real-time performance for this system. The proposed system will be implemented as an assistive technology that can be used in daily life.

Prior-Free Dependent Motion Segmentation Using Helmholtz-Hodge Decomposition Based Object-Motion Oriented Map

Motion segmentation in moving camera videos is a very challenging task because of the motion dependence between the camera and moving objects. Camera motion compensation is recognized as an effective approach. However, existing work depends on prior-knowledge on the camera motion and scene structure for model selection. This is not always available in practice. Moreover, the image plane motion suffers from depth variations, which leads to depth-dependent motion segmentation in 3D scenes. To solve these problems, this paper develops a prior-free dependent motion segmentation algorithm by introducing a modified Helmholtz-Hodge decomposition (HHD) based object-motion oriented map (OOM). By decomposing the image motion (optical flow) into a curl-free and a divergence-free component, all kinds of camera-induced image motions can be represented by these two components in an invariant way. HHD identifies the camera-induced image motion as one segment irrespective of depth variations with the help of OOM. To segment object motions from the scene, we deploy a novel spatio-temporal constrained quadtree labeling. Extensive experimental results on benchmarks demonstrate that our method improves the performance of the state-of-the-art by 10%~20% even over challenging scenes with complex background.

Continuous localization and mapping of a pan–tilt–zoom camera for wide area tracking

Pan---tilt---zoom (PTZ) cameras are well suited for object identification and recognition in far-field scenes. However, the effective use of PTZ cameras is complicated by the fact that a continuous online camera calibration is needed and the absolute pan, tilt and zoom values provided by the camera actuators cannot be used because they are not synchronized with the video stream. So, accurate calibration must be directly extracted from the visual content of the frames. Moreover, the large and abrupt scale changes, the scene background changes due to the camera operation and the need of camera motion compensation make target tracking with these cameras extremely challenging. In this paper, we present a solution that provides continuous online calibration of PTZ cameras which is robust to rapid camera motion, changes of the environment due to varying illumination or moving objects. The approach also scales beyond thousands of scene landmarks extracted with the SURF keypoint detector. The method directly derives the relationship between the position of a target in the ground plane and the corresponding scale and position in the image and allows real-time tracking of multiple targets with high and stable degree of accuracy even at far distances and any zoom level.

Stabilization of Target Tracking with 3-axis Motion Compensation for Camera System on Flying Vehicle

This paper presents a tracking system using images captured from a camera on a moving platform. A camera on an unmanned flying vehicle generally moves and shakes due to external factors such as wind and the ego-motion of the machine itself. This makes it difficult to track a target properly, and sometimes the target cannot be kept in view of the camera. To deal with this problem, we propose a new system for stable tracking of a target under such conditions. The tracking system includes target tracking and 3-axis camera motion compensation. At the same time, we consider the simulation of the motion of flying vehicles for efficient and safe testing. With 3-axis motion compensation, our experimental results show that robustness and stability are improved.

Non-uniformity Correction of Infrared Images by Midway Equalization

The non-uniformity is a time-dependent noise caused by the lack of sensor equalization. We present here the detailed algorithm and on line demo of the non-uniformity correction method by midway infrared equalization. This method was designed to suit infrared images. Nevertheless, it can be applied to images produced for example by scanners, or by push-broom satellites. The obtained single image method works on static images, is fully automatic, having no user parameter, and requires no registration. It needs no camera motion compensation, no closed aperture sensor equalization and is able to correct for a fully non-linear non-uniformity.

Segmenting moving objects from a freely moving camera with an effective segmentation cue

It is challenging to segment moving objects in a video sequence viewed from a freely moving camera. Within the conditional random field framework, we propose a new segmentation cue to solve this problem. The cue is defined as a spatial–color likelihood ratio map by combining the foreground and background appearance models which are extracted from the previous segmentation result. Specifically, to make the cue suitable for the current segmentation task in the presence of complex camera motions, we introduce a camera motion compensation step in the process of evaluating the likelihood ratio map. We validate the proposed segmentation cue using several video sequences taken by hand-held cameras in outdoor urban scenes.

Observation and analysis of high-speed human motion with frequent occlusion in a large area

The use of computer vision technology in collecting and analyzing statistics during sports matches or training sessions is expected to provide valuable information for tactics improvement. However, the measurements published in the literature so far are either unreliably documented to be used in training planning due to their limitations or unsuitable for studying high-speed motion in large area with frequent occlusions. A sports annotation system is introduced in this paper for tracking high-speed non-rigid human motion over a large playing area with the aid of motion camera, taking short track speed skating competitions as an example. The proposed system is composed of two sub-systems: precise camera motion compensation and accurate motion acquisition. In the video registration step, a distinctive invariant point feature detector (probability density grads detector) and a global parallax based matching points filter are used, to provide reliable and robust matching across a large range of affine distortion and illumination change. In the motion acquisition step, a two regions' relationship constrained joint color model and Markov chain Monte Carlo based joint particle filter are emphasized, by dividing the human body into two relative key regions. Several field tests are performed to assess measurement errors, including comparison to popular algorithms. With the help of the system presented, the system obtains position data on a 30 m × 60 m large rink with root-mean-square error better than 0.3975 m, velocity and acceleration data with absolute error better than 1.2579 m s−1 and 0.1494 m s−2, respectively.