Object Tracking Research Articles

TFITrack: Transformer Feature Integration Network for Object Tracking

AbstractDue to the ignoring of rich spatio-temporal and global contextual information with convolutional neural networks in features extraction, the traditional method is prone to tracking drift or even failure in complex scenario, especially for the tiny targets in aerial photography scenario. In this work, it proposes a transformer feature integration network (TFITrack) to obtain diverse and comprehensive target feature for the robust object tracking. Based on the typical transformer architecture, it optimizes encoder and decoder structure for aggregating discriminative spatio-temporal information and global context-awareness feature. Furthermore, the encoder introduces the similarity calculation layer and dual-attention module; the aim is to deepen the similarity between features and make corrections for channel and spatial dimensions, and feature representation is improved. Finally, with the introduction of the temporal context filtering layer, unimportant feature information is ignored adaptively, obtaining a balance between the parameters number reduction and stable performance. Experimental results show that the proposed tracking algorithm exhibits excellent tracking performance on seven benchmark datasets, especially on the aerial dataset UAV123, UAV20L, and UAV123@10fps, which presents the advantages of the novel method in dealing with fast motion and external interference.

Self-supervised learning for RGB-D object tracking

Recently, there has been a growing interest in RGB-D object tracking thanks to its promising performance achieved by combining visual information with auxiliary depth cues. However, the limited volume of annotated RGB-D tracking data for offline training has hindered the development of a dedicated end-to-end RGB-D tracker design. Consequently, the current state-of-the-art RGB-D trackers mainly rely on the visual branch to support the appearance modelling, with the depth map utilised for elementary information fusion or failure reasoning of online tracking. Despite the achieved progress, the current paradigms for RGB-D tracking have not fully harnessed the inherent potential of depth information, nor fully exploited the synergy of vision-depth information. Considering the availability of ample unlabelled RGB-D data and the advancement in self-supervised learning, we address the problem of self-supervised learning for RGB-D object tracking. Specifically, an RGB-D backbone network is trained on unlabelled RGB-D datasets using masked image modelling. To train the network, the masking mechanism creates a selective occlusion of the input visible image to force the corresponding aligned depth map to help with discerning and learning vision-depth cues for the reconstruction of the masked visible image. As a result, the pre-trained backbone network is capable of cooperating with crucial visual and depth features of the diverse objects and background in the RGB-D image. The intermediate RGB-D features output by the pre-trained network can effectively be used for object tracking. We thus embed the pre-trained RGB-D network into a transformer-based tracking framework for stable tracking. Comprehensive experiments and the analysis of the results obtained on several RGB-D tracking datasets demonstrate the effectiveness and superiority of the proposed RGB-D self-supervised learning framework and the following tracking approach.

Multiple object tracking based on appearance and motion graph convolutional neural networks with an explainer

Multiple object tracking based on appearance and motion graph convolutional neural networks with an explainer

Advancing defense capabilities through integration of electro-optical systems and computer vision technologies

This paper presents a comprehensive study on the integration of cutting-edge technologies to enhance defense capabilities, focusing on object detection, tracking, and distance measurement tasks. Also, the integration of IMX219-77 cameras and Nvidia Jetson Nano for object detection, tracking, and distance measurement in electro-optical systems is proposed. The decision to combine these components was based on their individual strengths: the high resolution and wide field of view of the IMX219-77 cameras, and the low-power, high-performance capabilities of the Jetson Nano. Software-wise, OpenCV and GStreamer were chosen for their widespread use and capabilities in computer vision and multimedia processing tasks, respectively. The MOSSE algorithm was selected for object tracking due to its speed, efficiency, and resilience to lighting changes. Additionally, Depth Estimation (Stereo Vision) was employed for distance measurement by analyzing images captured from a pair of cameras. The proposed algorithm involves transmitting images from the cameras to the Jetson Nano, where OpenCV and GStreamer are utilized to process the data. The MOSSE algorithm is employed for object detection and tracking, while Depth Estimation generates depth maps to measure distances between objects and the camera. The study's results demonstrate accurate distance measurements with a margin of error ranging from 0 to 2 mm, falling below the acceptable threshold mentioned in academic literature. This aligns with similar studies in the field, showcasing the efficacy of the proposed integration for object detection, tracking, and distance measurement in electro-optical systems.

Electric Vehicle Battery Disassembly Using Interfacing Toolbox for Robotic Arms

This paper showcases the integration of the Interfacing Toolbox for Robotic Arms (ITRA) with our newly developed hybrid Visual Servoing (VS) methods to automate the disassembly of electric vehicle batteries, thereby advancing sustainability and fostering a circular economy. ITRA enhances collaboration between industrial robotic arms, server computers, sensors, and actuators, meeting the intricate demands of robotic disassembly, including the essential real-time tracking of components and robotic arms. We demonstrate the effectiveness of our hybrid VS approach, combined with ITRA, in the context of Electric Vehicle (EV) battery disassembly across two robotic testbeds. The first employs a KUKA KR10 robot for precision tasks, while the second utilizes a KUKA KR500 for operations needing higher payload capacity. Conducted in T1 (Manual Reduced Velocity) mode, our experiments underscore a swift communication protocol that links low-level and high-level control systems, thus enabling rapid object detection and tracking. This allows for the efficient completion of disassembly tasks, such as removing the EV battery’s top case in 27 s and disassembling a stack of modules in 32 s. The demonstrated success of our framework highlights its extensive applicability in robotic manufacturing sectors that demand precision and adaptability, including medical robotics, extreme environments, aerospace, and construction.

Hierarchical mussel farm reconstruction from video with object tracking

ABSTRACT As the mussel farming industry expands, particularly in regions such as New Zealand, there is a growing need for advanced monitoring and management solutions to ensure sustainability and operational efficiency. The current reliance on manual infrequent observation of aquaculture structures limits farmers' ability to monitor them in real time. Addressing these challenges, large-scale 3D reconstruction provides a practical solution by facilitating the creation of replicable depictions of mussel farm scenes and floatation buoys derived from recorded video, supporting off-site evaluation and enabling precise decision making. This paper introduces a novel approach to enhance the visualisation and monitoring of mussel farm floatation through a hierarchical reconstruction process. In contrast to earlier studies, we focus on recovering not only the overall environment (background) but also the finer details of key elements such as buoys to create a comprehensive representation of mussel farm geometry and appearance. We propose to segment the scene into the background and granular object instances and reconstruct them separately in a multi-stage process. The initial 3D scene reconstruction is performed using the Structure-from-Motion (SfM) technique, leveraging video footage captured by a vessel-mounted camera. This coarse reconstruction serves as the foundation for subsequent fine-grained enhancements. To recover finer details, object tracking is applied and the trajectories obtained are then conjunct with geometry triangulation to determine the real-world positions of individual buoys. A multiple-scale denoising method, grounded in dominant direction correlation, is implemented to eliminate non-reliable tracking objects and reduce reconstruction artifacts, ensuring the accuracy of the final results. This hierarchical reconstruction approach contributes to the advancement of mussel farm management by offering a powerful technology for comprehensive visualisation, enabling farmers to make informed decisions based on a detailed understanding of the mussel farm's geometry and dynamics.

Edge computing based real-time Nephrops (Nephrops norvegicus) catch estimation in demersal trawls using object detection models

In demersal trawl fisheries, the unavailability of the catch information until the end of the catching process is a drawback, leading to seabed impacts, bycatches and reducing the economic performance of the fisheries. The emergence of in-trawl cameras to observe catches in real-time can provide such information. This data needs to be processed in real-time to determine the catch compositions and rates, eventually improving sustainability and economic performance of the fisheries. In this study, a real-time underwater video processing system counting the Nephrops individuals entering the trawl has been developed using object detection and tracking methods on an edge device (NVIDIA Jetson AGX Orin). Seven state-of-the-art YOLO models were tested to discover the appropriate training settings and YOLO model. To achieve real-time processing and accurate counting simultaneously, four frame skipping ideas were evaluated. It has been shown that adaptive frame skipping approach, together with YOLOv8s model, can increase the processing speed up to 97.47 FPS while achieving correct count rate and F-score of 82.57% and 0.86, respectively. In conclusion, this system can improve the sustainability of the Nephrops directed trawl fishery by providing catch information in real-time.

A structural description of pedestrian movement behavior in multiple surveillance videos

This study purpose of recording pedestrians' activities in all surveillance videos and then convert these records into structured descriptions to analyze a large number of surveillance videos. This paper proposes a four-stage processing method. The first stage uses convolutional neural network (CNN) to detect objects in the surveillance video and classify them. The second stage finds the correlation of pedestrians in continuous images and describes multiple objects' trajectories in the surveillance video. The third stage calculates the similarity of the pedestrian object sets of the surveillance video at different times as the basis for pedestrian re-identification. In the fourth stage, extract the walking path of the pedestrian in each surveillance video. Each pedestrian's actions in each surveillance video, including whether to stand, walking path, direction, clothing color and walking time. To construct a correlation table of pedestrian motion in multiple surveillance videos.This experiment uses two image datasets for experimentation and analysis to evaluate this method's effectiveness for pedestrian movement in multiple surveillance videos. The image datasets are the Multi-Camera Object Tracking (MCT) Challenge and PETS 2009 Benchmark Data. This experiment includes pedestrian detection, pedestrian identification, and multi-pedestrian tracking of pedestrians in single and multiple surveillance cameras. The results show that in the pedestrian detection experiment, the convolutional neural network can effectively solve the problems caused by static objects, light changes, similar colors, crowded pedestrians, and incomplete objects. In the pedestrian recognition and re-recognition experiments, the accuracy is reduced due to the influence of low-resolution surveillance videos. This detection method of the pedestrian is more accurate than traditional dynamic object detection methods. After the pedestrian tracking and re-identification are completed, the pedestrian relationships in multiple surveillance cameras are integrated, and a pedestrian behavior association table is built for surveillance videos.

MEDAVET: Traffic Vehicle Anomaly Detection Mechanism based on spatial and temporal structures in vehicle traffic

Road traffic anomaly detection is vital for reducing the number of accidents and ensuring a more efficient and safer transportation system. In highways, where traffic volume and speed limits are high, anomaly detection is not only essential but also considerably more challenging, given the multitude of fast-moving vehicles, often observed from extended distances and diverse angles, occluded by other objects, and subjected to variations in illumination and adverse weather conditions. This complexity has meant that human error often limits anomaly detection, making the role of computer vision systems integral to its success. In light of these challenges, this paper introduces MEDAVET - a sophisticated computer vision system engineered with an innovative mechanism that leverages spatial and temporal structures for high-precision traffic anomaly detection on highways. MEDAVET is assessed in its object tracking and anomaly detection efficacy using the UA-DETRAC and Track 4 benchmarks and has its performance compared with that of an array of state-of-the-art systems. The results have shown that, when MEDAVET’s ability to delimit relevant areas of the highway, through a bipartite graph and the Convex Hull algorithm, is paired with its QuadTree-based spatial and temporal approaches for detecting occluded and stationary vehicles, it emerges as superior in precision, compared to its counterparts, and with a competitive computational efficiency.

Pilot performance during simulated point and boundary avoidance tracking tasks

Abstract Helicopters are used in complex and harsh operational environments, such as search and rescue missions and firefighting, that require operating in ground proximity, tracking targets while avoiding impacting obstacles, namely a combination of point tracking (positive) and boundary avoidance (negative) objectives. A simulation task representing simplified helicopter dynamics is used to investigate point tracking and boundary avoidance tasks. The variance and regression analysis are used to study the effects of task conditions on participants’ tracking errors and input aggression. The overall tracking error shows a negative correlation with input aggression. The participants tend to have higher input aggression and lower tracking error near the boundaries, exposing the switching of manipulation input strategies under different task conditions. It also suggests a potential way of designing simulation tasks for human operators manipulating helicopters and a trigger for investigating pilots’ biodynamic feedthrough.

FPGA Implementation of a Pipelined Kalman Filter for Object Tracking in Two Dimensions

In a location tracking system for a moving object, not only accuracy but also real-time processing are important factors. The Kalman filter, as a recursive function, stands out as one of the prominent algorithms for object tracking. It continuously compares measured data with predicted values based on the system characteristics in real time, and then corrects the error of the predicted values while considering the noise of both system and measured data. This paper focuses on designing a hardware-based Kalman filter for object tracking in two dimensions. Following an analysis of the Kalman filter algorithm, the blocks capable of parallel processing are identified and configured to be processed in parallel, effectively reducing data processing time. The clock speed is enhanced by using the pipeline technique. In addition, the time-sharing technique is applied to increase the utilization of hardware resources and reduce the area. Data was processed at 32-bit floating points to uphold accuracy comparable to software-implemented Kalman filters. The proposed Kalman filter architecture is designed using verilog HDL and then simulated in Synopsys VCS/Verdi. And the accuracy is verified by comparing results with a software-based Kalman filter designed using MATLAB. It was implemented using Zynq ZYNQ-7 ZC702 Programmable logic via Xilnix Vivado, and can operate at 33MHz. It takes a total of 44 clocks, or 1.32 us, to process one data. Therefore, it was confirmed that the designed Kalman filter hardware is suitable for real-time processing.

A bonus task boosts people's willingness to offload cognition to an algorithm

With the increased sophistication of technology, humans have the possibility to offload a variety of tasks to algorithms. Here, we investigated whether the extent to which people are willing to offload an attentionally demanding task to an algorithm is modulated by the availability of a bonus task and by the knowledge about the algorithm’s capacity. Participants performed a multiple object tracking (MOT) task which required them to visually track targets on a screen. Participants could offload an unlimited number of targets to a “computer partner”. If participants decided to offload the entire task to the computer, they could instead perform a bonus task which resulted in additional financial gain—however, this gain was conditional on a high performance accuracy in the MOT task. Thus, participants should only offload the entire task if they trusted the computer to perform accurately. We found that participants were significantly more willing to completely offload the task if they were informed beforehand that the computer’s accuracy was flawless (Experiment 1 vs. 2). Participants’ offloading behavior was not significantly affected by whether the bonus task was incentivized or not (Experiment 2 vs. 3). These results combined with those from our previous study (Wahn et al. in PLoS ONE 18:e0286102, 2023), which did not include a bonus task but was identical otherwise, show that the human willingness to offload an attentionally demanding task to an algorithm is considerably boosted by the availability of a bonus task—even if not incentivized—and by the knowledge about the algorithm’s capacity.

Deep SORT Related Studies

Computer vision is the field of computer science in which computers are made capable to see and recognize like human being. Deep learning is using multiple layers for the purpose of understanding and recognizing various objects. Deep Simple Real Time Tracker is the area in which the objects are tracked in real time from multiple images and videos. Many researchers have contributed to the field and various algorithms have been proposed. The current study presents the deep SORT related studies in which the various algorithms have been presented for the sake of understanding and starting point for the researchers interested in computer vision and deep sorting. The single shot detection, feature extraction, have been explained along with the research conducted. Feature selection and extraction, matching recognition, object tracking through frames have been appended to the current study.

Using Ant Colony Optimization Method to Detect Dominant Points

This paper uses the ant colony optimization method to detect dominant points. It develops a dominant point detection system for closed target images with simple backgrounds. The process is to perform image preprocessing on the captured object images, then perform object contour tracking and breakpoint detection, and finally use the ant colony optimization method to detect the object's dominant points. In such problems, ant colony calculus is applied to such problems by first defining a fitness value. In addition, three different hybrid strategies, including the minimum error method, the maximum distance method, and the random selection method, were also experimentally compared. Experimental results show that the used hybrid strategy can improve the computational efficiency of all three types of problems.

Fusion-driven deep feature network for enhanced object detection and tracking in video surveillance systems

Object detection and tracking (ODT) is a crucial research area in video surveillance (VS) systems and poses a significant challenge in computer vision and image processing. The primary objective is to identify objects of various classes from video sequences for detection and tracking. The object detection and tracking process involves extracting moving objects from frames and tracking them over time. With the advancement of computer intelligence, this study proposes a novel optimized deep fused learning (ODFL) model for efficient object detection and tracking in video surveillance systems. The optimized deep fused learning model focuses primarily on detecting and tracking multiple objects present in video frames. Initially, the input video undergoes pre-processing by converting it into video frames and applying Gaussian filtering to eliminate noise and enhance frame quality. Next, the feature fusion model employs the Dense Convolution Feature Fusion Network (D-ConvFFN) to extract and fuse relevant features. Subsequently, the Enhanced RefineDet-based Fire Hawk (ERFH) object detection module is utilized by the optimized deep fused learning model to efficiently recognize multiple objects in the video frames. The hyperparameters are optimized using the Fire Hawk optimizer. Once the objects are recognized, a softmax classifier categorizes them into different classes. The Hungarian-based SORT model is used for multi-object tracking. The proposed optimized deep fused learning model is implemented in Python, evaluated on the PETS S2 2009 and UA-DETRAC datasets, and assessed in terms of various evaluation measures. The optimized deep fused learning model’s performance is compared with prevailing architectures, and it achieves a maximum accuracy of 99.16% for the PETS S2 2009 dataset and 99.42% for the UA-DETRAC dataset, outperforming existing classifiers for multi-class object detection and tracking.

PAW: Prediction of wildlife animals using a robot under adverse weather conditions

AbstractImage dehazing and object detection are two different research areas that play a vital role in machine learning. When merged together and implemented in real‐time, it is a boon in the field of artificial intelligence, specifically robotics. Object detection and tracking are two of the major implementations in almost the entire robot's training and learning. The learning of the robot depends on the images; these images can be camera‐captured images or a pretrained data set. Real‐time outdoor images clicked in bad weather conditions, such as mist, haze, smog, and fog, often suffer from poor visibility, and the consequences are incorrect results and hence an unexpected robot's behavior. To overcome these consequences, we have presented a novel approach to object detection and identification during adverse weather conditions. This method is proposed to be implemented in a real‐time environment to monitor animal behavior near railway tracks during fog, haze, and smog. This is not limited to specific application areas but can be used to identify endangered species and take active steps to save them from mishap. The deployment is done in a real‐time indoor environment using Tortoisebot mobile robot with a robot operating system framework.

Channel Self-Attention Based Low-Light Image Enhancement Network

Low-light image enhancement is crucial in applications such as traffic safety and medical imaging. Besides having characteristics like low luminance and poor visibility, low-light images are inevitably affected by noise. Noise not only covers image details, introduces artifacts, and decreases image quality, but also interferes with downstream computer vision tasks like object detection, image segmentation, and object tracking. Previous methods in image enhancement often overlook noise handling or fail to accurately suppress noise in adaptive denoising processes, resulting in more severe noise artifacts in the enhanced images. To effectively suppress noise, this paper proposes a Channel Self-Attention Based Low-Light Image Enhancement Network (CAENet), which leverages Transformers and CNNs to model long-range and short-range pixel dependencies, extract global and local features, and construct a Noise Suppression Transformer Block that adaptively suppresses noise regions guided by signal-to-noise ratio priors and attention maps. After adaptive noise suppression, the resulting images exhibit fewer noise artifacts and improved details. The experimental results show that the network in this paper outperforms other state-of-the-art methods overall on five representative paired datasets as well as six unpaired datasets, improving the image quality while effectively suppressing the noise.

A Canvas of Air and Signs: Integrating Voice Activated Hand Sign Recognition and Air Canvas for Hearing Impaired and Non-Verbal People

Moving hands in air and watching your screen writes for you and at the same time making different Signs with hands and the system displays it as well as speak what Sign you are making.This seems to be very futuristic approach towards the realm of Image Processing and Gesture Recognition . In this paper, we present a very interesting and a novel approach towards an interactive learning platform where one can draw the content on screen while moving their hand in air and can also use hand sign language to communicate with an ease with the Hearing Impaired and Dumb Community. Our system combines both technologies to create a smooth and engaging experience for users. It can be used in interactive art setups or virtual reality setups. Air canvas enables users to draw and manipulate digital content in mid-air with object tracking using Computer Vision and Mediapipe framework, while hand gesture recognition allows for real-time interpretation of Hand Signs to perform actions or commands within the system. This Model not only recognizes the sign but also speaks it loud using pyttsx3 a text-to-speech conversion Library, ensuring a good communication between a normal human and people with Non-Verbal and Hearing Impaired disability. To enhance the performance of the model We validate the model with a real dataset trained by us. This training was essential for refining the accuracy and efficiency of the model.

Distance-based feature repack algorithm for video coding for machines

Nowadays, the use of video data for machine (VCM) tasks has become increasingly prevalent, with deep learning and computer vision requiring large volumes of video data for object detection, object tracking, and other tasks. However, the features required for machine tasks are different from those used by humans, and a new approach is needed to encode and compress video data for machine consumption. Video coding for machines has received considerable attention, with many approaches focusing on compressing features rather than the video itself. However, a key challenge in this process is repacking the features in an efficient and effective manner. This paper proposes a distance-based patch tiling and intra-block quilting method to repack feature sequences in a manner that is better suited for existing video codecs, based on statistical analysis of feature characteristics in the channel dimension. Experimental results demonstrate that our method achieves an 65.54% BD-rate gain compared to benchmark methods. This research has significant implications for improving the efficiency of video coding for machine applications, and future work could explore the use of feature dimensionality reduction and combination of neural network (NN) codec to optimize the repacking of features for compression.

Virtual Reality Distraction for Needle-Related Pain and Distress in Children: A Multicenter Randomized Controlled Trial.

This international multicenter randomized controlled trial aimed to compare the effectiveness of virtual reality (VR) distraction with an identical non-VR game in reducing needle-related pain and anxiety in children undergoing venous blood draw. The study involved 304 children aged 5-9 years undergoing a blood draw procedure, randomly allocated to one of three groups: VR distraction, non-VR distraction, and control group (usual care). The distraction task was based on the Multiple Object Tracking (MOT) paradigm, and the game was identical in design and gameplay for both VR and non-VR distraction groups. The primary outcome was self-reported pain intensity using the Faces Pain Scale-Revised (FPS-R). Secondary outcomes included child distress, attention/distraction to the blood draw, and parent and medical staff satisfaction with procedure. Analyses were conducted using analysis of variance and multivariable linear regression models. The results showed that VR distraction and non-VR distraction performed similarly, showing large effect sizes compared with standard care. There was no significant difference between the two types of distraction. The study's findings suggest that VR and non-VR distraction are similarly effective in reducing needle-related pain and anxiety in children undergoing venous blood draw. This is the first well-powered study comparing modern VR distraction with an identical task displayed on a smartphone or monitor screen. The study's results have important implications for using VR in clinical settings and suggest that investing in expensive VR equipment for acute pain management may not be necessary. The study protocol was pre-registered on Open Science Framework at https://osf.io/frsyc.