Appearance Descriptors Research Articles

Improved DeepSORT-Based Object Tracking in Foggy Weather for AVs Using Sematic Labels and Fused Appearance Feature Network.

The presence of fog in the background can prevent small and distant objects from being detected, let alone tracked. Under safety-critical conditions, multi-object tracking models require faster tracking speed while maintaining high object-tracking accuracy. The original DeepSORT algorithm used YOLOv4 for the detection phase and a simple neural network for the deep appearance descriptor. Consequently, the feature map generated loses relevant details about the track being matched with a given detection in fog. Targets with a high degree of appearance similarity on the detection frame are more likely to be mismatched, resulting in identity switches or track failures in heavy fog. We propose an improved multi-object tracking model based on the DeepSORT algorithm to improve tracking accuracy and speed under foggy weather conditions. First, we employed our camera-radar fusion network (CR-YOLOnet) in the detection phase for faster and more accurate object detection. We proposed an appearance feature network to replace the basic convolutional neural network. We incorporated GhostNet to take the place of the traditional convolutional layers to generate more features and reduce computational complexities and costs. We adopted a segmentation module and fed the semantic labels of the corresponding input frame to add rich semantic information to the low-level appearance feature maps. Our proposed method outperformed YOLOv5 + DeepSORT with a 35.15% increase in multi-object tracking accuracy, a 32.65% increase in multi-object tracking precision, a speed increase by 37.56%, and identity switches decreased by 46.81%.

Intelligent Traffic System: Yolov8 and Deepsort in Car Detection, Tracking and Counting

Abstract: As the number of vehicles on the road increases, it becomes essential to quickly and accurately identify them to better detect traffic congestion and improve traffic management. DeepSORT (simple online and real-time tracking with a deep association metric) multitarget tracking algorithm in vehicle tracking. Due to the strong dependence of the DeepSORT algorithm on target detection, a YOLOv8 vehicle detection algorithm was proposed based on YOLOv7, which provides accurate and fast vehicle detection data to the DeepSORT algorithm. DeepSORT introduces deep learning into the SORT algorithm by adding an appearance descriptor to reduce identity switches, making tracking more efficient. DeepSORT uses a better association metric that combines both motion and appearance descriptors. DeepSORT can be defined as the tracking algorithm that tracks objects not only based on the velocity and motion of the object but also the appearance of the object. This vehicle detection also uses the DeepSORT algorithm to help count the number of vehicles that pass in the video effectively. YOLOv8 and the DeepSORT algorithm collaborate to identify and follow vehicles creating a model, for vehicle recognition that showcases their ability to track cars efficiently. From this paper, the model Yolov8 has achieved state-of-the-art results

Efficient and Accurate Image Classification via Spatial Pyramid Matching and SURF Sparse Coding

Sparsely coded signal space representations do well in feature quantization. Instead of using standard vector quantization, the suggested method uses selective sparse coding to assemble the most important features of the appearance descriptors of nearby image patches. Inadequate coding also enables adjacent max pooling on some spatial scales, which, unlike the setup of average pooling in a histogram, links interpretation with scale invariance. The acquired visual illustration is the key contribution of this research; it performs outperform with linear-SVM, improves the model training's, which in turn speeds up testing with improves accuracy. The efficacy of the method we have employed has been substantiated through a series of experiments conducted on diverse datasets. Since top-performing image classification systems heavily rely on nonlinear SPM in mean of vector quantization, the trustworthy recommended linear SPM greatly increases the use of larger sets of training data. The method given herein deduces that the sparse coding of SURF feature’s function hampered a more comprehensive local appearance descriptor for general-purpose image processing. Experiments and comparisons are conducted on standard datasets such as Caltech-101, FTVL, and Corel-1000, using state-of-the-art techniques and descriptors. When compared over several other image categories and descriptors, the method provided here comes out on top.

MBT3D: Deep learning based multi-object tracker for bumblebee 3D flight path estimation.

This work presents the Multi-Bees-Tracker (MBT3D) algorithm, a Python framework implementing a deep association tracker for Tracking-By-Detection, to address the challenging task of tracking flight paths of bumblebees in a social group. While tracking algorithms for bumblebees exist, they often come with intensive restrictions, such as the need for sufficient lighting, high contrast between the animal and background, absence of occlusion, significant user input, etc. Tracking flight paths of bumblebees in a social group is challenging. They suddenly adjust movements and change their appearance during different wing beat states while exhibiting significant similarities in their individual appearance. The MBT3D tracker, developed in this research, is an adaptation of an existing ant tracking algorithm for bumblebee tracking. It incorporates an offline trained appearance descriptor along with a Kalman Filter for appearance and motion matching. Different detector architectures for upstream detections (You Only Look Once (YOLOv5), Faster Region Proposal Convolutional Neural Network (Faster R-CNN), and RetinaNet) are investigated in a comparative study to optimize performance. The detection models were trained on a dataset containing 11359 labeled bumblebee images. YOLOv5 reaches an Average Precision of AP = 53, 8%, Faster R-CNN achieves AP = 45, 3% and RetinaNet AP = 38, 4% on the bumblebee validation dataset, which consists of 1323 labeled bumblebee images. The tracker's appearance model is trained on 144 samples. The tracker (with Faster R-CNN detections) reaches a Multiple Object Tracking Accuracy MOTA = 93, 5% and a Multiple Object Tracking Precision MOTP = 75, 6% on a validation dataset containing 2000 images, competing with state-of-the-art computer vision methods. The framework allows reliable tracking of different bumblebees in the same video stream with rarely occurring identity switches (IDS). MBT3D has much lower IDS than other commonly used algorithms, with one of the lowest false positive rates, competing with state-of-the-art animal tracking algorithms. The developed framework reconstructs the 3-dimensional (3D) flight paths of the bumblebees by triangulation. It also handles and compares two alternative stereo camera pairs if desired.

Sequential Monte Carlo localization in topometric appearance maps

Representing the scene appearance by a global image descriptor (BoW, NetVLAD, etc.) is a widely adopted choice to address Visual Place Recognition (VPR). The main reasons are that appearance descriptors can be effectively provided with radiometric and perspective invariances as well as they can deal with large environments because of their compactness. However, addressing metric localization with such descriptors (a problem called Appearance-based Localization or AbL) achieves much poorer accuracy than those techniques exploiting the observation of 3D landmarks, which represent the standard for visual localization. In this paper, we propose ALLOM (Appearance-based Localization with Local Observation Models) which addresses AbL by leveraging the topological location of a robot within a map to achieve accurate metric estimations. This topology-assisted metric localization is implemented with a sequential Monte Carlo Bayesian filter that applies a specific observation model for each different place of the environment, thus taking advantage of the local correlation between the pose and the appearance descriptor within each region. ALLOM also benefits from the topological structure of the map to detect eventual robot loss-of-tracking and to effectively cope with its relocalization by applying VPR. Our proposal demonstrates superior metric localization capability compared to different state-of-the-art AbL methods under a wide range of situations.

Don’t get lost in the crowd: Graph convolutional network for online animal tracking in dense groups

Tracking the behaviour of animals in group-housed situations is a critical area of study for precision livestock farming, but it poses challenges in diverse and crowded environments. Existing methods often struggle with false positives and false negatives due to the complexities of these scenarios. In this study, we propose a robust computer vision algorithm for long-term (>10mins) animal tracking, with a primary focus on group-housed pigs. Our method addresses the limitations of current approaches by effectively handling errors from the detector in challenging environments. Our approach integrates Graph Convolutional Networks.(GCNs) with deep learning-based object detection techniques. We represent the data as a graph structure, with nodes corresponding to multiple animal detections over several frames. Edges connect thedetections that do not appear in the same frame. The model's objective is to perform edge classification, where each edge is associated with a scalar representing the probability of being the same object. To enhance the robustness of edge classification, we combine the classifier withan ensemble predictor, enabling accurate tracking of animal identitiesfor extended periods. Notably, our graph convolutional model achievesaccurate re identification of non-consecutive animal detections inreal-time without the need for trajectory prediction or ID association. We evaluate our method by comparing its performance against othertracking methods, including DeepSORT—a state-of-the-art animal trackingmethod that relies on a Kalman filter, a deep appearance descriptor, and the Hungarian algorithm for trajectory prediction and ID association.Our comparison demonstrates the advantages of our method, as it (1)improves the IDF1 score, indicating enhanced ID association accuracy, by 1.72%; (2) mitigates 93% of errors resulting from ID-switch anddeviation; (3) extends the tracking duration of each animal byapproximately 66% in challenging conditions; and (4) achieves aprocessing speed that is 1000 times faster than DeepSORT when operatingat 22fps with the detector. The potential applications of our method extend to various aspects of livestock management. By accurately tracking individual animals, we can monitor behaviours such as feeding, drinking, and aggressive interactions. This information can lead to improvements in animal welfare, resource allocation, and breeding strategies. Follow-ups of this research are available at: https://gitlab.kuleuven.be/m3-biores/public/m3pig.

Multiple Object Tracking Based on YOLOv5 and Optimized DeepSORT Algorithm

As a widely discussed issue in academic and industrial fields, multiple object tracking (MOT) has a huge impact on various aspects, such as video surveillance, human-machine interaction, viral reality and autonomous driving. Among all the MOT algorithms, DeepSORT enjoys a high reputation for its speed, accuracy and robustness to frequent occlusion circumstance. With the help of DeepSORT, the main objective of this research work is to propose an algorithm with a better performance on frequent occlusion and long-time occlusion issues. To accomplish such a mission, an improved deep appearance descriptor was constructed and trained off-line. To achieve a higher performance, the object detector YOLOv5, which has been proved to be faster than previous detectors, is employed in this research as a substitution of the original detector Faster R-CNN.

Towards High-level, Intuitive Descriptors of Material Appearance

Towards High-level, Intuitive Descriptors of Material Appearance

Marine Object Segmentation and Tracking by Learning Marine Radar Images for Autonomous Surface Vehicles

This paper proposes a radar image segmentation and tracking method by learning radar images for autonomous surface vehicles. To identify marine objects from radar images, we propose a deep neural network named the dual path squeeze and excitation network (DPSE-Net). By learning the radar images, the proposed DPSE-Net is designed to segment every pixel of the radar images into four classes: marine objects, land, noise, and background. The proposed DPSE-Net shows the best performance in radar image segmentation while operating in real-time, compared to state-of-the-art real-time image segmentation network models. In addition, we design a real-time moving object tracking algorithm for estimating the position and velocity of marine objects based on deep simple online real-time tracking with a deep association metric (DeepSORT), a widely used tracking algorithm. The existing DeepSORT algorithm uses the intersection over union (IoU) metric and a deep appearance descriptor for data association, but since they are not suitable for radar images, successive tracking is difficult. To solve this problem, a new data association metric suitable for radar images is proposed. The field tests in ocean environments confirm that the proposed method performs better in marine object segmentation and tracking.

Multi-Vehicle Tracking Based on Monocular Camera in Driver View

Multi-vehicle tracking is used in advanced driver assistance systems to track obstacles, which is fundamental for high-level tasks. It requires real-time performance while dealing with object illumination variations and deformations. To this end, we propose a novel multi-vehicle tracking algorithm based on a monocular camera in driver view. It follows the tracking-by-detection paradigm and integrates detection and appearance descriptors into a single network. The one-stage detection approach consists of a backbone, a modified BiFPN as a neck layer, and three prediction heads. The data association consists of a two-step matching strategy together with a Kalman filter. Experimental results demonstrate that the proposed approach outperforms state-of-the-art algorithms. It is also able to solve the tracking problem in driving scenarios while maintaining 16 FPS on the test dataset.

A Study on N-Spherical Histogram for Unbounded Tensor Features in Human Action Recognition

Recent years, Dense Trajectory features has a crucial role in extracting implicit features for action recognition. The method encloses motion and appearance descriptors to specify characteristics of each trajectory. Moreover, combining gradient and optical flow field using tensor product has made a strong positive impact on the result as we introduced in our previous work. In this paper, a breakthrough concept of encoding a high dimensional unbound space using spherical coordinate is introduced and imposed to obtain sophisticated spherical tensor features. The experimental result shows that our propose features outperforms other conventional ones and the combination of all feature channels achieves the highest accuracy rate in our selfrecorded dataset.

Segmentation of Infant Brain Using Nonnegative Matrix Factorization

This study develops an atlas-based automated framework for segmenting infants’ brains from magnetic resonance imaging (MRI). For the accurate segmentation of different structures of an infant’s brain at the isointense age (6–12 months), our framework integrates features of diffusion tensor imaging (DTI) (e.g., the fractional anisotropy (FA)). A brain diffusion tensor (DT) image and its region map are considered samples of a Markov–Gibbs random field (MGRF) that jointly models visual appearance, shape, and spatial homogeneity of a goal structure. The visual appearance is modeled with an empirical distribution of the probability of the DTI features, fused by their nonnegative matrix factorization (NMF) and allocation to data clusters. Projecting an initial high-dimensional feature space onto a low-dimensional space of the significant fused features with the NMF allows for better separation of the goal structure and its background. The cluster centers in the latter space are determined at the training stage by the K-means clustering. In order to adapt to large infant brain inhomogeneities and segment the brain images more accurately, appearance descriptors of both the first-order and second-order are taken into account in the fused NMF feature space. Additionally, a second-order MGRF model is used to describe the appearance based on the voxel intensities and their pairwise spatial dependencies. An adaptive shape prior that is spatially variant is constructed from a training set of co-aligned images, forming an atlas database. Moreover, the spatial homogeneity of the shape is described with a spatially uniform 3D MGRF of the second-order for region labels. In vivo experiments on nine infant datasets showed promising results in terms of the accuracy, which was computed using three metrics: the 95-percentile modified Hausdorff distance (MHD), the Dice similarity coefficient (DSC), and the absolute volume difference (AVD). Both the quantitative and visual assessments confirm that integrating the proposed NMF-fused DTI feature and intensity MGRF models of visual appearance, the adaptive shape prior, and the shape homogeneity MGRF model is promising in segmenting the infant brain DTI.

PlaneLoc2: Indoor Global Localization Using Planar Segments and Passive Stereo Camera

This paper introduces PlaneLoc2 - a novel indoor global localization system designed to harness the potential of stereo cameras. A need for robust global localization that does not produce incorrect results (false positives) is present in almost every life-long autonomy task. We show that planar segments extracted from stereo vision data by a neural network enable such robust localization. Planar segments are easier to discriminate than keypoint features and provide easy-to-use geometric constraints. We propose an architecture that exploits a single deep neural network (DNN) to detect planar segments, produce appearance descriptors, and estimate segment geometry. Moreover, we introduce a novel view-based segment map and a novel pose retrieval procedure that considers the uncertainty of features to efficiently use the geometric constraints provided by them. We also show that the new learned descriptor provides better discrimination than the hand-crafted one. Finally, we present experimental results that show that our solution outperforms other state-of-the-art global localization methods and does not produce incorrect agent poses. For both test scenes it recognizes at least 15% more poses than the second best method without incorrect recognitions.

DeepPlayer-Track: Player and Referee Tracking With Jersey Color Recognition in Soccer

In real-world sports video analysis, identity switching caused by inter-object interactions is still a major difficulty for multi-player tracking. Due to similar appearances of players on the same squad, existing methodologies make it difficult to correlate detections and retain identities. In this paper, a novel approach (DeepPlayer-Track) is proposed to track the players and referees, by representing the deep features to retain the tracking identity. To provide identity-coherent trajectories, a sophisticated multi-player tracker is being developed further, encompassing deep features of player and referee identification. The proposed methodology consists of two parts: (i) the You Only Look Once (YOLOv4) can detect and classify players, soccer balls, referees, and background; (ii) Applying a modified deep feature association with a simple online real-time (SORT) tracking model which connects nodes from frame to frame using cosine distance and deep appearance descriptor to correlate the coefficient of the player identity (ID) which improved tracking performance by distinct identities. The proposed model achieved a tracking accuracy of 96% and 60% on MOTA and GMOTA metrics respectively with a detection speed of 23 frames per second (FPS).

BI-RADS Ultrasound Lexicon Descriptors and Stromal Tumor-Infiltrating Lymphocytes in Triple-Negative Breast Cancer

Increased levels of stromal tumor-infiltrating lymphocytes (sTILs) have recently been considered a favorable independent prognostic and predictive biomarker in triple-negative breast cancer (TNBC). The purpose of this study was to determine the relationship between BI-RADS (Breast Imaging Reporting and Data System) ultrasound lexicon descriptors and sTILs in TNBC. Patients with stage I-III TNBC were evaluated within a single-institution neoadjuvant clinical trial. Two fellowship-trained breast radiologists used the BI-RADS ultrasound lexicon to assess pretreatment tumor shape, margin, echo pattern, orientation, posterior features, and vascularity. sTILs were defined as low <20 or high ≥20 on the pretreatment biopsy. Fisher's exact tests were used to assess the association between lexicon descriptors and sTIL levels. The 284 patients (mean age 52 years, range 24-79 years) were comprised of 68% (193/284) with low-sTIL tumors and 32% (91/284) with high-sTIL tumors. TNBC tumors with high sTILs were more likely to have the following features: (1) oval/round shape than irregular shape (p = 0.003), (2) circumscribed or microlobulated margins than spiculated, indistinct, or angular margins (p = 0.0005); (3) complex cystic and solid pattern than heterogeneous pattern (p = 0.006); and (4) posterior enhancement than shadowing (p = 0.002). There was no significant association between sTILs and descriptors for orientation and vascularity (p = 0.06 and p = 0.49, respectively). BI-RADS ultrasound descriptors of the pretreatment appearance of a TNBC tumor can be useful in discriminating between tumors with low and high sTIL levels. Therefore, there is a potential use of ultrasound tumor characteristics to complement sTILs when used as stratification factors in treatment algorithms for TNBC.

Infrared maritime dim small target detection based on spatiotemporal cues and directional morphological filtering

The existing small target detection algorithms will suffer severe performance deterioration when the weak target is submerged in heavy cluttered infrared (IR) maritime image. To settle this problem, a small target detection scheme is developed based on spatiotemporal cues and multidirectional morphological filtering. Initially, considering the small target will be isotropic because of the point spread function of the long-distance thermal imaging, while background edge clutters are generally local directional, the multidirectional structuring elements (MSEs) are constructed. By incorporating constructed MSEs and morphological operations, the directional morphological filtering (DMF) is established to measure the multidirectional differences between target region and local surroundings. Then, according to the brightness and isotropic Gaussian-like shape of small target in local spatial domain, the local intensity difference measure (LIDM) and the local Gaussian-like shape measure (LGSM) are presented to differentiate the small target from background clutters. After that, in order to integrate the two spatial properties, the multidirectional improved top-hat filter (MITHF) is proposed by fusing LIDM and LGSM to highlight potential small targets and suppress strong structural edge clutters. Finally, considering the consistency of targets and the fluctuation of sea clutters in temporal domain, a new appearance descriptor namely histogram of oriented morphological filtering (HOMF) is designed to characterize the small target by fully exploiting the advantages of DMF, and the HOMF-based multi-frame confirmation strategy (HMCS) is developed to further identify real small targets and eliminate false alarms. Extensive experiments prove that proposed algorithm is superior to the compared methods, especially for dim small target buried in intricate maritime background clutter.

Appearance-based passenger counting in cluttered scenes with lateral movement compensation

Autonomous passenger counting in public transportation represents an integral part of an intelligent transportation system, as it provides vital information to improve the efficiency and resource management of a public transportation network. However, counting passengers in highly crowded scenes is a challenging task due to their random movement, diverse appearance settings and inter-object occlusions. Furthermore, state-of-the-art methods in this domain rely heavily on additional custom cameras or sensors instead of existing onboard surveillance cameras, which consequently limits the feasibility of such systems for large-scale deployment. Hence, this paper puts forward an enhanced appearance descriptor with lateral movement compensation, which addresses the difficulty in counting passengers bidirectionally in cluttered scenes. We first construct a head re-identification dataset, which is used to train an appearance descriptor. This dataset addresses the absence of a person re-identification dataset, which in turn allows for accurate tracking of passengers in cluttered scenes. Then, a novel technique of applying a fedora counting line is introduced to count the number of passengers entering and exiting a bus. This technique compensates the impact of passengers’ lateral movement, which crucially increases the accuracy of bidirectional passenger counting using onboard bus surveillance cameras. In addition, a real-time implementation of the proposed method, which includes the integration of DeepStream and fedora counting line, is also presented. Experimental results on a challenging test dataset demonstrate that the proposed method outperforms benchmarked techniques with an average counting accuracy of 93.21% for entering and 96.10% for exiting public buses. Furthermore, the proposed system achieves this accuracy at an average frame rate of 16 frames per second, which represents a practical solution to a real-time application.

Person re-identification from appearance cues and deep Siamese features

Automated person re-identification in a multi-camera surveillance setup is very important for effective tracking and monitoring crowd movement. In this paper, we propose an efficient hierarchical re-identification approach in which color histogram-based comparison is employed to find the closest matches in the gallery set, and next deep feature-based comparison is carried out using the Siamese network. Reduction in search space after the first level of matching helps in improving the accuracy as well as efficiency of prediction by the Siamese network by eliminating dissimilar elements. A silhouette part-based feature extraction scheme is adopted in each level of hierarchy to preserve the relative locations of the different body parts and make the appearance descriptors more discriminating. The proposed approach has been evaluated on five public data sets and also a new data set captured in our laboratory. Results reveal that it outperforms most state-of-the-art approaches in terms of overall accuracy.

Emotional characterisation of chairs by descriptors, neural network and verbalised analysis

In this article, we assess the emotional value of the chairs from the collections design, objects and furniture of the museum of modern and contemporary art in Saint-Etienne. The aim of this research is to study emotional responses to visual perception of chairs from declarative measurements based on: 1) emotional measurements made using the circumplex model which collects valence and arousal; 2) the evaluation of six appearance descriptors which describe aesthetics, safety and comfort. Using a mathematical model based on a neural network, an 'estimated valence' is then derived from the evaluation of these emotional descriptors. This 'estimated valence' is finally compared to the 'verbalised valence'. The results show that there is a significant correlation between verbalised and 'estimated valence'. These results attest that the evaluation of the six proposed descriptors could predict the emotional value of the chairs.

Emotional characterisation of chairs by descriptors, neural network and verbalised analysis

In this article, we assess the emotional value of the chairs from the collections design, objects and furniture of the museum of modern and contemporary art in Saint-Etienne. The aim of this research is to study emotional responses to visual perception of chairs from declarative measurements based on: 1) emotional measurements made using the circumplex model which collects valence and arousal; 2) the evaluation of six appearance descriptors which describe aesthetics, safety and comfort. Using a mathematical model based on a neural network, an 'estimated valence' is then derived from the evaluation of these emotional descriptors. This 'estimated valence' is finally compared to the 'verbalised valence'. The results show that there is a significant correlation between verbalised and 'estimated valence'. These results attest that the evaluation of the six proposed descriptors could predict the emotional value of the chairs.