Extract Appearance Features Research Articles

Enhanced Vehicle Recognition and Collision Alert System Using Yolo+DeepSORT Algorithm

Abstract This paper sets out to develop and deploy a deep learning-driven vehicle collision avoidance warning system. Initially, it delves into the foundational principles of traditional collision avoidance warning systems, alongside key safety driving concepts. It then elaborates on leveraging deep learning technology to enhance the precision and reliability of target tracking. This includes detailed discussions on techniques such as appearance feature extraction, Kalman filters, the Hungarian algorithm, and more. In the technical section, the paper outlines the specific implementation of utilizing the YOLO algorithm for target detection and the DeepSORT algorithm for target tracking within the system architecture. The experimental section showcases the system’s performance and stability through formatted experimental results, followed by a comprehensive analysis of the data obtained. Concluding, the paper summarizes its research findings, underscoring the significance and feasibility of a deep learning-based vehicle collision avoidance warning system.

Multimodal and multiscale feature fusion for weakly supervised video anomaly detection

Weakly supervised video anomaly detection aims to detect anomalous events with only video-level labels. In the absence of boundary information for anomaly segments, most existing methods rely on multiple instance learning. In these approaches, the predictions for unlabeled video snippets are guided by the classification of labeled untrimmed videos. However, these methods do not account for issues such as video blur and visual occlusion, which can hinder accurate anomaly detection. To address these issues, we propose a novel weakly supervised video anomaly detection method that fuses multimodal and multiscale features. Firstly, RGB and optical flow snippets are input into pre-trained I3D to extract appearance and motion features. Then, we introduce an Attention De-redundancy (AD) module, which employs an attention mechanism to filter out task-irrelevant redundancy in these appearance and motion features. Next, to mitigate the effects of video blurring and visual occlusion, we propose a Multi-scale Feature Learning module. This module captures long-term and short-term temporal dependencies among video snippets to provide global and local guidance for blurred or occluded video snippets. Finally, to effectively utilize the discriminative features of different modalities, we propose an Adaptive Feature Fusion module. This module adaptively fuses appearance and motion features based on their respective feature weights. Extensive experimental results demonstrate that our proposed method outperforms mainstream unsupervised and weakly supervised methods in terms of AUC. Specifically, our proposed method achieves 97.00% AUC and 85.31% AUC on two benchmark datasets, i.e., ShanghaiTech and UCF-Crime, respectively.

A deep image classification model based on prior feature knowledge embedding and application in medical diagnosis

Aiming at the problem of image classification with insignificant morphological structural features, strong target correlation, and low signal-to-noise ratio, combined with prior feature knowledge embedding, a deep learning method based on ResNet and Radial Basis Probabilistic Neural Network (RBPNN) is proposed model. Taking ResNet50 as a visual modeling network, it uses feature pyramid and self-attention mechanism to extract appearance and semantic features of images at multiple scales, and associate and enhance local and global features. Taking into account the diversity of category features, channel cosine similarity attention and dynamic C-means clustering algorithms are used to select representative sample features in different category of sample subsets to implicitly express prior category feature knowledge, and use them as the kernel centers of radial basis probability neurons (RBPN) to realize the embedding of diverse prior feature knowledge. In the RBPNN pattern aggregation layer, the outputs of RBPN are selectively summed according to the category of the kernel center, that is, the subcategory features are combined into category features, and finally the image classification is implemented based on Softmax. The functional module of the proposed method is designed specifically for image characteristics, which can highlight the significance of local and structural features of the image, form a non-convex decision-making area, and reduce the requirements for the completeness of the sample set. Applying the proposed method to medical image classification, experiments were conducted based on the brain tumor MRI image classification public dataset and the actual cardiac ultrasound image dataset, and the accuracy rate reached 85.82% and 83.92% respectively. Compared with the three mainstream image classification models, the performance indicators of this method have been significantly improved.

Enhanced YOLOv5s + DeepSORT method for highway vehicle speed detection and multi-sensor verification

Addressing the need for vehicle speed measurement in traffic surveillance, this study introduces an enhanced scheme combining YOLOv5s detection with Deep SORT tracking. Tailored to the characteristics of highway traffic and vehicle features, the dataset data augmentation process was initially optimized. To improve the detector’s recognition capabilities, the Swin Transformer Block module was incorporated, enhancing the model’s ability to capture local regions of interest. CIoU loss was employed as the loss function for the vehicle detection network, accelerating model convergence and achieving higher regression accuracy. The Mish activation function was utilized to reduce computational overhead and enhance convergence speed. The structure of the Deep SORT appearance feature extraction network was modified, and it was retrained on a vehicle re-identification dataset to mitigate identity switches due to obstructions. Subsequently, using known references in the image such as lane markers and contour labels, the transformation from image pixel coordinates to actual coordinates was accomplished. Finally, vehicle speed was measured by computing the average of instantaneous speeds across multiple frames. Through radar and video Multi-Sensor Verification, the experimental results show that the mean Average Precision (mAP) for target detection consistently exceeds 90%. The effective measurement distance for speed measurement is around 140 m, with the absolute speed error generally within 1–8 km/h, meeting the accuracy requirements for speed measurement. The proposed model is reliable and fully applicable to highway scenarios.

DeepSORT with siamese convolution autoencoder embedded for honey peach young fruit multiple object tracking

The young fruit bagging robot usually preferentially selects the object with largest prediction box size for bagging. However, due to the uncertainty in the output of the traditional models, the expected endpoint of mechanical motion may change constantly, which affects the bagging efficiency. This paper proposes a novel tracking algorithm for honey peach young fruit targets. Firstly, according to the proportion of young fruit prediction box in the original image, a staged data preprocessing strategy is designed. Then, the siamese convolution autoencoder (SCAE) is developed by combining autoencoding mechanism with siamese training mode, and the SCAE is trained in two-stages to ensure the discriminability of the extracted appearance features. Finally, the SCAE is inserted into the DeepSORT architecture to generate the DeepSORT-SCAE object tracking algorithm, which can measure the spatially continuous trajectory of multiple young fruit targets within video streams. The effectiveness of the developed method is verified by experiments.

A Convolutional Neural Network Based on Soft Attention Mechanism and Multi-Scale Fusion for Skin Cancer Classification

The seven most common skin diseases are melanocytic nevus, melanoma, benign keratosis, basal cell carcinoma, actinic keratosis, vascular lesions, and dermatofibroma. Among them, melanoma has been identified as one of the deadliest cancers based on medical studies and research. The current trend in disease detection revolves around the use of machine learning and deep learning models. Regardless of the model used, the crucial aspect is achieving accurate classification for these diseases. With the emergence of powerful convolutional neural networks (CNNs), significant progress has been made in classification of skin cancer lesions in recent years. However, various challenges hinder the development of practical and effective solutions. First, due to the specific nature of skin cancer lesion images, the current deep neural network architectures and training strategies have poor adaptability to medical images. They are also prone to gradient vanishing issues during network iteration, which hinders the construction of high-performance deep learning models that leverage distinctive characteristics of skin lesion images. Second, there exists a discordance between skin lesion images and deep learning network structures. To address these issues, this study introduces a soft attention mechanism to enhance adaptability to skin cancer lesion images and improve the extraction of informative features from medical images. Additionally, a novel multi-scale fusion convolutional neural network model is proposed to overcome the mismatch between deep learning CNN architectures and skin lesion images. This model autonomously extracts appearance features from raw dermatological medical images. Comparisons with other popular techniques demonstrate the effectiveness of the proposed model, which can achieve an accuracy of 93.9% on HAM10000 dataset. There is ongoing research to overcome the remaining challenges and further enhance the performance of skin cancer classification algorithms.

Improve multi-object tracking using yolov5 and deep-sort for Chinese yam counting model

<p>Accurate counting information is important for crop yield and quality. Obtaining yield estimation of Chinese Yam is critical to improving productivity. Referring to the method of counting pedestrian flow in surveillance video and its accuracy, a method for counting the number of Chinese Yams harvested in the field is proposed by improving YOLOv5s detection combined with Deep-SORT tracking. In order to improve the recognition effect of the detector, the attention module CBAM is fused with the Neck part of the YOLOv5s network to improve the feature extraction ability of the network; CIoU Loss is used instead of GIoU Loss as the target bounding box regression loss function to speed up the bounding box regression rate while improving positioning Accuracy; use DIoU-NMS to replace NMS to improve the missed detection problem when the target is crowded. Adjust the structure of the Deep SORT appearance feature extraction network and retrain on the yam re-identification dataset to reduce the identity switching caused by target occlusion. Connect the improved YOLOv5s detector and Deep SORT, and set a virtual detection line in the video to count the number of yams. The experimental results show that the number of yams can be counted more accurately. Compared with the original algorithm, the improved YOLOv5s has an average accuracy rate of 2.1 percentage points. Combined with Deep-SORT tracking, the counting statistics accuracy rate reaches 92.7%.<strong></strong></p>

Progressive Instance-Aware Feature Learning for Compositional Action Recognition.

In order to enable the model to generalize to unseen "action-objects" (compositional action), previous methods encode multiple pieces of information (i.e., the appearance, position, and identity of visual instances) independently and concatenate them for classification. However, these methods ignore the potential supervisory role of instance information (i.e., position and identity) in the process of visual perception. To this end, we present a novel framework, namely Progressive Instance-aware Feature Learning (PIFL), to progressively extract, reason, and predict dynamic cues of moving instances from videos for compositional action recognition. Specifically, this framework extracts features from foreground instances that are likely to be relevant to human actions (Position-aware Appearance Feature Extraction in Section III-B1), performs identity-aware reasoning among instance-centric features with semantic-specific interactions (Identity-aware Feature Interaction in Section III-B2), and finally predicts instances' position from observed states to force the model into perceiving their movement (Semantic-aware Position Prediction in Section III-B3). We evaluate our approach on two compositional action recognition benchmarks, namely, Something-Else and IKEA-Assembly. Our approach achieves consistent accuracy gain beyond off-the-shelf action recognition algorithms in terms of both ground truth and detected position of instances.

Video Action Recognition by Combining Spatial-Temporal Cues with Graph Convolutional Networks

Video action recognition relies heavily on the way spatio-temporal cues are combined in order to enhance recognition accuracy. This issue can be addressed with explicit modeling of interactions among objects within or between videos, such as the graph neural network, which has been shown to accurately model and represent complicated spatial- temporal object relations for video action classification. However, the visual objects in the video are diversified, whereas the nodes in the graphs are fixed. This may result in information overload or loss if the visual objects are too redundant or insufficient for graph construction. Segment level graph convolutional networks (SLGCNs) are proposed as a method for recognizing actions in videos. The SLGCN consists of a segment-level spatial graph and a segment-level temporal graph, both of which are capable of simultaneously processing spatial and temporal information. Specifically, the segment-level spatial graph and the segment-level temporal graph are constructed using 2D and 3D CNNs to extract appearance and motion features from video segments. Graph convolutions are applied in order to obtain informative segment-level spatial-temporal features. A variety of challenging video datasets, such as EPIC-Kitchens, FCVID, HMDB51 and UCF101, are used to evaluate our method. In experiments, it is demonstrated that the SLGCN can achieve performance comparable to the state-of-the-art models in terms of obtaining spatial-temporal features.

DOMOPT: A Detection-Based Online Multi-Object Pedestrian Tracking Network for Videos

Due to the problem of low tracking accuracy and weak tracking stability of current multi-object pedestrian tracking algorithms in complex scenes for videos, a Detection-based Online Multi-Object Pedestrian Tracking (DOMOPT) network is proposed. First, a Multi-Level Feature Fusion (MLFF) pedestrian detection network is proposed based on the Center and Scale Prediction (CSP) algorithm. The pyramid convolutional neural network is used as the backbone to enhance the feature extraction capability for small objects. The shallow features and deep features at multiple levels are integrated to fully obtain the position and semantic information to further improve the detection performance for small objects. Then, on the basis of Joint Detection and Embedding (JDE) architecture, a Multi-Branch Pedestrian Appearance (MBPA) feature extraction network is proposed and added into the pedestrian detection network to extract the appearance feature vector corresponding to each pedestrian. The pedestrian appearance feature extraction is treated as a classification task jointly training with the pedestrian detection task, using the multi-task learning strategy. Experimental results show that the proposed network has better tracking accuracy and stability compared with state-of-the-art algorithms.

외형특징 추출 기능을 개선한 딥러닝 기반의 다중객체 추적

Multi-Object Tracking (MOT) aims to track the trajectories of multiple objects simultaneously in a video over time. In MOT, object detection is used to locate instances of objects in images or videos, and re-identification is employed to match objects with the same ID. Among recent MOT methods, one-shot MOT, which relies on a single neck and head for object detection and appearance feature extraction, faces a conflict between object category classification and unique ID classification. To address this challenge, we propose a solution that integrates an FPN neck-based appearance feature extraction module into YOLOX. This method effectively mitigates the conflict between object category classification and unique ID classification in one-shot MOT. Furthermore, the proposed variable offset mechanism enables correction of the extraction position, even in cases of object occlusion. The proposed technique achieved 83.7% mIDF1 and 75% mMOTA in the experimental results for the TITAN dataset. This improved mIDF1 and mMOTA by 0.2% and 0.2% respectively over the IOU-based MOT, and by 1.2% and 0.7% respectively over the one-shot MOT.

Bi-READ: Bi-Residual AutoEncoder based feature enhancement for video anomaly detection

Video anomaly detection (VAD) refers to identifying abnormal events in the surveillance video. Typically, reconstruction based video anomaly detection techniques employ convolutional autoencoders with a limited number of layers, which extracts insufficient features leading to improper network training. To address this challenge, an end-to-end unsupervised feature enhancement network, namely Bi-Residual Convolutional AutoEncoder (Bi-ResCAE) has been proposed that can learn normal events with low reconstruction error and detect anomalies with high reconstruction error. The proposed Bi-ResCAE network incorporates long–short residual connections to enhance feature reusability and training stabilization. In addition, we propose to formulate a novel VAD model that can extract appearance and motion features by fusing both the Bi-ResCAE network and optical flow network in the objective function to recognize the anomalous object in the video. Extensive experiments on three benchmark datasets validate the effectiveness of the model. The proposed model achieves an AUC (Area Under the ROC Curve) of 84.7% on Ped1, 97.7% on Ped2, and 86.71% on the Avenue dataset. The results show that the Bi-READ performs better than state-of-the-art techniques.

Detector–Tracker Integration Framework for Autonomous Vehicles Pedestrian Tracking

Pedestrian tracking is an important aspect of autonomous vehicles environment perception in a vehicle running environment. The performance of the existing pedestrian tracking algorithms is limited by the complex traffic environment, the changeable appearance characteristics of pedestrians and the frequent occlusion interaction, which leads to the insufficient accuracy and stability of tracking. Therefore, this paper proposes a detector–tracker integration framework for autonomous vehicle pedestrian tracking. Firstly, a pedestrian objects detector based on the improved YOLOv7 network was established. Space-to-Depth convolution layer was adopted to improve the backbone network of YOLOv7. Then, a novel appearance feature extraction network is proposed, which integrates the convolutional structural re-parameterization idea to construct a full-scale feature extraction block, which is the optimized DeepSORT tracker. Finally, experiments were carried out on MOT17 and MOT20 public datasets and driving video sequences, and the tracking performance of the proposed framework was evaluated by comparing it with the most advanced multi-object tracking algorithms. Quantitative analysis results show that the framework has high tracking accuracy. Compared with DeepSORT, MOTA improves by 2.3% in the MOT17 dataset and MOTA improves by 4.2% in the MOT20 dataset. Through qualitative evaluation on real driving video sequences, the framework proposed in this paper is robust in a variety of climate environments, and can be effectively applied to the pedestrian tracking of autonomous vehicles.

Behavior-Based Video Summarization System for Dog Health and Welfare Monitoring

The popularity of dogs has been increasing owing to factors such as the physical and mental health benefits associated with raising them. While owners care about their dogs' health and welfare, it is difficult for them to assess these, and frequent veterinary checkups represent a growing financial burden. In this study, we propose a behavior-based video summarization and visualization system for monitoring a dog's behavioral patterns to help assess its health and welfare. The system proceeds in four modules: (1) a video data collection and preprocessing module; (2) an object detection-based module for retrieving image sequences where the dog is alone and cropping them to reduce background noise; (3) a dog behavior recognition module using two-stream EfficientNetV2 to extract appearance and motion features from the cropped images and their respective optical flow, followed by a long short-term memory (LSTM) model to recognize the dog's behaviors; and (4) a summarization and visualization module to provide effective visual summaries of the dog's location and behavior information to help assess and understand its health and welfare. The experimental results show that the system achieved an average F1 score of 0.955 for behavior recognition, with an execution time allowing real-time processing, while the summarization and visualization results demonstrate how the system can help owners assess and understand their dog's health and welfare.

FPGA-Based Vehicle Detection and Tracking Accelerator.

A convolutional neural network-based multiobject detection and tracking algorithm can be applied to vehicle detection and traffic flow statistics, thus enabling smart transportation. Aiming at the problems of the high computational complexity of multiobject detection and tracking algorithms, a large number of model parameters, and difficulty in achieving high throughput with a low power consumption in edge devices, we design and implement a low-power, low-latency, high-precision, and configurable vehicle detector based on a field programmable gate array (FPGA) with YOLOv3 (You-Only-Look-Once-version3), YOLOv3-tiny CNNs (Convolutional Neural Networks), and the Deepsort algorithm. First, we use a dynamic threshold structured pruning method based on a scaling factor to significantly compress the detection model size on the premise that the accuracy does not decrease. Second, a dynamic 16-bit fixed-point quantization algorithm is used to quantify the network parameters to reduce the memory occupation of the network model. Furthermore, we generate a reidentification (RE-ID) dataset from the UA-DETRAC dataset and train the appearance feature extraction network on the Deepsort algorithm to improve the vehicles' tracking performance. Finally, we implement hardware optimization techniques such as memory interlayer multiplexing, parameter rearrangement, ping-pong buffering, multichannel transfer, pipelining, Im2col+GEMM, and Winograd algorithms to improve resource utilization and computational efficiency. The experimental results demonstrate that the compressed YOLOv3 and YOLOv3-tiny network models decrease in size by 85.7% and 98.2%, respectively. The dual-module parallel acceleration meets the demand of the 6-way parallel video stream vehicle detection with the peak throughput at 168.72 fps.

Silicon energy bulk material cargo ship detection and tracking method combining YOLOv5 and DeepSort

Silicon widely contained in sand is expected to become a new energy material with environmental protection, safety and low cost. The intelligent management of the exploitation and transportation of such energy resources has become an urgent demand. Aiming at solving the problem of low detection rate of silicon energy bulk material cargo ship targets in river monitoring videos, this paper proposes a multi-ship detection and tracking method based on YOLOv5x combined with DeepSort algorithm. In order to improve the detector recognition efficiency, CIoU Loss is used as the target bounding box regression loss function instead of GIoU Loss to speed up the bounding box regression rate while improving the localization accuracy; NMS is replaced by DIoU-NMS to tackle the problem of missed detection when the targets are dense. The structure of DeepSort appearance feature extraction network is adjusted and trained on a self-built ship dataset to reduce the identity switching caused by target occlusion. The experimental results show that YOLOv5x trained on Pytorch framework makes the model converge fast and accurate with mAP reaching 95.6%, and then combined with DeepSort tracking can achieve the detection and tracking of multiple types of ships. This can provide effective technical support for the supervision of illegal mining and transportation of silicon energy bulk material.

Enhanced robust spatial feature selection and correlation filter learning for UAV tracking

Spatial boundary effect can significantly reduce the performance of a learned discriminative correlation filter (DCF) model. A commonly used method to relieve this effect is to extract appearance features from a wider region of a target. However, this way would introduce unexpected features from background pixels and noises, which will lead to a decrease of the filter’s discrimination power. To address this shortcoming, this paper proposes an innovative method called enhanced robust spatial feature selection and correlation filter Learning (EFSCF), which performs jointly sparse feature learning to handle boundary effects effectively while suppressing the influence of background pixels and noises. Unlike the ℓ2-norm-based tracking approaches that are prone to non-Gaussian noises, the proposed method imposes the ℓ2,1-norm on the loss term to enhance the robustness against the training outliers. To enhance the discrimination further, a jointly sparse feature selection scheme based on the ℓ2,1 -norm is designed to regularize the filter in rows and columns simultaneously. To the best of the authors’ knowledge, this has been the first work exploring the structural sparsity in rows and columns of a learned filter simultaneously. The proposed model can be efficiently solved by an alternating direction multiplier method. The proposed EFSCF is verified by experiments on four challenging unmanned aerial vehicle datasets under severe noise and appearance changes, and the results show that the proposed method can achieve better tracking performance than the state-of-the-art trackers.

RISTrack: Robust Infrared Ship Tracking With Modified Appearance Feature Extraction and Matching Strategy

Infrared (IR) ship tracking is becoming increasingly important in various applications. However, it remains a challenging task as the information that can be obtained from infrared images is limited. Aiming at enhancing IR ship tracking accuracy, we propose an innovative approach by presenting feature integration module (FIM) and backup matching module (BMM). FIM takes appearance feature, complete intersection over union (CIoU), and motion direction metrics into account. Regarding appearance feature extraction, an end-to-end characteristic learning strategy with a cross-guided multi-granularity fusion network is proposed to obtain more integral appearance features and enhance re-identification accuracy, which helps to distinguish individual IR ship targets better. Besides, a backup matching strategy is then used to match the unmatched tracks and detections after cascaded matching. Virtual trajectories are generated for the matched tracks to optimize parameters by parameter optimization module (POM). The accumulation of errors caused by the lack of observations in the Kalman filter is reduced. Thus, the position of IR ships can be estimated more accurately, and more robust IR ship tracking can be achieved. In addition, we present a sequential frame IR ship tracking dataset, providing the first public benchmark for testing IR ship tracking performance. Experimental results indicate that the MOTA, MOTP and IDs of the proposed method are 73.441, 80.826, and 32, respectively, outperforming other state-of-the-art methods. This demonstrates the superior robustness of the proposed method, particularly when the IR ships are occluded or the target texture information is lacking. Our dataset is available at https://github.com/echo-sky/SFIST.

A person-following method based on monocular camera for quadruped robots

This paper proposes a person-following method based on monocular vision, which allows quadruped robots to track a target person in both indoor and outdoor environments with different illumination conditions. Our method is composed of a person detector, a Kalman filter (KF) tracker, and a re-identification module. To be more specific, the person detector uses a human pose estimation method to detect persons. The KF is applied to predict the position of the target person and update its state with detection results. A re-identification module is proposed to deal with distractions, where the Convolutional Channel Features (CCF) is used to extract appearance features and Online Boosting is used to distinguish the target person from others. Especially, we design a target recapture mechanism based on the Recurrent Neural Network (RNN). Combining motion information with appearance features, the system can accurately re-identify the target person. Without extra customized markers, our method can track the target person steadily in real-time only using a monocular camera. Experiments results can validate the robustness and effectiveness of the proposed method.

A motion-appearance-aware network for object change detection

Object change detection (OCD), which aims to segment moving objects from an input frame, has attracted growing attention. Most OCD algorithms rely on scene diversity or ignore the interframe spatiotemporal structural dependence, which limits their applicability. In this paper, we propose a motion-appearance-aware network (MAAN) for learning robust feature representations. Specifically, a module for mining information at multiple time scales, which can adaptively adjust information elements, is designed to refine the motion feature. Meanwhile, salient object knowledge is obtained with the help of the extracted appearance features. To enhance the semantic consistency and trim redundant connections, we construct a fusion module called multi-view feature evolution, which effectively fuses motion and appearance information by global communication and local guidance, respectively. Moreover, we develop two strategies for obtaining uniform and consistent moving objects during information propagation. One is to feed the predicted mask of the previous frame into the decoder, and the other is to match different levels of motion cues at multiple time scales to the decoder. Finally, extensive experiments on four public datasets (i.e., LASIESTA, CDnet2014, INO, and AICD) indicate that the proposed approach outperforms other methods.