Neural Network Based Object Detection Research Articles

Misleading attention and classification: An adversarial attack to fool object detection models in the real world

Object detection is a hot topic in computer vision (CV), and it has many applications in various security fields. However, many works have demonstrated that neural network-based object detection is vulnerable to adversarial attacks. In this paper, we study adversarial attacks on object detectors in the real world and propose a new adversarial attack called Misleading Attention and Classification Attack (MACA), which can generate adversarial patches to mislead the object detectors. Specifically, we propose a new scheme to generate adversarial patches to fool the object detector. Our scheme restricts the noise of the adversarial patches and aims to ensure that the generated adversarial patches are visually similar to natural images. The attack simulates the complex external physical environment and the 3D transformations of non-rigid objects to increase the robustness of adversarial patches. We attack the up-to-date object detectors (e.g., Yolo-V5), and we prove that our technique has strong transferability among different detectors. Extensive experiments show that it is feasible to transfer the digital adversarial patches to the real world while maintaining the transferability of adversarial patches among different models and the success rate of adversarial attacks.

At-scale evaluation of weight clustering to enable energy-efficient object detection

Accelerators implementing Deep Neural Networks (DNNs) for image-based object detection operate on large volumes of data due to fetching images and neural network parameters, especially if they need to process video streams, hence with high power dissipation and bandwidth requirements to fetch all those data. While some solutions exist to mitigate power and bandwidth demands for data fetching, they are often assessed in the context of limited evaluations with a scale much smaller than that of the target application, which challenges finding the best tradeoff in practice.This paper sets up the infrastructure to assess at-scale a key power and bandwidth optimization – weight clustering – for You Only Look Once v3 (YOLOv3), a neural network-based object detection system, using videos of real driving conditions. Our assessment shows that accelerators such as systolic arrays with an Output Stationary architecture turn out to be a highly effective solution combined with weight clustering. In particular, applying weight clustering independently per neural network layer, and using between 32 (5-bit) and 256 (8-bit) weights allows achieving an accuracy close to that of the original YOLOv3 weights (32-bit weights). Such bit-count reduction of the weights allows shaving bandwidth requirements down to 30%–40% of the original requirements, and reduces energy consumption down to 45%. This is based on the fact that (i) energy due to multiply-and-accumulate operations is much smaller than DRAM data fetching, and (ii) designing accelerators appropriately may make that most of the data fetched corresponds to neural network weights, where clustering can be applied. Overall, our at-scale assessment provides key results to architect camera-based object detection accelerators by putting together a real-life application (YOLOv3), and real driving videos, in a unified setup so that trends observed are reliable.

SLMS-SSD: Improving the balance of semantic and spatial information in object detection

With the development of deep learning technology, the research on convolutional neural network-based object detection is becoming more and more mature. However, most methods are unsatisfactory in dealing with the issue of semantic and spatial information imbalance. In this article, we extend the single-shot multibox detector SSD and propose a self-learning multi-scale object detection network by balancing the semantic information and spatial information, named SLMS-SSD. We first construct a shallow feature enhancement module to enhance the representation of small objects by extracting richer context information. Second, in terms of feature connectivity, we design a multi-scale feature selection module for intermediate layer features with a combination of top-down and direct up-samplings. Finally, in terms of feature strength, we design a self-learning feature fusion module for measuring the feature importance. We validate our model on the PASCAL VOC and MS COCO datasets, and the results demonstrate that it can effectively improve the accuracy of object detection, especially the accuracy of small object detection.

A paced multi-stage block-wise approach for object detection in thermal images.

The growing advocacy of thermal imagery in applications, such as autonomous vehicles, surveillance, and COVID-19 detection, necessitates accurate object detection frameworks for the thermal domain. Conventional methods could fall short, especially in situations with poor lighting, for instance, detection during night-time. In this paper, we propose a paced multi-stage block-wise framework for effectively detecting objects from thermal images. Our approach utilizes the pre-existing knowledge of deep neural network-based object detectors trained on large-scale natural image data to enhance performance in the thermal domain constructively. The employed, multi-stage approach drives our model to achieve higher accuracies. And the introduction of the pace parameter during domain adaption enables efficient training. Our experimental results demonstrate that the framework outperforms previous benchmarks on the FLIR ADAS dataset on the person, bicycle, and car categories. We have also illustrated further analysis of the framework, such as the effect of its components on accuracy and training efficiency, its generalizability to other thermal datasets, and its superior performance on night-time images in contrast to state-of-the-art RGB object detectors.

Energy-Efficiency-Oriented Vision Feedback Control of QCSP Systems: Linear ADRC Approach

How to save the energy of unmanned aerial vehicles (UAVs) and then enable long-distance transport is a very real and difficult task. However, for UAVs, the classic object detection algorithm, such as the deep convolutional neural network–based object detection algorithm and the classic flight control algorithm, such as the PID-based position control algorithm, require significant energy, which limits the application scenarios of the UAV system. In view of this problem, this paper proposes a lightweight object detection network and a linear active disturbance rejection controller (LADRC) for the quadrotor with the cable-suspended payload (QCSP) system to improve energy efficiency. The system uses a YOLOV3 network and embeds it into the Jesson NX mobile platform to accurately detect the target position. Furthermore, a nonlinear velocity controller with a cable-suspended structure to control the velocity of the payload, a LADRC algorithm is adopted to achieve fast and accurate control of the payload position. Simulations and real flight experiments show that the proposed object detection algorithm and the LADRC control strategy can save the energy of drone effectively.

Detection of Locomotive Signal Lights and Pedestrians on Railway Tracks Using Improved YOLOv4

Intelligent detection of locomotive signal lights and pedestrians on railway tracks is of great significance to the safe operation of locomotives, especially under bad illumination conditions. Due to the highly complicated operation environment of railway locomotives, it is relatively difficult to apply deep neural network-based object detection methods in the recognition of locomotive signal lights and railway pedestrians. This work, for the first time, proposes a real-time detection method based on improved YOLOv4 to recognize locomotive signal lights and railway pedestrians. In our method, the Region-of-Interest is introduced into the original network of YOLOv4 to improve the detection precision of pedestrians on railway tracks. Importantly, we establish a dataset called detection of locomotive signal lights and railway pedestrians (DLSLRP), which is dedicated to the training, testing, and validation of related convolutional neural networks. We evaluated the proposed detector on DLSLRP dataset, the experimental results suggest that our method can detect locomotive signal lights and railway pedestrians with high speed and accuracy under different illumination conditions. The mAP reaches 93.52%, and the detection speed achieves 25 FPS.

Intelligent driving system at opencast mines during foggy weather

ABSTRACT The fog in mining operations minimises the visibility, preventing drivers from a clear view, causing accidents and vehicle collisions. This paper provides an intelligent driving system for heavy earthmoving machinery operators in opencast mines, including hardware and software. Hardware contains high definition and thermal cameras, a global navigation satellite system (GNSS), radar, laser light, wireless devices, graphical processing unit, touch screen, etc. The software covers image stitching, image enhancement, and convolution neural network-based object detection. The display dashboard is divided into four windows. Each window represents a different view, i.e. 180° panorama view of the driving lane, GNSS tracking map, proximity radar detection view, and rear thermal camera view. An additional colour transfer method has been used in the existing image stitching method to reduce misalignment and ghost effect in the panorama output. The proposed method outperformed the existing methods, namely contrast limited adaptive histogram equalisation (CLAHE) and dark channel prior (DCP). The proposed image enhancement technique has increased contrast, entropy, and colour average by 0.069, 0.43, and 13.96, respectively, than CLAHE, and 0.994, 0.43, and 42.07 than DCP. The accuracy of the object detection model is 97%, and the overall processing time of all the algorithms is 0.44949 seconds.

Prediction of future paths of mobile objects using path library

Abstract In situational awareness, the ability to make predictions about the near future situation in the area under surveillance is often as essential as being aware of the current situation. We introduce a privacy-preserving instance-based prediction method, where a path library is collected by learning earlier paths of mobile objects in the area of surveillance. The input to the prediction is the most recent coordinates of the objects in the scene. Based on similarity to short segments of currently tracked paths, a relative weight is associated with each path in the library. Future paths are predicted by computing the weighted average of the library paths. We demonstrate the operation of a situational awareness system where privacy-preserving data are extracted from an inexpensive computer vision which consists of a camera-equipped Raspberry PI-based edge device. The system runs a deep neural network-based object detection algorithm on the camera feed and stores the coordinates, object class labels, and timestamps of the detected objects. We used probabilistic reasoning based on joint probabilistic data association, Hungarian algorithm, and Kalman filter to infer which detections from different time instances came from the same object.

Cross-domain learning using optimized pseudo labels: toward adaptive car detection in different weather conditions and urban cities

Convolutional neural networks based object detection usually assumes that training and test data have the same distribution, which, however, does not always hold in real-world applications. In autonomous vehicles, the driving scene (target domain) consists of unconstrained road environments which cannot all possibly be observed in training data (source domain) and this will lead to a sharp drop in the accuracy of the detector. In this paper, we propose a domain adaptation framework based on pseudo-labels to solve the domain shift. First, the pseudo-labels of the target domain images are generated by the baseline detector (BD) and optimized by our data optimization module to correct the errors. Then, the hard samples in a single image are labeled based on the optimization results of pseudo-labels. The adaptive sampling module is approached to sample target domain data according to the number of hard samples per image to select more effective data. Finally, a modified knowledge distillation loss is applied in the retraining module, and we investigate two ways of assigning soft-labels to the training examples from the target domain to retrain the detector. We evaluate the average precision of our approach in various source/target domain pairs and demonstrate that the framework improves over 10% average precision of BD on multiple domain adaptation scenarios on the Cityscapes, KITTI, and Apollo datasets.

CRDet: Improving Signet Ring Cell Detection by Reinforcing the Classification Branch.

Detecting signet ring cells on histopathologic images is a critical computer-aided diagnostic task that is highly relevant to cancer grading and patients' survival rates. However, the cells are densely distributed and exhibit diverse and complex visual patterns in the image, together with the commonly observed incomplete annotation issue, posing a significant barrier to accurate detection. In this article, we propose to mitigate the detection difficulty from a model reinforcement point of view. Specifically, we devise a Classification Reinforcement Detection Network (CRDet). It is featured by adding a dedicated Classification Reinforcement Branch (CRB) on top of the architecture of Cascade RCNN. The proposed CRB consists of a context pooling module to perform a more robust feature representation by fully making use of context information, and a feature enhancement classifier to generate a superior feature by leveraging the deconvolution and attention mechanism. With the enhanced feature, the small-sized cell can be better characterized and CRDet enjoys a more accurate signet ring cell identification. We validate our proposal on a large-scale real clinical signet ring cell data set. It is shown that CRDet outperforms several popular convolutional neural network-based object detection models on this particular task.

Feedback-driven loss function for small object detection

In the recent years, Convolutional Neural Network-based object detection has experienced impressive progress. Despite these improvements, the performance on the small object detection leaves much to be desired, small objects are often missed or wrongly detected. The reasonable explanation is that the supervisory signals on the small objects are insufficient, and through the analysis of loss distribution over different scales in the iterations, there is a significant gap between the loss provided by the small objects and large objects. In order to balance the loss distribution and alleviate the insufficient supervisory on the small objects, a Feedback-driven loss function is presented in this paper. The Feedback-driven loss function uses the loss distribution information as the feedback signal, compared with the original loss function, the Feedback-driven loss function can supervise small objects more effectively, and train the detectors in a more balanced way. Experiments have been conducted on various detectors, backbones, training periods and datasets. Compared to the current state-of-the-art method, the novel Feedback-driven loss function can achieve 2.3% relative improvement on the Mean Average Precision, and especially 3.5% improvement on the detection of small objects, with nearly no additional computation both in training and testing stages.

Adapting Gaussian YOLOv3 with transfer learning for overhead view human detection in smart cities and societies

Nowadays, deep neural networks are widely applied in sustainable smart cities and societies, including smart manufacturing, healthcare, industries, agriculture, surveillance, and various artificial intelligence-based real-life applications. In this regard, the human detection system has gained notable attention since it is recognized as a crucial task in intelligent surveillance applications. Researchers practiced a variety of computer vision and deep neural networks-based techniques for human detection-based applications; however, they often focused on the frontal view camera perspective. Thus, in this work, we have introduced a human detection system for intelligent surveillance in smart cities and societies with a completely distinct perspective, i.e., an overhead perspective that can provide sufficient visibility and coverage of a scene in congested and obstructed environments. However, human appearance can be difficult from such an extreme point of view, as there are significant variations in humans’ poses and appearances. Therefore, in this work, leveraging the deep neural network-based object detection technique, the Gaussian YOLOv3 algorithm is used for human detection. The algorithm determines the bounding box uncertainty by modeling its coordinates as a Gaussian parameter, improving accuracy and reducing false positives. A Gaussian YOLOv3 is combined with channel attention and feature intertwine modules to improve specific feature maps. The channel attention module is combined with the feature map to learn each channel's weight autonomously, improve the key features, and enhance the network's ability to discriminate between humans and background. At the same time, different channels of the feature map are intertwined to obtain more representative features. Finally, the features obtained from the attention and feature intertwine modules are fused to form an improved feature map. In addition, to further increase the detection accuracy of the algorithm for human detection, transfer learning is adopted. The experimental outcomes reveal that training improves the Gaussian YOLOv3 algorithm's potential for human detection with an overall detection accuracy of 94%.

Investigating response time and accuracy in online classifier learning for multimedia publish-subscribe systems

The enormous growth of multimedia content in the field of the Internet of Things (IoT) leads to the challenge of processing multimedia streams in real-time. Event-based systems are constructed to process event streams. They cannot natively consume multimedia event types produced by the Internet of Multimedia Things (IoMT) generated data to answer multimedia-based user subscriptions. Machine learning-based techniques have enabled rapid progress in solving real-world problems and need to be optimised for the low response time of the multimedia event processing paradigm. In this paper, we describe a classifier construction approach for the training of online classifiers, that can handle dynamic subscriptions with low response time and provide reasonable accuracy for the multimedia event processing. We find that the current object detection methods can be configured dynamically for the construction of classifiers in real-time, by tuning hyperparameters even when training from scratch. Our experiments demonstrate that deep neural network-based object detection models, with hyperparameter tuning, can improve the performance within less training time for the answering of previously unknown user subscriptions. The results from this study show that the proposed online classifier training based model can achieve accuracy of 79.00% with 15-min of training and 84.28% with 1-hour training from scratch on a single GPU for the processing of multimedia events.

Infrared object detection network compression using Lp normalized weight

In view of the characteristic that the infrared image has less texture compared with RGB image, an infrared object detection network compression method using Lp normalized weight was proposed. It aimed at improving the adaptability of convolutional neural network based object detection framework to the infrared images, and compressing the scale of network while improving its generalization ability. Firstly, the phenomenon that the sparsity of Lp normalized weight can be precisely controlled by adjusting p was revealed. Based on the phenomenon, a sparsification method for object detection network was proposed. It respectively trained the backbone network and the detector with Lp spherical gradient descent and classical gradient descent, to balance the network scale and fitting accuracy. The tests on simulated infrared image dataset show that, the proposed method is superior to the dense model on both of network scale and detection accuracy: in terms of network scale, the sparsification reduces the effective parameters of Faster R-CNN, Single Shot multibox Detector (SSD) and YOLOv3 by 52%, 78% and 66% respectively; it also improves the mean Average Precision (mAP) of Faster R-CNN, SSD and YOLOv3 by 0.1%, 0.3% and 0.2%, thus verifying the effectiveness of the proposed method.

Zero-Centered Fixed-Point Quantization With Iterative Retraining for Deep Convolutional Neural Network-Based Object Detectors

In the field of object detection, deep learning has greatly improved accuracy compared to previous algorithms and has been used widely in recent years. However, object detection using deep learning requires many hardware (HW) resources due to the huge computations for high performance, making it very difficult to run real-time on embedded platforms. Therefore, various compression methods have been studied to solve this problem. In particular, quantization methods greatly reduce the computational burden of deep learning by reducing the number of bits used for weights and activation functions in deep learning. However, most of these existing studies targeted only object classification and cannot be applied to object detection. Furthermore, most of the existing quantization studies are based on floating-point operations, which requires additional effort when implementing HW accelerators. This paper proposes an HW-friendly fixed-point-based quantization method that can also be applied to object detection. In the proposed method, the center of the weight distribution is adjusted to zero by subtracting the mean of weight parameters before quantization, and the retraining process is iteratively applied to minimize the accuracy drop caused by quantization. Furthermore, while applying the proposed method to object detection, performance degradation is minimized by considering the minimum and maximum values of weight parameters of deep learning networks. When applying the proposed quantization method to representative one-stage object detectors, You Only Look Once v3 and v4 (YOLOv3 and YOLOv4), detection accuracy similar to the original networks (i.e., YOLOv3 and YOLOv4) with a single-precision floating-point format (32-bit) is maintained despite expressing weights with only about 20% of the bits compared to a single-precision floating-point format in COCO dataset.

The Role of a Mask - Understanding the Performance of Deep Neural Networks to Detect, Segment, and Extract Cellular Nuclei from Microscopy Images

Cell segmentation is the task of identifying cell nuclei instances in fluorescence microscopy images. It plays a key role in biomedical analysis tasks like cell characterization, cancer cell identification, and gene expression measurement. There has been a recent proliferation of deep neural network-based object detection techniques for cell segmentation. However, previous literature does not suitably address the proper understanding of the different methods and the merits of the various neural architectures proposed. One key design point relates to the learning of segmentation masks and how neural methods go about detecting, segmenting, and extracting nuclei from unseen images. The goal of this paper is to benchmark the performance of representative deep learning techniques for cell nuclei segmentation using standard datasets and common evaluation criteria. We investigate whether for cell nuclei segmentation, learning the nuclei masks in parallel with boundary detection features of the image provides superior performance and significantly cleaner separation of nuclei from images. We further explore the trade-off between increased accuracy, achieved through more complex deep learning models, and the heavy requirements imposed on both computational resources and training times. We believe this paper establishes an important baseline for cell nuclei segmentation, enabling researchers to continually refine and deploy neural models for real-world clinical applications.

Generative adversarial networks and faster-region convolutional neural networks based object detection in X-ray baggage security imagery

X-ray images have now been widely used in public transportation security. However, the ever-increasing number of dangerous object types, along with the complexity of the baggage's contents, makes it unsuitable for clearly displaying various baggages and hazards on X-ray images obtained directly from the inspection system. This article aims to improve the detection of dangerous objects by proposing a data augmentation method for enriching the X-ray prohibited item images using a Generative Adversarial Networks(GANs) based approach. Using the improved GANs model, a new X-ray images were generated with better quality and diversity, and the performance of the proposed method was evaluated using FID(Fréchet Inception Distance) score and Faster R-CNN (Region-based Convolutional Neural Network). Experimental results show that the new augmented X-ray security inspection image dataset can improve the detector performance. If other representative training datasets are utilized, we believe that our methodology could aid in the detection of other kinds of threat objects.

Toward plant organs in nature: a new database for plant organ system

The detection of plant organs is an important research field of plant recognition area. However, due to the lack of database of plant organs, the application of convolutional neural network-based object detection on plant species is very limited. A database of plant organs for deep learning-based object detection is constructed. A huge number of plant images are clawed using specific keywords through keyword search engines such as Baidu and Google. After that, an automatic junk image cleaning method is performed to remove junk images. Finally, artificial labeling is used to delineate plant organ regions. To evaluate the quality of the database, experiments in different object detection models are implemented. Results show that the established plant organ database has good performance in plant organs positioning and classification.

Reverse-engineering bar charts using neural networks

Reverse-engineering bar charts extract textual and numeric information from the visual representations of bar charts to support application scenarios that require the underlying information. In this paper, we propose a neural network-based method for reverse-engineering bar charts. We adopt a neural network-based object detection model to simultaneously localize and classify textual information. This approach improves the efficiency of textual information extraction. We design an encoder-decoder framework that integrates convolutional and recurrent neural networks to extract numeric information. We further introduce an attention mechanism into the framework to achieve high accuracy and robustness. Synthetic and real-world datasets are used to evaluate the effectiveness of the method. To the best of our knowledge, this work takes the lead in constructing a complete neural network-based method of reverse-engineering bar charts.

Semantic Loop Closure Detection With Instance-Level Inconsistency Removal in Dynamic Industrial Scenes

A novel semantic loop closure detection (SLCD) method is proposed in this article for visual simultaneous localization and mapping systems. SLCD aims to relieve the instance-level semantic inconsistency issue that arose from dynamic industrial scenes (e.g., autonomous driving in big cities). As the first step in this direction, SLCD fully exploits both low- and high-level video frame information, in a coarse-to-fine way. In SLCD, we adopt a convolutional neural network based object detection to acquire object information from the consecutive frames. Meanwhile, we perform a bag of visual words based similarity calculation to narrow the frames to coarse loop closure candidates. For these candidates, we perform an object matching on them to find their semantic inconsistency cases and remove involved semantic inconsistencies according to their cases. Then, we recalculate the similarity scores for these candidates. Finally, loop closures are determined by the similarity scores and a geometrical verification. Favorable performance of the proposed method is demonstrated by comparing it to other state-of-the-art methods using data from several public datasets and our new Dynamic Scenes dataset.