Vision-based Object Detection Research Articles

Advanced deep learning framework for underwater object detection with multibeam forward-looking sonar

Underwater object detection (UOD) is an essential activity in maintaining and monitoring underwater infrastructure, playing an important role in their efficient and low-risk asset management. In underwater environments, sonar, recognized for overcoming the limitations of optical imaging in low-light and turbid conditions, has increasingly gained popularity for UOD. However, due to the low resolution and limited foreground-background contrast in sonar images, existing sonar-based object detection algorithms still face challenges regarding precision and transferability. To solve these challenges, this article proposes an advanced deep learning framework for UOD that uses the data from multibeam forward-looking sonar. The framework is adapted from the network architecture of YOLOv7, one of the state-of-the-art vision-based object detection algorithms, by incorporating unique optimizations in three key aspects: data preprocessing, feature fusion, and loss functions. These improvements are extensively tested on a dedicated public dataset, showing superior object classification performance compared to the selected existing sonar-based methods. Through experiments conducted on an underwater remotely operated vehicle, the proposed framework validates significant enhancements in target classification, localization, and transfer learning capabilities. Since the engineering structures have similar geometric shapes to the objects tested in this study, the proposed framework presents potential applicability to underwater structural inspection and monitoring, and autonomous asset management.

AN EFFCIENT SYSTEM FOR DETECTING TRAFFIC VIOLATIONS SUCH AS OVER SPEED, DISREGARDING SIGNALS, AND INSTANCES OF TRIPLE RIDING

In recent time surveys, the deaths and injuries due to traffic violations have increased chiefly in Indian roads. So, this needed the assistance of an automated computer vision-based object detection model, as manually identifying the vehicles violating traffic is hectic. The principle of this paper is to detect multiple violations using single video frames. The input video stream obtained from the surveillance camera is processed and annotated to carry out multiple processes. The dataset used for red-light jumping is COCO and the dataset for over boarding is created by annotating the images obtained from google. The model is trained, and the output is visualized using tensor board. The accuracy for red light skipping is 93% and the mAP value for over boarding is 0.5:0.95. This system utilizes the video stream at its maximum to detect various violations.

Stereo vision based object detection for autonomous navigation in space environments

Obstacle detection and avoidance are the major issues in autonomous navigation for partially known or unknown environments. With the proliferation of space debris, researchers are actively investigating debris removal to facilitate future space operations. This calls for the development of autonomous navigation techniques for space missions. Free-space object detection is a crucial task in intelligent systems, particularly for path planning. In this study, we propose a stereo vision-based intelligent system for space object detection, using two vertically aligned omnidirectional stereo cameras separated by 10 cm. Firstly, a single-shot multibox detector (SSD) based on deep learning is employed to identify the objects present in the image. Then, the triangulation method is used to determine the distance between the object and the system. The proposed system can provide object depth information up to a maximum range of 1.1 km in a space environment.

MR Object Identification and Interaction

The increasing number of objects in ubiquitous computing environments creates a need for effective object detection and identification mechanisms that permit users to intuitively initiate interactions with these objects. While multiple approaches to such object detection -- including through visual object detection, fiducial markers, relative localization, or absolute spatial referencing -- are available, each of these suffers from drawbacks that limit their applicability. In this paper, we propose ODIF, an architecture that permits the fusion of object situation information from such heterogeneous sources and that remains vertically and horizontally modular to allow extending and upgrading systems that are constructed accordingly. We furthermore present BLEARVIS, a prototype system that builds on the proposed architecture and integrates computer-vision (CV) based object detection with radio-frequency (RF) angle of arrival (AoA) estimation to identify BLE-tagged objects. In our system, the front camera of a Mixed Reality (MR) head-mounted display (HMD) provides a live image stream to a vision-based object detection module, while an antenna array that is mounted on the HMD collects AoA information from ambient devices. In this way, BLEARVIS is able to differentiate between visually identical objects in the same environment and can provide an MR overlay of information (data and controls) that relates to them. We include experimental evaluations of both, the CV-based object detection and the RF-based AoA estimation, and discuss the applicability of the combined RF and CV pipelines in different ubiquitous computing scenarios. This research can form a starting point to spawn the integration of diverse object detection, identification, and interaction approaches that function across the electromagnetic spectrum, and beyond.

A Millimeter-Wave Radar-Aided Vision Detection Method for Water Surface Small Object Detection

Unmanned surface vehicles (USVs) have wide applications in marine inspection and monitoring, terrain mapping, and water surface cleaning. Accurate and robust environment perception ability is essential for achieving autonomy in USVs. Small object detection on water surfaces is an important environment perception task, typically achieved by visual detection using cameras. However, existing vision-based small object detection methods suffer from performance degradation in complex water surface environments. Therefore, in this paper, we propose a millimeter-wave (mmWave) radar-aided vision detection method that enables automatic data association and fusion between mmWave radar point clouds and images. Through testing on real-world data, the proposed method demonstrates significant performance improvement over vision-based object detection methods without introducing more computational costs, making it suitable for real-time application on USVs. Furthermore, the image–radar data association model in the proposed method can serve as a plug-and-play module for other object detection methods.

Object Detection for Agricultural Vehicles: Ensemble Method Based on Hierarchy of Classes.

Vision-based object detection is essential for safe and efficient field operation for autonomous agricultural vehicles. However, one of the challenges in transferring state-of-the-art object detectors to the agricultural domain is the limited availability of labeled datasets. This paper seeks to address this challenge by utilizing two object detection models based on YOLOv5, one pre-trained on a large-scale dataset for detecting general classes of objects and one trained to detect a smaller number of agriculture-specific classes. To combine the detections of the models at inference, we propose an ensemble module based on a hierarchical structure of classes. Results show that applying the proposed ensemble module increases mAP@.5 from 0.575 to 0.65 on the test dataset and reduces the misclassification of similar classes detected by different models. Furthermore, by translating detections from base classes to a higher level in the class hierarchy, we can increase the overall mAP@.5 to 0.701 at the cost of reducing class granularity.

MsFireD-Net: A lightweight and efficient convolutional neural network for flame and smoke segmentation

With the rising frequency and severity of wildfires across the globe, researchers have been actively searching for a reliable solution for early-stage forest fire detection. In recent years, Convolutional Neural Networks (CNNs) have demonstrated outstanding performances in computer vision-based object detection tasks, including forest fire detection. Using CNNs to detect forest fires by segmenting both flame and smoke pixels not only can provide early and accurate detection but also additional information such as the size, spread, location, and movement of the fire. However, CNN-based segmentation networks are computationally demanding and can be difficult to incorporate onboard lightweight mobile platforms, such as an Uncrewed Aerial Vehicle (UAV). To address this issue, this paper has proposed a new efficient upsampling technique based on transposed convolution to make segmentation CNNs lighter. This proposed technique, named Reversed Depthwise Separable Transposed Convolution (RDSTC), achieved F1-scores of 0.78 for smoke and 0.74 for flame, outperforming U-Net networks with bilinear upsampling, transposed convolution, and CARAFE upsampling. Additionally, a Multi-signature Fire Detection Network (MsFireD-Net) has been proposed in this paper, having 93% fewer parameters and 94% fewer computations than the RDSTC U-Net. Despite being such a lightweight and efficient network, MsFireD-Net has demonstrated strong results against the other U-Net-based networks.

Efficient GEMM Implementation for Vision-Based Object Detection in Autonomous Driving Applications

Convolutional Neural Networks (CNNs) have been incredibly effective for object detection tasks. YOLOv4 is a state-of-the-art object detection algorithm designed for embedded systems. It is based on YOLOv3 and has improved accuracy, speed, and robustness. However, deploying CNNs on embedded systems such as Field Programmable Gate Arrays (FPGAs) is difficult due to their limited resources. To address this issue, FPGA-based CNN architectures have been developed to improve the resource utilization of CNNs, resulting in improved accuracy and speed. This paper examines the use of General Matrix Multiplication Operations (GEMM) to accelerate the execution of YOLOv4 on embedded systems. It reviews the most recent GEMM implementations and evaluates their accuracy and robustness. It also discusses the challenges of deploying YOLOv4 on autonomous vehicle datasets. Finally, the paper presents a case study demonstrating the successful implementation of YOLOv4 on an Intel Arria 10 embedded system using GEMM.

Wood Veneer Defect Detection Based on Multiscale DETR with Position Encoder Net

Wood is one of the main building materials. However, defects on veneers result in substantial waste of wood resources. Traditional veneer defect detection relies on manual experience or photoelectric-based methods, which are either subjective and inefficient or need substantial investment. Computer vision-based object detection methods have been used in many realistic areas. This paper proposes a new deep learning defect detection pipeline. First, an image collection device is constructed and a total of more than 16,380 defect images are collected coupled with a mixed data augmentation method. Then, a detection pipeline is designed based on DEtection TRansformer (DETR). The original DETR needs position encoding functions to be designed and is ineffective for small object detection. To solve these problems, a position encoding net is designed with multiscale feature maps. The loss function is also redefined for much more stable training. The results from the defect dataset show that using a light feature mapping network, the proposed method is much faster with similar accuracy. Using a complex feature mapping network, the proposed method is much more accurate with similar speed.

Vision-Based Robotic Object Grasping—A Deep Reinforcement Learning Approach

This paper focuses on developing a robotic object grasping approach that possesses the ability of self-learning, is suitable for small-volume large variety production, and has a high success rate in object grasping/pick-and-place tasks. The proposed approach consists of a computer vision-based object detection algorithm and a deep reinforcement learning algorithm with self-learning capability. In particular, the You Only Look Once (YOLO) algorithm is employed to detect and classify all objects of interest within the field of view of a camera. Based on the detection/localization and classification results provided by YOLO, the Soft Actor-Critic deep reinforcement learning algorithm is employed to provide a desired grasp pose for the robot manipulator (i.e., learning agent) to perform object grasping. In order to speed up the training process and reduce the cost of training data collection, this paper employs the Sim-to-Real technique so as to reduce the likelihood of damaging the robot manipulator due to improper actions during the training process. The V-REP platform is used to construct a simulation environment for training the deep reinforcement learning neural network. Several experiments have been conducted and experimental results indicate that the 6-DOF industrial manipulator successfully performs object grasping with the proposed approach, even for the case of previously unseen objects.

Research Challenges, Recent Advances, and Popular Datasets in Deep Learning-Based Underwater Marine Object Detection: A Review.

Underwater marine object detection, as one of the most fundamental techniques in the community of marine science and engineering, has been shown to exhibit tremendous potential for exploring the oceans in recent years. It has been widely applied in practical applications, such as monitoring of underwater ecosystems, exploration of natural resources, management of commercial fisheries, etc. However, due to complexity of the underwater environment, characteristics of marine objects, and limitations imposed by exploration equipment, detection performance in terms of speed, accuracy, and robustness can be dramatically degraded when conventional approaches are used. Deep learning has been found to have significant impact on a variety of applications, including marine engineering. In this context, we offer a review of deep learning-based underwater marine object detection techniques. Underwater object detection can be performed by different sensors, such as acoustic sonar or optical cameras. In this paper, we focus on vision-based object detection due to several significant advantages. To facilitate a thorough understanding of this subject, we organize research challenges of vision-based underwater object detection into four categories: image quality degradation, small object detection, poor generalization, and real-time detection. We review recent advances in underwater marine object detection and highlight advantages and disadvantages of existing solutions for each challenge. In addition, we provide a detailed critical examination of the most extensively used datasets. In addition, we present comparative studies with previous reviews, notably those approaches that leverage artificial intelligence, as well as future trends related to this hot topic.

Computer vision with smartphone microphotography for detection of carious lesions

ObjectivesTo evaluate the similarities in microphotographic images across different smartphones and to establish whether computer vision can use microphotographs to successfully classify dental caries. MethodA universal clip-type microscope with 60x optical zoom was selected to perform in vitro microphotography of extracted teeth. For the first objective, areas of cariogenic interest were physically labelled by dentists and eight smartphones were used to capture images of tooth decays with the microscope fitted over the primary camera lens. For the second objective, 233 microphotography images were acquired and virtually augmented to produce 1631 images that were categorized digitally by an international caries classification system for computer vision-based object detection (YOLO.v4). Five practitioners independently labelled randomly selected images from the test dataset following the caries classification system which were subsequently used to evaluate the diagnostic test accuracy of the YOLO model. ResultA significant overall mean square error [F (df) = 4.03 (6); P < 0.05] was observed while Bhattacharya's distance evaluation produced no significant differences [F (df) = 1.60 (6); P > 0.05] across all eight smartphone derived datasets. Index and reference test comparisons determined an overall sensitivity of 0.99 and specificity of 0.94 for the trained YOLO.v4 and highly significant correlations (r > 0.9, P < 0.001) to the classifications labelled by the dental practitioners. ConclusionNon-standardized images of tooth caries captured by different smartphones generated an accurate diagnostic model for classifying carious lesions that was similar to the visual assessments performed by experienced dental practitioners.

Towards optimal foreign object debris detection in an airport environment

Foreign object debris (FOD) can critically damage aircraft engines as well as injure personnel in an airport environment. Airfield inspectors routinely inspect runways for the presence of FOD items, which differ in material, shape, and color, using conventional and automated methods. The major shortcoming of the current methods is their inability to detect all types of foreign objects in an accurate and timely manner for removal from the airport runways. In this study, we address this shortcoming, i.e., the lack of accuracy and timeliness in detection, by developing an object detection framework to detect FOD for quick removal from the airfields. Our proposed FOD detection framework consists of (i) unmanned aerial system (UAS) for inspecting and collecting data from the airfields, (ii) data preprocessing and augmentation techniques to counter the issue of learning on limited types of foreign objects, weather conditions, and airport surface materials that are present in the data sets, and (iii) a computer vision-based object detection model to attain high accuracy and faster inference time for deployment in a real-world airport environment. We generated the training data with the UAS at an air force range and developed various models, including the You Only Look Once (YOLO) object detector family of models (one-stage object detectors) in our framework. Our models are evaluated on previously unseen data collected by the UAS and a publicly available data set of FOD images. The experiment results demonstrate that our proposed approach with a version of the YOLO model (YOLOv4) with transfer learning provides a faster inference time for FOD detection and outperforms the other models in the precision and recall metric values.

Blind-Spot Collision Detection System for Commercial Vehicles Using Multi Deep CNN Architecture.

Buses and heavy vehicles have more blind spots compared to cars and other road vehicles due to their large sizes. Therefore, accidents caused by these heavy vehicles are more fatal and result in severe injuries to other road users. These possible blind-spot collisions can be identified early using vision-based object detection approaches. Yet, the existing state-of-the-art vision-based object detection models rely heavily on a single feature descriptor for making decisions. In this research, the design of two convolutional neural networks (CNNs) based on high-level feature descriptors and their integration with faster R-CNN is proposed to detect blind-spot collisions for heavy vehicles. Moreover, a fusion approach is proposed to integrate two pre-trained networks (i.e., Resnet 50 and Resnet 101) for extracting high level features for blind-spot vehicle detection. The fusion of features significantly improves the performance of faster R-CNN and outperformed the existing state-of-the-art methods. Both approaches are validated on a self-recorded blind-spot vehicle detection dataset for buses and an online LISA dataset for vehicle detection. For both proposed approaches, a false detection rate (FDR) of 3.05% and 3.49% are obtained for the self recorded dataset, making these approaches suitable for real time applications.

Indigenous design of a Traffic Light Control system responsive to the local traffic dynamics and priority vehicles

Automation of traffic light control (TLC) systems is the major focus area compliant with Industry 4.0. The majority of the adaptive TLC systems are designed for homogeneous traffic of four-wheeler vehicles and are responsive to the prevailing traffic conditions at a junction. Further, they do not serve priority vehicles such as public transport (buses) or emergency vehicles (ambulance, fire trucks, etc.). The traffic dynamics considered in this work are less-lane-disciplined, heterogeneous traffic (two-three-four-wheeler vehicles), as found in the majority of developing countries. This work proposes a novel adaptive TLC system for heterogeneous traffic scenarios which is responsive to the prevailing traffic conditions at a junction, and also serves priority vehicles at a preference than the regular traffic. The proposed system supports multiple levels of priority among the priority vehicles, and efficiently schedules traffic lights at a junction to cause minimal interruption to the other regular traffic.For heterogeneous traffic monitoring, we use the Passenger Car Unit (PCU) count of the vehicles, which are computed using computer vision-based object detection models. The performance of deep learning-based object detection algorithms such as YOLO, RCNN, is analyzed experimentally for the computation requirements, and accuracy in computing PCU count. Simulations are carried out to analyze the effect of varying error rates in PCU count, and congestion level on the performance of the proposed adaptive TLC. The effect of the proposed adaptive TLC on the average travel and waiting times, of the priority vehicles and the other regular traffic is assessed. The simulation results suggest that the proposed algorithm can tolerate 20% error in the PCU count without degrading the performance. Additionally, this work demonstrates that the traffic information with the required accuracy can be processed in real-time using the available micro-controllers (e.g., Raspberry Pi or Jetson Nano).

Unmanned aerial vehicles object detection based on image haze removal under sea fog conditions

Unmanned aerial vehicles (UAVs) have gradually become a major air threat to ships because of small size, good maneuverability, and low cost. Vision-based UAV detection offers one of the main ways to identify and protect against UAVs. Unlike land environment, the weather is complicated at sea. The visibility of an object is undermined by such factors as sea fog and sunlight, which makes it difficult to detect UAVs at sea through vision-based object detection. For the purpose of object detection at sea, this paper proposes a UAV object detection method based on image haze removal. In the proposed method, an improved dark channel haze removal (DCHR) algorithm is utilized to remove haze for and restore video images. Additionally, co-ordinate attention (CoordAttention, CA) is introduced to the lightweight algorithms of You Only Look Once (YOLO) for the object detection in restored video images, so as to improve the precision and speed of detection and reduce the miss rate. Some video images are also taken for detection experiments to verify the feasibility and effectiveness of the proposed method.

Vehicle wheel weld detection based on improved YOLO v4 algorithm

In recent years, vision-based object detection has made great progress across different fields. For instance, in the field of automobile manufacturing, welding detection is a key step of weld inspection in wheel production. The automatic detection and positioning of welded parts on wheels can improve the efficiency of wheel hub production. At present, there are few deep learning based methods to detect vehicle wheel welds. In this paper, a method based on YOLO v4 algorithm is proposed to detect vehicle wheel welds. The main contributions of the proposed method are the use of k-means to optimize anchor box size, a Distance-IoU loss to optimize the loss function of YOLO v4, and non-maximum suppression using Distance-IoU to eliminate redundant candidate bounding boxes. These steps improve detection accuracy. The experiments show that the improved methods can achieve high accuracy in vehicle wheel weld detection (4.92 % points higher than the baseline model with respect to AP75 and 2.75 % points higher with respect to AP50). We also evaluated the proposed method on the public KITTI dataset. The detection results show the improved method’s effectiveness.

New Metric for Evaluation of Deep Neural Network Applied in Vision-Based Systems

Vision-based object detection plays a crucial role for the complete functionality of many engineering systems. Typically, detectors or classifiers are used to detect objects or to distinguish different targets. This contribution presents a new evaluation of CNN classifiers in image detection using a modified Probability of Detection reliability measure. The proposed method allows the evaluation of further image parameters affecting the classification results. The proposed evaluation method is implemented on images and comparisons made on parameters with the best detection capability. A typical certification standard (90/95) denoting a 90% probability of detection at 95% reliability level is adapted and successfully applied. Using the 90/95 standard, comparisons are made between different image parameters. A noise analysis procedure is introduced, permitting the trade-off between the detection rate, false alarms, and process parameters. The advantage of the novel approach is experimentally evaluated for vision-based classification results of CNN considering different image parameters. With this new POD evaluation, classifiers will become a trustworthy part of vision systems.

CARL-D: A vision benchmark suite and large scale dataset for vehicle detection and scene segmentation

Vision-based object detection and scene understanding are becoming key features of environment perception and autonomous driving. In the past couple of years, numerous large-scale datasets for visual object detection and semantic understanding have been released which has enormously benefited the environment perception in self-driving cars. However, these datasets like Kitti and Cityscapes only focus on well organized and urban road traffic scenarios of European countries while ignoring the dense and unpattern traffic conditions of sub-continent countries like Pakistan, India, Bangladesh, and Sri Lanka. Consequently, the environment perception system developed on these datasets cannot efficiently assist self-driving cars in traffic scenarios of sub-continent countries. To this end, we present CARL-D, a large-scale dataset and benchmark suite for develop 2D object detection and instance/pixel-level segmentation methods for self-driving cars. CARL-D comprises large-scale stereo vision-based driving videos captured from more than 100 cities of Pakistan, including motorways, dense and unpattern traffic scenarios of urban, rural, and hilly areas. As a benchmark selection, 15,000 suitable images are labeled for 2D object detection and recognition. Whereas semantic segmentation benchmark contains 2500 images with pixel-level high-quality fine annotations and 5000 coarse-annotated images which could help in enabling the deep neural networks to leverage the weakly labeled data. Alongside the dataset, we also present transfer learning-based 2D vehicle detection and scene segmentation methods to evaluate the performance of existing state-of-the-art deep neural networks on our dataset. Lastly, an extensive experimental evaluation along with the comparative study has been carried out which demonstrates the upper edge of our dataset in terms of interclass-diversity, scene-variability, and annotations richness. The proposed benchmark suite is available at https://carl-dataset.github.io/index/.

Research on Object Detection of PCB Assembly Scene Based on Effective Receptive Field Anchor Allocation

Vision-based object detection of PCB (printed circuit board) assembly scenes is essential in accelerating the intelligent production of electronic products. In particular, it is necessary to improve the detection accuracy as much as possible to ensure the quality of assembly products. However, the lack of object detection datasets in PCB assembly scenes is the key to restricting intellectual PCB assembly research development. As an excellent representative of the one-stage object detection model, YOLOv3 (you only look once version 3) mainly relies on placing predefined anchors on the three feature pyramid layers and realizes recognition and positioning using regression. However, the number of anchors distributed in each grid cell of different scale feature layers is usually the same. The ERF (effective receptive field) corresponding to the grid cell at different locations varies. The contradiction between the uniform distribution of fixed-size anchors and the ERF size range in different feature layers will reduce the effectiveness of object detection. Few people use ERF as a standard for assigning anchors to improve detection accuracy. To address this issue, firstly, we constructed a PCB assembly scene object detection dataset, which includes 21 classes of detection objects in three scenes before assembly, during assembly, and after assembly. Secondly, we performed a refined ERF analysis on each grid of the three output layers of YOLOv3, determined the ERF range of each layer, and proposed an anchor allocation rule based on the ERF. Finally, for the small and difficult-to-detect TH (through-holes), we increased the context information and designed improved-ASPP (Atrous spatial pyramid pooling) and channel attention joint module. Through a series of experiments on the object detection dataset of the PCB assembly scene, we found that under the framework of YOLOv3, anchor allocation based on ERF can increase mAP (mean average precision) from 79.32% to 89.86%. At the same time, our proposed method is superior to Faster R-CNN (region convolution neural network), SSD (single shot multibox detector), and YOLOv4 (you only look once version 4) in the balance of high detection accuracy and low computational complexity.