Learning-based Object Detection Research Articles

Detection of illegal airstrips in digital images using artificial intelligence techniques

Ecuador is considered a transit country for drug trafficking and smuggling activities. Being a current problem in the northern and coastal border of the national territory, unauthorized airstrips are used for these illegal activities where controlled substances, money, weapons, ammunition, and explosives are transported. The research will be based on the design and development of a methodology using artificial intelligence techniques for the analysis and processing of images that will be obtained from reconnaissance missions carried out by Ecuadorian Air Force airplanes. For the recognition of clandestine airstrips, the YOLO deep learning-based object detection method and segmentation techniques will be used.

Crosswalk Detection from Satellite Imagery for Pedestrian Network Completion

Pedestrian network creation and completion for various tasks in transportation planning engineering generally requires laborious manual labeling of sidewalks and crosswalks. In this study, we employed the deep learning-based object detection model YOLO v5 for detecting crosswalks using Google Maps satellite imagery data of four North American cities. We also tested for two additional scenarios involving two image-processing techniques to exploit the special properties of crosswalks. Observations showed that the original images performed better than with binary thresholding and nearly the same even when noisy regions outside the right-of-way were removed. We proposed an algorithm to assign the detected crosswalks to intersection legs. We demonstrated the effectiveness of this technique for Washington, D.C. and Los Angeles, CA, showing classification accuracy rates of 71% and 89%, respectively. We also showed the influence of increasing distance threshold, a tolerance radius of prediction accuracy, in degrading the classification performance. This method, when combined with existing methods for sidewalk detection from aerial and street view images, could be reasonably used to help complete urban pedestrian networks in cities where high-resolution satellite images are available.

Impact of ISP Tuning on Object Detection.

In advanced driver assistance systems (ADAS) or autonomous vehicle research, acquiring semantic information about the surrounding environment generally relies heavily on camera-based object detection. Image signal processors (ISPs) in cameras are generally tuned for human perception. In most cases, ISP parameters are selected subjectively and the resulting image differs depending on the individual who tuned it. While the installation of cameras on cars started as a means of providing a view of the vehicle's environment to the driver, cameras are increasingly becoming part of safety-critical object detection systems for ADAS. Deep learning-based object detection has become prominent, but the effect of varying the ISP parameters has an unknown performance impact. In this study, we analyze the performance of 14 popular object detection models in the context of changes in the ISP parameters. We consider eight ISP blocks: demosaicing, gamma, denoising, edge enhancement, local tone mapping, saturation, contrast, and hue angle. We investigate two raw datasets, PASCALRAW and a custom raw dataset collected from an advanced driver assistance system (ADAS) perspective. We found that varying from a default ISP degrades the object detection performance and that the models differ in sensitivity to varying ISP parameters. Finally, we propose a novel methodology that increases object detection model robustness via ISP variation data augmentation.

Deep-Learning-Based Anti-Collision System for Construction Equipment Operators

Due to the dynamic environment of construction sites, worker collisions and stray accidents caused by heavy equipment are constantly occurring. In this study, a deep learning-based anti-collision system was developed to improve the existing proximity warning systems and to monitor the surroundings in real time. The technology proposed in this paper consists of an AI monitor, an image collection camera, and an alarm device. The AI monitor has a built-in object detection algorithm, automatically detects the operator from the image input from the camera, and notifies the operator of a danger warning. The deep learning-based object detection algorithm was trained with an image data set composed of a total of 42,620 newly constructed in this study. The proposed technology was installed on an excavator, which is the main equipment operated at the construction site, and performance tests were performed, and it showed the potential to effectively prevent collision accidents.

Microorganism Detection in Activated Sludge Microscopic Images Using Improved YOLO

Wastewater has detrimental effects on the natural environment. The activated sludge method, a widely adopted approach for wastewater treatment, has proven highly effective. Within this process, microorganisms play a pivotal role, necessitating continuous monitoring of their quantity and diversity. Conventional methods, such as microscopic observation, are time-consuming. With the widespread integration of computer vision technologies into object detection, deep learning-based object detection algorithms, notably the You Only Look Once (YOLO) model, have garnered substantial interest for their speed and precision in detection tasks. In this research, we applied the YOLO model to detect microorganisms in microscopic images of activated sludge. Furthermore, addressing the irregular shapes of microorganisms, we developed an improved YOLO model by incorporating deformable convolutional networks and an attention mechanism to enhance its detection capabilities. We conducted training and testing using a custom dataset comprising five distinct objects. The performance evaluations used in this study utilized metrics such as the mean average precision at intersections over a union threshold of 0.5 (mAP@0.5), with the improved YOLO model achieving a mAP@0.5 value of 93.7%, signifying a 4.3% improvement over the YOLOv5 model. Comparative analysis of the improved YOLO model and other object detection algorithms on the same dataset revealed a higher accuracy for the improved YOLO model. These results demonstrate the superior performance of the improved YOLO model in the task of detecting microorganisms in activated sludge, providing an effective auxiliary method for wastewater treatment monitoring.

Deep learning-based object detection in maritime unmanned aerial vehicle imagery: Review and experimental comparisons

With the advancement of maritime unmanned aerial vehicles (UAVs) and deep learning technologies, the application of UAV-based object detection has become increasingly significant in the fields of maritime industry and ocean engineering. Endowed with intelligent sensing capabilities, the maritime UAVs enable effective and efficient maritime surveillance. To further promote the development of maritime UAV-based object detection, this paper provides a comprehensive review of challenges, relative methods, and UAV aerial datasets. Specifically, in this work, we first briefly summarize four challenges for object detection on maritime UAVs, i.e., object feature diversity, device limitation, maritime environment variability, and dataset scarcity. We then focus on computational methods to improve maritime UAV-based object detection performance in terms of scale-aware, small object detection, view-aware, rotated object detection, lightweight methods, and others. Next, we review the UAV aerial image/video datasets and propose a maritime UAV aerial dataset named MS2ship for ship detection. Furthermore, we conduct a series of experiments to present the performance evaluation and robustness analysis of object detection methods on maritime datasets. Eventually, we give the discussion and outlook on future works for maritime UAV-based object detection. The MS2ship dataset is available at https://github.com/zcj234/MS2ship.

Feasibility of Detecting Sweet Potato (Ipomoea batatas) Virus Disease from High-Resolution Imagery in the Field Using a Deep Learning Framework

The sweet potato is an essential food and economic crop that is often threatened by the devastating sweet potato virus disease (SPVD), especially in developing countries. Traditional laboratory-based direct detection methods and field scouting are commonly used to rapidly detect SPVD. However, these molecular-based methods are costly and disruptive, while field scouting is subjective, labor-intensive, and time-consuming. In this study, we propose a deep learning-based object detection framework to assess the feasibility of detecting SPVD from ground and aerial high-resolution images. We proposed a novel object detector called SPVDet, as well as a lightweight version called SPVDet-Nano, using a single-level feature. These detectors were prototyped based on a small-scale publicly available benchmark dataset (PASCAL VOC 2012) and compared to mainstream feature pyramid object detectors using a leading large-scale publicly available benchmark dataset (MS COCO 2017). The learned model weights from this dataset were then transferred to fine-tune the detectors and directly analyze our self-made SPVD dataset encompassing one category and 1074 objects, incorporating the slicing aided hyper inference (SAHI) technology. The results showed that SPVDet outperformed both its single-level counterparts and several mainstream feature pyramid detectors. Furthermore, the introduction of SAHI techniques significantly improved the detection accuracy of SPVDet by 14% in terms of mean average precision (mAP) in both ground and aerial images, and yielded the best detection accuracy of 78.1% from close-up perspectives. These findings demonstrate the feasibility of detecting SPVD from ground and unmanned aerial vehicle (UAV) high-resolution images using the deep learning-based SPVDet object detector proposed here. They also have great implications for broader applications in high-throughput phenotyping of sweet potatoes under biotic stresses, which could accelerate the screening process for genetic resistance against SPVD in plant breeding and provide timely decision support for production management.

Detection of Small Objects in Side-Scan Sonar Images Using an Enhanced YOLOv7-Based Approach

Deep learning-based object detection methods have demonstrated remarkable effectiveness across various domains. Recently, there has been growing interest in applying these techniques to underwater environments. Conventional optical imaging methods face severe limitations when operating in underwater conditions, restricting their ability to identify objects with good visibility and at close distances. Consequently, side-scan sonar (SSS) has emerged as a common equipment choice for underwater detection due to its compatibility with the characteristics of sound waves in water. This paper introduces a novel method, termed the Enhanced YOLOv7-Based Approach, for detecting small objects in SSS images. Building upon the widely-adopted YOLOv7 method, the proposed approach incorporates several enhancements aimed at improving detection accuracy. First, a dedicated detection layer tailored for small objects is added to the original network architecture. Additionally, two attention mechanisms are integrated within the backbone and neck components of the network, respectively, to strengthen the network’s focus on object features. Finally, the network features are recombined based on the BiFPN structure. Experimental results demonstrate that the proposed method outperforms mainstream object detection algorithms. In comparison to the original YOLOv7 network, it achieves a Precision of 95.5%, indicating a significant improvement of 4.8%. Moreover, its Recall reaches 87.0%, representing an enhancement of 5.1%, while the mean Average Precision (mAP) at an IoU threshold of 0.5 (mAP@.5) reaches 86.9%, reflecting a 6.7% improvement. Furthermore, the mAP@.5:.95 reaches 55.1%, a 4.8% enhancement. Therefore, the method presented in this paper enhances the performance of YOLOv7 for object detection in SSS images, providing a fresh perspective on small object detection based on SSS images and contributing to the advancement of underwater detection techniques.

Scalable Surveillance of E-Cigarette Products on Instagram and TikTok Using Computer Vision.

Instagram and TikTok, video-based social media platforms popular among adolescents, contain tobacco-related content despite the platforms' policies prohibiting substance-related posts. Prior research identified themes in e-cigarette-related social media posts using qualitative or text-based machine learning methods. We developed an image-based computer vision model to identify e-cigarette products in social media images and videos. We created a data set of 6999 Instagram images labeled for 8 object classes: mod or pod devices, e-juice containers, packaging boxes, nicotine warning labels, e-juice flavors, e-cigarette brand names, and smoke clouds. We trained a DyHead object detection model using a Swin-Large backbone, evaluated the model's performance on 20 Instagram and TikTok videos, and applied the model to 14 072 e-cigarette-related promotional TikTok videos (2019-2022; 10 276 485 frames). The model achieved the following mean average precision scores on the image test set: e-juice container: 0.89; pod device: 0.67; mod device: 0.54; packaging box: 0.84; nicotine warning label: 0.86; e-cigarette brand name: 0.71; e-juice flavor name: 0.89; and smoke cloud: 0.46. The prevalence of pod devices in promotional TikTok videos increased by 15% from 2019 to 2022. The prevalence of e-juices increased by 33% from 2021 to 2022. The prevalence of e-juice flavor names and e-cigarette brand names increased by about 100% from 2019 to 2022. Deep learning-based object detection technology enables automated analysis of visual posts on social media. Our computer vision model can detect the presence of e-cigarettes products in images and videos, providing valuable surveillance data for tobacco regulatory science (TRS). Prior research identified themes in e-cigarette-related social media posts using qualitative or text-based machine learning methods. We developed an image-based computer vision model to identify e-cigarette products in social media images and videos. We trained a DyHead object detection model using a Swin-Large backbone, evaluated the model's performance on 20 Instagram and TikTok videos featuring at least two e-cigarette objects, and applied the model to 14 072 e-cigarette-related promotional TikTok videos (2019-2022; 10 276 485 frames). The deep learning model can be used for automated, scalable surveillance of image- and video-based e-cigarette-related promotional content on social media, providing valuable data for TRS. Social media platforms could use computer vision to identify tobacco-related imagery and remove it promptly, which could reduce adolescents' exposure to tobacco content online.

Simplifying YOLOv5 for deployment in a real crop monitoring setting

Deep learning-based object detection models have become a preferred choice for crop detection tasks in crop monitoring activities due to their high accuracy and generalization capabilities. However, their high computational demand and large memory footprint pose a challenge for use on mobile embedded devices deployed in crop monitoring settings. Various approaches have been taken to minimize the computational cost and reduce the size of object detection models such as channel and layer pruning, detection head searching, backbone optimization, etc. In this work, we approached computational lightening, model compression, and speed improvement by discarding one or more of the three detection scales of the YOLOv5 object detection model. Thus, we derived up to five separate fast and light models, each with only one or two detection scales. To evaluate the new models for a real crop monitoring use case, the models were deployed on NVIDIA Jetson nano and NVIDIA Jetson Orin devices. The new models achieved up to 21.4% reduction in giga floating-point operations per second (GFLOPS), 31.9% reduction in number of parameters, 30.8% reduction in model size, 28.1% increase in inference speed, with only a small average accuracy drop of 3.6%. These new models are suitable for crop detection tasks since the crops are usually of similar sizes due to the high likelihood of being in the same growth stage, thus, making it sufficient to detect the crops with just one or two detection scales.

A Novel Object Detection Method for Solid Waste Incorporating a Weighted Deformable Convolution

Rapid detection of solid waste with remote sensing images is of great significance for environmental protection. In recent years, deep learning-based object detection methods have been widely studied. In contrast to regular objects such as airplanes or buildings, solid wastes commonly h ave arbitrary shapes with difficult‐to‐distinguish boundaries. In this study, a solid waste detection network with a weighted deformable convolution and a global context block based on Feature Pyramid Network (FPN) model was proposed. The designed feature extraction structure can help to enhance the boundary and shape features of solid waste. The effectiveness of the proposed method was verified on the well-known DetectIon in Optical Remote sensing images data set and further on a solid waste data set, which was collected by the authors manually. The experimental results show that the proposed method outperforms other traditional object detection methods and a maximum improvement of 5.27% was obtained compared to the FPN method.

Deep learning-based text detection and recognition on architectural floor plans

An important aspect of automatic floor plan analysis is the extraction of textual information, as it is essential for a thorough understanding of the drawing. This paper presents a text extraction approach utilizing a deep learning-based object detection model and state-of-the-art Optical Character Recognition (OCR) methods. The paper contributes to the research community in three ways: First, it introduces additional annotations to existing data sets to encompass text elements. Second, it proposes a specialized data synthesis pipeline, allowing for generating training images that mimic important characteristics of real data. Finally, it documents a comparative study of deep learning-based object detection architectures (Tesseract, EAST, CRAFT, Faster R-CNN, YOLOv5, YOLOR, YOLOv7, and YOLOv8) and OCR tools (PARSEq, MATRN, EasyOCR, and Tesseract) for the task. Results indicate that YOLOv7 yields the best text detection performance (up to 97.5% wmAP) and PARSEq excels in character recognition (85.2% CER). The data sets are made available.

Shine: A deep learning-based accessible parking management system

The ongoing expansion of urban areas facilitated by advancements in science and technology has resulted in a considerable increase in the number of privately owned vehicles worldwide, including in South Korea. However, this gradual increment in the number of vehicles has inevitably led to parking-related issues, including the abuse of disabled parking spaces (hereafter referred to as accessible parking spaces) designated for individuals with disabilities. Traditional license plate recognition (LPR) systems have proven inefficient in addressing such a problem in real-time due to the high frame rate of surveillance cameras, the presence of natural and artificial noise, and variations in lighting and weather conditions that impede detection and recognition by these systems. With the growing concept of parking 4.0, many sensors, IoT and deep learning-based approaches have been applied to automatic LPR and parking management systems. Nonetheless, the studies show a need for a robust and efficient model for managing accessible parking spaces in South Korea. To address this, we have proposed a novel system called, ‘Shine’, which uses the deep learning-based object detection algorithm for detecting the vehicle, license plate, and disability badges (referred to as cards, badges, or access badges hereafter) and verifies the rights of the driver to use accessible parking spaces by coordinating with the central server. Our model, which achieves a mean average precision of 92.16%, is expected to address the issue of accessible parking space abuse and contributes significantly towards efficient and effective parking management in urban environments.

Tryp: a dataset of microscopy images of unstained thick blood smears for trypanosome detection

Trypanosomiasis, a neglected tropical disease (NTD), challenges communities in sub-Saharan Africa and Latin America. The World Health Organization underscores the need for practical, field-adaptable diagnostics and rapid screening tools to address the negative impact of NTDs. While artificial intelligence has shown promising results in disease screening, the lack of curated datasets impedes progress. In response to this challenge, we developed the Tryp dataset, comprising microscopy images of unstained thick blood smears containing the Trypanosoma brucei brucei parasite. The Tryp dataset provides bounding box annotations for tightly enclosed regions containing the parasite for 3,085 positive images, and 93 images collected from negative blood samples. The Tryp dataset represents the largest of its kind. Furthermore, we provide a benchmark on three leading deep learning-based object detection techniques that demonstrate the feasibility of AI for this task. Overall, the availability of the Tryp dataset is expected to facilitate research advancements in diagnostic screening for this disease, which may lead to improved healthcare outcomes for the communities impacted.

Object Detection and Distance Measurement Algorithm for Collision Avoidance of Precast Concrete Installation during Crane Lifting Process

In the construction industry, the process of carrying heavy loads from one location to another by means of a crane is inevitable. This reliance on cranes to carry heavy loads is more obvious when it comes to high-rise building construction. Depending on the conditions and requirements on-site, various types of construction lifting equipment (i.e., cranes) are being used. As off-site construction (OSC) is gaining more traction recently, cranes are becoming more important throughout the construction project as precast concrete (PC) members are major components of OSC calling for lifting work. As a result of the increased use of cranes on construction sites, concerns about construction safety as well as the effectiveness of existing load collision prevention systems are attracting more attention from various parties involved. Besides the inherent risks associated with heavy load lifting, the unpredictable movement of on-site workers around the crane operation area, along with the presence of blind spots that obstruct the crane operator’s field-of-view (FOV), further increase the accident probability during crane operation. As such, the need for a more reliable and improved collision avoidance system that prevents lifted loads from hitting other structures and workers is paramount. This study introduces the application of deep learning-based object detection and distance measurement sensors integrated in a complementary way to achieve the stated need. Specifically, the object detection technique was used with the application of an Internet Protocol (IP) camera to detect the workers within the crane operation radius, whereas ultrasonic sensors were used to measure the distance of surrounding obstacles. Both applications were designed to work concurrently so as to prevent potential collisions during crane lifting operations. The field testing and evaluation of the integrated system showed promising results.

Deep learning-based object detection for visible dust and prevention measures on construction sites

Activities on construction sites inevitably generate large amounts of dust, which can pose a significant threat to air quality and public health. Detection of construction dust has received increasing attention. In this study, we proposed an object detection method for visible dust and relevant prevention measures on construction sites based on YOLOv7 detector. Deformable Convolutional Networks and Wise-IoU were introduced to improve the precision of the detector in detecting non-rigid objects, such as visible dust. We constructed a publicly available dataset containing 7,500 realistic images covering various construction site conditions. The results indicate that the improved detector achieved high precision with mean average precision of 68.1% and detection speed of 80.6 frames per second. The proposed method enables real-time and effective detection of visible dust and relevant prevention measures on construction sites, which will help advance the research of dust detection and prevention in open space environments.

HAM-Transformer: A Hybrid Adaptive Multi-Scaled Transformer Net for Remote Sensing in Complex Scenes

The quality of remote sensing images has been greatly improved by the rapid improvement of unmanned aerial vehicles (UAVs), which has made it possible to detect small objects in the most complex scenes. Recently, learning-based object detection has been introduced and has gained popularity in remote sensing image processing. To improve the detection accuracy of small, weak objects in complex scenes, this work proposes a novel hybrid backbone composed of a convolutional neural network and an adaptive multi-scaled transformer, referred to as HAM-Transformer Net. HAM-Transformer Net firstly extracts the details of feature maps using convolutional local feature extraction blocks. Secondly, hierarchical information is extracted, using multi-scale location coding. Finally, an adaptive multi-scale transformer block is used to extract further features in different receptive fields and to fuse them adaptively. We implemented comparison experiments on a self-constructed dataset. The experiments proved that the method is a significant improvement over the state-of-the-art object detection algorithms. We also conducted a large number of comparative experiments in this work to demonstrate the effectiveness of this method.

Deep Learning Based Obstacle Awareness from Airborne Optical Sensors

Aviation statistics identify collision with terrain and obstacles as a leading cause of helicopter accidents. Assisting helicopter pilots in detecting the presence of obstacles can greatly mitigate the risk of collisions. However, only a limited number of helicopters in operation have an installed helicopter terrain awareness and warning system (HTAWS), while the cost of active obstacle warning systems remains prohibitive for many civil operators. In this work, we apply machine learning to automate obstacle detection and classification in combination with commercially available airborne optical sensors. While numerous techniques for learning-based object detection have been published in the literature, many of them are data and computation intensive. Our approach seeks to balance the detection and classification accuracy of the method with the size of the training data required and the runtime. Specifically, our approach combines the invariant feature extraction ability of pretrained deep convolutional neural networks (CNNs) and the high-speed training and classification ability of a novel, proprietary frequency-domain support vector machine (SVM) method. We describe our experimental setup comprising the CNN + SVM model and datasets of predefined classes of obstacles—pylons, chimneys, antennas, TV towers, wind turbines, helicopters—synthesized from prerecorded airborne video sequences of low-altitude helicopter flight. We analyze the detection performance using average precision, average recall, and runtime performance metrics on representative test data. Finally, we present a simple architecture for real-time, onboard implementation and discuss the obstacle detection performance of recently concluded flight tests.

Enhanced YOLOv5 network-based object detection (BALFilter Reader) promotes PERFECT filter-enabled liquid biopsy of lung cancer from bronchoalveolar lavage fluid (BALF)

Liquid biopsy of cancers, detecting tumor-related information from liquid samples, has attracted wide attentions as an emerging technology. Our previously reported large-area PERFECT (Precise-Efficient-Robust-Flexible-Easy-Controllable-Thin) filter has demonstrated competitive sensitivity in recovering rare tumor cells from clinical samples. However, it is time-consuming and easily biased to manually inspect rare target cells among numerous background cells distributed in a large area (Φ ≥ 13 mm). This puts forward an urgent demand for rapid and bias-free inspection. Hereby, this paper implemented deep learning-based object detection for the inspection of rare tumor cells from large-field images of PERFECT filters with hematoxylin-eosin (HE)-stained cells recovered from bronchoalveolar lavage fluid (BALF). CenterNet, EfficientDet, and YOLOv5 were trained and validated with 240 and 60 image blocks containing tumor and/or background cells, respectively. YOLOv5 was selected as the basic network given the highest mAP@0.5 of 92.1%, compared to those of CenterNet and EfficientDet at 85.2% and 91.6%, respectively. Then, tricks including CIoU loss, image flip, mosaic, HSV augmentation and TTA were applied to enhance the performance of the YOLOv5 network, improving mAP@0.5 to 96.2%. This enhanced YOLOv5 network-based object detection, named as BALFilter Reader, was tested and cross-validated on 24 clinical cases. The overall diagnosis performance (~2 min) with sensitivity@66.7% ± 16.7%, specificity@100.0% ± 0.0% and accuracy@75.0% ± 12.5% was superior to that from two experienced pathologists (10–30 min) with sensitivity@61.1%, specificity@16.7% and accuracy@50.0%, with the histopathological result as the gold standard. The AUC of the BALFilter Reader is 0.84 ± 0.08. Moreover, a customized Web was developed for a user-friendly interface and the promotion of wide applications. The current results revealed that the developed BALFilter Reader is a rapid, bias-free and easily accessible AI-enabled tool to promote the transplantation of the BALFilter technique. This work can easily expand to other cytopathological diagnoses and improve the application value of micro/nanotechnology-based liquid biopsy in the era of intelligent pathology.

PBA-YOLOv7: An Object Detection Method Based on an Improved YOLOv7 Network

Deep learning-based object detection methods address the problem of how to trade off the object detection accuracy and detection speed of the model. This paper proposes the PBA-YOLOv7 network algorithm, which is based on the YOLOv7 network, and first introduces the PConv, which lightens the ELAN module in the backbone network structure and reduces the number of parameters to improve the detection speed of the network and then designs and introduces the BiFusionNet network, which better aggregates the high-level semantic features and the low-level semantic features; and finally, on this basis, the coordinate attention mechanism is introduced to make the network focus on more critical features without increasing the model complexity. The coordinate attention mechanism is introduced to make the network focus more on important feature information and improve the feature expression ability of the network without increasing the model complexity. Experiments on the publicly available KITTI’s dataset show that the PBA-YOLOv7 network model significantly improves both detection accuracy and detection speed compared to the original YOLOv7 model, with 4% and 7.8% improvement in mAP0.5 and mAP0.5:0.95, respectively, and six frames improvement in FPS. The improved algorithm in this paper weighs the model’s detection accuracy and detection speed in the detection task. It performs well compared to other algorithms, such as YOLOv7 and YOLOv5l.