Deep Learning-based Computer Vision Research Articles

Application of deep-learning based computer vision in medical image analysis

The analyzing process of medical image has always been a crucial part for disease detection, diagnosis, surgery assisting and drug delivery. With the outbreak of population worldwide and the constant emergence of all types of complicated diseases, doctors nowadays are facing tremendous workloads and having difficulty in making precise diagnoses or performing more assured operations. Many experts have been striving to develop a wide range of techniques based on deep learning to ease doctors burden and improve patients recovery process or survival chances. This general review paper will mainly focus on providing readers with a brief understanding and knowledge about deep-learning based computer vision applied in modern medical image analysis. Several academic journals or books related to computer vision applications in medical image analysis were selected for review. Despite the advantages and convenience of this technique, this review finds out that potential obstacles still exist and can be overcome or amended in the future.

Deep transfer learning-based computer vision for real-time harvest period classification and impurity detection of Porphyra haitnensis

Deep transfer learning-based computer vision for real-time harvest period classification and impurity detection of Porphyra haitnensis

Deep learning based computer vision under the prism of 3D point clouds: a systematic review

Deep learning based computer vision under the prism of 3D point clouds: a systematic review

Deep Learning-Based Computer Vision Methods for Complex Traffic Environments Perception: A Review

Deep Learning-Based Computer Vision Methods for Complex Traffic Environments Perception: A Review

Plant Disease Detection using Convolutional Neural Network

The current agriculture system of the world is facing a massive challenge to agricultural productivity. Agricultural productivity must be increased to maintain enough food. Due to the various plant diseases, the expected agricultural productivity is impossible to achieve. Using Deep Learning (DL) in the agriculture field, farmers can monitor continuously rather than manually checking plants’ health. ResNet, DenseNet, GoogleNet, Inception, and YOLO are deep learning-based computer vision models. The most significant advantage of Deep Learning is that we do not need to extract features from the image manually. You feed the raw pixels in the images directly as inputs to the network. The network learns to extract features while training the model. The basic objective of this work is to propose multiple convolutional neural network-based models and identify the best model suitable for classifying the individual plant image into healthy and disease (multiple diseases) classes. This will help farmers and agriculturists to have a 360° insight into the cultivation status of the plants. Furthermore, to assess the proposed models; different quality metrics have been computed (i.e., accuracy and loss on the training, validation, and test datasets) to perform the comparative analysis between our proposed and existing techniques. These proposed models achieved a validation accuracy of 97% using MobileNet V1, 96% using MobileNet V1, and 83% using Vanilla CNN. While on 33 challenging images, a test accuracy of 82% using MobileNet V1, 76% using MobileNet V1, and 53% using Vanilla CNN has been achieved.

Improving detection of asphalt distresses with deep learning-based diffusion model for intelligent road maintenance

Research on road infrastructure structural health monitoring is critical due to the increasing problem of deteriorated conditions. The traditional approach to pavement distress detection relies on human-based visual recognition, a time-consuming and labor-intensive method. While Deep Learning-based computer vision systems are the most promising approach, they face the challenges of reduced performance due to the scarcity of labeled data due, high annotation costs misaligned with engineering applications, and limited instances of minority defects. This paper introduces a novel generative diffusion model for data augmentation, creating synthetic images of rare defects. It also investigates methods to enhance image quality and reduce production time. Compared to Generative Adversarial Networks, the optimal configuration excels in reliability, quality and diversity. After incorporating synthetic images into the training of our pavement distress detector, YOLOv5, its mean average precision has been enhanced. This computer-aided system enhances recognition and labelling efficiency, promoting intelligent maintenance and repairs.

Aerial visual data-driven approach for berthing capacity estimation in restricted waters

Coastal ship berth planning and management necessitate accurate identification of ship position and size, yet this is complicated by the presence of diverse ship types and mixed distribution patterns. Traditional survey-based and position-tracking ship monitoring methods have limitations in reflecting the precise spatial information of coastal ships and their surroundings. This paper introduces a novel aerial visual data-driven methodology to estimate the berthing capacity of sheltered space for stationary ships in geographically dispersed water areas. To measure the berthing capacity, image data from an unmanned aerial vehicle (UAV) survey is collected to reflect the spatial information of all types of ships. A deep learning-based computer vision algorithm is used to automatically detect, identify, and classify ships in sample images. Additionally, we introduce the occupancy factor to empirically quantify the spatial proportional correlation between the ship size and the requisite berthing capacity. A validation test of the proposed method is conducted across all sheltered spaces in Hong Kong, and the UAV survey yields over 7,000 informative images of all types of local ships. The ship identification algorithm achieves an average identification rate of 98.6%, demonstrating higher accuracy and reduced labor and time consumption than observational survey and manual counting methods. Findings indicate that the suggested sheltered space for individual ships is 7.75 times the ship size. The approach provides greater flexibility in estimating berthing capacity for mixed berthing forms in extensive or geographically dispersed water bodies, thereby having the potential to support the design of berths, anchorage areas, and typhoon shelter facilities in coastal regions.

Factors affecting the labelling accuracy of brain MRI studies relevant for deep learning abnormality detection.

Unlocking the vast potential of deep learning-based computer vision classification systems necessitates large data sets for model training. Natural Language Processing (NLP)-involving automation of dataset labelling-represents a potential avenue to achieve this. However, many aspects of NLP for dataset labelling remain unvalidated. Expert radiologists manually labelled over 5,000 MRI head reports in order to develop a deep learning-based neuroradiology NLP report classifier. Our results demonstrate that binary labels (normal vs. abnormal) showed high rates of accuracy, even when only two MRI sequences (T2-weighted and those based on diffusion weighted imaging) were employed as opposed to all sequences in an examination. Meanwhile, the accuracy of more specific labelling for multiple disease categories was variable and dependent on the category. Finally, resultant model performance was shown to be dependent on the expertise of the original labeller, with worse performance seen with non-expert vs. expert labellers.

ReplicAnt: a pipeline for generating annotated images of animals in complex environments using Unreal Engine

Deep learning-based computer vision methods are transforming animal behavioural research. Transfer learning has enabled work in non-model species, but still requires hand-annotation of example footage, and is only performant in well-defined conditions. To help overcome these limitations, we developed replicAnt, a configurable pipeline implemented in Unreal Engine 5 and Python, designed to generate large and variable training datasets on consumer-grade hardware. replicAnt places 3D animal models into complex, procedurally generated environments, from which automatically annotated images can be exported. We demonstrate that synthetic data generated with replicAnt can significantly reduce the hand-annotation required to achieve benchmark performance in common applications such as animal detection, tracking, pose-estimation, and semantic segmentation. We also show that it increases the subject-specificity and domain-invariance of the trained networks, thereby conferring robustness. In some applications, replicAnt may even remove the need for hand-annotation altogether. It thus represents a significant step towards porting deep learning-based computer vision tools to the field.

Towards a novel cyber physical control system framework: a deep learning driven use case

PurposeThe health crisis has highlighted the shortcomings of the industry sector which has revealed its vulnerability. To date, there is no guarantee of a return to the “world before”. The ability of companies to cope with these changes is a key competitive advantage requiring the adoption/mastery of industry 4.0 technologies. Therefore, companies must adapt their business processes to fit into similar situations.Design/methodology/approachThe proposed methodology comprises three steps. First, a comparative analysis of the existing CPSs is elaborated. Second, following this analysis, a deep learning driven CPS framework is proposed highlighting its components and tiers. Third, a real industrial case is presented to demonstrate the application of the envisioned framework. Deep learning network-based methods of object detection are used to train the model and evaluation is assessed accordingly.FindingsThe analysis revealed that most of the existing CPS frameworks address manufacturing related subjects. This illustrates the need for a resilient industrial CPS targeting other areas and considering CPSs as loopback systems preserving human–machine interaction, endowed with data tiering approach for easy and fast data access and embedded with deep learning-based computer vision processing methods.Originality/valueThis study provides insights about what needs to be addressed in terms of challenges faced due to unforeseen situations or adapting to new ones. In this paper, the CPS framework was used as a monitoring system in compliance with the precautionary measures (social distancing) and for self-protection with wearing the necessary equipments. Nevertheless, the proposed framework can be used and adapted to any industrial or non-industrial environments by adjusting object detection purpose.

Construction Instance Segmentation (CIS) Dataset for Deep Learning-Based Computer Vision

Deep learning-based computer vision (DLBCV) techniques have played an important role in intelligent construction. Image datasets are essential for developing DLBCV algorithms. However, a large-scale construction-specific dataset of major construction elements, such as precast components (PCs), PC delivery trucks, and workers not wearing safety helmets, remains absent. This paper presents the Construction Instance Segmentation (CIS) dataset, a new image dataset aimed at advancing state-of-the-art instance segmentation in the field of construction management. It contains 50,000 images with ten object categories belonging to construction workers, machines, and materials. Two rounds of algorithmic analysis have been conducted to refine and balance the dataset. Finally, a detailed statistical analysis of the dataset is presented.

PACMAN: A Framework for Pulse Oximeter Digit Detection and Reading in a Low-Resource Setting

In light of the COVID-19 pandemic, patients were required to manually input their daily oxygen saturation (SpO2) and pulse rate (PR) values into a health monitoring system—unfortunately, such a process trend to be an error in typing. Several studies attempted to detect the physiological value from the captured image using optical character recognition (OCR). However, the technology has limited availability with high cost. Thus, this study aimed to propose a novel framework called PACMAN (Pandemic Accelerated Human-Machine Collaboration) with a low-resource deep learning-based computer vision. We compared state-of-the-art object detection algorithms (scaled YOLOv4, YOLOv5, and YOLOR), including the commercial OCR tools for digit recognition on the captured images from the pulse oximeter display. All images were derived from crowdsourced data collection with varying quality and alignment. YOLOv5 was the best-performing model against the given model comparison across all datasets, notably the correctly orientated image dataset. We further improved the model performance with the digits auto-orientation algorithm and applied a clustering algorithm to extract SpO2 and PR values. The accuracy performance of YOLOv5 with the implementations was approximately 81.0-89.5%, which was enhanced compared to without any additional implementation. Accordingly, this study highlighted the completion of the PACMAN framework to detect and read digits in real-world datasets. The proposed framework has been currently integrated into the patient monitoring system utilized by hospitals nationwide.

The Potential of Visual ChatGPT for Remote Sensing

Recent advancements in Natural Language Processing (NLP), particularly in Large Language Models (LLMs), associated with deep learning-based computer vision techniques, have shown substantial potential for automating a variety of tasks. These are known as Visual LLMs and one notable model is Visual ChatGPT, which combines ChatGPT’s LLM capabilities with visual computation to enable effective image analysis. These models’ abilities to process images based on textual inputs can revolutionize diverse fields, and while their application in the remote sensing domain remains unexplored, it is important to acknowledge that novel implementations are to be expected. Thus, this is the first paper to examine the potential of Visual ChatGPT, a cutting-edge LLM founded on the GPT architecture, to tackle the aspects of image processing related to the remote sensing domain. Among its current capabilities, Visual ChatGPT can generate textual descriptions of images, perform canny edge and straight line detection, and conduct image segmentation. These offer valuable insights into image content and facilitate the interpretation and extraction of information. By exploring the applicability of these techniques within publicly available datasets of satellite images, we demonstrate the current model’s limitations in dealing with remote sensing images, highlighting its challenges and future prospects. Although still in early development, we believe that the combination of LLMs and visual models holds a significant potential to transform remote sensing image processing, creating accessible and practical application opportunities in the field.

Toward Automated Field Ballast Condition Evaluation: Development of a Ballast Scanning Vehicle

Ballast plays an essential role in the response of a railroad track to repeated loading. Ballast degradation may lead to poor drainage, lateral instability, and excessive settlement. Extreme levels of ballast degradation may cause service interruptions and safety concerns. Therefore, ballast condition evaluation is of great importance in ensuring safe and efficient operations. Current state-of-the-practice evaluation methods are heavily dependent on visual inspection, field sampling, and laboratory testing, which are subjective and labor-intensive. Meanwhile, existing inspection systems for railroads have not been customized for conducting in-depth evaluation of the ballast layer, including determination of the level of fouling and aggregate size and shape characteristics. For this reason, there is an urgent need for the development of a novel, vison-based ballast scanning platform. This paper introduces the ballast scanning vehicle (BSV), an automated platform that acquires high-quality images, videos, and 3-D height maps of ballast from plan and profile views of cut sections and trenches. The BSV is capable of performing data analysis and generating accurate and comprehensive evaluations of ballast conditions through geotechnical analyses. The essential design components, prototyping, and development stages are described. Further, preliminary data collected from testing on two in-service railroad tracks behind a shoulder ballast cleaner are presented to validate the functions of the BSV. The fully developed BSV serves as a data collection device for ballast evaluation and provides continuous and high-quality images for a deep learning-based computer vision algorithm for field ballast condition evaluation and geotechnical analyses.

Optimized deep learning vision system for human action recognition from drone images.

There are several benefits to constructing a lightweight vision system that is implemented directly on limited hardware devices. Most deep learning-based computer vision systems, such as YOLO (You Only Look Once), use computationally expensive backbone feature extractor networks, such as ResNet and Inception network. To address the issue of network complexity, researchers created SqueezeNet, an alternative compressed and diminutive network. However, SqueezeNet was trained to recognize 1000 unique objects as a broad classification system. This work integrates a two-layer particle swarm optimizer (TLPSO) into YOLO to reduce the contribution of SqueezeNet convolutional filters that have contributed less to human action recognition. In short, this work introduces a lightweight vision system with an optimized SqueezeNet backbone feature extraction network. Secondly, it does so without sacrificing accuracy. This is because that the high-dimensional SqueezeNet convolutional filter selection is supported by the efficient TLPSO algorithm. The proposed vision system has been used to the recognition of human behaviors from drone-mounted camera images. This study focused on two separate motions, namely walking and running. As a consequence, a total of 300 pictures were taken at various places, angles, and weather conditions, with 100 shots capturing running and 200 images capturing walking. The TLPSO technique lowered SqueezeNet's convolutional filters by 52%, resulting in a sevenfold boost in detection speed. With an F1 score of 94.65% and an inference time of 0.061 milliseconds, the suggested system beat earlier vision systems in terms of human recognition from drone-based photographs. In addition, the performance assessment of TLPSO in comparison to other related optimizers found that TLPSO had a better convergence curve and achieved a higher fitness value. In statistical comparisons, TLPSO surpassed PSO and RLMPSO by a wide margin.

Offloading Deep Learning Powered Vision Tasks From UAV to 5G Edge Server With Denoising

Offloading computationally heavy tasks from an unmanned aerial vehicle (UAV) to a remote server helps improve the battery life and can help reduce resource requirements. Deep learning based state-of-the-art computer vision tasks, such as object segmentation and object detection, are computationally heavy algorithms, requiring large memory and computing power. Many UAVs are using (pretrained) off-the-shelf versions of such algorithms. Offloading such power-hungry algorithms to a remote server could help UAVs save power significantly. However, deep learning based algorithms are susceptible to noise, and a wireless communication system, by its nature, introduces noise to the original signal. When the signal represents an image, noise affects the image. There has not been much work studying the effect of the noise introduced by the communication system on pretrained deep networks. In this work, we first analyze how reliable it is to offload deep learning based computer vision tasks (including both object segmentation and detection) by focusing on the effect of various parameters of a 5G wireless communication system on the transmitted image and demonstrate how the introduced noise of the used 5G wireless communication system reduces the performance of the offloaded deep learning task. Then solutions are introduced to eliminate (or reduce) the negative effect of the noise. The proposed framework starts with introducing many classical techniques as alternative solutions first, and then introduces a novel deep learning based solution to denoise the given noisy input image. The performance of various denoising algorithms on offloading both object segmentation and object detection tasks are compared. Our proposed deep transformer-based denoiser algorithm (NR-Net) yields the state-of-the-art results on reducing the negative effect of the noise in our experiments.

A novel occupant-centric stratum ventilation system using computer vision: Occupant detection, thermal comfort, air quality, and energy savings

Traditional ventilation and air conditioning systems typically operate on a predetermined schedule with fixed operating parameters. Occupant-centric control (OCC) strategies have been proposed to reduce system operation energy consumption without sacrificing thermal comfort. Indoor occupancy detection in real time is a critical step in successfully implementing the OCC strategy. Thus, the deep learning-based computer vision method was adopted in the first step of the study, and the detection performance and camera position were analyzed in an office scenario. Next, the proposed OCC strategy was used to regulate the supply air parameters and outdoor air volume in stratum ventilation based on the monitored occupant number. The traditional static control strategy was then compared to two control strategies: constant air volume and variable air volume. Occupant detection performance results showed the mean NRMSD for the five most common relative positions of the occupants and camera was 0.1109, with sitting back to camera having the lowest accuracy. Subjective response results demonstrated that, when compared to the traditional control strategy, thermal comfort was improved by 43%–73%, perceived air quality was maintained at an acceptable level, CO2 concentration was less than 700 ppm, and energy could be saved by 2.3%–8.1%. Furthermore, the lower the occupancy, the greater the improvement in comfort and the greater the energy savings. This research focused on how the stratum ventilation system responds to dynamic changes in occupancy and provided insights into reducing unnecessary energy waste while maintaining comfort.

Study of convolutional neural network-based semantic segmentation methods on edge intelligence devices for field agricultural robot navigation line extraction

Smart agricultural machinery is emerging as the future trend for field robots, and the fully automatic robot has a great application prospect. However, it is a big challenge for robots to navigate in complex farmland environments. In this research, 5 deep learning-based computer vision methods under different field scenes for field navigation line extraction were studied and successfully deployed on an embedded system, which can be integrated into robots for automatic navigation in the future. The field road was segmented by the semantic segmentation algorithm at first, and then the navigation line is extracted from the segmented image by a polygon fitting method. Finally, all the models are transformed through the TensorRT library and deployed on the edge computing device Jetson Nano. In the experiment, five reprehensive semantic segmentation networks namely UNet, Deeplabv3+, BiseNetv1, BiseNetv2, and ENet networks were selected. Among the five networks, Deeplabv3+ is the most accurate. In five scenes, its average segmentation accuracy is 84.87 %, and the navigation line error is 9.59 pixels. Especially in the third scene with shadow and occlusion, it performs best, with only 8.34 pixel error, But the speed of Deeplabv3+ is only 9.7 FPS. ENet, BiseNetv1, and BiseNetv2 are lightweight networks. The speed of ENet is 16.8 FPS, BiseNetv2 is 17 FPS, and BiseNetv1 is 15.8 FPS. In segmentation accuracy and navigation line error, ENet performs better than BiseNet series networks, which are 84.94 % and 10.73 pixels, respectively. In the third scene with shadow and occlusion, it also performs slightly better than BiseNet series networks. In summary, deep learning-based semantic segmentation methods have strong robustness and stability in complex environment compared with previous research. Among all currently available neural networks, ENet has the best performance and good application potential in field navigation.

Two-Speed Deep-Learning Ensemble for Classification of Incremental Land-Cover Satellite Image Patches

High-velocity data streams present a challenge to deep learning-based computer vision models due to the resources needed to retrain for new incremental data. This study presents a novel staggered training approach using an ensemble model comprising the following: (i) a resource-intensive high-accuracy vision transformer; and (ii) a fast training, but less accurate, low parameter-count convolutional neural network. The vision transformer provides a scalable and accurate base model. A convolutional neural network (CNN) quickly incorporates new data into the ensemble model. Incremental data are simulated by dividing the very large So2Sat LCZ42 satellite image dataset into four intervals. The CNN is trained every interval and the vision transformer trained every half interval. We call this combination of a complementary ensemble with staggered training a “two-speed” network. The novelty of this approach is in the use of a staggered training schedule that allows the ensemble model to efficiently incorporate new data by retraining the high-speed CNN in advance of the resource-intensive vision transformer, thereby allowing for stable continuous improvement of the ensemble. Additionally, the ensemble models for each data increment out-perform each of the component models, with best accuracy of 65% against a holdout test partition of the RGB version of the So2Sat dataset.

Artificial intelligence for sustainability: Facilitating sustainable smart product-service systems with computer vision

Recent advances in artificial intelligence in general, and deep learning in particular, enable innovations that have a massive impact on society and industries. Autonomous driving, facial recognition, drug discovery, and speech recognition are examples of fundamental innovations facilitated by deep learning. However, the usage and impact of deep learning for cleaner production and sustainability purposes remain little explored. This work shows how deep learning can be harnessed to increase sustainability in production and product usage. Specifically, we utilize deep learning-based computer vision to determine the wear states of products. The resulting insights serve as a basis for novel product-service systems with improved integration and result orientation. Moreover, these insights are expected to facilitate product usage improvements and R&D innovations. We demonstrate our approach on two products: machining tools and rotating X-ray anodes. From a technical standpoint, we show that it is possible to recognize the wear state of these products using deep-learning-based computer vision. In particular, we detect wear through microscopic images of the two products. We utilize a U-Net for semantic segmentation to detect wear based on pixel granularity. The resulting mean dice coefficients of 0.631 and 0.603 demonstrate the feasibility of the proposed approach. Consequently, experts can now make better decisions, for example, improve machining process parameters. To assess the impact of the proposed approach on environmental sustainability, we perform life cycle assessments that show gains for both products. The results indicate that the emissions of CO2 equivalents are reduced by 12% for machining tools and by 44% for rotating anodes. This work can serve as a guideline and inspire researchers and practitioners to utilize computer vision in similar scenarios to develop sustainable smart product-service systems and enable cleaner production.