Deep Learning-based Computer Vision Research Articles

Self-supervised learning unveils urban change from street-level images

Cities around the world are grappling with multiple interconnected challenges, including population growth, shortage of affordable and decent housing, and the need for neighborhood improvements. Despite its critical importance for policy, our ability to effectively monitor and track urban change remains limited. Deep learning-based computer vision methods applied to street-level images have been successful in the measurement of socioeconomic and environmental inequalities but did not fully utilize temporal images to track urban change, as time-varying labels are often unavailable. We used self-supervised methods to measure change in London using 15 million street images taken between 2008 and 2021. Our novel adaptation of Barlow Twins, Street2Vec, embeds urban structure while being invariant to seasonal and daily changes without manual annotations. It outperformed generic pretrained embeddings, successfully identified point-level change in London's housing supply from street-level images, and distinguished between major and minor change. This capability can provide timely information for urban planning and policy decisions towards more liveable, equitable, and sustainable cities.

Improving deep learning based bluespotted ribbontail ray (Taeniura Lymma) recognition

This paper presents the novel task of bluespotted ribbontail (BR) ray (Taeniura lymma) recognition using deep learning based computer vision methods to enable the identification of specific individuals of this species. Mapping the specific individuals in relation to location and time will allow marine researchers to understand their movement patterns, habitat choice, life span, size of the population and more – data which could allow monitoring and establishing a tailor-made conservation plan for this species.Our work is pioneer on this recognition problem. We give a detailed description of the three basic steps of detection, feature extraction and recognition in this vision problem and perform experiments to explore the system configuration and what improves the performance. A feature extraction enhancement as well as a crucial effect of a split into different main poses are demonstrated. Though the precision results achieved in this paper are still moderate and should be further improved, they are nevertheless promising and reasonable for practical use if the six best matches are chosen. For this scenario, almost 85% precision for upper-pose model, and almost 80% precision for left- and right-pose models, are achieved demonstrating the feasibility of the pipeline suggested as well as opportunities for improvement.

Artificial intelligence model for automated surgical instrument detection and counting: an experimental proof-of-concept study

BackgroundRetained surgical items (RSI) are preventable events that pose a significant risk to patient safety. Current strategies for preventing RSIs rely heavily on manual instrument counting methods, which are prone to human error. This study evaluates the feasibility and performance of a deep learning-based computer vision model for automated surgical tool detection and counting.MethodsA novel dataset of 1,004 images containing 13,213 surgical tools across 11 categories was developed. The dataset was split into training, validation, and test sets at a 60:20:20 ratio. An artificial intelligence (AI) model was trained on the dataset, and the model’s performance was evaluated using standard object detection metrics, including precision and recall. To simulate a real-world surgical setting, model performance was also evaluated in a dynamic surgical video of instruments being moved in real-time.ResultsThe model demonstrated high precision (98.5%) and recall (99.9%) in distinguishing surgical tools from the background. It also exhibited excellent performance in differentiating between various surgical tools, with precision ranging from 94.0 to 100% and recall ranging from 97.1 to 100% across 11 tool categories. The model maintained strong performance on a subset of test images containing overlapping tools (precision range: 89.6–100%, and recall range 97.2–98.2%). In a real-time surgical video analysis, the model maintained a correct surgical tool count in all non-transition frames, with a median inference speed of 40.4 frames per second (interquartile range: 4.9).ConclusionThis study demonstrates that using a deep learning-based computer vision model for automated surgical tool detection and counting is feasible. The model’s high precision and real-time inference capabilities highlight its potential to serve as an AI safeguard to potentially improve patient safety and reduce manual burden on surgical staff. Further validation in clinical settings is warranted.

Image-based discrimination of the early stages of mesenchymal stem cell differentiation.

Mesenchymal stem cells (MSCs) are self-renewing, multipotent cells, which can be used in cellular and tissue therapeutics. MSCs cell number can be expanded in vitro, but premature differentiation results in reduced cell number and compromised therapeutic efficacies. Current techniques fail to discriminate the "stem-like" population from early stages (12 hours) of differentiated MSC population. Here, we imaged nuclear structure and actin architecture using immunofluorescence and used deep learning-based computer vision technology to discriminate the early stages (6-12 hours) of MSC differentiation. Convolutional neural network (CNN) models trained by nucleus and actin images have high accuracy in reporting MSC differentiation; nuclear images alone can identify early stages of differentiation. Concurrently, we show that chromatin fluidity and heterochromatin levels or localization change during early MSC differentiation. This study quantifies changes in cell architecture during early MSC differentiation and describes a novel image-based diagnostic tool that could be widely used in MSC culture, expansion and utilization.

Application of deep-learning based computer vision in medical image analysis

The analyzing process of medical image has always been a crucial part for disease detection, diagnosis, surgery assisting and drug delivery. With the outbreak of population worldwide and the constant emergence of all types of complicated diseases, doctors nowadays are facing tremendous workloads and having difficulty in making precise diagnoses or performing more assured operations. Many experts have been striving to develop a wide range of techniques based on deep learning to ease doctors burden and improve patients recovery process or survival chances. This general review paper will mainly focus on providing readers with a brief understanding and knowledge about deep-learning based computer vision applied in modern medical image analysis. Several academic journals or books related to computer vision applications in medical image analysis were selected for review. Despite the advantages and convenience of this technique, this review finds out that potential obstacles still exist and can be overcome or amended in the future.

Deep transfer learning-based computer vision for real-time harvest period classification and impurity detection of Porphyra haitnensis

Deep transfer learning-based computer vision for real-time harvest period classification and impurity detection of Porphyra haitnensis

Deep learning based computer vision under the prism of 3D point clouds: a systematic review

Deep learning based computer vision under the prism of 3D point clouds: a systematic review

Deep Learning-Based Computer Vision Methods for Complex Traffic Environments Perception: A Review

Deep Learning-Based Computer Vision Methods for Complex Traffic Environments Perception: A Review

Plant Disease Detection using Convolutional Neural Network

The current agriculture system of the world is facing a massive challenge to agricultural productivity. Agricultural productivity must be increased to maintain enough food. Due to the various plant diseases, the expected agricultural productivity is impossible to achieve. Using Deep Learning (DL) in the agriculture field, farmers can monitor continuously rather than manually checking plants’ health. ResNet, DenseNet, GoogleNet, Inception, and YOLO are deep learning-based computer vision models. The most significant advantage of Deep Learning is that we do not need to extract features from the image manually. You feed the raw pixels in the images directly as inputs to the network. The network learns to extract features while training the model. The basic objective of this work is to propose multiple convolutional neural network-based models and identify the best model suitable for classifying the individual plant image into healthy and disease (multiple diseases) classes. This will help farmers and agriculturists to have a 360° insight into the cultivation status of the plants. Furthermore, to assess the proposed models; different quality metrics have been computed (i.e., accuracy and loss on the training, validation, and test datasets) to perform the comparative analysis between our proposed and existing techniques. These proposed models achieved a validation accuracy of 97% using MobileNet V1, 96% using MobileNet V1, and 83% using Vanilla CNN. While on 33 challenging images, a test accuracy of 82% using MobileNet V1, 76% using MobileNet V1, and 53% using Vanilla CNN has been achieved.

Improving detection of asphalt distresses with deep learning-based diffusion model for intelligent road maintenance

Research on road infrastructure structural health monitoring is critical due to the increasing problem of deteriorated conditions. The traditional approach to pavement distress detection relies on human-based visual recognition, a time-consuming and labor-intensive method. While Deep Learning-based computer vision systems are the most promising approach, they face the challenges of reduced performance due to the scarcity of labeled data due, high annotation costs misaligned with engineering applications, and limited instances of minority defects. This paper introduces a novel generative diffusion model for data augmentation, creating synthetic images of rare defects. It also investigates methods to enhance image quality and reduce production time. Compared to Generative Adversarial Networks, the optimal configuration excels in reliability, quality and diversity. After incorporating synthetic images into the training of our pavement distress detector, YOLOv5, its mean average precision has been enhanced. This computer-aided system enhances recognition and labelling efficiency, promoting intelligent maintenance and repairs.

Aerial visual data-driven approach for berthing capacity estimation in restricted waters

Coastal ship berth planning and management necessitate accurate identification of ship position and size, yet this is complicated by the presence of diverse ship types and mixed distribution patterns. Traditional survey-based and position-tracking ship monitoring methods have limitations in reflecting the precise spatial information of coastal ships and their surroundings. This paper introduces a novel aerial visual data-driven methodology to estimate the berthing capacity of sheltered space for stationary ships in geographically dispersed water areas. To measure the berthing capacity, image data from an unmanned aerial vehicle (UAV) survey is collected to reflect the spatial information of all types of ships. A deep learning-based computer vision algorithm is used to automatically detect, identify, and classify ships in sample images. Additionally, we introduce the occupancy factor to empirically quantify the spatial proportional correlation between the ship size and the requisite berthing capacity. A validation test of the proposed method is conducted across all sheltered spaces in Hong Kong, and the UAV survey yields over 7,000 informative images of all types of local ships. The ship identification algorithm achieves an average identification rate of 98.6%, demonstrating higher accuracy and reduced labor and time consumption than observational survey and manual counting methods. Findings indicate that the suggested sheltered space for individual ships is 7.75 times the ship size. The approach provides greater flexibility in estimating berthing capacity for mixed berthing forms in extensive or geographically dispersed water bodies, thereby having the potential to support the design of berths, anchorage areas, and typhoon shelter facilities in coastal regions.

Factors affecting the labelling accuracy of brain MRI studies relevant for deep learning abnormality detection.

Unlocking the vast potential of deep learning-based computer vision classification systems necessitates large data sets for model training. Natural Language Processing (NLP)-involving automation of dataset labelling-represents a potential avenue to achieve this. However, many aspects of NLP for dataset labelling remain unvalidated. Expert radiologists manually labelled over 5,000 MRI head reports in order to develop a deep learning-based neuroradiology NLP report classifier. Our results demonstrate that binary labels (normal vs. abnormal) showed high rates of accuracy, even when only two MRI sequences (T2-weighted and those based on diffusion weighted imaging) were employed as opposed to all sequences in an examination. Meanwhile, the accuracy of more specific labelling for multiple disease categories was variable and dependent on the category. Finally, resultant model performance was shown to be dependent on the expertise of the original labeller, with worse performance seen with non-expert vs. expert labellers.

ReplicAnt: a pipeline for generating annotated images of animals in complex environments using Unreal Engine

Deep learning-based computer vision methods are transforming animal behavioural research. Transfer learning has enabled work in non-model species, but still requires hand-annotation of example footage, and is only performant in well-defined conditions. To help overcome these limitations, we developed replicAnt, a configurable pipeline implemented in Unreal Engine 5 and Python, designed to generate large and variable training datasets on consumer-grade hardware. replicAnt places 3D animal models into complex, procedurally generated environments, from which automatically annotated images can be exported. We demonstrate that synthetic data generated with replicAnt can significantly reduce the hand-annotation required to achieve benchmark performance in common applications such as animal detection, tracking, pose-estimation, and semantic segmentation. We also show that it increases the subject-specificity and domain-invariance of the trained networks, thereby conferring robustness. In some applications, replicAnt may even remove the need for hand-annotation altogether. It thus represents a significant step towards porting deep learning-based computer vision tools to the field.

Towards a novel cyber physical control system framework: a deep learning driven use case

PurposeThe health crisis has highlighted the shortcomings of the industry sector which has revealed its vulnerability. To date, there is no guarantee of a return to the “world before”. The ability of companies to cope with these changes is a key competitive advantage requiring the adoption/mastery of industry 4.0 technologies. Therefore, companies must adapt their business processes to fit into similar situations.Design/methodology/approachThe proposed methodology comprises three steps. First, a comparative analysis of the existing CPSs is elaborated. Second, following this analysis, a deep learning driven CPS framework is proposed highlighting its components and tiers. Third, a real industrial case is presented to demonstrate the application of the envisioned framework. Deep learning network-based methods of object detection are used to train the model and evaluation is assessed accordingly.FindingsThe analysis revealed that most of the existing CPS frameworks address manufacturing related subjects. This illustrates the need for a resilient industrial CPS targeting other areas and considering CPSs as loopback systems preserving human–machine interaction, endowed with data tiering approach for easy and fast data access and embedded with deep learning-based computer vision processing methods.Originality/valueThis study provides insights about what needs to be addressed in terms of challenges faced due to unforeseen situations or adapting to new ones. In this paper, the CPS framework was used as a monitoring system in compliance with the precautionary measures (social distancing) and for self-protection with wearing the necessary equipments. Nevertheless, the proposed framework can be used and adapted to any industrial or non-industrial environments by adjusting object detection purpose.

Construction Instance Segmentation (CIS) Dataset for Deep Learning-Based Computer Vision

Deep learning-based computer vision (DLBCV) techniques have played an important role in intelligent construction. Image datasets are essential for developing DLBCV algorithms. However, a large-scale construction-specific dataset of major construction elements, such as precast components (PCs), PC delivery trucks, and workers not wearing safety helmets, remains absent. This paper presents the Construction Instance Segmentation (CIS) dataset, a new image dataset aimed at advancing state-of-the-art instance segmentation in the field of construction management. It contains 50,000 images with ten object categories belonging to construction workers, machines, and materials. Two rounds of algorithmic analysis have been conducted to refine and balance the dataset. Finally, a detailed statistical analysis of the dataset is presented.

PACMAN: A Framework for Pulse Oximeter Digit Detection and Reading in a Low-Resource Setting

In light of the COVID-19 pandemic, patients were required to manually input their daily oxygen saturation (SpO2) and pulse rate (PR) values into a health monitoring system—unfortunately, such a process trend to be an error in typing. Several studies attempted to detect the physiological value from the captured image using optical character recognition (OCR). However, the technology has limited availability with high cost. Thus, this study aimed to propose a novel framework called PACMAN (Pandemic Accelerated Human-Machine Collaboration) with a low-resource deep learning-based computer vision. We compared state-of-the-art object detection algorithms (scaled YOLOv4, YOLOv5, and YOLOR), including the commercial OCR tools for digit recognition on the captured images from the pulse oximeter display. All images were derived from crowdsourced data collection with varying quality and alignment. YOLOv5 was the best-performing model against the given model comparison across all datasets, notably the correctly orientated image dataset. We further improved the model performance with the digits auto-orientation algorithm and applied a clustering algorithm to extract SpO2 and PR values. The accuracy performance of YOLOv5 with the implementations was approximately 81.0-89.5%, which was enhanced compared to without any additional implementation. Accordingly, this study highlighted the completion of the PACMAN framework to detect and read digits in real-world datasets. The proposed framework has been currently integrated into the patient monitoring system utilized by hospitals nationwide.

The Potential of Visual ChatGPT for Remote Sensing

Recent advancements in Natural Language Processing (NLP), particularly in Large Language Models (LLMs), associated with deep learning-based computer vision techniques, have shown substantial potential for automating a variety of tasks. These are known as Visual LLMs and one notable model is Visual ChatGPT, which combines ChatGPT’s LLM capabilities with visual computation to enable effective image analysis. These models’ abilities to process images based on textual inputs can revolutionize diverse fields, and while their application in the remote sensing domain remains unexplored, it is important to acknowledge that novel implementations are to be expected. Thus, this is the first paper to examine the potential of Visual ChatGPT, a cutting-edge LLM founded on the GPT architecture, to tackle the aspects of image processing related to the remote sensing domain. Among its current capabilities, Visual ChatGPT can generate textual descriptions of images, perform canny edge and straight line detection, and conduct image segmentation. These offer valuable insights into image content and facilitate the interpretation and extraction of information. By exploring the applicability of these techniques within publicly available datasets of satellite images, we demonstrate the current model’s limitations in dealing with remote sensing images, highlighting its challenges and future prospects. Although still in early development, we believe that the combination of LLMs and visual models holds a significant potential to transform remote sensing image processing, creating accessible and practical application opportunities in the field.

Toward Automated Field Ballast Condition Evaluation: Development of a Ballast Scanning Vehicle

Ballast plays an essential role in the response of a railroad track to repeated loading. Ballast degradation may lead to poor drainage, lateral instability, and excessive settlement. Extreme levels of ballast degradation may cause service interruptions and safety concerns. Therefore, ballast condition evaluation is of great importance in ensuring safe and efficient operations. Current state-of-the-practice evaluation methods are heavily dependent on visual inspection, field sampling, and laboratory testing, which are subjective and labor-intensive. Meanwhile, existing inspection systems for railroads have not been customized for conducting in-depth evaluation of the ballast layer, including determination of the level of fouling and aggregate size and shape characteristics. For this reason, there is an urgent need for the development of a novel, vison-based ballast scanning platform. This paper introduces the ballast scanning vehicle (BSV), an automated platform that acquires high-quality images, videos, and 3-D height maps of ballast from plan and profile views of cut sections and trenches. The BSV is capable of performing data analysis and generating accurate and comprehensive evaluations of ballast conditions through geotechnical analyses. The essential design components, prototyping, and development stages are described. Further, preliminary data collected from testing on two in-service railroad tracks behind a shoulder ballast cleaner are presented to validate the functions of the BSV. The fully developed BSV serves as a data collection device for ballast evaluation and provides continuous and high-quality images for a deep learning-based computer vision algorithm for field ballast condition evaluation and geotechnical analyses.

Optimized deep learning vision system for human action recognition from drone images.

There are several benefits to constructing a lightweight vision system that is implemented directly on limited hardware devices. Most deep learning-based computer vision systems, such as YOLO (You Only Look Once), use computationally expensive backbone feature extractor networks, such as ResNet and Inception network. To address the issue of network complexity, researchers created SqueezeNet, an alternative compressed and diminutive network. However, SqueezeNet was trained to recognize 1000 unique objects as a broad classification system. This work integrates a two-layer particle swarm optimizer (TLPSO) into YOLO to reduce the contribution of SqueezeNet convolutional filters that have contributed less to human action recognition. In short, this work introduces a lightweight vision system with an optimized SqueezeNet backbone feature extraction network. Secondly, it does so without sacrificing accuracy. This is because that the high-dimensional SqueezeNet convolutional filter selection is supported by the efficient TLPSO algorithm. The proposed vision system has been used to the recognition of human behaviors from drone-mounted camera images. This study focused on two separate motions, namely walking and running. As a consequence, a total of 300 pictures were taken at various places, angles, and weather conditions, with 100 shots capturing running and 200 images capturing walking. The TLPSO technique lowered SqueezeNet's convolutional filters by 52%, resulting in a sevenfold boost in detection speed. With an F1 score of 94.65% and an inference time of 0.061 milliseconds, the suggested system beat earlier vision systems in terms of human recognition from drone-based photographs. In addition, the performance assessment of TLPSO in comparison to other related optimizers found that TLPSO had a better convergence curve and achieved a higher fitness value. In statistical comparisons, TLPSO surpassed PSO and RLMPSO by a wide margin.

Offloading Deep Learning Powered Vision Tasks From UAV to 5G Edge Server With Denoising

Offloading computationally heavy tasks from an unmanned aerial vehicle (UAV) to a remote server helps improve the battery life and can help reduce resource requirements. Deep learning based state-of-the-art computer vision tasks, such as object segmentation and object detection, are computationally heavy algorithms, requiring large memory and computing power. Many UAVs are using (pretrained) off-the-shelf versions of such algorithms. Offloading such power-hungry algorithms to a remote server could help UAVs save power significantly. However, deep learning based algorithms are susceptible to noise, and a wireless communication system, by its nature, introduces noise to the original signal. When the signal represents an image, noise affects the image. There has not been much work studying the effect of the noise introduced by the communication system on pretrained deep networks. In this work, we first analyze how reliable it is to offload deep learning based computer vision tasks (including both object segmentation and detection) by focusing on the effect of various parameters of a 5G wireless communication system on the transmitted image and demonstrate how the introduced noise of the used 5G wireless communication system reduces the performance of the offloaded deep learning task. Then solutions are introduced to eliminate (or reduce) the negative effect of the noise. The proposed framework starts with introducing many classical techniques as alternative solutions first, and then introduces a novel deep learning based solution to denoise the given noisy input image. The performance of various denoising algorithms on offloading both object segmentation and object detection tasks are compared. Our proposed deep transformer-based denoiser algorithm (NR-Net) yields the state-of-the-art results on reducing the negative effect of the noise in our experiments.