Computer Vision Approach Research Articles

Advancements in Geological Disaster Monitoring and Early Warning Systems: A Deep Learning and Computer Vision Approach

Advancements in Geological Disaster Monitoring and Early Warning Systems: A Deep Learning and Computer Vision Approach

Deep Learning in Precision Agriculture: Artificially Generated VNIR Images Segmentation for Early Postharvest Decay Prediction in Apples.

Food quality control is an important task in the agricultural domain at the postharvest stage for avoiding food losses. The latest achievements in image processing with deep learning (DL) and computer vision (CV) approaches provide a number of effective tools based on the image colorization and image-to-image translation for plant quality control at the postharvest stage. In this article, we propose the approach based on Generative Adversarial Network (GAN) and Convolutional Neural Network (CNN) techniques to use synthesized and segmented VNIR imaging data for early postharvest decay and fungal zone predictions as well as the quality assessment of stored apples. The Pix2PixHD model achieved higher results in terms of VNIR images translation from RGB (SSIM = 0.972). Mask R-CNN model was selected as a CNN technique for VNIR images segmentation and achieved 58.861 for postharvest decay zones, 40.968 for fungal zones and 94.800 for both the decayed and fungal zones detection and prediction in stored apples in terms of F1-score metric. In order to verify the effectiveness of this approach, a unique paired dataset containing 1305 RGB and VNIR images of apples of four varieties was obtained. It is further utilized for a GAN model selection. Additionally, we acquired 1029 VNIR images of apples for training and testing a CNN model. We conducted validation on an embedded system equipped with a graphical processing unit. Using Pix2PixHD, 100 VNIR images from RGB images were generated at a rate of 17 frames per second (FPS). Subsequently, these images were segmented using Mask R-CNN at a rate of 0.42 FPS. The achieved results are promising for enhancing the food study and control during the postharvest stage.

Exploring Stochastic Autoregressive Image Modeling for Visual Representation

Autoregressive language modeling (ALM) has been successfully used in self-supervised pre-training in Natural language processing (NLP). However, this paradigm has not achieved comparable results with other self-supervised approaches in computer vision (e.g., contrastive learning, masked image modeling). In this paper, we try to find the reason why autoregressive modeling does not work well on vision tasks. To tackle this problem, we fully analyze the limitation of visual autoregressive methods and proposed a novel stochastic autoregressive image modeling (named SAIM) by the two simple designs. First, we serialize the image into patches. Second, we employ the stochastic permutation strategy to generate an effective and robust image context which is critical for vision tasks. To realize this task, we create a parallel encoder-decoder training process in which the encoder serves a similar role to the standard vision transformer focusing on learning the whole contextual information, and meanwhile the decoder predicts the content of the current position so that the encoder and decoder can reinforce each other. Our method significantly improves the performance of autoregressive image modeling and achieves the best accuracy (83.9%) on the vanilla ViT-Base model among methods using only ImageNet-1K data. Transfer performance in downstream tasks also shows that our model achieves competitive performance. Code is available at https://github.com/qiy20/SAIM.

ECR Spotlight – Jamie McCoy

ECR Spotlight is a series of interviews with early-career authors from a selection of papers published in Journal of Experimental Biology and aims to promote not only the diversity of early-career researchers (ECRs) working in experimental biology during our centenary year, but also the huge variety of animals and physiological systems that are essential for the ‘comparative’ approach. Jamie McCoy is an author on ‘A phenomics approach reveals interspecific differences in integrated developmental responses to chronic elevated temperatures’, published in JEB. Jamie is a postdoctoral research associate in the lab of Dr Oliver Tills at the School of Biological and Marine Sciences, University of Plymouth, UK, investigating the use of bioimaging and computer vision approaches to examine the impacts of environmental change on the development of marine and freshwater invertebrates.

Computer vision and IoT research landscape for health and safety management on construction sites

AimsPerform a systematic review of current literature to evaluate and summarise the health and safety hazards on construction sites. MethodsScience Direct, SCOPUS and web of science databases were searched for research articles published from 2013 to 2021. From an initial search of 350 research articles, we removed the duplicate articles and carried out an analysis of the abstract and full text that focused on health, safety, hazards, behaviour, on-site health and safety and the digital technologies leaving a total of 66 studies included. ResultsComputer vision and Internet of Things (IoT) are the dominant technologies for health and safety management. A comparison of the two technologies reveals that computer vision is dominant because of its non-intrusive approach to data collection; thus, supporting the scalability of computer vision approach at the expense of cost and development time. It will help to prevent on-site health and safety hazards and injuries on construction site. ConclusionComputer vision offers non-intrusive benefits over Internet of Things (IoT); being able to detect the health and safety hazards. Computer vision has proved to be beneficial for better accuracy prediction, real time data monitoring, and model development for onsite health and safety analytics on the construction site.

Image-based and risk-informed detection of Subsea Pipeline damage

As one of the most important assets in the transportation of oil and gas products, subsea pipelines are susceptible to various environmental hazards, such as mechanical damage and corrosion, that can compromise their structural integrity and cause catastrophic environmental and financial damage. Autonomous underwater systems (AUS) are expected to assist offshore operations personnel and contribute to subsea pipeline inspection, maintenance, and damage detection tasks. Despite the promise of increased safety, AUS technology needs to mature, especially for image-based inspections with computer vision methods that analyze incoming images and detect potential pipeline damage through anomaly detection. Recent research addresses some of the most significant computer vision challenges for subsea environments, including visibility, color, and shape reconstruction. However, despite the high quality of subsea images, the lack of training data for reliable image analysis and the difficulty of incorporating risk-based knowledge into existing approaches continue to be significant obstacles. In this paper, we analyze industry-provided images of subsea pipelines and propose a methodology to address the challenges faced by popular computer vision methods. We focus on the difficulty posed by a lack of training data and the opportunities of creating synthetic data using risk analysis insights. We gather information on subsea pipeline anomalies, evaluate the general computer vision approaches, and generate synthetic data to compensate for the challenges that result from lacking training data, and evidence of pipeline damage in data, thereby increasing the likelihood of a more reliable AUS subsea pipeline inspection for damage detection.

Comparison of deep learning methods for grapevine growth stage recognition

Monitoring the phenological development stages of grapes represents a challenge in viticulture. It includes the phenological distinction of the growth stages of grapevines and the continuous technological developments, especially in computer vision, enabling a detailed classification of economically relevant development stages of grapes. In the present work, we show that based on a cascading computer vision approach, the development stages of grapes can be classified and distinguished at the micro level. In a comparative experiment (ResNet, DenseNet, InceptionV3), it could be shown that a ResNet architecture provides the best classification results with an average accuracy of 88.1%.

Predicting Ballast Fouling Conditions Through RGB-Based Statistical Quantity Analysis

Evaluating railway ballast fouling is critical to assessing track conditions and planning proper ballast maintenance. Varying material properties associated with different ballast fouling conditions can be used to evaluate the severity of fouling. Although approaches using ground penetration radar, impulse response, surface wave, and SmartRock have been developed to estimate the fouling conditions, these methods require special sensors and equipment, or well-trained technicians. Recently, convolutional neural network (CNN) based computer vision approaches have become popular in performing particle segmentation to obtain the ballast grain size distribution. The coarse aggregate fraction can be evaluated by the CNN method, but it is not easy to segment fine particles accurately with these approaches. This study proposes a novel image analysis approach to directly estimate the ballast fouling conditions. First, fouled ballast images with different fouling conditions are taken as references. The RGB (red, green, blue) color distributions of the fouled ballast images are then processed through statistical analysis. A strong linear correlation is found between Fouling Index (FI) and variance and this is used to establish the FI prediction model. The established FI prediction model is tested and validated by additional fouled ballast samples. The FI prediction model proposed in this study can be used to quantify ballast fouling conditions with promising performance.

A computer vision approach to the assessment of dried blood spot size and quality in newborn screening

BackgroundDried blood spot (DBS) size and quality affect newborn screening (NBS) test results. Visual assessment of DBS quality is subjective. MethodsWe developed and validated a computer vision (CV) algorithm to measure DBS diameter and identify incorrectly applied blood in images from the Panthera DBS puncher. We used CV to assess historical trends in DBS quality and correlate DBS diameter to NBS analyte concentrations in 130,620 specimens. ResultsCV estimates of DBS diameter were precise (percentage coefficient of variation < 1.3%) and demonstrated excellent agreement with digital calipers with a mean (standard deviation) difference of 0.23 mm (0.18 mm). An optimised logistic regression model showed a sensitivity of 94.3% and specificity of 96.8% for detecting incorrectly applied blood. In a validation set of images (n = 40), CV agreed with an expert panel in all acceptable specimens and identified all specimens rejected by the expert panel due to incorrect blood application or DBS diameter > 14 mm. CV identified a reduction in unsuitable NBS specimens from 25.5% in 2015 to 2% in 2021. Each mm decrease in DBS diameter decreased analyte concentrations by up to 4.3%. ConclusionsCV can aid assessment of DBS size and quality to harmonize specimen rejection both within and between laboratories.

A Siamese Transformer Network for Zero-Shot Ancient Coin Classification.

Ancient numismatics, the study of ancient coins, has in recent years become an attractive domain for the application of computer vision and machine learning. Though rich in research problems, the predominant focus in this area to date has been on the task of attributing a coin from an image, that is of identifying its issue. This may be considered the cardinal problem in the field and it continues to challenge automatic methods. In the present paper, we address a number of limitations of previous work. Firstly, the existing methods approach the problem as a classification task. As such, they are unable to deal with classes with no or few exemplars (which would be most, given over 50,000 issues of Roman Imperial coins alone), and require retraining when exemplars of a new class become available. Hence, rather than seeking to learn a representation that distinguishes a particular class from all the others, herein we seek a representation that is overall best at distinguishing classes from one another, thus relinquishing the demand for exemplars of any specific class. This leads to our adoption of the paradigm of pairwise coin matching by issue, rather than the usual classification paradigm, and the specific solution we propose in the form of a Siamese neural network. Furthermore, while adopting deep learning, motivated by its successes in the field and its unchallenged superiority over classical computer vision approaches, we also seek to leverage the advantages that transformers have over the previously employed convolutional neural networks, and in particular their non-local attention mechanisms, which ought to be particularly useful in ancient coin analysis by associating semantically but not visually related distal elements of a coin's design. Evaluated on a large data corpus of 14,820 images and 7605 issues, using transfer learning and only a small training set of 542 images of 24 issues, our Double Siamese ViT model is shown to surpass the state of the art by a large margin, achieving an overall accuracy of 81%. Moreover, our further investigation of the results shows that the majority of the method's errors are unrelated to the intrinsic aspects of the algorithm itself, but are rather a consequence of unclean data, which is a problem that can be easily addressed in practice by simple pre-processing and quality checking.

AntTracker: A low-cost and efficient computer vision approach to research leaf-cutter ants behavior

Leaf-cutter ants play a crucial role in agroecosystems, and understanding their behavior is key to developing effective damage control strategies. While several tracking solutions exist for ants in controlled environments or on aerial images, accurately measuring ant behavior in the wild remains a challenge. In this work, we propose a three-stage processing pipeline that segments individual ants, tracks their movement, and classifies whether they are carrying a leaf using a convolutional neural network. The output of the pipeline includes a timestamped record of the activity on the trail, accounting for parameters such as ant velocity, size and if it is going from or to the nest. We use the recently developed portable device AntVideoRecord to register video of a selected ant trail. To validate our approach, we collected a labeled dataset and evaluated each stage using standard metrics, achieving a median F2 score of 83% for segmentation, MOTA of 97% for tracking and F1 of 82% for detecting ants carrying a leaf. We then carried out a larger use case in the wild, demonstrating the effectiveness of our approach in capturing the intricate behaviors of leaf-cutter ants. We believe our method has the potential to inform the development of more effective ant damage control strategies in agroecosystems.

Few-Shot Learning for Clinical Natural Language Processing Using Siamese Neural Networks: Algorithm Development and Validation Study

Background Natural language processing (NLP) has become an emerging technology in health care that leverages a large amount of free-text data in electronic health records to improve patient care, support clinical decisions, and facilitate clinical and translational science research. Recently, deep learning has achieved state-of-the-art performance in many clinical NLP tasks. However, training deep learning models often requires large, annotated data sets, which are normally not publicly available and can be time-consuming to build in clinical domains. Working with smaller annotated data sets is typical in clinical NLP; therefore, ensuring that deep learning models perform well is crucial for real-world clinical NLP applications. A widely adopted approach is fine-tuning existing pretrained language models, but these attempts fall short when the training data set contains only a few annotated samples. Few-shot learning (FSL) has recently been investigated to tackle this problem. Siamese neural network (SNN) has been widely used as an FSL approach in computer vision but has not been studied well in NLP. Furthermore, the literature on its applications in clinical domains is scarce. Objective The aim of our study is to propose and evaluate SNN-based approaches for few-shot clinical NLP tasks. Methods We propose 2 SNN-based FSL approaches, including pretrained SNN and SNN with second-order embeddings. We evaluate the proposed approaches on the clinical sentence classification task. We experiment with 3 few-shot settings, including 4-shot, 8-shot, and 16-shot learning. The clinical NLP task is benchmarked using the following 4 pretrained language models: bidirectional encoder representations from transformers (BERT), BERT for biomedical text mining (BioBERT), BioBERT trained on clinical notes (BioClinicalBERT), and generative pretrained transformer 2 (GPT-2). We also present a performance comparison between SNN-based approaches and the prompt-based GPT-2 approach. Results In 4-shot sentence classification tasks, GPT-2 had the highest precision (0.63), but its recall (0.38) and F score (0.42) were lower than those of BioBERT-based pretrained SNN (0.45 and 0.46, respectively). In both 8-shot and 16-shot settings, SNN-based approaches outperformed GPT-2 in all 3 metrics of precision, recall, and F score. Conclusions The experimental results verified the effectiveness of the proposed SNN approaches for few-shot clinical NLP tasks.

Image processing-based classification of pavement fatigue severity using extremely randomized trees, deep neural network, and convolutional neural network

ABSTRACT Fatigue failure is a major structural defect found in the asphalt pavement subjected to repeated traffic loadings. In order to establish cost-effective maintenance plans, timely detection of pavement fatigue and classification of its severity are crucial. This study aims at developing an advanced image processing method based on Gaussian steerable filters, projection integrals, and texture descriptors for automating the tasks of interest. The extremely randomized trees (ERT) and deep neural network (DNN) are used to analyze the features extracted from the aforementioned image processing methods. The performance of ERT and DNN is also benchmarked against that of the convolutional neural network. A dataset consisting of 6000 samples has been collected in Da Nang city (Vietnam) to construct and verify the proposed computer vision approaches. Experimental results supported by Wilcoxon signed-rank tests confirm that the ERT-based method has achieved the most desired classification performance with an accuracy rate > 0.93.

Pose detection using deep learning for gait assessment

This approach of gait analysis helps in solving the problems associated with the health of aged people by concentrating on the frailty and senility syndromes that affect elderly individuals. Deterioration of cognitive and motor abilities is a common result of ageing and has an effect on elderly people's quality of life. Some studies have connected these changes in gait patterns to the decline in cognitive and motor function. As a result, gait analysis is a useful tool for diagnosing senility and frailty diseases. By transferring the pose estimation data to a correct statistical analysis, gait analysis can be carried out utilizing machine learning and computer vision approach, allowing physiotherapists to analyse and treat patients accordingly. The main problem observed is that physiotherapists will analyse stroke patients' walking patterns on their own. As a result, this approach solves the problem by providing an accurate result of the observation. As everything is computerised, proper treatment can be provided with accurate data. A physiotherapist could also use machine learning classification techniques to analyse and classify the observed disorder and correct it. Our study mainly focuses on using machine learning approach instead of traditional sensor based or consultation just by vision. This will serve as a newer and better proposition of the same.

Design and Evolution of Deep Convolutional Neural Networks in Image Classification – A Review

Convolutional Neural Network(CNN) is a well-known computer vision approach successfully applied for various classification and recognition problems. It has an outstanding power to identify patterns in 1D and 2D data. Though invented in 80’s, it became hugely successful after LeCun’s work on digit identification. Several CNN based models have been developed to record splendid performance on ImageNet and other databases. Ability of the CNN in learning complex features at different hierarchy from the data had made it the most successful among deep learning algorithms. Innovative architectural designs and hyper parameteroptimization have greatly improved the efficiency of CNN in pattern recognition. This review majorly focuses on the evolution and history of CNN models. Landmark CNN architectures are discussed with their categorization depending on various parameters. In addition, this also explores the architectural details of different layers, activation function, optimizers and other hyperparameters used by CNN. Review concludes by shedding the light on the applications and observations to be considered while designing the network.

A computer vision approach to identifying the manufacturer of posterior thoracolumbar instrumentation systems.

Knowledge of the manufacturer of the previously implanted pedicle screw systems prior to revision spinal surgery may facilitate faster and safer surgery. Often, this information is unavailable because patients are referred by other centers or because of missing information in the patients' records. Recently, machine learning and computer vision have gained wider use in clinical applications. The authors propose a computer vision approach to classify posterior thoracolumbar instrumentation systems. Lateral and anteroposterior (AP) radiographs obtained in patients undergoing posterior thoracolumbar pedicle screw implantation for any indication at the authors' institution (2015-2021) were obtained. DICOM images were cropped to include both the pedicle screws and rods. Images were labeled with the manufacturer according to the operative record. Multiple feature detection methods were tested (SURF, MESR, and Minimum Eigenvalues); however, the bag-of-visual-words technique with KAZE feature detection was ultimately used to construct a computer vision support vector machine (SVM) classifier for lateral, AP, and fused lateral and AP images. Accuracy was tested using an 80%/20% training/testing pseudorandom split over 100 iterations. Using a reader study, the authors compared the model performance with the current practice of surgeons and manufacturer representatives identifying spinal hardware by visual inspection. Among the three image types, 355 lateral, 379 AP, and 338 fused radiographs were obtained. The five pedicle screw implants included in this study were the Globus Medical Creo, Medtronic Solera, NuVasive Reline, Stryker Xia, and DePuy Expedium. When the two most common manufacturers used at the authors' institution were binarily classified (Globus Medical and Medtronic), the accuracy rates for lateral, AP, and fused images were 93.15% ± 4.06%, 88.98% ± 4.08%, and 91.08% ± 5.30%, respectively. Classification accuracy decreased by approximately 10% with each additional manufacturer added. The multilevel five-way classification accuracy rates for lateral, AP, and fused images were 64.27% ± 5.13%, 60.95% ± 5.52%, and 65.90% ± 5.14%, respectively. In the reader study, the model performed five-way classification on 100 test images with 79% accuracy in 14 seconds, compared with an average of 44% accuracy in 20 minutes for two surgeons and three manufacturer representatives. The authors developed a KAZE feature detector with an SVM classifier that successfully identified posterior thoracolumbar hardware at five-level classification. The model performed more accurately and efficiently than the method currently used in clinical practice. The relative computational simplicity of this model, from input to output, may facilitate future prospective studies in the clinical setting.

A computer vision approach for analyzing label free leukocyte trafficking dynamics on a microvascular mimetic.

High-content imaging techniques in conjunction with in vitro microphysiological systems (MPS) allow for novel explorations of physiological phenomena with a high degree of translational relevance due to the usage of human cell lines. MPS featuring ultrathin and nanoporous silicon nitride membranes (µSiM) have been utilized in the past to facilitate high magnification phase contrast microscopy recordings of leukocyte trafficking events in a living mimetic of the human vascular microenvironment. Notably, the imaging plane can be set directly at the endothelial interface in a µSiM device, resulting in a high-resolution capture of an endothelial cell (EC) and leukocyte coculture reacting to different stimulatory conditions. The abundance of data generated from recording observations at this interface can be used to elucidate disease mechanisms related to vascular barrier dysfunction, such as sepsis. The appearance of leukocytes in these recordings is dynamic, changing in character, location and time. Consequently, conventional image processing techniques are incapable of extracting the spatiotemporal profiles and bulk statistics of numerous leukocytes responding to a disease state, necessitating labor-intensive manual processing, a significant limitation of this approach. Here we describe a machine learning pipeline that uses a semantic segmentation algorithm and classification script that, in combination, is capable of automated and label-free leukocyte trafficking analysis in a coculture mimetic. The developed computational toolset has demonstrable parity with manually tabulated datasets when characterizing leukocyte spatiotemporal behavior, is computationally efficient and capable of managing large imaging datasets in a semi-automated manner.

Quantifying the movement, behaviour and environmental context of group-living animals using drones and computer vision.

Methods for collecting animal behaviour data in natural environments, such as direct observation and biologging, are typically limited in spatiotemporal resolution, the number of animals that can be observed and information about animals' social and physical environments. Video imagery can capture rich information about animals and their environments, but image-based approaches are often impractical due to the challenges of processing large and complex multi-image datasets and transforming resulting data, such as animals' locations, into geographical coordinates. We demonstrate a new system for studying behaviour in the wild that uses drone-recorded videos and computer vision approaches to automatically track the location and body posture of free-roaming animals in georeferenced coordinates with high spatiotemporal resolution embedded in contemporaneous 3D landscape models of the surrounding area. We provide two worked examples in which we apply this approach to videos of gelada monkeys and multiple species of group-living African ungulates. We demonstrate how to track multiple animals simultaneously, classify individuals by species and age-sex class, estimate individuals' body postures (poses) and extract environmental features, including topography of the landscape and animal trails. By quantifying animal movement and posture while reconstructing a detailed 3D model of the landscape, our approach opens the door to studying the sensory ecology and decision-making of animals within their natural physical and social environments.

NeuroSuitUp: System Architecture and Validation of a Motor Rehabilitation Wearable Robotics and Serious Game Platform

This article presents the system architecture and validation of the NeuroSuitUp body-machine interface (BMI). The platform consists of wearable robotics jacket and gloves in combination with a serious game application for self-paced neurorehabilitation in spinal cord injury and chronic stroke. The wearable robotics implement a sensor layer, to approximate kinematic chain segment orientation, and an actuation layer. Sensors consist of commercial magnetic, angular rate and gravity (MARG), surface electromyography (sEMG), and flex sensors, while actuation is achieved through electrical muscle stimulation (EMS) and pneumatic actuators. On-board electronics connect to a Robot Operating System environment-based parser/controller and to a Unity-based live avatar representation game. BMI subsystems validation was performed using exercises through a Stereoscopic camera Computer Vision approach for the jacket and through multiple grip activities for the glove. Ten healthy subjects participated in system validation trials, performing three arm and three hand exercises (each 10 motor task trials) and completing user experience questionnaires. Acceptable correlation was observed in 23/30 arm exercises performed with the jacket. No significant differences in glove sensor data during actuation state were observed. No difficulty to use, discomfort, or negative robotics perception were reported. Subsequent design improvements will implement additional absolute orientation sensors, MARG/EMG based biofeedback to the game, improved immersion through Augmented Reality and improvements towards system robustness.

Compliance With the US Food and Drug Administration's Guidelines for Health Warning Labels and Engagement in Little Cigar and Cigarillo Content: Computer Vision Analysis of Instagram Posts.

Health warnings in tobacco advertisements provide health information while also increasing the perceived risks of tobacco use. However, existing federal laws requiring warnings on advertisements for tobacco products do not specify whether the rules apply to social media promotions. This study aims to examine the current state of influencer promotions of little cigars and cigarillos (LCCs) on Instagram and the use of health warnings in influencer promotions. Instagram influencers were identified as those who were tagged by any of the 3 leading LCC brand Instagram pages between 2018 and 2021. Posts from identified influencers, which mentioned one of the three brands were considered LCC influencer promotions. A novel Warning Label Multi-Layer Image Identification computer vision algorithm was developed to measure the presence and properties of health warnings in a sample of 889 influencer posts. Negative binomial regressions were performed to examine the associations of health warning properties with post engagement (number of likes and comments). The Warning Label Multi-Layer Image Identification algorithm was 99.3% accurate in detecting the presence of health warnings. Only 8.2% (n=73) of LCC influencer posts included a health warning. Influencer posts that contained health warnings received fewer likes (incidence rate ratio 0.59, P<.001, 95% CI 0.48-0.71) and fewer comments (incidence rate ratio 0.46, P<.001, 95% CI 0.31-0.67). Health warnings are rarely used by influencers tagged by LCC brands' Instagram accounts. Very few influencer posts met the US Food and Drug Administration's health warning requirement of size and placement for tobacco advertising. The presence of a health warning was associated with lower social media engagement. Our study provides support for the implementation of comparable health warning requirements to social media tobacco promotions. Using an innovative computer vision approach to detect health warning labels in influencer promotions on social media is a novel strategy for monitoring health warning compliance in social media tobacco promotions.