Computer Vision-based Methods Research Articles

Design and Implementation of a Low-Cost, Linear Robotic Camera System, Targeting Greenhouse Plant Growth Monitoring

Automated greenhouse production systems frequently employ non-destructive techniques, such as computer vision-based methods, to accurately measure plant physiological properties and monitor crop growth. By utilizing an automated image acquisition and analysis system, it becomes possible to swiftly assess the growth and health of plants throughout their entire lifecycle. This valuable information can be utilized by growers, farmers, and crop researchers who are interested in self-cultivation procedures. At the same time, such a system can alleviate the burden of daily plant photography for human photographers and crop researchers, while facilitating automated plant image acquisition for crop status monitoring. Given these considerations, the aim of this study was to develop an experimental, low-cost, 1-DOF linear robotic camera system specifically designed for automated plant photography. As an initial evaluation of the proposed system, which targets future research endeavors of simplifying the process of plant growth monitoring in a small greenhouse, the experimental setup and precise plant identification and localization are demonstrated in this work through an application on lettuce plants, imaged mostly under laboratory conditions.

Accurate semantic segmentation of small-body craters for navigation

Feature extraction and matching play integral roles in autonomous vision navigation. As a typical morphological feature, craters can be used as navigation landmarks. However, the spin of small bodies introduces substantial variations in illumination and viewing angles within acquired images, thereby posing challenges to feature extraction and matching algorithms. Addressing these challenges, a deep learning-based approach emerges as pivotal in mitigating the limitations of existing navigation algorithms. This paper presents an algorithm designed to segment noteworthy craters and irregular regions on such small bodies, thereby furnishing crucial autonomous navigation data for missions involving uncharted terrains. By seamlessly integrating deep learning techniques with conventional computer vision-based feature description methods, the algorithm achieves matching segmentation results of noteworthy precision. The experimental results show that the algorithm has better results compared to the existing mainstream segmentation networks. And it can achieve the accurate matching of segmentation results.

Variational mode decomposition-based multirate data-fusion framework for estimating structural dynamic displacement by integrating vision- and acceleration-based measurements

Dynamic displacement is a crucial parameter in structural health monitoring (SHM) for assessing the safety, dependability, and suitability of structures under various types of excitations. Computer vision-based methods for dynamic displacement estimation have attracted much interest owing to their cost-effectiveness and convenience. However, these methods are limited by their low sampling rates and high data sensitivity. To compensate for these limitations, methods for combining data obtained from other sensors have been proposed. In this study, an experimental data-fusion framework for displacement estimation based on variational mode decomposition (VMD) was developed to leverage the advantages of vision- and acceleration-based measurements. The measurements were decomposed into ensembles of modes and recomposed to reconstruct the displacement with a higher accuracy and over a wider frequency range. An optimal mode recomposition method was proposed to achieve optimal mode combinations. Furthermore, this study introduced an improved vision-based displacement measurement method and a VMD-based indirect acceleration measurement method. The proposed framework was validated through four-story RC structure tests, which demonstrated that the method could enhance the accuracy of displacement estimation and extend the feasible frequency range compared with single-source displacement measurements. The method provides a promising solution for more effective health monitoring of modern structures subjected to a wide variety of dynamic loads.

A target-free vision-based method for out-of-plane vibration measurement using projection speckle and camera self-calibration technology

In recent decades, computer vision-based methods have been introduced into engineering structures to provide full-field, non-contact vibration measurement. However, the vision-based vibration measurement method is challenging to measure the targetless structures. This paper proposed a target-free out-of-plane vibration measurement method using projection speckle and camera self-calibration technology. In this method, the camera extrinsic parameter estimation result based on the direct method is used as the initial value in the proposed method. Then, a nonlinear inequality constraint on the extrinsic parameters is established regarding the epipolar condition to minimize the reprojection error. Furthermore, the projection relationship between the projection speckle displacement and the actual displacement of the structure is studied so that the deviation of the out-of-plane vibration displacement is corrected. In the experimental verification, 3D reconstruction and rigid translation experiments are designed, respectively. The results show that the self-calibration accuracy of the binocular camera proposed in this paper is notably improved compared with the existing methods. At the same time, the deviation correction effect of the vibration displacement is significant. Finally, the typical flat plate structure and composite wing structure are taken as vibration measurement cases to demonstrate that the proposed approach is reliable and accurate in measuring the dynamic responses of engineering structures.

Yoga Pose Estimation Using Angle-Based Feature Extraction.

This research addresses the challenges of maintaining proper yoga postures, an issue that has been exacerbated by the COVID-19 pandemic and the subsequent shift to virtual platforms for yoga instruction. This research aims to develop a mechanism for detecting correct yoga poses and providing real-time feedback through the application of computer vision and machine learning (ML) techniques. This study utilized computer vision-based pose estimation methods to extract features and calculate yoga pose angles. A variety of models, including extremely randomized trees, logistic regression, random forest, gradient boosting, extreme gradient boosting, and deep neural networks, were trained and tested to classify yoga poses. Our study employed the Yoga-82 dataset, consisting of many yoga pose images downloaded from the web. The results of this study show that the extremely randomized trees model outperformed the other models, achieving the highest prediction accuracy of 91% on the test dataset and 92% in a fivefold cross-validation experiment. Other models like random forest, gradient boosting, extreme gradient boosting, and deep neural networks achieved accuracies of 90%, 89%, 90%, and 85%, respectively, while logistic regression underperformed, having the lowest accuracy. This research concludes that the extremely randomized trees model presents superior predictive power for yoga pose recognition. This suggests a valuable avenue for future exploration in this domain. Moreover, the approach has significant potential for implementation on low-powered smartphones with minimal latency, thereby enabling real-time feedback for users practicing yoga at home.

Automatic Facial Palsy Detection—From Mathematical Modeling to Deep Learning

Automated solutions for medical diagnosis based on computer vision form an emerging field of science aiming to enhance diagnosis and early disease detection. The detection and quantification of facial asymmetries enable facial palsy evaluation. In this work, a detailed review of the quantification of facial palsy takes place, covering all methods ranging from traditional manual mathematical modeling to automated computer vision-based methods. Moreover, facial palsy quantification is defined in terms of facial asymmetry indices calculation for different image modalities. The aim is to introduce readers to the concept of mathematical modeling approaches for facial palsy detection and evaluation and present the process of the development of this separate application field over time. Facial landmark extraction, facial datasets, and palsy grading systems are included in this research. As a general conclusion, machine learning methods for the evaluation of facial palsy lead to limited performance due to the use of handcrafted features, combined with the scarcity of the available datasets. Deep learning methods allow the automatic learning of discriminative deep facial features, leading to comparatively higher performance accuracies. Datasets limitations, proposed solutions, and future research directions in the field are also presented.

An Improved Object Detection and Trajectory Prediction Method for Traffic Conflicts Analysis

Although computer vision-based methods have seen broad utilisation in evaluating traffic situations, there is a lack of research on the assessment and prediction of near misses in traffic. In addition, most object detection algorithms are not very good at detecting small targets. This study proposes a combination of object detection and tracking algorithms, Inverse Perspective Mapping (IPM), and trajectory prediction mechanisms to assess near-miss events. First, an instance segmentation head was proposed to improve the accuracy of the object frame box detection phase. Secondly, IPM was applied to all detection results. The relationship between them is then explored based on their distance to determine whether there is a near-miss event. In this process, the moving speed of the target was considered as a parameter. Finally, the Kalman filter is used to predict the object's trajectory to determine whether there will be a near-miss in the next few seconds. Experiments on Closed-Circuit Television (CCTV) datasets showed results of 0.94 mAP compared to other state-of-the-art methods. In addition to improved detection accuracy, the advantages of instance segmentation fused object detection for small target detection are validated. Therefore, the results will be used to analyse near misses more accurately.

Oolong tea cultivars categorization and germination period classification based on multispectral information.

Recognizing and identifying tea plant (Camellia sinensis) cultivar plays a significant role in tea planting and germplasm resource management, particularly for oolong tea. There is a wide range of high-quality oolong tea with diverse varieties of tea plants that are suitable for oolong tea production. The conventional method for identifying and confirming tea cultivars involves visual assessment. Machine learning and computer vision-based automatic classification methods offer efficient and non-invasive alternatives for rapid categorization. Despite advancements in technology, the identification and classification of tea cultivars still pose a complex challenge. This paper utilized machine learning approaches for classifying 18 oolong tea cultivars based on 27 multispectral characteristics. Then the SVM classification model was executed using three optimization algorithms, namely genetic algorithm (GA), particle swarm optimization (PSO), and grey wolf optimizer (GWO). The results revealed that the SVM model optimized by GWO achieved the best performance, with an average discrimination rate of 99.91%, 93.30% and 92.63% for the training set, test set and validation set, respectively. In addition, based on the multispectral information (h, s, r, b, L, Asm, Var, Hom, Dis, σ, S, G, RVI, DVI, VOG), the germination period of oolong tea cultivars can be completely evaluated by Fisher discriminant analysis. The study indicated that the practical protection of tea plants through automated and precise classification of oolong tea cultivars and germination periods is feasible by utilizing multispectral imaging system.

A Study on Wetland Cover Map Formulation and Evaluation Using Unmanned Aerial Vehicle High-Resolution Images

Wetlands possess significant ecological value and play a crucial role in the environment. Recent advancements in remote exploration technology have enabled a quantitative analysis of wetlands through surveys on the type of cover present. However, the classification of complex cover types as land cover types in wetlands remains challenging, leading to ongoing studies aimed at addressing this issue. With the advent of high-resolution sensors in unmanned aerial vehicles (UAVs), researchers can now obtain detailed data and utilize them for their investigations. In this paper, we sought to establish an effective method for classifying centimeter-scale images using multispectral and hyperspectral techniques. Since there are numerous classes of land cover types, it is important to build and extract effective training data for each type. In addition, computer vision-based methods, especially those that combine deep learning and machine learning, are attracting considerable attention as high-accuracy methods. Collecting training data before classifying by cover type is an important factor that which requires effective data sampling. To obtain accurate detection results, a few data sampling techniques must be tested. In this study, we employed two data sampling methods (endmember and pixel sampling) to acquire data, after which their accuracy and detection outcomes were compared through classification using spectral angle mapper (SAM), support vector machine (SVM), and artificial neural network (ANN) approaches. Our findings confirmed the effectiveness of the pixel-based sampling method, demonstrating a notable difference of 38.62% compared to the endmember sampling method. Moreover, among the classification methods employed, the SAM technique exhibited the highest effectiveness, with approximately 10% disparity observed in multispectral data and 7.15% in hyperspectral data compared to the other models. Our findings provide insights into the accuracy and classification outcomes of different models based on the sampling method employed in spectral imagery.

Crack detection of continuous casting slab by evolutionary topology backbone search

In recent years, computer vision-based methods have been widely used in steel defect detection. Traditional image detection methods mainly rely on manually extracted features, resulting in poor generalization. Deep learning methods are sensitive to the number of samples, and the network structure design relies heavily on manual experience. To address these problems, a backbone network search algorithm based on evolutionary topology is proposed in this paper for crack detection on continuous casting surfaces. Firstly, a variable-length genetic encoding scheme is designed for industrial defect problems with different data complexity, which can improve the applicability of the algorithm and extend the search space. Secondly, to effectively solve the channel redundancy problem in densely connected CNNs, a random pruning strategy for network connection channels is proposed to reduce the topological space and the complexity of the model. Finally, a computational resource allocation mechanism based on a dynamic surrogate model is devised. The surrogate model predicts the individual performance to ensure that computational resources can be concentrated on individuals with better quality. In addition to the steel crack image dataset, the proposed method also uses the workpiece crack image dataset for a supplementary experiment. Experimental results show that the proposed algorithm can achieve better detection performance with fewer computational resources compared to manually designed deep learning algorithms and classical approaches that use evolutionary algorithms to search network architectures.

Transfer learning-driven ensemble model for detection of diabetic retinopathy disease.

In this study, we propose an ensemble model for the detection of diabetic retinopathy (DR) illness that is driven by transfer learning. Due to diabetes, the DR is a problem that affects the eyes. The retinal blood vessels in a person with high blood sugar deteriorate. The blood arteries may enlarge and leak as a result, or they may close and stop the flow of blood. If DR is not treated, it can become severe, damage vision, and eventually result in blindness. Medical experts study the colored fundus photos for this reason in order to manually diagnose disease, however this is a perilous technique. As a result, the condition was automatically identified utilizing retinal scans and a number of computer vision-based methods. A model is trained on one task or datasets employing the transfer learning (TL) technique, and then the pre-trained models or weights are applied to another task or dataset. Six deep learning (DL)-based convolutional neural network (CNN) models were trained in this study using huge datasets of reasonable photos, including DenseNet-169, VGG-19, ResNet101-V2, Mobilenet-V2, and Inception-V3. We also applied a data-preprocessing strategy to improve the accuracy and lower the training costs in order to improve the results. The experimental results demonstrate that the suggested model works better than existing approaches on the same dataset, with an accuracy of up to 98%, and detects the stage of DR.

Wood Veneer Defect Detection Based on Multiscale DETR with Position Encoder Net

Wood is one of the main building materials. However, defects on veneers result in substantial waste of wood resources. Traditional veneer defect detection relies on manual experience or photoelectric-based methods, which are either subjective and inefficient or need substantial investment. Computer vision-based object detection methods have been used in many realistic areas. This paper proposes a new deep learning defect detection pipeline. First, an image collection device is constructed and a total of more than 16,380 defect images are collected coupled with a mixed data augmentation method. Then, a detection pipeline is designed based on DEtection TRansformer (DETR). The original DETR needs position encoding functions to be designed and is ineffective for small object detection. To solve these problems, a position encoding net is designed with multiscale feature maps. The loss function is also redefined for much more stable training. The results from the defect dataset show that using a light feature mapping network, the proposed method is much faster with similar accuracy. Using a complex feature mapping network, the proposed method is much more accurate with similar speed.

Evaluation of Hand Washing Procedure Using Vision-Based Frame Level and Spatio-Temporal Level Data Models

Hand hygiene is obligatory for all healthcare workers and vital for patient care. During COVID-19, adequate hand washing was among recommended measures for preventing virus transmission. A general hand-washing procedure consisting several steps is recommended by World Health Organization for ensuring hand hygiene. This process can vary from person to person and human supervision for inspection would be impractical. In this study, we propose computer vision-based new methods using 12 different neural network models and 4 different data models (RGB, Point Cloud, Point Gesture Map, Projection) for the classification of 8 universally accepted hand-washing steps. These methods can also perform well under situations where the order of steps is not observed or the duration of steps are varied. Using a custom dataset, we achieved 100% accuracy with one of the models, and 94.23% average accuracy for all models. We also developed a real-time robust data acquisition technique where RGB and depth streams from Kinect 2.0 camera were utilized. Results showed that with the proposed methods and data models, efficient hand hygiene control is possible.

Dynamic fry counting based on multi-object tracking and one-stage detection

Fry counting is an essential task in aquaculture that provides valuable information for feedstuff feeding, culture density adjustment, and fish economic benefit estimation. However, most current computer vision-based fry counting methods are designed for statically counting, with a fixed number of fry in a container. As a result, these methods are not suitable for dynamic scenarios, such as flowing water, where the effects of factors like water velocity and uneven lighting can cause errors in counting. To address this issue and improve accuracy, we propose a dynamic fry counting method capable of dealing with large-number fry counting. Specifically, we regard fry counting as a Multiple Object Tracking (MOT) problem based on Tracking-By-Detection (TBD) framework. Firstly, a dataset of streaming fry is constructed for training the detection model, and videos with varying numbers of fry are manually counted for counting tests. Secondly, due to the highly similar appearance of fry, Simple Online and Realtime Tracking (SORT) is combined with You Only Look Once version 5 Nano (YOLOv5-Nano) to perform fry tracking. To achieve stable tracking of fry movement, the SORT algorithm is improved based on multi-matching and trajectory recovery, reaching 82.6 % multi-object tracking accuracy. Finally, fry counting is realized based on the intersection of a counting line and moving trajectories. The results indicate that the proposed counting method achieves 96.4 % accuracy and can be implemented on CPU at 5–10 frames per second (fps) and GPU at 15–35 fps. In conclusion, the proposed method automatically counts the number of fry in videos with higher speed and accuracy, facilitating automated fishery farming.

A novel cell contour-based instance segmentation model and its applications in HER2 breast cancer discrimination

Breast cancer is a vital disease that seriously threatens women's health. Fluorescence In Situ Hybridization (FISH) is an effective method to detect breast cancer and can assist in utilizing effective targeted therapies. However, manual and computer vision-based methods have shortcomings in accuracy and real-time performance in post-processing screening samples for breast cancer detection by FISH. We obtain highly accurate cell contour polygons using our original method, called Bio-Snake. Specifically, we first apply B-channel Topology Analysis on breast cancer FISH samples based on boundary tracking algorithm in order to automatically determine the initial cell contour. Afterwards, we apply Adaptive LSTM Alignment method on cell contour optimization in order to more accurately utilize the local contours information. The Weighted Loss is used to train the cell contour segmentation model. Finally, we propose HER2 & Chromosome 17 copy number cell attribution model to assist in sample statistics. Our experimental results demonstrate that our proposed Bio-Snake achieves 51.4 average precision (AP) and 34.0 Mask-AP on the standard benchmarks with an inference speed of 11.1 FPS, which is 34% higher and 14% faster than the current state-of-the-art method.

Bottleneck Detection in Modular Construction Factories Using Computer Vision.

The construction industry is increasingly adopting off-site and modular construction methods due to the advantages offered in terms of safety, quality, and productivity for construction projects. Despite the advantages promised by this method of construction, modular construction factories still rely on manually-intensive work, which can lead to highly variable cycle times. As a result, these factories experience bottlenecks in production that can reduce productivity and cause delays to modular integrated construction projects. To remedy this effect, computer vision-based methods have been proposed to monitor the progress of work in modular construction factories. However, these methods fail to account for changes in the appearance of the modular units during production, they are difficult to adapt to other stations and factories, and they require a significant amount of annotation effort. Due to these drawbacks, this paper proposes a computer vision-based progress monitoring method that is easy to adapt to different stations and factories and relies only on two image annotations per station. In doing so, the Scale-invariant feature transform (SIFT) method is used to identify the presence of modular units at workstations, and the Mask R-CNN deep learning-based method is used to identify active workstations. This information was synthesized using a near real-time data-driven bottleneck identification method suited for assembly lines in modular construction factories. This framework was successfully validated using 420 h of surveillance videos of a production line in a modular construction factory in the U.S., providing 96% accuracy in identifying the occupancy of the workstations and an F-1 Score of 89% in identifying the state of each station on the production line. The extracted active and inactive durations were successfully used via a data-driven bottleneck detection method to detect bottleneck stations inside a modular construction factory. The implementation of this method in factories can lead to continuous and comprehensive monitoring of the production line and prevent delays by timely identification of bottlenecks.

CANet: Context-aware Aggregation Network for Salient Object Detection of Surface Defects

Surface defect detection has become more and more important in the industrial manufacture and engineering application in recent years. However, due to the lack of overall perception and interaction among features layers, lots of computer vision-based detection methods cannot grab the complete details of defects when dealing with complex scenes, such as low contrast and irregular shape. Therefore, in this paper, we propose a Context-aware Aggregation Network (CANet) to accurately pop-out the defects, where we focus on the mining of context cues and the fusion of multiple context features. To be specific, embarking on the multi-level deep features extracted by encoder, we first deploy a sufficient exploration to dig the context information by deploying the weighted convolution pyramid (WCP) module, which extracts multi-scale context features, transfers the information flow between different resolution features, and fuses the features with same resolution. By this way, we can obtain the effective context pyramid features. Then, the decoder deploys the weighted context attention (WCA) module to filter the irrelevant information from context features and employs the cascaded fusion structure (CFS) to aggregate the multiple context cues in a hierarchical way. Following this way, the generated high-quality saliency maps can highlight the defects accurately and completely. Extensive experiments are performed on four public datasets, and the results firmly prove the effectiveness and superiority of the proposed CANet under different evaluation metrics.

SDDet: An Enhanced Encoder–Decoder Network With Hierarchical Supervision for Surface Defect Detection

Surface defect detection (SDDet) is extremely critical for quality control and routine maintenance, and computer vision-based methods have delivered promising performance in various industrial fields. However, there remains big challenges in region consistency and boundary localization, due to complicated appearance in defects, interference of artifacts, and low contrast. To overcome these issues, in this article, we propose an enhanced encoder–decoder network with hierarchical supervision for SDDet. Specifically, deformable convolution is adopted to enable feature learning in a free-form way, thus making the network more robust to various defect structures. Then, a bridge module is applied to capture richer multilevel contextual information, which is conducive to distinguishing defects from the background artifacts. Next, the learned features from adjacent layers are adaptively combined by selective feature aggregation (SFA) module to enhance discriminative features and suppress valueless features. Meanwhile, the bottleneck refinement (BR) module is added to further refine the boundary of defect region, resulting in a satisfying boundary details. Finally, a hybrid three-level loss is developed to guide the network to focus more on error-prone hard pixels. Additionally, a hardware system embedded with our SDDet is built to achieve automatic and accurate defect inspection. Experimental results over four datasets demonstrate the effectiveness and generalization of our method compared with state-of-the-art methods.

Three-dimensional structural displacement measurement using monocular vision and deep learning based pose estimation

Computer vision-based displacement measurement methods have received increasing attention for the structural health monitoring of buildings and infrastructures owing to their advantages over traditional contact sensors. Meanwhile, surveillance cameras widely equipped in urban areas can record a large number of images and videos of buildings and infrastructure, which have the potential to support structural analysis in structural health monitoring or engineering investigations. The three-dimensional (3D) displacement of structures is important for structural analysis. It is challenging for the existing vision-based measurement methods to obtain all the 3D displacement components because they require either multi-view camera systems or additional specially designed targets, which makes it difficult to meet the requirements of measurement applications based on urban surveillance cameras. Therefore, this study proposes a 3D structural displacement measurement method using monocular vision and deep learning based pose estimation. The method uses virtual rendering to synthesize the training set based on the 3D models of the target objects, then trains the deep learning model DPOD (Dense Pose Object Detector) to estimate the poses of the target object, and finally measures the 3D translation of the structures based on the original and destination poses or the original pose and keypoint matching. The effectiveness of the proposed method was validated through static and dynamic experiments. The results showed that the method can meet the needs of obtaining 3D structural displacement and has good accuracy in identifying the principal frequencies of the dynamic responses. The proposed method can support the 3D displacement measurements of buildings and infrastructure based on urban surveillance cameras.

Target-free vision-based approach for modal identification of a simply-supported bridge

Accurate and efficient vibration response measurement is one of the key tasks in structural health monitoring (SHM) of civil engineering structures. Normally, dynamic responses such as displacement responses, are acquired by physical sensors and data acquisition systems installed on the structures. However, the installation of sensors may be time-consuming and costly. In recent decades, computer vision-based methods have been introduced into civil engineering to provide alternatives for traditional displacement measurements. However, the camera specifications, such as limitations on field of view (FOV) and relatively low resolution of economic lens, have prevented the vision-based method from being widely used for vibration displacement measurement on relatively large-scale structures. This paper proposes a target-free vision-based approach for dynamic displacement measurement and modal identification of simply-supported bridges using a single camera. In this study, a series of videos are captured in separate experimental tests using a consumer-class GoPro on a long-span beam subjected to a moving load. The radial distortion of videos caused by the wide-angle lens is offset by distortion adaption, and the scale factor is fitted using a static video shooting experiment. The tracking process consists of three basic steps, namely, region of interest (ROI) selection, natural feature extraction and features tracking. The results are compared with those measured by linear variable differential transducer (LVDT), demonstrating a high accuracy by the proposed approach. Vibration characteristics, including natural frequencies and mode shapes of the bridge, are identified from the measured displacement. The overall mode shapes are obtained by combining the mode shapes identified from different segmental videos of the bridge obtained by roving the single camera along the bridge for measurements. Comparisons of the natural frequencies and mode shapes between the results from the proposed approach and LVDTs demonstrate the effectiveness and accuracy of the proposed approach for displacement measurement and modal identification of long bridges under moving loads.