Vision-based Method Research Articles

Highly Accurate Visual Method of Mars Terrain Classification for Rovers Based on Novel Image Features.

It is important for Mars exploration rovers to achieve autonomous and safe mobility over rough terrain. Terrain classification can help rovers to select a safe terrain to traverse and avoid sinking and/or damaging the vehicle. Mars terrains are often classified using visual methods. However, the accuracy of terrain classification has been less than 90% in read operations. A high-accuracy vision-based method for Mars terrain classification is presented in this paper. By analyzing Mars terrain characteristics, novel image features, including multiscale gray gradient-grade features, multiscale edges strength-grade features, multiscale frequency-domain mean amplitude features, multiscale spectrum symmetry features, and multiscale spectrum amplitude-moment features, are proposed that are specifically targeted for terrain classification. Three classifiers, K-nearest neighbor (KNN), support vector machine (SVM), and random forests (RF), are adopted to classify the terrain using the proposed features. The Mars image dataset MSLNet that was collected by the Mars Science Laboratory (MSL, Curiosity) rover is used to conduct terrain classification experiments. The resolution of Mars images in the dataset is 256 × 256. Experimental results indicate that the RF classifies Mars terrain at the highest level of accuracy of 94.66%.

A computer vision-based lifting task recognition method

Low-back musculoskeletal disorders (MSDs) are major cause of work-related injury among workers in manual material handling (MMH). Epidemiology studies show that excessive repetition is one of major risk factors of low-back MSDs. Thus, it is essential to monitor the frequency of lifting tasks for an ergonomics intervention. In the current field practice, safety practitioners need to manually observe workers to identify their lifting frequency, which is time consuming and labor intensive. In this study, we propose a method that can recognize lifting actions from videos using computer vision and deep neural networks. An open-source package OpenPose was first adopted to detect bony landmarks of human body in real time. Interpolation and scaling techniques were then applied to prevent missing points and offset different recording environments. Spatial and temporal kinematic features of human motion were then derived. These features were fed into long short-term memory networks for lifting action recognition. The results show that the F1-score of the lifting action recognition is 0.88. The proposed method has potential to monitor lifting frequency in an automated way and thus could lead to a more practical ergonomics intervention.

A Self-Supervised Metric Learning Framework for the Arising-From-Chair Assessment of Parkinsonians With Graph Convolutional Networks

The onset and progression of Parkinson’s disease (PD) gradually affect the patient’s motor functions and quality of life. The PD motor symptoms are usually assessed using the Movement Disorder Society-sponsored revision of the Unified Parkinson’s Disease Rating Scale (MDS-UPDRS). Automated MDS-UPDRS assessment has been recently required as an invaluable tool for PD diagnosis and telemedicine, especially with the recent novel coronavirus pandemic outbreak. This paper proposes a novel vision-based method for automated assessment of the arising-from-chair task, which is one of the key MDS-UPDRS components. The proposed method is based on a self-supervised metric learning scheme with a graph convolutional network (SSM-GCN). Specifically, for human skeleton sequences extracted from videos, a self-supervised intra-video quadruplet learning strategy is proposed to construct a metric learning formulation with prior knowledge, for improving the spatial-temporal representations. Afterwards, a vertex-specific convolution operation is designed to achieve effective aggregation of all skeletal joint features, where each joint or feature is weighted differently based on its relative factor of importance. Finally, a graph representation supervised mechanism is developed to maximize the potential consistency between the joint and bone information streams. Experimental results on a clinical dataset demonstrate the superiority of the proposed method over the existing sensor-based methods, with an accuracy of 70.60% and an acceptable accuracy of 98.65%. The analysis of discriminative spatial connections makes our predictions more clinically interpretable. This method can achieve reliable automated PD assessment using only easily-obtainable videos, thus providing an effective tool for real-time PD diagnosis or remote continuous monitoring.

Vision-Based Quality Assurance of Composites Printed by Extrusion-Based 3D-Printers

Additive manufacturing (AM) has been emerging as a promising production approach due to its expanded design freedoms and cost-effective batch size. Yet, its commercial acceptance is hindered by technical challenges in terms of quality assurance and defects of printed parts. Conventionally, this results in failed parts, waste of time, labor, and material. Though there have been successful attempts for process monitoring and quality assurance for single-material additive manufacturing processes, limited work has been reported for multi-material printing applications. This paper proposes a vision-based method for characterizing the material deposition accuracy of multi-material printing. More specifically, layer-wise images of a printed part are acquired, segmented, and compared with its equivalent digital reference, and position accuracy is measured by pixel differences, which provides a feasible solution for evaluating the overall performance of the multi-material AM process. An exploratory image acquiring setup using an offline camera is presented to experimentally validate the feasibility of the proposed workflow. The study concludes that detecting material deposition errors for two contrasting materials offers promising feasibility by using image processing methods and using the histogram of the image can quantify the degree of inaccuracy. This research provides a preliminary base for further research in the direction of integrating image acquisition systems with the printer, automating the image processing stage and extending the image histogram reading techniques for higher precision.

Optical flow based measurement of flow field in wave-structure interaction

Accurate measurement of the flow field in the wave-structure interaction, which is fundamental to the study of flow mechanism, is still a challenging task to date. The widely used Particle Image Velocimetry (PIV) technique is a promising approach to measure the flow field. However, it requires having the laser sheet to highlight the artificial particles seeded in the water and is only suitable for the clear water condition. It is incapable for the scenario of wave impacting on the structure, since a large number of bubbles interfere the tracking of particles. In this study, a computer vision-based method, i.e. the Optical Flow (OF) based method, is proposed and adopted to measure the flow field, and further to explore the underlying flow mechanism in wave-structure interaction. Firstly, the OF method is validated by comparing with the direct numerical simulation (DNS) and PIV measured results, involving the simple case of open-channel flow and complicated case of vortex flow. It is shown that the OF method can reasonably capture the primary flow features, as well as obtain the velocity with comparable accuracy of PIV method. Then, the OF method is used to study the flow field in the process of wave impacting on a rectangular structure. From the OF measured flow field around the structure, it is found that some typical flow phenomena, e.g. wave splash, jet flow, air entrainment, and wake vortex, exist at different phases during the wave-structure interaction. It also demonstrates that the flow acceleration and kinetic energy of wave impacting on the structure are increased with the increase of wave height due to the increase of flow velocity. Therefore, the OF method could be a useful non-intrusive approach for flow measurement in the wave-structure interaction.

Dynamic identification of crane load fall zone: A computer vision approach

Crane activities pose significant safety risks on construction sites. Thus, numerous studies have applied computer vision techniques to identify crane load fall zones or areas where the crane load could fall during an accident. However, past studies typically assumed that the fall zone directly below the load was static and had a fixed size. In reality, the fall zone is dynamic because its size changes with different crane load types and heights. Thus, this study proposes and validates a vision-based method that dynamically identifies the fall zone directly below the load. The proposed dynamic fall zone detection and tracking method first detects the key points of the load, and then the fall zone is estimated by projecting the area of the lifted load to the construction floor. Once the fall zone is detected and tracked, coupled with a previously established approach for worker detection and tracking, workers entering the fall zone can be automatically detected. Data were collected from a public housing project in Singapore to train and evaluate the proposed method. The proposed method achieved precision and recall of 0.86 and 0.83, respectively. The main contribution of this study is the development of a novel method to detect and track the crane load fall zone. Besides providing real-time safety alerts to crane operators and site supervisors, the proposed method can accumulate statistics on unsafe behaviour and situations to facilitate safety management.

A computer vision-based method to identify the international roughness index of highway pavements

The International Roughness Index (IRI) is one of the most critical parameters in the field of pavement performance management. Traditional methods for the measurement of IRI rely on expensive instrumented vehicles and well-trained professionals. The equipment and labor costs of traditional measurement methods limit the timely updates of IRI on the pavements. In this article, a novel imaging-based Deep Neural Network (DNN) model, which can use pavement photos to directly identify the IRI values, is proposed. This model proved that it is possible to use 2-dimensional (2D) images to identify the IRI other than the typically used vertical accelerations or 3-dimensional (3D) images. Due to the fast growth in photography equipment, small and convenient sports action cameras such as the GoPro Hero series are able to capture smooth videos at a high framerate with built-in electronic image stabilization systems. These significant improvements make it not only more convenient to collect high-quality 2D images, but also easier to process them than vibrations or accelerations. In the proposed method, 15% of the imaging data were randomly selected for testing and had never been touched during the training steps. The testing results showed an averaged coefficient of determination (R square) of 0.6728 and an averaged root mean square error (RMSE) of 0.50.

Automated Denudation of Oocytes.

Denudation is a technique for removal of the cumulus cell mass from oocytes in clinical intracytoplasmic sperm injection (ICSI). Manual oocyte denudation requires long training hours and stringent skills, but still suffers from low yield rate and denudation efficiency due to human fatigue and skill variations across operators. To address these limitations, this paper reports a robotic system for automated oocyte denudation. In this system, several key techniques are proposed, including a vision-based contact detection method for measuring the relative z position between the micropipette tip and the dish substrate, recognition of oocytes and the surrounding cumulus cells, automated calibration algorithm for eliminating the misalignment angle, and automated control of the flow rate based on the model of oocyte dynamics during micropipette aspiration and deposition. Experiments on mouse oocytes demonstrated that the robotic denudation system achieved a high yield rate of 97.0 ± 2.8% and denudation efficiency of 95.0 ± 0.8%. Additionally, oocytes denuded by the robotic system showed comparable fertilization rate and developmental competence compared with manual denudation. Our robotic denudation system represents one step towards the automation and standardization of ICSI procedures.

Vision-Based 40-nm-Accuracy Liquid Level Detection Compliant With Micromanipulation

Recent efforts in single-cell research call for precise liquid level detection (LLD) for end-effectors to manipulate small volume of liquid samples such as nano- to pico-liter droplets. Existing LLD methods, however, are yet satisfactory for such delicate micromanipulation tasks. This article presents a vision-based contact type LLD method, which adopts the hardware settings normally used in typical micromanipulation scenarios. The detection principle is based on the generation of patterned ripples when the end-effector (e.g., glass micropipette), that is being vibrated, moves to get contact with the liquid surface. Through the microscopic imaging system in micromanipulation, the image frames of patterned ripples such as concentric circles are recorded, from which the wavelength is extracted to determine the contact moment. The vision-based LLD method was cross-validated by the electrical impedance-based method. Experiments revealed that the best detection accuracy was 40 nm, only limited by the micromanipulator's positioning resolution. Moreover, the method was robust to various micromanipulation settings, such as illumination intensity, microscopy magnification, exposure time, and focal plane. As a demonstration, this method was successful in detecting the surface of droplets with a volume as low as 450 pL for different liquids. Given its generality, this method is expected to suit many scenarios involving liquid handling tasks in micromanipulation and industry.

Machine vision-based testing action recognition method for robotic testing of mobile application

The explosive growth and rapid version iteration of various mobile applications have brought enormous workloads to mobile application testing. Robotic testing methods can efficiently handle repetitive testing tasks, which can compensate for the accuracy of manual testing and improve the efficiency of testing work. Vision-based robotic testing identifies the types of test actions by analyzing expert test videos and generates expert imitation test cases. The mobile application expert imitation testing method uses machine learning algorithms to analyze the behavior of experts imitating test videos, generates test cases with high reliability and reusability, and drives robots to execute test cases. However, the difficulty of estimating multi-dimensional gestures in 2D images leads to complex algorithm steps, including tracking, detection, and recognition of dynamic gestures. Hence, this article focuses on the analysis and recognition of test actions in mobile application robot testing. Combined with the improved YOLOv5 algorithm and the ResNet-152 algorithm, a visual modeling method of mobile application test action based on machine vision is proposed. The precise localization of the hand is accomplished by injecting dynamic anchors, attention mechanism, and the weighted boxes fusion in the YOLOv5 algorithm. The improved algorithm recognition accuracy increased from 82.6% to 94.8%. By introducing the pyramid context awareness mechanism into the ResNet-152 algorithm, the accuracy of test action classification is improved. The accuracy of the test action classification was improved from 72.57% to 76.84%. Experiments show that this method can reduce the probability of multiple detections and missed detection of test actions, and improve the accuracy of test action recognition.

A Novel Smart Lightweight Visual Attention Model for Fine-Grained Vehicle Recognition

Vehicle Make and Model Recognition (VMMR) requires fast and accurate recognition of a vehicle’s information. Generally, the vision-based VMMR method recognizes different vehicle models that mainly rely on locating and extracting the discriminative part features of a vehicle. In this paper, we propose a Lightweight Recurrent Attention Unit (LRAU) to enhance the feature extraction ability of the standard Convolutional Neural Network (CNN) architectures for VMMR. The proposed LRAU extracts the discriminative part features by generating attention masks to locate the keypoints of a vehicle (e.g., logo, headlight). The attention mask is generated based on the feature maps received by the LRAU and the preceding attention state generated by the preceding LRAU. By adding LRAUs to receive the multi-scale feature maps generated by the standard CNN architecture, discriminative features of different scales can be efficiently extracted and combined. We conduct comprehensive experiments on three challenging VMMR datasets to evaluate the proposed VMMR models. Experimental results show our models have a stable performance under different environmental conditions. Our models achieve state-of-the-art results with 93.94% accuracy on the Stanford Cars dataset, 98.31% accuracy on the CompCars dataset, and 99.41% accuracy on the NTOU-MMR dataset. Moreover, we demonstrate that our models outperform the traditional machine learning-based VMMR models in terms of recognition accuracy and processing speed. In addition, we construct a one-stage Vehicle Detection and Fine-grained Recognition (VDFR) model by combining our LRAU with the general object detection model. Results show the proposed VDFR model can achieve excellent performance with real-time processing speed.

A vision-based method for narrow weld trajectory recognition of arc welding robots

A vision-based method for narrow weld trajectory recognition of arc welding robots

Vision-based method for semantic information extraction in construction by integrating deep learning object detection and image captioning

Recently, vision-based monitoring has been widely adopted in construction management to improve crew productivity, reduce safety risks, and facilitate site planning. However, automated retrieval of semantic information (e.g., objects, activities, and interactions between objects) from construction images remains challenging due to the complex nature of construction sites. This paper proposes a novel semantic information extraction method by integrating deep learning object detection and image captioning, which aims to explore salient information from construction images or videos. In the proposed method, object detection has been employed as an encoder to extract the feature maps of construction object zones and the holistic image. The image captioning has been selected as the decoder to extract the semantic information. A post-processing method has been proposed to parse the semantic information into a graph format for better accessibility and visualization. In experiments, the proposed method has achieved the Consensus-based Image Description Evaluation (CIDEr) of 1.84. By adopting the proposed method, semantic information behind construction images can be presented to construction managers to assist their decision-making.

A Vision-based inventory method for stacked goods in stereoscopic warehouse

A Vision-based inventory method for stacked goods in stereoscopic warehouse

Perceiving Humans: From Monocular 3D Localization to Social Distancing

Perceiving humans in the context of Intelligent Transportation Systems (ITS) often relies on multiple cameras or expensive LiDAR sensors. In this work, we present a new cost-effective vision-based method that perceives humans’ locations in 3D and their body orientation from a single image. We address the challenges related to the ill-posed monocular 3D tasks by proposing a neural network architecture that predicts confidence intervals in contrast to point estimates. Our neural network estimates human 3D body locations and their orientation with a measure of uncertainty. Our proposed solution (i) is privacy-safe, (ii) works with any fixed or moving cameras, and (iii) does not rely on ground plane estimation. We demonstrate the performance of our method with respect to three applications: locating humans in 3D, detecting social interactions, and verifying the compliance of recent safety measures due to the COVID-19 outbreak. We show that it is possible to rethink the concept of “social distancing” as a form of social interaction in contrast to a simple location-based rule. We publicly share the source code towards an open science mission.

Automated detection and analysis of piglet suckling behaviour using high-accuracy amodal instance segmentation

Piglet suckling behaviour is a critical indicator of piglet liveability and health status; however, automated detection of this important behaviour is rarely reported in literatures. In this study, we proposed a two-step computer vision-based detection method for piglet suckling behaviour. In the first step, an anchor-free deep learning network was employed in instance segmentation of individual sows and piglets. Firstly the localization head detected piglets to obtain the features of the region of interest (ROI). The features extracted from ROI were passed to a novel attention graph convolution-based structure to distil element-wise features. The distilled features were further encoded by a graph convolutional network and were fed into the boundary head and the mask head for piglet contour and mask prediction, respectively. In the second step, the piglets, in adhesion with the sow, were tracked by intersection over union (IOU) which was calculated between adjacent frames. Piglet motion features were derived from the maximum, minimum, variance, and average values in IOU sequence within the 21-frame (3-s) independent processing units. The extracted motion features were input into an SVM, classifying a piglet into suckling or nonsuckling. The dataset, for training and verifying the proposed network respectively, was composed of 100 1-min and 7-fps short video clips as well as one 8-h long video episode, from seven pens of Large White sows and piglets. Our method achieved favourable detection performance with F1 score of 93.6%, Recalls of 92.1%, and Precisions of 95.2% in short video clips, which showed that detecting suckling behaviours for piglets using amodal instance segmentation was feasible. The time budgets of at least one piglet, more than half of the piglets, or all piglets exhibiting suckling behaviour were 74.0 min, 65.6 min, and 1.1 min in an 8-h long video.

Pork primal cuts recognition method via computer vision

Pork accounts for more than 33% of global meat consumption and dominates meat consumption in China. With the improvement of people's quality of life, people pay more and more attention to the quality of pork. There are many factors that affect the quality of pork, and the cutting position of pork is also one of them. The quality of different pork primal cuts varies greatly. Aiming at the difficulty of distinguishing pork primal cuts, this study proposes a computer vision-based method to identify different pork primal cuts, using images of four different pork primal cuts (ham, loin, belly, and neck) as the experimental data, the results show that the method proposed in this paper can identify the original cuts of pork well. It also proves that computer vision technology has the potential to help people identify pork cuts.

Damage detection of structures from motion videos using high-spatial-resolution mode shapes and data fusion

Despite extensive research into damage detection based on mode shapes from vision-based methods in the past decades. There are still some notable insufficiencies, one of which is that vision-based damage detection requires a speckle pattern or mounting the high contrast markers on the surface, and the other is that damage detection accuracy in noisy environments is low, especially when detecting slight damage. In order to address these shortcomings, we proposed a high-precision damage detection strategy by combining an advanced vision-based measurement method with a signal analysis method. In terms of overcoming the requirement for a speckle pattern on the surface, a novel technique called phase-based optical flow is introduced to provide high-precision mode shapes. Then, considering the mode shape curvature is sensitive to damage feature, the Teager energy operator (TEO) together with wavelet transform (WT) to process the mode shape curvature, producing WT-TEO mode shape curvature to search for damage features in noisy environments. Finally, a data fusion algorithm based on Bayesian fusion theory is used to further eliminate the uncertain of noise interference by fusing the damage features across multi-scale space. The results of numerical and experiments demonstrate that the proposed strategy has the capability to detect single and multiple damages with high-precision in noisy environments.

A deep neural network approach for accurate 3D shape estimation of soft manipulator with vision correction

Soft manipulator is a strong non-linear and uncertain system with infinite degrees of freedom. The real-time 3D shape estimation is the guarantee for control and application, but a single kind of sensor technology always has inherent limitations. To address the above issue, firstly, the constant curvature (CC) kinematics is proposed to roughly estimate the 3D shape of the soft manipulator, because this method becomes inaccurate when the manipulator is bent with large deformation. Secondly, a vision-based high-precision shape estimation method is developed. The self-organizing map (SOM) adaptive algorithm is introduced to identify the centerline of the manipulator from the image data. While vision sensing has strict requirements for the environment. Therefore, a learning approach based on a deep neural network (DNN) is designed to correct the CC kinematics with accurate visual estimation results. Finally, the performance of the trained DNN is evaluated on the test set and a real-time bending deformation experiment. The results indicate that the DNN approach has high accuracy and stability ability to learn the entire 3D shape of the soft manipulator in real time.

3D object deforming and manipulating through dynamic hand gestures

Deformation of 3D objects is increasingly used in human–computer interaction applications and computer games. Achieving a natural user interface like hand gestures is the most important challenge in these fields. In this paper, we propose a system for 3D object deformation through hand natural gestures. The purpose of this research is to represent the whole process of hand involvement in different tasks from start to the end, describe the challenges in each step, and resolve them in proper ways. A part of the study focuses on constraints and rules of the hand. In the proposed system, we use a camera and colored gloves to create a vision-based method to track hands. Compared to similar systems, more complex scenarios of deformations and more natural interactions are provided in our proposed framework. Furthermore, a model of the hand is created with the intention of giving a better sense of hand-object interaction to users. The hand model simulates the hand movements in the desktop environment. We conducted a set of experiments to evaluate different parts of the proposed system. The results in each step show that the proposed system can provide a precise interaction and a good experience for users to deform objects.